Evidence-Based Policy Making: Some Observations of Recent Canadian Experience

David Zussman
President of the Public Policy Forum
Canada

Abstract

David Zussman talks about Canada's experience with evidence-based policy making.  He shares three examples of current policy discussions in Canada that illustrate the challenges and problems involved in trying to integrate evidence into the decision-making process.  He also describes a Canadian government initiative aimed at creating and sharing an evidential base on its own performance.  The project, "Canada's Performance 2002", provides evidence on how well or how poorly the government is meeting the objectives it has set for itself.

Introduction

As we all know, evidence-based decision making is not new.  Whether it was to build a bridge or wage a war, presidents and prime ministers have always based their plans on evidence of some sort.  But the recent increase in interest in evidence-based policy making is a response to a perception that governments need to improve the quality of their decision making in a world typified by rapid change and scarce resources.

Many critics argue that policy decisions are too often driven by inertia or short-term political pressures.  Why would any government ever develop policy, or continue to apply policy that was not based on evidence, or worse, that is contrary to available evidence?  Unfortunately, we all know that this happens regularly

Take the following examples from the social policy areas.  Evidence shows that many prisoners in Canadian prisons use drugs and share needles.  Evidence also shows that needles are the main way HIV/AIDS spreads through the prison population.  Yet, so far, we do not have a clean needle exchange programme in our prisons.  Evidence also shows that cigarettes kill more people annually than does marijuana.  Yet cigarettes are legal and marijuana is not.

How can we explain these apparent inconsistencies?  Why wouldn't a government act when the evidence is clear?  Is it because people do not know the evidence?  Don't they care?  Or are there other, deeper reasons?

In my experience, there are many reasons why governments do not always look to scientific evidence to decide what to do.

When it comes to policy making, evidence is not always as clear or consistent as we would like.  It may not even be readily available.  Policy making also characteristically requires many different kinds of evidence.

Another extremely valuable source of evidence lies in the records of any government's own transactions and actions.  Governments have always been famous for keeping huge volumes of files, but recent developments in electronic file management open huge and interesting possibilities for good governance.  The Public Policy Forum has recently published a report, Information Management to Support Evidence-Based Governance in the Electronic Age, addressing ways that new technology can be used to develop records that are authentic, reliable, accessible, understandable and useable.

In addition to this kind of hard evidence, we also get and use other kinds of evidence - softer, more intuitive and certainly more political.  This kind of evidence, while hard to quantify, is just as important as the other kind.  It might include prevailing values and accepted norms, which may appear, in the first instance, to be unrelated to the issue at hand.

For example, a country like Canada has to be careful of how our neighbours might react to policy initiatives.  What would be the impact of legalising the possession of marijuana, for example, on our ability to cross the Canada-United States border at a time when the United States government has a no-tolerance attitude toward drug use?  Many things go into policy making.  I think it was Bismarck who once said that watching policy making is like watching someone make a sausage.  If you saw what went into it, you would never eat the stuff.

Another, related, challenge is sorting out cause and effect.  Sometimes clear scientific evidence is not enough to figure out what to do.  I was struck recently by a newspaper report that medical evidence shows that men who are married tend to be happier than those who do not get married.  The policy conclusions for bachelors seem clear.  However, more recent research seems to indicate that happy men get married more often than those who are chronically unhappy.  Which way does the causality run?  Does marriage make men happy?  Or does happiness move men to get married?  It may even be true that it works both ways.

Perhaps the best way to approach the subject would be to describe three real-life policy-making exercises in Canada.  I will briefly describe each example, and draw some lessons from each.

Social policy example

Canada has a fairly well developed social security system.  This includes both unemployment insurance (a partial payment to those who temporarily lose their jobs), and welfare (a lower level of support that is given to those who are unable to support themselves).  Many of these people are single mothers with young children, for whom leaving the house to work is a major problem.  Although the level of welfare payments for single mums is low, it often amounts to more than they would get if they found a job at or near the minimum wage and had to pay childcare expenses.

Because obtaining a job means you are "off" welfare, it is argued that the current welfare system is a trap that can prevent people from entering the job market, and develops a welfare culture counterproductive to social objectives.  In 1992 the Canadian Government initiated a project aimed at testing a different approach.  It was designed to allow people to receive some level of public assistance - after they took paid employment as an incentive to get out of the trap and back into the workforce.

The programme group received assistance after they rejoined the workforce while the control group continued to function under the old rules.  After spending 10 years and several million dollars, results show  that the programme group had increased employment and earnings and used less welfare.  The average earnings of the programme group were more than 20% greater than the control group and children of the programme group did significantly better in school.  Despite this evidence, the government has yet to introduce any changes in the social welfare programme.

Environmental policy example

Canada participated in the Kyoto Conference on Climate Change in 1997.  The conference was prompted in part by a gradual consensus of scientific evidence on the impact of greenhouse gases on the atmosphere.  Going into that conference, Canada had laboriously worked out a Canadian policy of stabilising greenhouse gas emissions at the 1990 level.  But in the end, what was signed at Kyoto was a further 6% reduction from 1990 levels, to be achieved by 2010.  While we had agreed to the objective, nobody really knew how much it would cost, and what it would take to reach it.

Shortly afterwards, Canada underwent a massive internal consultative exercise involving 16 sectoral tables, 450 experts, representatives of industry and environmental groups, and academics.

In the end, almost every table filed a report on its understanding of the current situation and what would have to happen to meet the stated target.  In few cases, however, was there agreement on either the current state or on the projected costs.  In some cases, the divergences were quite significant.  There was lots of evidence - on both sides of the ledger - but in the face of the never-ending bickering among stakeholders, our Prime Minister announced last November in South Africa that Canada would ratify the Kyoto Agreement.

A sports policy example

Ice hockey is our national game.  All young Canadian boys and - increasingly - girls play ice hockey.  Canadians take professional hockey very seriously.  When I was a boy all the best hockey teams were Canadian.  Several cities in the northern United States had hockey teams, but all their players were Canadians.

In the last couple of decades, however, hockey has swept the United States.  American cities like Miami, Los Angeles and Dallas now have hockey teams.  Furthermore, they pay a lot more, so gradually they get all the best players.  Hockey is a sport, but it is also a big business, and like most businesses they frequently claim they are overtaxed and cannot compete with American rivals.  In fact, in August 2000 one owner argued fiercely that if he did not get relief within a year he would move his team to the United States.

This would have been a serious embarrassment for the government.  However, the idea of offering tax relief to millionaire hockey players or their millionaire owners is not a very attractive political proposition.  The Government asked the Public Policy Forum to try to look into the matter.

We undertook to try to find out what the evidence was.  Our interviewees included the players' association, the owners, provincial governments and local municipalities.  We asked for evidence on revenues, salaries, costs and taxes.  We explored with the different parties what might happen in different scenarios.

But in the end we found that hard evidence was hard to get and that none of the parties was willing to make a major concession to help keep professional hockey in Canada.  The hockey players wanted their salaries, the owners wanted their revenues and the municipalities wanted their taxes.  Consequently, the federal government opted to do nothing.  Interestingly, three years later the situation continues about the same and rumours still abound about teams planning to move to the United States.

Limitations of evidence-based decision making

These three examples - from social policy, environmental policy and sports policy - are quite typical examples of policy making at the federal level of government in Canada.  Policy makers are called on to recommend policy to politicians based on an estimate of what might happen in the future.  Evidence is often inconsistent and confusing but a decision has to be made.

In the social policy example, there was quite powerful evidence of a dramatic difference between the programme group and the control - a 20% increase in employment and income.  However, it had to be weighed against some other evidence - namely that the total cost to government had increased, which was contrary to one of the stated objectives of the government.  More confusingly, the longer-term evidence was much less clear.  It revealed the dramatic difference existed only as long as there was a subsidy.  After participants were weaned off the supplement, differences dissipated almost completely.  There were no identifiable long-term effects.  Putting this starkly, we can say that after waiting 10 years and spending $5 million dollars on evaluation, the government ends up with a handful of conflicting evidence.

It raises one important limitation with respect to evidence-based decision making.  Governments are under constant pressure to solve problems, and characteristically do not have the time to study things as much as they want.  Consequently, they are often required to make a policy decision based on an informed guess today instead of waiting for a more complete answer.

In the Kyoto example, policy makers were dealing with two quite different kinds of evidence.  On the one hand, we had quite a bit of scientific evidence about greenhouse gases and climate change.  The science is not unanimous, but there is clearly a growing consensus that greenhouse gases do indeed contribute to global warming.  Conversely, we had almost no evidence about what we would have to do to reach our goals, or about what the consequences of various options would be for industries, provinces or consumers.

What we got was a lot of informed, but contradictory, speculation about the potential impacts of different approaches.  Predictably, environmental groups thought the economic impacts would be minimal and that business and the economy could continue as usual.  From the business side, in particular from the energy sector, the view was that the costs would be horrendous, and our competitive situation would be adversely impacted.  They all had evidence to back up their case, based on explicit or implicit assumptions, and a good deal of scientific analysis, but most of it of necessity was speculation.

This leads me to my second observation about the limitations of evidence-based policy making.  Many policy initiatives require assumptions about the future that are fundamentally untestable.  Policy makers have to carefully weigh predictions about future impacts against the credibility of the sources of the information and their own common sense.

Finally, in the sports example other problems surfaced.  Many participants were willing to share evidence with us privately, but for competitive reasons were not willing to have such information made public.  Without hiring a team of auditors, the policy makers had to rely on the assertions of interested parties.  We do not think that anyone misrepresented their situation, but there are often different ways to interpret financial statements.  Some of the teams produced evidence to show that without tax relief they would incur unsustainable losses and would be forced to leave the country.  As it turns out, this has not happened yet, because the teams are still operative.  They might have been wrong or, they might have been crying wolf.  Or they might have been right all along but wrong on the timeframe.

This leads me to a third observation about evidence-based policy making.  Decision makers in government are often aware of information which, for various reasons, cannot be made public and which is therefore not contestable.  Some of this evidence is real and some is hype, leaving policy makers to use their own judgement to sort out which is which.

Implications for decision making

There are at least six implications for successful decision making.  Decision makers need to:

• promote an environment in which public servants are encouraged to lay the facts on the table, even if they do not squarely support the government's policy or intentions;
• become consumers of research, not relying solely on the work of their own staff, however capable;
• make more information available to the public, including outcome measures;
• ask for more systematic and rigorous programme evaluations;
• commit to long-term measures of success; and
• be humble - not every public intervention will be successful.  Accept policy failures, learn from them, and move on.

Policy makers in turn need to:

• recognise the importance of taking an interdisciplinary approach to policy making;
• involve as many stakeholders as possible in the development of policy;
• recruit policy analysts from a wide range of skills, including technical and managerial skills;
• specify objectives and measure results, however awkward that can be at times; and
• involve outside as well as inside experts.

As decision makers, the Canadian government is involved with the project "Canada's Performance 2002", which has revolutionised the way we measure the government's achievement of objectives.  The project makes public, on an annual basis, data on indicators that reflect Canada's economic and social objectives.

The initiative owes some intellectual debt to the original thinking done in New Zealand a decade or so ago around the identification of outputs and outcomes as measures of government performance.  Prior to that, many governments had been much more focused on inputs than outputs, and few had ever thought about separating outputs from outcomes.

Today, every Canadian department or agency has to publicly identify specific outcomes and indicators by which its performance can be judged.  The Canada Performance 2002 project goes a step further.

It aims to create an evidential base for the country's performance as a whole, and make it public - primarily for members of Parliament and the Canadian population, but beyond that for the whole world via the internet.  An initial step was to consult with parliamentarians and Canadians from all walks of life on the list of indicators that should be used.  The advice regarding the indicators was consistent.  Information must be:

• relevant - indicators must reflect Canadian values;
• temporal - data must highlight trends over time and show progress toward goals;
• available - data must be easily accessible;
• comparable - it must be possible to compare with data from other countries; and
• understandable - data must be easily grasped by various audiences.

In the end, 19 indicators were chosen, grouped according to four main themes:

• economic opportunities and innovation in Canada;
• the health of Canadians;
• the Canadian environment; and
• the strength and safety of Canadian communities.

For each of these indicators, there is a summary of the issue, a brief trend analysis and a link to a summary report on Canada's current performance, as well as links to the relevant government programmes.

Canada's Performance 2002 has three simple objectives: to build trust in government, improve transparency and accountability to parliamentarians, and encourage a focus on results.  As the Canada Performance 2002 initiative is only two years old, it will take a few years before we know whether it has made a difference.  Nevertheless, it is an interesting and innovative approach to enhancing evidence-based policy making. 

I strongly believe that good policy capacity requires the research and analytical skills to seek out and analyse information with respect to various policy options.  However, in the end, policy making will always be as much art as science.  Judgement will always come into play when, as is so often the case, the data are unclear, inconsistent, lacking, costly to obtain, too long in coming, or from unreliable sources.

Good policy capacity requires the capacity to understand the limitations of the evidence and to make reasonable inferences based on incomplete information.  It also requires the ability to integrate that scientific kind of evidence with the soft but very powerful evidence that members and ministers get every day as they knock on doors, attend Rotary Club events and eat dinners in church basements.

In the final analysis, Canada, like the United Kingdom, Australia and New Zealand, is a democracy.  Governments, ministers and members of Parliament have to answer to their constituencies for the policies they adopt, and it is our obligation to provide the evidence in a format and manner that decision makers can understand and use effectively.