From Research to Social Policy and Back Again: Translating Scholarship into Practice through the Starry Eyes of a Sometimes Scarred Veteran

David T. Ellwood
Professor of Political Economy
John F Kennedy School of Government
Harvard University

Abstract

Professor Ellwood shares his unique perspective of being both a highly regarded academic and a former senior policy maker in the Clinton administration. He describes how three important research ideas and findings have influenced the poverty policy debate in the United States. First he illustrates the key findings, and explores how and why they influenced policy. Next, he discusses how the contrasting cultures of policy makers and scholars create inevitable divisions, but also opportunities for mutual benefit and connection. Finally, Professor Ellwood discusses specific measures that could improve the connection process.

Introduction

This conference is about both substance and process, and so I find myself in a quandary. For most of my adult life I have been a scholar of poverty and social policy. As a younger professor I worked with policy makers on a regular basis, but the experience was often frustrating, as these policy makers often ignored what I perceived to be rich scholarly insights and bold new initiatives. Then for approximately three years, I joined the Clinton administration and was part of a leadership troika for welfare reform. We tried to use academics, many of them close friends and colleagues, but often, we found them out-of-touch and calling on us to move in a completely different direction than our president had promised during the election. So I have felt the schizophrenic impulses of living in two worlds.

Since much of my life I have been a scholar, my natural inclination is to discuss critical scholarly ideas of policy relevance. But since I have also had the rare privilege of also wearing a policy hat, I have some desire to talk instead about how connections between social science research and policy succeed and fail.

In this paper, I first discuss how the contrasting cultures of policy makers and scholars create inevitable divisions, but also opportunities for mutual benefit. I describe how three important research ideas and findings have influenced the debate on poverty policy in the United States. They are the dynamics of poverty and welfare, incentives and work, and randomised control evaluation. I seek both to illustrate the key findings and to explore how and why these research findings influenced policy. I try to generalise from these examples to show how and when policy and research can connect successfully. Finally, I discuss specific measures that I think could improve the connection process - some of which are already under way in New Zealand.

Let me preface my comments by noting that throughout this paper I am treating policy makers as one homogeneous group and scholars as another. Of course, policy is usually developed by professional staff working closely for political leadership. I treat the policy maker as one entity, with a political orientation. So I speak of political policy makers. In fact, in many nations the professional civil servants may have superb scholarly training, and may work very hard to keep up with the latest developments in research. For my purposes it is easier to separate the political from the research world, and so the policy maker is placed in the former. For those who regard themselves as research oriented, politically neutral policy makers, I offer my apologies for this oversimplification.

Contrasting cultures and essential roles: policy makers and scholars

The relationship between policy makers and research scholars is inevitably awkward and imperfect because each exists in very different cultures.

• Ideology versus Truth: First and foremost, political policy makers live in a world where values and ideology are central. As the public cannot possibly master the details of even a single piece of legislation, they elect representatives, as much for their values as for their specific policy positions. The competition of ideologies is at the very heart of democracy. Scholars by contrast are supposed to seek out "truth" using the models and methods of academic disciplines. They are suspicious of ideologically driven findings, even to the point where scholars who become seen as ideological often lose their academic credibility
• Political Allies/Constituencies versus Independence: Politicians rely heavily on building constituencies and political allies. They pay particular attention to the interest and concerns of those who are most loyal to the party and its candidates. Scholars value independence above all else. It is the very basis for tenure - to protect scholars from the pressures of outside interest groups. Any involvement with particular constituencies, particularly the politically or financially powerful, smacks of conflict of interest.
• New Policies versus New Ideas: Success for the aspiring policymaker is the creation of a new policy. But to most scholars it is ideas that really matter. Scholars fantasise about creating a new idea or insight that fundamentally reshapes the way in which people explore issues. Sure they want to influence policy - but through their ideas. Politicians want to get a law passed or a policy adopted.
• Timing is Everything versus Deliberation: Anyone in government knows that timing is everything. Issues become ripe or hot at rare and opportune moments. These moments are often brought on be unforeseen external events - a dramatic news story or a seemingly catastrophic (if isolated) policy failure. That magic moment requires immediate action if a policy is to be enacted. Yet scholars don't even seem to have timeliness as a value, as any conference organiser seeking papers to be deliver by a particular date can tell you. Truth and good ideas cannot be rushed. The data and findings are ready when it is ready. To the scholar, more study is always necessary.
• Simplicity and Certainty versus Rich Complexity and Acknowledged Limitations: Politicians must explain and sell their proposals. They also find that any doubts or ambiguity they voice about a path that has been chosen is immediately seized by the press and exploited by opponents. Thus both simplicity and certainty are the hallmarks of successful policy proposals. Truth is rarely simple and results always ambiguous. The academic seeks to carefully delineate the limitations of his or her findings, and emphasises the richness of the issue and the findings.
• Compromise versus Principle: Governance always involves compromise. It involves recognising the political realities, protecting key constituencies, selling the idea and that means accepting messy deals. Scholars stand on principle above all else.
• Implementation versus Uninteresting Details: Ironically even though scholars thrive on complexity, they are rarely interested in the operational details of policy design. But implementation is often far more important than the big idea or the legislative language. Few scholars have any interest or patience with practice. Ironically though, few law-makers actually spend much time thinking about implementation either. Arguably the failure to think hard about implementation is one of the greatest examples of the limits of government and provide an opportunity for collaboration.

Thus, it is not surprising that scholars see politicians and policy makers as ideologically driven, "oversimplifiers", often more interested in their constituencies than in doing the right thing, who often rush to judgements that lack a solid intellectual basis. Politicians see academics as "complexifiers", isolated in their own world, with little interest for the strategy and compromises central to the adoption of new policy. Indeed, perhaps the miracle is that the groups can work together at all!

Experience and trust often makes the situation better. Even so, a close relationship carries dangers to both sides. A politician who relies too heavily on a scholar will often find that scholar is more loyal to his or her ideas than to the politician, and when inevitable compromises are made, the scholar may abandon and damage the efforts of the politician. Conversely, a scholar who gets too close to a particular government or official will often be seen as subjugating his or her principles to maintain influence with the party or politician. As someone who has moved back and forth between these worlds, something that is rare nearly everywhere, the differences can be frustrating. As a scholar who joined an administration at a senior level, I found most unpleasant the need for secrecy, the call to publicly deny essentially all the uncertainties and risks inherent in any particular proposal, and the requirement that one must strongly defend in public, policies that one adamantly opposed in private deliberations.

My struggle to resolve this contradiction, as both a scholar and policy maker, was illustrated by a conversation I had early on with a friend and senior political advisor within the administration.

"David, you realise, of course, that when the final deal is cut on welfare reform, you will not be in the room."

"What? I am one of the three people in charge of welfare reform."

"Yes, but in the end, you care more about poor people than you do about Bill Clinton and the political future of this party."

Although this comment irked me at the time and smacked of cynical political expediency, the longer I stayed in government, the more I understood the logic. On those occasions when I was quite candid with the press, allowing that we were not completely certain how things would work out or acknowledging honest disagreements within the administration, I was rewarded with a front-page story selectively using my words to suggest disarray and dissention within the administration. The White House was not pleased.

And while I did call on scholarly colleagues occasionally, seeking help on problems as they arose, they often seemed to have incredibly creative answers to the wrong question. They wanted to talk about the big picture, whether the Clinton vision was the right one, not the specifics of the policy, such as a particular time limit. Most had little time or interest in doing a quick and dirty analysis when the data were weak and the impact of the work uncertain. They seemed far removed from the issues and minutiae that occupy most days for a policy maker. And even if their aid was enlisted, one would subsequently endure public criticisms from them about the plan that emerged.

Essential roles: values and means

In spite of these contrasting cultures, I believe good policy cannot be achieved unless politicians and researchers work together while occupying different roles. Again and again I find that without a clear sense of values and goals, policy development flounders. During the administration of Jimmy Carter two visions for social policy reform competed: one group, based in the Department of Health, Education and Welfare (HEW, now HHS) advocated a guaranteed annual income (a negative income tax). A second group wanted a work-based reform with increased work requirements and guaranteed government jobs. Carter had never been clear in articulating his own vision, and so for months duelling "micro-simulation models" predicting work, poverty and costs were used in an attempt to prove that one policy or another was superior. But value conflicts are rarely resolved with data and the effort itself ultimately became mired in the competing visions.

In a democracy the vision and values must be entrusted to the political process. The notion that people who work should not be poor can in theory be derived from philosophical reasoning, but it is not a testable or disprovable proposition in the sense that social scientists use the terms. Similarly, the notion that it is both reasonable and appropriate to require lone parents eventually to enrol in training or to go to work outside the home is fundamentally about values. Many people argue, quite legitimately, that the most appropriate thing a society can do is to nurture its children and that since parents are the best nurturers, a nation ought to be willing to provide sufficient support to allow lone parents (and married parents as well) to stay home with their children. Social science can offer some insights into whether children fare better when their mother does or does not work (answer: it's a complicated problem!), but ultimately this is a question of values. And as more and more married mothers began working in the labour market, and as the reasons for single-parenthood changed, so too did public and political attitudes about this question in the United States.

Scholars often chafe at what they see as the cynical, narrow and exploitive ways in which politicians describe social problems and social policies. And the public is often badly informed - believing that public assistance is a far bigger share of the budget than it is, or that minorities receive the bulk of aid. A critical role for scholars and policy makers is to ensure that information that is more accurate is available. Yet over the years I have been struck by the realisation that even when confronted with the basic facts, the values regarding work and responsibility in different nations at particular times reflect something far deeper than limited information. These involve what people want and hope and expect their nation to reflect. I recognise increasingly that when I am particularly frustrated with political processes, I am arguing that my enlightened and informed values are somehow superior to those of the public, a disturbingly élitist position. As an economist, I once saw politics as a constraint - preventing the adoption of the "right" policy. In fact, in its awkward and sometimes misguided manner, politics is the way in which the public conveys its sense of values and priorities - to an economist, its social welfare function. Scholars can and should try to shape the values and ideals of policy makers. Indeed, one of the critical roles of scholarship is to frame the way in which people think about a problem, its causes and its consequences.

Still, in a democracy, the value-laden decisions are the responsibility of the polity. When and perhaps only when there are clear values and goals, research and scholarship represent essential tools for finding the most effective ways to achieve those ends.

Scholars can be far better than policy makers at understanding behaviour, examining the long-term causes of deprivation or advancement, determining which programme will actually achieve the desired effect, recognising potentially unintended consequences, and considering "what if" scenarios. Scholars will still disagree among themselves about the best and most effective means to ends, but this is a ground where scholars can productively argue using their various methods of research.

I am not arguing for a complete divorce in roles, for both are deeply interconnected. Indeed, scholars can change how a problem is viewed and the values associated with policy, and lawmakers often have better insights into how to move bureaucracy or constituents than scholars do. I am arguing, however, that each party has a comparative advantage in some realm. When people operate in an atmosphere of mutual respect, there are enormous gains from trade.

Let me now turn to three examples where I believe research played a critical role in policy development, albeit not always an entirely positive one, to help illustrate the opportunities and limits to connection research and policy.

Three critical research insights for poverty policy

By the late 1980s political support had nearly collapsed for the existing programmes of public assistance designed to aid lone parents and their children in the United States. Unlike many other industrialised nations, in the United States social assistance programmes have always been conditioned on evidence that the recipient is either unable or not expected to work. Our "Social Security" system provides aid to experienced workers who reach retirement age or who have become disabled and unable to work. Unemployment insurance offers time-limited support to experienced workers who lose a job (usually for no more than 26 weeks). Federally supported means-tested cash assistance has generally been provided only to single parents, or people who are beyond retirement age, or disabled. In fact, the only universal programme in the United States for low-income families is "food stamps", designed to provide enough food coupons to ensure that everyone can afford a minimally nutritious diet.

The most controversial of these programmes provided means-tested cash support for lone parents. Aid to Families with Dependent Children (AFDC) was unpopular even when it was designed during the depression in the 1930s, even though at that time most single parents were widows, few married mothers worked, and jobs were scarce. By the late 1980s, however, most AFDC recipients were never-married mothers, and most married mothers worked. Long-term cash aid for lone parents seemed much harder to justify. And in most states, AFDC payments were remarkably low, and recipients disliked the programme nearly as much as its opponents did. The expression "welfare", used almost exclusively as a derogatory term, came to be associated primarily with this programme.

States began, with federal approval, to experiment with changes. Calls for "welfare reform" reached a crescendo with the election of Bill Clinton, who campaigned for an end to "welfare as we know it", a position previously unheard of for a Democratic presidential candidate.

In the design of Clinton's original plan, as well as the design of the final welfare reform bill, scholarship played an important, if not entirely constructive, role. I will focus on three specific issues where scholarship was most relevant and which provide lessons that are broadly applicable to poverty policy in many nations.

The dynamics of poverty and welfare

In the 1960s several visionary scholars, foundations and government officials saw the need to begin collecting longitudinal data. The availability of this new type of data, which tracked people over extended periods, shifted the focus of poverty research and ultimately the nature of policy.

Cross-sectional data only really allow one to ask, "Who are the poor?" or "Who are the recipients of public aid?" These questions and their answers lead mostly to static types of policies. The policy questions become, "Who should receive aid?" and "How much should they get?" Longitudinal data allow one to ask a very different sort of question about poverty and benefit receipt: "How long does it last?" and "What events lead people into and out of need?" And by changing the questions, ultimately policy makers were moved to look for rather different sorts of answers.

The data revealed that poverty and public assistance have two sides. Most of those who were just becoming needy for the first time and most of those who were just beginning aid for the first time had relatively short episodes. Thus, most of those who ever had an episode of poverty or public assistance receipt moved off assistance fairly quickly. Yet, seemingly in contradiction, most of those poor at a point in time or receiving aid at a given moment were in the midst of a much longer period of need. Table 1 illustrates this point.

Table 1 Distribution of Time in Spell and Total Time On Public Assistance Among Single Parents in the United States

Source: Bane (1994).

The first column shows that nearly half (48.9%) of women beginning a spell of AFDC will end that spell within two years, and a tiny fraction (14%) will remain on welfare for 10 years or more. Of course people may have multiple spells. If one looks at the total time on aid that women just starting a first spell will accumulate over their lifetimes, the numbers are somewhat larger. The second column shows that 36% of newly enrolled women will receive aid for less than two years, and 22% for 10 or more years. Still, from these first two columns one would be inclined to conclude that welfare is generally a short-lived event. The picture changes quite dramatically if one asks a slightly different question: Of all the women found on AFDC at any point in time, what is the distribution of total time they will receive aid? Only 8.5% of current recipients are in the midst of lifetime episodes of two years or less, and a remarkable 56.6% will remain on welfare for 10 years or more.

All of the columns of this table are based on exactly the same data and they are all completely consistent with each other. How can welfare appear to be both a short-term and a long-term phenomenon? A hospital analogy illustrates these seemingly contradictory findings. Suppose one were trying to characterise the duration of hospital stays. One might go to the admitting ward and find that the vast majority of patients were entering with a short-term acute problem. Only a few would be entering with a condition requiring long-term hospitalisation. Thus, one might reasonably conclude that hospitals mainly treat short-term acute episodes. The same is true for poverty and welfare - most episodes are short.

However, if one got out of the admitting ward and walked around and looked at the patients in the beds, one might be shocked to discover that the overwhelming majority of patients in the hospital at any point in time were in the midst of long-term stays with a chronic condition. The reason is that the long-term patients accumulate. One patient who stays for 52 weeks uses as many bed days as 52 patients who stay for just one week. The small proportion of entering patients who stayed long term account for the overwhelming majority of bed days (and hospital resources).

And so, on AFDC, even though they were a small minority of new entrants, at any point in time, the long-term users represented a majority of current recipients and the majority of funds spent over any given period went to providing their benefits.

Dynamic policy implications

This information on the on-going use of welfare assistance had at least three significant policy implications.

Firstly, one could no longer claim, as some liberals had for many years, that welfare was almost exclusively a short-term, transitional source of support. Most of the money actually went to long-term recipients. Alternatively, conservatives had to admit that welfare was not the "trap" that they often described, since the majority of those who had ever gotten assistance used it only briefly. Now both sides could agree that long-term receipt existed and was troubling, but they also could agree that welfare assistance often met short-term needs. Because of this research, policy makers and scholars alike tended to increasingly focus on ways of reducing long-term welfare use.

Secondly, the study of welfare dynamics led scholars to examine not just who was poor, but how recipients moved into and out of poverty. In general, dynamic analyses are driven by beginnings and endings - changes in status. This perspective gives more insight into the "why" of neediness. And it tends to direct policy makers and researchers to think more about mechanisms to dampen movements into poverty and facilitate movements out of poverty rather than the allocation of public assistance.

One striking finding from this research was the realisation that changes in family structure - marriage, divorce, childbearing - played a large role in explaining entry into and out of poverty and public assistance. Work seemed less important than people expected, especially for "permanent" exits. Since no one knew how to change family structure patterns, this finding reinforced a concern with limited work and low pay. Slogans like "If you work, you shouldn't be poor" and "Make work pay" evolved with the recognition that many working families remained poor. The finding also led to a greater sense by both researchers and policy makers that escaping welfare and poverty required encouraging or requiring recipients to work more, while also raising financial supports for working families.

Finally, the focus on welfare dynamics also led to change in the focus of policy itself: it too became oriented toward dynamics and durations. The question asked by policy makers evolved from how much money should the nation provide, to what should happen when a recipient first enters the door, and how should the programme change as time on the programme increases?

By the mid- to late 1980s states began experimenting with plans that included gradually escalating participation requirements - in either education or work. In 1992, campaigning for a "hand-up" and not a "hand-out", Clinton called for a system in which significant support for education and training was available at first, but after two years, most recipients would be required to work "preferably in a private unsubsidised job, but in a subsidised private or public job if necessary".

Limits and lessons of dynamic analysis

Dynamic analysis proved highly influential in policy making in the United States and it appears to be influencing policy in other nations. As an active participant in both the development of these dynamic models and as a policy maker who used them in developing policy, I am acutely aware of how this analysis was used to aid policy making, but also the ways in which it was ignored or misused.

First, dynamics and duration analysis alone cannot guide policy unless accompanied by a set of values regarding support and responsibility.

There is no inherent reason why providing aid to recipients for 10 years should be troublesome. In fact, the United States Social Security programme, which provides long-term income support to retirees and disabled workers, is one of the most popular and least controversial of any federal programmes. It is only in a cultural and economic context where people are expected to provide for themselves that long durations of support become seen as dependency.

A second concern is that the rich and important nuances of research are lost in the translation to policy. One of the most important lessons of dynamic research is the extreme heterogeneity of the population. People's lives and loves are complicated and filled with unexpected setbacks and successes. But in a political world of sound bites and short attention spans, simple tends to win out.

Clinton campaigned on the welfare reform slogans "two years and you are off" and an "end to welfare as we know it". The two-year time frame was almost completely arbitrary - driven by his values and political sense. When our team began to design the programme, we sought to build in flexibility by creating a two-year time clock that stopped and started depending on the recipient's situation. But when the Republicans took control of the agenda they adopted a far simpler system: states were required to impose simple time limits, preferably of a few years, but no longer than five years. There was no carefully calibrated clock. In many states the policy really became "two years and you are off".

"Two years and you are off" had another disturbing feature. Clinton was actually proposing "two years and you work". Anyone who understood the dynamic research on durations had to believe that a literal two-year time limit followed by the cessation of aid would cause significant hardship. After all, less than 10% of current recipients had total durations of two years or less. Clinton's original notion was that at some point recipients would be expected to work and a job would be provided, if necessary. But apparently, political worries about the suggestion of large-scale government job creation led to a far more dangerous sound bite. Consequently, the adopted policy in a Republican Congress ultimately became five years (or less) and off. No provision was made for any sort of assured job or work, regardless of the state of the economy. And the weakened Clinton was not willing to fight for his original vision of a job guarantee.

Incentives and work

There was a period in the United States, at least, where economic incentives were widely discussed and tested in social policies. When a guaranteed annual income was proposed, economists busily set about estimating the effect of various guarantees and taper rates on work and costs. But the energy appeared to dissipate as painful trade-offs between targeting, work incentives, and costs seemed to create an "iron triangle", whereby no practical way could be found to generously support the poor, provide them with strong work incentives, and keep the budget manageable.

Ironically, discussion of incentives was not central to the welfare reform debates of the mid-1990s in the United States. But the effect of the policy choices was to sharply alter incentives, nonetheless. And the subsequent changes in work behaviour have re-ignited an interest in strategies for improving work incentives in several nations.

Policy Impact of Incentives

The Clinton agenda to make work pay was driven in part by a desire to increase work incentives, but it rested heavily on the philosophical proposition that working families should not be poor. One of the first actions of the Clinton administration was to increase tax credits sharply for the working poor - the Earned Income Tax Credit (EITC).

The administration also fought to sharply raise medical and childcare support for the working poor. These policies resulted in a remarkable expansion in aid to working families. In real terms, federal spending on programmes supporting working poor families grew from roughly $11 billion in 1988 to $70 billion in 1999. The amount spent in real terms on the EITC alone in 1999 was greater than federal expenditures for AFDC in any past year.

When AFDC reform finally passed, 3½ years into the administration, states were given block grants and instructed to focus heavily on moving people quickly from welfare into work. And so states began to reduce their support for non-workers, while expanding aid for working families.

Thus, the true story of welfare reform in the United States was not a cut in expenditures. Expenditures actually increased considerably, with an expanded earned income tax credit, as well as more childcare and medical support for working families. Instead, the policy represented a sizeable shift of resources, with considerably more support for working families and less aid for non-working ones. Those changes signalled a serious shift in work incentives.

To illustrate this shift, in 1988 a lone mother moving into a full-time minimum-wage job would see an increase in income of just $2,325, and that only if she had low childcare expenses. She would probably lose her health insurance coverage as well, something probably worth roughly $2,000 to her. In sum, there was little financial incentive to work at all. By 1999, however, the financial gain to working for a single mother formerly on welfare was over $7,000 and her children would retain their medical coverage. Moreover, she would receive strong administrative pressure to leave welfare for work, and ultimately might be denied aid altogether if she failed to find work quickly. Thus, there was far more pressure to work and far greater rewards for doing so.

And in keeping with expectations, given the shifting incentive, what we observed after the reform of AFDC was a remarkable surge in work by less-educated single parents. For almost two decades the percentage of working single mothers in the lowest education levels hovered at about 35%. Then, in the mid- to late 1990s it suddenly shot up to 55%. The move to work was no doubt greatly aided by the exceptionally strong economy during the latter half of the 1990s. Still, this unprecedented change disproportionately reflects changes in policy.

Limits and Lessons from Modelling Incentives

While scholars often wanted to talk of incentives, lawmakers were focused on sending a clear message that work was expected, even demanded, and that workers would be rewarded. They wanted to send a clear message about work. The programme was renamed: Transitional Assistance for Needy Families (TANF) to emphasise its short-term transitional nature. And far from emphasising work incentives in the usual economic sense, policy makers focused instead on administrative mandates: people were required to work or they would lose benefits.

States took up the challenge with relish. Long before legislation was actually passed, administrators began to signal a new message to caseworkers: get people off welfare and into work.

Most advocates and scholars were appalled by the heavy-handedness of the policies and the mean-spiritedness of some of the rhetoric and the policies. Many predicted massive increases in poverty. But a remarkable thing happened. Work increased dramatically. And poverty - at least as measured in the United States - fell sharply. The messages signalled by the administration and the Congress and thereafter appear to have changed the whole culture of welfare offices. Welfare seems no longer to be about cheque writing and benefits. It is about work.

There are plenty of reasons to be fearful of the longer-term implications of this policy. With a recession upon us we must question: what happens when you force people to work but there is no work available?

Politicians with their preference for powerful messages and uncomplicated policies may be more effective in achieving a particular outcome - though their failure to properly consider the risk of a recession was a serious flaw. There is perhaps no better reason for researchers and policy makers to collaborate.

As one whose home is in academia, from this example I take away the lesson that scholars need to think as systematically about administrative incentives and the ways in which recipients are treated in welfare offices as about financial incentives. But there is another critical lesson: the language of politics, even with its focus on sound bites and ideology, has a powerful role to play in designing effective policy.

Randomised control evaluation of welfare-to-work plans

Controlled evaluations where subjects are randomly assigned to the treatment and control groups are currently the almost universal standard for evaluating the effectiveness of new drugs. More recently, in social science these methods have been gaining favour in the evaluation of social policy programmes.

AFDC has historically been a shared federal and state programme. During the 1980s and early 1990s states increasingly sought to innovate by using a special provision in the federal AFDC legislation that allowed for waivers of key federal provisions of the act for any "experimental, pilot, or demonstration projects that the Secretary [judged] likely to help in promoting the objectives of the programme." With remarkable foresight, federal bureaucrats chose to be relatively permissive in the granting of waivers provided that the state used a randomised controlled experiment to evaluate the results of their proposed programme.

The effect of this policy was a virtual explosion in knowledge about what programmes seemed to be effective. The striking feature of well-constructed randomised controlled experiments is that their results are almost always accepted as accurate by nearly all sides of the debate. This stands in sharp contrast to evaluations conducted using any other method, no matter how sophisticated. Critics could always find something to legitimately criticise in these evaluations and thus discount the findings.

Policy Implications of Evaluation

Nowhere is the potential for reliable evaluations to influence public policy more evident than in the case of so-called human capital versus work-first welfare-to-work strategies. Conventional wisdom had always been that training and education were essential stepping-stones for helping recipients gain employment. A few sceptics claimed, however, that most education and training programmes available were so superficial and of such poor quality that they provided few real skills. Moreover, many welfare recipients had poor skills precisely because traditional learning methods had not worked well. And finally, the sceptics argued, even if one advocated education or training, it was better placed after people had worked for a while, since recipients would be far more motivated to learn as they discovered they needed education to advance.

In addition, there was considerable ongoing debate over whether programmes could be voluntary or needed to be mandatory. A plethora of welfare-to-work programmes were evaluated, some voluntary, some mandatory; some explicitly education focused, some more employment focused. Some of the most recently available results are summarised below (Hamilton 2002).

Figure 1 compares earnings of treatments and controls in various settings. Thus, the five-year earnings of recipients in a job-search-first, high-enforcement (i.e. mandatory) programme were $19,866 as compared to $17,680 for the randomly assigned control group; the programme raised earnings by $2,187, on average. By contrast, an education-focused programme with low enforcement of participation raised earnings by just $788. Though these results are taken from a 2002 report, much of this information was beginning to be available in the early to mid-1990s when the Clinton administration's welfare reform legislation was being drafted.

What seemed very clear to policy makers from the evaluation literature was that mandatory programmes were more effective than voluntary or "low-enforcement" programmes. More importantly, over five years employment-focused programmes were more successful in increasing earnings than were programmes oriented towards human capital development. Hamilton (2002) concludes:

In side-by-side comparisons in the same sites, the LFA [labor force attachment] and HCD [human capital development] approaches' five-year impacts on employment, earnings, months on welfare, and welfare payments were not the same, but the differences were generally not statistically significant - that is, it could not be confidently concluded that the differences in impacts did not occur by chance. Where there were statistically significant differences between the effects of the two types of programs, however - such differences were found for some early follow-up years and for some subgroups and outcomes - the LFA programs always came out ahead.

The impact of these findings on welfare reform was dramatic. Even advocates of education found themselves hard pressed to dismiss the emerging set of consistent findings. The legislation that finally passed focused nearly exclusively on pushing people to go to work immediately. Education and training activities were downplayed. States adopted strict "work-first" programmes.

Limits and Lessons from Evaluation

This very strong emphasis on work, even to the exclusion of training, probably had an important role in changing the expectations of both welfare recipients and programme administrators.

Preliminary evaluations of the results of welfare reform were generally positive. Still, as was the case with the welfare dynamics example, the tendency for the political process to excessively distil and oversimplify the lessons from research was evident. The results primarily revealed that disembodied training programmes - programmes focused exclusively on education without a clear connection to employment and specific jobs - fared poorly. Moreover, the level of training involved was often quite modest, likely either lasting 13 weeks, or designed to give people a high school equivalency.

As the results above suggest, and as findings continue to filter in, a more nuanced interpretation makes more sense. In fact, the most successful programmes seem to combine a strong employment focus with some job-specific education and training. A programme in Portland, Oregon, was among the most successful. Hamilton concludes:

The Portland program's success suggests that the following are key features of very effective programs: an employment focus, the use of both job search and short-term education or training, and an emphasis on holding out for a good job.

But the rigid application of "work-first" often led states to deemphasise training, even when it was closely and effectively connected to work. Currently, the United States is actively renewing the welfare reform legislation. These more nuanced results are now much more widely available, providing an interesting test of whether more refined evaluation can reshape policy. Unfortunately, early indications are not promising. Still, it should be emphasised that millions of recipients have moved from welfare to work, far more than most critics of the final legislation expected, including me. Evaluations gave real credibility to a set of highly work-oriented strategies that probably would have lacked this credibility otherwise.

Specific gains from trade

The three examples of research influencing policy cited above each illustrate one broad mechanism whereby research can help in the formulation of effective and thoughtful social policy: framing the issue, understanding and modelling behaviour, and evaluating effectiveness.

Framing the Issue

Probably the most fundamental way in which scholarship influences policy is by creating the intellectual framework for analysing the problem.

When William Julius Wilson (1987) described concentrated urban poverty, identifying the complex interaction between economic decline, deprivation, isolation and culture, he provided a way to think about ghetto poverty that quickly spread into both academic and political circles. Although Wilson's work was rather light on policy prescription, he reshaped the dialogue, the analysis and ultimately policies surrounding poverty.

Using dynamics to examine durations of poverty and the use of public assistance had some of the same effect. By illustrating that welfare had both a long and short-term element, this research pointed to a new understanding of public aid and the attendant policy concerns. Moreover, the dynamic approach helped shift policy toward support for the working poor and strategies to move people into work.

To many scholars, coming up with an idea that is powerful enough to shift both the intellectual and the policy dialogue is the ultimate dream - evidence of real impact. But scholars whose ideas do help shape the debate often become quite controversial - precisely because they are influential. And these ideas and findings are often used in ways far afield from the original conception. In some cases, the research itself is fundamentally misunderstood or misused. More commonly, those on either side of an ideological or political divide choose to use the ideas selectively to reinforce their positions. This is always a source of irritation among scholars. Of course, the reality is that scholars with big ideas are rarely very good at the detailed work needed to craft a functional policy, much less at satisfying the political demands of policy making.

Loss of control is the price a scholar often pays for developing a powerful new idea that enters the public and political domain. Often the idea is badly distorted to justify policies that the scholar opposes. Some might argue that this danger of distortion is a reason to be extremely careful with scholarship that challenges conventional orthodoxy and that might be used by those whose values are different from those of the scholar. In my view, self-censorship of important new ideas is a recipe for both mediocre scholarship and moribund policy.

How might one get more of this reframing research? This is a domain where government has the hardest time providing targeted support. Most scholars work in relative obscurity, focusing on the problems they believe are most important and most amenable to their research skills. Occasionally, powerful ideas emerge. But they are hard to predict. It is nearly impossible to target resources only to settings where "the big idea" seems sure to emerge. Government can, however, provide essential building blocks for research: research grants, research and policy conferences, and most significantly, data. It can also provide mechanisms to help bridge the worlds of ideas and policy.

Understanding and Modelling Behaviour

Behavioural impacts lie at the very heart of any social policy and they are the central domain of social science and related fields. Thus, social scientists, who study behaviour, can be invaluable in designing social policies.

Nearly all policies aimed only at aiding the needy must confront the possibility of moral hazard: that by providing aid for people in some condition, such as unemployment or lone parenthood or even old age, the policy could perversely act to reduce work or family formation or savings.

It seems essential that scholars be involved in the discussion of particular policy designs and strategies. Without the rigour of scholarship, politicians and policy makers have a tendency to implicitly self-reflect - "How would I respond?" - or to stereotype - "These people need to be forced to work." Scholars can provide vastly more reliable ideas for effective social policies.

The news that scholars have to offer is not always what the policy maker wants to hear. Trade-offs abound in social policy. Behaviour can be very difficult to change. Costs often conflict with incentives. Modest shifts in policy rarely alter larger social forces much. But these are circumstances where academic voices should be heard. Ignorance of the real behavioural forces is not bliss.

Economics seems to have a sizeable lead in the influence category, probably in large part because it is one of the few disciplines that offers compact ways of describing some of the forces that shape behaviour, and economists alone seem willing to quantitatively predict how responsive people will be to changed policies. Economics can offer more concrete (though not necessarily more accurate) answers to specific policy questions. Increasingly, the language of economics and the statistical analysis tools that economists favour are being applied to policy design. As an economist I applaud these developments, even as I also believe that the significant limits of economics should not be forgotten.

Economics, with its tendency to focus on the individual, is rightfully criticised for often ignoring the larger societal forces that constrain opportunity, shape identity, and limit ambition. The very notion that poverty should be principally defined as having an income below some cut-off is challenged by a diverse array of scholars, now perhaps most prominently Amartya Sen (1999). Social exclusion rather than poverty per se really is the issue.

In recent years in the United States and other nations, a very positive trend has emerged as ethnographers, social psychologists, sociologists and others have begun to learn from each other and to apply methods across disciplines.

Evaluation

Nowhere is scholarship more valuable than when it provides a highly credible evaluation of a policy being considered. The debate over human capital versus work-focused approaches benefited considerably from the random experiment evaluations. Of course, evaluations themselves are only as reliable as their methods and authors.

I believe that this emergence of randomised controlled experiments is one of the most positive and important developments in the chequered history of scholarship and policy.

Such experiments are expensive, they require years of data gathering, they raise ethical concerns, and they inevitably examine only a small number of programmes or approaches. But lots of less-expensive evaluations are done and they often have very little influence or - worse yet - they come to erroneous conclusions and have significant influence. In my view, the payoff per dollar invested is vastly higher for randomised controlled trials.

In his book Politics and the Professor (1987), Henry Aaron wrote:

Analysts not familiar with government decisionmaking are surprised and often shocked by how small a direct contribution research makes…. One suspects that an awareness of the often precarious objectivity and short life expectancy of most research helps explain why policy makers often place less weight on analytical findings than analysts feel they merit and why they tend to place the most weight on findings that support their predilections. Far from being a sign of intellectual mendacity, such behavior may reflect the fact that the person who finds research in conflict with deeply held beliefs may think errors in the research are more likely than in his preconceptions.

With randomised experiments, the social science community and ultimately the political policy makers have come to largely accept the findings. That a method could be found that is widely seen as reliable represents a significant step forward in the capacity of scholarship to influence policy.

And the payoff grows as the number of demonstrations grows. When several programmes seem to show similar patterns, or when one programme performs much better or worse that others, larger patterns of what works or does not work begins to emerge. So it has been with more recent evaluations of welfare-to-work programmes. "Work first" works better than human-capital-only programmes, but the best training programmes are probably those that are embedded within a firm or industry where the trainee is prepared for a specific job opportunity.

Improving the connections

The contrasting cultures of scholars and policy makers will always provide barriers to effective collaboration. But much more can be done. The list of proposals in Connections, Resources and Capacities: How Social Science Research Can Better Inform Social Policy Advice (Social Science Reference Group 2001) is excellent. Let me briefly reinforce and supplement those ideas.

Data

Data are the lifeblood of social science. It is fair to say that if longitudinal data had not existed in the United States, welfare reform would not have taken the shape it did. Unfortunately, data are moderately expensive to produce, and governments understandably worry about costs, burdens to citizens and confidentiality. If governments want more policy-relevant research, however, almost nothing is more effective in stimulating it than supporting readily available and easy-to-use data that include information on incomes, public assistance use, deprivation, and whatever else people can agree on. Longitudinal data are particularly valuable because issues of causality and behavioural change are far easier to isolate.

However, the issue is often less one of cost than of confidentiality. Governments are understandably worried that people may not be willing to cooperate if their information is shared. In reality, I believe thoughtful statistical agencies can quite effectively protect confidentiality while releasing most data (though often with only very broad geographic identifiers). In the United States, where data are widely available, I am not aware of any major confidentiality scandals. Of course, public perception may not entirely reflect reality. Nevertheless, if there is anything governments can do to facilitate research and policy connections, it is to collect and release data. Many countries, including New Zealand, I fear, can do better.

Evaluation

Evaluations, particularly randomised controlled evaluations, are exciting for scholars, respected by their peers, and ultimately very important for policy. The best demonstrations are not ones crafted by clever academics to test pet ideas. The best demonstrations reflect ideas or policies advocated by a city, state or community group. These ideas are much more likely to resemble something that might actually be adopted. Governments must find the courage and resources to encourage innovation, while ensuring genuine accountability through first-rate credible evaluation.

Grants for Policy-Relevant Social Science Research

Grant programmes can ultimately yield many of the big framing ideas that can shift policy paradigms. And they can help refine understandings of behavioural impacts of particular policies and incentives. But a critical element is how the selection process for such grants is implemented. If dominated by disciplinary scholars, the research will look highly disciplinary. If scholars and government officials are jointly involved, one can get both the rigour of the disciplines and the relevance of the government.

Policy Centres / Policy Schools

The rewards for genuinely helpful policy research are limited in most academic settings. Indeed, the reaction of peers can be quite negative. But a sizeable fraction of those drawn to scholarship in policy-relevant areas are pulled by a genuine desire to advance the public good. If there are environments where policy-connected scholarship is rewarded, and where governments actually listen to these ideas, plenty of serious scholars will participate. Policy centres can be linked directly to political science and policy departments, sometimes offering quick responses to important questions, sponsoring conferences and research, and facilitating meetings between key scholars and policy makers. Moreover, the more that scholars are exposed to the questions and issues of the government, the more helpful they will become.

Small and Relatively Informal Gatherings of Senior Policy Makers and Scholars

When senior policy makers get a chance to spend extended time together, discussing and socialising, more bridges are probably built than during any other activity.

Scholar / Civil Servant Exchanges

There are small programmes in several countries encouraging scholars to spend time in government agencies, and reciprocal programmes encouraging senior civil servants to get further education or to spend time at a major research institute. These are an excellent way to build connections, though I am unsure how often these lead to direct impacts on policy.

Scholars as Policy Entrepreneurs

Occasionally, scholars seek to genuinely bridge the two cultures, seeking to develop and advocate specific policy approaches that can consequently be implemented. These scholars do more than provide advice about strategies that might be more or less effective in achieving specific ends; they become advocates for a particular framing of goals and they seek to promote a set of policy ideas. This is a dangerous road for a scholar, for it can taint his or her objectivity. But for those who do seek to advance ideas, the rewards can be many.

The power of politics is the opportunity to act, to really make a difference, to fulfil principles and advance the public good. The life of the scholar is reflective, designed to stimulate action in others. For me, personally, the most exciting professional times of my life were probably the years I worked in government. At its best, I was immensely proud of many of the ideas and the policies that I had a chance to be involved in creating. At its worst, I wondered whether I had sacrificed my integrity for political expediency. Still, what I was doing felt important, far more important than any paper or book that I could ever write. But it also felt like a time when I was consuming my human capital without replenishing it. To me, that signalled that as a policy maker I needed the ideas and insights that scholars had to offer. Serious social policy can combine the values and the power of political vision and the reflection and rigour of scholarship.

Melding scholarship and research will always be a challenge, precisely because of the cultural clashes. But there is so much to be gained. The struggles facing people and polities are simply too important not to find ways to collaborate.

References

Aaron, Henry J. (1978) Politics and the Professors: The Great Society in Perspective, The Brookings Institution, Washington, D.C.

Bane, M.J. and D.T. Ellwood (1994) Welfare Realities: From Rhetoric to Reform, Harvard University Press, Cambridge, Massachusetts.

Hamilton, Gayle (2002) Moving People from Welfare to Work. Lessons from the National Evaluation of Welfare-to-Work Strategies, Manpower Demonstration Research Corporation, New York City.

Sen, A. (1999) Development as Freedom, Random House, Inc., New York.

Social Science Reference Group (2001) Connections, Resources and Capacities: How Social Science Research Can Better Inform Social Policy Advice, Ministry of Social Development, Wellington.

Wilson, W.J. (1987) The Truly Disadvantaged: The Inner City, the Underclass, and Public Policy, University of Chicago Press.