Evaluation 95; Evaluation For A New Century, A Global Perspective

Marianne Bray
Social Policy Agency

First Global Meeting

Evaluation '95 was the first international evaluation conference. The conference was co-sponsored by the Canadian and American Evaluation Associations with help from the Australasian, Central American and European Evaluation Societies.

This review describes the conference in general, talks about the stage evaluation has reached both internationally and in New Zealand, summarises the key debates and tries to reach some resolution about some key evaluation issues on a global level.

Key Themes

The conference was designed to provide a forum to share experiences and learn about new development in evaluation thinking and practice for the 21st century. It centred around three themes:

• Global Issues: highlighting international experience in evaluating the effects of policies or programmes dealing with world-wide problems including sustainable development, status of women, quality management and mainstreaming of disadvantaged populations.
• Evaluation Lessons Learned: disseminating accumulated wisdom in the traditional areas of planning, monitoring and evaluation. This included education and training, organising the evaluation function, prospective and retrospective methods, maximising evaluation credibility, ensuring the use of evaluation findings and understanding the political culture of evaluation.
• Changing Face of Evaluation: emphasising new evaluation approaches which suit the cultural context, abilities and technical expertise of evaluators, programme staff and beneficiaries.

Vast Scale

The conference was vast. Sixteen hundred evaluators and users from business, government, academia and voluntary sectors, from 65 countries, gathered in Vancouver to discuss evaluation. They used a variety of activities - plenary and parallel sessions, poster displays, social events, workshops, topical interest groups and site visits.

The scale, which made it exciting, was also a drawback. In total there were over 350 sessions. For every 90 minute session there were 30 different parallel sessions to choose from, featuring several papers. Because the conference sought to capture a wide audience - from those who knew a great deal about evaluation to those just beginning to practice it - the sessions ranged from critical debates to basic evaluation practice.

Because of this vastness, it is impossible to say whether the conference achieved a global consensus. However the conference was extremely useful in indicating the state of evaluation in different countries, and highlighting some of the key debates and issues on a global level.

To What Stage Has Evaluation Come Internationally?

There was agreement among participants that evaluation is a young discipline, with many different uses and users, still trying to define its role and relationships to other disciplines. While it originated in the education sector, it now includes aspects of auditing and performance monitoring. Many participants hope that, one day, evaluation will have an equal status with the accountancy discipline in organisational management.

For evaluation to have a higher standing (comparable to accountancy or economics), several speakers argued there was a need to develop a sounder theoretical base for the discipline. They felt that while some knowledge has been generated on new theoretical models, there has been little theory development in recent years. Evaluation theory tends not to be developed because evaluators focus on practice, and are commissioned to evaluate rather than read or develop theory. Many evaluators do not think about theory other than as guiding principles to govern action. A need was expressed to go beyond practice to theory, as otherwise evaluation will only ever be applied social science research. Professor Thomas Cook from North Western University in Chicago used the examples of economics: the discipline is strong because it is related to money and because it is generalisable.

For evaluation to become a discipline in its own right, and to have power like economics, a more substantive knowledge base is required, synthesised from a wide range of studies and experience, to establish "good practice" - what works and does not work in different circumstances. This kind of development is needed to confirm and validate evaluation. Wider theoretical questions also need to be answered. For example, how do you ensure a place for objective evaluation in a government policy-making setting, when government and government workers are political actors in a political process, trying to validate the success of their policies and programmes to Ministers and to voters? How do you ensure a political commitment to evaluation, when policy focuses on budget approvals and what can be gained through that process, rather than the evaluation of progress?

The US Scene

Of the different regions represented at the conference, the United States stood out. Evaluation is widely funded and practised in the education, labour market, criminology, social policy and health sectors. Increasing interest in programme and organisational evaluation has been driven by economic and fiscal factors, state and federal legislatures (the Federal government's new Performance Results Act) and the executive (the US Office of Budget and Management pushing agency performance audits and evaluation units). The United States also has a number of academics who are willing to work on evaluation theory, and provide a well-spread training capability. Private foundations and businesses are also getting onto the evaluation bandwagon. Despite this high level of resourcing, it was interesting to note that not all American evaluations are large, well funded, net-impact studies using experimental methods; some were small scale, lacking resources and having trouble involving key stakeholders.

The European Scene

It was difficult to gauge the development of evaluation in Europe, with different countries at different stages. There was a feeling among European evaluators that Europe gained a lot from the evolutionary development of evaluation in North America and was able to avoid repeating some of the same experiences. However, nine reviews undertaken by the European Commission on European Union evaluations were less positive.

The Commission found that only large-spending organisations had evaluation units many with small budgets. While these units were well skilled, there was a low level of awareness of the importance and utility of evaluation in programme management. Evaluation activity was often carried out in isolation and a major challenge is to develop inter-government evaluations. Evaluations often had vague objectives, and focused on output evaluations, rather than outcome or impact evaluations. Harmony with the policy cycle was difficult and there appeared to be a missing link between evaluation and budget implications.

The Place of New Zealand

New Zealand is often seen as lagging behind the United States in terms of its evaluation progress. Ian Trotman, one New Zealand participant, noted that the work being done in Australasia, although on a much smaller scale with far fewer resources, could be held up to work being carried out in North America. He cited Inland Revenue's award-winning evaluation project as an example. He also noted areas where this part of the world was ahead, such as in benchmarking, client satisfaction, professionalism and the move to accreditation and legislative recognition.

While the emphasis in the conference was very much on the North American way of doing evaluations, with the Europeans expressing a disappointment at the lack of discussion on issues of power and politics, the international scope of the conference was useful, highlighting a "coming of age" for the international evaluation community. Of special interest was the raising of similar debates and issues across the globe.

The Debates

Because evaluation theory is not well developed, evaluators are still debating a number of dichotomies across the world:

• the quantitative versus qualitative debate, when in fact, good research requires both,
• summative and formative evaluation, with the addition of a third element, meta-evaluation,
• external or internal evaluation, influencing the forms, utilisation and objectiveness of reporting,
• interactive or distance (away from the programme) evaluations,
• individual studies or a synthesis of studies (rather than basing conclusions on one or two possibly flawed studies, gathering information through a synthesis of studies),
• evaluation for the purpose of taking action (making decisions) or evaluation aimed at "enlightenment" (increasing knowledge across the board).

A keynote plenary session by Professor Thomas Cook looked at some of the above issues. He argued for a balance and linkage of these elements depending on the purpose and contest of the evaluation. He noted that evaluation will not play a more central role without wider debate around such issues.

Beyond the dichotomies listed above, speakers discussed:

• the extent to which evaluators should be making value judgment or recommendations;
• the implications of increased emphasis on performance and outcomes measurement in government,
• the extent and impact of stakeholder involvement,
• the need for effectiveness measures to link to decision making and government outcomes,
• the need for the institutionalisation of evaluation to ensure regular quality evaluations,
• the need for a common language for evaluation, and
• the idea that evaluators are rarely welcomed by parties to the evaluation.

Attempts at Resolution

The conference reached a synthesis of minds at a later session when four American evaluation gurus, Michael Patton, Michael Scriven, William Shadish and Ernest House, listed the evaluation issues they agreed and disagreed upon. Despite their different starting points (for example Scriven is a believer in external evaluations to make "judgments in truth", while Patton tries to draw stakeholders into the conclusion so they own the product), the four theorists agreed upon the following issues:

Theory of use

• Evaluation is used in many ways; evaluators are concerned with both instrumental and conceptual use of evaluation process and findings.
• Evaluations are more likely to be used if they specify intended uses by intended users, and the processes by which use will occur.
• The evaluator is accountable to intended users for intended uses.
• There are multiple, varied and usually conflicting interests around any evaluation and limited resources with which to do evaluation. It is necessary to prioritise intended uses and intended users in the evaluation design, and thus, also, the political inherency of evaluation.

Theory of valuing

•  Evaluators/evaluation cannot be value-free.

Theory of knowledge

• Epistemology and methodology are essential topics in evaluation.
• In the US, programme evaluation is an empirical endeavour in a social science tradition.
• Knowledge of many different kinds (causation, generalisation, description of implementation) must be constructed in most evaluations, but the emphasis given to each differs over studies.
• No social science method can be automatically rejected from the evaluator's repertoire.
• All methods are fallible.
• Multiple studies are preferred to single studies for reliable and valid knowledge generation.
• Over the long-term, quality of knowledge increases with critical public scrutiny of it.

Theory of practice

• Evaluation occurs under time and resource constraints that require difficult trade-offs.
• Active and skilful facilitation by the evaluator is needed to increase the amount and quality of use of the evaluation process and findings.
• Evaluations should be judged by their utility, feasibility, ethics, accuracy and relevance.

It is clear from this conference that because the theory of evaluation is not developed, spans so many areas and is shaped mostly by the theory of practice, it is open to many interpretations and to much debate. The conference sought a resolution to these debates by getting key theorists together. This synthesis of minds merely highlighted the many different uses and users of evaluation. A key challenge heading into the 21st century is to develop a founding theory to bring evaluation to an equal standing with other disciplines such as economics and accountancy.