"Evaluation: Making Performance Count" The International Conference of the Australisian Evaluation Society
Ian Trotman
Senior Advisor
State Services Commission
This paper summarises the main events, key themes and most interesting papers delivered at the Annual Conference of the Australian Evaluation Society (AES) held in Canberra in September of this year. The key themes of the conference were:
• Quality management and associated benchmarking and promotion of best practice.
• Performance measures including a growing interest in the place of "soft" measures relating to staff commitment which is seen as essential and to the other emerging interest of customer satisfaction.
• Growing interest in evaluation and policy analysis, where NZ is further ahead (acknowledged by one of the key speakers).
• Several papers promoting the use of evaluation in strategic planning.
• Overview papers of evaluation in the federal government, NSW public service and Queensland; and
• Three notable papers on Evaluation in France, developing an evaluation system to meet ISO9000 standards and on meta-evaluation and how to compare evaluations.
The Opening Hypothetical
The first session of the Conference was a very interesting hypothetical which was effectively a role play situation involving an issue of following through a Minister's bright idea without adequate analysis and trying to retrieve it by evaluation as it began to fall apart. Of particular interest were the calibre of people playing the roles with a real Senate Minister, one of the leading TV interviewers as the moderator, permanent secretaries, officials, union leader, industry association consultants all playing their normal but exaggerated roles.
This piece of theatre brought out the importance of designing in evaluation from the start, the importance of up-front planning, trying to envisage likely questions and planning for their answers, the importance of "no surprises" which can be reduced by wider consultation, recognition of the risk of trying new ideas and the importance of managing expectations.
Of interest was the comment of the editor of the Canberra Times, who made the point that Ministerial mistakes associated with bureaucratic minding are always good copy and the newspaper records usually are able to hype up the stories by linking them to similar examples in the past. This was a very useful session and was referred to by a number of the speakers during the rest of the conference. A video tape of this session can be purchased through the AES.
Colloquium
This was a panel involving two permanent heads, Dr Michael Keating, the Secretary of the Department of Prime Minister and Cabinet, and Andrew Podger, Secretary of the Commonwealth Department of Housing and Regional Development; Dr Arnold Live, President of the Evaluation Society and Dr Ilana McLean, President of the Australasian Evaluation Society. The focus of the colloquium was supposed to be on where evaluation was going, with emerging key issues and ideas for the future.
Andrew Podger advocated the use of the evaluations as a means of providing fundamental review of major policy and considered that they were a must and an important basis for ongoing management, particularly linked with corporate planning. He suggested that evaluations should be more exposed to the evaluation community for peer review by exposure in journals etc.
Ilana McLean spoke briefly and focused on the importance of evaluation for providing the necessary information required for a wide range of management issues and stressed the importance of looking at policies from the client perspective and from an integrated perspective across the range of activities contributing to major programmes.
Arnold Love addressed the topic most effectively referring to a recent Canadian Auditor-General report which had three chapters on programme evaluation. He noted that the conclusions were somewhat similar to Auditor Generals' reports he has seen in Australia for 1991 and 1992. The AG had noted that evaluation was essential for resource decisions but had been used very poorly; it should be used as a basis for improving programmes, and to provide better accountability for results where it was of marginal effect. He noted that there was a proliferation of small scale "stove pipe" evaluations without adequate cross-linking between activities or agencies which impinged upon the programme or activity being considered.
In Canada there is much more linking of evaluation to total quality management and the learning organisation. There was concern about an absence of central or external (more objective) evaluation and a lack of publication of more critical evaluations. There was also concern that the low level of training for doers resulted in their picking up the skills on the fly.
Major Themes for Significant Papers
Quality Management
There were a number of papers reflecting the theme of quality management and associated emphasis on benchmarking best practices and the importance of moving to ISO standards for international competitiveness, including where public service activities contribute to processes for producing goods or services.
These ranged from an enthusiastic presentation by Steve Bartos on benchmarking explaining it and its application already within the Australian public sector; to a clutch of papers dealing with best practice and an Australian industry scheme giving away significant sums for organisations to be recognised and demonstrate good practices in the export sector; to a very interesting paper by Richard Hunter who has prepared a framework for evaluation which is believed will meet the ISO9000 standards. This includes, among other things, detailed documentation of all the steps of the process which are signed off, critical definitions, and an audit trail. He has developed a simple system based on a framework and a number of templates aimed at facilitating the adoption of this procedure by the various directorates of his organisation with:
- emphasis on encouraging and empowering evaluations;
- following improving evaluated techniques;
- providing a framework rather than a manual;
- procedures capable of broad application; and
- tools for evaluation.
Performance Measures
There were a number of papers demonstrating a significant shift in evaluation to measuring soft performance measures primarily relating to the performance of employees who are the major internal force influencing organisation performance and to customers who are the key to the organisation's success. A very interesting paper by Peter Bycroft developed models for a corporate health index and performance in priority matrices and similar service and product index and performance and priority matrices. This approach was touched on by several other papers, although one by Michael Gill expressed some caution on the pitfalls associated with measuring client's satisfaction and emphasised that it was more important to understand the nature of relationships involving staff and clients and to manage those particularly through the development of a learning organisation mentality. His premise was the importance of relationships rather than services.
An interesting paper was prepared by Dianne Grady who had been in the public sector and taken a period out in the private sector and recently returned. From this combined experience she had developed seven lessons on performance management and measurement. (As an aside, it was interesting this year to pick up an increasing trend which reflects the North American experience, of many more people in Australia moving between the public and private sectors). These seven lessons include:
- more care on how information is collected and communicated.
- taking a spotlight approach to focus on key information compared to trying to cover everything.
- the "quarantining" of some information to avoid overload and fudging.
- being aware of actions that people will take to meet targets before installing new measures – people are eager to look good on measures which may have counter-effects to improved service.
- treating the new emphasis on soft measures seriously, and
- recognising strategic planning as a flexible tool which can serve a number of different purposes.
Evaluation and Policy Analysis
There is a considerable interest in improved policy analysis including a paper by Dr Meredith Edwards, Deputy Secretary of the Department of Prime Minister and Cabinet, on the issue of unemployment that traced through the delivery of both green and white papers together with the associated background evaluation material on which they were based. This is a very comprehensive paper outlining the approaches used and possibly shining a torch on aspects which were not widely appreciated.
There was a very interesting paper by Professor Fred Hilmer, currently Dean of the Australian Graduate School of Management, who is on the Business Council of Australia and has been involved in chairing a number of major reviews for the Commonwealth Government. He was asked to prepare a paper on how to recognise good policy and admitted freely that he had not really thought about this topic before but found it very interesting to do so. Unfortunately, his paper was not available at the Conference and is one amongst several which will be distributed subsequently.
Evaluation for Strategic Planning
There was a small group of papers linking evaluation to strategic planning. One was by Faye Lambert and John Owen, who stressed the importance of shifting from seeing many activities as individual and linear processes to integrated and circular systems with feedback loops. In addition, there were more practical papers evaluating strategic planning or processes associated with it, e.g. Ken Fishpool who reviewed the planning of agricultural quarantine.
Overviews of Public Sector Evaluation in Australia
There were a few papers providing an overview of public sector evaluation in various parts of Australia. Steve Sedgwick, Secretary of the Department of Finance, emphasised the importance of evaluation in government decision-making, especially the Budget round. Jane Diplock, Director-General of the New South Wales Department of Industrial Relations, emphasises the evolving practice of consulting community clients as the ultimate recipients of the services. Robert Simpson of Queensland noted that a survey of 18 agencies showed a low implementation of evaluation recommendations, the need for more commitment by senior management and leadership from central agencies, and emphasised the desirability of steering committees.
Evaluation in France
Dr Silvie Trosa has been at the heart of the French Government's Scientific Council of Evaluation, a group of eleven mixed academic and senior officials who are responsible for overseeing every Cabinet approved evaluation to ensure adequate quality, whilst also providing the role of mediator between different agencies contributing to large-scale evaluations. The council also ensures that satisfactory steering committees are set up for each evaluation which involve dialogue with the stakeholders. Hers was a very interesting paper which described how evaluation has led to improvement in French departments working together. It also indicated the importance of the person interest and involvement of the Prime Minister to ensure that the Council operates effectively.
Meta-evaluation
Another significant paper was one by Denise Conroy on meta-evaluation, i.e. evaluation of evaluations. She produced a framework in five charts bringing together all the key elements of the types of evaluation, purposes, interest of main stakeholders, approaches to management, programme logic and performance measurement frameworks and a model for evaluation utilisation. This framework, together with a one-page summary of assessment criteria for meta-evaluation was applied to two major evaluations: one on industrial relations and the other on the Defence Force Academy as a basis for comparing them.
