Challenges to Implementing Good Practice Guidelines for Evaluation with Māori A Pākehā Perspective

Mathea Roorda¹
Independent evaluator
evalue research ltd

Robin Peace
Associate Professor
School of People, Environment and Planning
Massey University

Abstract

Evaluation contractors working in the Aotearoa / New Zealand government sector, whether Māori or non-Māori, are expected to use culturally appropriate processes when evaluating mainstream programmes where Māori are a significant subgroup. For independent evaluators, this expectation is generally made explicit in requests for proposals. A range of formal guidelines has been developed over the last decade to support both commissioning agencies and independent evaluators to conduct culturally appropriate evaluations. However, few of the processes suggested in these guidelines are fully incorporated into Pakeha evaluation practice. A case study of an anonymised "Programme X" identifies and reviews some of the challenges to good practice and process.

Introduction

Over the past 10 years evaluators in New Zealand have had access to different sets of guidelines that focus specifically on research involving Māori (Health Research Council 1998, Te Puni Kokiri 1999, Ministry of Social Development 2004). These guidelines have led to greater awareness of a range of considerations entailed in research and evaluation involving Māori. The launch of the Social Policy Evaluation and Research Committee (SPEaR) Good Practice Guidelines2  (Guidelines) is a renewed prompt for evaluators and social policy agency officials to reflect further on the processes they engage in when evaluating government-sponsored programmes where Māori are participants. The Guidelines also have relevance to independent evaluators and researchers who work for government sector agencies (SPEaR 2008).

In a New Zealand social policy context there is an expectation -- generally made explicit in the selection criteria of requests for proposals (RFPs) for research and evaluation work -- that culturally appropriate processes will be adopted wherever Māori are a significant group or subgroup in the programme under review. Although an evaluator or evaluation team may start with strong intentions of observing the advice and direction of such guidelines, they often fall short of what is outlined as good practice. The focus of this paper is on some of the complex issues that attend the implementation of the Guidelines in real-world research and evaluation contexts. We show that, for independent Pakeha evaluators, the "intention of observance" and the apparent "failure of practice" pose ongoing challenges.

Identifying and understanding some of the factors that might contribute to the gap between intention and practice is one of the purposes of this paper. A second purpose is to stimulate dialogue among evaluators and commissioning agency staff about current practice and how "the standard of research and evaluation practice across the social sector as a whole" (SPEaR 2008:5) could be enhanced. Insights drawn from a review of the process of a government-commissioned evaluation, and experience in independent evaluation practice, inform the discussion in this paper.

We begin with a short contextual discussion of guideline development in New Zealand, and then move to the analysis of "Programme X". In this case study all identifying material has been removed so that the evaluators, commissioning agency and participants have their anonymity preserved. Notwithstanding the constraints and frustrations that were identified in this evaluation, all those involved worked hard, within complex constraints, to deliver a responsive, robust evaluation.

What are Guidelines for Research and Evaluation with Māori?

Guidelines, as the term implies, are not sets of rules or laws, but rather provide principled advice and direction. In the case of research and evaluation, such guidelines may steer the conduct of a specific project in the direction of ideal practice. Professional evaluators who belong to associations or societies such as the Australasian Evaluation Society (AES), or the recently formed Aotearoa / New Zealand Evaluation Association (ANZEA), will also work within ethical guidelines promulgated by the organisation. AES, for example, promotes the ethical practice of evaluation and aims to "foster continuing improvement in the theory, practice and use of evaluation" (Australasian Evaluation Society 2002:2). The ANZEA constitution identifies its purpose as being to:

Promote excellence in evaluation in Aotearoa New Zealand with a focus on the maintenance of appropriate ethical standards for the evaluation profession, development of effective practice and craft, and the promotion of reflective learning as a strategy for evaluation for the public and community well being. (ANZEA 2007)

Guidelines for research and evaluation involving Māori are intended to improve the quality of evaluation practice within Aotearoa / New Zealand government agencies and to "enhance our ability to carry out effective and appropriate research (and evaluation) with Māori" (Ministry of Social Development 2004:2).

Tauri (2004) outlines the chronological development of guidelines for research and evaluation involving Māori and notes that a number of initiatives implemented in the 1980s and 1990s by government agencies acted as a signal of increased recognition and commitment to the Treaty of Waitangi. The first to be developed were Guidelines for Researchers on Health Research Involving Māori (Health Research Council 1998). These were aimed at researchers in the health sector and focus on processes for consultation. They were followed a year later by Te Puni Kokiri's Evaluation for Māori: Guidelines for Government Agencies (Te Puni Kokiri 1999). In 2004 the Ministry of Social Development's (MSD) Centre for Social Research and Evaluation (CSRE) published Nga Ara Tohutohu Rangahau Māori: Guidelines for Research and Evaluation with Māori, aimed at both staff and contractors. These two most recent guidelines include suggestions for processes to consider during all stages of an evaluation, from planning through to reporting.

In 2002 SPEaR began developing a set of guidelines relevant to all stakeholders in social policy research and evaluation in Aotearoa / New Zealand. SPEaR's mandate is to "oversee the government's social policy research and evaluation purchase" (SPEaR, n.d. Purpose, para. 1). One of its functions is to "promote the utilisation of 'best practice' approaches, tools and techniques through development (where necessary) and/or dissemination" (SPEaR, n.d. Roles and functions, para. vii). The SPEaR Good Practice Guidelines are designed to:

… provide practice-based advice aimed at improving social sector research and evaluation systems and processes, enhancing the professional conduct of government officials and external research contractors, improve the generation of information that informs policy development and service delivery, support the saving and sharing of research data and encourage the development of respectful and ethical working relationships between all participants in social sector research and evaluation. (SPEaR 2008:5)

As stated in the Guidelines, good practice is seen to rest on five principles: integrity, respect, responsiveness, competency and reciprocity. The practice expectations that are included are consistent with those outlined in the previous guidelines (SPEaR 2008, Te Puni Kokiri 1999, MSD 2004). The SPEaR Guidelines inform the evaluation of Programme X that follows.

Given that considerable intellectual and organisational work underpins the development of the range of guidelines in the last decade, it is reasonable to expect some success stories in relation to their implementation. No such stories currently circulate, however, and the analysis that follows indicates how stories that do emerge are likely to highlight more challenges than successes.

Evaluation of Programme X

Programme X was a year-long evaluation of a government-funded programme that included Māori as a significant subgroup of participants. The programme that was evaluated was delivered by Māori staff employed by community-based service providers, most of which are iwi-based organisations. Community stakeholders were not involved in key decisions about the evaluation design prior to the request for proposals being developed because there were significant time and budget constraints (Agency contract manager, personal comment, 2008). The evaluation was contracted to an independent evaluation company and led by two experienced Pakeha evaluators. In terms of strategies designed to manage the limited budget, a senior Māori evaluation advisor from the commissioning agency provided 15 hours' support to the project team, and Māori staff, also from the agency, were recruited to assist with fieldwork and analysis.

Method of Review

The review of the activities in Programme X was undertaken prior to the finalisation of the Guidelines, consequently the draft Guidelines were the point of reference. However, the substantive items that comprised the review framework remained consistent between the Draft and final versions. For the analysis of the evaluation programme's uptake and implementation of advice and direction, five systematic (although not exhaustive) steps were followed:

1. An inventory of the practice expectations listed under each of five guideline principles was prepared.
2. The field notes and records made during the evaluation were scanned and additional notes, from the reviewer's own recollections of the process, were compiled.
3. The evaluative activities were broken down and matched with relevant expectations.
4. The matched findings were tabulated.
5. The review process was reviewed by another independent evaluator and the agency contract manager, both of whom confirmed the findings.

A summary of these review findings is given in Table 1³.

How this five-step review process worked can be followed in the present context. First, the reviewer considered the principle of respect in the Guidelines, which states that there is an expectation to "Involve Māori participants (whanau, hapu, iwi and community organisations) as early as possible in the designing of research that will impact on them and their communities" (SPEaR 2008:34). Notes and recollections from the evaluation identified a number of activities that were undertaken early in the evaluation. For example, the evaluation objectives were defined by the commissioning agency; the evaluation design development occurred as part of the tender proposal process; and prior to fieldwork beginning the evaluators met face to face with the service providers to introduce the evaluation team, explain the evaluation design as it had been signed off by the sponsoring agency, negotiate access to clients, and explain the proviso that providers would be able to veto clients from the participant sample. The reviewer then entered these findings in the table of findings and made an assessment of the extent to which the Guidelines had been incorporated in the Programme X evaluation. The possible responses were "yes" (implemented), "no" (not implemented), or "partial" (partly implemented). The findings were later corroborated by the independent evaluator and the agency contract manager.

In what follows, the main findings are identified and then discussed according to whether the expectations were met, partially met or unmet. In the discussion, the processes that occurred during the evaluation are described and the range of constraints and barriers that were encountered during the evaluation are briefly identified in order to set the scene for the discussion that follows in the final section of the paper.

Expectations Met

Overall, only one SPEaR guideline was effectively incorporated in the evaluation. Under the principle of reciprocity, the expectation of ensuring Māori participants⁴  receive all relevant information about the evaluation could be considered to have been achieved. The purposes of the evaluation were explained (both verbally and in writing) to all potential participants, and participants were invited to sign a consent form. The evaluators provided information on what participation and confidentiality (including who will have access to their information) would involve, and clarified that participation was voluntary and that participants had the right to refrain from answering any questions and could end the interview at any time. This expectation is similar to standard ethical guidelines that must be enacted with any group of participants, and it is therefore not surprising that it was incorporated.

Expectations Partially Met

Eight of the expectations were judged to have been partially met. The first three partially met expectations fall under the principle of respect and include the need to:

Involve Māori participants (whanau, hapu, iwi and community organisations) as early as possible in the designing of research that will impact on them and their communities

Use the project-planning phase13 to identify the appropriate protocols that need to be observed during engagement with Māori participants and stakeholders now and in the future

Ensure that the budget for the project is adequate and includes sufficient resources for consultation, reciprocity, compensation for contribution/participation and feedback  (SPEaR 2008:34)

The fourth partially met expectation comes under the principle of integrity, and entails the application of ethical guidelines, while the fifth, under the principle of "responsiveness", includes the development of processes that enable Māori participants to maintain contact with the project team throughout the project. These are stated in the Guidelines as:

Conduct the research according to the agreed protocols (i.e., of the research ethics committee if applicable, the project management committee/Advisory Group) and in accordance with legal requirements and professional guidance (e.g. the code of practice of the professional body they are members of, the procedures of the agency they are employed by) (SPEaR 2008:25).

Develop processes that enable Māori participants to maintain contact with the project team throughout the life of the initial research project, or future unspecified projects, and which enables the project team to keep participants informed of the progress of the project(s) (SPEaR 2008:37)

The sixth and seventh partially met expectations fall under the principle of competency and entail the inclusion of officials or external advisors with an appropriate level of experience and knowledge of tikanga and kawa, and the inclusion of people with experience and knowledge of methodologies and methods applicable to Māori evaluation contexts. In the Guidelines, these are stated as:

Include in the project team, officials or external advisors with an appropriate level of experience and knowledge of the tikanga (protocols) applicable to the Māori entities involved in the project, with experience in research involving hapu, iwi and or other Māori organisations, and with experience and knowledge of methodologies and methods applicable to Māori research contexts. (SPEaR 2008:39).

The final expectation that met with some partial success fell under the principle of reciprocity, and entails the "Release research findings (and in appropriate formats) as agreed in the contract and with regard to the consent given by (and any additional guarantees given to) research participants" (SPEaR 2008:22).

The first SPEaR guideline element, respect, outlines expectations that protocols for engagement with Māori participants and stakeholders be identified during project planning, and the reasons why there was only partial success in this aspect can be identified in a re-telling of what happened: the process story. In the first instance, the contracted evaluators set out to establish contact with the Māori stakeholders by contacting senior managers from the commissioning agency who worked in locations where fieldwork was to be conducted. The evaluators' aims were to use both the existing organisational hierarchy and local knowledge to find locally-based Māori staff to assist the Pakeha evaluator in the field, to ensure the evaluator observed appropriate tikanga⁵,  to optimise rapport building, and to assist with making sense of the data collected.

What ensued was a complex set of circumstances that evolved beyond the evaluators' control. In one area, a Māori manager assigned a Pakeha staff member who they said had adequate knowledge of tikanga to support both the Pakeha evaluator and the Māori respondents. The Pakeha evaluators questioned the appropriateness of another Pakeha assisting them with the interviews but felt obliged to accept the manager's decision. After an initial meeting the designated staff member withdrew from the project, stating they did not have time to participate. Further, a relatively inexperienced Māori postgraduate student was sub-contracted at short notice to take up the research assistant role in one fieldwork location, while in a second location a Māori staff member was assigned by their manager to work with the evaluation team. While a job description had been developed for the role, the Pakeha evaluators were not in a position to assess the skills of either of their research assistants. The evaluators had little or no knowledge of tikanga or te reo Māori, and were thus reliant on others to identify research assistants with the appropriate expertise. Time constraints also meant they were not in a position to pick and choose from a pool of candidates.

In the field it became obvious that the Māori staff member had no knowledge of tikanga or ability to converse in te Reo. The impact of this lack of appropriate skill on the respondents is not known, but it was a stressful experience for the Pakeha evaluators. Although not openly discussed, it was obvious the situation was not ideal for the research assistant, who was enthusiastic about participating in the evaluation but clearly unable to provide cultural expertise to the project. In an attempt to be useful, the research assistant assumed responsibility for locating respondents' addresses and driving the evaluator to meet with participants. During interviews and in the analysis workshop they remained largely silent.

In contrast, the postgraduate student spoke te Reo confidently and was able to build rapport with Māori respondents that led to rich, in-depth data being collected. The debrief notes from one interview provide evidence of the difference it made for respondents to have a researcher with the appropriate skills:

The interview was shifted from Tuesday to Friday as the couple had to go to a tangi. Husband left a message on my cell phone telling us he was sorry but they'd had a family bereavement. When we arrived (at the prearranged time) he had popped out to drop his mokopuna [grandchild] somewhere locally. His wife welcomed us into their home. We started the introduction and said we would wait for her husband (to do the interview). She said she really had no idea why we were there; she was thinking we might have come because they were doing something wrong. When husband arrived, he responded to [research assistant] in Māori, saying he (also) didn't understand why were there really, but he was willing to listen. Once the consent form was signed and [research assistant] asked the first question, husband said he would like to start further back (in time), which is what he did. It was important that we allow him to start his story from where he was comfortable and this really helped (with building rapport). Once the interview was over we were invited to stay for a cup of tea. (Debrief notes from interview)

Respondents were all recruited by telephone. Those who agreed to participate were sent a letter with information about the evaluation and confirmation of the date and time of the interview. As the fieldwork progressed, the evaluator realised that people still needed to be reassured verbally and in person about the purpose of their visit. For some, particularly older, respondents it was clear they felt more comfortable if the introduction was conducted in te Reo.

It is difficult to state with any certainty the extent to which respondents participated in an interview because of the presence of someone who had the appropriate cultural expertise. It is possible that the postgraduate assistant's ability to converse in te Reo, and their knowledge of tikanga, resulted in a good experience for both respondents and the researchers. It is also possible that these interactions enhanced the relevance of the information collected for the evaluation. Further and more systematic research would be required to confirm these speculations.

Partial implementation, therefore, was achieved for a range of elements across the evaluation process, but not in any clear, linear or cumulative way. It was difficult to identify the precise phasing of some elements or the logic of why some elements were attended to and others not. Much of what eventuated was beyond the evaluators' control. During the evaluation it was difficult for any of the participants to stand back and assess where practice was or was not following the Guidelines, and only the retrospective reconstruction of events, facilitated by the review process, made such assessment possible.

Unmet Expectations

Overall, at least nine of the processes considered important in the Guidelines were not implemented during the evaluation. The details of these appear in Table 1 and some examples are chosen for further clarification below.

Under the principles of integrity and responsiveness, a number of guidelines highlight the need for early and on-going involvement of stakeholders and participants. In particular, the Guidelines point to the need for officials to ensure the integrity of their work with Māori and:

• Develop a consultation plan for engaging with Māori, Māori organisations, hapu and iwi that have been identified as likely participants in the project (2008:35).
• Use the planning and consultation phases to identify whether there are likely to be actions required for addressing intellectual and cultural property issues or concerns now and in the future. Ensure future consultation is enacted when the data is reused (2008:35).
• Ensure that the budget for the project is adequate and includes sufficient resources for consultation, reciprocity, compensation for contribution/ participation and feedback (2008:34).
• Check the validity of the analysis and/or reporting of data with Māori participants (2008 36).

In addition:

To ensure the responsiveness of their processes, officials should:

Involve Māori participants in the design of the project - including the design the research question(s), the methodology, the methods, analytical framework and mechanisms for disseminating results. (SPEaR 2008:36)

In the case of Programme X, however, few of these principles were observed. The evaluation objectives were defined by the agency prior to the RFP being posted on the Government Electronic Tenders Service (GETS) website. Contractors responding to the RFP were expected to put forward an evaluation approach as part of their proposal (due three weeks after the RFP was posted). This approach allowed little opportunity for considering Māori interests and the level and nature of Māori involvement in the project. Although the providers were Māori organisations whose clients were predominantly Māori, no community stakeholders were consulted about key aspects of the evaluation design. According to Moewaka-Barnes, this approach to evaluation design is relatively common in the New Zealand government sector:

The reality is that frequently an evaluation is imposed on a programme and the researchers have little control over its parameters, other than deciding whether they will do it or not. Consultation in this context is not about being able to offer those who are consulted any level of control over the evaluation. (Moewaka-Barnes 2003:148)

The evaluator's proposal for the evaluation of Programme X worked around the issue of consultation by stating that the evaluators would consult with "Māori evaluation colleagues, Te Puni Kokiri staff and others undertaking related research and evaluation projects". In retrospect it is clear that, at best, the evaluators were in a position to negotiate aspects of their design. As Justice McGechan has noted, consultation is a rather different process from negotiation, which involves:

… setting out a proposal not fully decided upon; adequately informing a party about relevant information upon which the proposal is based; listening to what the others have to say with an open mind (in that there is room to be persuaded against the proposal); undertaking that task in a genuine and not cosmetic manner; and reaching a decision that may or may not alter the original proposal. (Health Research Council 1998:5)

Once the evaluation plan had been approved by the commissioning agency, but prior to fieldwork commencing, the evaluators visited staff from the Māori provider organisations selected by the commissioning agency for participation in the evaluation. The primary purpose was to introduce the evaluation team, build rapport and negotiate access to clients who had participated in the programme being evaluated. Theoretically, the providers were entitled to state that they did not wish to participate in the evaluation, but in reality they had little choice because involvement was included as a condition of their contract with the funding agency.

Lack of community involvement at the design stage produced repercussions for both the commissioning agency and the evaluators at the end of the evaluation. As with other aspects of the evaluation, both time and budget constraints made implementation according to the Guidelines problematic. The government evaluator responsible for commissioning the evaluation had to manage internal pressures to provide evaluative information within a policy-dictated timeframe and a budget that involved trade-offs in the sampling strategy. The evaluation findings were presented at a national hui attended by all the providers contracted to deliver the programme. The evaluation design was criticised by a number of providers because it focused on a small number of providers/areas and ignored the experiences of all other providers. The criticism that surfaced at the hui resulted in the evaluation findings being disregarded by a key stakeholder group.

In summary, Programme X achieved partial success in implementing the Guidelines, but it also illustrated some endemic failures. The intention of observance" was evident in how the evaluators developed a number of work-arounds and innovative, on-the-run solutions as opportunities presented themselves. Overall, however, Programme X, like many other evaluations, exemplifies an apparent "failure of practice" in the face of ongoing challenges. In the next section some of these challenges are discussed and questions for further debate are put forward.

Discussion

What Authority do the Guidelines have?

The first point is that neither the evaluation team nor the commissioning agency were held to account for falling far short of the Guidelines, both in terms of what was planned and what was implemented. No one questioned the lack of consultation with community stakeholders in the evaluation design, and the design was not openly questioned or criticised by any of the government, community or evaluation stakeholders directly involved in the evaluation. Furthermore, although the evaluation plan was independently reviewed by an experienced evaluator, and a Māori evaluation advisor provided support to the project team, the Guidelines were not an explicit point of reference for the evaluation. Ultimately, no major repercussions appeared to result from the independent evaluators' or the commissioning agency's approach other than that some providers dismissed the findings (when they were presented in a hui to both participants and non-participants) because their programmes were not included in the evaluation.

The SPEaR Best Practice Guidelines: Background Paper makes it clear that the guidelines are not, and should not be, prescriptive and that constraints are pervasive and unavoidable:

In government agencies, the decision to undertake a particular research or evaluation project usually does not lie with an individual researcher or policy analyst. Managers are accountable for finances and people resources. Most funding and in-house capacity is fully allocated to negotiated work programmes. A new project does not necessarily mean additional resources are readily available and priorities can change quite rapidly. This can be a challenge for research and evaluation ideals about lead times and coverage of breadth and depth… There will usually be a time line which is not very flexible, a set limit on the funds available and limits on the time "in-house" people can make available. (Good 2004:4)

One explanation for the observation silence is that the Guidelines have little authority and there is a general acceptance that "this is how it happens" in a government/provider contractual relationship. Although most parties to the evidence-building processes that now characterise government are aware of the need to observe ethical guidelines, a range of factors make such observance problematic. Among the factors that frustrate opportunities for observance are such things as requirements to participate, unrealistic ideas of what would constitutes sufficient time and resources, confusion over obligations under the Treaty on Waitangi (Treaty), the need for agencies to maintain control in order to manage political and fiscal risk, and the credibility of culture as a marker of robust research and evaluation practice. Each of these factors is considered in more detail below.

Requirements to Participate

Moewaka-Barnes (2003:148) notes that "consultation [in the context of evaluations] is undertaken because it is a requirement of a funding or ethics body". Thus, service providers can expect to be scrutinised by funding agencies both because it is a requirement and because ideas about financial accountability are given prominence in policy discourse. As Lunt (2003: 13) suggests, "a number of factors are exerting pressure for evidence-based activity … a key driver here is the continuing emphasis within the public sector to secure value for money". Community providers responsible for delivering Programme X had contracts that stipulated they participate in an evaluation. A provider's motive, or toleration, for participating in an evaluation may also include demonstrating the programme's success so they can secure ongoing funding.

Sufficient Time and Resources

Another reason for the partial or incomplete observance of the Guidelines is that few officials are aware of just how much time is "sufficient time" or the kind of resources necessary to allow more than a cursory consultation with community stakeholders. The gap between intention and responsiveness may be both unavoidable and unintentional and, at the end of the day, the deadlines for reporting are a more persuasive motivation than perfect process. The United Kingdom Evaluation Society (2003) suggests that:

It would be helpful if commissioners:

• hold preliminary consultations with all parties to the evaluation to support a relevant, realistic and viable specification
• clarify the constraints that commissioners operate under, e.g. timescales, budgets, deadlines, and accountability.

In the case of Programme X, a meeting was held between a staff member from the commissioning agency, the evaluator and a Te Puni Kokiri analyst, but there was no budget or time allocated to involve community stakeholders in the design phase.

Obligations under the Treaty

The lack of clarity about government's rights and obligations under the Treaty⁶  may also create confusion for staff from commissioning agencies. Two versions of the Treaty exist: an English-language and a Māori-language version. The two versions are not precise translations of each other. This has been the source of ongoing tension and debate, more recently in relation to the Crown's obligations toward Māori in the social policy area (Barrett and Connelly-Stone 1998). Tauri (2004) notes, for example, that the English-language version of Article 1 of the Treaty is interpreted as providing the Crown with the right to govern and make laws. Tauri's view is that government agencies use this interpretation to underpin the legitimacy of their research and evaluation units to undertake research and evaluation involving Māori:

Article 1 is also interpreted as providing Government and, by extension, agencies and officials with the authority to set the policy and research agendas, including deciding what to research and on what issues Government's research spend will be focused. (Tauri 2004:8)

However, this interpretation is not shared by proponents of kaupapa Māori research, which is based on a premise that the Treaty gives Māori the authority to control the research (and, by implication, evaluation) process (Pipi et al. 2004, Tauri 2004). A key component of a kaupapa Māori approach is meaningful consultation with Māori in the development of the evaluation design. As Te Awekotuku (1991:17) has stated, "A researcher's responsibility, when working with people, is to the people themselves. This responsibility transcends sponsors; these individuals must come first."

Managing Risk

In the political and risk-averse environment in which most evaluation currently occurs, government agencies maintain tight control over the evaluation scope, design and reporting of findings. Publication of findings, for example, may not occur until the agency's chief executive officer, or the Minister responsible, has given their permission. There is a reluctance to publish any information that might portray an agency, or the current government, in a negative light. Reporting findings back to participants may take months, and in some cases never occurs (McKegg 2003). Such "risk management" processes may seem frustrating from an evaluation participant's perspective but are seen as both necessary and expedient from a policy perspective.

Credibility of Culture

When government agencies fail to involve Māori stakeholders, they risk designing and conducting evaluations and delivering evaluative information that are not credible to Māori. As Johnson et al. (2008) state, culturally appropriate processes are not an end in themselves but a means to an end. They encourage evaluation to be "conducted in a way that supports credibility, that data reported accurately represent the phenomena of interest, and that interpretations are trustworthy" (Johnson et al. 2008:201).

It is in a government's interest to obtain evaluative information that derives from rigorous, robust evaluation designs. Definitions of rigour and robustness, however, are socially constructed and subject to culturally distinct interpretations. As Smith (2005), in her discussion of the "tricky ground" on which research practice is constituted, suggests:

It is considered [by indigenous people] a sign of success when the Western world, through one of its institutions, pauses even momentarily to consider an alternative possibility. Indigenous research actively seeks to extend that momentary pause into genuine engagement with indigenous communities and alternative ways of seeking to live with and in the world. (Smith 2005:103)

The credibility and robustness of evaluation practice, when enacted against a set of well-crafted guidelines, may constitute such a pause, leading to genuine engagement.

Summary

Although the review of practices implemented in one government-sponsored evaluation has identified a number of structural barriers that were beyond the evaluation team's control, it has also highlighted gaps in the evaluation team's ability to work with Māori. These capability issues have been noted by Duignan (2002) and the State Services Commission, which has stated that:

… officials commissioning and using evaluative activity to inform policy often do not have an adequate understanding of tikanga and te reo Māori or of Māori research and evaluation methodology. (State Services Commission 2003:29)

Professional evaluation organisations can play an important role in supporting members to develop their cultural capability. The Aotearoa New Zealand Evaluation Association (ANZEA) has as its goals to:

Promote and facilitate the development of evaluation practices and standards which are relevant to Aotearoa New Zealand, with particular reference to the principles and obligations established by Te Tiriti o Waitangi and reflecting the unique bi-cultural context of Aotearoa New Zealand, while also providing a framework from which multi-culturalism can be embraced and responded to. (ANZEA website 2008)

The Australasian Evaluation Society (AES) also has an agenda to develop evaluators' capability to practise appropriately within indigenous contexts (Wehipeihana, 2008; Scougill 2006) as does, in a less direct way, the American Evaluation Association in the discussion and debate that underpins the Cultural Reading of the Program Evaluation Standards (AEA 2004).

Conclusions

The SPEaR Guidelines are a useful resource for contractors working in the government sector, not just at the planning stage but throughout an evaluation and as a framework for reflecting on current practice. The final version, refined through an extensive consultation process, offers complex and demanding material for consideration. Despite the existence of such guidelines, however, there are a number of challenges that make it difficult to implement good practice approaches when conducting government evaluations where Māori are a significant subgroup. These challenges include practical considerations such as budget and time constraints, and lack of clarity about what consultation is required when involving Māori. More subtle challenges are evident in unspoken requirements for groups to participate in evaluations, or for evaluators to deliver robust findings. Complex institutional challenges exist in New Zealand in relation to obligations under the Treaty, and the extent to which agencies will invest authority in culturally responsive guidelines. There are political challenges in the need for agencies to maintain control in order to manage political and fiscal risk.

At an individual level, contractors are expected to have a wide range of competencies, one of which is the ability to be able to engage with and involve Māori individuals, families and communities in ways that are authentic, appropriate and ethical. As this review has highlighted, an evaluation team may proceed with good intentions but difficulties encountered at different stages of the process mean the ideal is seldom realised.

The dissemination of thoughtful and well-crafted guidelines is one necessary strategy to mitigate the failure of practice, but it is not sufficient. Evaluators need to reflect on their practice, share their experiences and explore new approaches to evaluation. New approaches may align with or surpass existing practice guidelines. If government is serious about promoting best practice approaches, there needs to be structural change in the ways evaluations are currently conceived and commissioned. Debate and dialogue about the nature and role of culturally appropriate practice are critical.

References

American Evaluation Association (2004) A Cultural Reading of the Program Evaluation Standards (2nd ed), Final Report of the Diversity Committee American Evaluation Association, [accessed 06/11/08].

Aotearoa New Zealand Evaluation Association (2007) Constitution of the Aotearoa New Zealand Evaluation Association Incorporated (ANZEA) www.anzea.org.nz/ [accessed 1/5/08].

Aotearoa New Zealand Evaluation Association (2008) ANZEA’s goals www.anzea.org.nz/about.htm [accessed 1/5/08].

Australasian Evaluation Society Inc. (2002) Guidelines for the Ethical Conduct of Evaluations, Australasian Evaluation Society, Canberra, ACT.

Barrett, M., and K. Connolly-Stone (1998) “The Treaty of Waitangi and social policy” Social Policy Journal of New Zealand, 11, http://www.msd.govt.nz/about-msd-and-our-work/publications-resources/journals-and-magazines/social-policy-journal/spj11/treaty-of-waitangi-and-social-policy.html [accessed 31/1/08].

Belgrave, M., M. Kawharu and D. Williams (2005) Waitangi Revisited: Perspectives on the Treaty of Waitangi, Oxford University Press, Melbourne.

Duignan, P. (2002) “Building social policy evaluation capacity” Social Policy Journal of New Zealand, 19:179--194.

Health Research Council (1998) Guidelines for Researchers on Health Research Involving Mäori, Health Research Council of New Zealand, Auckland.

Johnson, E., Kirkhart, K., Madison, A., Noley, G., and G. Solano-Flores (2007) “The impact of narrow views of scientific rigor on evaluation practices of underrepresented groups” in N. Smith and P. Brandon (eds.) Fundamental Issues in Evaluation, Guildford Press, New York.

Lunt, N. (2003) “Knowledge for policy: The emergence of evaluation research within New Zealand” in N. Lunt, C. Davidson and K. McKegg (eds.) Evaluating Policy and Practice: A New Zealand Reader, Pearson, Auckland.

McKegg, K. (2003) “From margins to mainstream: The importance of people and context in evaluation utilisation” in N. Lunt, C. Davidson and K. McKegg (eds.) Evaluating Policy and Practice: A New Zealand Reader, Pearson, Auckland.

Ministry of Social Development (2004) Ngā Ara Tohutohu Rangahau Māori: Guidelines for Research and Evaluation with Māori, Centre for Social Research and Evaluation, Wellington.

Moewaka Barnes, H. (2003) “Māori and evaluation: Some issues to consider” in N. Lunt, C. Davidson and K. McKegg (eds.) Evaluating Policy and Practice: A New Zealand Reader, Pearson, Auckland.

Pipi, K., F. Cram, R. Hawke, S. Hawke, T.M. Huriwai, T. Mataki, et al. (2004) “A research ethic for studying Māori and iwi provider success” Social Policy Journal of New Zealand, 23:141--153.

Scougall, J. (2006) “Reconciling tensions between principles and practice in Indigenous evaluation”, Evaluation Journal of Australasia, 6:2 49-55.

Smith, L.T. (2005) “On tricky ground: Researching the native in the age of uncertainty”. in N. Denzin and Y. Lincoln (eds.) The SAGE Handbook of Qualitative Research, Sage, Thousand Oaks, CA.

SPEaR (Social Policy Evaluation and Research) Committee (n.d.) Purpose of the Social Policy Evaluation and Research (SPEaR) Committee, www.spear.govt.nz/about-us/index.html [accessed 22/10/07].

SPEaR (Social Policy Evaluation and Research) Committee (2001) Terms of Reference http://www.spear.govt.nz/about-us/what-is-spear/tor.html [accessed 22/10/07].

SPEaR (Social Policy Evaluation and Research) Committee Working Party (2005) Draft SPEaR Good Practice Guidelines Package for Workshop, 10--11 May 2005, Wellington.

SPEaR (Social Policy Evaluation and Research) Committee (2008) Good Practice Guidelines. http://www.spear.govt.nz/documents/good-practice/good-practice-guidelines-june-2008-final-version.pdf [accessed 6/11/06].

State Services Commission (2003) Doing the Right Things and Doing Them Right: Improving Evaluative Activity in the New Zealand State Sector, State Services Commission, Wellington, www.ssc.govt.nz/display/document.asp?DocID=3507 [accessed 19/5/06].

Tauri, J. (2004) “Key issues in the development of government agency guidelines for research and evaluation with Māori” paper presented at the Australasian Evaluation Society 2004 international conference, Adelaide, 13--15 October.

Te Awekotuku, N. (1991) He Tikanga Whekaro – Research Ethics in the Māori Community: A Discussion Paper, Ministry of Māori Affairs, Wellington.

Te Puni Kōkiri (1999) Evaluation for Māori: Guidelines for Government Agencies,a report prepared by Aroturuki me te Arotakenga Monitoring and Evaluation Branch, Te Puni Kōkiri, Wellington.

UK Evaluation Society (2003) Guidelines for Good Practice in Evaluation,

Wehipeihana, N. (2008) “Indigenous evaluation: A strategic objective of the Australasian Evaluation Society”, Evaluation Journal of Australasia, 8:1 40-44.

Footnotes

¹Acknowledgements
The authors would like to acknowledge colleagues in Massey University's Post Graduate Diploma in Social Sector Evaluation Research among whom debate and discussion on topics such as this take place.
Correspondence:Mathea Roorda, evalue research ltd., Wellington. Mathea@paradise.net.nz
Robin Peace, School of People, Environment and Planning, Massey University, Wellington campus.

²These guidelines sit within a broader programme of Good Practice Guidelines for social policy research and evaluation, undertaken and commissioned by New Zealand government agencies. The draft guidelines (SPEaR 2005), publicised on the MSD website and in a number of conferences and hui, were the subject of extensive consultation. The formally approved guidelines (2008) are available on the SPEaR website

³Note that, for brevity, some of the wording in the table has been abridged. The full text of the Guidelines can be found on the SPEaR website (www.spear.govt.nz/) and the full text version appears in the discussion. The final version of the guidelines will also be published on this site.

⁴The SPEaR Guidelines do not define “participant” but identifies them in the context of “many stakeholders in social policy research and evaluation - public servants, academics, students, private sector or third sector/NGO researchers and evaluators, research participants or communities of interest” (2008:5).. The authors assume the term also refers to those who answer the questions in an evaluation (otherwise known as “respondents”), and may also include those who are more passively involved in an evaluation (sometimes known as “subjects” or “stakeholders”).

⁵Māori customs and values

⁶The Treaty of Waitangi was signed by representatives of the British Crown and New Zealand’s Māori chiefs (rangatira) in 1840. For a more detailed account of the Treaty and perspectives on the Treaty, the reader is referred to Belgrave et al.(2005).