The Measurement Of Training Opportunities Course Outcomes: An Effective Policy Tool?

Ottilie Stolte¹
Department of Geography
University of Waikato

Abstract

Training Opportunities is an active labour market policy initiative, and part of a response to the entrenched problems of unemployment in Aotearoa New Zealand. The funding and implementation of Training Opportunities are determined in part by a particular system for measuring course outcomes. This paper argues that this measuring system should not be used for policy development, due to measurement errors and problems assigning causality to the intervention. Consequently, various disincentives arise that contradict the objectives of Training Opportunities. While accountability is important, the over-reliance on the narrowly defined Training Opportunities outcomes undermines the ability of providers to assist the unemployed, and thereby contribute to the policy goals of reducing unemployment and labour market disadvantage in New Zealand.

Introduction

Despite varying levels of economic growth, unemployment and underemployment² are now entrenched features of Aotearoa New Zealand society. One active labour market policy that offers direct assistance to specified groups of unemployed adults is Training Opportunities. ³ This training policy aims to address unemployment by assisting individuals to overcome their impediments to full labour market participation. Like most other areas of public spending, Training Opportunities is under constant scrutiny and must operate in accordance with business accountability principles.

The prime focus of Training Opportunities is to assist learners to acquire a critical bundle of foundation skills that will enable them to move effectively into sustainable employment and/or higher levels of tertiary education.

This paper focuses on two aspects of research conducted on Training Opportunities (Stolte 2001). First, the paper considers the emergence of Training Opportunities as a particular response to unemployment in New Zealand. A brief historical context is followed by an outline of the operational context for the policy initiative. Second, this paper argues that the particular design of policy mechanisms for Training Opportunities is counterproductive to broader policy goals. Specifically, the paper asks whether the current outcome measurement system is an appropriate tool for policy development to ensure that these courses are a prudent and constructive response to unemployment. This is an important question not only because of the need for accountability, but also because of the persistence of unemployment and labour market disadvantage.

Background: A Study Of Training Opportunities

The research informing this paper grew out of a concern about unemployment in New Zealand and the failure of economic growth as a "solution" to unemployment. Bertram (1988) and Higgins (1997) assert that there is a need for scholarly investigation of the policy frameworks and mechanisms for employment assistance and training programmes in New Zealand. In particular, they stress the need for work at a practical level that draws on participatory research approaches.

When the research began in 2000, Training Opportunities was seen as a major form of employment assistance available to unemployed adults. The pilot phase of the research indicated that course outcomes were a controversial issue for both the providers and funders of Training Opportunities. A qualitative methodology appeared most useful to investigate why there were problems with the system for measuring outcomes. The research combined case studies, discourse analysis and theoretical engagement.⁴  The fieldwork included open-ended key informant interviewing with funders and providers. Informal encounters and participant observation occurred in various training course settings. These approaches were less viable within the public sector organisations, so the research contained a somewhat greater focus on the providers' experiences. Secondary data were derived from policy documents, media releases and government publications.

Due to time and resource constraints, most of the fieldwork was located in Hamilton, although I did visit Wellington on two occasions to discuss the research with staff in central government agencies.

Context: The Emergence Of Training Opportunities

This section provides the context for the emergence of Training Opportunities as a historically specific response to the problem of unemployment in Aotearoa New Zealand (Wallace 1998). Since the 1970s many different employment assistance initiatives and training schemes have been developed in response to the persistence of unemployment.

The Entrenchment of Unemployment in Aotearoa New Zealand

The political and economic reforms that began after the 1984 elections were based on the assurance that principles of free-market economics would lead to growth and prosperity, and would eventually reduce unemployment. Yet between 1984 and 1998 economic growth remained virtually static and the unemployment rate doubled (Chatterjee 1999:65-67).

Many academics and commentators have detailed the entrenchment of unemployment in New Zealand since the 1970s (Easton 1989, 1997, Green 1994, Kelsey 1993, 1995, Morrison 1991, Waldegrave and Coventry 1987). In contrast, more optimistic reports claim that unemployment rates are declining overall and that the labour market is expanding into new areas (Brash 2000, Cocrombe et al. 1991, Rose 1990). Another group of researchers, however, highlight the increasing prevalence of underemployment, labour market disadvantage and income inequality, arguing that this prevalence often escapes the variables currently used in labour market research (Briar 2000, Clogg 1979, Callister 2001, Easton 1996, 1997, Martin 2000, Peace 1999, Waldegrave 1998). In effect, they say, the possibility of stable and reasonable employment conditions is now an elusive goal for large groups of the population.

The increase of casual and impermanent work means that many people officially recorded as being "in the labour force" are faced with frequent occurrences of poverty and disadvantage (Brown and Scase 1991). In Aotearoa New Zealand being unemployed or underemployed usually means being poor (Waldegrave and Coventry 1987). The logical progression is that assisting people to move towards more stable forms of employment becomes a crucial factor in addressing poverty and creating a more inclusive society. However, radical social theorists such as Beck (2000) and Gorz (1999) highlight the irony of basing social inclusion on paid employment in a society where full-time paid work is a dwindling prospect. Can the Training Opportunities providers be expected to produce employment outcomes when there are too few jobs?

Training Opportunities

In 1993 the Training Opportunities Programme (TOP) was introduced to replace ACCESS, which involved employment schemes for low-skilled workers.⁵  The decreased availability of low-skilled work undermined the rationale of ACCESS. Although such work schemes were of some value to participants, they were questioned overall because what was the point in training people for jobs that did not exist (O'Connor 1983)? In response to the impasse, employment assistance policies were realigned towards improving the individual performance of participants in the labour market. The TOP courses were introduced as an integral part of the National government's education strategy, which was aimed at:

… raising achievement levels; increasing the participation of under-represented groups and individuals in education and training; increasing opportunities in the post-school sector; and ensuring that the system is more responsive to changing needs. (ETSA 1992:8)

The TOP courses involved a greater emphasis on learning job-seeker strategies,⁶ improving general work attitude⁷ and providing entry-level skills targeted at areas of high labour demand. For some trainees, Training Opportunities served as a stepping-stone to further training and education (ACNielsen 1999).⁸

In their current form, the Training Opportunities courses are designed to assist people who are regarded to be at risk of "long-term unemployment" (DWI/WINZ 2000). These training courses are delivered by private training establishments (referred to as providers in this paper) -- organisations associated with churches, iwi groups, community trusts; some private businesses; educational institutions such as polytechnics; and employers. The providers must be approved and registered with the New Zealand Qualifications Authority before they can tender for a contract to deliver a course (ETSA 1992). The Training Opportunities providers offer an adult learning environment for skills acquisition and to build self-esteem and confidence. In 2001, there were 413 training providers; as at 31 July 2001 there were 9,043 trainees, with 21,600 participating over the calendar year (Skill New Zealand 2001). The total cost of Training Opportunities for the year was $94 million (Statistics New Zealand 2001).

The Training Opportunities courses are "targeted", with criteria intended to prevent disadvantaged individuals from being "crowded out" by less disadvantaged people (Ministry of Education 2002). The eligibility criteria are:

• Aged 18--19 years with low qualifications, left school in the last 26 weeks and registered with Work and Income;⁹ or
• Registered with Work and Income as an unemployed job seeker for at least 26 weeks, with low qualifications; or
• Registered with Work and Income for fewer than 26 weeks, with low qualifications and assessed by Work and Income as being at risk of long-term unemployment; or
• Registered with Work and Income as an unemployed job seeker for at least 26 weeks, with more than two School Certificate passes or more than 40 credits and assessed by Work and Income as lacking foundation skills; or
• Has Refugee status, with higher qualifications and registered with Work and Income; or
• Participated in Youth Training in the last three months and granted approval by the TEC to enter Training Opportunities to complete training.
• Low qualifications are generally defined as no more than two School Certificate passes and no qualification higher than Sixth Form Certificate.

The Training Opportunities Outcome Measurement System

The following is a brief explanation of the particular process for the measurement of Training Opportunities course outcomes. Collecting the outcome measurements is the responsibility of the provider. The first requirement is that exactly two months after the completion of a course, the provider must contact all ex-trainees to ask about their employment status. Once (and if) the ex-trainees are contacted, the responses given need to be coded according to the outcome categories determined by the funding agency (the Tertiary Education Commission, formerly Skill New Zealand) and recorded on the 2-Month Labour Market Outcome Form (Figure 1). The preferred outcome category, according to my informants at the time of the research, was full-time employment. Following each course, the provider had to achieve a quota of employment-type outcomes. There was also an allocation for education or further training outcomes.

Figure 1 The Outcome Measurement Form: "Two- Month Labour Market Outcome Form" Download file under Documents. Source: Courtesy of TEC

The category classifications of the outcomes are precise. For instance, to register for a "further education" outcome, the individual must have been participating in education on the actual day contacted. If the individual is enrolled but the course has not started, they should be classed as a "non-successful" outcome. At the time of the research, the outcomes supplied the primary information that Skill New Zealand considered in their purchase of Training Opportunities courses. Although the providers maintained narrative accounts of the trainees' progress, the "snap-shot" two-month outcome results were treated as the main determinant of a provider's effectiveness.

A major problem for the providers I interviewed was trying to track down ex-trainees who may move house frequently, often in search of work, although one provider mentioned that the popularity of cell phones made it easier to locate people. Failing to contact an ex-trainee results in a failed outcome for the provider. Another provider mentioned that on several occasions trainees who had grievances had "got back at him", either by not maintaining contact with the organisation or by lying about their employment status. Whereas the providers are expected to "be tough" on the trainees and not tolerate any inappropriate behaviour (for example, absenteeism), they are also dependent on the goodwill of the trainees to obtain the necessary post-course outcomes.¹⁰

The Administration And Funding Of Training Opportunities

The Education Training and Support Agency (ETSA) launched the Training Opportunities Programme (TOP) courses in 1993. The initial vision of TOP was to provide a stepping-stone for people who had not succeeded in mainstream education, or had experienced some kind of obstacle to participating fully in the labour market (ACNielsen 1999). In 1998, ETSA became Skill New Zealand Pükenga Aotearoa. Changes to the implementation of TOP courses resulted from a Cabinet decision in 1998 to split TOP into two separate programmes: Youth Training and Training Opportunities. The budget for Training Opportunities transferred to Vote:Employment and was administered by Skill New Zealand under a Memorandum of Understanding with Work & Income. Youth Training remained under Vote:Education. At the same time, part of the TOP budget was transferred to Work and Income for discretionary use by Work & Income regions. From the beginning of 1999 the TOP courses were replaced by Training Opportunities, in the case of programmes aimed at clients of Work and Income, and Youth Training, in the case of programmes directed at school leavers lacking foundation skills. (Skill New Zealand has since been absorbed into the Tertiary Employment Commission.)

Public Sector Changes

During the 1980s and 1990s the New Zealand public sector underwent rapid and far-reaching changes. Economic decline and a general dissatisfaction with the public sector (perceived to be overly bureaucratic and wasteful) led to the introduction of business accountability principles and financial management techniques. A succession of legislation changed the operation of the public sector, including the operation of Training Opportunities. These reforms are outlined in government publications (Schick 1996, Audit Office 1989, Treasury 1989, 1996) and are examined in many other sources (Ball 1987, Clarke 1990, Boston et al. 1996, Scott 2001, Tozer and Hamilton 1998).

Accountability: Measuring Efficiency and Effectiveness

In the training sector the cause for the new regime of accountability began with an increase in financial statement auditing. However, central government agencies voiced the concern that public sector organisations focused on meeting budgets, rather than seeking more innovative ways to deliver policy (The Audit Office 1989, Treasury 1989). Previously, public sector auditing involved the reporting of inputs (resources) and outputs (the products or services delivered). An extreme focus on financial accountability could just lead to agencies very efficiently producing things that were not needed (Ball 1992). These arguments advanced the cause for new ways to measure the effectiveness of what an organisation does, in terms of its effects on society. The concept of the outcome was introduced to measure the effects of a policy, and to determine whether the outputs of agencies (such as training providers) were aligned with overall policy goals. Consequently, the financial management techniques used to measure inputs and outputs (which are usually cost-based) were transferred to the (non-financial) notion of effectiveness, to create the measurement criteria for outcomes.

To begin with, it was envisaged that outcomes would be useful at a policy decision-making level. First, the government would determine its outcome priorities; for instance, reducing unemployment. Second, by drawing on policy advice and analyses of the relationships between outputs and outcomes, the government could select the most appropriate outputs. Before making decisions about what active labour market policies to fund, the government could take expert advice to ascertain the range of interventions most likely to help individuals and communities address unemployment. The assumption was that the competent delivery of interventions (such as Training Opportunities) would remedy unemployment. Third, the success of the government's strategies would then (it was hoped) be reflected in improvements in national measures such as the unemployment rate.

Empirical Findings: Those Controversial Outcomes

During the fieldwork it became apparent that the measurement of outcomes was a contentious issue. When the Training Opportunities providers in the study were queried about their main concerns they invariably raised frustrations with the outcomes, both in terms of the way outcomes were measured and how the outcomes were often a poor reflection of the work they did with the trainees. The providers felt that there were too few opportunities for them to express their concerns and suggestions. They felt that their professional ability and on-the-ground knowledge of the specific issues in their regions were sidelined. Despite the difficulties, many providers were determined to continue to provide training of some sort, because they saw a clear need for their services in their communities. The government employees interviewed emphasised the importance of outcomes as the principal tool for decision making and for providing evidence of operations. Their responses included the "success stories" (the providers with "good" outcomes) and the "poor" providers who failed to adapt and perform under the new accountability systems.

The Pressure to Produce "Good" Outcomes

Since the providers must secure funding on a contract-by-contract basis, it is fundamental to their continued operation that they fulfil the expectations of the funder (Skill New Zealand) by producing "good outcomes". At the time of the research a good labour market outcome meant that the ex-trainee was in a job or further education on the exact day of the post-course measurement (i.e. two months after leaving a course). The system provided no way to measure or take into account the quality or durability of the employment, but the dominant assumption was that being in a job was the best outcome.¹¹

In the course of the fieldwork, providers made the point that (due to the targeting criteria) many people eligible for Training Opportunities courses cannot be easily or quickly made employable because of their educational or skills disadvantage in relation to the rest of the workforce. Furthermore, local labour market fluctuations affect the availability of jobs at various times, which can influence employment outcomes positively or negatively. Several providers maintained that when unemployment rates are low in their region the work of training providers is harder because the more employable people are absorbed into the workforce, leaving the more difficult cases.

An important measure of success for training courses such as TOP is employment, yet many individuals who are targeted through the eligibility criteria face a combination of personal, health or social problems in addition to a lack of vocational skills. The increasing emphasis on employment outcomes has been accompanied by an increasing emphasis on the screening of trainees. A discussion paper on youth training emphasised that training providers need to refine their selection processes to ensure that the trainees recruited are "ready to make progress" (Skill New Zealand 2000a:20). The report commented that training providers "cannot be all things to all people", therefore they need to specialise in catering for particular groups.

Many of the providers contacted in the study have found this "selection" of trainees problematic. First, potential trainees may have problems of a personal nature (for example, domestic abuse or health issues), which can be difficult to ascertain in a selection process. Secondly, trainees may face various combinations of issues, so it can be hard for providers to "specialise" in dealing with only one type of issue. Third, the providers repeatedly raised the frustration that they were not getting enough referrals at the right times, and that they were pressured to take "less-suitable" applicants to maintain course occupancy.¹²  Fourth, several providers raised concerns about the long-term consequences to society if they continually turn these "difficult" people away. While training courses are intended to deal specifically with employment-related issues, they are often also a time when deeper problems can surface and be worked on. I asked a government employee if unemployed people are likely to seek help for personal issues themselves. Her reply:

"It's highly unlikely eh! They don't have the motivation or the confidence. If you don't grab them while you have them held captive on a course, then you lose an opportunity that the learner won't, more than likely, take up of their own accord."

Clearly the time and resources required to assist long-term unemployed and low-skilled individuals varies greatly. While the providers recited success stories, these successes had often taken considerable time and effort. Despite these difficulties, the providers contacted were committed to their work as they saw the courses as an important step for assisting people out of a cycle of deprivation. The providers may see the need within their communities to offer assistance to the most disadvantaged people (especially youth). Yet, to stay in business providers are required to select the less disadvantaged of the disadvantaged.

Training Opportunities providers can offer a supply-side aspect of active labour market policy. Apart from addressing the obstacles to labour market participation and developing "work-readiness", the role of a training provider involves connecting (or reconnecting) the trainee with a range of opportunities in the form of further education or training and work experience, with the final goal of stable employment.

Measurement Problems

All of the interviewees agreed about the need for accountability, but the providers argued that the drive for short-term employment gains must be weighed against the possibility of a long-term focus towards stable employment. The providers interviewed believed that understanding the various situations of unemployed people is important in identifying and addressing obstacles to employment. They observed that many of the "good outcomes" involved casual, temporary, low-quality job placements. While employment in a "MacJob"¹³ ensures a good outcome and provides some work experience, several providers commented that they would prefer to encourage trainees to consider further education, training, voluntary work and other strategies that could lead to long-term career development. With the rigid two-month post-course outcome measurement and the lack of continuous funding for post-placement support, such goals become unrealistic. The providers felt they were forced to push their trainees into any job (to stay in business) rather than being able to set the trainees on a path towards sustainable employment.

The providers in the study agreed that the two-month time frame was too short. Mary (Interview 5, Stolte 2001) commented:

"I guess that they [Skill New Zealand] argue that if it is at one month, two months, three months or five months, makes no difference! Well, no, I'm simply saying that from our experience our results tend to show better, more positive results with the six-month period."

Peter (Interview 1, Stolte 2001) also expressed with some amusement that at times he felt the outcome measurements were quite inaccurate and irrelevant:

"We've had instances where Skill New Zealand rings up and asks what's the outcome of this person? And we say "they are employed full-time" and that week it's true (laughs) because they had a one-week job. All they want to know is are they in full-time employment. And we will say yes. We could get them vacuuming the carpet for a week!"

Peter was, however, less cheerful about the reality of the system that placed his organisation under constant threat: "It comes down to money in the end because if we don't meet the specific outcomes that they set [for] us, then they [Skill New Zealand] can withdraw funding". Peter also referred to anecdotal instances of less scrupulous providers who were "cheating" with their outcomes. This cheating (also referred to by other providers) was conducted either through dishonest reporting of outcome results, or by arranging employment for ex-trainees on the days that outcomes needed to be recorded. Clearly this cheating needs to be eliminated, as it undermines the efforts of the honest providers. Some auditing of results exists, but increased monitoring and surveillance of the providers would be costly and is unlikely to be well received. It would be preferable to concentrate on designing more detailed assessments of the type, quality and duration of the jobs, or other post-course activities of the ex-trainees. This would provide incentives for a continual educative improvement of Training Opportunities. There was concern, however, that increased reporting and monitoring expends time and resources that are better spent on helping the trainees.

The providers were concerned that the pressure to meet outcome quotas forced them to "weigh-up" potential trainees to decide how easily they could be placed into employment. The providers all warned that this situation creates pressure to reject potential trainees who may not appear to be "good outcome prospects". While Training Opportunities "is targeted towards assisting those who are most disadvantaged in the labour market" (Skill New Zealand 2000b:10), the measurement of outcomes makes this policy goal extremely problematic. Paul (Interview 3, Stolte 2001) explained how difficult it is for his organisation to take on trainees from the health and disability sector:

"We used to work a lot with people referred to us from the health sector as a stepping-stone between one phase of life and another. And that proved our undoing in terms of the TOP programme, specifically all those Workbridge referrals. The person in that [funding] agency, she sat in front of me and said: 'You can't afford to take those kinds of people. Make a business decision!' That is exactly what happened!"

Peter, commented on his discomfort with the increasing pressure to screen potential trainees:

"It's not for us to decide. Who are we to say whether students can or can't get jobs? But, we filter them to see if they will fit on the course. Are they going to suit being in the class? Yeah, sometimes we end up having to take some of the students because we have no choice; we do that because we have a moral responsibility."

It could be argued that Peter's "moral responsibility" is to not take such clients. If the primary objective of Training Opportunities is getting people into employment and if this client is unlikely to become employed (in the specified period), then the client may be better off elsewhere. The problem for the providers is that there are very few "elsewheres" to which these individuals can be referred. A year later I met Peter in the supermarket. He mentioned that he no longer worked as a training provider. When I enquired why he had left his old job, he gave an exasperated reply "those outcomes!" Peter had a wealth of experience in the training sector, but felt disillusioned about having to turn the people with the greatest needs away. By definition, most Training Opportunities trainees experience some kind of barrier to attaining employment -- otherwise they would not qualify for the programme. The measurement system for Training Opportunities fails to account for social issues, as though the simple act of attending courses and learning skills immediately transforms these unemployed individuals into work-ready job seekers (who automatically slot into the presumably available jobs).

All of the providers in the study felt that there were too few opportunities, independent of the contract negotiations, for them to express their views. The providers felt they were under some pressure to appear as an agreeable and "good" provider, which made it a lot harder to voice contentious ideas. This research was not a detailed examination of how public sector employees are constrained, although at the time it did appear that there were few opportunities for government employees to question and consider the limitations of the system.

Policy Goals And Consistency

This section explores the inconsistency between the Training Opportunities objectives and accountability mechanisms in relation to social policy. A public sector initiative such as Training Opportunities should, in theory, align with central policy goals. A major policy priority for the 1999 Labour government was social participation. The Minister of Social Services and Employment stated: "The key to social security is social participation." To achieve this requires the "removal of obstacles to employment" (Maharey 1999:6), "to ensure that individuals have the opportunity to earn income over their lifetime sufficient to meet their needs" (p.5). I believe the Minister's speeches to be in keeping with the promotion of employment assistance and active labour market policies (Maharey 1999, 2000).

Overall, the notion of social participation relies on policies that encourage individuals to extend themselves through education, skills development and paid employment. In August 2001 the Government embarked on a review of Training Opportunities and Youth Training, and circulated a consultation document (Ministry of Education 2001). The final report identified a need for courses that raise the "foundation skills"¹⁴  of individuals (who have not succeeded well in education), so that they may enter employment and participate more fully in society (Ministry of Education 2002).

Another policy priority for government is business accountability in the public sector. The use of financial management techniques is important for fiscal prudence and transparency. The systems devised to ensure financial accountability, however, should not obstruct social development goals. In its current form, the measurement system of Training Opportunities outcomes is not an appropriate tool for policy decisions, due to inherent problems that limit the possibility for the courses to improve social participation. Paul, a training provider summarised:

"We have these global objectives, which have high-minded vision and philosophical what-have-you. But we are using tools which just don't let it happen."

Rationales for Outcomes

The particular design of the system for measuring Training Opportunities outcomes was developed during the 1990s. The notion of the outcome is in essence a qualitative concept. It is also in this context a very political one. Furthermore, complex social outcomes are hard to define and are not easily translated into numerical values (Tozer and Hamilton 1998). Despite these difficulties, there were continuing debates during the 1990s about the need for outcomes as measures of effectiveness (Birch 1992). As a consequence, outcomes became a performance measure for individual agencies (for example, Training Opportunities providers) operating at the community level. The emphasis on the effectiveness and performance of public sector agencies was reinforced by the 1988 State Sector Act, which instituted the funder--provider split. The devolution of responsibility (and accountability) for the delivery of public services to the lowest level was a central aim. It was hoped that this would ensure that services would be more closely aligned with the needs of the people for whom they were intended.

The changes were good in theory. People working at the coalface often possess a good working knowledge of what is required in their particular local situation (Shuman 1998). Instead of a national blueprint for social service provision, individual agencies were contracted to provide services as they saw fit. Such an arrangement necessitated a clear prescription of responsibilities and expectations, which in many cases were represented by outcome statistics. However, the difficulty of translating complex notions of effects into outcomes has, in this case, led to measurement and causality problems (Ball 1992).

On the one hand, accountability is necessary to ensure that the recipients of employment assistance or training (for example, the unemployed) receive high-quality and appropriate services. The approved Training Opportunities providers have a responsibility to deliver excellent services for their clientele, and it should be well within their scope and expertise to achieve this. Therefore, providers should be assessed according to the benefits for the client groups.

On the other hand accountability is necessary to meet the general public's expectations of a cost-effective public sector that advances incremental improvements in society. Cost-effectiveness can be measured and managed relatively simply, and providers must report their income and expenditure meticulously. Consequential improvements in participants and communities are a lot harder to define and measure. Yet, this is what the outcome statistics represent. How can individual providers be responsible, and thus be made accountable, for wide social and economic trends and the actions of the unemployed individual? The providers do have a responsibility to provide high-quality services and their operational performance can be measured, certainly in terms of delivering qualifications or progress towards them. It is the complex, difficult-to-measure outcomes that involve multiple agents (including the subject and the choices he or she makes) that are problematical.

Causality Problems

The concept of the employment outcome is linked to assumptions made at a central government level about what will work. For example, the selection of a particular active labour market policy is, hypothetically, based on knowledge available about the correlations between outputs and outcomes in relation to the problem of unemployment. If a policy is implemented according to the highest quality standards and with consideration to the economic and social context in which it is launched and the suitability of the participants, then theoretically it should work and produce a positive effect on unemployment. If it does not, then there are a number of possibilities.

To start with, the initial assumptions could be wrong, meaning that the policy was designed from misleading information on outputs--outcomes correlations. Alternatively, the assumptions could be feasible, but exogenous factors may have affected the policy in unexpected ways. Exogenous factors are always exerting their influence over social development programmes, so perhaps the problem is a lack of analysis and anticipation of external influences. Another possibility is that despite high standards of policy delivery, the services failed in some way as a response to unemployment. In this case, further research should occur not only in the process or technical issues of the policy, but to see if the problems are part of broad intersectoral issues in society.

In the Training Opportunities sector the accountability mechanisms tend to focus on the performance of the provider. The provider is responsible for creating an employment outcome. While providers may be able to capitalise on their knowledge of the local employment situation, there are still many factors outside their control. The providers are in the business of delivering training, so this is what they should be measured on. The emphasis on employment outcomes does seem unrealistic, as the providers are not contracted or funded to create jobs. Another possibility is for a return to some kind of outputs-based funding focused on courses provided. An outputs model might be feasible if it were accompanied by readily accessible research on local labour market conditions and the training needs of the unemployed. This should help to ensure that training schemes are indeed relevant.

Gross Outcomes and Disincentives

The employment outcome as defined by Skill New Zealand is a poor operationalisation of the intended construct because it fails to record incremental changes resulting from the courses (although of course it does record progress towards qualifications). The outcome measures, however, do not reflect the differential characteristics of the trainees, or the amount of effort expended by the provider to assist them. Instead, the system creates disincentives for providers to help the more disadvantaged individuals for whom the courses are designed. In addition, gross outcomes disregard the educative quality of training. The providers felt pressured to focus on short-term employment gains rather than long-term personal and career development. The courses are often targeted to sectors that feature casual and temporary employment with high staff turnovers, so the trainees may not have significantly improved their overall position in the labour market.

The providers in the research were torn between their desire to help people and the pressure to secure funding. It is hardly surprising that some providers resort to "creaming", or selecting trainees who are more likely to produce the outcomes sought by Skill New Zealand. In addition, some providers thought that a relaxation of the Training Opportunities eligibility criteria was necessary, to ensure a more constant supply of suitable trainees for their courses. Paul, a provider, reveals the paradox between social policy goals and the policy mechanisms for Training Opportunities:

"They [Skill New Zealand] say welcome everybody, let's let everybody in. Let's all be enabling and empowering, but let's get these outcomes! So, where does that leave the training provider? Either to go with the objective and to go out of business, or to compromise and stay in business, and jettison people who are a liability."

Other Analyses of Training Opportunities

More in-depth and contextualised information is crucial for understanding the complexity of unemployment issues, and to take account of the various personal and external factors that have an impact on the outcomes. The providers interviewed suggested that qualitative information alongside outcome statistics could be invaluable to all those working with Training Opportunities to identify and respond to the various situations that occur, and to allow more informed decision making. However, the providers were concerned about being overburdened with paperwork, which leaves less time for working with the trainees.

Skill New Zealand conducted and commissioned various reports on its programmes (Skill New Zealand 1999, 2000a, 2000c, 2000d). One report documents the "success stories" of ex-trainees who are now employed or self-employed (Skill New Zealand 2000d). Another report, specifically on Training Opportunities, identifies some of the benefits and limitations in terms of what the courses do for the trainees (ACNielsen 1999). A report on Maori training raises the many challenges such as the failure of the mainstream education system and structural economic problems, and suggests that training courses on their own will not solve the problem (Skill New Zealand 1999). These qualitative reports are useful to strengthen the case for Training Opportunities at a central policy level, and create awareness of the broader context for the training sector.

In 2001, a Te Puni Kokiri Audit of Training Opportunities and Youth Training established that the courses are an important "second chance" education opportunity for Maori; however, the audit raised the concern that Maori were being "turned away" from courses because they were perceived to be less likely to produce "good outcomes".

Despite the concerns raised in the reports and the audit, the day-to-day funding decisions that affect individual providers are still based to a very great extent on outcome statistics (in addition to such factors as learner needs and local labour market needs). The qualitative reports have not generated significant improvements in the measurement or assessment of the courses.

Further Considerations

Clearly the current system for measuring the quality of Training Opportunities courses needs to be improved, to remove disincentives and more accurately reflect the incremental improvements of the trainees. While further research is required, a more workable system could be one that combined financial accountability of inputs and outputs with quantitative and qualitative information on the progress of each trainee. Making incremental improvements a focus of measurement could align with the policy goals of social participation. In the absence of job creation policies, the best training providers can do is to assist unemployed individuals to engage in positive activities and learning, with the intention that this will lead to some form of economic independence.

Employment assistance policy initiatives such as TOP courses are routinely subject to criticism in both public and academic debates. Aside from all the attention on the measurement and quality of courses, one could argue that Training Opportunities is a rather half-hearted response to unemployment and social exclusion. Martin (1998) suggests that active labour market policies are often more for show, rather than being a determined effort to address unemployment and social exclusion. The research for this paper suggests that training policies can have positive results for trainees, although these results often escape accountability measures. In the absence of other initiatives for individuals disadvantaged in the labour market, a more pressing issue is to assess how policies are applied and whether they meet community needs.

More regular and sustained attempts to include both providers and funders in decision making could help to improve relationships, which would be beneficial for the training sector as a whole. Unemployment and labour market disadvantage are serious and complex problems that require collaborative approaches, where employment assistance and/or training form one aspect of a coordinated active labour market policy response. Furthermore, care needs to be taken that the drive for competitiveness does not erode the consolidation of resources and expertise in the training sector.

O'Brien (1994:128) makes the simple but compelling comment that "unemployment and poverty are not merely technical issues".¹⁵  Technical measures, such as those used to define training course outcomes, are essential for central government to inform macro-scale decision making. Such measures must continually be reviewed for conceptual blind spots (Briar 2000). While various measures may be useful for accountability purposes, we should not lose sight of the realities behind the figures.

Conclusion

The purpose of this paper is to inquire whether the current system for measuring Training Opportunities outcomes is an appropriate tool for making policy decisions. My research findings indicate that the Training Opportunities outcome measurement system has low validity, and creates a number of distorted responses and side effects contrary to the intention of the policy and programme objectives. The final report on the review of Training Opportunities and Youth Training acknowledges that the two-month outcome measure is problematic, but recommends, "the measure should be retained in the interim" while the Ministry of Social Development develops measures for "sustainable employment" (Ministry of Education 2002:45). In addition, the review team recommended that there is a need to develop new measurement criteria for "foundation skills". The current outcome measurement system is still regarded as a central policy tool and there is a continuing emphasis on the "evaluation of results" (Skill New Zealand 2002:24).

From the interviews, participant observation and document research of Training Opportunities I have come to several conclusions. First, there is a consensus among providers and funders in this study that accountability mechanisms are important. Second, the current methods for measuring the Training Opportunities outcomes (on their own) do not necessarily provide accurate or useful accounts of effectiveness. Third, there is a need for improved accountability systems that also take causality into account the problematical nature of measuring employment outcomes. The providers I interviewed seemed to really care about quality, not just about securing the next funding contract, and were becoming increasingly demoralised by the current system. The danger is that these people will leave the sector and with them will go years of experience, knowledge and community networks.

The Training Opportunities outcome measurement system does not necessarily encourage integrity and quality, as the main incentive is to satisfy outcome criteria regardless of whether this is beneficial for the people involved. This paper emphasises the need to develop accountability methods that also foster social development. Furthermore, there is a need to position training and employment assistance within a broad framework of active labour market policy. The social and economic reality of unemployment and labour market disadvantage require comprehensive policy approaches that include a range of responses, to support the efforts of the many organisations and individuals sufficiently concerned about unemployment to take action.

References

Audit Office (1989) Central Government Management: A New Approach, Wellington.

ACNielsen (1999) Training Opportunities: Research into Longer-Term Outcomes, Skill New Zealand, Wellington.

Ball, I. (1987) Public Sector Accounting Concepts, New Zealand Society of Accountants, Wellington.

Ball, I. (1992) "Outcome specification" paper presented to Public Sector Convention, The Public Sector Challenge: Defining, Delivering and Reporting Performance, Wellington, November.

Beck, U. (2000) The Brave New World of Work, Polity Press, Cambridge.

Bertram, G. (1988) "Evaluation of employment programmes: a background paper" Discussion Paper No. 50, Department of Economics, Victoria University of Wellington.

Birch, W. (1992) "Political expectations of the public sector performance" paper presented to Public Sector Convention, The Public Sector Challenge: Defining, Delivering and Reporting Performance, Wellington, November.

Boston, J., J. Martin, J. Pallot and P. Walsh (1996) Public Management: The New Zealand Model, Oxford University Press, Auckland.

Brash, D.T. (2000) The Building Blocks of Economic Growth, The Reserve Bank of New Zealand, Wellington.

Briar, C. (2000) "In search of gender-sensitive concepts and measures of poverty, inequality and well-being" Social Policy Journal of New Zealand, 14.

Brown, P. and R. Scase (eds.) (1991) Poor Work: Disadvantage and the Division of Labour, Open University Press, Milton Keynes, Philadelphia.

Callister, P. (2001) "A polarization into work-rich and work-poor households in New Zealand? Trends from 1986 to 2000" Department of Labour Occasional Paper Series, Paper 3, http://www.lmpg.govt.nz.

Chatterjee, S. (ed.) (1999) The New Politics: A Third Way for New Zealand, Dunmore Press, Palmerston North.

Clarke, F. (1990) Public Sector Accounting Concepts Update, New Zealand Society of Accountants, Wellington.

Clogg, C.C. (1979) Measuring Underemployment: Demographic Measures for the United States, Academic Press, New York.

Crocombe, G.T., M.J. Enright and M.E. Porter (1991) Upgrading New Zealand's Competitive Advantage, Oxford University Press, Auckland.

DWI/WINZ (2000) Department of Work and Income website http:\\www.winz.govt.nz, accessed 15 March 2000.

Easton, B. (ed.) (1989) The Making of Rogernomics, Auckland University Press, Auckland.

Easton, B. (1996) "Some macroeconomics of the Employments Contracts Act" in P. Morrison (ed.) Labour, Employment and Work in New Zealand 1996, Victoria University.

Easton, B. (1997) In Stormy Seas: The Post-War New Zealand Economy, University of Otago Press, Dunedin.

ETSA (1992) "The Government's new Training Opportunities Programme" Education and Training Support Agency, Wellington.

Gorz, A. (1999) Reclaiming Work: Beyond the Wage-Based Society, Polity Press, Cambridge.

Green, P.F. (ed.) (1994) Studies in New Zealand Social Problems, Dunmore Press, Palmerston North.

Higgins, J. (1997) "No longer a burning social issue? Employment assistance policy and the closure of the unemployment debate in New Zealand" Australian and New Zealand Journal of Sociology, 33(2):137-152.

Kelsey, J. (1993) Rolling Back the State: Privatisation of Power in Aotearoa/New Zealand, Bridget Williams Books, Wellington.

Kelsey, J. (1995) The New Zealand Experiment: A World Model for Structural Adjustment? Auckland University Press with Bridget Williams Books, Auckland.

Maharey, S. (1999) "Labour's employment policy launch" Wellington, 26 October.

Maharey, S (2000) "Social exclusion: a threat to social cohesion" New Zealand Government Executive press release, Wellington, 9 August.

Martin, B. (2000) "Change in the distribution of income of the working age population 1976-96" Labour Market Bulletin, http://www.lmpg.govt.nz.

Martin, R. (1998) "Regional dimensions of Europe's unemployment crisis" in P. Lawless, R. Martin and S. Hardy (eds.) Unemployment and Social Exclusion, Regional Studies Association, London.

Ministry of Education (2001) "Building Futures: Training Opportunities and Youth Training Review: Preliminary Consultation Document" Wellington.

Ministry of Education (2002) "Building Futures: The Final Report on the Review Of Training Opportunities and Youth Training" Wellington

Morrison, P.S. (1991) "Job loss and unemployment in New Zealand during the 1980s" Discussion Paper, Department of Geography, Victoria University of Wellington.

O'Brien, M. (1994) "Ideology, poverty and unemployment" in P.F. Green (ed.) Studies in New Zealand Social Problems, Dunmore Press, Palmerston North.

O'Connor, M. (1983) The Fallacies of Work Skills and Youth Training: Once Bitten, Twice Shy, Employment Department, Auckland City Council.

Peace, R. (1999) "Surface tension: place/poverty/policy: from 'poverty' to 'social exclusion': implications of discursive shifts in European Union poverty policy 1975-1999" Doctoral thesis, Department of Geography, University of Waikato, Hamilton.

Rose, W.D. (1990) The Fully Employed High Income Society, New Zealand Planning Council, Wellington.

Scott, G. (2001) Public Management in New Zealand: Lessons and Challenges, New Zealand Business Roundtable, Wellington.

Schick, A. (1996) The Spirit of Reform: Managing the New Zealand State Sector in a Time of Change, State Services Commission and Treasury, Wellington.

Shuman, M. (1998) Going Local: Creating Self-Reliant Communities in a Global Age, Free Press, New York.

Skill New Zealand (1999) Meeting the Challenge: a Report on Innovative Maori Skills Training and Enterprises, Wellington, September.

Skill New Zealand (2000a) Building a Future for Young People, Skill New Zealand Discussion Papers, January, Wellington.

Skill New Zealand (2000b) Annual Report of the Education and Training Support Agency, Wellington, October.

Skill New Zealand (2000c) Pathways to Independence: Effective Youth Transitions, Skill New Zealand, Wellington, December.

Skill New Zealand (2000d) Journeys: 10 years of Skills, Innovation and Growth, Skill New Zealand, Wellington, July.

Skill New Zealand (2001) "Skill New Zealand Training Opportunities Update 2001" http://www.tec.govt.nz/downloads/a2z_publications/trainingopportunitesreport2001.pdf

Skill New Zealand (2002) Skill New Zealand Post-Election Briefing to the Minister of Education, Wellington, September.

Sklar, H. (1995) Chaos or Community? Seeking Solutions Not Scapegoats for Bad Economics, South End Press, Boston.

Statistics New Zealand (2001) Census of Population and Dwelling: Provisional Results, http://www.stats.govt.nz.

Stolte, O. (2001) "Outcomes, outliers outcasts: unemployment, employment assistance policy and TOP training courses in Hamilton, Aotearoa/New Zealand" Master's thesis, Department of Geography, University of Waikato, Hamilton.

Tozer, L.E. and F.S.B. Hamilton (1998) Public Sector Auditing in New Zealand: A Decade of Change, Department of Accountancy and Business Law, Massey University, Palmerston North.

Treasury (1989) Putting it Simply: An Explanatory Guide to Financial Management Reform, Wellington.

Treasury (1996) Putting It Together: An Explanatory Guide to the New Zealand Public Sector Financial Management System, Wellington.

Wallace, D. (1998) "Writing policy: from monologue to dialogue" Social Policy Journal of New Zealand, 10:58-69.

Waldegrave, C. (1998) "Balancing the three E's: equality, efficiency and employment" Social Policy Journal of New Zealand, 10:1-15.

Waldegrave, C. and R. Coventry (1987) Poor New Zealand: An Open Letter on Poverty, Platform Publishing, Wellington.

Footnotes

¹ Acknowledgements
This paper is based on my Master’s thesis (Stolte 2001). I appreciate the academic encouragement provided by the Department of Geography at the University of Waikato. A University of Waikato Master’s Scholarship supported the research. I gratefully acknowledge the valuable input of the research participants, Robyn Longhurst, David Neilson, James Swindells, Marlene Levine and two anonymous referees.
Correspondence
E-mail: oees@waikato.ac.nz
Readers’ feedback is welcome. I am working on my doctoral thesis, “Unemployment and training programmes: Reconciling business accountability with social development.”

² When referring to unemployment in this paper, the author acknowledges that there are various other forms of labour market disadvantage, such as underemployment, that generate social injustices.

³ Training Opportunities (for targeted clients of Work and Income) and Youth Training (for school leavers lacking foundation skills) replaced the former Training Opportunities Programme from 1999. There have been considerable changes to the programme since then, which are acknowledgeded in the course of the paper.

⁴ The fieldwork included case studies of three Training Opportunities providers and two government institutions associated with Training Opportunities. Data were derived over a three-month period from five in-depth interviews, several informal discussions and participant observation. The development of a theoretical framework guided analysis and drew from the work of feminist, political economic and social theorists.

⁵ Although the ACCESS schemes were designed to assist people into low-skilled work, this did not prevent individual providers from including broader educative approaches in the courses.

⁶ The job-seeker strategies included CV preparation, canvassing employers, improving individual presentation (for example, wardrobe selection and grooming) and practising interview responses and telephone manner.

⁷ Improving trainees’ “work attitudes” involved emphasising the importance of punctuality, a work ethic and responding positively to demands from employers and customers.

⁸ The Tertiary Education Commission commented that one of the most important factors distinguishing TOP from ACCESS was that TOP enabled learners to gain nationally recognised qualifications or credits towards them. In terms of outcomes, the programme always had a dual focus: the achievement of unit standards on the National Qualifications Framework and moving into work or further education and training (personal communication).

⁹ At the time of the original research, the Department of Work and Income administered the payments of benefits and encouraged job seekers to find work. In October 2001 the Department of Work and Income merged with the Ministry of Social Policy (the policy advisory body) to become the Ministry of Social Development.

¹⁰ According to Tertiary Education Commission in commenting on this paper, employment and entry into further education outside Training Opportunities are considered as completely equal outcomes. Although there may be have times in the past when more emphasis was placed on employment, even then, further training would have been accepted as a positive outcome. While at a policy level, employment and further training outcomes are treated equally, some regions may have negotiated particular targets for employment and training with providers as part of the contracts. Training Opportunities has always had a dual outcome focus -- achievement of credits towards national qualifications is also measured on an ongoing basis. Thus there is more to measuring the outcomes of the programme than simply looking at the labour market outcomes (personal communication).

¹¹ The Tertiary Education Commission points out that while it is true that at times, employment has been seen as the preferred option, nevertheless, in policy terms, further training was always given equal weight, and achievement of credits towards qualifications has always been one of the outcomes sought (personal communication).

¹² During the research period there was an increasing trend for providers to do their own advertising and networking, to reduce reliance on referrals. Even so, providers still complained that they often do not get enough “suitable” applicants.

¹³ The term “MacJob” has arisen with the dominance of global corporations such as the MacDonald’s fast-food chain. The interviewees used the term. Writers such as Sklar (1995) use “MacJobs” to refer to the growing number of casual, temporary and low-paid jobs that offer little in terms of personal fulfilment, security and advancement.

¹⁴ Foundation skills are defined as the generic skills such as literacy and numeracy that form the basis for further skill development and education. The final review report states that work is required to more accurately define foundation skills and develop suitable measures (Ministry of Education 2002).

¹⁵ O’Brien (1994) argues for the interconnections between “the poor” and “the unemployed”. I agree that there is overlap between the two terms, but recognise that the direct substitution of the one term for the other can be problematic.