Evaluation report - June 2014

Executive summary

This report details findings from the evaluation of Youth Service undertaken by Insights MSD, covering the Youth Service’s first 18 months of operation. The evaluation involved:

• an outcomes evaluation of young people’s participation in education, training or work-based learning, budgeting and parenting activities, and achievement of NCEA qualifications
• an impact assessment of Youth Service on young people’s benefit receipt, using a comparison group drawn from young people who started a main benefit before the introduction of the service
• a process evaluation of the Youth Service implementation, six months after it started, that assessed whether it operated as intended, what worked well, and what could have been improved.

Youth Service is designed to reduce the dependence of young people on the benefit system over time. Given the long-term nature of the changes sought from the service, the evaluation findings are preliminary and underestimate its full effect. In the 18 months covered by the evaluation (20 August 2012 to end March 2014) most young people had only a relatively short engagement with the service and impacts are yet to be seen for participants who began the service toward thfe end of the evaluation period. A more comprehensive assessment will be carried out in 2015.

Key findings

In the first 12 months of enrolment in Youth Service, Youth Payment (YP) and Young Parent Payment (YPP) participants are more likely to gain NCEA credits

Nearly two-thirds (63 per cent) of YP participants increased the number of NCEA credits they held in their first 12 months of participating in Youth Service, compared to one quarter (24 per cent) of their comparison group (Figure 7). Amongst YPP participants, over two-fifths (43 per cent) increased their number of NCEA credits in their first 12 months, compared to one-fifth (20 per cent) of their comparison group.

YP and YPP participants are more likely to meet the requirements[1] of NCEA Level 2 within their first 12 months in Youth Service

Within their first 12 months in Youth Service, 14 per cent of YP participants and 7 per cent of YPP participants met the requirements of NCEA Level 2. Over this period, both YP and YPP were more likely to meet the requirements for NCEA Level 2 relative to similar young beneficiaries before Youth Service was established (a nine percentage point gain for YP, and a two percentage point gain for YPP).

NEET[2] young people in Youth Service are achieving NCEA credits and qualifications

Of these young people:

• half increased their number of NCEA credits within 12 months of starting Youth Service
• 15 per cent met the requirements of NCEA Level 2 within their first 12 months in the Service
• 17 per cent had already met the requirements of NCEA Level 2 prior to starting the Service.

Early evidence suggests that Youth Payment participants spend less time on benefit under Youth Service

After one year, fewer Youth Payment (YP) participants are receiving a main benefit relative to similar young beneficiaries before Youth Service was established. These findings suggest that the service is improving YP participant’s prospects of moving off benefit. We expect this is due to participants taking up employment or full-time tertiary training.[3]

Fewer Youth Payment participants are transitioning to a working-age benefit

As a consequence of more YP participants exiting benefit after one year, the proportion of young people eligible for YP[4] who transition from a youth benefit onto a working-age main benefit has fallen by eight percentage points since Youth Service was implemented.

It is too early to assess the impact of Youth Service for Young Parent Payment[5]

At present, the number of Young Parent Payment (YPP) participants is too small to observe a meaningful impact on benefit receipt. Significant impacts are not anticipated until four to five years after teen parents start Youth Service, based on timeframes observed by the evaluation of the Training Incentive Allowance for sole parents. Because of their childcare responsibilities, we expect it is unlikely that many YPP participants can take full advantage of any gains in qualifications under Youth Service to move into employment until their children reach school age.

Under Youth Service, most young people are participating in education, training or work-based learning

At the end of March 2014, four out of five young people enrolled in Youth Service were engaged in an education, training or work-based learning activity. The majority of these young people were engaged in some form of education. The small number of participants who were not engaged in an education or training activity are likely to be new to the service and not yet had an activity assigned to them.

It can take several months to engage a young person in an education or training activity, as many enter Youth Service disengaged from school and can have other complex issues

Many young people who enter Youth Service have experienced difficulties with mainstream education, are disengaged from school, and have serious or complex issues that require intensive support. As a result, it can often take several months before some participants are ready to enrol in an education or training activity. Correspondingly, the longer a young person is enrolled in the service, the more likely they are to be engaged in education and training activities.

Young people are enjoying Youth Service’s focus on education and training

Process evaluation feedback on the education and training focus of Youth Service was positive, and the majority of young people interviewed were happy with their current education or training programme. In addition, the evaluation found that education and training courses can also help address a young person’s social, emotional and confidence issues.

Most young people undertake budgeting activities within their first five months of enrolment

Around 55 per cent of YP and YPP participants are engaged in budgeting activities five months after starting Youth Service. Enrolment in this activity declines thereafter.

YPP participation in parenting programmes increases with duration in the Youth Service

YPP enrolment in parenting activities increases with the time a young parent spends in the service. After one month’s duration, only nine per cent of YPP participants are enrolled in a parenting activity, compared to 50 per cent after 10 months.

The process evaluation found that Youth Service was generally implemented and operating as intended, and was a positive initiative for the contracted providers and young people involved

The process evaluation was carried out in February 2013 to assess Youth Service’s initial implementation and operation. It found that the service was generally implemented as intended and was operating as intended. Contracted providers and young people interviewed for the evaluation were positive about their involvement in the service.

As the Youth Service was a new service delivery mechanism for both Work and Income and providers, the evaluation also identified several areas for improvement in the early stages of the service’s operation. Providers and Work and Income worked proactively together to address the issues identified and continue to work cooperatively.

[1] Data on NCEA qualifications was not available for this analysis. Instead we have counted young people as having ‘met the requirements’ for NCEA Level 2 when they had gained 80 NCEA Level 2 credits.

[2] The evaluation was able to match NZQA qualifications data for approximately 17,500 (89 per cent) of the 20,000 young people receiving Youth Service for NEET. Historical information on NEET young people who did not receive a benefit from Work and Income is not available for us to construct a comparison group for Youth Service: NEET educational achievement

[3] Analysis of employment outcomes will be able to be undertaken once MSD has completed linking with the Integrated Data Infrastructure (IDI) at Statistics New Zealand (within 12 months).

[4] Young people without children who are receiving a benefit from Work and Income.

[5] In addition, we are unable to assess the impact of Youth Service on the benefit recipient of NEET participants. Historical information on NEET young people who did not receive a benefit from Work and Income is not available for us to construct a comparison group.

Background

Youth Service is a new approach to working with young people. The service was implemented on 20 August 2012 under Welfare Reform and targets 16-18 year old young people who are at risk of long-term benefit dependency. It aims to help young people build an independent future and reduce their risk of transitioning to a working-age benefit after age 18, through achieving a qualification of NCEA Level 2 or higher and developing life skills.

Under the Youth Service, MSD contracts community-based service providers to work intensively with young people and provide a wrap-around service to support them into education, training or work-based learning. Provider funding is based on an incentivised outcome-based contract consisting of administration fees, milestone payments and success fees for achieving outcomes.

The Youth Service is available to all young people aged between 16 and 18 years who receive financial assistance from Work and Income, as well as other young people who are not engaged in employment, education or training (NEET). Work and Income set up a centralised unit, the Youth Service Support Unit (YSSU), to administer financial assistance to young people receiving the service, process referrals and applications, and provide administrative support to providers.

Youth Service for Work and Income clients: Youth Payment and Young Parent Payment

Young people aged between 16 and 18 who apply for financial assistance from Work and Income will receive either Youth Service: Youth Payment (YP), or Youth Service: Young Parent Payment (YPP). Young people aged 16 and 17 without children receive YP. YP replaces the previous Independent Youth Benefit (IYB). Teen parents under the age of 19 receive YPP, which replaces the Emergency Maintenance Allowance for teen parents aged 16-17 and the relevant working-age benefit for those aged 18. Young people who are a partner of a main beneficiary also receive YP or YPP as a primary beneficiary under Youth Service.

Participants receiving Youth Service (YP and YPP) have clear obligations to:

• remain in, or be available for education, training or work-based learning that is leading towards at least NCEA Level 2 or equivalent qualification
• participate in other approved programmes such as budgeting and parenting courses in return for financial assistance.

Specific features of the Youth Service (YP and YPP) include:

• Money Management to assist young people with managing their financial assistance
• Guaranteed Childcare Assistance Payment (GCAP) for those with children under the age of five, to ensure childcare costs are not a barrier to participation in education, training or work-based learning.

Youth Service for NEET participants not receiving Work and Income assistance

Youth Service: NEET is a voluntary programme which focuses on youth at risk for NEET who are not in receipt of a main benefit. It aims to actively identify, engage and support 16–17 year olds (and some 15 year olds[1]) who are or are at-risk of becoming NEET, to return to education, training or employment.

Under Youth Service: NEET, providers work with young people to enable them to:

• participate in education, training or work-based training
• achieve an NCEA Level 2 or an equivalent qualification
• plan for their future employment, education, or training after exiting the service.

Specific features of Youth Service: NEET include:

• data sharing between MSD and the Ministry of Education (MoE) to improve the targeting of services according to risk profiles[2]
• an incentivised contract model focused on education and training outcomes.

Expected outcomes

The expected outcomes of Youth Service are that young people:

• reduce their risk of long-term benefit dependency
• achieve at least NCEA Level 2 or an equivalent qualification
• develop important life skills, such as budgeting and parenting, to improve their social outcomes and those of their children (for young parents).

Youth Service: NEET has the additional outcomes for young people to:

• sustain participation in education, training or work-based training for at least three months
• have an achievable plan for employment, further education, or training upon exiting the service
• not be on benefit or serving a custodial sentence three months after the end of the school year/end of training course following their 18th birthday.[3]

Rationale

A strong focus on youth is required to reduce the number of people at risk of long-term benefit receipt.

Although the number of new youth entrants to benefit each year is low, the most recent actuarial valuation shows that these participants account for a significant proportion of the liability over time. In fact, more than 70 per cent of the forward liability is in respect of people who first received a benefit before the age of 20 – indicating that many of these young people remain vulnerable to benefit dependency their whole lives. Early entrants to the benefit system have a high average liability of $109,000 per client. This is even higher for clients who meet the eligibility for YP within the early entrants’ cohort, who have an average liability at $185,000.

The Youth Service aims to address this through equipping young people at risk of long-term benefit dependency with the skills and education to build an independent future, to reduce the likelihood they will require a working-age benefit after age 18.

[1] This group consists of 15 year olds who have been granted early leaving exemptions from school or those who will turn 16 during the school holidays and have no intention of returning to school.

[2] Each individual referred to Youth Service for NEET is given a risk rating that estimates their likelihood of long-term benefit dependency based on information about that individual held by MSD and MoE. The risk rating also links to payments to providers for the achievement of milestones and success fees.

[3] It is too early for the evaluation to assess this outcome.

Purpose of the report

This report details findings from the evaluation of Youth Service’s first 18 months of operation.

It brings together findings from a number of evaluations undertaken by Insights MSD on the operation and impact of the Youth Service, specifically:

• an outcomes evaluation for young people who participated in the service in its first 18 months (20 August 2012 to end March 2014). The evaluation is based on MSD administrative data and considers young people’s participation in education, training or work-based learning over the period, enrolment in budgeting and parenting activities, and achievement of NCEA qualifications.
• an assessment of the service’s impact on young people’s benefit receipt. The impact was assessed using a comparison group of similar participants who started a main benefit at ages 16-18 before the introduction of Youth Service. This comparison group was made up of young people who received Independent Youth Benefit, Emergency Maintenance Allowance and Domestic Purposes Benefit.
• a process evaluation of the service’s implementation and early operation, which assesses at an early stage whether the service is operating as intended, what is working well, and what could be improved. The evaluation draws primarily on interviews with Youth Service providers, staff and participants undertaken in February 2013, soon after the service was implemented. Areas for improvement identified by the process evaluation have since been addressed by MSD.

Limitations of the evaluation

Youth Service is designed to reduce the dependence of young people on the benefit system over time.

Given the long-term nature of the changes sought from the Youth Service, the evaluation findings in this report are preliminary and underestimate the service’s full effect, covering only the first 18 months of operation (20 August 2012 to end March 2014). Over this short period, most young people had only a relatively short engagement with the service and impacts are yet to be seen for participants who began the service toward the end of the study period.

A more comprehensive assessment will be carried out in 2015, when the service has been in operation for three years. At this time, more young people would have engaged with the service for at least 18 months, allowing for the more long-term effects of the service to be observed.

Limitations and caveats for each of the methodologies used by the evaluation are included in this report, where applicable, with the corresponding findings. The findings should be read and interpreted with this information in mind.

Education, training and work-based learning

This section reports findings on young people’s participation in education, training or work-based learning, budgeting and parenting activities over the first 18 months of Youth Service.

Under Youth Service, most young people are engaged in education, training or work-based learning

At the end of March 2014, 4 out of 5 young people enrolled in the Youth Service were engaged in at least one full-time or part-time education, training or work-based learning activity.[1] The majority of these young people were engaged in some form of education (72 per cent for YP; 79 per cent for YPP; and 64 per cent for NEET) (Table 1). Engagement rates have increased over the last year and have remained relatively stable over the past six months (Figure 1 below and Table 2 in Appendix 1). The small number of young people who were not participating in an education, training or work-based learning activity are likely to have been new to Youth Service and not yet had an activity assigned to them.

[1] Some these young people may have been engaged in short courses, such as obtaining a driver’s license.

Table 1: Engagement in education, training and work-based learning activities (either full-time or part-time); current Youth Service participants at 31 March 2014

Activity	YP	YPP	NEET
Education	72%	79%	64%
Training	25%	17%	18%
Work-based learning	2%	1%	4%
Any education, training or work-based learning	88%	87%	77%
No education, training or work-based learning activity	12%	13%	23%

Note: Counts are based on data reported by Youth Service providers in the Activity Reporting Tool (ART) and include short courses, such as obtaining a driver’s license. Counts for Education, Training and Work-based Learning are not mutually exclusive: people may participate in more than one activity.

Source: Activity Reporting Tool (ART), MSD

Figure 1: Youth Service participants enrolled in education, training or work-based learning (either full-time or part-time); at quarter end, March 2013 to March 2014

Note: Counts are based on data reported by Youth Service providers in the Activity Reporting Tool (ART) and include short courses, such as obtaining a driver’s license.

Source: Activity Reporting Tool (ART), MSD

It can take several months to engage a young person in an education or training activity, as many enter Youth Service disengaged from school and can have other complex issues

The process evaluation found that most young people receiving Youth Service have experienced difficulties with mainstream education, and the majority are disengaged from school. In addition, many of these young people require intensive support with serious or complex issues which are often health and family related. For example, a review of YP grants by Work and Income found that 38 per cent of applicants were victims of domestic violence, 76 per cent suffered emotional neglect and 5 per cent were homeless or victims of sexual abuse.

As a result, it can often take several months to enrol a participant in an education or training activity, with providers and coaches instead working with the young person to stabilise their living situation or other factors in their life first. Correspondingly, engagement in education, training, or workplace learning increases with the length of time a young person is enrolled in Youth Service. After two months in the service, only 54 per cent of YP, 43 per cent of YPP and 63 per cent of NEET participants were participating (either full-time or part-time) in education, training or work-based learning activities, compared to 83, 74 and 87 per cent respectively after 10 months (Figure 2 below and Table 3 in Appendix 1).

Figure 2: Proportion of participants enrolled in education, training or work-based learning (either full-time or part-time); by duration in Youth Service

Note: The total number of participants enrolled in the Youth Service generally declines across the 18-month period. For example, in the case of YP participants, the total number of participants still enrolled in the Youth Service at the 18-month mark is approximately 400, compared to approximately 6,600 at the 1-month mark. This means that small variations in activity enrolment will have a larger effect on the percentage enrolled at later months, compared to the earlier months. This effect is purely statistical in nature.

Counts are based on data reported by Youth Service providers in the Activity Reporting Tool (ART) and include short courses, such as obtaining a driver’s license.

Source: Activity Reporting Tool (ART), MSD

Figures 9, 10 and 11 in Appendix 1 provide a further breakdown of engagement for YP, YPP and NEET participants by duration in the service.

The process evaluation found that transport and enrolment timeframes can also impact on the ability of young people to enrol in education, training or work-based learning courses. Limited access to transport can prevent young people from attending their preferred course. In addition, set enrolment periods (e.g. February and March intake) can mean that young people who start the service outside the enrolment timeframe are unable to enrol in a specific education or training programme.

Young people are enjoying the Youth Service’s focus on education and training

Process evaluation feedback on the education and training focus of Youth Service was positive, and the majority of young people interviewed were happy with their current education or training programme.

I love the course, it’s better than sitting at home. Tutors help out a lot, even with stuff that is not to do with the course. Unit standards take a long time at school but it’s one-on-one at the course, they help you out until you achieve them. (Youth Service participant)

The evaluation found that education and training courses can serve additional purposes for young people beyond general learning, in addressing social, emotional and confidence issues.

I’ve matured a lot. Since I’ve been on [Youth Service] I’ve had more discipline, like, I have to be in training. I can’t just slack off and say I can’t be bothered. It’s been a good encouragement for me. I feel I have achieved a lot. The opportunity to be in training has been a whole lot better. (Youth Service participant)

We are unable to assess whether Youth Service has increased participation in education and training for young people on benefit

Education and training data were not collected for young people receiving benefit prior to the introduction of Youth Service. As a result, the evaluation is unable to estimate what young people’s rates of participation in education and training activities would have been in the absence of the service.

However, prior to the establishment of Youth Service many young people who received Work and Income assistance at age 16 or 17, such as the Independent Youth Benefit (IYB), were incentivised to look for employment as a first choice, accessing training or education only if they were unable to gain a full-time job. As a result, many of these young people gained employment in low-skilled industries that did not equip them with enough skills to prevent them requiring income support in the future, and IYB participants had one of the highest average benefit liabilities

Given the opposite focus of Youth Service, we expect that more young people on benefit are now engaging in education or training instead of moving into employment in the short-term.

Most young people undertake budgeting activities within their first five months of enrolment

Over half of YP and YPP participants are engaged in budgeting activities five months after starting Youth Service (Figure 3).[1] Enrolment in this activity declines thereafter, falling to around 1 in 5 of YP and YPP participants by the 18-month mark.

Very few NEET participants undertake budgeting activities. As they are not receiving financial assistance from Work and Income, they are not required to participate.

[1] Some YP and YPP clients are not on benefit for at least three months and although they may receive budgeting assistance, do not participate for at least three months in order to receive incentive payments.

Figure 3: Proportion of participants enrolled in budgeting activities; by duration in Youth Service

Note: The total number of participants enrolled in the Youth Service generally declines across the 18-month period. This means that small variations in activity enrolment will have a larger effect on the percentage enrolled at later months, compared to the earlier months. This effect is purely statistical in nature.

Source: Activity Reporting Tool (ART), MSD

YPP participation in parenting programmes increases with duration in the Youth Service

Participants show increasing enrolment in parenting activities over their time in Youth Service. After one month’s duration, only nine per cent of YPP participants are enrolled in a parenting activity, compared to 50 per cent after 10 months (Figure 4).

Figure 4: Proportion of YPP participants enrolled in parenting activities; by duration in Youth Service

Note: The total number of participants enrolled in the Youth Service generally declines across the 18-month period. This means that small variations in activity enrolment will have a larger effect on the percentage enrolled at later months, compared to the earlier months. This effect is purely statistical in nature.

Source: Activity Reporting Tool (ART), MSD

Impact on benefit receipt

This section reports findings to date on the impact of Youth Service on young people’s main benefit receipt in the first 18 months following enrolment. Impact is assessed using a comparison group drawn from young people who started a youth main benefit before the introduction of Youth Service and were case managed by Work and Income. Main benefit receipt is compared between the two groups.

The analysis covers young people receiving YP and YPP and excludes NEET young people enrolled in Service. Historical information on NEET young people who did not receive a benefit from Work and Income is not available for us to construct a comparison group for Youth Service: NEET.

The comparison group for YP and YPP has been matched on a small number of participant characteristics[1]and are taken from participants starting youth benefits between 2008 and August 2012. As a result, we cannot rule out the possibility that reported impacts are due to uncontrolled differences in participant profiles (eg school achievement) or because of changing labour market conditions before and after the introduction of Youth Service. To eliminate these alternative explanations will require more rigorous analysis. A more robust estimate that better controls for differences in the participant profiles will be developed for the 2015 evaluation update.

As the evaluation found that Youth Service has a different impact for teen parents (those receiving YPP) compared to young people who start the service without children (those receiving YP), the findings for YP and YPP participants are discussed separately in this section.

Early evidence suggests that, over the medium term, Youth Payment participants spend less time on benefit under Youth Service

In the first six months after starting the Youth Service, young people receiving YP initially spend slightly longer on benefit than their comparison group (Figure 5). The most likely explanation for this observed impact is the change from Work and Income’s focus on employment assistance, to a greater emphasis on education and training under Youth Service. There is a large body of evidence to show that education and training assistance initially increases the time participants spend on benefit. This effect occurs for a number of reasons, the most important are:

• while on education and training programmes participants have less time to engage in job search and therefore are less likely to move off benefit into employment during this time
• gaining a qualification provides a strong incentive to complete training, rather than to exit the course if a job opportunity arises.

However, one year after enrolment in the service we begin to see lower levels of benefit receipt for YP participants relative to similar young beneficiaries before the Youth Service was established (Figure 5). A year and a half after starting the service, 54 per cent of YP participants are on a main benefit (either youth related or working-age) compared to 62 per cent of the comparison group.

These findings suggest that the service is improving YP participants’ prospects of moving off benefit in the medium-term. We expect the positive impact after one year is from either movement into employment or full-time tertiary training. Further analysis is required to confirm this.[2]

[1] These include whether they were a teen parent, time from starting a main benefit, and age at first benefit start date.

[2] Analysis of participation in training and education using Ministry of Education data will be carried out later in the year. Analysis of employment outcomes will be able to be undertaken once MSD has completed linking with the Integrated Data Infrastructure (IDI) at Statistics New Zealand (within 12 months).

Figure 5: Youth Payment participants and comparison group active on a main benefit

Notes: Participants: started the Youth Service after 20 August 2012.

Comparison: matched to the participants and started a youth related benefit between 1 January 2008 and July 2012.

Any main benefit: any spell on youth or working age main benefit.

Source: MSD administrative data (research data, not official statistics)

Fewer Youth Payment participants are transitioning to a working-age benefit

As a consequence of more YP participants exiting benefit after one year, the proportion of young people eligible for YP[1] who transition from a youth benefit onto a main working-age benefit has fallen by eight percentage points under Youth Service:

• over the 12 months to March 2014, 30 per cent of YP participants transferred to a main working-age benefit
• over the 12 months to March 2012, 38 per cent of Independent Youth Benefit participants transferred to a main working-age benefit.

It is too early to assess the impact of Youth Service for participants receiving Young Parent Payment

At present, the number of YPP participants is too small to observe a meaningful impact. As shown in Figure 6, over the first year and half after starting Youth Service we see little difference in benefit receipt between YPP participants and their comparison group.

We believe the absence of any observable impact of Youth Service for teen parents is because these participants are caring for young children. Child care responsibilities will impinge on the impact of the service in two related ways:

• participants will have less time in the day to undertake education and training, with greater logistical challenges fitting study around childcare.[2]Consequently, we expect it to take longer for teen parents in the Youth Service to gain marketable qualifications
• care for pre-school children can limit employment opportunities for teen parents, therefore it unlikely that many teen parents can take full advantage of any gains in qualifications under Youth Service until their children reach school age.

For these reasons we believe it unlikely that we will see reduction in benefit receipt for teen parents resulting from the Youth Service until at least four to five years following their enrolment. Such a timeframe is consistent with the evaluation of Training Incentive Allowance (TIA) for sole parents, which found that it took three to four years after initial TIA receipt before sole parents became more likely to move off benefit than their comparison group. In addition, it was only after eight to ten years following the initial receipt date that TIA began to reduce the overall time that sole parents spent on benefit relative to their comparison group (MSD, 2013).[3]

[1] Young people without children who are receiving a benefit from Work and Income.

[2] The Guaranteed Childcare Payment will help offset the cost of childcare, but cannot completely eliminate the issue of childcare.

[3] MSD. (2013). Cost-effectiveness of Work and income employment assistance. Wellington, Ministry of Social Development.

Figure 6: Young Parent Payment participants and comparison group active on a main benefit

Notes: Participants: started the Youth Service after 20 August 2012.

Comparison: matched to the participants and started a youth related benefit between 1 January 2008 and July 2012.

Any main benefit: any spell on youth or working age main benefit.

Source: MSD administrative data (research data, not official statistics)

Educational achievement

This section reports findings on the difference Youth Service is making to educational achievement for YP and YPP participants in the first 12 months following enrolment. The difference is determined using the same comparison group we used to analyse the impacts on main benefit receipt.[1]

In addition to the limitations of the comparison group stated in the previous section, a further limitation of this analysis is that the comparison group was selected without consideration of prior educational achievement. This is because the NZQA qualifications data was not available at the time the matching process was carried out to select the comparison group. As a result, a higher proportion of young people in the comparison group had met the requirements for NCEA Level 2 at the point of entry to benefit than YP and YPP participants. The effects of this are discussed below.

A descriptive analysis of NCEA achievement for NEET young people in Youth Service is also included. As noted in the previous section, we are not able to construct a comparison group for Youth Service: NEET.

In the first 12 months of enrolment in Youth Service, YP and YPP participants are more likely to gain NCEA credits

Nearly two-thirds (63 per cent) of YP participants increased the number of NCEA credits they held in their first 12 months of participating in Youth Service, compared to one quarter (24 per cent) of their comparison group (Figure 7). Amongst YPP participants, over two-fifths (43 per cent) increased their number of NCEA credits, compared to one-fifth (20 per cent) of their comparison group.

[1] A descriptive analysis of NEET educational achievement will be provided, once the data become available.

Figure 7: Proportion of Youth Payment and Young Parent Payment participants, and their comparison group, who gained NCEA credits within 12 months of Youth Service

Notes: Youth Service clients: YP and YPP participants who started Youth Service after 20 August 2012.

Comparison: matched to the participants and started a youth related benefit between 1 January 2008 and July 2012.

Source: MSD administrative data (research data, not official statistics)

YP and YPP participants are more likely to meet the requirements[1] of NCEA Level 2 within their first 12 months in Youth Service

Within their first 12 months in Youth Service, 14 per cent of YP participants and 7 per cent of YPP participants met the requirements of NCEA Level 2 (Figure 8).

Over this period, both YP and YPP were more likely to meet the requirements for NCEA Level 2 relative to similar young beneficiaries before Youth Service was established (a nine percentage point gain for YP, and a two percentage point gain for YPP).

However, 12 months after enrolment in the Youth Service, YP and YPP participants remained less likely to have met the requirements of NCEA Level 2 than young people in the comparison group

At the point of entry to Youth Service, YP and YPP participants were less likely to have met the requirements of an NCEA Level 2 qualification than their comparison group. There are two reasons for this:

• the comparison group was selected without consideration of prior educational achievement, as NZQA qualifications data was not available at the time of the matching process;
• Youth Service targets young people who have lower levels of educational achievement.

The gains made by YP and YPP participants toward NCEA Level 2 as a result of Youth Service have not been large enough to compensate for this difference (Figure 8). After one year in the service, YP and YPP participants remain less likely to have met the requirements of NCEA Level 2 than their comparison group.

A more robust estimate that controls for differences in prior educational achievement will be developed for the 2015 evaluation update.

[1] Data on NCEA qualifications was not available for this analysis. Instead we have counted young people as having ‘met the requirements’ for NCEA level 2 when they had gained 80 NCEA Level 2 credits.

Figure 8: Proportion of Youth Payment and Young Parent Payment participants, and their comparison group, who met the requirements of NCEA Level 2 within 12 months of enrolling in the service

Notes: Youth Service clients: YP and YPP participants who started Youth Service after 20 August 2012.

Comparison: matched to the participants and started a youth related benefit between 1 January 2008 and July 2012.

Source: MSD administrative data (research data, not official statistics)

NEET[1] young people in Youth Service are achieving NCEA credits and qualifications

Of these young people:

• Half increased their number of NCEA credits within 12 months of starting Youth Service
• 15 percent met the requirements of NCEA Level 2 within their first 12 months in the Service
• 17 per cent had already met the requirements of NCEA Level 2 prior to starting the Service.

[1] The evaluation was able to match NZQA qualifications data for approximately 17,500 (89 per cent) of the 20,000 young people receiving Youth Service for NEET. Historical information on NEET young people who did not receive a benefit from Work and Income is not available for us to construct a comparison group for Youth Service: NEET educational achievement

Appendix 1: Additional tables and graphs

Table 2: Youth Service participants enrolled in education, training or work-based learning (either full-time or part-time); at quarter end, March 2013 to March 2014

Quarter end	YP (%)	YPP (%)	NEET (%)
March 2013	65	46	52
June 2013	76	57	66
September 2013	87	78	75
December 2013	83	81	77
March 2014	88	87	77

Source: Activity Reporting Tool (ART), MSD

Table 3: Proportion of participants enrolled in education, training or work-based learning (either full-time or part-time); by duration in Youth Service

Months after starting in the Youth Service	YP (%)	YPP (%)	NEET (%)
2	54	43	63
4	65	51	73
6	75	60	81
8	79	66	85
10	83	74	87
12	85	79	88
14	83	81	88
16	80	83	90
18	74	81	92

Source: Activity Reporting Tool (ART), MSD

Figure 9: Youth Payment: Participants enrolled in education, training or work-based learning (either full-time or part-time); by duration in Youth Service

Source: Activity Reporting Tool (ART), MSD

Figure 10: Young Parent Payment: Participants enrolled in education, training or work-based learning (either full-time or part-time); by duration in Youth Service

Source: Activity Reporting Tool (ART), MSD

Figure 11: NEET: Participants enrolled in education, training or work-based learning (either full-time or part-time); by duration in Youth Service

Source: Activity Reporting Tool (ART), MSD