Building the evidence of system performance What are we learning about the tertiary education system? ATEM NZ Conference 10 July 2007

Building the evidence of system performance

What are we learning about the tertiary education system?

ATEM NZ Conference

10 July 2007

Page 2 New Zealand Ministry of Education

Lies, damned lies and statistics

▲ Government collects large amounts of data about education – and tertiary education in particular. From it, we can work out many facts.

▲ But facts without context are not much more than trivia. And data is useful only if it helps us to answer ‘important’ questions.

▲ The really important questions:

▲ how well do our students do in completing their studies?

▲ how well do they do after they leave study and in what ways?

▲ how much does our system add value?

▲ It is all about gathering evidence


This presentation

▲ Looks at the Ministry of Education’s tertiary education analytical programme

▲ Where it came from ▲ Current priorities▲ Some examples

▲ What do you want? And what is missing?


Building evidence of performance in tertiary education

▲ Government, institutions and students all have a stake.

▲ Need to improve study outcomes for many.

▲ Need to get good value for money.

▲ Policy shift from emphasis on participation to outcomes.

▲ Funding changes.

▲ Historically, lack of evidence has hampered our understanding.


Where we came from

▲ Unit record data has been collected since early 90s on parts of the tertiary education system

▲ From 2000, the collection was enhanced – with the SDR ▲ So there was a great and rich resource ▲ But little use was made of it, except internal analyses to underpin

some policy projects▲ With the 2002 reforms, the ministry needed to take a system view of

performance – and saw a greater need for public accountability information

▲ So we set about creating an evidence base from the available data ▲ to underpin policy▲ to provide accountability to public▲ and to support sector planning


Where we came from

▲ Where to start ….

▲ First, we analysed information and evidence needs▲ Conclusions

▲ a focus on performance ▲ and on outcomes▲ a system focus

▲ But to assess the performance of a system, we need to look at value created

▲ And that means primarily looking outside the system itself▲ what was a person’s status before▲ and what happens after leaving▲ plus something about how things work inside


Where we came from

▲ While we have great data, one issue was its usability▲ A key issue was the lack of links across time and across

providers▲ NSNs would eventually resolve that issue, but waiting

would take years▲ So in the meantime, statistical matching enabled us to do

completion rates


Where we came from

▲ But looking at SDR data – even with matching – didn’t give us outcomes data

▲ Especially given the need to assess performance by what happens after study

▲ The integrated dataset on Student Loan Scheme borrowers allows us to establish precise linkages between an individual’s education profile and post-study income – a good proxy for outcomes

▲ Although coverage is limited – it applies only to those who use the loan scheme – this was also a breakthrough

▲ But issues remain – data on industry training and targeted programmes is held separately – and how do we measure pre-entry status

▲ The NSN is now at a stage where we can start to address these issues


We set ourselves some goals

▲Our mission ▲ leadership in analysis ….▲ production and dissemination ….▲ timely, authoritative and relevant

• the tertiary education system • its interaction with the wider education sector • its contribution to New Zealand’s longer term social and economic

outcomes.

▲Our role▲ we look at a system that is made up of components▲ the TEC looks at components within a system▲ but we apply common or complementary analytical approaches to

common data


What is happening now

▲ We do summary reports▲ like Profile and Trends and TES monitoring▲ they are comprehensive▲ and summarise analyses

▲ And more detailed reports on aspects of the system▲ Mainly our approach is quantitative not qualitative▲ We focus on four themes

▲ Learners▲ Post-study outcomes▲ Supporting the system▲ Research and knowledge creation

▲ Education Counts has over 60 reports on many aspects of the system▲ Examples …





More examples▲ Who doesn’t pay back their loans?▲ What determines the research performance of staff? ▲ How long do people spend in tertiary education?▲ A changing population and the NZ tertiary sector▲ Participation in tertiary education – a birth cohort approach▲ He tini manu reo▲ Maori student performance in degrees


So what do we find?

▲ Some examples▲ Post-study outcomes▲ Passing courses and completing qualifications▲ Research performance


Outcomes – estimated earnings by level and gender

0

10

20

30

40

Leve

l 1-3

cer

tific

ates

Leve

l 4 c

ertif

icat

es

Leve

l 5-7

dip

lom

as

Leve

l 7 b

ache

lors

Leve

l 8-1

0ho

nour

s/m

aste

rs/d

octo

rate

s

Leve

l 1-3

cer

tific

ates

Leve

l 4 c

ertif

icat

es

Leve

l 5-7

dip

lom

as

Leve

l 7 b

ache

lors

Leve

l 8-1

0ho

nour

s/m

aste

rs/d

octo

rate

s

3 y ears after study 5 y ears after study

Estim

ated

ear

ning

s $

(000

)

Female Male


Outcomes – estimated earnings 5 years post-study by level and ethnic group

0

10

20

30

40

Lev el 1-3 certificates Lev el 4 certificates Lev el 5-7 diplomas Lev el 7 bachelors Lev el 8-10

honours/masters/doctorates

Estim

ated

ear

nings

$ (0

00)

Asian European Māori Other Pasifika


So what do we find? Completions

▲ 50%-55% completion rate for bachelors degree.▲ 80% pass rate for bachelors courses.▲ Rates are higher at postgraduate level and lower at sub-

degree level.▲ 25% pass all their courses without gaining a qualification.▲ Full-time students have much higher rates, even allowing

for the extra time part-timers need to complete.▲ NZ compares below average internationally, but

reasonably well when adjusted for its high level of part-time study.


So what do we find? Completions

Completion rates for 1995 starters over time

0%

10%

20%

30%

40%

50%

60%

70%

1 2 3 4 5 6 7 8 9 10 11

Years in study

Bachelors PhDs


Bibliometric analysis of university research

0.0

0.2

0.4

0.6

0.8

1.0

1.2

1.4

Relat

ive im

pact

Biological sciences Medicine & public health Health


Bibliometrics vs PBRF scoresBiological sciences

0

1

2

3

4

5

6

0 10 20 30 40 50 60 70 80

Citations per FTE

PB

RF

ave

rage

qua

lity

scor

e

AUT

LN VUW MY

WK

CY OT

AK


Where are we going?

▲ Breakthroughs in data infrastructure lead to analytical breakthroughs▲ Links between agencies’ datasets▲ Broadening of integrated data

▲ Some projects▲ More on research and on the PBRF

• How has the PBRF affected attrition in research degrees?• Statistical analyses of shifts

▲ Industry training▲ Transitions – how does school performance impact on tertiary

participation and on tertiary achievement

▲ Improving statistics – quality, timeliness and usability


The challenge for the future

▲ As we break new ground, we shift the frontier▲ We are changing the way we look at and talk about

tertiary education▲ And the new Tertiary Education Strategy – what demands

is that making?▲ Hard to get at information▲ For instance ….

▲ Improving research connections and linkages …▲ … to create economic opportunities▲ Increasing language, literacy and numeracy levels for the

workforce


And how can you help

▲ What do you need to know?▲ What is missing?


▲ Our task – build the evidence base▲ Our achievement

▲ lots more material▲ better statistics▲ better analysis

▲ But a way to go▲ Our challenge

▲ data infrastructure▲ improve the relevance of the analyses▲ the hard questions

http://educationcounts.edcentre.govt.nz/

Documents

Building the evidence of system performance What are we learning about the tertiary education system? ATEM NZ Conference 10 July 2007