Quality Management, Performance

Measurement and Indicators in Higher

Education Institutions: Between Burden,

Inspiration and Innovation

Cláudia S. SarricoHigher Education TeamOECD Directorate for Education and Skills

SQELT International Evaluation Workshop1-2 July 2019Danube University Krems

Rationale for performance and quality management in higher education

2

Higher education is rapidly expandingShare of 24-34 year-olds with a tertiary degree across OECD and G20 countries

Note: The figures in these graphs are estimates based on available data. The population estimations are based on the OECD annual

population projections.

Source: OECD (2015), "How is the global talent pool changing (2013, 2030)?”, Education Indicators in Focus, No. 31,

https://doi.org/10.1787/5js33lf9jk41-en.

China, 17%

United States, 14%

India, 14%

Russian Federation, 10%

Japan, 6%

Indonesia, 4%

Brazil, 4%

Korea, 4%

Mexico, 3%

United Kingdom, 3%

France, 2%

Germany, 2%

Canada, 2%

Turkey, 2%

Spain, 2%Poland, 2%

Argentina, 1%Italy, 1%

Australia, 1%Saudi Arabia, 1% Other, 6%

137 million 25-34 year-olds with tertiary

education

China, 27%

United States, 8%

India, 23%

Russian Federation, 4%

Japan, 3%

Indonesia, 5%

Brazil, 5%

Korea, 2%

Mexico, 2%

United Kingdom, 2%

France, 1%Germany, 2%

Canada, 1%

Turkey, 2%Spain, 1%Poland, 1%

Argentina, 2%Italy, 1%Australia, 1%

Saudi Arabia, 3% Other, 6%

300 million 25-34 year-olds with tertiary

education

2013 2030

3

4

But there are quality challenges…Percentage of graduates with low literacy and numeracy

0 20 40 60 80

JapanNetherlands*

Czech RepublicFinland

SwedenFlanders*Norway*Australia

Slovak RepublicUnited StatesNew Zealand

AustriaNorthern Ireland…

PolandFrance

DenmarkEngland (UK)

GermanyKorea

IrelandAverageCanada

Estonia*Slovenia

LithuaniaItaly

SpainIsrael

GreeceTurkey

Chile

Literacy skills

0 10 20 30 40 50 60 70 80

Czech RepublicFlemish Community*

Netherlands*Slovak Republic

JapanSwedenAustria

Norway*Finland

DenmarkGermany

FranceLithuaniaSloveniaAustraliaAverage

Northern Ireland (UK)New Zealand

PolandEstonia*

United StatesEngland (UK)

KoreaIreland

CanadaItaly

SpainGreece

IsraelTurkey

Chile

Numeracy skills

Below level 1 Level 1 Level 2

Source: OECD (2016), Skills Matter: Further Results from the Survey of Adult Skills, https://doi.org/10.1787/9789264258051-en.

5

And equity challenges…Access rate gaps for 18-24 year-olds

-80

-70

-60

-50

-40

-30

-20

-10

0

Panel A - Bachelor’s or long first degree programmes Percentage change in the probability to enter a higher education programme for 18-24 year-olds whose

parents did not attain higher education and for those whose parents are foreign-born

-100

-80

-60

-40

-20

0

20

40

60

Chile Slovenia Norway* Average France Sweden Latvia Portugal

Panel B - Short-cycle programmesPercentage change in the probability to enter a higher education programme for 18-24 year-olds whose

parents did not attain higher education and for those whose parents are foreign-born

Parents without higher education Foreign-born parents

How to read this chart: Panel A: In Slovenia, 18-24 year-olds without tertiary educated parents are about 40% less likely to enter a bachelor’s or long first degree programme than other

18-24 year-olds. Panel B: In Chile, 18-24 year-olds without tertiary educated parents are about 40% more likely to enter a short-cycle programme than other 18-24 year-olds.

Source: Indicators of Education Systems (INES) Survey on Equity in Tertiary Education.

6

And relevance challenges…Businesses collaborating on innovation with higher education or research institutions

0

5

10

15

20

25

30

Universities or other higher education institutions Government or public research institutes Private research institutes

53 42

Source: Adapted from Eurostat (2018), Community Innovation Survey, Science, Technology and Innovation Indicators,

http://ec.europa.eu/eurostat/web/science-technology-innovation/data/database.

7

And the costs of higher education are becoming

increasingly difficult to manage…

90

100

110

120

130

140

2005 2010 2015

Expenditure on education institutions and number of students by education level, 2005=100

Total expenditure, higher education Number of students, higher educationExpenditure per student, higher education Total expenditure, primary to post-secondary non-tertiary educationNumber of students, primary to post-secondary non-tertiary education Expenditure per student, primary to post-secondary non-tertiary education

Source: Adapted from OECD (2018[3]), Education at a Glance 2018, https://doi.org/10.1787/eag-2018-en.

From Burden to Inspiration and Innovation

8

Questions

Erosion of trust

Accountability mechanisms

Attempt to restore trust

Quality?

Equity?

Relevance?

Financial sustainability?

Normative perspective on trust

Rationalist-instrumental perspective on trust

(Stensaker and Harvey 2011)

(Stensaker and Gornitzka 2009)

Burden

9

10

Benchmarking Higher Education Systems

Highlights

https://doi.org/10.1787/be5514d7-en

Higher education provides graduates with favourable economic and social outcomes, but the low basic skills of some graduates is a cause for concern

Inequity of access by socio-economic and migration background is a persistent challenge

Only 4 in 10 bachelor’s students are able to complete on time, and 2 in 10 do not complete at all

Young doctorate holders in higher education employment find less job security than their predecessors and their peers in other sectors

Higher education research and development relies heavily upon public funding, and establishes limited collaboration with businesses on innovation, especially for small and medium enterprises

There is an increasing focus on engagement activities, but frameworks for measuring activities do not yet exist

Open access to scientific documents remains limited

Although quality is difficult to measure, governments are increasingly trying to link funding and other policies to the quality of teaching and research

Data limitations prevent comprehensive performance assessment of higher education systems, but improvements in measurements are possible

11

Benchmarking Higher Education Systems

Source of inspiration and innovation

https://doi.org/10.1787/be5514d7-en

Benchmarking approach

Higher education

system benchmark

Higher education

system

Policy

benchmarking

Comparison of

policies

New ideas, new policies,

new practices

Creativity and innovation

Peer learning

Comparison of

metrics

Metric

benchmarking

Inspiration

Innovation

12

Country reviews

The mobilisation, allocation, and management of financial resources: public funding for teaching, research, and infrastructure; private revenues raised by higher education institutions; student fees and student financial support

Human resources management: attracting, recruiting and selecting the higher education workforce, the structuring of the higher education workforces, and inducing the desired performance from the higher education workforce

Resource governance and coordination: coordinating demand and supply (study places, programmes, and institutions), the network of provision (institutional collaboration, alliance, and mergers), and student pathways.

New Higher Education Resources Project

Country reviews and benchmarking policy briefs

Attracting, recruiting and selecting the

HE workforce

Recruitment process

Staff profile

Staff numbers

Structuring the higher education

workforce

Employment status of academic staff

Academic roles and working time

Digitalisation of teaching and learning

Inducing the desired performance from the HE workforce

Staff appraisal

Promotion

Remuneration

Professional development

Mobility

Retirement

14

New Higher Education Resources Project

Human Resources

What is happening to quality and performance measurement and

management?

15

Performance Model

objectives inputs activities outputs

intermediate outcomes

final outcomes

needssocio-

economic situation

efficiency

cost-effectivenesseconomy

value and sustainability

effectivenessrelevance

higher education system

Context

(Talbot 2010, Bouckaert and Halligan 2008)

(Sarrico, 2018) 16

• Different measurement and management instruments

• Steering mechanisms: professional, state and market

regulation

• Performance indicators

– Economy

– Efficiency

– Quality: from internal to external quality, from intrinsic to

extrinsic motivations, from improvement to accountability

17

Towards integration and squaring the circle

Towards integration and squaring the circle

• Integrative frameworks, at system, institution and unit

level

– National and supra-national (ENQA, INQAAHE, OECD,

UNESCO, World Bank)

– Institutional – increasingly integrated:

• In wider management and governance arrangements

• Different missions: learning and teaching, research and scholarship,

engagement

• Different organisational levels: institution, sub-units

• Different quality and performance dimensions

18

What has resulted from the quest for performance in higher education?

19

More and better?

• Research productivity and quality up

– Economic and social impact?

• Rankings, reputation and the quest for world-class universities

– Lack of attention to the quality of teaching and meaningful engagement

with the wider world

• Poor education provision less likely

– Study success increasingly addressed

– Learning outcomes and learning gain?

– Graduate labour market outcomes?

– Wider social outcomes of graduates?

20

More and better?

• Engagement

– Emphasis on technology transfer and commercialisation of research

– Less on wider civic and social engagement

– Difficulty in measuring ‘valorisation’

• Performance-based funding in addition to basic government allocations

– Ex-post – reward for good past performance

– Ex-ante – performance agreements

• Growing importance of third-party funding in addition to core funding and student fees

– Continuing Education

– Knowledge and technology transfer

– Service provision

– Endowments and other philanthropic donations

21

What does the future hold?

22

Some possible trajectories

• Self-accountability -> societal accountability -> societal engagement(Hazelkorn, 2016)

• Bias towards research addressed

– Measurement of learning outcomes and learning gain

– Initiatives to improve the quality of teaching

• More attention to human resources management and professional

development

• Valorisation agenda

– More emphasis on the social impact of higher education

– More engaged graduates, knowledge exchange, and civic and social

engagement

• From ‘world-class universities’ to ‘world-class systems’

23