Assessment for Effectiveness and Equity:

Lessons from a Longitudinal Study

Caine Rolleston

REAL, CambridgeJune 15, 2016

• Extent of ‘learning crisis’ becoming clearer – e.g. from citizen led assessments UWEZO etc.

• Cross-sectional data & benchmarking key altho still absent in many contexts

• But other key questions (especially effectiveness/equity) require more sophisticated data, including longitudinal or combined household and school data:

• Which schools/school systems are more effective (add more value) & why?

• To what extent a crisis of school quality? (not only learning)• When do gaps arise/develop during the life-course?• What are the causes of/remedies for poor attainment?• Cost-effectiveness, intervention choices

• Some potentially require combined longitudinal HH plus longitudinal school data – E.g. Are schools equally effective for more and less advantaged pupils?

ASSESSMENT IN DEVELOPING COUNTRIES

Young Lives longitudinal survey of children, households & communities every 3 years since 2002

• 12,000 index children in two cohorts (now aged 13 & 19)

• Ethiopia, India, Peru, Vietnam• 20 sentinel sites in each country• Tested in maths at each round with common

items• Primary school surveys implemented since

2010• Secondary school surveys from 2016

Allows comparison of • Learning levels• Learning trajectories• Change over time between cohorts

YOUNG LIVES STUDY

YOUNG LIVES STUDY

BENCHMARKING COMPETENCY LEVELS (ETHIOPIA)Competency level Number of Pupils Percentage of

PupilsCumulative Percentage

0 (Below Level 1) 853 8.66 8.661 (Early Foundational) (~ G1) 2,121 21.54 30.212 (Foundational) (~G2-3) 5,152 52.33 82.533 (Emerging) (~G3-4) 1,473 14.96 97.494 (Grade level) (~G4-5) 247 2.51 100.00Total 9,846 100.00

020

4060

8010

0pe

rcen

t

Rural Urban5 4 5 4

Level 0 Level 1Level 2 Level 3Level 4

LONGITUDINAL STUDY WITHIN A STUDY

.000

5.0

01.0

015

.002

.002

5D

ensi

ty

200 400 600 800Score

Test 1 Test 2

Maths Scores at First and Second Round Tests

• Mean Test 1 = 500, SD= 100. Mean Test 2=530 (gain 0.3 SD)

MEASURING PROGRESS OVER THE SCHOOL YEAR (ETHIOPIA)

INTERVALE-SCALE METRICS

Mathematics Reading Comprehension

October 2012

May 2013

Gain October 2012

May 2013

Gain

Mean

500.0 530.0 30.0 500.0 530.6 30.6

Gender

Boy 502.1 532.5 30.4 498.0 527.5 29.5

Girl 497.9 527.6 29.7 501.9 533.6 31.7

Difference 4.2 4.9 0.7 -3.9 -6.1 -2.2

Location

Urban 517.6 548.5 30.9 521.1 551.1 30.0

Rural 456.4 481.2 24.8 447.3 476.6 29.3

Difference 61.2 67.3 6.1 73.8 74.5 0.7

PREDICTORS OF ATTAINMENT AND PROGRESS OVER TIME (ETHIOPIA)

VARIABLES Maths T1 Maths T2 Reading T1 Reading T2

Girl -10.2031 *** -4.4330 *** 0.1650 4.8107 ***

Has 3+ meals per day 18.2975 *** 6.6820 *** 12.6032 *** 2.1339

PCA pupil durable assets 4.5648 *** 1.7210 *** 4.9427 *** 0.8929 *

% days absence W1-W2 -3.3064 *** -1.8387 *** -3.1255 *** -1.4661 ***

Orphan (single or double) 2.8872 -3.2279 * 2.6308 -3.3137 **

No-one in household literate -9.1339 ** -5.7685 * -14.3049 *** -1.8481

Attended pre-school 2.6854 -0.3207 6.3172 *** 3.3394 **

Ever repeated a grade -39.6920 *** -5.0135 *** -38.7614 *** -4.2894 ***

Ever dropped-out -6.1516 ** -2.1293 -13.4413 *** -3.7113 **

Reads books at home 17.5734 *** 6.2704 *** 17.3460 *** 3.6314 *

Child learns in home language 3.8083 -2.2792 14.5838 *** 5.4488 *

Pastoralist -24.1437 *** -4.3839 -37.9640 *** -8.0455 **

Pupil spends time on paid work -10.4830 *** 0.2447 -12.1998 *** -1.0128

SCHOOL-LEVEL VALUE-ADDED (ETHIOPIA)

-100

-50

050

100

Sch

ool V

alue

-Add

ed

0 20 40 60 80 100School Rank

Urban Rural

DUAL COHORT STUDY WITH A 7 YEAR INTERVAL:IMPROVING TEST SCORES, MIXED PATTERNS OF

EQUITYPeru Vietnam

Test score gains with/without equity improvement

Children aged 12 in 2006 and 2013

DECLINING TEST SCORES WITH WIDENING INEQUALITY

India (AP) EthiopiaChildren aged 12 in 2006 and 2013

020

4060

80M

edia

n M

aths

Sco

re R

3 %

0 20 40 60 80 100CDA-Q Score R2 %

Ethiopia PeruIndia Vietnam

WIDENING GAPS ARE DRIVEN BY DIFFERENCES IN LEARNING PROGRESS OVER TIME BETWEEN SYSTEMS:

AGE 5 TO 8

0.2

.4.6

01

23

4

-2 0 2 4 -2 0 2 4

Ethiopia Vietnam

Most Poor Least Poor

Mat

h sc

ores

(201

3)

Math scores (2009)

Graphs by country

Maths scoresLearning divergence by wealth groups

• Steeper gains by prior score in Ethiopia, broadly similar gaps by wealth• Pupils typically make more gains over time in Vietnam

• Distribution of school quality accentuates disadvantage

• E.g. differences in school quality explain more of the differences in test scores in India and Peru than Vietnam (schools are more heterogeneous)

• Differential effectiveness may mean triple disadvantage (two ways in which schools widen gaps)

• Reasons could include elitism, curriculum, language of instruction, discrimination

• A recent study Glewwe, Krutikova & Rolleston (EDCC, forthcoming) compares Vietnam and Peru – using both longitudinal household and school data

EQUITY ISSUES

Vietnam Peru-0.3

-0.2

-0.1

0

0.1

0.2

0.3

0.4

0.5

-0.18

0.41*

Proportion of 1 SD of maths test score distribut...

Difference in test score gains by increasing school quality by 1SD between richest 40% and poorest 60% of pupils

In Vietnam, schools equally effective in teaching Maths

In Peru, schools significantly less effective at teaching children from disadvantaged backgrounds

WITHIN THE SAME SCHOOL, DISADVANTAGED PUPILS MAKE LESS PROGRESS IN PERU, BUT NOT

IN VIETNAM

www.younglives.org.ukc.rolleston@ucl.ac.uk

FINDING OUT MORE