Mike TreadawayDirector of ResearchFischer Family Trust

Contents

FFT Live – Key Analyses

SecondarySecondary

FFT Live – Key Reports – Value Added

FFT Live – Key Reports - Estimates

FFT Live – Developments in 2010

KS4 (and KS5) Subject VA and Estimates

Value AddedValue Added

EstimatesEstimates

To Single Grade or Not to Single Grade?

Single Grade Estimates: Database• Single Grade estimate in the database• Calculated for subject groups

Single Grade Estimates: Database

FFTLive• FFTLive introduced a full range of probabilities

FFTLive

FFTLive – Single Grade

G F E D C B A A*Billy Onion 1% 8% 10% 20% 30% 20% 10% 1%

• Highlight the highest probability

Narrowing to the Middle• Let’s imagine a class of 10 pupils with exactly the same

estimates to Billy Onion

• Highlighted grade exported to the school MIS• The subject teacher sees.....

Estimate

Abigail Onion CBilly Onion CChristina Onion CDavid Onion CEmily Onion C

Estimate

Fatima Onion CGeoff Onion CHarry Onion CIsobel Onion CJanice Onion C

G F E D C B A A*Billy Onion 1% 8% 10% 20% 30% 20% 10% 1%

Narrowing to the MiddleEstimates should be averaged across the group

G F E D C B A A*Abigail Onion 1% 8% 10% 20% 30% 20% 10% 1%Billy Onion 1% 8% 10% 20% 30% 20% 10% 1%Christina Onion 1% 8% 10% 20% 30% 20% 10% 1%David Onion 1% 8% 10% 20% 30% 20% 10% 1%Emily Onion 1% 8% 10% 20% 30% 20% 10% 1%Fatima Onion 1% 8% 10% 20% 30% 20% 10% 1%Geoff Onion 1% 8% 10% 20% 30% 20% 10% 1%Harry Onion 1% 8% 10% 20% 30% 20% 10% 1%Isobel Onion 1% 8% 10% 20% 30% 20% 10% 1%Janice Onion 1% 8% 10% 20% 30% 20% 10% 1%Average 1% 8% 10% 20% 30% 20% 10% 1%Estimated no. of pupils 0 0 1 2 3 2 1 0

KS4 Development Report

Feature Details

Highlighting off as default Can be switched on if required

Chances shown as % for each grade

Option to show as cumulative

Option to highlight grades likely for top x% of students removed

Users found it confusing to have selections for both this and for school rank

Additional subjects Specific subjects within Science, MFL and Technology

Two Estimate Grades Est-2 : Grade likely to be achieved by, at most, 25% of studentsEst-1 : Somewhat more complicated …………………..

Est-1 : Explaining the mystery

Est-1 : Pros and Cons

Adding up highest probability grades

Too few A*/A and F/G gradesToo few A*/A and F/G grades

Calculating Points (from ordinal regression)

Issues – Accuracy and FairnessParticularly if used in context of evaluating progress of different teaching groups Issues – Accuracy and FairnessParticularly if used in context of evaluating progress of different teaching groups

Analysing Subject VA• A common approach is to use ordinal regression

• Issues with this are:– Fails (U grades)– Linearity– Granularity– Intervals

Fails and Linearity• KS2 Average English Level

• Similar pattern for over 80% of GCSE subjects at KS4• Using linear regression introduces significant errors for A*, A

and G grades

Granularity

• Regression Analysis (OLS) works well where inputs AND outputs are on a continuous scale

• Inputs are OK (fine grades)• Outputs are not – they are in clusters (grades)

– If we had e.g. UMS points … but we don’t!

Mathematics (random sample of 1000 records)

KS2 -> KS3 KS2 -> KS4

IntervalsIs the difference between A*/A, C/D and F/G grades the same?

YesYes NoNo Not sure, but probably noNot sure, but probably noYesYes

Responses are what we find whenever we ask subject leaders this question

<10% 30% 60%

We can debate whether or not their ‘gut feelings’ are justified

If, though, we can find a method of analysis which doesn’t care whether or not grade intervals are equal………

Solution

Outputs are chances not estimated points

NominalRegression

Likely Developments : Grades for Estimates

Estimates - Example

G F E D C B A A* A*C TM HTM TQ9% 23% 32% 26% 9% 0% 0% 0% 9% E 35% C3% 11% 24% 34% 25% 2% 0% 0% 27% D 27% B0% 0% 0% 2% 23% 38% 30% 7% 97% B 37% A*0% 0% 0% 4% 27% 39% 26% 4% 96% B 30% A*0% 0% 0% 1% 14% 34% 38% 12% 99% A 12% A*2% 8% 21% 35% 30% 3% 0% 0% 33% D 33% B0% 0% 1% 10% 45% 32% 11% 1% 89% C 44% A2% 8% 22% 39% 26% 2% 0% 0% 28% D 28% B0% 0% 0% 1% 13% 33% 39% 14% 99% A 14% A*0% 0% 0% 0% 6% 24% 44% 26% 100% A 26% A*

Estimates - TM

G F E D C B A A* A*C TM HTM TQ9% 23% 32% 26% 9% 0% 0% 0% 9% E 35% C3% 11% 24% 34% 25% 2% 0% 0% 27% D 27% B0% 0% 0% 2% 23% 38% 30% 7% 97% B 37% A*0% 0% 0% 4% 27% 39% 26% 4% 96% B 30% A*0% 0% 0% 1% 14% 34% 38% 12% 99% A 12% A*2% 8% 21% 35% 30% 3% 0% 0% 33% D 33% B0% 0% 1% 10% 45% 32% 11% 1% 89% C 44% A2% 8% 22% 39% 26% 2% 0% 0% 28% D 28% B0% 0% 0% 1% 13% 33% 39% 14% 99% A 14% A*0% 0% 0% 0% 6% 24% 44% 26% 100% A 26% A*

Estimates - TQ

G F E D C B A A* A*C TM HTM TQ9% 23% 32% 26% 9% 0% 0% 0% 9% E 35% C3% 11% 24% 34% 25% 2% 0% 0% 27% D 27% B0% 0% 0% 2% 23% 38% 30% 7% 97% B 37% A*0% 0% 0% 4% 27% 39% 26% 4% 96% B 30% A*0% 0% 0% 1% 14% 34% 38% 12% 99% A 12% A*2% 8% 21% 35% 30% 3% 0% 0% 33% D 33% B0% 0% 1% 10% 45% 32% 11% 1% 89% C 44% A2% 8% 22% 39% 26% 2% 0% 0% 28% D 28% B0% 0% 0% 1% 13% 33% 39% 14% 99% A 14% A*0% 0% 0% 0% 6% 24% 44% 26% 100% A 26% A*

Estimates - HTM

G F E D C B A A* A*C TM HTM TQ9% 23% 32% 26% 9% 0% 0% 0% 9% E 35% C3% 11% 24% 34% 25% 2% 0% 0% 27% D 27% B0% 0% 0% 2% 23% 38% 30% 7% 97% B 37% A*0% 0% 0% 4% 27% 39% 26% 4% 96% B 30% A*0% 0% 0% 1% 14% 34% 38% 12% 99% A 12% A*2% 8% 21% 35% 30% 3% 0% 0% 33% D 33% B0% 0% 1% 10% 45% 32% 11% 1% 89% C 44% A2% 8% 22% 39% 26% 2% 0% 0% 28% D 28% B0% 0% 0% 1% 13% 33% 39% 14% 99% A 14% A*0% 0% 0% 0% 6% 24% 44% 26% 100% A 26% A*

Exports

Issue 7 : Specific Subjects and Subject Types at KS4