College readiness and Cambridge International Examinations: what can the research show us and what does it really mean?

Val Sismey, Stuart Shaw, & John BarnhillAACRAO

April 3rd, 2012

Outline

• Setting the scene• The need for the research• The challenges• The data• The findings and their meaning

• Applications for admissions personnel

Setting the scene• A variety of acceleration mechanisms in the

USA

• AP

• Dual enrollment

• IB

• CLEP

• School enrichment programs

• Cambridge

Structure of the AICE program

• To gain the Cambridge AICE Diploma students must pass with at least six credits– Each International AS Level passed exam counts as one full credit– Each International A Level passed exam counts as a double credit

• Students must pass at least one subject exam from each group: Mathematics & Science, Languages, Arts & Humanities

• On completion of the Cambridge AICE Diploma, students receive either a Distinction, Merit or Pass

Development of the AICE ProgramPilot Study Comparing AICE with IB from 1997-2000 conducted by Florida DOE. Findings:

• Equally rigorous academically

• Both offer well-balanced curricula, high academic standards, practical real world applications, and international perspectives

• AICE emphasizes extremely flexible curriculum that can be tailored to individual students’ interests and future plans

School districts in Florida receive the same recognition, support

and state level funding for AS and A levels as AP and IB.

Cambridge International Education worldwide

9000 schools

160 countries

40 governments9000

schools160 countries

40 governments

CIE in the USA

Future developments

• Increasing reach in USA

• Excellence for All (BES) integrating relevant and rigorous curricula, professional development and assessments

• Collaboration with College Board AP

Why do we need to do research?

• To meet changing educational needs

• To feed into the assessment development process

• To reassure stakeholders that the students will be well prepared for university

The challenge of research work on college readiness: general issues

For tests used for university selection purposes it is vital to demonstrate predictive strength through relating secondaryschool performance to later academic performance

Long-term research purpose - highlight predictive strength ofCambridge Assessments and other students’ characteristics topredict success at U.S universities in terms of 1st year GPA

Establishing predictive strength is fraught with practicaldifficulties

The challenge of research work on college readiness: predictive success measure

Choice of predictive success measure

• a challenge to all models interested in prediction is the choice of predictive success measure

• SAT or ACT or HS GPA- reasonable predictors of university GPA (Cohn, Manion and Morrison, 2004; Noble

and Sawyer, 1987; Noble, 1991; Betts and Morrell, 1999)

- Culpepper and Davenport (2009) highlight the need to control for race in models involving SAT scores

- Kobrin, Patterson, Shawattern and Barbuti (2008) - SAT is an excellent predictor of how students perform in their first year at university

The challenge of research work on college readiness: university success measure

Choice of university success measure

• average GPA for 1st year (or other years if available)• number of courses passed• number of courses excelled in • GPA in certain courses e.g. science/mathematics vs.

humanities• university enrolment status (as of second fall after high

school graduation)• university retention i.e. re-enrolment in a 2nd year at same

institution

The challenge of research work on college readiness: design of study

1st Year

Undergraduate experience

End of first yearAverage freshman GPA

GraduationCumulative GPA

Predicting success

No Credit

Acceleration programme

(AICE, IB, AP, Dual

Enrolment)

AC

T /

SA

T

sco

res

University

Range of explanatory

variables

A

B C

Explore relationship between A and B; A and C

Explore A, B and C in light of other student characteristics(explanatory variables)

The challenge of a case study approach

• Case study methodology is “both a process of inquiry about the case and the product of that inquiry” (Stake, 2008, p. 121)

• justified as a mode of situated inquiry

• favouring uniqueness over generalizability

• Common challenge - controlling for selection bias– choice of educational program not necessarily random– High Schools have different characteristics and in mixed

Cambridge /non-Cambridge high schools students may have a choice

The challenge of a case study approach

• Study takes a case study approach: Florida State University

• Data from 3 cohorts of students (2007/2008; 2008/2009; 2009/2010) • largest data set (n = 6382) contained information on the SAT (or ACT) score - ‘no

credit’• Cambridge AICE credit (n = 144), with AP credit (n = 1188) and IB credit (n = 806)

• Data include information about each student’s performance at high school, ethnicity, gender and first year GPA

• Multilevel modelling used to investigate relationships between variables • in particular to determine best indicators of academic success at university, whilst

taking into account effects of individual high schools

FSU Overview

• Location: Tallahassee, Florida

• Type: Comprehensive, Research, Public

• Enrollment: 40,416 total (31,418 undergrad)

• FTIC Class: 6,155

• FTIC Profile: 3.7-4.2, 26-30 ACT, 1670-1870 SAT; 23% First Generation; 30% Minority

• Most popular FTIC Names: Sarah, Michael

FsusnLast Yr Enrolled

ClassDiv

Race SexDegree Hrs

FSUGpaH S Gpa

Mat Yr

Sat V

Sat M

Sat Total

Act

HS Code

MajorIB Hrs

AD0547664 2008 3 BD 1 F 61.00 3.94 0.00 2007 580 580 1160 25 100254219999

33.00

AM7719817 2010 4 CO 5 F 119.00 3.35 3.90 2007 610 630 1240 101466 418510 30.00

AT4238559 2010 4 VA 1 F 119.00 3.89 3.80 2007 550 520 1070 26 101583 420610 33.00

AV4125313 2010 4 VA 1 M 115.00 3.48 4.20 2007 510 610 1120 26 101514 428010 30.00

AV6039830 2010 4 BU 1 M 179.00 3.17 4.50 2007 630 680 1310 101530 214310 23.00

AZ1785985 2010 4 AS 1 M 146.00 4.00 4.70 2007 730 690 1420 34 100254 114212 37.00

BC2310720 2010 4 AS 1 M 115.00 3.47 3.60 2007 640 580 1220 27 100287 115210 30.00

BD7352404 2010 4 SS 1 M 130.00 3.43 3.90 2007 660 590 1250 28 100515328710

23.00

BH5728431 2010 4 IE 1 M 119.00 2.95 3.70 2007 540 680 1220 28 000000 555010 22.00

BH8767312 2010 4 AS 3 M 117.00 3.56 4.10 2007 510 560 1070 23 100254 111110 30.00

BQ2341259 2010 4 SS 5 M 114.00 2.58 3.50 2007 630 570 1200 101466329910

30.00

BQ4753086 2010 4 ED 1 F 126.00 3.58 4.30 2007 750 620 1370 101902220903

37.00

BR3793285 2010 4 HE 1 F 118.00 3.47 4.00 2007 590 550 1140 23 100289 255315 27.00

BS2413745 2010 4 BU 1 F 130.00 2.93 4.00 2007 620 670 1290 27 100254 216710 29.00

BS3684934 2010 4 HE 1 F 131.00 3.60 4.80 2007 560 560 1120 28 101437 254440 42.00

BS7834534 2010 4 SS 5 F 97.00 2.80 4.00 2007 640 640 1280 000000324910

34.00

BV7280201 2010 4 AS 1 F 131.00 2.66 4.30 2007 690 690 1380 100924 115210 35.00

BZ8785277 2010 4 HE 1 F 99.00 3.47 3.00 2007 570 550 1120 24 101663 255315 30.00

CB2794892 2010 3 AS 1 F 70.00 1.58 3.60 2007 600 560 1160 26 101663 114212 40.00

CH8903129 2010 4 AS 5 M 94.00 2.03 3.10 2007 590 530 1120 000000 117141 30.00

Four student data sets were provided: IB, AP, AICE and No Credit.

The findings and their meaning: hypotheses

Hypothesis 1: students who follow Cambridge AICE program achieve a higher1st university GPA than students who follow AP/IB/no credit programs (given sameSAT scores) Hypothesis 2: difference between males and females is smaller in terms of 1st yearuniversity GPA among students who follow Cambridge AICE program thanstudents who follow AP/IB/no credit programs (given same SAT scores) Hypothesis 3: difference between student ethnic groups is smaller in terms of1st year university GPA among students who follow Cambridge AICE programthan students who follow AP/IB/no credit programs (given same SAT scores) Hypothesis 4: difference between student ethnic groups and between gender issmaller in terms of 1st year university GPA among students who follow CambridgeAICE program than students who follow AP/IB/no credit programs (given same SATscores)

The findings and their meaning: multilevel outcomes

• Students who took the AICE attained, on average, a GPA of 0.35 higher than those with no credit, given the same SAT score

• Students who took the AP attained, on average, a GPA of 0.36 higher than those with no credit, given the same SAT score

• Students who took the IB attained, on average, a GPA of 0.22 higher than those with no credit, given the same SAT score

Base – No credit Regression Coefficient (Standard Error)

AICE 0.351 (0.053)

AP 0.359 (0.023)

IB 0.222 (0.026)

Compared to students with no credit (and controlling for the effects of SAT scores, gender and race) having taken the AICE, AP or IB programmes were all associated with significantly higher first year GPAs

Multilevel modelling findings

There is no evidence of any statistical difference between Cambridge AICE

and AP students on all of the tests carried out

It should be noted that school-level effects appeared to be much smaller

than the individual-level effects:

there is no statistical difference between schools

Findings: Hypothesis 1 - educational program

Effect of educational program (given equivalent SAT scores) on FSU GPA

Given equivalent SAT scores, the Cambridge AICE exam is associated with, on average,

0.142 higher on their FSU GPA than the IB

and 0.389 higher than having no extra credit

Base – Cambridge AICE Regression Coefficient (Standard Error)

AP with SAT -0.026 (0.058) IB with SAT -0.142 (0.060) no credit with SAT -0.389 (0.056)

Findings: Hypothesis 2 - gender

Base – Cambridge AICE, male Regression Coefficient (Standard Error)

AP -0.0096 (0.057) IB -0.139 (0.059) no credit -0.354 (0.055)

Effect of gender (given equivalent SAT scores) on FSU GPA

Cambridge AICE is associated with, on average, 0.354 higher GPA than no credit, controlling for gender and given equivalent SAT scores.

It is also associated with, on average, 0.139 higher GPA than the IB

Controlling for gender and SAT score closes the gap in FSU GPA between males and females for all groups of exam program

Findings: Hypothesis 3 - raceBase – Cambridge AICE, white Regression Coefficient

(Standard Error) AP 0.005 (0.056) IB -0.120 (0.058) no credit -0.377 (0.054) Black -0.250 (0.021) Asian -0.109 (0.033) Native American 0.101 (0.077) Hispanic -0.048 (0.019) Unreported -0.054 (0.060) Hawaiian -0.014 (0.143)

Effect of race (given the same SAT scores) on FSU GPA

Black students perform on average 0.25 points less well on their FSU GPA compared with white students, which is better (a smaller gap in performance) than when SAT score is not controlled for

Cambridge AICE students get, on average, 0.12 higher on FSU GPA than IB students, which is significant.

Asian and Hispanic students also do less poorly compared to white students, given equivalent SAT score, than if SAT score is not considered

Findings: Hypothesis 4 - gender and raceBase – Cambridge AICE, white Regression Coefficient

(Standard Error) AP 0.021 (0.055) IB -0.118 (0.057) no credit -0.343 (0.053) Black -0.249 (0.020) Asian -0.091 (0.033) Native American 0.092 (0.075) Hispanic -0.047 (0.019) Unreported -0.060 (0.059) Hawaiian -0.044 (0.141)

Effect of gender and race (given the same SAT scores) on FSU GPA

Black students have an FSU GPA that is 0.249 points lower than white students, which is a smaller gap in performance compared to when SAT score is not controlled for.

Cambridge students achieve, on average, 0.118 higher on FSU GPA than IB students, after controlling for race, gender and SAT score.

There is also a smaller gap in performance between Asian and Hispanic students compared to white students, given equivalent SAT score and after controlling for gender, than if SAT score is not considered.

Study limitations

Given: – smaller numbers in AICE, AP and IB groups– case study nature of research– possible presence of unknown confounding variables between groups

Unwise to draw conclusions about relative predictive strength of three

acceleration programmes

Further work required to collect more data from FSU and other U.S.

universities

Cambridge has already obtained considerably smaller datasets from

– universities of Maryland, Virginia, Michigan

Process of data collection expected to continue

Overview of findings

• Confirmation that more rigorous curriculum in high school pays off in better performance in college.

• Confirmation that our newest program, AICE, is just as good, if not better, than the programs with which we are most familiar.

• Confirmation that AICE prepares all students for college success.

• The need to do more research, particularly related to graduation and retention.

How can US admissions staff apply the findings to inform their policies?

• The importance of rigor in a h.s. curriculum. All gpa’s are not equal.

• Acceptance of AICE/Cambridge AS&A Levels as a program of rigor.

• Promote credit for AICE/Cambridge AS&A levels similar to policies that govern credit for AP and IB.

• The need for institutional specific research to shape your individual policies.