1

The impact of preparation on TSA and

BMATtestresults–aninstitutionalcase

studyatOxfordUniversity

FinalReportMay2019

Jenny Lenkeit, Daniel Caro and Hubert Ertl, Sam Cheadle, Gosia

Turner,AlisonMatthews,SaminaKhanandJo-AnneBaird

withShailenPopatandSamanthaCurle

DepartmentofEducation,UniversityofOxford

UndergraduateAdmissions,UniversityofOxford

StudentDataManagementandAnalysis,UniversityofOxford

Project sponsored by Cambridge English Language Assessment (Admissions TestingService)

OUCEA/19/3

2

Contents

Executivesummary..........................................................................................................................................5

Projectcontext................................................................................................................................................5

Theresearch....................................................................................................................................................5

Keyfindings.....................................................................................................................................................5

Testpreparationmatters......................................................................................................................5

Testpreparationdiffersbyschooltype..........................................................................................5

Resultson theTSAandBMATwere related to student characteristics, potentiallyraisingequityissues................................................................................................................................6

Implications.....................................................................................................................................................6

1. Introduction................................................................................................................................................7

Testpreparation............................................................................................................................................8

2. AnalysisofOxfordadmissiondata.................................................................................................13

2.1 Sample..............................................................................................................................................14

2.2 Variables.........................................................................................................................................14

2.3 Descriptiveanalysis...................................................................................................................16

2.3.1 TSAandBMATperformancebycharacteristicsofapplicants...........................17

2.4 Regressionresults.......................................................................................................................21

2.5 Consistencychecks.....................................................................................................................24

3. AnalysisofBrasenoseCollegesurveydata.................................................................................27

3.1 Amountofpreparation.............................................................................................................27

3.2 Supportfromschools................................................................................................................28

3.3 Useofexampreparationresources....................................................................................28

3.4 Perceptionsofthetests............................................................................................................28

4. InterviewswithOxfordstudents....................................................................................................29

5. Discussionofresultsfromtheexplorationofrelationshipbetweenstudentcharacteristics,testperformanceandtestpreparation(Phase1)...........................................31

Researchquestions...............................................................................................................................31

Limitations................................................................................................................................................33

6. Oxfordsurveydataandadmissiondata2017/2018cohort..............................................33

6.1 Development of survey instrument on Oxford student test preparation andadministration.............................................................................................................................................33

6.2 Data...................................................................................................................................................34

3

6.2.1 Responseratesandanalyticalsample..........................................................................34

6.2.2 Variables.....................................................................................................................................34

6.3 Descriptionoftheanalyticsampleandnon-respondents.........................................36

7. AnalysisofOxfordsurveydataandadmissiondata2017/2018cohort.......................37

7.1 Descriptiveanalysisoftheextentandnatureoftestpreparation........................37

7.1.1 Extentandnatureoftestpreparationbytesttakers..............................................37

7.1.2 Extentandnatureoftestpreparationbystudentbackgroundcharacteristics 44

7.2 Associations between student background characteristics, test preparationandtestperformance...............................................................................................................................50

7.3 Mediatingandmoderatingeffectsofstudentbackgroundcharacteristics.......55

7.3.1 Mediatingeffectsof studentbackgroundcharacteristics for theassociationbetweentestpreparationandtestperformance.....................................................................55

7.3.2 Moderatingeffectsofstudentbackgroundcharacteristicsfortheassociationbetweentestpreparationandtestperformance.....................................................................56

7.4 Associations between time of application, test preparation and testperformance.................................................................................................................................................57

7.5 Learningstrategiesandtestperformance.......................................................................57

7.5.1 Thelearningstrategyscales..............................................................................................57

7.5.2 Associations between learning strategies, student backgroundcharacteristicsandtestperformance...........................................................................................59

8. Discussionandlimitations.................................................................................................................60

Limitations....................................................................................................................................................61

Discussionofkeyfindings......................................................................................................................62

Towhatextentdostudentsprepareforuniversityadmissionstestsandwhatisthenature of this preparation amongst students with different backgroundcharacteristics?.......................................................................................................................................62

What are the connections between student characteristics, including priorqualifications,andtestpreparationandperformance?........................................................62

Isthetimeofapplicationconnectedwithpreparationfor,andbetterperformancein,tests?.....................................................................................................................................................64

Whatlearningstrategiesareconnectedwithbettertestperformance?.......................64

Implications..................................................................................................................................................64

References.........................................................................................................................................................65

Appendices........................................................................................................................................................68

4

AppendixA.TSAandBMATtables.....................................................................................................68

AppendixB....................................................................................................................................................72

AppendixC.BrasenoseCollegesurvey:selectedquotes...........................................................73

B1.Amountofpreparation................................................................................................................73

B.2Supportfromschools...................................................................................................................73

B.3Useofexampreparationresources.......................................................................................74

B.4Perceptionsofthetests...............................................................................................................74

AppendixD.Interviewguide.................................................................................................................76

AppendixE:SurveyInstrument...............................................................................................................79

5

Executivesummary

Projectcontext

TheprojectevaluatedtheimpactoftestpreparationforUniversityadmissionstestsontestperformancewithdatafromOxfordUniversity.Thetopicisofparticularrelevancebecauseanincreasingnumberofundergraduateprogrammes–nationallyintheUKbutalso internationally - require applicants to take an admissions test. The outcomes ofthese tests are considered for admission to University in addition to prior schoolattainment measures such as GCSE scores. However, how students prepare foradmissionstestsandifthispreparationhasanimpactontheirperformanceinthetestisanunder-researchedarea.

Theresearch

This project first explored the relationships between student characteristics and testperformance with Oxford University Admissions data from 2010 to 2015. Second,drawingoninformationfromtheBrasenoseCollegesurveyonhowstudentsprepareforadmissions tests, findings from a literature review and qualitative interviews with asample of students, a survey was developed. It asked for information on how far inadvance students started preparing for the test, the sources they used for testpreparation,thesupporttheyreceivedfromvariousagentsandthestrategiesstudentsemployedtoprepareforthetest.Third,thesurveywasadministeredtostudentswhocommencedtheirstudiesatOxfordUniversityinOctober2017anddatawasanalysedwithmultivariateanalysistechniques.

Keyfindings

Testpreparationmatters.

Nearlyall studentsprepare to take the test to someextentand frequentlyuseawiderange of online resources. The most popular sources were the Oxford UniversityWebsite,theofficialtestproviderwebsiteandTheStudentRoom,withthemajorityofthosewhousedthemfindingthemtobequiteorextremelyhelpful.

The most relevant aspect of test preparation that is positively related to testperformanceistheamountoftimestudentsstartpreparinginadvanceofthetest.

Testpreparationdiffersbyschooltype.

Schools play an important role in supporting students to take and prepare for theadmissionstestbyofferingthemopportunitiestoasksubject-specificquestionsorworkwith a subject teacher, receive general support/mentoring from a teacher andpreparinginworkshopsorclassesfortheadmissionstest.

6

Theamountofsupport thatstudentsreceivetoprepare for thetestsdiffersbyschooltypeandindicatorsofsocio-economicdeprivation.Studentsfromstateschoolsreceivefewer opportunities from schools to prepare for the admissions test. Students fromstate schools and those with an Overall WP Flag also have the least access to otherforms of support: they received feedback on the past test papers they practiced lessoften and were less likely to have talked to former pupils at their school/college orformerorcurrentOxfordstudentswhotookthetest.

ResultsontheTSAandBMATwererelatedtostudentcharacteristics,potentially

raisingequityissues.

A-Level results were highly associated with TSA and BMAT test performance andthereforesuccessofbeingadmittedtoanOxfordUniversityundergraduateprogramme.A-Level results remain positively related to test performance even for students whowereadmittedtothecourseforBMATbutnotforTSA.

Other characteristics being equal, BME applicants, female applicants and those fromdeprivedsocio-economicareasperformed lesswell inboth theTSAandBMAT.Thesecharacteristics were associated with lower performances in TSA and BMATperformanceevenforstudentswhowereadmittedtoacourse.Students’demographicand socio-economic background characteristics explain a larger variance of BMATscoresthanTSAscores.

Startingtoprepareearlyforthetestscounterstherelativedisadvantageassociatedwithsocio-economic deprivation, state school attendance and gender in TSA performancebutnotinBMATperformance.

Implications

The results of this project give evidence on how prior attainment and studentbackground characteristics are related to performance in admissions tests. Thisunderstanding is essential for Universities’ policies on equality. For the University ofOxford, specifically, the resultshavedirect implications for itsoutreachwork, as theyprovide clear evidence that admissions tests and support in preparing for them is ahigherhurdleforapplicantsfromunder-representedgroups–especiallythoseinstateschoolsandfromsocio-economicallydeprivedareas–forwhomadviceandsupportfortaking these tests is essential. In addition, test providers may re-evaluate availablepreparation materials as not all of them are used with similar frequency or areperceivedashelpfulbyallstudents.

7

1. Introduction

This study investigated the preparation of applicants for admissions tests forundergraduate programmes at the University of Oxford. At Oxford, admissionsprocessesofagrowingnumberofundergraduateprogrammesincludeadmissionstestsand the results of the tests influence decisions onwhether applicants are invited forinterviewsand,potentially,whethertheyareofferedaplace.

Admissions tests are becoming more important in admissions processes in selectiveEnglish higher education institutions (HEIs) but preparation is an under-researchedarea (Kirkup et al., 2010).While a 2010BIS report concluded that SAT-style tests do‘not add any additional information, over and above that of GCSEs and A levels (orGCSEsalone),atasignificantlyusefullevel’(Kirkupetal.,2010,4),selectiveuniversitieshave foundadmissions testshelpful indistinguishingbetween largenumbersofhigh-achievingapplicants.

HEIs in theUKoften considerprior attainment atGCSEalongwithother informationfromcandidates for admission touniversity.Withevidenceonprior attainment,HEIsaim to judge the suitability of candidates for the course and their ability to completecourses successfully. Admissions selection criteria are consistent with meta-analysesshowingthatpriorattainmentiscorrelatedwithperformanceinuniversitycourses.Therelationship,however,isnotlarge.Previousacademicperformancehasaneffectsizeof.30 in medicine, explaining for example 23% of variance in overall performance inmedicalschool(Ferguson,James,&Madeley,2002).Forsocialscienceandhumanities,effectsizesofd=.31 tod=.67were found(Peers& Johnston,1994).Particularly,priorattainmentdoesnotdiscriminatewellamonghighlyqualifiedcandidates.ThisisoneofthereasonswhysomeHEIswithhighlyselectivecourseshave introducedadmissionstestsaspartofadmissioncriteria.AnotherreasonisthatmanystudentsapplytoHEIswith different qualifications and, compared to the constantly changing nationalqualificationstakenbyschoolstudents,admissionstestsprovideacommonbenchmarkofacademicability.Andstillanotherreasonisthatadmissionstestsmakeitpossibletocomparestudentsfromverydifferenteducationalbackgrounds.

AtOxford, an increasingnumberof undergraduateprogrammes require applicants totake an admissions test. Of the 248 undergraduate courses offered in 2016, 232requiredapre-interviewtestortestsandonly14courseshavenotestatall.Thereare11separate tests,althoughsomeof these,suchasModernLanguages,aremadeupofmultiple individual papers. In total 85%of all applicants registered for pre-interviewtestsin2015.TheThinkingSkillsAssessment(TSA)iscurrentlyusedforadmissionstonine Oxford undergraduate programmes (Economics and Management, ExperimentalPsychology, Geography, History and Economics, Human Sciences, Philosophy andLinguistics, PPE, Psychology and Linguistics, Psychology and Philosophy). TheBiomedical AdmissionsTest (BMAT) is used forMedicine andBiomedical Sciences at

8

Oxford.TheTSAandBMATaredevelopedandadministeredbyCambridgeAssessmentAdmissionTesting(CAAT)andarisefromthethinkingskillstradition(Fisher,2005).

The thinking skills tradition emerges from work in psychology and philosophy.Although there aredifferent strands, they share the common idea that thinking skillscanbetaughtdirectlyaspartofthecurriculum,andthattheseskills involvecognitiveprocessessuchasarguingacase,decision-making,problemsolving,explainingcauses,evaluating, comparing and contrasting, judging credibility, clarifying and interpretingideasandotherhigher-orderthinkingskills(Fisher,2005).

Extensive research, including meta-analyses and experimental studies, shows thatthinking skills teaching programmes can promote higher educational attainment andtransferable skills across different subject areas and contexts (Adey & Shayer, 1994;Fisher, 2005; Higgins & Hall, 2004). Thinking skills are particularly relevant at thehighereducation level,wherestudentsaregenerallyexpected to think for themselvesandworkindependently.Consistently,researchshowsthatthinkingskillstestsliketheTSAandBMATpredictcoursedegreemarksinuniversityoverandaboveA-levels(e.g.,Black, 2012; Emery, 2009;O’Hare&McGuinness, 2015). That is another reasonHEIsconsiderthesetestsaspartoftheadmissionselectioncriteria.

TheTSAhas two sections. The first is composed of 50multiple-choice questions andstudentshave90minutestocompletethem.Thesecondsectionisawritingtask,whichstudentshave30minutestocomplete.Asthewritingsectionscoresarenotusedaspartof the selection process toOxfordUniversity courses, data from section 2 of the TSAwere not included in this research. TSA section 1 addresses “problem-solving skills,including numerical and spatial reasoning. Critical thinking skills, includingunderstandingargumentandreasoningusingeverydaylanguage.”1

The BMAT is composed of three sections; candidates are allowed 60minutes for thefirst sectionand30minutes for sections2 and3,making the examination2hours induration.The firstsection iscomposedof35multiple-choicequestionsandaddresses“generic skills in problem solving, understanding arguments and data analysis andinference.” Section2 contains27multiple-choicequestions,which test “The ability toapplyscientificknowledgetypicallycoveredinschoosScienceandMathematicsbytheageof16(forexample,GCSEintheUKandIGCSEinternationally).”Thefinalsectionisawriting task inwhich candidates choose fromoneof threequestionsdesigned to test“Theabilitytoselect,developandorganiseideas,andtocommunicatetheminwriting,conciselyandeffectively.”2

Testpreparation

Testpreparationhasbeenthesubjectofconsiderableresearchmorebroadlyandwasintensively researched in the 1980s in the US, in relation to the SATs (university

1 https://www.admissionstesting.org/for-test-takers/thinking-skills-assessment/tsa-oxford/about-tsa-oxford/

2https://www.admissionstesting.org/for-institutions/about-our-tests/biomedical-admissions-test/

9

entrance tests).A general conclusion from thisworkwas that no test candistinguishbetween aptitude and performance: they assess performance (Powers, 1985; Slack&Porter, 1980). It hadpreviously been argued that some tests could assess underlyingpotential,evenifastudentwasnotcurrentlyperformingatahighlevel.Arecentcaseinwhich this notion was disabused was Durham University’s Centre for EducationalMeasurement’s11+test,asresearchindicatedthattestpreparationhadaneffectuponoutcomes.3 Thisrecentexampledemonstratesthatdespitetheearlierresearchinthisarea,perceptionsthatapureformofpotentialcanbeassessedpersistinsomequarters.

Studies on test preparation, cramming and drilling have found increases in students’scores(e.g.Bangert-Drowns,Kulik,andKulik1983;BuntingandMooney2001;MessickandJungeblut1981;Powers1986;Sturman2003).Accesstotestpreparationmaterials,similarityofthematerialstotheactualassessments,thenumberofpracticetests,timeontaskandhigherabilitystudentshaveallbeenassociatedwithhigherperformances(Kulik, Bangert-Drowns & Kulik, 1984; Powers and Swinton, 1984; Griffin, Carless &Wilson,2013).Analyticalandquantitativetasksshowhighergainsfromrepeatedtest-takingthandoverbaltasks(Hausknecht,Halpert,DiPaolo&MoriartyGerrard,2007).These studies have typically been conducted for tests that do not have an associatedcurriculum,justliketheBMATandtheTSA(unlike,forexample,A-levels).Section2ofthe BMAT does assess scientific knowledge, though, so may operate more likecurriculum-relatedassessmentsforthatproportionofthetest.

Fourexplanations forstudents’ scoregainsexist in the literature (Arendasy,Sommer,Guiterrez-Lobos&Punter,2016)(Table1.1).Thefirststatesthattestfamiliarityinitselfis a cause of increased scores, even if this gain is not relevant to the construct beingassessed.Suchgainsdonotimpactuponstudents’abilities,onlyontheirperformances.Asecondmodelpositsthatdrillingontest-specifictechniques(suchashowtoeliminateoptionalresponseswithoutadeepunderstandingof thequestion)wouldhavesimilareffects to model 1 in terms of impact upon performance and ability. A third modelinvolvesteachingconstruct-relevantmaterialtostudentssotheyperformbetterontestquestionsandthisimpactstheirabilityinadomain-specificmanner.Afourthmodelisthatofgeneralteachingandlearningstrategieswhichwouldbeexpectedtobothimpactuponperformanceandproducegeneralisedimprovementsinstudents’abilities.

Table1.1:Testpreparationmodels

Model Construct

relevance

Cause Impacton

performance

Impacton

ability

1 No Testfamiliarity Yes No

2 No Testcoachingtipsandtricks

Yes No

3https://www.theguardian.com/education/2016/sep/12/tutor-11plus-test-grammar-schools-disadvantaged-pupils

10

3 Yes Itemsolutionstrategies

Yes Yes:test-ordomain-specific

4 Yes Teaching Yes Yes

InresearchonamedicalentranceexaminationattheUniversityofVienna,Arendasyetal.(2016)demonstratedthatModel3bestexplainedtheirfindings.Testpreparation,intheir sample of students, had test-specific and domain-specific effects but in keepingwith other studies, the gains from test preparation did not transfer to generalintelligence(g)oreventogeneralnaturalscienceknowledge.Itwaspreviouslyknownthat practising test items did not increase general intelligence (Arendasy & Sommer,2013),butthefindingwithrespecttodomain-specificgeneralisationwasnew.

Preparation for thinking skills admissions tests like the TSA and BMAT is an under-researchedarea.There is evidence thatGCSEandA-levels results correlatepositivelywithSAT-styletests(Kirkupetal.,2010),butlittleisknownaboutpreparationfortheTSAandBMATtests.InarecentstudyGallacher,McElweeandCheung(2017)studiedpreparationstrategiesfortheBMATwithsurveydatafromaconveniencesample.TheauthorsfoundthatthegreatmajorityofcandidatesusetheBMATwebsiteforaccessingpreparationmaterials. Regression analysis results suggested that attempting practicepapersundertimedconditionswasaneffectivestrategyforhighertestscoresacrossallBMAT sections. There was no indication that extra help from schools, commercialcoursesorcomingfromanIndependentschoolwaspositivelyrelatedtohigherBMATscoresonceotherfactorswerecontrolledfor.

Conceptually, enhancing thinking skills through educational programmes and studies(GCSE,A-Levels,thinkingskillscourses)shouldbedistinguishedfrompreparationsthataimto improve test-takingskills.While the formerwouldsupportskillsnecessary forstudents’ study, the latter would merely develop strategies to enhance test-takingskills.4

ThisstudywillinvestigatepreparationfortheTSAandBMATwithdatafromon-courseUniversityofOxfordstudents.Thefollowingresearchquestionswillbeaddressed.

Researchquestions

Theguidingresearchquestionthisstudyseekstoansweris:

- What is the impact of preparation for university admissions tests on testperformance?

Thisquestioncomprisesanumberofsub-questions:

4 This project did not aim to disentangle the two concepts of preparation for thinking skills tests(theoreticallyorempirically)andinstrumentshavenotbeendesignedaccordingly.Whereappropriateadistinctionisattemptedinthediscussionofresults.

11

- Towhatextentdostudentsprepareforuniversityadmissionstestsandwhatisthe nature of this preparation amongst students with different backgroundcharacteristics?

- Whataretheconnectionsbetweenstudentcharacteristicsandtestpreparationandperformance?

- Whichpriorqualificationsandsubjectsareconnectedwithtestpreparationandbettertestperformance?

- Is the time of application connected with preparation for, and betterperformancein,tests?

- Whatamount, timingandnatureofpreparationare connectedwithbetter testperformance?

- Istheinfluenceoftestpreparationontestperformancemediatedormoderatedbystudentcharacteristicsandbackground?

- What learningstrategiesandsourcesofsupportareconnectedwithbetter testperformance?

Studyphases

Theresearchquestionsareaddressedintwoconnectedphases.

Phase1:Explorationofstudentcharacteristics, testperformanceandadmissionstestspreparation

1. Exploration of the relationship between student characteristics and testperformanceMultiple regression analysis examined the association between applicantcharacteristics and performance in TSA andBMAT tests for students from theUK. A number of characteristicswere considered, such as BME status, gender,priorattainment(GCSEandA-Levelresults),andschooltypeorigin.One of the underlying questions was whether certain GCSEs and A-Levelsincreasethelikelihoodofgoodtestresultsornot,i.e.whichqualificationsseemto provide a good preparation for these two admissions tests. This analysisallowedustodeterminewhethertestresultdifferencesthatseemtobelinkedtocandidatebackgroundpersistwhenGCSEandA-Levelattainmentwascontrolledfor. The work of student-level data also provided insights into connectionsbetween test results and other background variables (e.g., type of schooling,POLARquintile).

2. AdditionalanalysisbasedonstudentsurveyresponsesInaseparatesurvey,collectedbyBrasenoseCollegeatOxfordoverthelastthreeyears, studentswere asked in an open-ended questionwhether and how they

12

preparedforadmissionstests,includingwhethertestpreparationwasprovidedby students’ schools orbasedon students’ own initiative.Thematic analysis ofthis data allowed us to identify potential connections between amount andnatureofpreparation,typeofschoolingandotherbackgroundvariables.

3. Application date as a proxy for the amount of test preparationPreliminary analysis of with Brasenose College admissions data has revealedthat thedateof application is significantly related to admissions success.Aftercontrolling for GCSE performance, applicants who apply early have a higherchanceofobtaininganofferandgainingaplace.Onepossibleexplanationisthatearlyapplyingapplicantshavemoretimeandsupporttoprepareforadmissionstests and are thereforemore likely to achieve higher test scores and a higherproportion of offers. The study will explore this and alternative explanationsfurther, using the university wide dataset, and directly examining therelationship between application date and TSA and BMAT performance.

4. Qualitative investigation of the impact of preparation on test experience andresultsBasedoninsightsgainedin1–3andpreviousresearchontestpreparationthisstudy generates qualitative data to explore the usefulness of differentpreparationmaterials andmodes. This was achieved by conducting follow-upinterviewswithasampleofsurveyrespondentswhohaveexperienceddifferentlevels and types of preparation. The interviews allowed us to explore in aqualitativemannersomeofthewidely-heldassumptionsregardingpreparationforuniversityadmissionstests,forinstancethatmaterialrecommendedbytestdevelopers is the focus of students’ preparations. Most importantly theinterviewsinformedthedevelopmentofthesurveyinstrumentinPhase2.

Phase2:Analysisofadmissionstestspreparationandtestperformance

Basedonthe findingsofPhase1asurveywasdevelopedandpiloted(AppendixE) tonewlyadmittedundergraduatestudentswhocommencedtheirstudiesinOctober2017.Thesurveyincludesitemsinthefollowingareas:

• Howearlystudentsstartedpreparingfortheadmissionstests• Howmuchstudentspreparedfortheadmissionstests• Useofpasttestpapersandwaysofpracticingwiththem• Sourcesofsupportinpreparationprocess• Usefulnessofdifferenttypesofpreparationmaterials

13

Additionally, the survey included the scales on student learning strategies (e.g.,memorisation, elaboration, and control strategies), widely used in the context of thePISAstudy(Marsh,Haug,Artelt&Baumert,2006;Säälik,Nissinen&Malin,2015)andadaptedforhigh-stakestests inpreviousresearchbytheOxfordUniversityCentreforEducational Assessment (Baird, Hopfenbeck, Elwood, Caro & Ahmed, 2015). Thequestionnaire includes various types of questions, ranging from closed questions, toLikert scale items and open-ended questions. The survey is administered by OxfordUniversity’sStudentDataManagementandAnalysisteamviaOnlineSurveys(formerlyBOS) and piloted with a substantial sample of incoming students commencing theirstudiesinOctober2017.Thesurveyislinkedtostudentrecords,whichallowsanalysissimilar to thatundertaken for smaller samplesof students inPhase1, but employingmuchmoredetaileddataontestpreparationandsophisticatedmethods.

Next to descriptive statistics, multivariate analysis is employed. Factor analysisevaluatedthestructureofstudentresponses(e.g.,learningstrategiesscale).Regressionanalysis examined the association between student preparation variables and testresultswhilst taking intoaccountdifferentialeffects forbackgroundcharacteristicsofstudents (e.g., BME status and gender). The latter is in linewith findings from large-scale assessment suggesting that preparation strategies vary by the student's genderandsocio-economicstatus(e.g.,OECD,2010).Thedatafromthesurveywillalsobeusedto explorewhether the influenceof testpreparationon testperformance ismediated(explained) or moderated (differentially affected) by student characteristics andbackground.

Aimandstructureofthereport

ThisFinalReportpresentstheresultsofPhase1andPhase2.Itisstructuredasfollows:

Sections 2 to 5 discuss the findings from Phase 1. Section 2 reports analysis of theassociation between TSA/BMAT scores and characteristics of students (e.g., GCSE/A-levelsandsocio-economicbackground)usinglarge-scaleOxfordadmissiondataacrossfiveyearsanddifferentcourses.Section3presentsopen-endedresponsesofstudentsatBrasenoseCollegeonhowtheyprepared foradmissionexams.Section4presents theprocedure of developing and administering interviews with 10 candidates. Section 5discussesmainresultsofPhase1.

Sections6to8reportonthefindingsfromPhase2.Section6outlinesthedevelopmentof the survey, the sample and used variables for the 2017/2018 cohort. Section 7presents the analysis and results including descriptive statistics and associationsbetween test preparation, student background characteristics and test performance.Section8discussedtheoverallfindingsoftheproject.

2. AnalysisofOxfordadmissiondataThis sectionpresents resultsof analysisofOxfordadmissiondataon the relationshipbetween test scores on admissions tests (i.e., TSA and BMAT) and applicants’

14

characteristics. Results of descriptive analysis and regression analysis are presented.Twosamplesareconsideredforanalysis,onefortheTSAtestandanotherfortheBMATtest.

2.1 Sample

The total TSA sample consisted of 9,886 home (UK domiciled) applicants across the2010 – 2015 Universities and Colleges Admissions Service (UCAS) cycles. The totalBMATsampleconsistedof7,175homeapplicantsacrossthe2009–2015UCAScycles.Formultivariateanalysis,thesamplewasfurtherrestrictedtoapplicantswithGCSE/A-level results and relevant socio-economic characteristics (available by UK postcode),resultinginafinalTSAsamplesizeof7,032andafinalBMATsamplesizeof5,301.TheTSA sample included applicants for the following courses: Economics&Management(n=2,169),ExperimentalPsychology(n=741),Geography(n=674),‘Politics,Philosophyand Economics’ (n=3,168) and ‘Psychology, Philosophy and Linguistics’ (n=280). TheBMAT sample included applicants to the following courses: Biomedical Sciences(n=190)andMedicine(n=5,111).5

2.2 Variables

ForbothTSA6andBMATscorestheoverallscoreswereusedintheanalysis.TSAandBMATscoreshavebeenstandardisedbyZ-scoring.NotethatscoreswerestandardisedwithineachUCAScycleandcoursegroupseparately.Thiswascarriedoutfortwomainreasons. (1) Standardisationwithin eachUCAS cycle aims to account for year to yearfluctuationsinthedifficultyofthetest.Notethatwithincyclestandardisationwasalsoapplied to GCSE and A-level attainment measures. (2) Due to the limitations in thecentral admissions database (ADSS), scores supplied by Cambridge Assessment havebeen recorded for all courses taking TSA as well as medicine applicants taking theBMATacrosstheUCAScycles.InBiomedicalSciences,BMATscoreshavebeenrecordedin a standardised format across the UCAS cycles. This left no other option than tostandardisealltestscoreswithineachcourseseparately.7 Further,BMATscoreswereweightedaccordingtoOxford’sselectionprocedures.8

The following variables were considered as statistical predictors of TSA and BMATscores.

5Note:NumbersdeviatefromtheInterimReport,astestscoresfortheBiomedicalSciencescoursehavebeen incorrectly assigned to TSA due to ADSS test storage issues. Subsequent analysis has beencorrectedforthiserrorinthepresentreport.

6FortheTSAonlythemultiplechoicesectionsassessingcriticalthinkingandproblemsolvingskillsareincludedintheoverallscore.

7Note:StandardisationofBMATtestscoresacrossMedicineandBiomedicalSciencescoursesdoes,dueto thedifferent formats inwhich thescoreswererecorded,noteliminatecomparability issuesof thescores. However, Appendix B compares the analysis conducted in Table 2.3 for BMAT with thecombinedMedicineandBiomedicalSciencescoreswithresultsforBMATscoresofMedicinestudentsonly.DifferencesinresultsarenegligibleandsubsequentanalysisforBMATisconductedandreportedwiththecombinedstandardisedscoresfromMedicineandBiomedicalSciences.

8https://www.medsci.ox.ac.uk/study/medicine/pre-clinical/statistics

15

• School type: Dummy variables indicate applicant’s school type (state,independent,orother).Stateandotherschoolsarecomparedwithindependentschoolsinregressionmodels.The‘other’categoryismainlyoverseasschools.

• BME:Dummyvariablesindicateblackandminorityethnic(BME)status(yes,no,ordon’tknow/refuse).

• Application time: The number of days between 1st September and theapplication date (UCAS form received by Oxford). The reference date of 1stSeptember was chosen because this is close to the application deadline. Theprecise reference date does not affect the statistical results. Smaller valuesreflectearlyappliers.Resultingregressioncoefficientsweremultipliedby10foreaseofinterpretation.

• ACORNFLAG:Dummyvariable indicates socio-economic adversity according toUKpostcode.Whereacandidate’spostcodefallsintoACORNgroupsfourorfive('FinanciallyStretched'and'UrbanAdversity')theapplicationwasflagged.

• POLAR3FLAG:Dummyvariable indicates lowparticipation inhigher educationaccordingtoUKpostcode.ThePOLARclassificationplacesregionsintoquintilesbased on the rate of young participation inHigher Education. Applicants fromthelowesttwoquintileswereflagged.

• PRE16SCHOOLFLAG: Dummy variable indicates performance of applicant’sschoolatGCSEisbelownationalaverage.

• POST16SCHOOLFLAG: Dummy variable indicates performance of applicant’sschoolatA-levelisbelownationalaverage.

• WPOVERALLFLAG(wideningparticipation):Dummyvariableindicateswhethertheapplicanthasobtainedoneofmoreschoolflags(preorpost16)andoneormoregeodemographicflags(ACORNorPOLAR3)orhaspreviouslybeenincare.

• GCSEA*s:ThenumberofA* grades (only including applicantswith5ormoreGCSEs).GCSEscoresweretransformedintoz-scores.

• A-levelA/A*s:ThenumberofAorA*grades. A-levelsscoresweretransformedinto z-scores. (Note that A-levels are taken after the TSA or BMAT for moststudents,butwere included in themodelasaproxy forapplicants’ concurrentattainmentlevels.)

• Schoolhistoryofapplication:Itisequaltotherawnumberofapplicantsbetween2011and2016,perschool.ItreflectstheschoolhistoryofapplyingtoOxford.

• UcasCycle:This is theapplication cycle inwhichanapplicationwasmade, andrefers to the year of entry for the majority of those applying. For example,applications made in the 2014 Universities and Colleges Admissions Service(UCAS) Cycle were made by 15 October 2013, for entry in October 2014(majority),ordeferredentryinOctober2015(minority).

• A-level subjects: Dummy variables indicate applicant’s A-levels choices (Art,Biology, Business Studies, Chemistry, Computing, Economics, English, EnglishLanguage, English Literature, Further Mathematics, Further Additional,Geography, History, Languages, Law, Mathematics, Philosophy, Physics,Psychology,ReligiousEducation,ScienceOther,SocialScience,andOther).

16

2.3 Descriptiveanalysis

Table 2.1 and 2.2 report descriptive statistics for selected variables in the TSA andBMATsamplesanalysed.Percentageofapplicantsandmeantestscoresarereportedbyschooltype.

Table 2.1 and 2.2 show that the percentage of applicants from disadvantaged areasaccording to ACORNFLAG and POLAR3FLAG is relatively low in independent schools(around4%forTSAapplicantandaround7%forBMATapplicants),buthigherinstateandotherschools(around14%inTSAapplicantandbetween16%and20%inBMATapplicants). In termsof educational attainment, candidates from independent schoolsperformbetteratGCSEandA-levelsthancandidatesfromstateandotherschools.GCSEandA-levelperformanceisparticularlylowforcandidatesfromotherschools.

Table2.1:TSAanalyticsample:descriptivestatistics(%ofapplicantsandmean

scores).Home(UKdomiciled)applicantsonly(N=9,886).

School,type Independent Other StatePercentagesFemale 38% 34% 42%BME 22% 30% 24%ACORNFlag 3% 14% 13%POLAR3Flag 5% 15% 14%Pre16SchoolFlag 2% 7% 12%Post16SchoolFlag 1% 7% 29%OverallWPFlag 1% 6% 10%MeansGCSE(z-score) 0.44 -0.68 -0.18A-levels(z-score) 0.14 -0.33 -0.13TSA* 62.31 59.31 60.45TSA(z-score) 0.36 -0.10 0.11

*Excludes recorded TSA scores of less than 10 to exclude outliers which were considered to beunexplainedmissingdata.Note:HomeapplicantswhohavetakentheTSAonly(2010–2015UCAScycles).Forcomparability,GCSEandA-levelsscoreswerescaled(i.e.,z-cores)tohaveameanofzeroandastandarddeviationofone.

Table2.2:BMATanalyticsample:descriptivestatistics(%ofapplicantsandmean

scores).Home(UKdomiciled)applicantsonly(N=7,175).

School,type Independent Other StatePercentagesFemale 52% 60% 54%BME 35% 44% 38%ACORNFlag 6% 20% 16%POLAR3Flag 8% 16% 17%

17

Pre16SchoolFlag 3% 6% 15%Post16SchoolFlag 1% 7% 29%OverallWPFlag 1% 8% 13%MeansGCSE(z-score) 0.38 -0.85 0.00A-levels(z-score) 0.23 -0.54 0.00BMAT* 56.85 49.09 54.13BMAT(z-score) 0.35 -0.39 0.09

*ExcludesrecordedBMATscoresoflessthan10.Note:Homeapplicantswhohave taken theBMATonly (2009–2015UCAS cycles). For comparability,GCSEandA-levelsscoreswerescaled(i.e.,z-cores) tohaveameanofzeroandastandarddeviationofone.

2.3.1 TSAandBMATperformancebycharacteristicsofapplicantsAs with GCSE and A-levels results, Table 2.1 and 2.2 suggest that applicants fromindependent schools perform slightly better in the TSA and BMAT compared toapplicants from state schools and other schools.Note thatmature students aremorelikelytobeincludedinthe‘other’category,withapproximately17%beingaged19oroverinthisgroup,comparedwith5%intheoveralldataset.

Wewantedtoevaluatewhetherperformancegapsinadmissionstestsacrossdifferentschools were explained by prior attainment of candidates. Figure 2.1 displays TSAperformance by school type for candidates with equivalent GCSE results, that is,belongingtothesameGCSEattainmentquartile(seealsoTableA1inAppendixA).Thex-axis locates candidates according to GCSE quartiles and the y-axis depicts TSAstandardisedscores.

When we look at candidates with equivalent GCSE results, evidence of a TSAperformance gap favouring independent schools can only partly be confirmed.CandidatesofindependentschoolsandstateschoolsperformfairlysimilarlyintheTSAtest, particularly at higher GCSE levels. There is only weak evidence that candidatesfrom independent schools perform better than candidates from state schools at thelowestGCSEquartile.ANOVAanalysisconfirmedthatthedifferenceissignificant(F(1,9068) = 6.06,p = 0.014). Still, it is the attainment in GCSE that explainsmost of thevariationinTSAscoresbetweenstateandindependentschools.

These results are explored further in regression analysis controlling for morecharacteristicsinthenextsection.

Figure 2.1: AverageTSA standardised scores byGCSE quartiles and school type

(homeapplicantsonly)

18

Figure 2.2 presents the TSA performance for female and male candidates by GCSEquartile (see also Table A2 in Appendix A). Results consistently show that malecandidates performed better than female candidates in TSA across GCSE attainmentlevels.ANOVAanalysisconfirmedthatthedifferenceissignificant(F(1,9293)=340.02,p < 0.001). The gender gap is relatively stable across students with different GCSElevels.

Figure2.3depictsthegapinTSAoutcomesforBMEandWhitecandidatesacrossGCSEquartiles(seealsoTableA3inAppendixA).CandidateswithBMEbackgroundperformworse inTSA thanWhitecandidates.ANOVAanalysisconfirmedthat thedifference issignificant (F (1, 9289) = 130.30, p < 0.001). The TSA score gap related to BMEbackground appears to be larger for students with lower GCSE scores, in the firstquartile(F(3,9289)=6.39,p<0.001).

Figure2.2:AverageTSAstandardisedscoresbyGCSEquartilesandgender(home

applicants only)

-1.0

-0.5

0.0

0.5

1.0

Q1 Q2 Q3 Q4

Sta

nd

ard

ise

d T

SA

te

st

sco

re

GCSE attainment quartile

Independent State

-1.0

-0.5

0.0

0.5

1.0

Q1 Q2 Q3 Q4

Sta

nd

ard

ise

d T

SA

te

st

sco

re

GCSE attainment quartile

Female Male

19

Figure 2.3: Average TSA standardised scores by GCSE quartiles and BME

background

Similarly,Figures2.4to2.6displayperformancegapsinBMATscoresacrossdifferenttypes of school, student gender and BME status by prior attainment of candidates.Candidates from independent schools score significantlyhigher than those fromstateschools(F(1,6595)=48.50,p<0.001).ThedifferenceisstableacrossGCSEquartiles(F(3,6595)=0.96,p=0.41.N.S.)(Figure2.4).

Figure 2.5 presents the BMAT performance for female andmale candidates by GCSEquartile.ResultsconsistentlyshowthatmalecandidatesperformedsignificantlybetterthanfemalecandidatesinBMATacrossGCSEattainmentlevels(F(1,6745)=324.48,p<0.001).ThegendergapisstrongeratlowerandhigherGCSElevels(F(3,6745)=2.77,p<0.05).

Figure 2.6 presents the BMAT performance for BME status by GCSE quartile. ResultsconsistentlyshowthatcandidateswithaBMEstatusperformedsignificantlylowerthancandidateswithoutBMEstatus inBMATacrossGCSEattainment levels (F(1,6423)=182.90,p < 0.001). The gap is stronger at lowerGCSE levels (F (3, 6423) = 9.83,p <0.001).

-1.0

-0.5

0.0

0.5

1.0

Q1 Q2 Q3 Q4

Sta

nd

ard

ise

d T

SA

te

st

sco

re

GCSE attainment quartile

White BME

20

Figure2.4:AverageBMATstandardisedscoresbyGCSEquartilesandschooltype

Figure2.5:AverageBMATstandardisedscoresbyGCSEquartilesandgender

Figure2.6:AverageBMATstandardisedscoresbyGCSEquartilesandBMEstatus

-1.0

-0.5

0.0

0.5

1.0

Q1 Q2 Q3 Q4

Sta

nd

ard

ise

d B

MA

T t

est

sco

re

GCSE attainment quartile

Independent State

-1.0

-0.5

0.0

0.5

1.0

Q1 Q2 Q3 Q4

Sta

nd

ard

ise

d B

MA

T t

est

sco

re

GCSE attainment quartile

Female Male

21

2.4 Regressionresults

Linear regression models were used to investigate statistical prediction of TSA andBMAT scores. Unstandardised regression coefficients and associated standard errorsarereportedinTable2.3.

Table 2.3: Regression of TSA and BMAT scores on student and school

characteristics

VariablesTSA BMAT

Coef. SE Sig. Coef. SE Sig.Intercept 16.70 11.79 -63.25 11.03 ***Gender(Female=1) -0.21 0.02 *** -0.25 0.02 ***Schooltype(Other=1) -0.13 0.23 0.43 0.30 Schooltype(State=1) 0.07 0.02 ** -0.04 0.02

BME(don'tknow/refuse=1) -0.18 0.04 *** -0.29 0.05 ***BME(Yes=1) -0.39 0.02 *** -0.27 0.02 ***ACORNFlag -0.10 0.04 * -0.11 0.04 **POLAR3Flag 0.04 0.03 0.01 0.03 Pre16SchoolFlag -0.05 0.04 -0.05 0.03 Post16SchoolFlag -0.09 0.03 ** -0.05 0.03 OverallWPFlag9 -0.06 0.05 -0.13 0.05 **Applicationtime -0.43 0.19 * -0.45 0.19 *GCSEA*s(zscored) 0.23 0.01 *** 0.19 0.01 ***A-levelA/A*s(zscored) 0.20 0.02 *** 0.21 0.01 ***Schoolhistoryofapplication 0.04 0.01 *** 0.01 0.01 *UcasCycle -0.01 0.01 0.03 0.01 ***

9TSA:467applicantswereWPflaggedoutofasampleof7,032.BMAT:479applicantswereWPflaggedoutofasampleof5,301.

-1.0

-0.5

0.0

0.5

1.0

Q1 Q2 Q3 Q4

Sta

nd

ard

ise

d B

MA

T t

est

sco

re

GCSE attainment quartile

White BME

22

A-LevelArt 0.02 0.05 -0.01 0.06 A-LevelBiology 0.01 0.03 0.06 0.06 A-LevelBusinessStudies 0.03 0.06 0.17 0.28 A-LevelChemistry -0.04 0.03 0.31 0.14 *A-LevelComputing 0.18 0.17 0.26 0.33 A-LevelEconomics 0.10 0.03 *** 0.24 0.07 **A-LevelEnglish -0.01 0.07 0.22 0.14 A-LevelEnglishLanguage -0.08 0.06 0.04 0.17 A-LevelEnglishLiterature -0.07 0.03 * -0.07 0.05 A-LevelFurtherMathematics 0.17 0.03 *** 0.27 0.04 ***A-LevelFurtherAdditional 0.00 0.00 * 0.00 0.00 A-LevelGeography -0.11 0.03 ** -0.11 0.06 A-LevelHistory -0.06 0.03 * -0.01 0.04 A-LevelLanguages -0.09 0.03 ** -0.05 0.03 A-LevelLaw -0.08 0.10 -0.63 0.37 A-LevelMathematics 0.22 0.03 *** 0.12 0.03 ***A-LevelOther 0.13 0.03 *** 0.09 0.06 *A-LevelPhilosophy 0.29 0.05 *** 0.25 0.15 *A-LevelPhysics 0.16 0.03 *** 0.11 0.03 ***A-LevelPsychology 0.11 0.04 ** 0.07 0.06 A-LevelReligiousEducation 0.10 0.04 ** 0.17 0.07 *A-LevelScienceOther 0.07 0.15 -0.31 0.20 A-LevelSocialScience 0.07 0.08 -0.14 0.21

p<0.05(*),p<0.01(**)andp<0.001(***)

Thecombinedsetofpredictorsexplains27%ofthevarianceoftheTSAscoreandtheBMAT score. Overall, results of the TSA model with the full set of control variablesindicated:

Significantlyhigherperformancefor,

• Stateschoolsapplicants(relativetoindependentschoolapplicants)• ApplicantsfromschoolswithahistoryofapplyingtoOxford• ApplicantswithhigherA-LevelandGCSEscores• Applicantswhoappliedearlier• Applicants who chose the following A-level courses: Economics, Further

Mathematics,Mathematics,Philosophy,Physics,Psychology,ReligiousEducationandOther.

Andsignificantlylowerperformancefor,

• BMEapplicants(relativetoWhiteapplicants)• Femaleapplicants• ACORNflaggedapplicants

23

• Post16schoolflaggedapplicants• A-levelcourses:EnglishLiterature,Geography,History,Languages

Overall,resultsoftheBMATmodelindicated:

Significantlyhigherperformancefor,

• ApplicantsfromschoolswithahistoryofapplyingtoOxford• ApplicantswithhigherA-levelandGCSEscores• Applicantswhoappliedearlier• ApplicantswhoappliedinalaterUCAScycle• A-level subjects: Chemistry, Economics, Mathematics, Further Mathematics,

Philosophy,Physics,andReligiousEducationandOther.

Significantlylowerperformancefor,

• Femaleapplicants• BMEapplicants(relativetoWhiteapplicants)• ACORNflaggedapplicants• OverallWPapplicants(relativetonon-flaggedapplicants)

In descriptive analysis we had shown that applicants from independent schoolsperformedbetterthancandidatesfromstateschoolsintheTSAandBMATtests.FortheTSA,wehad seen that this differencewas significant, however, that the gapbetweencandidates from independent and state schools was largely explained by GCSE priorattainment. Additionally, regression results in Table 2.3 indicate that applicants fromstateschoolsperformslightlybetterinTSAthancandidatesfromindependentschoolsonce the full set of socio-demographic and prior attainment characteristics areconsidered. The table also shows that there is no significant difference betweencandidates from independent or state schools in BMAT, once the same set of socio-demographicandpriorattainmentcharacteristicsareconsidered.

That is, applicants from state schools perform better (TSA) or equal to (BMAT)applicants from independent schools with equivalent GCSE/A-levels results, socio-economicbackground,andothercharacteristicsconsidered inmodels (seeTable2.3).Whilsta reducednumberof candidates in stateand independent schoolsmight shareprior attainment and socio-economic background characteristics included in models,whatregressionresultsdocommunicateclearlyisthatlowerTSA/BMATperformanceof candidates from state schools is not an attribute of state schools, but rather isexplained by prior attainment and socio-economic characteristics of candidatesattendingindependentandstateschools.

Consistently with results in Figures 2.2 and 2.5, regression models show that malecandidates performbetter than female candidates.Also, consistentlywith Figures 2.3and 2.6, regression models indicate that BME candidates perform worse thanWhite

24

applicants.TheTSA/BMATgapinperformancebetweenBMEandWhitecandidates isnotexplainedbytheirpriorattainmentorsocio-economicbackground.ThusthereareothercharacteristicsnotincludedinregressionmodelsexplainingtheperformancegapbetweenBMEandWhitecandidates.

Regression results also show that early applicants perform better in TSA/BMATadmissionsteststhanthosewhoapplylater.Theapplicationtimevariablereflectsdaysbetween1stSeptemberandtheapplicationdate,withsmallervaluesindicatingearlierapplications.Bi-variatecorrelationswith timeofapplicationarenegative forboth theTSAandBMATresults (Pearson’s r= -0.12), indicating thatearlierapplicantsperformbetter in the test, consistentlywith regressionresults.Furtheranalysis in thisprojectwill aim to explain the mechanisms for the positive effect of applying early onTSA/BMATresults.

2.5 Consistencychecks

As the survey in Phase 2 was administered to on-course Oxford students only,consistency checkswereperformed to evaluate inconsistencies in regression findingsfor1)allapplicantsand2)on-coursestudentpools.Regressionanalysiswerererunforon-courseTSA(N=1,478)andBMAT(N=835)studentsonly.ResultsareshowninTable2.4.

Table 2.4: Regression of TSA and BMAT scores on student and school

characteristics,includingon-coursestudentsonly

VariablesTSA BMAT

Coef. SE Sig. Coef. SE Sig.Intercept 2.64 24.54 59.71 24.10 *Gender(Female=1) -0.26 0.04 *** -0.24 0.05 ***Schooltype(Other=1) 0.36 0.45 0.62 0.74 Schooltype(State=1) 0.03 0.05 -0.05 0.05

BME(don'tknow/refuse=1) 0.10 0.14 -0.16 0.17 BME(Yes=1) -0.22 0.05 *** 0.05 0.05 ACORNFlag -0.21 0.09 * -0.23 0.11 *POLAR3Flag 0.07 0.07 0.03 0.08 Pre16SchoolFlag -0.04 0.09 -0.10 0.08 Post16SchoolFlag -0.15 0.06 * -0.19 0.07 **OverallWPFlag -0.01 0.11 -0.11 0.13 Applicationtime 0.22 0.36 -0.01 0.36 GCSEA*s(zscored) 0.03 0.03 -0.11 0.04 **A-levelA/A*s(zscored) 0.02 0.05 0.15 0.08 Schoolhistoryofapplication 0.02 0.02 0.03 0.02 UcasCycle 0.00 0.01 -0.03 0.01 *A-LevelArt -0.05 0.11 0.01 0.11

25

A-LevelBiology -0.04 0.08 -0.11 0.15 A-LevelBusinessStudies 0.22 0.20 0.13 0.63 A-LevelChemistry 0.04 0.08 0.22 0.37 A-LevelComputing 0.59 0.53 0.81 0.63 A-LevelEconomics 0.09 0.06 0.11 0.15 A-LevelEnglish 0.02 0.18 0.37 0.26 A-LevelEnglishLanguage -0.29 0.18 0.00 0.00 A-LevelEnglishLiterature 0.00 0.07 0.02 0.13 A-LevelFurtherMathematics 0.20 0.07 ** 0.20 0.10 *A-LevelFurtherAdditional 0.00 0.00 0.00 0.00 A-LevelGeography -0.16 0.08 -0.12 0.15 A-LevelHistory 0.02 0.07 0.05 0.11 A-LevelLanguages -0.04 0.07 -0.05 0.09 A-LevelLaw -0.15 0.28 0.00 0.00 A-LevelMathematics 0.31 0.08 *** 0.16 0.10 A-LevelOther 0.06 0.06 0.17 0.15 A-LevelPhilosophy 0.05 0.10 -0.29 0.32 A-LevelPhysics 0.12 0.08 0.12 0.09 A-LevelPsychology 0.00 0.08 0.23 0.16 A-LevelReligiousEducation 0.16 0.07 * 0.10 0.17 A-LevelScienceOther 0.18 0.43 0.00 0.44 A-LevelSocialScience -0.33 0.19 0.00 0.00 p<0.05(*),p<0.01(**)andp<0.001(***)

Asexpectedanumberof,butnotall,thesamecharacteristicsweresignificantstatisticalpredictorsofTSAperformanceforon-coursestudentsonly:

Significantlylowerperformancefor,

• BME(relativetoWhite)applicants• Femaleapplicants• Acornflaggedapplicants• Post16schoolflaggedapplicants

Note,however,thatthefollowingcharacteristicsarenolongersignificantbasedonthelimitedon-coursepool:

• Schooltype• SchoolhistoryofOxfordapplications• Candidateswhoappliedearlier• ApplicantswithhigherA-LevelandGCSEscores

Also, a smallernumberofA-Level courses are significant statisticalpredictorsofTSAperformance.

A-levelcoursesassociatedwithhigherperformance:

26

• FurtherMathematics,Mathematics,ReligiousEducation

For BMAT, statistical predictors of performance for on-course students only are thefollowing:

Significantlylowerperformancefor,

• Femaleapplicants• Acornflaggedapplicants• Post16schoolflaggedapplicants• ApplicantswithhigherGCSEscores• ApplicantswhoappliedinanearlierUCAScycle

Note,however,thatthefollowingcharacteristicsarenolongersignificantbasedonthelimitedon-coursepool:

• BMEapplicants(relativetoWhiteapplicants)• OverallWPapplicants(relativetonon-flaggedapplicants)• SchoolhistoryofOxfordapplications• Candidateswhoappliedearlier• ApplicantswithhigherA-Levelscores

Only Further Mathematics A-Levels is a significant statistical predictor of BMATperformance.

StudentsacceptedtooneoftheUniversity’scoursesareamoreselectiveandthereforeheterogeneous group than the group of applicants in focus in Table 2.3. We canthereforeassumethatthisispartlythereasonwhysomecharacteristicsarenolongerpredictiveoftestperformancewhenlookingaton-coursestudentsonly,becausetheirvariance in the on-course student group is smaller. Hence, we see that, while schooltype, schoolhistoryofOxfordapplications,earlierapplicationandhigherA-LevelandGCSE scores were statistically predictive of TSA test performance for the applicantgroup,thesecharacteristicsarenolongerassociatedwithtestperformancewithintheon-course student group.Within this group there is however still sufficient variancewithregardtoBMEstatus,genderandsocio-economicdeprivation(ACORNFlag,post16schoolflag),forthosecharacteristicstopredictTSAtestperformance.

Similarly, for BMAT test scores, BME status, overall WP flag, the school’s history ofOxford applications, earlier application and higher A-Level scores applicants do notstatisticallypredicttestperformancewithintheon-coursestudentgroup,whiletheydidwithintheapplicantgroup.Wecanthusassumethattheon-coursestudentsinMedicineandBiomedical science courses aremore similarwith regard to these characteristicsand their related test scores. They aremore heterogeneous, however,with regard togender, indicators of socio-economic deprivation and GCSE scores, as thesecharacteristicsremainsignificantlyrelatedtoBMATtestperformance.

27

3. AnalysisofBrasenoseCollegesurveydata

First year students at Brasenose College were asked about their preparation foradmissionstestsinthecontextofasurvey.Surveyresponsesinformedthepreparationof interviews and, ultimately, the development of a survey administered to OxfordstudentsinPhase2ofthisproject.

Brasenose College provided a file with data for 141 respondents (47 in 2013, 44 in2014,50 in2015),ofwhich116 indicatedthat theyhadsatanadmissionstest(38 in2013,37in2014,41in2015).

Theywereaskedtorespondtothefollowingopenendeditem:

‘Ifyouhadtotakeanaptitudetestduringtheadmissionsprocess–couldyousayhowyou prepared, how much input your school had, and any other thoughts orrecollections?’

Theresponsesreceivedforthisitemrangedfromoneorafewwordstoagoodnumberof longer sentences. Most responses included information on the type and extent ofpreparationandtheroleoftheschoolinpreparation.Onlyaminorityofresponsesalsoprovidedotherthoughtsandrecollections.Table3.1presentsanoverviewofresults.

Studentswho responded to the surveywerenot identified as takersof a specific test(TSA,BMAT,other),henceresponsesmayrelatetoanyadmissionstesttaken.However,survey responses could be grouped in the following thematic areas: amount ofpreparation,supportfromschools,useofexampreparationresources,andperceptionsof the test.Selectedquotes foreachof theseareasarepresented inAppendixC.Mainresultsarepresentedbelow.

Table3.1:BrasenoseCollege:preparationandsupportbysubjectarea

2013 2014 2015 Total % Prepared

‘yes’ %Littleornosupport

fromschool%

Usableresponses 38 37 41* 116* 98° 84.4 51 44.0Arts&Humanities 12 17 12 41 35.3 29 70.7 20 48.8JointDegree 7 4 1 12 10.3 12 100 2 16.7Sciences 9 10 15 34 29.3 32 94.1 14 41.2SocialSciences 10 6 7 23 19.8 20 87.0 13 56.5*includes6respondentswhosesubjectareaisnotclear;°includes5respondentswhosesubjectareaisnotclear

3.1 Amountofpreparation

Amongst the116respondents thathadsatadmissions tests therewerevery fewwhoindicatedtheyhadnotpreparedforthetest.Thevastmajorityofstudentshadpreparedfor the admissions test (84.4%), but the self-reported degree of preparation variedvastly,with14respondentsindicatingthattheyonlyhadverylimitedpreparation.The

28

reasonsgivenforlimitedpreparationvariedfromalackofguidanceonhowtopreparetootherprioritiesandunforeseencircumstances.

MostofthestudentswhoreportedverylimitedtestpreparationwerestudyingasubjectintheArtsandHumanities(9),whereasonlyoneSciencesstudentfellintothisgroup.Overall, just over 70% of Arts andHumanities students explicitly indicated that theyhad undertaken substantial levels of test preparation, compared with nearly 85% ofstudentsoverall(seeTable3.1).

3.2 Supportfromschools

Overall,51of116studentsindicatedsubstantialsupportfromtheirschoolinresponseto the item stated above, 25 indicated little or very little support and a further 26explicitlymade itclear that theyhadnohelp fromtheirschool,apart fromtheschoolproviding some administrative support such as helping in registering for the test. Inmanycases,studentswhoreportedlimitedtestpreparationalsoindicatedthattheyhadnoorverylittlesupportfromtheirschool.Supportfromschoolswasmostwide-spreadforstudentsintheSocialSciences(seeTable3.1).

3.3 Useofexampreparationresources

Irrespective of whether students received support from their school, the mostfrequentlymentionedtypeofpreparationwasworkingthroughpastorpracticepapers.A clear majority of students made, often extensive, use of these papers (i.e., 70students). In a good number of responses it becomes clear that students tried tosubstitutethelackofsupportfromschoolbyworkingextensivelywithpasttestpapers.

Othermaterial/sourcesmentionedincludedprintedhandbooksonadmissionstests,thewebsitesoftestproviders,onlinepracticetests,subject-relevanttextbooks,chatrooms,andsourcesmadeavailableonOxfordCollegeorDepartmentwebpages.

3.4 Perceptionsofthetests

In terms of themore general perceptions of the admissions test by students, a goodnumberofrespondentspointed toasignificant levelofuncertaintyregardingwhat toexpect in the test. This perception seems to be unrelated to the level of preparationstudents did, and in someways developing a strategy for dealingwith the feeling ofuncertaintybecamepartofthepreparationprocess.

Studentsalsodescribedhowthisuncertaintywas‘resolved’,inasfarastheactualteststurned out to be much harder than they had expected. However, there were someinstances in which the experience of taking the test was described as enjoyable orrewarding,notleastbecausetheyweredifferenttowhatstudentshadtoprepareforA-Levels.

29

4. InterviewswithOxfordstudents

To develop a survey for Phase 2, Brinkmann and Kvale’s (2015) seven stages of aninterviewprocesswereusedasaguide.BrasenoseCollegesurveyresponses(Section3)informedthedevelopmentofinterviewswithstudentsatOxfordandinadditionseveralthemesemergedfromtheliteraturereviewandthroughdiscussionswithintheresearchteam which guided the development of interview questions. Further, a pilot wasconductedwiththreecurrentOxfordUniversitystudentsatBrasenoseCollegewhohadpreviouslytakenanOxfordUniversityadmissionstest.Thepilottogetherwithresultsofthe literature reviewand theBrasenoseCollege surveypointed to six thematic areas.The themes are presented in Table 4.1. The themes together with more detailedprompts for follow-up questions were useful in maintaining the researchers’ focusduringtheinterviewprocessastheybridgetheinterviewquestionswiththeresearchaims. The wording of the questions was open aiming at evaluative and descriptiveresponses.ThefullinterviewguideispresentedinAppendixD.

Table4.1:ThematicInterviewGuide

ThemeA TheStudentThemeB PreparationfortheTestThemeC UsefulnessofthePreparationThemeD LogisticsSurroundingtheTestThemeE PerceptionsoftheRelevanceandImportanceoftheTestThemeF AdviceforFutureStudents

These multiple perspectives were considered important for the development of anuancedviewofthefindings(Flyvbjerg2006).ParticipantswereviewedaswhatFoley(2012)calls ‘a respondent’as theywerechosenbecause theyhavespecialknowledgearound preparation for admissions tests and were thus positioned as experts. Thisreassured the participants that they were not being tested and ensured that theinterviewwasmore factual and descriptive rather than emotional (Foley 2012). Thequestionsrequiredparticipantstobereflectivewhichwaslikelytoleadtoavarietyofresponsesdependingupontheparticipants’opennessandabilitytoarticulate.Priortomeetingthem,theresearcherswerenotawareofthepersonalityof theparticipantorhowmotivatedtheyweretoparticipate.

Theaimoftheinterviewswastoprovidethickdescriptionandrichdetailpertainingtopreparation undertaken for Oxford University admissions tests. Ten students wereselected for the main interview. The selection process was as follows. The samplingframe was the Definitive Dataset that contains information on all applicants whoappliedtoOxfordin2016(tocommencetheirstudiesin2017).Itwaslimitedtothosewhometandacceptedtheirofferandtocoursesthatrequiretakinganadmissionstest.It was important to achieve a good representation of all school types. A stratifiedrandomsamplingwasusedtoselect10studentsfromindependentschools,10students

30

from state schools and 8 students from ‘other’ schools. In total 48 students wereselected.Thenthelistwascross-checkedwiththelistofcurrentstudentstorevealthattwoofthechosenstudentsdidnotcommencetheirstudies leaving46students inthesample.

The46studentswereinvitedbyemailtoparticipateintheinterviewintwoblocks,firsttwentythentwentysix,aimingatobtaining10positiveresponses.Asthetakeupfromthestudentswastoolow,itwasnecessarytoapproachanother44studentsinordertosummon10interviewees.Thesewereselectedinexactlythesamewayastheoriginalsample of 46. The 10 students in the final interview sample received each a £10incentiveperinterview.TableA4(AppendixA)givesanoverviewofthecharacteristicsoftheinvitedsampleandthe10studentswhodidtheinterview.

Interviews were conducted in a private room so that the conversation could berecorded confidentially andwithout interruption.Noteswere takenof any reflectionsby the researchers to aid the formulation of follow-up questions. The interviewschedules were semi-structured and were used as a guide with flexibility to exploreotheraspects.Theresearchersactivelylistenedtowhatwassaidandhowitwassaidsothatitcouldbedecidedwhichaspectsofananswertopursuewithsecondaryquestions,beinginformedbyknowledgeofthetopicandguidedbyempathy(Brinkmann&Kvale2015). AsRubinandRubin (2005)advise ‘youshowempathybyaskingquestions toobtainthedetailsthatallowyoutoimaginewhatyourintervieweeshaveexperienced’(2005:81).The researcherswereaware thatparticipantswouldvary in thequalityofdata that they can provide and recognised that somemay be reticent to share theirhonest views, so they attempted to ‘motivate and facilitate interviewee accounts’throughtheirtoneandbodylanguage(Brinkmann&Kvale2015:193).Datacollectionand some initial analysis occurred simultaneously so that judgements could bemadeabout which secondary questions to ask. Interviews followed criteria for a qualityinterviewprovidedbyBrinkmannandKvale’s(2015)andlistedinTable4.2.

Table4.2:BrinkmannandKvale’s(2015:181)criteriaforaqualityinterview

Rich,specific,andrelevantanswersfromtheinterviewee

Theshortestinterviewer'squestionsandlongestsubjects'answerspossible

The degree to which the interviewer follows up and clarifies the meanings of therelevantaspectsoftheanswers

The degree to which the interviewer verifies their interpretations of the subject'sanswersoverthecourseoftheinterview

QualitativeanalysisoftheinterviewdatacontributedtodevisingasurveyforPhase2oftheproject.ThefinalsurveyquestionnaireispresentedinAppendixE.

31

5. Discussionofresultsfromtheexplorationofrelationshipbetweenstudent characteristics, test performance and test preparation

(Phase1)

The aim of this project is to study the impact of preparation for TSA and BMATadmissions testsusing the caseofOxford.Anumberof researchquestionshavebeenpostulated.ThissectionprovidespreliminaryresponsestotheresearchquestionswiththedatasetsutilisedinPhase1.

Researchquestions

Towhatextentdostudentsprepareforuniversityadmissionstestsandwhatisthenature of this preparation amongst students with different backgroundcharacteristics?Results of the Brasenose College survey indicated that the greatmajority of studentsprepared for admissions tests (84%). The degree of preparation, however, variedremarkablyacrosscourseswithstudentsintheArtsandHumanitiesreportinglimitedtest preparation compared to students in the Sciences. About half of the studentsindicated substantial support from schools, one quarter indicated very little supportfrom schools, and another quarter indicated explicitly that they had no help fromschools. Similarly, interviewswith ten students indicated great variability in termsofschool support, with students who had received regular and intense support fromschools to studentswho had not received support at all. Very few students reportedsupportfromfamilyandfriends.

Beyond school support, students indicated making extensive use of past papers inpreparingforadmissionstests.

The Brasenose College survey is restricted to one College. The survey in Phase 2examines further the preparation for admissions tests for students with differentbackgroundcharacteristics.

Whataretheconnectionsbetweenstudentcharacteristicsandtestpreparationandperformance?ResultsofOxfordadmissiondatasetsin2010–2015acrossdifferentcoursesrevealedconnections between student characteristics and performance in the TSA and BMATtest.

Consistently, applicants with higher A-level and GCSE scores and applicants fromschoolswithahistoryofapplyingtoOxfordperformedbetterintheTSAandBMATtestthantherest,evenaftercontrollingforanumberofstudentandschoolcharacteristics.

BMEand female applicants aswell as applicants fromdeprived socio-economic areasperformedworseinbothtests,independentlyofothercharacteristics.

32

Candidates from independent schools performed better in the TSA and BMAT thancandidatesfromstateschools.However,candidatesfromStateandindependentschoolswith equivalent GCSE results performed fairly similar in admissions tests. That is,underperformance of state schools in admissions tests is mainly explained by priorattainmentofcandidates.

ThereisaninteractionbetweentheschooltypeandGCSEperformance,withcandidatesfromindependentschoolsperformingslightlybetterthancandidatesfromstateschoolsatlowerGCSElevelsfortheTSAtest.

Which prior qualifications and subjects are connectedwith test preparation andbettertestperformance?OxfordadmissiondatasetsmadeitpossibletolookattherelationshipbetweenA-Levelchoices and performance in the TSA and BMAT tests. For both tests, specific A-Levelchoices such as Economics, Mathematics, Physics, and Religious Education relatedpositivelytoperformanceinTSAandBMATtests,whereastheGeographyA-levelwasnegatively related to performance in both tests even after student and schoolcharacteristicsaretakenintoaccount.

Isthetimeofapplicationconnectedwithpreparationfor,andbetterperformancein,tests?Previousanalysis(Section2)hadshownthatearlyappliersweremorelikelytoobtainan offer. Thiswas true independently of the course,UCAS cycle, the applicant’s priorattainment,schooltype,andschoolhistoryofapplyingtoOxford.Theassociationwithtime of application was around one fifth as predictive as GCSEs for explaining offerdecisions. Given this evidence, it was postulated that the time of application wasassociatedwithadmissionstestsresults.

EvidencefromOxfordadmissionsdataindicatethatearlyappliersperformbetterintheTSAandBMATtests.Theresultsholdevenafteranumberofcharacteristicsrelatedtothe applicant and his/her schools are taken into account. Differentmechanismsmayexplainthisresult(e.g.,preparationtime,culturalcapital,motivation).Forexample,itispossible that early appliers use more time for preparing for exams, or that earlyappliers and their families are better informed about the system, or even that earlyappliers aremoremotivated than late appliers. Analysis in Phase 2with survey dataexploredsomeofthepossiblemechanismsunderlyingtheassociationbetweentimeofapplicationandadmissionstestscores.

What amount, timing, and nature of preparation are connected with better testperformance?ThisquestionisaddressedinPhase2andresultspresentedaccordingly.

Istheinfluenceoftestpreparationontestperformancemediatedormoderatedbystudentcharacteristicsandbackground?ThisquestionisaddressedinPhase2andresultspresentedaccordingly.

33

What learning strategies and sources of support are connectedwith positive testperformance?ThisquestionisaddressedinPhase2andresultspresentedaccordingly.

Limitations

Likely themost important limitationof this study is thatourdata is restricted toon-coursestudentsatOxford.AsitissuggestedbytheBrasenosedata,themajorityofon-coursestudentspreparedforadmissionstests(84%).However,wehavenoinformationonpreparation for those candidateswhowerenotofferedaplace.One couldassumethat the amount of preparation and preparation strategies varied for candidatesadmittedandnotadmittedtoOxford.Infact,variationbetweenthesetwogroupsmightbeveryvaluableforidentifyingeffectivepreparationstrategies.Totheextentthatouranalysisislimitedtoon-coursestudents,ourfindingsdonotrepresentthewholerangeof performance in admissions tests and some effects might be underestimated. Forexample, the association with socio-economic deprivation will be underestimated ifnon-admitted candidates disproportionallymore often come from socio-economicallydeprivedareas.

6. Oxfordsurveydataandadmissiondata2017/2018cohort

6.1 Development of survey instrument on Oxford student test

preparationandadministration

The survey instrumenthasbeendevelopedon thebasis of the findings fromPhase1and in particular those of the qualitative interviews. It was administered to fresherswhocommencedtheirstudiesinOctober2017.Thequestionnaire(AppendixE)asksanumberofquestionsabouthowstudentshavepreparedfortheadmissionstest.Theseinclude: how far in advance students started preparing for the test, the sources theyusedfortestpreparationandthesupporttheyreceivedfromvariousagents.Inadditionthesurveyincludedascaleonstudentlearningstrategies,widelyusedinthecontextofthe PISA study (Marsh et al., 2006) and adapted for high-stakes tests in previousresearch by the Oxford University Centre for Educational Assessment (Baird et al.,2015).

TheonlinesurveywasadministeredbyOxfordUniversity’sStudentDataManagementandAnalysisteamviaOnlineSurveys(formerlyBOS).Thesurveywasopenbetween9thofMarchand22ndofApril2018, inordertoallowstudentssufficient timetorespondbetweentheendofHilaryandthebeginningofTrinityTerm.

Information collected through the survey was linked to respondents’ studentadmissions recordswhich allowed for similar analysis to that undertaken for smallersamples of students in Phase 1, but employing much more detailed data on testpreparationandmoresophisticatedmethods.

34

6.2 Data

Table6.1givesanoverviewof thecohortandrespondent information. InMichaelmasTerm2017atotalof3,160studentscommencedtheirundergraduatestudiesatOxfordUniversity. Of these, 2,737 (86.6%) students needed to take an admissions test. 703studentstooktheTSAtestand181tooktheBMATtest,thatis,22.2%and5.7%ofthetotalcohort,respectively.

Table6.1:Overviewofcohortandrespondents

StudentscommencinginMT2017

Studentstaking

admissionstest

StudentstakingTSA

StudentstakingBMAT

Surveyrespondents

BMATtesttakers

TSAtesttakers

N 3160 2737 703 181 783 70 202%ofcohort 100.0 86.6 22.2 5.7 24.8 2.2 6.4%ofrespondents 8.9 25.8

6.2.1 Responseratesandanalyticalsample783studentsrespondedtothesurveywhichconstitutes28.6%responserate,ofwhich202(6.4%ofthetotalcohort;25.8%ofrespondents)areTSAtesttakersand70(2.2%ofthetotalcohort;8.9%ofrespondents)areBMATtesttakers(Table6.1).TheresponserateamongTSAtesttakerswas28.7%andamongBMATtesttakerswas38.6%.

Ofthe202TSAtesttakerseightstudentsdidnotindicateinthesurveythattheytooktheTSA test, although theywere identifiedby theUniversity’s admission informationdataasTSAtesttakers.ThosestudentsareexcludedfromtheTSAsampleastheymighthave answeredquestions in the survey in relation to another test. Table 6.2 gives anoverviewoftheanalyticalsample.

Table6.2:Overviewofanalyticalsample

Survey

respondentsBMAT

respondentsTSA

respondentsOthertestrespondents

N 783 70 194 519

%ofrespondents 100 8.9 24.8 66.3

6.2.2 VariablesArangeof informationhasbeencollected through thequestionnaire,notallofwhichcanbereasonablyaddressedinthecontextofthisreport.Variablesusedfordescriptiveanalysisareself-explanatoryregardingtheircategoriesandarenotlistedbelow.Forthemultivariateanalyses(sections7.2to7.5)thefollowingvariableshavebeenused:

35

• TSAtestscore: istheaverageoftwoseparatemarksfromtheCriticalThinkingandProblemSolvingtests,eachweightedequally.ThevariableiscontinuousandhasbeenstandardisedtohaveameanofM=0andastandarddeviationofSD=1.

• BMATtest score: isa combinedscore fromthreeBMAT test sections.Foreachsection students receive amark that is converted into a score. The combinedtotal score weights sections 1 and 2 with 40% and section 3 with 20%. Thevariable is continuouswith a range of 48 to 91 and has been standardised tohaveameanofM=0andastandarddeviationofSD=1.

• Gender:isabinaryvariableindicatingwhetherthestudentisfemale(0)ormale(1);

• BME:isabinaryvariableindicatingwhetherthestudenthasaBMEstatus(1)ornot(0);

• Fee status: is represented by three dummy variables indicating whether thestudenthasaHome,EUornon-EUfeestatus;

• SchoolGroupUK:isabinaryvariableindicatingwhetherthestudentattendedanindependent(0)orstate(1)schoolintheUK;

• SchoolGroupInternational: isabinaryvariable indicatingwhether thestudentattendedaschoolintheUK(0)oraninternationalschooloutsidetheUK(1);

• A-LevelBand:indicateswhetherthestudentachieved‘Less than A*A*B, A*AB or

AAB’ (0), AAB (1), AAA(2),A*AB(3),A*AA(4),A*A*B(5),A*A*A*(6),3A*s(7),4A*s (8) or 5A*s or better (9). For themultivariate analysis the variable wasstandardisedtohaveameanofM=0andastandarddeviationofSD=1;

• Application time: is a continuous variable indicating the number of days thathavelapsedbetweenthestudents’applicationandtheapplicationdeadline(15-OCT-2016).Largervaluesreflectearlierappliers;

• WPOVERALLFLAG(wideningparticipation): isadummyvariable that indicateswhethertheapplicanthasobtainedoneormoreoftheschoolperformanceflags(preorpost16)andoneormoresocio-economic flags(ACORNorPOLAR3)orhaspreviouslybeenincare(1)ornot(0);

• Timepreparation:isacategoricalvariableindicatinghowmuchtimeinadvancestudents started preparing for the test: 'not at all' (0), 'just before' (1), '1-2monthsbefore'(2),'3-6monthsbefore'(3),'6-12monthsbefore'(4),'Morethanayearbefore'(5).Notethatthisvariablegivesnoindicationofeffortordegreeofpreparationinthistimeduration;

• Amount practiced: is a variable combining information from two questions:whetherstudentspracticedpastpapersandhowmuchtheypracticedpasttestpapers.Thevariableindicatesthatstudents‘didnotpracticeatall’(0),‘practicedverylittleorsomewhat’(1),‘practicedquiteabitoragreatdeal’(2);

• Onlineresources: isabinaryvariable indicatingwhetherstudentsoverallusedonlineresources ‘very littleorsomewhat’(0) ‘quiteabitoragreatdeal’ (1)to

36

prepare for the test. The variable originally had four categories, but to ensuresufficient numbers on each category they were collapsed into two formultivariateanalysis;

• Family support: is a binary variable indicating whether students receivedsupportfromtheirfamily(1)ornot(0);

• Schoolencouragement:isabinaryvariableindicatingwhetherastudent’sschoolencouragedthemtoapplytoOxford(1)ornot(0);

• Controlstrategies:isascaleindicatingtheextenttowhichstudentsusecontrolstrategies to prepare for the test (M=0; SD=1; α=0.81110). It is based on fivevariables. Example item: ‘I tried to figure out which ideas I had not reallyunderstood’. Students answered all items by indicating how often they used aparticularstrategyona5-pointLikertscale(always–never).

• Memorisation: is a scale indicating the extent to which students usememorisationstrategiestoprepareforthetest(M=0;SD=1;α=0.916).Itisbasedon four variables. Example item: ‘I tried tomemorise asmuch of the revisionmaterialaspossible’.Studentsansweredall itemsby indicatinghowoftentheyusedaparticularstrategyona5-pointLikertscale(always–never).

• Elaboration:isascaleindicatingtheextenttowhichstudentsusememorisationstrategies to prepare for the test (M=0; SD=1; α=0.792). It is based on fourvariables. Example item: ‘I tried to understand the revisionmaterial better byrelating it to what I already knew’. Students answered all items by indicatinghow often they used a particular strategy on a 5-point Likert scale (always –never).

6.3 Descriptionoftheanalyticsampleandnon-respondents

Thesamplecomprised58%females.Themajorityofrespondents(84%)wereWhiteasopposed toBME.Threequarters of respondentswereUK students (75%), 11%werefrom the EU and 13% were non-EU. There were 14% of respondents who had anOverall WP Flag11. There were 37% of respondents who came from independentschools (as opposed to state schools) and 23% of respondents came from overseasschools.TableA5(Appendix)givesanoverviewofthesamplesstatistics.

Inordertocheckwhetheroursampleofrespondentsissubstantiallydifferentfromthenon-respondentsacomparisonofbackgroundcharacteristicswasconducted.Figure6.1summarisesthedifferencesbetweenthetwogroups.

10 The Cronbach’s Alphameasures the internal consistency of the scale, α values higher than 0.8 areconsideredrobustscales.

11StudentsareflaggedwiththeOverallWPFlagiftheyhaveatleastoneofschoolperformanceflag(pre-16orpost-16flag)ANDatleastonesocio-economicflag(ACORNorPOLAR)ORiftheywerepreviouslyincare.

37

Figure 6.1: Background characteristics by survey respondents and non-

respondents

Chi-Squaretestsrevealthat,apartfromthepercentageofstudentswithBMEstatus,alldifferences between the two groups are significant. Thus, the respondents areoversampled in the sense that significantly more female students, those from EUcountries,thosefromstateschools(UKonly)andinternationalschoolsandthosewithanOverallWPFlaghaveansweredthequestionnaire.Inaddition,respondentsachievesignificantlymoreoftenanA*ABA-Levelscorethannon-respondents(notpictured).

ThereisnosignificantdifferencebetweentheTSAtestscoresofNon-respondents(M=68.6, SD = 6.13) andRespondents (M= 67.9, SD = 6.17; t (688) = 1.236, p = .0.746).Further,nosignificantdifferencebetweentheBMATtestscoresofNon-respondents(M=64.09,SD=6.72)andRespondents(M=66.33,SD=7.93;t(176)=-2.022,p=.263)wasidentified.

7. Analysis of Oxford survey data and admission data 2017/2018cohort

7.1 Descriptiveanalysisoftheextentandnatureoftestpreparation

7.1.1 ExtentandnatureoftestpreparationbytesttakersThis section looks only at the identified analytical sample (n=783) and distinguishesbetweenTSA,BMATandothertesttakerswhereappropriate.

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Female

BME

UK

EU

Non-EU

Overall WP flag

Independent School (vs. State School [UK only])

International School (vs. UK school)

% Non-Respondents % Respondents

38

Themajorityofstudentsstartedpreparingfortheadmissionstest1-2monthsormoreinadvanceofthetestdate(Figure7.1).Thisis independentofwhethertheypreparedfor the TSA, BMAT or another admissions test. Only a small number of studentsindicatedthattheydidnotprepareatallforthetest.TSAtesttakersstartpreparingforthe test later (1-2 months before) than other test takers (including BMAT) who aremorelikelytostarttheirpreparations3-6monthsbeforethetestdateorevenearlier.Thesedifferencesaresignificant.

Figure7.2indicatesthatover90%oftesttakerspracticedpastpaperswhenpreparingfor theadmissions test.Chi-Square tests reveal thatother test takersare significantlyless likely and TSA takers are significantly more likely to prepare for the test bypracticingpastpapers.Moreover,BMATtakerspreparesignificantlymorethanTSAtesttakers(Figure7.3).

Therewerefurtherquestionsabouthowstudentspracticedpastpapers.77%indicatedthattheypracticedundertimedconditions,35.2%indicatedreceivingfeedbackonthepaperstheypracticedand88.8%statedtheyusedamarkscheme.Figure7.4providesfurther information about howuseful students perceived thesemodes to be for theirpreparation.

Figure7.1:Timestudentsstartpreparingfortheadmissionstest

16%

20%

11%

44%

55%

53%

26%

17%

26%

9%

5%

9%

Other

TSA

BMAT

When did you start preparing for the test?

I dont remember I did not prepare at all Just before the test

1-2 months before the test 3-6 months before the test 6-12 months before the test

More than a year before the test

39

Figure7.2:Studentsconfirming that theypracticedpastpaperswhenpreparing

fortheadmissionstest

Figure7.3:Amountstudentspractisedpastpapers

Both practicing past papers under timed conditions and receiving feedback on themwere perceived as extremely helpful in the preparation process by the majority ofstudentswhousedthosemodes.Themarkschemewasalsomainlyperceivedasquitehelpful.Onlyasmallpercentageofstudents(<15%)indicatedthattheydidnotperceivethedifferentwaysofpractisingpastpapersashelpful.

Inordertopreparefortheadmissionstest,studentsusedonlineresources.Figure7.5displays the differences between overall use of online resources by test takers. Chi-Square significance tests reveal that BMAT test takers indicated they used onlineresourcessignificantlymoreoftenthanothertesttakersincludingTSAtesttakers.This

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Other

TSA

BMAT

Did you practise past papers when preparing for the test?

7%

4%

3%

23%

24%

16%

36%

44%

32%

34%

28%

49%

Other

TSA

BMAT

Very little Somewhat Quite a bit A great deal

40

finding ishoweverhardlysurprisingasmoreonlineresourcesareavailable forBMATthanforTSA.

Figure7.4:Helpfulnessofdifferentmodestopracticepastpapers

Figure7.5:Overalluseofonlineresourcestopreparefortheadmissionstest

Thesurveyinstrumentalsoinquiredinmoredetailaboutthetypesofonlineresourcesstudents used to prepare for the admissions tests. Overall most students used theOxfordUniversity’swebsite,theofficialtestproviderwebsitesandtheStudentRoominorder to prepare for the admissions tests. They less often accessed other onlineresourcessuchasYouTubeorschool/collegewebsites.Figure7.6indicatesthatthere

13%

11%

4%

51%

39%

47%

36%

50%

49%

Mark scheme

Feedback received

Timed conditions

Not at all helpful Not very helpful Quite helpful Extremely helpful

20%

16%

9%

25%

26%

11%

28%

28%

36%

27%

30%

44%

Other

TSA

BMAT

Overall, how much did you use online resources?

Very little Somewhat Quite a bit A great deal

41

are substantial differences between the different test takers and their use of specificonlineresources.Chi-Squaretestsrevealthefollowingsignificantdifferences:OthertesttakersaresignificantlymorelikelytousetheOxfordUniversity’swebsiteandlesslikelytouseanofficial testproviderwebsite. Incontrast,BMATtest takersaresignificantlylesslikelytousetheOxfordUniversity’swebsiteandmorelikelytousetheofficialtestproviderwebsiteorotherwebsitesdedicatedtothetest.TSAtesttakersarealsomorelikely to use the official test provider website but less likely to use other websitesdedicatedtothetest.

The majority of students who used the Oxford University website, the official testproviderwebsiteandotherwebsitesdedicated to the testperceived themasquiteorextremelyhelpful(Figure7.7). But, therearealsosubstantialproportionsofstudentswho perceived the Oxford University website (29%), The Student Room (44%) andotherwebsitesdedicatedtothetest(19%)asnotveryhelpful.

Figure7.6:Useofonlineresourcestopreparefortheadmissionstest

Figure7.7:Perceivedhelpfulnessofdifferentonlineresourcesforadmissionstest

preparation

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Another school/college website(s)

Other student forum websites (besides the

Student Room)

Your school/college website

Other

YouTube

Other websites dedicated to the test

The Student Room

Oxford University website

Official test provider website

TSA BMAT Other

42

Studentswerefurtheraskedtoprovideinformationaboutsupporttheyreceivedfromtheir families in the process of preparing for the test. Only a few students receivedsupport from their families and the differences between the test takers are notsignificant (Figure 7.8). Themajority of studentswho did receive support from theirfamilies indicated that this was financial support (13%), working out how to bestapproachthetest(10%)goingthroughpasttestpapers(9%),revisingsubject-specificmaterial(4%)andotherformsofsupport(6%).

Figure7.8:Familysupportreceived

Withregardtosupportreceivedfromtheirschools,themajorityofstudentsindicatedthat theywere already considering applying to Oxford and their schools encouraged

9%

25%

7%

28%

24%

17%

26%

62%

61%

49%

53%

62%

35%

38%

80%

83%

29%

14%

44%

19%

14%

48%

35%

20%

17%

Oxford University website

Official test provider website

The Student Room

Other websites dedicated to the test

YouTube

Other

Your school/college website

Another school/college website(s)

Other student forum websites (besides The Student

Room)

Extremely helpful Quite helpful Not at all/not very helpful

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Other

BMAT

TSA

43

them(Figure7.9).Onlyafewstudents indicatedthattheirschools infactdiscouragedthem to apply. Similar proportions of students indicate that their schools supportedthem in preparing for the admissions test by providing opportunities to ask subject-specificquestionsorworkwithasubjectteacher,providingworkshopsandmentoring.Schoolslessoftenprovidedtheopportunitytouseclassroomperiodstoprepareforthetest(Figure7.10).Notethatresultsforschoolsupportarenotdistinguishedbytesttypebecausethereisnoreasonableassumptionthatschoolsupportshoulddifferbythetestthatstudentstake.Differencesbystudentbackgroundcharacteristicsarepresentedinsection7.1.2.

Figure7.9:Schoolencouragementreceived

Figure7.10:Schoolsupportreceived

Students fell back on a range of other strategies and resources to prepare for theadmissionstest(Figure7.11)suchasstudyingthesubjectsyllabusortextbooks,talking

0% 20% 40% 60% 80% 100%

No, they actually discouraged me from

applying

No, they did not

Yes, I was already considering it and

they encouraged me

Yes, they suggested I should consider

applying

Did your school/college encourage you to apply to Oxford?

0% 20% 40% 60% 80% 100%

Opportunity to use some class periods

to prepare for the test

General support/mentoring from a

teacher at your school/college

Provide workshops or classes to help

prepare for the admissions test

Opportunity to ask subject-specific

questions or work with a subject

teacher at your school/college

Forms of school support

44

tocurrentorformerOxfordstudentswhotookthetestorprivatetutorials.Chi-Squaretestsrevealanumberofsignificantdifferencesbetweenthetesttakers.Comparedwithstudents takingotherentrancexaminations in thedataset,BMATandTSA test takerssignificantlymoreoftenusedcommerciallypublishedtestpreparationbookstopreparefor the test (which may not be available for all of the other tests); BMAT takerssignificantlymoreoftenattendedcommercialpreparationcoursesandusedtheservicesofanexternalagencyinordertoprepareforthetest;BMATandothertesttakersusedsubjectsyllabus/textbookssignificantlymoreoften;other test takers talked to formerpupilsof their schoolor current/formerOxford students significantlymoreoftenandattendedOxfordOpenDaysmoreoften.

Figure7.11:Otherstrategiesandresourcesusedforpreparation

7.1.2 Extent and nature of test preparation by student backgroundcharacteristics

Thissectionlooksattheidentifiedanalyticalsample(n=783)anddifferentiatesresultsby different student background characteristics. All descriptive analyses wereconductedforthefollowingbackgroundcharacteristics:Gender,BMEstatus,Feestatus,OverallWPFlag,SchoolGroup(UKonly),SchoolGroup(International).However,onlydifferences that are significant between different student groups are reported anddiscussed.

Figures 7.12 to 7.14 present the differences ofwhen students start preparing for theadmissions test by their characteristics. UK students were significantly less likely toreportnopreparationforthetestthanothers,andnon-EUstudentsweresignificantly

0% 20% 40% 60% 80% 100%

Other

Used services of an external agency

Had private tutorials

Attended commercial preparation course(s)

Talked to current or former Oxford students

Talked to former pupils at my school/college

Attended Oxford Open Days

Used subject syllabus/textbooks

Read commercially published test preparation book(s)

TSA BMAT Other

45

morelikelytoreportnopreparation.Non-EUstudentswerealsosignificantlylesslikelytoreportthattheystartedpreparing1-2monthsbeforethetestthanothers.

Significantlymore students fromUK state schools reported preparing just before thetestthanthosefromindependentschools,andtheywerealsosignificantlylesslikelytoreport preparing 3-6 or 6-12 months before the test than those from independentschools(Figure7.13).Further,studentsfromschoolsoutsidetheUKsignificantlymoreoftendidnotprepareforthetestatall.

Figure7.12:Timestudentsstartpreparingfortheadmissionstestbyfeestatus

Figure7.13:TimestudentsstartpreparingfortheadmissionstestbySchoolType

(UKonly)

5%

6%

16%

18%

20%

49%

52%

35%

24%

17%

27%

9%

6%

8%

UK

EU

Non-EU

I don't remember I did not prepare at all Just before the test

1-2 months before the test 3-6 months before the test 6-12 months before the test

More than a year before the test

9%

20%

43%

51%

32%

20%

13%

7%

Independent

State

I dont remember I did not prepare at all Just before the test

1-2 months before the test 3-6 months before the test 6-12 months before the test

More than a year before the test

46

Figure7.14:TimestudentsstartpreparingfortheadmissionstestbySchooltype

(International)

Almostallstudentspracticedpasttestpapersbutfemalestudents(Figure7.15),thosewith an Overall WP Flag (Figure 7.16) and those from International schools (Figure7.17)practicedpasttestpaperssignificantlylessoftenthantheircounterparts.

With regard to the ways in which they practiced (mark scheme, feedback, timedconditions), significant differences are only found for students from independentschoolswhoreceivedfeedbacksignificantlymoreoften(56.9%)comparedtothose instateschools(31.4%)and forstudents fromUKschoolsoverall (40.3%),comparedtointernationalschools(15.4%).

Figure7.15:Amountstudentspracticedpastpapersbygender

Figure7.16:AmountstudentspracticedpastpapersbyOverallWPFlag

6%

16%

19%

49%

45%

24%

21%

9%

6%

UK school

International School

I dont remember I did not prepare at all Just before the test

1-2 months before the test 3-6 months before the test 6-12 months before the test

More than a year before the test

0% 20% 40% 60% 80% 100%

% practiced test papers

Very little

Somewhat

Quite a bit

A great deal

Male Female

47

Figure 7.17: Amount students practiced past papers by School Group

(International)

Figure7.18:OveralluseofonlineresourcesbySchoolType(International)

0% 20% 40% 60% 80% 100%

% practiced test papers

Very little

Somewhat

Quite a bit

A great deal

Overall WP Flag No Overall WP Flag

0% 20% 40% 60% 80% 100%

% practiced test papers

Very little

Somewhat

Quite a bit

A great deal

International school UK school

20%

13%

23%

28%

30%

25%

27%

34%

UK school

International school

Very little Somewhat Quite a bit A great deal

48

SignificantdifferencesintheoveralluseofonlineresourcesarefoundonlyforstudentsfromUKschoolscomparedtostudentsfrominternationalschools,wherestudentsfromUKschoolindicatedthattheyuseonlineresources‘verylittle’significantlymoreoften(Figure7.18).

Differences in family support received are not significant between students fromdifferent background groupswith one exception; students from international schools(39.4%) received significantly more support from their families than those from UKschools(29.6%).Thedifferenceissignificantandisduetohigheramountsoffinancialsupport(29%)thatstudentsfromInternationalschoolsreceivecomparedtotheirpeersfromUKschools(8%).

Figure 7.19 presents the differences in school support by students fromdifferentUKschools.Onlya relativelysmallpercentageof students reported that their schoolsdidnotencouragethemtoapply(10%).Butstudentsfromstateschoolsindicatedthattheywere less often encouraged to apply (12%) than students from independent schools(6%).Theywerealsolessoftenencouragedwhentheywereconsideringitthemselvesthanstudentsfromindependentschools(63%vs.74%)..Thedifferencesaresignificant.This result prompts a closer look at the kind of support that independent schoolstudentsreceivecomparedtostateschoolstudents.

Figure7.19SchoolencouragementbySchoolGroup(UKonly)

Figure 7.20 and significance tests reveal that students in independent schools areofferedsignificantlymoreopportunitiestoasksubject-specificquestionsorworkwithasubjectteacher,receivegeneralsupport/mentoringfromateacherandmayprepareinworkshopsorclassesfortheadmissionstestthanstateschoolstudents.

0% 20% 40% 60% 80% 100%

No, they actually discouraged me from

applying

No, they did not

Yes, I was already considering it and

they encouraged me

Yes, they suggested I should consider

applying

State Independent

49

Figure7.20FormsofschoolsupportreceivedbySchoolGroup(UKonly)

Students from International schools (Figure 7.21) were also significantly less oftenencouraged to apply and less often encouraged when they were considering itthemselvesthanstudentsfromUKschools.

Figure7.21:SchoolsupportbySchoolType(International)

Female students (27.3%) read commercially published test books significantly moreoftenthanmalestudents(18.1%).BMEstudents(39.6%)readcommerciallypublishedtest books significantly more often than non-BME students (20.5%). This may beattributabletoahigherpercentageofBMEstudentstakingtheBMAT(36.1%)thantheTSA (12.4%). BME students were also more likely to have attended commercialpreparationcourses(13.5%vs.6.0%).

0% 20% 40% 60% 80% 100%

Provide workshops or classes to help

prepare for the admissions test

Opportunity to use some class periods

to prepare for the test

General support/mentoring from a

teacher at your school/college

Opportunity to ask subject-specific

questions or work with a subject

teacher at your school/college

State Independent

0% 20% 40% 60% 80% 100%

No, they actually discouraged me from

applying

No, they did not

Yes, I was already considering it and

they encouraged me

Yes, they suggested I should consider

applying

International school UK school

50

StudentswithanOverallWPFlag talked to formerpupilsat their school/collegewhotook the test significantly lessoften than thosewithoutanOverallWP flag (8.3%vs.19.7%) andWP students reported using other strategiesmore often (4.8%vs. 1.4%)(bothdifferencesaresignificant).

Studentsfromindependentschoolstalkedtoformerpupilsattheirschool/collegewhotookthetestsignificantlymoreoftenthanthosefromstateschools(32.3%vs.11.0%),andalsoweremorelikelythanstateschoolstudentstohavetalkedtocurrentorformerOxford students who they knew (21.4% vs. 10.8%) (both significant) and to haveattended Oxford Open Days (32.2% vs. 24.4%). Students from International schoolsusedtheservicesofanexternalagencysignificantlymoreoftenthanthoseatUKschools(6.0%vs.1.6%),andalso talked to formerpupilsat their school/collegewho tookanadmissionstestsignificantlylessoften(11.3%vs.18.4%).Studentsfrominternationalschools also attended Oxford Open Days less often (6.0% vs. 27.9%) than those atschoolintheUK.

7.2 Associations between student background characteristics, test

preparationandtestperformance

This sections looks at associations between student background characteristics, testpreparationandtestperformance.Analysesareconductedforstudentsoftheanalyticalsample who took the TSA (n=194) and BMAT (n=70) test. Analyses are conductedseparately for the two groups but coefficients are comparable because dependentvariables(TSAtestscoreandBMATtestscore)havebeenstandardisedtohaveameanofM=0andastandarddeviationofSD=1.Further,becausesamplesarerelativelysmall,particularlyfortheBMATsample,aparsimoniousapproachtomodellingisadopted,i.e.where independent variables are identified as non-significant predictors of testperformance,theyareexcludedinthesubsequentanalysis.

Inthefirststepregressionanalysiswereconductedtopredicttestscoresbytheamountandnatureoftestpreparation.Table7.1summarisestheresults.Models1to5forTSAandModels7to11forBMATevaluatetherelationshipbetweeneachpredictorwiththetestscore individually. InModels6and12allvariableswereusedtogethertopredictthe testscore.Forboth tests,whenstudentsstartpreparing for the test issignificant.Thus,theearlierthestudentstartspreparingforthetestthehigherthetestscores.Theassociationsremainstatisticallysignificantwhenallotherpredictorsarecontrolledfor.Initially,theuseofonlineresourcesissignificantlyrelatedtoBMATtestscore,butwhenother predictors are controlled for, the association becomes insignificant. All otherindicatorsoftestpreparationarenotsignificantlyrelatedtotestscores.Consequently,theyareexcludedfromsubsequentanalysis.

51

Table7.1:Regressioncoefficientsfortestperformanceontestpreparationbytest

Coeff. SE sig. Coeff. SE sig. Coeff. SE sig. Coeff. SE sig. Coeff. SE sig. Coeff. SE sig.

fixedeffectsTimepreparation 0.173 (0.08) * 0.217 (0.09) *

Amountpracticed 0.187 (0.14) 0.045 (0.17)

Onlineresources 0.256 (0.15) 0.166 (0.15)

Familysupport -0.009 (0.16) -0.150 (0.16)

Schoolencouragement -0.013 (0.17) -0.112 (0.17)

randomeffectsexplainedvariance(in%) 12.1

Coeff. SE sig. Coeff. SE sig. Coeff. SE sig. Coeff. SE sig. Coeff. SE sig. Coeff. SE sig.

fixedeffectsTimepreparation 0.423 (0.13) ** 0.287 (0.15) *

Amountpracticed 0.432 (0.24) 0.233 (0.23)

Onlineresources 0.821 (0.28) ** 0.564 (0.32)

Familysupport 0.191 (0.27) 0.007 (0.26)

Schoolencouragement 0.115 (0.28) -0.003 (0.26)

randomeffectsexplainedvariance(in%) 17.30

p<0.05 (*), p<0.01 (**) and p<0.001 (***)

Model7

Model1 Model2 Model3 Model4

Model12Model11Model10Model9Model8

TSAtestscore

BMATtestscore

Model5 Model6

52

In a second step relationships between student background characteristics and test

preparation with test scores were evaluated (Table 7.2). As in the previous table,

Models1to8forTSAandModels11to18forBMATevaluatetherelationshipbetween

eachpredictorwith the testscore individually.Models9and10(TSA)and19and20

(BMAT)combineanumberofvariablestopredictthetestscore.

For the TSA test score, gender, OverallWP Flag and UK school group are significant

predictorswhenlookedatindividually.Malestudentsscorehigherwhilethosewithan

OverallWPflagandthosefromstateschoolsperformlower.Whenthosepredictorsare

evaluated in amultiple regression togetherwithwhen students startedpreparing for

thetestandtheiroveralluseofonlineresources(Model9),onlythetimewhenstudents

startedpreparingforthetestissignificantlypositivelyrelatedtotestscore.Thegender

coefficient closely misses the significance level. Model 10 additionally evaluates the

associationbetweenA-Levelbandandtestscores,butthisisnotasignificantpredictor

ofTSAtestscoreswheneitherlookedatindividually(Model8)orwhenothervariables

arecontrolledfor(Model10).

Similarly, for theBMAT test scores,havinganOverallWPFlagandhavingattendeda

state school are negatively associated with test performance when evaluated

individuallywhereasoverseasstudentsperformbetter.Moreover,studentswithbetter

A-Levels score significantly higher in the BMAT test. In the combined Model 19,

students from state schools perform significantly lower than those from independent

schoolsand femalestudentsperformsignificantly lesswell thanmalestudents.12Test

preparationvariablesarenotsignificantoncebackgroundcharacteristicsarecontrolled

for, i.e. characteristics that predict high performance in BMAT scores are out of

students’ control. Thismeans, in contrast to results for TSA,where preparing for the

testearliermitigateddisadvantagesfromdemographiccharacteristics,femaleandstate

schoolstudentsstillperformlowerinBMATdespitehavingequaltimepreparingforthe

tests as male and independent school students. Model 20 additionally evaluates the

associationbetweenpriorattainmentandtestscoresandthisissignificant,evenwhen

othervariablesarecontrolledfor.

ThevariablesinTable7.2explain19.7%of test scorevariance forTSA,but45.1% for

BMAT. Thus, students’ background characteristics seem to be substantially more

predictive of test scores for BMAT than for TSA. The amount of explained variance

remains fairly constant when additionally controlling for A-Level band (23.2% TSA;

52.6%BMAT).

Note, although Fee status (OS) is initially significant, it could not be included in the

combinedmodelforcomputationalreasons.Infact,studentsintheBMATsamplewith

anOverseasFeeStatusareonlyn=3.

12ItshouldbenotedthatmalestudentsareunderrepresentedintheBMATsample(20%)andthatthis

mayoverestimatethegendereffect.

53

54

Table7.2:Regressioncoefficientsfortestperformanceontestpreparationandstudentbackgroundcharacteristicsandpriorattainmentbytest

Coeff. SE sig. Coeff. SE sig. Coeff. SE sig. Coeff. SE sig. Coeff. SE sig. Coeff. SE sig. Coeff. SE sig. Coeff. SE sig. Coeff. SE sig. Coeff. SE sig.fixedeffectsGender(male) 0.428 (0.15) ** 0.311 (0.16) 0.277 (0.17)BME -0.25 (0.26)Feestatus(EU) 0.328 (0.21)Feestatus(OS) 0.254 (0.21)OverallWPFlag -0.51 (0.25) * -0.3 (0.25) -0.19 (0.27)SchoolgroupUK(State) -0.52 (0.18) ** -0.34 (0.19) -0.46 (0.21) *SchoolgroupINT(Intern.) 0.325 (0.17)A-Levelband -0.01 (0.08) -0.07 (0.08)Timepreparation 0.312 (0.10) ** 0.303 (0.10) **Onlineresources 0.028 (0.16) 0.012 (0.17)randomeffectsexplainedvariance 19.70 23.20

Coeff. SE sig. Coeff. SE sig. Coeff. SE sig. Coeff. SE sig. Coeff. SE sig. Coeff. SE sig. Coeff. SE sig. Coeff. SE sig. Coeff. SE sig. Coeff. SE sig.fixedeffectsGender(male) 0.571 (0.30) 0.72 (0.27) ** 0.515 (0.26) *BME 0.082 (0.26)Feestatus(EU) 0.17 (0.42)Feestatus(OS) 1.015 (0.57) *OverallWPFlag -0.99 (0.37) ** -0.58 (0.34) -0.58 (0.32)SchoolgroupUK(State) -0.98 (0.22) *** -0.68 (0.23) ** -0.81 (0.22) ***SchoolgroupINT(Intern.) 0.599 (0.40)A-Levelband 0.404 (0.17) * 0.272 (0.12) *Timepreparation 0.222 (0.13) 0.188 (0.13)Onlineresources 0.274 (0.28) 0.273 (0.28)randomeffectsexplainedvariance 45.10 52.60p<0.05(*),p<0.01(**)andp<0.001(***)

Model16 Model17 Model18 Model19 Model20

TSAtestscore

BMATtestscore

Model7 Model8 Model9 Model10

Model11 Model12 Model13 Model14 Model15

Model1 Model2 Model3 Model4 Model5 Model6

55

7.3 Mediating and moderating effects of student backgroundcharacteristics

In order to better understand the associations between student backgroundcharacteristics, test preparation and test performance, mediation and moderationanalysisisconducted.Mediationanalysisisusedtoinvestigatewhethertheassociationsbetween test preparation and test performance are to some extent associationsbetween student background characteristics and test performance because they areconfoundedwith each other, i.e. that for examplemale students use online resourcesmoreoftenandbecausemalestudentshaveonaveragehighertestperformanceitlookslikeonlineresourcesareassociatedwithtestperformance.Moderationanalysisisusedto investigatewhethertheassociationbetweentestpreparationandtestperformancevaries by different student background characteristics, i.e. that for example theassociationbetweenthetimestudentsstartpreparinginadvanceandtestperformanceisdifferentiallystrongforstudentswithorwithoutanOverallWPFlag.

7.3.1 Mediating effects of student background characteristics for theassociationbetweentestpreparationandtestperformance

In the regression models student gender, having an Overall WP Flag, being from anindependent or state school and prior attainment were at some point identified assignificant predictors of test performance on TSA or BMAT. Thus this selection ofbackgroundvariablesisusedinthemediationanalysis.Moreover,whenstudentsstartpreparing for the test and the overall use of online resources were identified assignificantpredictorsoftestperformance.Inaddition,theamountofpracticingpasttestpapers is included, as the previous descriptive analysis revealed several significantdifferencesonthisvariablebetweenstudentsofdifferentbackgroundcharacteristics.

Figure 7.21 shows the general mediation model conducted separately for the fourdifferent background characteristics. For each background characteristic a separatemediationmodelwasrunbothforTSAandBMATtestscoresasdependentvariables.

Neither of the selected student background variables significantlymediate any of theassociationsbetweentheTSAtestscoreandwhenstudentsstartpreparingforthetest,the amount they practice and their use of online resources. For BMAT, the onlysignificantmediationeffectwasfoundbetweentheuseofonlineresourcesandgender.That is, the positive association between the use of online resources is substantiallymediatedbygenderasthedirecteffectisinfactsmallerthantheindirecteffect.Thus,male students overall achieve higher on the BMAT test and because they use onlineresourcesmoreoftenthanfemales,itinitiallyappearsasiftheuseofonlineresourcesispositively related to test performance. However, a substantial amount of thisassociationisconfoundedwiththefactthatmalestudentsgenerallyattainhigherscoresinthetest.

56

Figure 7.21: Generalmediationmodel to evaluate the association between testpreparationandtestperformance

7.3.2 Moderating effects of student background characteristics for theassociationbetweentestpreparationandtestperformance

Again, the four student background characteristics (gender, Overall WP Flag, SchoolGroupUK,andpriorattainment)andthreetestpreparationvariables(thetimestudentsstart preparing in advance for the test, the overall use of online resources and theamountofpracticingpasttestpapers)wereselectedforthemoderationanalysisforthesamereasonsasoutlinedinsection7.2.1.

Figure7.22 shows thegeneralmediationmodel. Separatemodelswere conducted foreach of the four different background characteristics by each of the test preparationvariables.

Figure 7.22: Generalmediationmodel to evaluate the association between testpreparationandtestperformance

Time preparation

Amount practiced

Online resources

Background characteristics

Test score

Background characteristics

Test score Test preparation

57

Neither TSA nor BMAT test scoremodels show any significantmoderating effects ofstudents background characteristics on the relationship between test preparationvariables and test score. That is, the relationships between test preparation and testscores are not significantly different for students of different categories on thebackgroundcharacteristics.

7.4 Associationsbetweentimeofapplication,testpreparationandtestperformance

Thissectionevaluateswhetherthetimeelapsedbetweenthestudents’applicationandtheapplicationdeadline(15-OCT-2016)isrelatedtotestpreparationandperformance.Thevariablereflectsthedifferencebetweenthosetwodateswherelargervaluesreflectanearlierapplication.

Table7.3summarisestheresultsoftheregressionanalysesoftestperformanceontestpreparation and application time. The time that elapsed between the application andtheapplicationdeadlineisnotasignificantpredictoroftestperformance,eitherwhenevaluatedseparatelyorinrelationwithwhenstudentsstartpreparingforthetestandtheiruseofonlineresources.Whilethisisinlinewiththeresultsforon-coursestudentsinPhase1, it isworthremembering thatapplication timewassignificantlyassociatedwithtestscores in thesampleofapplicants,whereearlierapplicantsscoredhigher inbothtests.

7.5 Learningstrategiesandtestperformance

7.5.1 ThelearningstrategyscalesStudentswereaskedanumberofquestionsrelatingtothestrategiestheyappliedwhenpreparingfortheadmissionstest.Informationonstudents’learningstrategiesiswidelyusedinthecontextofthePISAstudy(Marshetal.,2006)inordertopredicteducationaloutcomes. The items used in this survey have been adapted for high-stakes tests inpreviousresearchbytheOxfordUniversityCentreforEducationalAssessment(Bairdetal.,2015).Overall,studentsprovidedinformationon14questions.Factoranalysiswasusedtomapthe14itemsontothreescalesidentifiedbyMarshandcolleagues(2006).These are: Control strategies, memorisation and elaboration. Table 7.4 gives anoverviewofitems,factorandscalecharacteristics.Withtheexceptionofone,allitemsweresuccessfullymappedontothethreescalesaccordingly.

58

Table7.3:Regressioncoefficientsfortestperformanceontestpreparationandapplicationtimebytest

TSAtestscore BMATtestscore Model1 SE Sig. Model2 SE Sig. Model1 SE Sig. Model2 SE Sig.fixedeffects Applicationtime -0.128 (0.65) -0.371 (0.65) 0.646 (0.97) 0.422 (0.89) Timepreparation 0.162 (0.08) * 0.318 (0.14) *Onlineresources 0.225 (0.15) 0.52 (0.31) randomeffects explainedvariancein% 3.6 16.5 p<0.05(*),p<0.01(**)andp<0.001(***)

59

Table 7.4: Overview of items used for learning strategy scales and statisticalinformationVariables Scale Eigenvalue Explained

VarianceReliability(α)

IstartedbyfiguringoutexactlywhatIneededtolearn.

Controlstrategies

2.895 48.1 0.811

I checked if I understood what I hadread.

Controlstrategies

I made sure that I remembered themost important points in the revisionmaterial

Controlstrategies

If I did not understand something, Ilooked for additional information toclarifyit.

Controlstrategies

I tried to figureoutwhich ideas Ihadnotreallyunderstood.

Controlstrategies

I tried to memorise all the materialrequired

Memorisation 3.22 74.5 0.916

Itriedtolearnmynotesbyheart. Memorisation

I tried to memorise as much of therevisionmaterialaspossible

Memorisation

ItriedtomemorisewhatIthoughtwasimportant.

Memorisation

I tried to relate new information toknowledgefromothersubjects.

Elaboration 2.469 49.7 0.792

I figured out how the informationmightbeusefulintherealworld.

Elaboration

I tried to understand the revisionmaterialbetterbyrelatingittowhatIalreadyknew.

Elaboration

I studied material that went beyondwhatwasexpectedforthetest

Elaboration

7.5.2 Associations between learning strategies, student backgroundcharacteristicsandtestperformance

RegressionanalysiswasusedtoevaluatewhetherthethreedifferentlearningstrategyscaleswereassociatedwithTSAandBMATtestscores.However,nosignificantresultscouldbereportedforeithertest.Nevertheless,acloserlookathowdifferenttesttakersandstudentswithdifferentbackgroundcharacteristicsmakeuseoflearningstrategieswhenpreparingforthetestrevealedsomesignificantdifferences.Specifically,studentspreparing for theBMAT test applied all three learning strategies – control strategies,memorisationandelaboration–moreoftenthanthosepreparingfortheTSA.ThesameistrueforBMEstudents,althoughitisworthnoting,thatsubstantiallymoreBMATtesttakersareBMEstudents(36.1%)thantheTSAtakers(12.4%).Further,femalestudents

60

applymemorisation strategiesmore often thanmale students. Considering that bothBMAT and TSA have major thinking skills components, this may not be the mostsuitablestrategyandmaypotentiallyberelatedtotheconsistentlylowertestresultsforfemalestudents in theBMAT. Inaddition,students fromschoolsoutsidetheUKapplymemorisationtechniqueslessoften.

Table 7.5: Descriptive statistics for learning strategies scale by test type andbackgroundvariables

Controlstrategies Memorisation Elaboration M SD Sig.# M SD Sig.# M SD Sig.#Testtypes

BMAT 0.52 0.56 *** 0.97 0.75 *** 0.51 0.71 ***TSA -0.19 0.97 -0.35 0.78 -0.18 0.90

Gender

Female 0.07 0.91

0.10 0.99 * 0.06 0.90

Male -0.13 0.96 -0.18 0.89 -0.10 0.91BMEstatus

Non-BME -0.11 0.95 *** -0.03 0.95 ** -0.07 0.87 **BME 0.48 0.56 0.60 1.04 0.31 0.84

OverallWPFlag

No -0.02 0.92

0.07 0.99

-0.01 0.87

Yes 0.18 0.87 0.22 1.05 0.10 0.95SchoolGroup(UK)

Independent 0.01 0.95

0.09 1.08

-0.06 0.95 State -0.01 0.90 0.07 0.95 0.03 0.82

SchoolGroup(International)

UK 0.00 0.92

0.08 1.00 ** -0.01 0.87

International -0.03 1.00 -0.32 0.80 -0.03 1.04#SignificancedifferenceinbetweenmeansistestedbyT-testsforindependentsamples.

8. Discussionandlimitations

TheprojectaimedtoevaluatetheimpactoftestpreparationforUniversityadmissionstestson testperformancewithdata fromOxfordUniversity.The topic isofparticularrelevancebecauseanincreasingnumberofundergraduateprogrammes–nationallyinthe UK but also internationally - require applicants to take an admissions test. TheoutcomesofthesetestsareconsideredforadmissiontoUniversityinadditiontopriorschoolattainmentmeasuressuchasGCSEscores.Howstudentsprepareforadmissionstestsandifthispreparationhasanimpactontheirperformanceinthetestishoweveranunder-researchedarea.Particularly,theprojectaddressedthefollowingoverarchingresearchquestions:

61

- Towhatextentdostudentsprepareforuniversityadmissionstestsandwhatisthe nature of this preparation amongst students with different backgroundcharacteristics?

- What are the connections between student characteristics, including priorqualifications,andtestpreparationandperformance?

- Is the time of application connected with preparation for, and betterperformancein,tests?

- Whatlearningstrategiesareconnectedwithbettertestperformance?

In order to address these questions the first phase of the project explored therelationships between student characteristics and test performance with OxfordUniversity Admissions data from 2010 to 2015. Drawing on information from theBrasenoseCollegesurveyonhowstudentsprepareforadmissionstests,findingsfromaliterature review and qualitative interviewswith a sample of students, a surveywasdeveloped. Itasked for informationonhow far inadvancestudentsstartedpreparingforthetest,thesourcestheyusedfortestpreparation,thesupporttheyreceivedfromvariousagentsandthestrategiesstudentsemployedtoprepareforthetest.Thesurveywas administered to students who commenced their studies at Oxford University inOctober2017.

Limitations

Inconsideringtheresults,itisimportanttobearinmindsomeofthelimitationsoftheresearchthatmayaffectourinterpretationofthem.

Althoughitwouldhavebeenvaluabletobeabletocomparethepreparationstrategiesbetweenstudentswhowereacceptedtotheircourseandthosewhowerenot,thiswasnotpracticallypossible,andthusthesurveysamplewaslimitedonlytothosewhoarecurrentstudents.

Thesestudentsthusrepresentalimitedbreadthoftheentireapplicantpoolandhenceamorehomogeneousgroup,particularlyintermsoftheirrangeoftestresults.It isalsothecase thatsomeof theresultsmightbeunderestimated(e.g.OverallWPFlag,BMEstatus),andthismayaccountforthelackofassociationseeninsomeareas,forexamplebetweentestresultsandschoolhistoryofapplying.

It is also important to note that the survey is based on self-reportedmeasures, andthere may be some differences between groups in the way in which they representthemselves-forexamplefemalesmaybelikelytobemorereservedintheirresponsesthanmales.

Neverthelesstherearekeyfindingsfromthisprojectwhicharediscussedbelow.Theseseemtobeallthemoreimportant,becausetheyareobserveddespitethesmallerandmorehomogeneousgroupofon-coursestudentsassessedinthisstudy.

62

Discussionofkeyfindings

Towhatextentdostudentsprepare foruniversityadmissions testsandwhat isthe nature of this preparation amongst students with different backgroundcharacteristics?Thereisevidencethatnearlyallstudentspreparetotakethetesttosomeextent.Onlineresourceswerecitedasfrequentlyusedbyallstudents.Whilethemostpopularonlinesource used was the Oxford University Website, students draw substantially on anumber of other sources to prepare for the test, such as the official test providerwebsiteandTheStudentRoom,withthemajorityofthosewhousedthemfindingthemtobequiteorextremelyhelpful.Moreover,studentsusedotherstrategiesandresourcestoprepareforthetest,suchasreadingcommerciallypublishedtestpreparationbooks,subject/syllabus books, attending Oxford Open Days and talking to current Oxfordstudentsorformerstudentswhotookthetest.

Allstudentspreparefortheadmissionstestindependentoftheirschooltypeandsocio-economic background.What is interesting though, is that the amount of support thatstudentsreceiveinordertoprepareforthetestsdiffersbythosecharacteristics.Mostnotably,studentsfromstateschoolsandthosewithanOverallWPFlagseemtohavetheleastaccesstosupport.Theyreceivedfeedbacklessoftenonthepasttestpaperstheypracticedandwerelesslikelytohavetalkedtoformerpupilsattheirschool/collegeorformerorcurrentOxfordstudentswhotookthetest.Itcanbeassumedthattheseformsofsupportarelessreadilyavailabletothem,ratherthantheyarelesslikelytodrawonthem.Inadditionstudentsfromstateschoolsaresomewhatlesslikelytobeencouragedto apply to Oxford and have fewer opportunities to prepare for the test in schoolcontexts,suchasteachermentoringorschoolinitiatedworkshops.Feweropportunitiesto prepare and weaker support networks may also be the reason why state schoolstudentsstartpreparing later forthetest.This istheonlyvariablerelatingtosupportthat is significantly positively related to test performance when other preparationstrategies are considered (both for TSA and BMAT) and when student backgroundvariables are controlled for (TSA only). This finding is in linewith previous researchthatevidencesthepositiverelationshipbetweentestpreparationandtestperformance(e.g.Bangert-Drowns,Kulik, andKulik1983;Bunting andMooney2001;Messick andJungeblut1981;Powers1986;Sturman2003).However,astheseresultsemergeinthegroup of on-course students only, it is reasonable to assume that the relative lack ofpreparation opportunities and support networks – through schools and former testtakers–playsanevenbiggerroleatanearlierstageintheselectionprocess.

What are the connections between student characteristics, including priorqualifications,andtestpreparationandperformance?Unsurprisingly, A-Level results are related to test scores for the applicant pool.However, when looking at on-course students only who took the TSA no significant

63

relationship was found anymore. This means, A-Level results seem to be highlyassociated with TSA scores and hence of being admitted to a course. The on-coursestudentsthenarearelativelyhomogeneousgroupintermsofA-Levelresultsandtestscores for which no significant relationship between A-Levels and test scores wasfound.

Withinthegroupofon-coursestudentswhotooktheBMAT,A-Levelswerestillhighlyassociatedwithtestperformance.ItseemsthatamoreheterogeneousgroupintermsofA-Levels(andpossibletestscores)hasbeenacceptedtocoursesthatrequiretheBMAT.Althoughnotpossiblewiththedataathand,itwouldhavebeeninterestingtoevaluatewhethertheyareinfactpredictiveofsuccessinthecoursesthemselves.

BME applicants, female applicants and those from deprived socio-economic areasperformedlesswellinboththeTSAandBMAT,othercharacteristicsbeingequal,whichconsequentlyaffectsthechancesofbeingacceptedtoacourse.Withinthegroupofon-coursestudentsthesizeoftheeffectsofthesecharacteristicshasdecreased,butlargelyremain to present as disadvantaging factors in TSA and BMAT performance. Theseresultscouldonlypartlybeconfirmedwiththemorerecent2017/2018studentcohort,possiblyduetotherestrictedsample.Here,femalestudents,thosefromdeprivedsocio-economicareasandfromstateschoolsperformedlesswellontheTSA.However,whenmultiple background characteristics and test preparation strategies are taken intoaccount,associationsbetweengender,socio-economicareasandtestperformancearelargelyexplained.ItisalsoworthnotingthatforTSA,testpreparationseemstocounterthe relative disadvantage associated with socio-economic deprivation, state schoolattendanceandgender.

ForBMAT,thepicture looksslightlydifferent.Students fromdeprivedsocio-economicareas, state schools and with lower A-Level results performed less well on the test.However,afteraccountingformultiplebackgroundcharacteristicsandtestpreparation,being female, from a state school and having lower A-Level results remainedsignificantly negatively associated with test performance. There is also evidencestudents’backgroundcharacteristicsexplainalargervarianceofBMATscoresthanTSAscores.

We did not find relevant evidence that student background variables mediate ormoderatetheassociationsbetweentestpreparationandtestscores.

Theseresultshaveimplicationsfordiscussionsaroundequityinadmissionstests.WhydoBME, femaleandapplicants fromsocio-economicallydeprivedareasunderperforminTSAandBMATdespiteequalGCSEandA-Level results?Similarly,whydostudentsfromstateschoolsunderperformincomparisontotheirpeersatindependentschools,despite GCSEs, A-Levels and other background characteristics being equal?With thedataathandwecannotanswerthesequestionsandtheremightbeothercharacteristicsnotincludedinouranalysisthatareabletoexplainthesegapsintestperformance.

64

Isthetimeofapplicationconnectedwithpreparationfor,andbetterperformancein,tests?ResultsprovideevidencethatapplicantswhoapplyearlierperformhigheronTSAandBMAT with other characteristics such as prior attainment, demographic and socio-economic characteristics being equal. However, for on-course students, it is the timestudents started preparing in advance for the test that positively predicts testperformanceinbothTSAandBMAT,ratherthanthetimeoftheirapplication.

Whatlearningstrategiesareconnectedwithbettertestperformance?Results show that students preparing for the BMAT test applied the three learningstrategies – control strategies,memorisation and elaboration –more frequently thanstudentspreparingfortheTSA.Also,femalestudentsusememorisationstrategiesmoreoften than male students. As both BMAT and TSA have major thinking skillscomponents,memorisationmaynotbethebeststrategyandmaypotentiallyberelatedtotheconsistentlylowertestresultsforfemalestudentsinBMAT.

Implications

There are practical implications of understanding these results. If preparation foradmissions tests has a positive impact on test performance, clearly those with lessopportunities andnetworks supportingpreparation are at a disadvantage.Advantagecanbegainedfrompractice,andthisneedstobecommunicatedexplicitlytothosewhowilltakethetests.Itseemsclearthatmoreadvantagedstudentsareprofitingfromthisknowledgealready,anditisfairthatthisknowledgebeshared.

Therearepracticalformsofoutreachthatmaybehelpful,includingdevelopingclearerguidance around the efficacy of preparation, and the importance of starting suchpreparationwell in advanceof the admissions round.This should includeadvice thattheattempttomemoriselotsofinformationmaynotbethebeststrategytoprepareforadmissions tests. Theremay also be consideration of how further ‘demystification’ oftheadmissionsprocesstoOxfordcanbetakenforward–clarifyingboththetimingandnatureoftheprocessesinvolved.

Moreover,previousresearchhasshownthattestpreparationaffectstestspecificskillsrather than general intelligence or even general subject knowledge (Arendasy et al.,2016).Consequently,theremayalsobemeritinreviewingtheemphasisplacedontestsintheadmissionsprocess,andwhethertheyareputtingalreadyadvantagedsubgroupsofstudentswithmoreopportunitiesandsupportforpreparationfurtherahead.

65

References

Adey, P.S. & Shayer, M. (1994). Really raising standards: Cognitive intervention andacademicachievement.London:Routledge

Arendasy,M.E.& Sommer,M. (2013).Quantitative differences in retest effects acrossdifferentmethodsusedtoconstructalternatetestforms.Intelligence,41(3),181–192.

Arendasy, M.E., Sommer, M., Guiterrez-Lobos, K., & Punter, F.J. (2016). Do individualdifferences in test preparation compromise the measurement fairness ofadmissiontests?Intelligence,55,44–56.

Baird, J.-A.,Hopfenbeck,T.N.,Elwood, J.,Caro,D.,&Ahmed,A.(2015).Predictability inthe Irish Leaving Certificate. Oxford University Centre for EducationalAssessmentandQueen’sUniversity,Belfast.

Bangert-Drowns,R.L.,Kulik, J.A.,&Kulik,C.-L.(1983).Effectsofcoachingprogramsonachievementtestperformance.ReviewofEducationalResearch,53(4),571–585

Black,B.(2012).AnoverviewofaprogrammeofresearchtosupporttheassessmentofCriticalThinking.ThinkingSkillsandCreativity,7,122–133.

Brinkmann,S.&Kvale,S.(2015).Sevenstagesofaninterviewinvestigation(PartII),in:S. Brinkmann (Ed.), InterViews: Learning the craft of qualitative researchinterviewing.Sage,LosAngeles,London,NewDelhi,Singapore,WashingtonDC.

Bunting,B.P.&Mooney,E. (2001).Theeffectsofpracticeandcoachingontestresultsfor educational selection at eleven years of age.Educational Psychology, 21(3),243–253.

Emery, J.L. (2009). The predictive validity of the BioMedical Admissions Test forpreclinicalexaminationperformance.MedicalEducation,43(6),557-564.

Flyvbjerg, F. (2006). Five misunderstandings about case-study research. QualitativeInquiry,12(2),219-245.

Ferguson,E.,James,D.,&Madeley,L.(2002).Factorsassociatedwithsuccessinmedicalschool:Systematicreviewoftheliterature.BritishMedicalJournal,324,952–957.

Fisher,A. (2005). ‘ThinkingSkills’ andadmission tohigher education.Reportpreparedfor the University of Cambridge Local Examinations Syndicate. Cambridge:UCLES.

Foley, L.J. (2012). Constructing the respondent. In: J. F Gubrium, J. A. Holstein, A. B.Marvasti, & K. D. McKinney (Eds.), Handbook of interview research: Thecomplexityofthecraft.ThousandOaks,CA:Sage,pp.305–315.

Gallacher, T., McElwee, S., & Cheung, K. (2017). BMAT 2015 test preparation surveyreport.CambridgeAssessment–AdmissionsTesting.WorkingPaper.

66

Griffin,B.,Carless,S.,&Wilson,I.(2013).Theeffectofcommercialcoachingonselectiontestperformance.MedicalTeacher,35(4),295-300.

Hausknecht,J.P.,Halpert,J.A.,DiPaolo,N.T.,&MoriartyGerrard,M.O.(2007).Retestinginselection:Ameta-analysisofcoachingandpracticeeffectsfortestsofcognitiveability.JournalofAppliedPsychology,92,373–385.

Higgins,S.&Hall,E.(2004).Pickingthestrawberriesoutof the jam:Thinkingcriticallyaboutsystematicreviews,narrativereviewsandmeta-analysis.BERAConferencePaper,UMISTSeptember2004.

Kulik, J.A., Bangert-Drowns,R.L.,&Kulik, C.-L.,C. (1984). Effectiveness of coaching foraptitudetests.PsychologicalBulletin,95(2),179-188.

Kirkup,C.,Wheater,R.,Morrison,J.,Durbin,B.,&Pomati,M.(2010).Useofanaptitudetestinuniversityentrance:avaliditystudy.BISResearchPaperno26(London).

Marsh,H.,Haug,K.-T.,Artelt,C.,&Baumert,J.(2006).OECD’sbriefself-reportmeasureof educational psychology’s most useful affective constructs: Cross-cultural,psychometriccomparisonsacross25countries. International JournalofTesting,6(4),311–360.

Messick, S. & Jungeblut, A. (1981). Time and method in coaching for the SAT.PsychologicalBulletin,89(2),191–216.

OECD(2010).PISA2009results:Learningtolearn–studentengagement,strategiesandpractices(VolumeIII).http://dx.doi.org/10.1787/9789264083943-en

O’Hare,L.&McGuinness,C.(2015).Thevalidityofcriticalthinkingtestsforpredictingdegree performance: A longitudinal study. International Journal of EducationalResearch,72,162–172.

Peers, I.S., & Johnston, M. (1994). Influence of learning context on the relationshipbetween A-level attainment and final degree performance: A meta-analyticreview.TheBritishJournalofEducationalPsychology,64(11),1–18.

Powers, D.E. (1985). Effects of test preparation on the validity of a of a graduateadmissionstest.AppliedPsychologicalMeasurement,9(2),179–190.

Powers, D.E. (1986). Relations of test item characteristics to test preparation/testpracticeeffects:aquantitativesummary.PsychologicalBulletin,100(1),67–77.

Powers,D.E.&Swinton,S.S.(1984).Effectsofself-studyforcoachabletest itemtypes.JournalofEducationalPsychology,76(2),206–278.

Rubin, H.J. & Rubin, I.S. (2005). Qualitative interviewing: The art of hearing data.ThousandOakes:SagePublications,Inc.

Säälik,Ü.,Nissinen,K.,&Malin,A.(2015).Learningstrategiesexplainingdifferencesinreading proficiency. Findings of Nordic and Baltic countries in PISA 2009.LearningandIndividualDifferences,42,36-43.

67

Slack,W.&Porter,D.(1980).TheScholasticAptitudeTest:Acriticalappraisal.HarvardEducationalReview,50(2),154–175.

Sturman, L. (2003). Teaching to the test: Science or intuition? Educational Research,45(3),261–273.

68

Appendices

AppendixA.TSAandBMATtables

Table A1. Average TSA standardised scores by CGSE quartiles and school typegroup(homeapplicantsonly)

GCSEquartiles

Schoolgroup Q1 Q2 Q3 Q4Independent -0.15(410) 0.18(838) 0.42(1169) 0.61(1396)

Other -0.2(123) 0.27(52) 0.65(30) 0.49(26)

State -0.31(1847) 0.13(1520) 0.38(1165) 0.6(1031)

Note:Samplesizeinparenthesis.

TableA2.AverageTSAstandardisedscoresbyCGSEquartilesandgender(homeapplicantsonly) GCSEquartiles

Gender Q1 Q2 Q3 Q4Female -0.52(847) -0.03(981) 0.19(983) 0.42(1110)

Male -0.14(1533) 0.27(1429) 0.55(1381) 0.75(1343)

Note:Samplesizeinparenthesis.

Table A3. Average TSA standardised scores by CGSE quartiles and BMEbackground(homeapplicantsonly)

GCSEquartiles

BMEbackground Q1 Q2 Q3 Q4White -0.14(1616) 0.2(1730) 0.48(1668) 0.69(1682)

Notknown/Refused -0.28(164) 0.25(179) 0.22(183) 0.48(194)

BME -0.65(600) -0.06(501) 0.24(513) 0.38(577)

Note:Samplesizeinparenthesis

69

TableA4. Frequenciesofbackgroundcharacteristics from the interviewsampleandanalyticalsample

Characteristics Sample(46) Analyticalsample(10)

N % N %Female 19 41.3 5 50Male 27 58.7 5 50Subjectstudy(collapsed)

Humanities 16 34.8 1 10MedicalSciences 7 15.2 3 30

MPLS 14 30.4 3 30SocialSciences 9 19.6 3 30

Subjectstudy(detailed)

Economics&Management 2 4.3

EngineeringScience 4 8.7 1 10English 8 17.4

English&ModLanguages 3 6.5

ExperimentalPsychology 1 2.2 1 10History 4 8.7 1 10

Law 2 4.3 1 10Mathematics 6 13

Medicine 6 13 1 10ModernLanguages 1 2.2

Physics 4 8.7 1 10PPE 5 10.9 2 20

Feestatus

UK 40 87 9 90EU 3 6.5 0

non-EU 3 6.5 1 10SchoolType

Independent 20 43.5 6 60State 19 41.3 3 30Other 7 15.2 1 10

70

TableA5:Descriptivestatisticsofsamplesusedintheanalysis

Respondents Non-Respondents TSA BMAT

N 783 2,737 194 70Variables % % % %

Female 57.6 47.1 57.7 80BME 16.5 18.1 12.4 36.1UK 75.5 87.9 71.6 87.1EU 11.5 7.4 13.4 8.6Non-EU 13 13.7 14.9 4.3OverallWPflag 14.2 10.9 12.2 11.5IndependentSchool 36.6 46.6 31.4 38.7InternationalSchool 22.6 18.8 27.3 10.4ALevelBand

LessthanA*A*B,A*ABorAAB 0.2 0.7 AAB 2.0 1.0 1.6 AAA 7.1 6.8 7.8 A*AB 3.4 1.7 2.3 A*AA 17.9 20.6 18.8 14.8A*A*B 0.7 1.1 A*A*A 22.9 26.0 21.9 183A*s 13.9 12.1 38.3 47.54A*s 30.9 28.7 9.4 19.7

5A*sorbetter 1.1 1.2 Practisingpastpapers*

Practicedpastpapers 98.4 97.1Usedmarkscheme 88.4 88.2Receivedfeedback 16.2 32.4

Practicedundertimedconditions 84.3 89.7Onlineresources*

OxfordUniversitywebsite 60.3 32.9Officialtestproviderwebsite 63.9 87.1

Otherwebsitesdedicatedtothetest 11.9 38.6Yourschool/collegewebsite 5.2 4.3

Anotherschool/collegewebsite(s) 0.5 2.9TheStudentRoom 32 35.7

Otherstudentforumwebsites 2.1 4.3YouTube 4.1 10Other 6.7 7.1

Familysupport* 30.1 28.6Schoolsupport*

Theydiscouraged 3.6 2.9Theydidnotencourage 19.7 21.4

Theyencouraged 58.0 57.1Theysuggested 16.1 18.6

Provideworkshops/classes 30.6 42.9Useclassperiods 1.6 2.9

Mentoring 21.2 24.3Subject-specificquestions 20.7 44.3

Otherstrategiesandresourcesused* Commerciallypublishedtest

preparationbooks 34 65.7

Commercialpreparationcourses 3.6 20Servicesofanexternalagency 0.5 7.1

Privatetutorials 3.6 4.3Subjectsyllabus/textbooks 5.2 65.7

71

Formerpupilsatmyschool/college 7.7 22.9CurrentorformerOxfordstudents 9.8 17.1

OxfordOpenDays 13.9 21.4Other 1 1.4

Respondents Non-Respondents TSA BMAT

Variables M SD M SD M SD M SDTSAscore(standardised) -0.07 1.00 0.03 1.00 0.00 1.00 BMATscore(standardised) 0.19 1.09 -0.12 0.92 0.00 1.00Timelapsed 13.05 52.61 6.39 6.14Time preparation (0 'Not at all' - 5'Morethan1yearbefore') 3.08 0.87 3.36 0.85

Amountpracticingpastpapers(0'Verylittle'-3'Agreatdeal') 1.95 0.83 2.26 0.84Usefulnessinwaysofpractisingpastpapers(0'Notatallhelpful'-3'Extremelyhelpful')

Markscheme 2.28 0.71 2.32 0.68Feedback 2.35 0.71 2.27 0.70

Practicingundertimedconditions 2.48 0.56 2.69 0.47Overalluseonlineresources(1'Verylittle'-4'Agreatdeal') 2.72 1.06 3.16 0.94Helpfulnessonlineresources(0'Notatall/notveryhelpful-3'Extremelyhelpful')

OxfordUniversitywebsite 0.73 0.57 0.91 0.60Officialtestproviderwebsite 1.11 0.60 1.46 0.62

Otherwebsitesdedicatedtothetest 0.91 0.79 1.19 0.68Yourschool/collegewebsite 1.50 0.71 0.33 0.58

Anotherschool/collegewebsite(s) 0.00 1.00 0.00TheStudentRoom 0.68 0.57 0.52 0.59

Otherstudentforumwebsites(besidestheStudentRoom) 0.75 0.50 0.67 0.58

YouTube 1.38 0.52 0.86 0.38Other 0.38 0.51 0.60 0.55

Learningstrategies Controlstrategies -0.19 0.97 0.52 0.56

Memorisationstrategies -0.35 0.78 0.97 0.75Elaborationstrategies -0.18 0.90 0.51 0.71

* Percentages relate to students who indicated that they used this particular strategy, resource orreceivedaparticularsupport.

72

AppendixB

Table A6: Regression results for combined BMAT test scores (Medicine andBiomedicalSciences)andBMATtestscoresforMedicineonly

VariablesBMAT BMAT-Medicineonly

Coef. SE Sig. Coef. SE Sig.

Intercept -63.25 11.03*** -64.09 11.06***Gender(Female=1) -0.25 0.02*** -0.23 0.02***Schooltype(Other=1) 0.43 0.30 0.41 0.30Schooltype(State=1) -0.04 0.02 -0.04 0.02BME(don'tknow/refuse=1) -0.29 0.05*** -0.31 0.05***BME(Yes=1) -0.27 0.02*** -0.28 0.02***ACORNFlag -0.11 0.04** -0.11 0.04**POLAR3Flag 0.01 0.03 0.00 0.03Pre16SchoolFlag -0.05 0.03 -0.03 0.04Post16SchoolFlag -0.05 0.03 -0.03 0.03OverallWPFlag -0.13 0.05** -0.15 0.05**Applicationtime -0.45 0.19* 0.00 0.00*GCSEA*s(zscored) 0.19 0.01*** 0.19 0.01***A-levelA/A*s(zscored) 0.21 0.01*** 0.21 0.02***Schoolhistoryofapplication 0.01 0.01* 0.00 0.00*UcasCycle 0.03 0.01*** 0.03 0.01***A-LevelArt -0.01 0.06 0.02 0.06A-LevelBiology 0.06 0.06 0.02 0.06A-LevelBusinessStudies 0.17 0.28 0.18 0.28A-LevelChemistry 0.31 0.14* 0.40 0.28A-LevelComputing 0.26 0.33 0.26 0.33A-LevelEconomics 0.24 0.07** 0.23 0.07**A-LevelEnglish 0.22 0.14 0.22 0.14A-LevelEnglishLanguage 0.04 0.17 0.11 0.18A-LevelEnglsihLiterature -0.07 0.05 -0.09 0.05A-LevelFurtherMathematics 0.27 0.04*** 0.26 0.04***A-LevelFurtherAddidional 0.00 0.00 0.00 0.00A-LevelGeography -0.11 0.06 -0.12 0.06*A-LevelHistory -0.01 0.04 -0.01 0.04A-LevelLanguages -0.05 0.03 -0.05 0.03A-LevelLaw -0.63 0.37 -0.64 0.37A-LevelMathematics 0.12 0.03*** 0.11 0.03**A-LevelOther 0.09 0.06* 0.14 0.07*A-LevelPhilosophy 0.25 0.15* 0.31 0.15*A-LevelPhysics 0.11 0.03*** 0.11 0.03***A-LevelPsychology 0.07 0.06 0.08 0.06A-LevelReligiousEducation 0.17 0.07* 0.17 0.07**A-LevelScienceOther -0.31 0.20 -0.25 0.21A-LevelSocialScience -0.14 0.21 -0.16 0.21p<0.05(*),p<0.01(**)andp<0.001(***)

73

AppendixC.BrasenoseCollegesurvey:selectedquotes

B1.Amountofpreparation‘I sat an aptitude test but didn't really do anything in the way of preparationbecause Iknew itwouldbeon topics Ihadn'tencounteredbeforeandwouldbelookingtoassessmymoregeneralskillsasahistorian,thatIcouldn'treallyrevise’(Arts&Humanities,2014).

B.2Supportfromschools‘TooktheTSA.Schoolhadzeroinput’(SocialScience,2013).

‘Ihad to take theBMAT,myschooldidn'thavea lotof input inmypreparation’(Sciences,2014).

‘…my schoolheld classesonce aweekgoingover them (past testpapers) for acoupleofmonthsbefore(thetestwastaken)’(Sciences,2014).

‘Myschoolgaveseveralhalfhourmeetingstodiscussgeneralwaysofapproachingthe…test’(Arts&Humanities,2014).

‘HadtotaketheTSA–hadacoupleofsessionsatschoolandpreparedusingthepastpapers/questionbooks’(SocialSciences,2013).

‘… our teacher gave us a powerpoint slide detailing the types of questions thatmight arise and how best to logically go about answering them’ (Joint Degree,2015).

‘preparedbydoingthepastpapers,schoolhassome(notveryuseful)classesonit’(ArtsandHumanities,2015).

‘Myschoolgavemesomeclasses inadvancebutthesewereof littleuse’(Arts&Humanities,2015).

‘Schoolheldworkshopsbutwerenotuseful’(Sciences,2015).

‘My schools was very supportive but couldn’t do too much to help’ (SocialSciences,2013).

‘Schoolhadnoinputbecausetheydidn’treallyunderstandwhatitwas…’(Arts&Humanities,2014).

‘…they(schoolteachers)weren’tquitesureexactlyhowbesttohelpallthetimeastheywereunfamiliarwiththesystem(Sciences,2014).

‘Iwasvery luckybecausemyschoolpreparedmewell forthetest, Idida lotofpracticetestsandmyteacherwaswillingtomarkthemandtalkmethroughsomefeedback’(Arts&Humanities,2015).

‘… a couple of meetings with a maths teacher at my school helped me to gothroughthem(practicemathsquestions)’(Science,2013).

74

‘Ididpastpapersandhadfeedbackfromahistoryteacher(…who)toldmewhereIwasgoingwrongandhowIcouldimprove(Arts&Humanities,2014).

‘Prepared quite a lot for the MAT. School put on twice weekly lunchtimeextensionsinpreparation.Bythetimeoftakingthetesthadcompletedallofthepastpaperquestionsavailableonthewebsite’(Science,2014).

‘IwroteouteverypastHAT testandmyschoolmarkedmyessaysandgavemefeedback’(Arts&Humanities,2014).

‘Didallthepastpapers,myschoolheldclassesonceaweekgoingoverthemforacoupleofmonthsbefore’(Science,2014).

B.3Useofexampreparationresources‘MyschoolhadverylittleinputbeyondadministeringtheTSA.IpreparedjustbydoingasmanypastpapersasIcouldfind’(SocialSciences,2013).

‘Ididalloftheavailablepracticetests,butreceivednoassistancefrommyschool’(SocialSciences,2013).

‘Iboughtthegrammarbookthatstudentshereuse(IwasadvisedinBlackwells)andIwentthroughit,makingnotes’(Arts&Humanities,2013).

‘I was given very little guidance bymy school so resorted to asking students Iknewhad studiedatOxfordabout thebestway toprepare’ (Arts&Humanities,2013).

‘Myschooldidnotorganisemuchtohelpme,however,aformerOxfordundergradwhonowworksatmyschoolgavemeafewpointersonwhattolookfor’(Arts&Humanities,2013).

‘ImainlypreparedbyrevisingmyGCSEsciencecourse’(Sciences,2014).

‘IdidtheHATsoIjustprintedoffeverypastpaperandmadesurethatIknewmyA level history syllabus so I could draw upon pre-existing knowledge’ (Arts &Humanities,2013).

‘…overthehalftermholidayIensuredIhadasoundknowledgeofmyAStopics’(JointDegree,2013).

‘Myschoolprovidedmewithallgrammartextbooks Ineed formyGermantest’(JointDegree,2013).

‘AsIdoModernLanguages,Ialsogenerallyrevisedgrammar’(Arts&Humanities,2013).

‘…thecontent(ofthePAT)wasfarremovedfromtheASsyllabus’(Science,2013).

B.4Perceptionsofthetests‘…wasquiteintimidating,andhadlittleideaaboutwhatIwasabouttosit’(SocialSciences,2013).

75

‘Didn’t really know what to expect from it (the test). Would have liked moreguidance…’(Arts&Humanities,2013).

‘Would have liked more guidance, especially on to how it featured in theadmissionsprocess’(Arts&Humanities,2013).

‘Itwasstressfulwithdifferentrumoursabouthowmuchweightingtheresulthadonadmissionprospects…’(Sciences,2014).

‘ThetestweremuchharderthananythingIhadpreparedfor’(Arts&Humanities,2013).

‘I remember the test being very challenging – harder than themock tests – butveryinterestingatthesametime’(Arts&Humanities,2014).

‘Thetestwasmoreenjoyablethanexpected’(JointDegree,2014).

‘Ireallyenjoyedsittingitstrangely’(Arts&Humanities,2014).

‘ThetestsweretoughbutIthinkrewardingandnecessary’(Sciences,2013).

‘The paper itselfwas unique and challenging, and a test in hindsight I probablyenjoyed’(Sciences,2015).

‘Test was challenging but enjoyable having the opportunity to write somethingentirelydifferenttoA-levels’(Arts&Humanities,2015).

76

AppendixD.Interviewguide

InformationandConsentOne of the aims of this research project is to learn how students prepare for Oxfordadmissionstests.Weareinterestedtoknowabouttheexperiencesthatyouhadsothatit can inform us about the connection between test preparation and admissionsoutcomes which we hope will contribute to developing effective support for futureapplicants.Youareinvitedtotakepartinaninterviewlastingaround45minutes. Theinterviewwillbeaudiorecordedandthetranscriptionwillbesenttoyouforverificationbeforeweanalyse it.The informationyouprovidewillbekept strictly confidential andyouranswers will be aggregated with those of other participants so that you remainanonymous.Theinformationyouprovidewillbeusedforthepurposesofresearchandstudentadviceonly.TheresearchisledbyDrDanielCarofromtheDepartmentofEducation,UniversityofOxford. Should you have any questions please contact him(daniel.caro@education.ox.ac.uk).For questions regarding the ethical approval process please contact the head ofDepartment of Education Research Ethics Committee, Dr Liam Gearon(liam.gearon@education.ox.ac.uk).Pleaseconfirmthat,

• youhavereadandunderstoodtheinformationaboutthisstudy.• youunderstandthatyoucanwithdrawfromthestudywithoutconsequenceat

anytimesimplybyinformingtheresearcher.• you are aware of who to contact should you have questions following my

participationinthisstudy.• you understand that this project has been reviewed by and received ethical

clearance through the University of Oxford Central University Research EthicsCommittee.

• youagreetoparticipateinthisstudy.Name(Interviewee): ________________________________Date: ________________________________Signature: ________________________________ Researcher(Interviewer): ________________________________Date: ________________________________ Signature: ________________________________

77

YouQ.Whichcoursedo/didyoustudyandwhendidyoustart?Q.Whichtestdidyoutakeaspartoftheadmissionsprocessandwhenwasthat?PreparationforthetestQ.Whendidyoufindoutthatthattestwaspartoftheadmissionsprocess?

o Whereandfromwhom?o Wereyouclearaboutwhatwasinvolvedinthetest?

Q. Whowould you say helped you to understandwhat the testwas about andwhat

preparationsyouwouldneedtomake?Weretheyexperts/experienced?o Howdideachpersonhelp?o Howmuchdidyoufigure-outforyourself?

UsefulnessofPreparationQ.Didyouprepareforthetest?Q.Ifso,whatdidyoudo?

o Organizeyourstudyspaceo Useflowchartsanddiagramso Practicepastexampaperso Explainyouranswerstootherso Organizestudygroupswithfriends

Q.Whichapproachesweremostusefultoyouandwhichwerelessso?Q.Whichmaterialsdidyoufindtobethemostusefulandwhichwerelessso?o PastPaperso Preparationbookso Preparationcourseso Oxfordwebsiteo Testproviderwebsiteo SubjectsyllabusQ.Wereyourpreparationsusefulforthetestorwasitdifferenttowhatyouexpected?Q.Wastheformateasytounderstand,doandcompleteontime?LogisticssurroundingthetestQ.Didyouregisterforthetestyourself?

o Didanyoneelsehelpwiththat?Q.Didyouhavetopayafeetotakethetest?

78

o Wasitmanageable?

Q.Wherewasthetestandhoweasywasittogetthere?Q.Whatwasitliketakingthetestinthatvenue?PerceptionsofImportanceofandRelevanceofthetestQ. When youwere preparing to apply, how important did you think the testwas ingettingyouaplaceatOxford?Q.Wasthetestreferredtolaterintheadmissionsprocess?Q.Whatwouldyouradvicebeforstudentswhowilltakethetestnextyear?

79

AppendixE:SurveyInstrument

80

81

82

83

84

85

86

87

88

89

90

91

92

93

94

95

96

97

98

99

100