ASSESSING HIGH-STAKES ASSUMPTIONS · 2017. 10. 17. · ASSESSING HIGH-STAKES ASSUMPTIONS A longitudinal mixed-methods study of university entrance language tests, and of the policy

ASSESSING HIGH-STAKES ASSUMPTIONS

Bart Deygers A

ssessing high-stakes assumptions

Bart Deygers

Supervisor Prof. dr. Kris Van den BrandenCo-supervisor dr. Koen Van Gorp

ISBN: 9789 079219070

ASSESSINGHIGH-STAKESASSUMPTIONS

Alongitudinalmixed-methodsstudyofuniversityentrancelanguagetests,andofthepolicythatreliesonthem.

ProefschriftingediendtothetbehalenvandegraadvanDoctorindeTaalkundedoorBartDeygers,2017

PromotorProf.dr.KrisVandenBranden

Copromotordr.KoenVanGorp

BegeleidingscommissieProf.dr.LieveDeWachterProf.dr.JohnNorris

Juryleden

Prof.dr.ClaudiaHarschProf.dr.ElkePeters

KULeuvenFaculteitLetterenOnderzoekseenheidTaalkundeOnderzoeksgroepTaal&Onderwijs©2017BartDeygersISBN:9789079219070PrintedbyACCODrukkerij–Herent,BelgiumThisworkwasfundedbytheResearchFoundationFlanders,FWOVlaanderen,underGrantnumberG078113N.Allrightsreserved.Nopartofthisthesismaybereproducedortransmitted,inanyformorbyanymeans,electronicormechanical,includingphotocopying,recordingorotherwise,withoutwrittenpermissionfromtheauthororfromthepublishersholdingthecopyrightofthepublishedarticles.

ForMax

I

IwakewiththesparrowsAndIhurryofftoworkTheneedforvalidation,babyGonecompletelyberserk

NickCave

II

APERSONALNOTE

ThisPhDstartedoutasaprojectproposal–anunreadabletitleandanimaginarytimeline.Gradually,itstoppedbeingaproject,andstartedtotrulymatter.It’staughtmealot,sharpenedmymindanddeepenedmythinking.Iwillmissit,andIwillalwaysbegratefultothepeoplewhohelpedshapeit.Thisisforyou.Thankyou,Krisfortheopendiscussions,andtherightkindofencouragement.Forgivingmefreedomandtrust,foraskingthought-provokingquestions,andforansweringmyoutburstswithapinchofstoicism.Thankyou,Koenforgivingmesomanyopportunities.Forencouragingmetodoresearchinthefirstplace.Forcountlessbeersonthreecontinents,and forabunchofmemorable latenights.Thankyoubothfortheextraeffortsyoumadeduringthefinalweeksofwriting.John.Youchangedeverythingwhenyou invitedmetoGeorgetown.Thankyouforlookingoutforme,foryourto-the-pointcomments,andformakingmy35thbirthdaysospecial.Thankyou,Lieve,foryourmoralsupport,yourconstructivecomments,andyour spontaneity.ThankyouElke, for thecoffeesand thesolidadvice.AndClaudia,forcombininginspiringresearchwithsocietalinvolvement.Cecilie, sparring partner extraordinaire, thank you so much for the animateddiscussions,thesolidadvice,thepersonaltalks,theunstoppablelaughsandtheunwavering support. To all my colleagues at the Centre for Language andEducation:thankyou!Abigshout-outtoKathelijne,Joke,Goedele,Lies,Carolienfortheupliftingtalks,andtotheCNaVTcrewforthehelp.Thisstudycouldnothavehappenedwithoutyourcooperation,andwithouttheassistanceofITNA.Ittakes a courageous and self-critical testdeveloper toparticipate in a study likethis. Thank you, Young A, Sandra, John, Tyler, Meg, Mina, Todd, Amy, andFrancesca –my dear Georgetown friends – for setting some impressively highstandards.Thankyou,Lourdes,forhelpingmegetChapter5tothenextlevel.Togetthisresearchofftheground,quiteafewpeoplehelpedout.Bart,Caroline,Carolien, Delphine, Sofie, Eefje, Ellen, Freek, Inge, Jordi, Kathelijne, Martien,Nele,Sara,Sarah,Sibo,Sien,Lisa,Jackie,Eva,Vanessa,Willemijn,mum,dad,andMarie-Paule–youguysaremagnificent.AndthankyouNick,Mariangela,Esther,Beate, Dina, and the other wonderful people in ALTE. Chapter 1 would havelookedverydifferentlywithoutyourhelp,ThankyouEllen.Forthetalks,theopenness,thelaughsandthewarmth.Fortheweird foodweeat.AndTom, for leadingbyexamplewhen itcomes todoingaPhD. 130 represent.Thankyou,Sibo for challengingmyassumptionsabout theworld, Sünbül for your unique blend of frankness, andMarieke for foolingmeannually.Thankyou, JordiandFilip forthebestkindofpeerpressure.Thomas

III

andLaure-Ann,forthescientificandmoralsupport,SaraandAlineforthethinktank.Kris,fortheno-frillsencouragement,andSaraforastellarC1performance.Andries, fordiscussing JohnRawlsonarandomtrainride.Sarah, forourwide-ranging lunchtime discussions, and for your take on justice and policy. Also,Mandy.Nobodyhasbeenatmysidequiteasmuchasyouhaveduringthesepastyears.ThankyouSheila,BernadetteandSuzanne,fortheoinksandthepokes.Thankyou for the loveof languageandeducation, and thehelpalong thewaymama,papa,VeerleandKatrien.Sofie!Ican’tthankyouenough.Forremindingmeofthebiggerpicture,forpushingmetoexplorelifetothefullest.Fortellingme not to cut corners, and for defusing the stressfulmoments that invariablyoccur in a project like this with disarming jokes, wisdom, and proofreadingwizardry.A very special word of thanks goes to the respondents. I am indebted to theacademicrespondentsandthepolicymakers,fortheircandorandparticipation.IowetheL2studentswhotookpartinthelongitudinalstudyaworldofgratitude.Thankyouforallowingmetobeapartofyourlivesduringaroughandeventfulyear. Thank you for making me see the full value of education and the storybehindascore.Thankyouformakingmerealizewhathighstakestrulymean.

IV

V

CONTENTSIntroduction:Examiningassumptions 12

TheuniversityentrancepolicyinFlanders,Belgium 12Validatingtestscoreuse 18Anoteonpolicy 21Researchgoals 22Availableevidence 27Researchdesign 29Structureandrelevanceofthisdissertation 34Anoteoncomposition 36

PART1:LEVELS&CONSTRUCTS 38Chapter1:UniversityadmissionpoliciesacrossEurope 40

Researchquestions 42Participants&methodology 42Results 45Discussion 51Conclusion 52

Chapter2:Content&levelrepresentativeness 54

Academiclanguagerequirements 54Justice 55Researchaims 57Participants&methodology 58Results 64Discussion 74Conclusion 76

PART2:SELECTION&DISCRIMINATION 78Chapter3:Level&constructequivalence 80

Equivalence 80Justice 82Researchquestions 83Participants&methodology 83Results 88 Discussion 94Conclusion 97

VI

Chapter4:Criterionequivalence 100

CriterionequivalenceandtheCEFR 101Researchquestion 103Participants&methodology 103Results 106Discussion 112Conclusion 113

Chapter5:ComparingL1andL2performance 116

ResearchintoL1andL2performance 116Researchquestions 120Participants&methodology 121Results 127Discussion 131Conclusion 134

PART3:GAINS&CONTEXT 136Chapter6:ExaminingL2gains 138

Researchquestions 143Participants&methodology 145Results 149Discussion 168Epilogue 173

PART4:POLICY,CONCLUSION&IMPLICATIONS 176Chapter7:Thepolicy-makingprocess 178

Examininguniversityadmissionpolicies 178Researchquestions 180Participants&methodology 180Results 182Discussion 186Conclusion 187

Chapter8:Summary&discussionoftheresearchfindings 190

Researchgoal1 190Researchgoal2 194Researchgoal3 199Researchgoal4 200Researchgoal5 202

VII

Chapter9:Limitations,implications&recommendations 204Afewwordsonstrengthsandlimitations 204Implications 206Recommendations 211“Theneedforvalidation,baby,gonecompletelyberserk” 214

References 216AcademicoutputrelatedtothisPhD 239SummaryinDutch 242Appendix 247

VIII

TABLES&FIGURES

IntroductionTable1.1.Internationalstudents 13Table1.2.Languagerequirements 15Table1.3.Doublecoding 33Figure1.1.Toulmin’sargumentstructure 19Figure1.2Validationscheme 20Figure1.3Researchdesign 30

Chapter1:UniversityadmissionpoliciesacrossEurope

Table2.1.Countriesandregionssurveyed 43 Table2.2.Testsacceptedforuniversityentry 45Table2.3.CEFRlevelrequiredforuniversityentrance 46Table2.4.IsB2enough 46Table2.5.Whodecidesonlanguagerequirements 47Table2.6.Empiricalresearch 48

Chapter2:STRT&ITNA:Content&levelrepresentativeness

Table3.1.Focusgroupsamples,arrangedbyCEFRlevel 62Table3.2.UniversitylecturesandSTRTlisteningprompts 66Table3.3.Academiclanguageskillsselectedinfocusgroups 71

Chapter3:level&constructequivalence

Table4.1.Researchvs.regularpopulation 85Table4.2.Descriptivestatistics 89Table4.3.Pass/Failcrosstab 89Table4.4.ITNAcomputerscores&STRTwrittenscores 90Table4.5.MFRAwrittentasks(equalweights) 91Table4.6.Promax-rotatedfactorloadings 92Table4.7.MFRAoralcriteria(equalweights) 93Table4.8.MFRAoralcriteria(actualweights) 94

Chapter4:criterionequivalence

Table5.1.Jaccardindexforratingdescriptorpairs 107Table5.2.FrequenciesofassignedCEFRlevels 108Table5.3.ProbabilityofattainingB2orhigher 108Table5.4.Relationshipbetweencorrespondingcriteria 109Table5.5.Multivariatelinearregression 110Table5.6.Linearregressiononcriterionlevel 111Table5.7.MFRA:STRTandITNA(bymeasure) 111Table5.8.MFRA:STRTandITNAcriteria(bymeasure) 112

IX

Chapter5:ComparingL1andL2performance

Table6.1.Multivariatelinearregression 122Table6.2.Demographicdataofparticipants 123Table6.3.Promaxrotatedfactorloadings 126Table6.4.Descriptivestatistics 128Table6.5.MFRAforfacet“Group” 128Table6.6.Wilcoxonsigned-ranktest:Flemish,L2FandL2I 129Table6.7.Wilcoxonsigned-ranktest:L1,G1.5,andL2F 129Table6.8.Logisticregression:Flemish~criterionscores 130Table6.9Multinomiallinearregression 131

Chapter6:examiningL2gains

Table7.1.Multivariatelinearregression 147Table7.2Languagegains 150Table7.3.Reduceddatamatrix:interpersonal 151Table7.4interpersonalrelationships 152Table7.5.Howdothinkyourclassmatesseeyou?(March) 158Table7.6.Reduceddatamatrix:institution 161

Chapter7:thepolicy-makingprocess

Table8.1.Policymakerrespondentcodes 180Table8.2.Datacodingcategories 181Table8.3.Exemptionsfromadmissionrequirements 183

Chapter8:summary&discussionoftheresearchfindings

Table9.1.STRT&ITNAtasktypes 195Table9.2.STRT&ITNAresultvs.academicsuccess 198

X

APPENDICES

Appendix1(1/3).STRTPart1:Listening-into-writing 247Appendix1(2/3).STRT,Part2:Reading-into-writing 248Appendix1(3/3).STRT,Part3:Speaking 249Appendix2(1/2).ITNA:computertest 250Appendix2(2/2).ITNA:Speakingtest 251Appendix3.L2Pparticipants 252Appendix4.L2Fparticipants 253Appendix5.Universitystaff 254

Introduction:Examiningassumptions

12

INTRODUCTIONEXAMININGASSUMPTIONS

Ifeverybodyhas theright toaneducation(UNGeneralAssembly, 1948),and ifeverybody has the right to pursue the goals that he or she deems valuable(Nussbaum, 2002; Sen, 2010), then an entrance policy that obstructs people’saccesstotheeducationof theirchoosingbymeansofanassessmentprocedurewouldrequirestrongempiricaljustification(Sen,2010).Itwouldhavetobeclearthat theadmissionpolicy facilitates theselectionof therightapplicants for therightreasons.Nevertheless, inmanycontexts,universityentrancerequirementsarebasedonassumptionsorclaimsthatareasyetunsupportedbyempiricaldata(McNamara&Ryan,2011).

Theprimary assumption supporting theuse of language tests to controlentrance to higher education is that a certain language proficiency level isrequired to verify whether L2 students will be able to meet the linguisticrequirements of academic studies. This assumption is rarely investigated orchallenged,however,eventhough its impact is substantial (McNamara&Ryan,2011). The purpose of this dissertation is to empirically assess the assumptionsthatsupporttheuseoflanguagetestsinonespecificcase:theFlemishuniversityentrance policy. When test scores are used in such a way that they mayfundamentally impact a test-taker’s opportunities in life, the stakes are high.Suchtests,andtheclaimsthataremadeonthebasisoftheirscores,requireclosescrutinyandrobustevidence.THEUNIVERSITYENTRANCEPOLICYINFLANDERS,BELGIUMFormoststudentswhohavegraduatedfromaFlemishhighschool,therearenoobligatoryorbindinguniversityentrancetests.OnlyFlemishstudentswhowishto pursue a degree inmedicine or dentistry need to pass an examination thattests their knowledge in exact sciences, and their reading skills. For all otherstudents with a Dutch high school degree there are no centralized subject-specifictestsorlanguagetestspriortouniversityentrance.

Therelativelyopenuniversityentrancepolicyhasresultedinlargegroupsof students in the first year, where ex cathedra teaching (i.e., one-waytransmission teaching) is the norm. Consequently, students are generally notexpectedtospeakorwritemuchinthecourseoftheirstudiesuntilthesecondorthethirdyear,whenthefirstwrittenpapersaredue(DeWachter,Heeren,Marx,&Huyghe,2013).Anotherconsequenceoftheopenregistrationpolicyisthatthedefactoselectionofstudentsoccursnotbeforethestartofacademicprogramsat


13

universitybutattheendofthefirstyear,whenaround60%ofthestudentsfailtheirexams(Amkreutz,2013).Studentswithanatypicaleducationalbackground,alowsocio-economicstatus,oranL1differentfromDutchareoverrepresentedinthegroupofstudentswhodonotpasstheirfirstyearatuniversity(DeWit,VanPetegem,&DeMaeyer,2000;Lievens,2016).

Thenumberof international studentsatFlemishuniversitieshasbeensteadily increasing inrecentyears(BeleidscelDiversiteitenGender,2016),eventhough the proportion of international students at Flemish universities is stillconsiderablylowerthanattheirBritishandNorthAmericancounterparts.

Table1.1.InternationalstudentsatFlemishuniversities(2015-‘16) Studentpopulation Internationalstudents UniversityofLeuven 41.500 19% GhentUniversity 41.000 11% UniversityofAntwerp 20.000 14% UniversityofBrussels 11.000 20% UniversityofHasselt 6.000 9% Note.ThesenumbersincludePhDstudents International students account for up to 20% of the population at the fiveFlemish universities (see Table 1.1). These publicly available percentages alsoincludePhDstudents,however,whoarenotrequiredtoattendcurricularclassesand who do not need to pass Dutch language tests. The proportion ofinternationalL2studentswhoarerequiredtopassalanguagetest(i.e.,thoseatthe bachelor ormaster level) is considerably lower than the publicly availablenumbers, but few universities are willing to disseminate detailed informationabout the actual composition of the non-PhD student population. At GhentUniversity, 1.7% of the newly registered freshmen in 2015 were internationalstudents (privatecommunication). Importantly,universitiesoftencountDutch-speaking students from the Netherlands – who are exempt from taking alanguage test – as international students.TheUniversityofAntwerpdoeskeeptrackoftheproportionofinternationalL2studentsattheundergraduatelevel.Atthisinstitution,lessthan4%oftheundergraduatestudentpopulationconsistsofinternationalL2students(privatecommunication),whichissubstantiallybelowthe publicly available figure of 14%. Given the disparity in definitions andinclusioncriteriaitisimpossibletostatehowmanyinternationalL2studentsareimpactedbytheuniversityentrancepolicy.


14

LanguageregulationsIthasbeenarguedthattheeducationallanguagepolicyinFlandersistoalargeextent based on ideology (Blommaert, 2011; Blommaert&VanAvermaet, 2008;Van Splunder, 2015). Policy makers do not always favor increasedinternationalization of the student population. In the minds of many, highereducation isstillprimarilyseenasaservicetoFlemishtaxpayers(Leliaert,2011;Truyts&Torfs,2015).Congruently,thelanguageofeducationinmostprogramsisDutch,theofficiallanguageofFlanders.Asaresultofnearlytwocenturiesoflanguage-related political turmoil an ideology of territorial monolingualism(Blommaert, 2011; Van Splunder, 2015) has permeatedmany aspects of Flemishsociety, including education.Many primary and secondary schools use a strictDutch-only policy (Agirdag, 2010; Blommaert & Van Avermaet, 2008; Strobbe,2016) and theofficial languagepolicyofhigher education inFlandershasbeeninfluencedbythesameideology.

The official language of Flemish higher education is Dutch, inadministrative and educational matters (Vlaamse Regering, 2013). Thegovernmental decrees that shape language regulations in Flemish highereducationstrivetowardsmaintainingDutchasanacademicallyviablelanguage.Recent rulings have become more lenient towards organizing education inlanguagesother thanDutch,butcanstillbeconsidered rather restrictivewhencompared toTheNetherlands (Wet op het hoger onderwijs enwetenschappelijkonderzoek, 1992), where over 60% of the bachelor and maser programs weretaught in English in 2016 (Bouma, 2016). A Flemish university cannot organizemorethan6%ofitsbacheloror35%ofitsmasterprogramsinalanguageotherthan Dutch. A program is considered non-Dutch when more than 18.33%(bachelorprograms)or 50% (masterprograms)of the classes arenot taught inDutch(DepartementOnderwijsenVorming,2015).

In contrast to their peers who graduated from a Dutch-medium highschool, international L2 students in Flanders need to prove a certain languageproficiency level. The level of language proficiency required by all Flemishuniversities for bachelor andmaster programs, is the B2 level of theCommonEuropean Framework of Reference for Languages (CEFR - Council of Europe,2001).

TheB2levelisthefourthofsixconsecutivelanguageproficiencylevelsontheCEFR(CouncilofEurope2001),whichstartsatA1andgoesuptotheveryadvanced C2 level. In the following chapters the B2 level, as well as itsapplicationsandapplicabilityinuniversityentrancetesting,willbediscussedindetail.Fornow,itsufficestodescribetheB2learnerassomebodywithalanguageproficiencythatiscomparabletoACTFLAdvancedMid(ACTFL,2016),whocanunderstandthemainideasofcomplextexts,interactfluentlyandspontaneously


15

withnative speakers,produceclear anddetailed texts, anddevelopa sustainedargumentation(CouncilofEurope,2001:24).

TheCEFRhasbeenwidelyadoptedbyeducationalpolicymakers,andits levels are used to determine entrance requirements in a wide variety ofcontexts (Figueras 2012). Unfortunately, however, the CEFR bands are ratherbroad(Fulcher2004;Hulstijn2014),andtwoteststhatlinktothesamelevelarenot necessarily equally difficult, even though policymay assume that they are(Green,Forthcoming).Becausethereissubstantialroomforvariationwithinoneand the same CEFR level, it is important not to assume equivalence of testssimplybecausetheysharethesameCEFRlevel,withoutempiricallyinvestigatingthatassumption(seethespecialissueofTheModernLanguageJournaleditedbyByrnes,2007).

InternationalL2studentswhowishtopursueaprograminwhichDutchisnotonlythemediumofinstruction,butalsothegoal(i.e.,translationstudies,orDutch literature)arerequiredtoproveC1proficiencyatsomeuniversities(e.g.,GhentUniversity,andUniversityofAntwerpasof2017),whileothersrequireC1for teacher trainingprograms (e.g.,University ofHasselt). B2, however, canbeconsideredthedefaultentrancerequirement(seeChapter6).Table 1.2. Language requirements for international L2 undergraduate students atFlemishuniversities

Level Acceptedproofoflanguageproficiency

B2req

uiremen

t

STRT&IT

NA

6

60creditsin

FlemishHE7

Flem

ishSE

8 deg

ree

STEX

II9

One

yea

rinFlemishSE

AccreditedB2

cou

rse

Med

icine/de

ntistry

admission

test

HOSP

10deg

ree

Limbu

rgH

Elang

uage

test

UniversityofLeuven1 ★ ★ ★ ★ ★ GhentUniversity2 ★ ★ ★ ★ ★ ★ ★ ★ UniversityofAntwerp3 ★ ★ ★ ★ ★ ★ UniversityofBrussels4 ★ ★ ★ ★ ★ ★ ★ ★ UniversityofHasselt5 ★ ★ ★ ★ ★ ★ ★ Note. 1KULeuven,2016,p.4;2UniversiteitGent,2016,p.19;3UniversiteitAntwerpen,2016,

p.3;4VrijeUniversiteitBrussel,2014,p.22,5UniversiteitHasselt,20166ITNA:InteruniversitaireTaaltoetsNederlandsvoorAnderstaligen(InterUniversitytestofL2Dutch),STRT:EducatiefStartbekwaam(Readytostarthighereducation),7HigherEducation, 8Secondary Education, 9Staatsexamen Programma II (State Exam,Netherlands), 10Hoger Onderwijs voor Sociale Promotie (higher education for socialpromotion)


16

DifferentuniversitiesallowfordifferentkindsofevidencebutcertaindocumentsareacceptedatallfiveuniversitiesasadequateevidenceofB2ability(Table1.2).A certificate of STRT or ITNA, two accredited B2 tests (see below), grantsadmission to every Dutch-medium program. Similarly, a degree of a Dutch-mediumhighschoolorsixtycredits inaFlemishhighereducationprogramareconsideredassufficientproofofB2ability.

The B2 requirement is not imposed on international students only;international teaching staff tooneed toproveB2ability if theydonot teach inDutch, C1 if they do (Vlaamse Regering, 2013).Within three years after beingappointedtheyarerequiredtopassITNA(DepartementOnderwijsenVorming,2015).STRT&ITNAITNA(InterUniversitytestofL2Dutch)isacomputer-basedandface-to-facetest,issued and developed by the Interuniversitair Testing Consortium of Flemishuniversitylanguagecenters.STRT(Readytostarthighereducation),atask-based,integrated-skilllanguagetest,istheonlyDutchlanguagetestattheB2levelthatis internationallyadministered. It isdevelopedat theUniversityofLeuven,andfunded by the Dutch Language Union, an international, intergovernmentalorganizationoverseeingtheDutch languagepolicy in theNetherlands,BelgiumandSuriname. ThecertificatesofthesetwotestsareacceptedbyallFlemishuniversitiesasproofoftherequiredB2ability.BothtestshavebeenlinkedtotheB2levelofthe CEFR (Council of Europe, 2001) following the familiarization, specification,standard settingandvalidationproceduresdescribedinFigueras,North,Takala,Verhelst,&VanAvermaet(2005).

STRTandITNAarecomparableonanumberofparametersotherthantheirCEFRlevel.BothtestshaveundergoneasuccessfulauditbytheAssociationof Language Testers in Europe (ALTE), offering an independent assessment oftheir validity, reliability, and consistency. They also share the same primarypurpose(i.e.,testingnon-nativespeakersofDutchforuniversityadmission),andrefer to communicatively oriented conceptualizations of language proficiency,suchasWeir's(2005)sociocognitiveframeworkandtheCEFR,asprimarysourcesof their construct. Finally, since both tests employ a pass/fail procedure,candidates either attain a B2 certificate, or not. In terms of operationalizationSTRT and ITNA show a number of substantial differences, especially in thewrittencomponent.Theoralcomponentsaremorecomparable,althoughtherearedifferences intheratingcriteriaused.Appendix 1and2giveanoverviewoftheoperationalizationofSTRTandITNArespectively.


17

ThewrittencomponentofSTRTispaper-basedandconsistsoftwocomponents.Inthewriting-from-listeningcomponentcandidateswriteanargumentativetextbased on audio input, and summarize a scripted lecture about a general topic.Thewriting-from-readingcomponentalsoincludesanargumentativetask,andasummary task with a substantial argumentative component. The writtencomponentofITNAiscomputer-basedandprimarilyincludesselected-responsequestiontypes.Candidatesdragjumbledparagraphstoorderthemcorrectly,filloutmissingwordsinatext,andanswermultiple-choicequestionsaboutreadingor listening prompts. At the B2 level, ITNA does not include writing tasks, aswritingismeasuredindirectlyusingselected-responseitemtypes.

Theoralcomponentsofboth testsdonot takemore than25minutes,includingpreparation time.Candidates interactwitha trainedexaminerduringtheoralcomponent,whichconsistsofapresentationandanargumentationtask.Theargumentationtaskinvitesthetesttakerstoweighanumberofalternativesolutions to a problem, and argue why their choice is the better one. In thepresentationtaskcandidatesbrieflypresentastudybyusinginputmaterialsuchas graphs and tables. Even though the oral tasks are similar in both tests, thescoring rubrics differ, because ITNA only takes into account linguistic criteria(Vocabulary, Grammar, Cohesion, Pronunciation, Fluency), while STRT alsofocuses on content (i.e., whether a performance contains the main pointsmentionedintheprompt).

STRTwritingtasksareratedbytwoindependenttrainedraterswhoscorecontent criteria in a binary way (i.e., whether the candidate mentions therequired aspectsornot) and linguistic criteriaon a four-point scale.The ITNAcomputer test is scoredelectronicallyusingabinary ratingprocedure,while itsoral component is scored in situby the examiner and an additional rater,whocometoajointoverallscoreforfivelinguisticcriteria.TheoralSTRTcomponentis administered by a trained examiner, recorded and centrally scored by twoindependent trained raters, using a rating scale that includes five criteria thatcorrespond to those used in ITNA, plus Register, Initiative and Content (i.e.,whetherthecandidatementionsallsalientpointsaskedforintheprompt).ITNAexaminersandraterstendtobeexperiencedL2teachersofDutchwhotypicallyattend training at least once a year and score oral tests at different timesthroughouttheyear.STRTratersareusuallynoviceraterswithabackgroundinlinguisticsorcommunicationstudieswhohavereceiveda two-daytrainingandhave shown that they are able to rate two sample exams with a satisfactorydegreeofaccuracyandconsistency.Thefirstdayoftrainingincludesbecomingacquaintedwiththetest, itspurposeandtheratingscales.Afterthis,candidateratersscoreeightstandardizedperformancesofeachtask(i.e.,48performances),comparingtheirscoretothestandardizedone.Attheendoftheseconddayofthetraining,candidateratersscoretwoexams(i.e.,12tasks).Theyareconsideredfitforratingwhenthescoresofallraterscorrespondtothestandardizedscores.


18

STRTcandidatesdonottypicallyreceiveadetailedscorereportthatlistsscoreson task or criterion level. Instead, it provides a certificate for candidates whoachievedanoverallRaschmeasureatorabove1.42(cutscorebasedonstandardsetting; private communication, 6 January 2016). ITNA informs candidateswhetherornottheypassedthecomputertestviae-mailonthesamedayastheadministration.Candidatesgettheirresultsoneachsectionofthecomputertest,and thosewho reached the cut scoreof 54%are invited to take theoral exam.After the oral component the overall cut score required to obtain ITNAcertificationis52.5%(privatecommunication,6March2015).

In line with the rising number of international students at Flemishuniversities, thecandidatureofSTRTandITNAhasbeengrowing.Thenumberof ITNA test takers has risen from 286 in 2010 (the year of the first ITNAadministration)toover1000in2014(InteruniversitairTestingConsortium,2015).STRTserved608candidatesin2010,and957in2014(CNaVT,2013,2016b).

VALIDATINGTESTSCOREUSETheapproachtovalidationadoptedinthisdissertationreliesonKane’sToulmin-inspired treatmentof explicit and implicit claimsconcerning the interpretationand use of test scores (Kane, 2013). Throughout the chapters, specificimplications and applications of Kane’s approachwill be explained, but at thispoint it isuseful todiscussthemoregeneral frameworkthat linksthedifferentchapters.

Test scores bear littlemeaning in a contextual vacuum. A score onlybecomes real when it has real-life consequences, such as access to a valuedposition, service or status. For that reason,most validation theories argue thatvalidating a test without considering its social context and consequences isinadequate(Bachman&Palmer,2010;Kane,2012,2013).Whatrequiresvalidationis not only the test itself (e.g., Borsboom&Markus, 2013), but also theway inwhich a score is interpreted and used (Kane, 2013). For Kane, validation is amatterofempiricallyinvestigatingtheclaimsthatsupportthewayinwhichscoreusersinterpretoruseascore.Theresponsibilityofvalidationthereforedoesnotfallonthetestdeveloperalone,butalsoonthescoreuser.Flawedtestsprecludevalid scoreuse, andas such the testdeveloper is responsible fordevelopinganinstrument of measurement that is valid for the purpose for which it wasintended (Norris, 2008). Score users bear the responsibility for proving anyadditionalclaimstheymaymake(Kane,2013).


19

Figure1.1.Toulmin’sargumentstructure

To substantiate claims related to score use, Kane proposes using Toulmin’sargument structure (Figure 1.1), which provides a transparent connectionbetween claims and the data onwhich these claims rely. In Toulmin’s logic, aClaim (C) isaconclusionbasedonData (D).Anyclaimthat isunsupportedbydata, is empty (Toulmin, 2003). Throughout this dissertation, we will examineclaimsthatsupporttheuniversityentrancepolicyinFlanders.Inpolicy,itisnotuncommonforclaimstoremainimplicit(Phillips,2007),andforthatreasonthetermAssumption(A)willbeusedthroughoutthisdissertationtorefertopolicyclaims.Anassumption,asdefinedbytheOxfordDictionaryrefersto“athingthatisacceptedastrueorascertaintohappen,withoutproof”.Typically,dataneedtobeinterpretedbeforetheycanleadtoaclaim.ThisconnectionbetweendataandtheclaimiscalledaWarrant(W).Backing(B)mayberequiredtooffercrediblesupport to the interpretation of the data (e.g., a publication that supports acertain use of the data). Finally,Qualifiers (Q) and Rebuttals (R) are used tospecifythedegreetowhichsomethingmaybetrue(Q),andtheconditionsunderwhichtheclaimmaynotapply(R). Figure1.2offersavisualrepresentationofthevalidationmodelemployedinthisstudy.Themodelshowsthedifferentstagesinvolvedinusingmorethanone language test in a high-stakes language testing policy. The upper halfconcernsclaimsthattestdevelopersneedtoprove,whereasthelowerhalfdealswithusesandinterpretationsthataretheresponsibilityofscoreusers.

The basic structure of the model’s upper half reminds of Bachman &Palmer's(2010)AssessmentUseArgument,andshowshowthedifferentstepsinthe testing process affect the outcome. A candidate’s performance on a test ismediated by the selection and the operationalization of test tasks (Bachman,2002; Sasayama, 2016;Weir, 2005). This performance is then translated into ascorebymeansofa ratingprocess,whichmay includescore transformationbymeansofstatisticalprocedures,butalwaysinvolvesaninterpretativecomponent,realizedbyhumanratersorpreprogrammedinratingalgorithms(Lumley,2002).Often, but not always, the test developer will then make a score-basedassumption concerning a candidate’s real-life ability, or link the score to anexternal frameworksuchas theCEFR.Testdevelopersdonotnecessarilymake


20

claimsaboutacandidate’sreal-lifelanguageability,however,andmayleavethematterofmakinginferencestothescoreuser(hencethedottedverticallinesinthemodel). Figure1.2.Validationscheme

Since a test score is influenced by a number of interconnected variables, testdevelopersneedtojustifyacertainamountofimplicitorexplicitclaimsrelatingtothetest’slevel,goal,orpopulation(Kane,2006;Norris,2008).Ataminimumitneedstobeclearthattheselectedtasksarerepresentativeforthetargetdomain(Weigle&Malone,2016;Weir,2005),thatsufficientmeasureshavebeentakentoreduce the influence of bias and construct irrelevant variance on test scores(Shaw&Imam,2013),thatscoresareassignedinareliableway,andthatthereisbackingfortheinferencewhichlinksascoretoreal-lifeabilityortoanexternalcriterion (see the various codes of practice: ALTE, 2001; EALTA, 2000; ILTA,2007).Throughouttheprocessofvalidation,thepurposeforwhichthetestwasdesigned–theintendeduse–needstobeclear(Kane,2001;Norris,2008).Ifthispreconditionisnotmet,therepresentativenessofthetasks,theadequacyofthecriteria, and the quality of the inference would be very difficult to assess.Determininghowfit-for-purposeatestis,seemsraterfutileifthepurposeofthetest in unclear. If a test user would decided to use a score for a purpose notintendedby thedeveloper, the scoreuserwouldneed toprovideevidence thatusing the test for an originally unintended purpose is warranted (Kane, 2013).Often,policymakersoradmissionofficersattheinstitutionaloratthenationallevelwill determine the social consequences of a test score. If this is the case,score users should assume part of the responsibility for validating claims that


21

impact real-world consequences (Kane, 2013). If an admission board acceptsdifferent scores on different tests for the same purpose (e.g., universityadmission)scoreuserswillberequiredtoprovideevidencetosupportthispolicy.

Thefinalstageofthevalidationframeworkusedinthisstudyconcernsthe decision made by score users on the basis of a score, and the socialconsequencesofthatdecision.Ifascoreuserconsiderstwotestcertificatestobeequivalent for a certain purpose, candidates who attain either one of thesecertificateswilltypicallygainaccesstothedesiredservice,position,orstatus(inthis sense, McNamara, 2012 compared language tests to shibboleths). Themagnitude of the impact of such decision often depends on the status of thescore user: For instance, the decisions made by governments of nation statesbasedon language test scoreswill typicallyhave a larger impact thatdecisionsmadebyalocalemployer.

Validation, as it is conceptualized in this dissertation, thus relies onprovidingevidenceforclaimsorassumptionsmadebytestdevelopersandscoreusers,whichhaveremainedlargelyunsubstantiated(Kane,2001,2012,2013).Theclaimsandassumptionsthatrequirevalidationrangefromtaskselectiontoscoreuse, and the burden of evidence is shared between test developers and scoreusers, depending on whomakes which implicit or explicit claim. Investigatingevery singleassumptionor claim thatunderlies a certainuseof a certain scorecould lead to a never-ending validation process. Kane (2013, 2017) thereforerecommends focusing on the assumptions that are potentially the mostproblematic,ortheleastlikely.Importantly,Kaneisratherstrictwhenitcomestoevidenceforhigh-stakesscoreuse.Themoreimpacttheinterpretationoruseofascorehasonanindividual’slife,thestrongertheevidenceshouldbe:Alltheevidencemustsupporttheassumptionsmadebytestdevelopersorscoreusers.Evidence that contradicts an assumption or a claim nullifies the validityargument.

ANOTEONPOLICY

Before introducing the research goals of this dissertation, it is important tobriefly introduce the topic of policy analysis, which will be considered moreelaborately inChapter7.Fornow it is important todefinepolicy,andtoaddadisclaimer concerning the research goals and the research design that will bedescribedbelow.

The term “policy”, as it is used in this dissertation, refers to the rulesandmeasureswrittendowninnationalandinstitutionallawsandregulationstopursueavaluedgoalorattainatarget(Grin,2003),andtoactionstakenbykeyactorsinthepolicy-makingprocess(Ball,Maguire,&Braun,2012).Thisdefinitionincorporates two levels of power identified inWilson (2006): thehighest level,


22

where policy objectives are defined – policy as text (Ball, 2015, p. 2) – and thelowerlevels,wherepolicy isenacted.It isatthis lowerlevelthatthesuccessorfailure of a policy is often decided (Wilson, 2006). In the context of thisdissertation the key actors are people who impact or implement universityentrance policy (such as civil servants, deans, educational directors, anduniversityadmissionofficers),andthemeasurestakentoimplementapolicyaretherulesstipulatedintheuniversityentrancerequirements(seeTable1.2).

Theoretical models (Lasswell's, 1956 model has been particularlyinfluential) showpolicy as a rather linear sequence of stages, startingwith thedefinition of a problem and endingwith a policy evaluation (Jann &Wegrich,2007). Empirical studieshave shown thismodel to be an idealization, however(Fischer,2007;Raaper,2016).Real-worldpolicymaking is influencedbybudgetconcerns,ideology,constraintsimposedbypre-existingpolicies,thinktanks,andthe like (Jann &Wegrich, 2007). Because policy is essentially a patchwork ofdecisions (Ball, 2015), the policy goals may be ill defined or absent, and therationaleforpolicymeasuresmaynotbespecified(Jann&Wegrich,2007).

Since admission policies of the Flemish universities did not stipulategoalsorrationales,assumptionswerededucedfromtheadmissionrequirements.Theassumptionsformulatedinthefollowingsectionsshouldthusbeseenasthestarting point for research, and as a falsifiable basis for the formulation ofresearch goals. This process of teasing out assumptions and arguments frompolicytextsforthepurposeofempiricalpolicyresearchisnotexceptional.Quitethe contrary; Fischer (2003) considers it an integral part of the policy analysts’job.

Makingassumptionsaboutassumptionsmaybehazardous,however.Assuch, policymakerswere consulted in order to explore the foundation for theadmission regulations. Chapter 7 confirms that most policy did indeed holdassumptions1,3and4tobetrue.Assumption2isnotfortestusers,butfortestdeveloperstoinvestigate.

RESEARCHGOALSTheresearchgoalsidentifiedinthisresearchprojectconcerndifferentaspectsoftheFlemishuniversityentrancepolicy.The first fourgoals focusonempiricallyexaminingassumptionsthatarepresentinthepolicytexts,thefifthgoalisaboutdeterminingwhetheranassumption,heldbypolicymakers, canbeempiricallyverified.Researchgoal6concernsthefoundationoftheadmissionpolicyitself.


23

Researchgoals1–4:ExaminingassumptionsBasedontheuniversityentrancerequirementsandontheassertionsmadebytheSTRT and ITNA test developers, four major assumptions (A1 – A4) can beidentifiedonwhichtheuniversityentrancepolicytowardsinternationalstudentsrests.Eachassumptionisintroducedhere,andwillbeexaminedindetailinthisdissertation. These assumptions were originally deduced from the admissionrequirements, but as Chapter 7 shows, they represent commonly held policymakerbeliefs.ConstructsandlevelsThefirsttwoassumptionsconcerntheadequacyofthetargetlanguagelevel(B2)andthelanguagetestconstructs.TheyareinvestigatedinChapter1andChapter2ofthisdissertation.A1 B2 isanadequatethreshold level todecideon internationalL2students’

accesstoaDutch-mediumuniversityinFlanders.Thefirstassumptionisofasomewhatdifferentorderthanthesubsequentones,since theB2demand iscentral toeveryuniversityentrancepolicy.While someuniversitiesrequireinternationalL2studentsofliteratureandlinguisticstopassaC1test(e.g.,UniversityofAntwerp),FlemishuniversitieshaveB2asacommonentrancerequirement.

“Adequatethresholdleveloflanguageproficiency”,here,isconsideredasthe point below which the language level of L2 students will likely preventsuccessfulparticipationinacademicstudiesatuniversity,butabovewhichsuchparticipationmay be possible. This conceptualization of an entrance level as aminimally acceptable cut off point relies on McNamara & Ryan (2011) and onCarlsen, (2017), who distinguishes between strong and weak interpretation ofentrancerequirements.Theformerimpliesthatstudentswhopasstheentrancerequirements are expected to succeed in the target setting, while the latterdenotesthatstudentswhodonotmeettheentrancerequirements,areexpectedtobeunsuccessfulinthetargetsetting.

If policy makers did not believe that the B2 level was an adequatethreshold level for university entrance, they would either knowingly admitstudentswhoarelikelytostrugglewiththereal-lifelanguagedemands,ordenyentrance tostudentswhomeet thosedemands.Sincebothoptionsareunlikelyand morally dubious, we hypothesized that, like in many other countries (Xi,Bridgeman,&Wendler,2013),policymakersconsidertheB2levelanacceptableuniversity entrance level. Some policy texts explicitly state that the B2 levelindicatessufficientknowledgeofDutchasamediumofinstruction(Universiteit


24

Antwerpen,2016,p.3;VUB,2014,p.22).Sincenoentrancepolicydifferentiatesbetweendifferentskills,wealsohypothesizedthatuniversityadmissionofficersconsiderB2anacceptableentrancelevelforreceptiveandproductiveskills.A2 STRT and ITNA are representative for the academic language

requirementsatFlemishuniversities.

Following the structure of Figure 1.2, verifying the second assumption is theresponsibility of test developers, as long as the test is used for a purposepromoted by the developer (Kane, 2013; but also see Norris, 2008 for theimportanceofaclearlystatedtestgoal).STRTismarketedasaB2testdesignedforpeoplewhointendtopursuehighereducationataDutch-mediuminstitution(CNaVT,2016a).Assuch,theSTRTdevelopersclaimthatthetestcanbeusedtoassesstheDutchlanguageproficiencyoflearnerswhointenttoattendaDutch-mediumuniversityoruniversitycollegeprograminFlandersortheNetherlands.

ITNA is used as an achievement test at the end of an L2 learningtrajectoryofferedat the languagecenters thatdevelop it,but the ITNAwebsiteexplicitly refers to the test’s use as a university entrance test (ITNA, 2016).Actually,morethan70%oftheITNAcandidatestakethetestforthepurposeofuniversity admission (Interuniversitair Testing Consortium, 2015), andinternationaluniversitystaffmemberswhoneedtomeettheB2requirementforDutchare required topass ITNAaswell (DepartementOnderwijs enVorming,2015).Moreover,theIUTC,theconsortiumdevelopingITNA,mentionsthefirstyear of higher education as the target language use context (InteruniversitairTestingConsortium,2015,p.12).

Sincelanguagetestdevelopersareresponsibleforvalidatingtheirtestsin contexts they explicitly promote or knowingly allow (Kane, 2013), and sincebothSTRTandITNAadvertisetheuseoftheirtestsforthepurposeofuniversityentrance, both test developers can be held accountable for validity claimspertaining to task selection, rating, and inferencemade in this context (Kane,2013;Weigle&Malone,2016;Weir,2005).BothSTRTandITNApromotetheuseof their tests formore than one purpose. This dissertation focuses on the onepurposetheyshare:admissiontouniversityinFlanders.Selection&DiscriminationAssumption 3 and 4 are concerned with how consistently the universityadmissionpolicyensurestheadmittanceofstudentswhohaveB2proficiency.A3 STRT and ITNA can be considered equivalent measures of B2 Dutch

languageproficiency.


25

BothSTRTandITNAclaima linktotheB2 level,andbothtests includeratingscalesthatarebasedonCEFRleveldescriptors.Likewise,allFlemishuniversitiesacceptboth STRTand ITNAasmeasures ofB2 ability (UniversiteitGent, 2016;KULeuven, 2016;UniversiteitAntwerpen, 2016;UniversiteitHasselt, 2016;VrijeUniversiteitBrussel,2014).Certificatesofbothtestshavethesamelegalvalueinthe admission process. Empirically investigating Assumption 3 is the focus ofChapter3andChapter4.A4 Students with a Flemish high school degree have obtained Dutch

languageproficiencyatB2level.The Flemish decree regarding the language regulations in higher education isactually more lenient than this assumption. Article II.193 (Vlaamse Regering,2013)statesthateverybodywithproofofsuccessfulcompletionofanyoneyearinDutch-mediumsecondaryeducationshouldbeallowedtoregisterwithouttakinga languagetest,also if theyhavenotattainedthe finaldiploma(theywould, inthatcase,berequiredtoshowadifferentdiplomaprovingthattheyfinishedhighschool somewhereelse). Inpractice this situationquite rarelyoccurs, except inthecaseofchildrenofexpats.Assumption4isthusmorecarefulthantheactualregulationsforL2studentsare.ItisbasedonthemostbasicprincipleoftheFlemishuniversityentrancepolicy,namely that there are no obligatory or binding university entrance tests forstudentswithadegreefromaFlemishhighschool.Chapter5examineshowsafeitistoassumethatallthesestudentshaveachievedtheB2level.

The requirement on which Assumption 4 relies may include otherassumptions(e.g.,regardingcontentknowledge),butthosearebeyondthescopeofthisdissertation.

In order to investigate to what extent these four assumptions aresupportedbyempiricaldata,fourresearchgoalswereidentified:1. Examine the empirical support for the B2 level as an entrance

requirement;2. Compare real-life language requirements at Flemish universities to

STRTandITNAoperationalizations;3. Empirically establish to what extent STRT and ITNA scores can be

consideredequivalent;4. Determine whether all students who enter university with a Flemish

highschooldegreepasstheB2threshold.


26

Researchgoal5:Measuringandexplainingpost-testlanguagegainsPart 1 of this dissertation discusses research conducted to assess the first twoassumptionsdescribedabove.Chapter 1 showsthatmanyprofessional languagetesters consider B2 an acceptable entrance level, but expect international L2studentstomakelanguagegainswhilestudying.Similarly,theresultspresentedin Chapter 2 shows that students with a B2 level struggle with the real-lifelinguistic demands at university. Chapters 2 and 7 indicate that policymakersassume that international students’ language proficiency level will increasebecausetheyliveandstudyinaDutch-mediumcontext.

If the B2 level is considered an entrance level, and if international L2studentsareexpectedtomakelanguagegainsovertime,itisworthdeterminingwhether thesegains are actuallymade.Consequently, the fifth researchgoalofthisstudyis:5. LongitudinallytracklanguagegainsmadebyinternationalL2students

who have passed STRT, ITNA or both, and explain these gains byanalyzingcontextualfactors.

Research goal 6: Tracing the origins and mechanisms of the FlemishuniversityadmissionpolicyEmpirical research conclusions are essential for useful policy evaluations, butthey are not the be-all and end-all. Science typically uses a different paradigmthan policy making. Academic research is premised on four basic principles:truth, autonomy, independent funding, and peer-driven quality assessment(Wollmann, 2007). These principles do not carry the same weight the highlyentangledandpoliticizeddomainofreal-worldpolicymaking,however.

Forthatreason,Chapter7givesvoicetopolicymakers,andshowshowthecurrentpolicycame tobe.Byhighlighting the roleofempiricaldata in theFlemish university admission process and by uncovering themechanisms thatimpact the policy-making process, this chapter offers essential information formakingrealisticpolicyrecommendations.6. Uncover the mechanisms and assumptions that have shaped the

Flemishuniversityadmissionpolicy.


27

AVAILABLEEVIDENCENot each of the above-mentioned research goals has remained completelyunexplored to date. Some have been the topic of research, but the results arescatteredandincomplete.Thissectionprovidesanoverviewoftheresearchthathasbeenconductedtoinvestigatethefiveresearchgoals.Researchgoal1ThefirstresearchgoalconcernstheB2requirementforinternationalL2students,whichiscentraltoalluniversityentrancepolicydocumentsandisreiteratedinthe STRT and ITNA validity arguments. Nevertheless, there is no publiclyavailableevidencetosupportthenear-universalB2requirement.PolicytextsandlegaldocumentstypicallyrefertotheB2levelas“sufficient”or“required”,butdonot offer any substantiationorproof (e.g.,Besluit van de Vlaamse Regering totcodificatie van de decretale bepalingen betreffende het hoger onderwijs, 2013;DepartementOnderwijsenVorming,2015).Researchgoal2Both STRT and ITNAhaveproduced validity arguments in order to obtain thesealofqualityawardedbytheAssociationofLanguagetestersinEurope(ALTE).ThisQMark is awarded to tests thathavebeen successfully audited.AnALTEaudit procedure involves a review based on seventeen minimum standards,including the construct, the purpose, the rating procedure, theCEFR link, andthe robustness of the statistical analyses. The audit reports themselves areconfidential,butbothSTRTandITNAhavegiveninsight intotheirpreparatorydocuments (CNaVT, 2014; Interuniversitair Testing Consortium, 2015). Thesereports and the audit outcomes show that both tests have satisfactory ratingprocedures(e.g.,STRTK=.8;ITNAK=.73)andinternalreliability(e.g.,inbothtestsCronbach’salpha< .9).BothtestsemployRaschmodelingtomonitorandcontrolthedifficultyoftheexam.BothtestdevelopersclaimthatcandidateswhoareawardedcertificationhavetheB2level.Internaldocumentsprovidedbybothtests reveal that both ITNA and STRT have been linked to the B2 level of theCEFR using the stepwise approach proposed in Figueras et al. (2005). Validityevidence pertaining to STRT has been presented in peer-reviewed journals(Deygers&VanGorp,2015)andbooks(Deygers,VanGorp,Luyten,&Joos,2013),and at conferences (Deygers, De Wachter, Van Gorp, & Joos, 2013). ResearchconcerningITNAhasbeenpresentedatconferences(e.g.,DeGeest,Steemans,&Verguts,2015;Steemans&Vlasselaers,2013).

Even though STRT and ITNA have provided quantitative dataconcerning internal reliability, rating, and the like, the evidence regarding


28

research goal 1 – representativeness for the target language use context – isscattered.STRTisspecificallydevelopedforfuturestudentsataDutch-mediuminstitutionofhighereducation,andarecentSTRTpublicationcontainstheclaimthat students who have passed the testmight well function adequately in thetargetsetting(Maes,2016).ThecurrentoperationalizationofSTRT(seeAppendix1)isbasedonaneedsanalysis,ontheCEFRdescriptorsfortheB2level,andonaliterature review focusing on language requirements in the academic domain(CNaVT, 2014). The needs analysis (Gysen&Avermaet, 2005; VanAvermaet &Gysen,2006),conductedin2000,appliedfactoranalysistoquestionnairedata,toidentify the perceivedneeds ofDutch language learners (N = 700) andDutchlanguageteachers(N=800).Theprimaryneedsintheeducationaldomainwereidentified as taking awritten and an oral exam, attending class, andwriting apaper(Gysen&Avermaet,2005,p.55).

ITNA makes no claims regarding future performance in the targetsetting, but the developers do state that most candidates take the test foruniversity entrance purposes. Consequently, the ITNA test tasks have beendesigned tomeet theneedsof thispopulation (ITNA,2016,p. 11).Additionally,thecomputertestisclaimedtoreliablyandvalidlyindicatewhetheracandidatehas achieved theB2 level (ITNA, 2016).Nevertheless, the rationale provided tosupport the use of the test tasks (see Appendix 2) does not refer to empiricalresearch, needs analyses or target context research conducted by the testdeveloper. The validity argument for the task types relies mainly on the B2descriptors themselves (e.g., the reading structure task), on test developerexperience (e.g., the word transformation tasks), or on published researchconductedinadifferentcontext(e.g.,cloze:Bachman, 1985;dictationtask:Cai,2013). No explicit rationale is given for the oral argumentation task or thepresentationtask.Researchgoal3AtallFlemishuniversitiesSTRTandITNAareacceptedasequivalentmeasuresofB2proficiency.Nevertheless,testuserswhoconsiderthemlegallyorlinguisticallyequivalenthaveprovidednoevidencetobacktheimplicitassumptionthattheB2abilitymeasured by both tests is comparable.However, a pilot study that wasconducted jointly by the STRT and ITNA test developers did investigate testequivalence. The results show strong correlations (see Cohen, 1988) for thewritten(r=.77,p<.000,N=77)andlowcorrelationsfortheoralcomponents(r=.15, ns, N = 38) (Deygers & Luyten, 2012; Van Gorp, Luyten, De Wachter, &Steemans,2014).


29

Researchgoal4TheassumptionthatstudentswhohaveobtainedaFlemishsecondaryeducationdegreewillalsomeettheB2 languagerequirements,hasnotbeenconfirmedinpublicly available research. One study (Van Houtven & Peters, 2010; VanHoutven, Peters, & El Morabit, 2010), conducted at four Flemish universitycolleges(N=176),showedthatnotallfirst-yearstudentspassedasummarytaskof the PTHO, the B2 test which preceded STRT. Secondly, DeWachter et al.(2013)showedthatthereissubstantialvariationintheL1proficiencyoffirst-yearstudents at the University of Leuven. However, the extent to which first-yearFlemishuniversitystudentsmeettheB2demandshasnotbeeninvestigated.

There are no centralized tests at the end of Flemish secondaryeducation,buttherearecommonattainmenttargetsoreducationalgoals,whichare operationalized and tested by teams of teachers or by individual teachers.TheseofficialattainmenttargetshavenotbeenlinkedtotheCEFR.Inprogramsthat typically prepare for university, educational goals are comparable to B2tasks.Insomevocationalprograms,however,theleveloftheattainmenttargetsmaybebelowB2(e.g.,OnderwijsVlaanderen,2015).Researchgoal5There is very little research into language gains made by international L2studentswhodonottakeadditionallanguagecourses.Todate,researchinthatfield has primarily focused on measuring writing gains in English-mediumcontexts (Knoch, Rouhshad, Oon, & Storch, 2015; Storch, 2009). The availableresults suggest that, international L2 students who do not receive additionallanguage classes are unlikely to make major language gains. Prior to thisdissertation,noresearchofthiskindhadbeenconductedinaFlemishsetting.Researchgoal6Todate, themechanismsofpolicymakingatFlemishuniversitieshasnotbeenthetopicofresearch.Internationallytoo,nopolicyresearchhasbeenconductedinthespecificdomainofuniversityadmission.

RESEARCHDESIGN

Gatheringandanalyzingdataconcerninglanguagegains,testscores,experiencesandopinionsrequiresavariedsetofapproacheswithinthesameoveralldesign.Mixed methods research uses both quantitative and qualitative approaches totacklethesameoverarchingresearchgoal inoneresearchproject(Davies,2010;


30

Xi,2010).Sinceeverychapterincludesadedicatedresearchmethodologysection,thisintroductiontotheresearchdesignprimarilyservestoshowhowthestudiesareconnected,ratherthantoofferdetailedinformationontheapproachesusedtoinvestigateeveryresearchgoal.Figure1.3summarizeswhichchaptersfocusonwhichresearchgoals,usingwhichdataandwhichresearchpopulation.

This dissertation includes data collected among different researchpopulations.Thepopulations,andthedataobtainedfromthem,areintroducedbelow,butmoredetailedinformationconcerningdataanddataanalysis,isgivenwhenrelevanttospecificchapters. Figure1.3.Researchdesign

ResearchpopulationsEuropeantestdevelopers

N: 30Data: StructuredinterviewsPeriod: November2014–April2015Analysis: Qualitative(Chapter1)

In order to determine common characteristics in the myriad of universityentrance policies across Europe, 30 experts representing 28 European regionswere interviewed. All respondents were professionally involved in language


31

testing, and all were members of ALTE (Association of Language Testers inEurope).TheoutcomesoftheseinterviewsarediscussedinChapter1.Flemishuniversitystaff

N: 24Data: FocusgroupPeriod: JanuaryandFebruary2014Analysis: Qualitative(Chapter2)Appendix: 5

In JanuaryandFebruary2014,24universitystaffmembers fromthe two largestFlemishuniversities(GhentUniversityandKULeuven)tookpartinsixdifferentfocusgroups inordertodelineatetheminimallyacceptableuniversityentrancelanguage levelandtoestablishwhichtasktypescanbeconsideredessential forstudentstomasteruponuniversityentrance.Chapter2showstheresultsofthesefocusgroups.InternationalL2students:L2P(Pilot)

N: 11Data: Semi-structuredinterviewsPeriod: October–December2012Analysis: Qualitative(Chapter2)Appendix: 3

These participants were international L2 students who had enrolled at GhentUniversityafterpassingSTRTorITNA.Theywereinterviewedatthestartandattheendof their firstsemesteratGhentUniversity,aspartofapilotstudy.Theanalyses of these interviews – included in Chapter 2 – offered informationconcerningthereal-lifelanguagedemandsatFlemishuniversities.InternationalL2students:L2F(tookSTRTinFlanders)

N: 135Data: Semi-structuredinterviews

STRT&ITNAscoresSTRTtest/retestscoresAcademicscoretranscripts

Period: June2014–May2015Analysis: MixedMethods(Chapters2and6)

Quantitative(Chapters3,4,5)Appendix: 4


32

During the summer of 2014, 135 L2 International L2 students who planned toenrollatamajorFlemishuniversity(GhentUniversity,KULeuven,UniversityofAntwerp) took STRT and ITNAwithin the sameweek. These respondents hadregisteredfor ITNAandhadagreedtoalsotakeSTRTfreeofcharge.Their testscores were used in analyses in all chapters of this dissertation, except forChapters1and7.Twentyrespondentswithinthispopulationagreedtobepartofa longitudinalstudythattracedthelinguistic,academicandsocialhurdlestheyencounteredduringtheirfirstyearatuniversity.Thelongitudinaldatagatheredfrom these twenty respondents, are discussed in Chapters 2 and 6 (which alsoincludesSTRTretestdata).InternationalL2students:L2I(tookSTRTattheirhomeuniversity)

N: 526Data: STRTtestscoresPeriod: May2015Analysis: Quantitative(Chapter5)

ThescoresofSTRTcandidateswhohadstudiedDutchattheirhomeuniversityand taken the test there were used to compare the pass probability and scoreprofilesofinternationalL2studentstoFlemishstudents.FlemishStudents

N: 159Data: STRTwritingscoresPeriod: October2015Analysis: Quantitative(Chapter5)

In order to determine whether incoming students with a Flemish high schooldegreehaveB2 languageproficiency, 159 first-year students of business studieswithaFlemishhighschoolbackgroundtooktwoSTRTwritingtasks.InChapter5theirscoresarecomparedtotheL2FandL2ISTRTwritingscores.FlemishPolicymakers

N: 15Data: Semi-structuredinterviewsPeriod: December2016–February2017Analysis: Qualitative(Chapter7)

Thecivilservantsresponsiblefortheuniversityentrancelanguageregulationsatgovernmental levelandthepolicymakersresponsiblefortheadmissioncriteriaat the five Flemish universities were recruited in order to examine how the


33

Flemish university admission policy takes shape. The results of this study areexaminedinChapter7.MethodologyAllqualitativedata,andallqualitativedataanalysesweredouble-checked.Everyinterview was audio recorded and transcribed by trained transcribers. Alltranscripts were checked by the researcher, andwere partly double coded. Allcoding was done using NVivo 11 For Mac, except the double coding for theinterviews with European test developers, which was manual. Table 1.3 showsthatthelevelofinter-coderagreementwasconsistentlyhigh.Table1.3.Doublecodinginter-rateragreement Doublecoded AgreementEuropeantestdeveloperinterviews 30% 86%Flemishuniversitystaff 20% 90%L2Finterviews 20% 90%In order to check the accuracy of the transcribers, three randomly selectedinterviews were fully transcribed. Prior to calculating the similarity of thesamples, the transcripts were preprocessed. First, transcripts of interviewerspeechandparticipantspeechwereseparated,sinceitwasassumedthataccentfamiliaritymightimpacttheoverlapinthetranscripts.Theoverlapbetweenthetranscription pairs was checked using the two indices used in informationretrievalmethodology.BoththeJaccardindexJandtheSørensen–DiceindexQSare commonly used and accepted indicators of similarity between two samples(Gomaa&Fahmy,2013).Inbothcases,twoidenticaldatasetsyieldanindexof1andtwoutterlydissimilarsetswouldresultinanindexof0.Theresultsshowaverylargedegreeofoverlapbetweenthetwosetsofinterviewer(J=.9;QS=.95)andintervieweetranscripts(J=.77;QS=.87).

Allquantitativedata–testscores, languagegainindices–wereanalyzedusing R (descriptive and inferential statistics), Facets (Multi-Faceted Raschanalysis),andPython(textpre-processingandtextsimilarityindices).WithintheR environment, the following packages were used: car (simple and multipleregression),exacti(McNemar’stest),ggplot2(dataplotting),Hmisc(correlation),irr (inter-rater reliability), MASS (principal component analysis), Mlogit(multinomialregression),Pastecs(dataplotting),Pgirmess(Kruskal-Wallistest),Plyr (summary statistics), prob (pass probability), psych (descriptive statistics,examining normality), QuantPsyc (regression). The specific methods used toanalyzethedatawillbeexplainedthroughoutthedissertation,whenrelevanttotheresearchgoalathand.


34

STRUCTUREANDRELEVANCEOFTHISDISSERTATIONThisdissertationaddressesseveralgapsintheexistingliterature.Itexaminestheimpact of policy measures, and determines whether these measures have thedesired effect. Policy impact research of this kind has been performed in thecontextofpublic(Nagel,2002),orenvironmentalpolicy(EEA,2001),andeveninthecontextofK12schoolpolicy(Ball,Maguire,&Braun,2012;Grin,2000,2003),butuniversityentrancepolicieshavetypicallynotbeenthesubjectoflarge-scale,triangulated studies (McNamara&Ryan, 2011). Similarly, the idea of justice, asrelated to high-stakes language testing for admission purposes, has remainedlargelyunderexplored(Khan&McNamara,2017),andthereisnocleardefinitionofwhatajustadmissiontestingpolicymightlooklike.Thisdissertationoffersawayforwardregardingbothissues.

The chapters following the introduction are based on research paperswhichhavebeenpublishedoracceptedbyinternationalpeer-reviewedjournals,andwhich have been edited to fit the format of this dissertation and to avoidredundancy. Each chapter focuses on one aspect of the Flemish universityentrance policy, but also has wider implications for the language testing orappliedlinguisticscommunity.Partone:Constructs&levelsThe first chapter isbasedon structured interviewswith 30 expert respondents.TheresultsshowthattheCEFRisomnipresentinEuropeanuniversityentrancelanguagetestsandthattheB2isthemostcommonlyusedlevelinthatcontext,butoftenwithoutempiricalsupport.Prior to thisstudy,noresearchhadeitherexaminedthecommunalitiesbetweendifferentEuropeanpolicies,ortheimpactof the CEFR on these policies. Chapter 1 frames the dissertation in a widercontext.Asofchapter2thefocusnarrowstoFlemishpolicy.

Chapter 2 brings together the opinions and experiences of 24 universitystaffmembers and31 internationalL2 students (L2P andL2F).Theoutcomesofthis study reveal that the actual receptive language requirements of universitystudiesmost likelyexceedtheexpectedB2 level,andthattheFlemishentrancetests include language tasks that are of little importance in the real-lifeuniversity-based studies of first-year students. By having a group of candidatestake both STRT and ITNA, it was possible to track the progress in the targetcontextofpeoplewhohadactually failedoneof the twoentrance tests.This isquite possibly the first study to partially bypass the truncated sample problem(Alderson, Clapham, & Wall, 1995) by using this approach. Drawing on theobservation that a number of students who failed the entrance test actuallymanaged quitewell at university, this chapter also discussesmatters of justicerelatedtoadmissiontesting.


35

Parttwo:PerformanceandequivalenceThe third chapter examines the implicit claim that STRT and ITNA can beconsidered equivalent measures of the same level of language proficiency.Additionally, this chapter aims to explain not only the strength but also thenatureof therelationshipbetweenthetwotests.Tothebestofmyknowledge,nostudieshaveyetbeenpublished inwhich researchershadaccess todetailedraterdatafromtwodifferent,high-stakesentrancetestsinordertocompareleveland construct equivalence.The resultsdiscussed inChapter 3 reveal importantdiscrepanciesbetweenSTRTandITNA.

Thefollowingchapterfocusesontheequivalenceofcorrespondingratingcriteria.STRTandITNAusethesamecriteriatoassessperformancesontheoralargumentationtasksandpresentationtasks.The leveldescriptorsusedtoscorethese criteria are based on the same CEFR descriptors. The results show,however, that the scores – assigned to the same candidates performing nearlyidentical tasks – deviate significantly. Prior to this research no study had yetassessedtheequivalenceofCEFR-basedratingdescriptorsacrossdifferenttests.

Chapter 5 investigateswhether all Flemish candidateshave aB2-level inDutch upon university entrance, and whether L1 test takers outperform L2candidates who learnedDutch at home or in Flanders. The results show that,eventhoughtheFlemishgroupoutperformedbothgroupsofL2candidates,notallFlemishcandidatesreachedtheB2level.Todate,nostudyhadcomparedL1andL2performanceonacentralizedhigh-stakesB2universityentrancetest.Partthree:Gains&contextThesixthchaptertellsthestoryoftwentyinternationalL2students(L2F)duringtheir first yearat threeuniversities inFlanders,Belgium.The results show thatafter eight months at university, the respondents had only made progress interms of written fluency. No other gains in the oral or writtenmodality wereobserved.Thequalitativedatausedtoexplainthelanguagegainsshowthattherespondents experienced little institutional support, and that few respondentshad gained access to the L1 academic community. Longitudinal studies thatfocusedexplicitlyonlanguagegainsbyinternationalL2studentswhoreceivednoadditional language support, are very limited in number, and have so far onlyconsideredwritinggains.Withinthiscontext,chaptersixpresentsthefirststudyto measure speaking gains. No existing studies have considered personal andacademicexperiencesofL2learnerstoexplainoralandwrittenlanguagegains.


36

Partfour:policy,conclusion&implicationsThe final component of this dissertation includes a discussion of how theuniversityadmissionpolicyismade.Chapter6showshowstakeholders’interestshave impacted regulations and shows that empirical data have not had ademonstrableinfluenceonthepolicymeasuresinplace.Theseobservationsareconnected to the conclusions drawn from the empirical research to formulateimplicationsandrecommendationsbasedonFischer’smodelofpolicyevaluationin the final Chapter. This triangulated and multifaceted approach to policyevaluationisnewtothefieldoflanguageassessment.

ANOTEONCOMPOSITIONThefirstsixchaptersofthisdissertationarebasedonresearchpapersthathavebeensubmittedto,acceptedorpublishedbypeerreviewedjournalsorbooks.Inorder to avoid unnecessary repetition across chapters, certain sections of theoriginal papers have been omitted from the chapters and are included in thisintroduction. The sections in this introduction dealing with the context ofresearch,withtheapproachtovalidation,orwiththeresearchpopulationhavebeen adapted from these research papers (Deygers, 2017; Deygers, Van denBranden,&Peters,2017;Deygers,VandenBranden,&VanGorp,2017;Deygers,VanGorp,&Demeester,2017).

Part1:Levels&constructs

38

PART1LEVELS&CONSTRUCTS

ThefirstpartofthisdissertationinvestigatestwoaspectsoftheFlemishuniversity entrance policy: required language level, and therepresentativenessofthemaingatekeepingtestsforthetargetcontext.

InordertosituatetheFlemishuniversityentrancepolicyinawidercontext,thefirstchapteroffersanoverviewofthelanguagerequirementsforinternationalL2students throughout Europe. The data show that inmany European countriesinternational L2 students are required to prove language proficiency at the B2level. At the same time, however, the B2 requirement seems to have becomerather pervasive without much solid empirical evidence. In most Europeancontextssurveyed–Flanders included–therewaslittleornopubliclyavailableempiricalevidencetosupporttherequireduniversityentrancelevel.InthesecondchapterthefocusshiftstoFlanders.ChaptertwoshowsthattheB2requirement, especially for receptive skills, does not correspond to the actuallanguage demands at university. Additionally, the data indicate that the twomajor university entrance tests used in Flanders, STRT and ITNA, mayoperationalizeorprioritize language tasks thatarenotnecessarily important inthetargetsetting,orarenotyetexpectedofincomingstudents.DrawingontheobservationthatL2studentsareassessedonthebasisoftasksthattheirL1peersare not expected to perform, this chapter also reflects onmatters of justice inuniversityentrancetesting.Chapters1and2arebasedon:

Deygers, B., Zeidler, D., Vilcu, D., & Carlsen C.H. (2017, in press). Oneframeworktounitethemall?TheuseoftheCEFRinEuropeanuniversityentrancepolicies.LanguageAssessmentQuarterly.

Deygers,B.,VandenBranden,K.,VanGorp,K.(2017,inpress).Universityentrancelanguagetests:amatterofjustice.LanguageTesting.

The papers have been formatted to fit the structure of this book. Certain components (e.g., methodology, participants) have been rewritten to avoid redundancy and benefit readability.


40

CHAPTER1UNIVERSITYADMISSIONPOLICIESACROSSEUROPEThe aimof this chapter is to investigate howwidespread the use of theCEFR, and more specifically the B2 level, is in European universityentrancetesting.Assuch,Chapter1framestheFlemishpolicyinalargercontext,andshowsthatthelackofpubliclyavailableevidencetosupportanadmissionpolicyisnotuniquetoFlanders.

The goal of the Common European Framework of Reference for Languages(CEFR,CouncilofEurope,2001)istopromotethefreemovementofpeopleandideas by increasing the transparency across educational systems through thecommonuseofthesameproficiencylevels(VanEk,1975).ThefirstdraftsofwhatwastobecometheCEFRappearedintheearly1970swiththedevelopmentoftheThresholdlevel,latercalledB1(VanEk,1975).Throughtheyearsnewlevelswereadded(VanEk&Trim,1991b;VanEk&Trim,2001),existinglevelswererefined(VanEk&Trim, 1991a), and some thirty years after the firstdrafts, theprojectculminated in the “blue book” we know today. The CEFR proposes sixconsecutive levels of language proficiency, ranging from A1 through to C2. Itfocuses on what learners putatively can do with language and includes 53illustrative scales, which list language-independent descriptions of eachproficiencylevelforagivenskillorability.Arguably,theseillustrativescaleshavebecome themost influential, but also themost heavily criticized aspect of theCEFR(Little,2007;Figueras,2012).

CEFR criticism is usually either political in nature or content-related(Figueras,2012).ThepoliticallyorientedcriticismseestheCEFRasaninstrumentof power that encourages a simplistic, level-driven logic at the expense of aneeds-based,user-drivenpolicy(McNamara,2007;Shohamy,2011).Accordingtothis strandof criticism theCEFR levels areoftenusedas anormative standardthatprovidespolicymakerswithaneasy tool toassist ingatekeeping (Fulcher,2004; 2012). In a defense against this criticism, North warns against confusingintended CEFR use with actual but inappropriate use (North, 2014a), andconsidersanynormativeuseoftheCEFRtofallunderthiscategory(Martyniuk,2010; North, 2014a). Furthermore, so the argumentation goes, the CEFRdiscourages a simplistic level-based gatekeeping policy, because it stimulatesdiscussionbetweendecisionmakersandlanguageexperts(Porto,2012).Content-relatedcriticismontheotherhandfocusesonthoseaspectsoftheCEFRthatareproblematic in the context of language testing – the field where the CEFR’sinfluence has been most keenly felt (Little, 2007). Some authors tackle thefoundationoftheCEFR,pointingoutthatitlacksempiricalvalidation(Fulcher,2012b),ignoresinsightsfromsecondlanguageacquisitiontheory(Alderson,2007;


41

Little, 2007; Hulstijn, 2007), does not pay equal attention to all skills (Weir,2005b; Alderson et al., 2006; Staehr, 2008) or does not cover the full range oflevelsineveryscale(Aldersonetal.,2006).Itisnowcommonlyrecognizedthatthe theoretical support for the descriptors of the receptive skills is not quiterobust(Alderson,2007;North,2014a),butagrowingbodyofresearchhasbeenproviding language-specific empirical validation for the CEFR (Salamoura &Saville,2009;Carlsen,2014).AsecondstrandofcriticismrelatedtothecontentfocusesonthedeficienciesoftheCEFRasacommonreferencepointinlanguagetesting.Thislineofcriticismmaintainsthatthelevelsarenotequidistantastheycontain overlaps and inconsistencies, and that the descriptors are general,language-independent, containing impressionistic terminology (Alderson et al.,2006;Alderson,2007;Papageorgiou,2010,Fulcher,2012).

ThisfiercecriticismtowardstheCEFRmayseemsomewhatoverstatedincomparisonwiththedocument’sintendeduse,whichappearsrelativelymodest.TheauthorshaverepeatedlypointedoutthattheCEFRdescriptorsweremeanttobegeneralandthatthelevelswerenevermeanttomakeupanintervalscale,so in part the content-related criticism points out a characteristic that waspurposefully built in to theCEFR (Little, 2007;North, 2014a). In their defense,CEFR authors also state that in spite of the general nature of the CEFRdescriptors,professionals inorganizations suchasALTEandEALTA(EuropeanAssociationforLanguageTestingandAssessment)interprettheCEFRlevelsinauniform fashion (North, 2014a). When it comes to the use of the CEFR, itsauthorsinsistthatitwasnotmeantasanormativedocument,butasamalleableheuristicthatstimulatesreflection,facilitatesdiscussionaboutlanguagelearning,and aids curriculum planning and language certification (North, 2007; 2014a;2014b).TheactualuseoftheCEFRhasnotalwaysbeenintunewithitsintendeduse however (North, 2014a), and many language tests across Europe haveexperiencedpressurefromstakeholderstolinktheirscoringsystemtotheCEFR(Fulcher, 2004), or have been redeveloped with the CEFR descriptors inmind(Galaczi, ffrench, Hubbard, & Green, 2011). Also, as many countries andinstitutions in the EU now use CEFR levels to set legally binding citizenshiprequirements (Van Avermaet & Pulinx, 2013), curriculum goals (University ofCambridge, ESOL Examinations, 2011) and university entrance demands (Xi,Bridgeman,andWendler,2014),languagetestersinEuropehavebeenrequiredtofollowsuit. Fifteenyearsafter itspublication, theCEFRhas inspireda largebodyofliterature, and has gained critics and champions. It has been praised forfacilitating dialogue (North, 2014a) and denounced for causing validity chaos(Fulcher,2012b).Ithasbeenstudiedinthecontextofratingscaledesign(Harsch&Martin,2012),andithasbeenthesubjectofsociolinguisticdebates(Roever&McNamara,2006). Ithasbeenadoptedbyusers inothercontextsandonothercontinents (Negishi, Takada,&Tono, 2013), and has been accused of linguistic


42

imperialism for it (Curtis, 2015). TheCEFRhas led to a lot of language testingresearch, but its actual impact on the lives of test takers has remained largelyunexplored.Still,becauseofitseffectonlanguagetests,ithaspotentiallyaffectedthe livesofmillions.Thestudypresented inthischapter focusesontheCEFR’simpactinonespecificfield:universityentrancelanguagetestingforL2studentsfromabroad.

RESEARCHQUESTIONSThereislittlereliableinformationabouttheimpactoftheCEFRonL2universityentrance languagepolicyacrossEurope,andthe informationthat isavailable isfragmented or scattered across the websites of Europe’s universities anddepartments of education. As such, the existing literature does not offer theopportunitytoframetheFlemishsituationinawidercontext.Forthatreason,anexplorativestudywassetup,withtwoguidingquestions:RQ1: What are the common features of university entrance language

requirementsforinternationalL2studentsacrossEurope?RQ2: What is the impact of the CEFR on university admission policies

anduniversityadmissionlanguagetestsinEurope?TheseresearchquestionshelptoframetheFlemishuniversityentrancelanguagepolicy.TheresultsofferinformationconcerningtheempiricalgroundforusingtheB2levelasathresholdforuniversityadmission.Assuch,thedatapresentedinthischapterarerelevanttoAssumption1(B2isanadequatethresholdleveltodecideoninternationalL2students’accesstoaDutch-mediumuniversityinFlanders).

PARTICIPANTS&METHODOLOGYRespondentsThe information required for this study is only knownby relatively fewpeoplewhoworkinthecontextofuniversityadmissionoruniversityentrancelanguagetesting.Consequently,collectingdataviarandomizedsamplingwasof littleuseand respondents were chosen through purposeful selection (Freeman, 2000),which implies identifying knowledgeable and information-rich respondents(Reybold, Lammert, & Stribling, 2013). All respondents were professionallyinvolvedinlanguagetesting,andinmanycasesinthedevelopmentoflanguage


43

tests for university entrance. The researchers contactedmembers of Europeanlanguage testing organizations that are full members or affiliates of theAssociation of Language Testers in Europe (ALTE). Thirty-nine organizationswere asked to take part in the study, and in the end representatives of 30organizations operating within 28 states or regions with autonomy overeducationalmatters agreed toparticipate.Although testoperationalizationwasnot a selection criterion, all university entrance language tests included in thisstudy (N = 25) are skill-based, seven of which also feature a grammar andvocabularysection.ScopeFor reasons of readability we will use the term “context” throughout the textwhenreferringtobothnationstatesandregionsthathave(quasi)autonomyovereducational matters. Consequently, even though this study covers 26 nationstates,thischapterreportson28contexts.Table2.1.CountriesandregionssurveyedR1 Austria R17 LuxembourgR2 Belgium(Flanders) R18 MaltaR3 Belgium(Wallonia) R19 NetherlandsR4 Bulgaria R20 NorwayR5 CzechRepublic R21 PolandR6 Denmark R22 PortugalR7 Estonia R23 RomaniaR8 Finland R24 SloveniaR9 France R25 Spain(Basque)R10 Germany R26 SpainR11 Germany R27 SwedenR12 Greece R28 SwitzerlandR13 Hungary R29 UnitedKingdomR14 Ireland R30 UnitedKingdomR15 Italy R16 Lithuania Becauseof the independenteducationalpolicies inWallonia,Flanders (both inBelgium)andtheBasqueregion(Spain), theseregionswereconsidereddistinctcontexts. In other countries, such as Switzerland, there are different linguisticregions, but the policy and the entrance tests in these regions share the samevisionandcharacteristics,andassuchtheyarecountedasonecontext.Insomecasesthereweretworespondentspercontextbecausetherewasmorethanonefullmember of ALTE for that country, andmore than onememberwished to


44

participate(R10andR11,R29andR30).Thesedoublesallowedtheresearcherstocomparetheanswerstothefactualquestions.Inbothcasesthesameinformationwasgivenbybothrespondents.Table2.1 liststhecodesoftherespondentsandthecountriessurveyedinthisstudy.DatacollectionandanalysisSince it was considered important to collect data that shed light on eachindividual context, but also allowed for meaningful comparison, structuredinterviewswereused.Respondentswereaskedafixedsetofquestions,buttheywere free to contextualize their answers in a way that questionnaires cannotaccommodate (Schwartz, Knäuper, Oyersman, & Stich, 2008). The interviewscenario consisted of three parts: (1) one concerning the university entrancepolicy, (2) one concerning the entrance tests, and (3) one concerning theinterviewee’spersonalopinionsabouttheCEFRanduniversityentrancelanguagetests(not includedinthischapter,seeDeygers,Zeidler,Vilcu,&Carlsen,2017).Respondentsreceivedthefactualquestionsthreedaysbeforehandviae-mailandwere encouraged to look up any information they did not have on hand. Thescenario was trialed twice in interview conditions, and the interviews wereconducted by four trained researchers via Skype and were recorded using theAudacitysoftware.

Allinterviewsweretranscribedverbatim.Thetranscriptionswerecheckedbytworesearcherswhoreplayedtheinterview,correctedanyinaccuracies,codedthe transcripts independently, and compared codes and outcomes afterwards.Codingwas done bothmanually and using the qualitative softwareNVivo ForMac, since a combination ofmanual and computer-assisted coding is likely toyield the most reliable results (Welsh, 2002). Both researchers used a set ofagreed-upon a priori codes that correspondedwith the questions asked in thestructured scenario. After the first round of coding, the exact inter-rateragreement was checked for the a priori coding categories (86.4%), based on arandomsamplerepresenting30%of thetotal transcribedtext.Bothcodersalsoemployed a grounded theory approach (Glaser & Strauss, 1967; Miles &Huberman,1994),whichmeansthattheyalsocodedsalientissuesthatemergedfromthedata.Thisdoubleapproachallowedtheresearcherstocomparefactualdataacrosscontexts,butalsotospotopinionsorviewsthatsurfacedduringtheinterviewswithout being explicitly probed for.After coding independently, theresearchers discussed their coding categories over several meetings untilconsensus was reached. The complexity of the coded data was reduced byquantifying recurring practices and patterns (Ziegler & Kang, 2016). Thesequantificationsarepresentedinthetablesbelow.


45

AllinterviewswereconductedinEnglishandthequotesincludedinthisarticleareliteraltranscriptionsthathavebeenlightlyeditedforthesakeofreadability.Editing was restricted to correcting grammatical flaws, and omitting wordrepetitionsandfilledpauses.Throughoutthechapter,“I”willbeusedtorefertothe interviewer, and respondents will be referred to by their code (i.e., R1).Misinterpretationofwhatwas saidduring the interview is alwayspossible. Forthatreason,thetranscriptionsweresentbacktotherespondentssotheycouldcommentonanyfactualflaws.Whenthishappened,theinformationwasusedinthedataanalysis,buttheoriginaltranscriptswerenotaltered.Asafinalstepintheverificationoftheinterpretationofthedata,therespondents,aswellasthemembersofALTE’sCEFRspecial interestgroupreceivedaprefinaldraftof thischapter,whichtheycouldamend.Norespondentrequestedanyrevisions.

RESULTSIn23contexts,passingalanguagetestismandatoryforuniversityentranceforL2students. In three of these countries, only one centralized test is accepted forfirst-year university entrance, but the twenty other contexts all have a systemwhere multiple tests are used for the same high-stakes purpose. In thirteencontexts both centralized tests and local tests – developed by the receivinguniversityitself–areaccepted(seeTable2.2).Table2.2.Testsacceptedforuniversityentry(N=28) NumberofcontextsMultiplecentralizedandlocaltests 13Multiplecentralizedtests 5Onecentralizedtest 3Multiplelocaltests 2Notest 5The respondents did not generally regard local tests positively and expressconcernovertheirquality, transparencyandcomparability.Outofthefourteenrespondents from contexts where both centralized and local tests can grantuniversityaccess,eightrespondentswishedtostreamlinetheuniversityentrancesystem. R10 [Thelocallydevelopedtests]don’tapplypiloting,pretestingandstatistical

analysis, so it’snot astonishing that the resultswereveryheterogeneousandtheexamsareofadifferentlevelofdifficulty.


46

In 22 contexts there is a university entrance language requirement that isexpressed inCEFR terms (seeTable 2.3). In five contexts (Finland,Luxemburg,PortugalandSpain, includingtheBasqueregion)therearenospecific languagerequirements,andinSwedentherequirementsarenotCEFR-related.Table2.3.CEFRlevelrequiredforuniversityentrance(N=28) NumberofcontextsA2andB2† 1 B1orhigher† 1 B2 9 B2+ 1 B2orC1† 8 C1orhigher† 2 RequirementnotCEFR-related 1Norequirement 5Note.†variesbyprogram ThefindingsofthestudyconfirmtheassumptionthatB2isthemostcommonlyrequired level for university entrance across Europe: in nine contexts B2 is theonlyrequiredlevel,andinanothertenit isoneoftherequiredlevels.ButeventhoughB2isthemostcommonlyusedCEFRlevelforuniversityentrance,thereisno agreement among the 30 individual respondents that B2 users have thelinguistic resources required to function at the start of academic studies atuniversity(seeTable2.4).Table2.4.IsB2enoughtofunctionlinguisticallyatthestartofuniversity?(N=30) Numberofcontexts No 7

B2+asaminimum (4) C1asaminimum (3)

NotQuite 8Ifadditionallanguagesupportisoffered (3) Dependsonstudentneeds (5)

Yes 10

B2isenough (3) B2istheabsoluteminimum (7)

Don’tknow 5The respondents may doubt the B2 level as an adequate level for universityentrance,buttestdevelopersarerarelytheoneswhomakethedecisiononwhereto set the entrance level requirements. In sixteen contexts (see Table 2.5) the


47

decision on language level requirements for university entrance is partly orcompletelyup touniversityor faculty staff. In fouroutof sixteen instances theministrydecidesonageneralrule,buttheuniversitydeterminestheactuallevel,andinthetwelveremainingcasesthelevelrequirementsarelefttotheuniversityentirely. In seven contexts there is a national regulation stipulating the levelrequirements.Table2.5.Whodecidesonlanguagerequirements(N=28) NumberofcontextsNorequirement 5Government(Generalrule)&university(specifics) 4Governmentorministryofeducation 7University,facultyordepartment 12Inmostcontextswherealanguagelevelisrequiredforuniversityentrance,thatlevelwas not determined on the basis of empirical data or needs analyses, therespondentsreport.R4 Thereisn’tanyofficialstudythatsaidB2istherightlevelforstudents.It’s

just intuitive, from the practice in universities. Knowing that othercountriesrequireB2theyjustdecidedtointroduceB2.

R24 [Theuniversity] askedus about our opinion, andwe said, inEurope it’s

mostlyB2.Andtheysaid,okay,itshouldbeB2.

R21 Idon’tthinkit’sbasedonanyempiricalground.It’sjustanadministrativedecision […] Those in charge of making decisions start with the levels.Then the whole field is trying to adjust to the decisions rather thanstarting with an analysis of the needs […] When the process has beenreversed,itisverydifficulttoturnitaround.

In the 23 contexts where L2 university entrance is regulated by language testslinkedtotheCEFR,languagerequirementsrarelyseemtobebasedonempiricaldata (see Table 2.6). Only one respondent claimed that the requirements inhis/hercontextwereempiricallyfounded.


48

Table2.6.Empiricalresearchtosupportrequiredlanguagelevel(N=23) Numberofcontexts NoEmpiricalfoundation 19 Followingotherinstitutions (6) Literaturestudy (6) Unknown (6) CEFR (3) Partlyempiricallyfounded 3 Needsanalysis (2) Expertcounsel (1) Empiricallyfounded 1Thisdoesnotimplythattheteststhemselvesarenotbasedonempiricalstudies.Some tests arebasedonextensive researchandneedsanalysis,butuniversitiesare free to decide on the required entrance level themselves. In many casesuniversities do not seem to base their requirements on needs analyses, but oncommonpracticeoronwhattheircompetitorsdo.R30 Thereisempiricalresearchandtherewasresearchdonewhenthetestwas

firstbeingdesignedandwhen the testwasbeing trialed. [...]Obviously,theofficialadviceisthat[peoplefromtheacceptinginstitution]shouldsitdownandevaluatethecourses;sayforacourseofthatnature,weneedaminimum standard of whatever. Now, some universities have done thatkindofthing,butothersareessentiallyrespondingtothemarketintermsof seeing what other people are asking for. […] In the sense of actuallydoingthatformalsitdown,standardsettingsession,Ithinkonlyalimitednumber have done that. […] I would be happier if I believed thatuniversities were really doing a standard setting, rather than simplyrespondingtowhattheircompetitorsandpeersweredoing.

Inquiteafewcontexts,CEFRrequirementsdependnotonneedsanalyses,butonfinancialconsiderations.SomeuniversitiesrequireaCEFRlevelthatislowerthanwhatisrequiredinotheruniversitiesinordertoattractstudents(R30above,andR13, R15 below). In at least eight contexts surveyed it is common practice foruniversities or ministries to use CEFR levels to control the flow of incomingstudentsforfinancialreasons,toattractacertaintypeofstudents,ortocontrolaccess to the labormarket.Level requirementswouldriseand fall,notbecausethelanguageneededtoparticipateinacademiclifechanges,butbecausecertainfacultiesrequiremoreorfewerstudents.


49

R15 In some universities in the south part where the students are very few,theyprovideveryeasyentrancetests[...]whileinthenorthpartwehavemoredifficulttests.

R13 Universities can decide to lower the level […] if they desperately need

students.R4 The situation for international students is differentdependingonwhich

countrytheycomefrom.IftheyarefromtheEuropeanUniontheyarenotobligedtohaveB2.

R6 TheC1level[…]that'sthenormalthing.Butwenowhaveaspecialcase:

medical doctors and nurses in health care that were educated in theirhomeland. […] They informed us from the ministry, that they neededpeoplewithin thehealth systemand they foundout that theC1 level, itwastoodifficulttopass.

R9 Wehaveaninstitute[ofhighereducation]workingwithoilandtherewe

don’taskanylevel,becauseweneedpeopleworkingwithoil.Of the 30 respondents involved in this study, 27 developed CEFR-relatedlanguagetests.MostoftheserespondentsstatedthattheCEFRwasusedtosetordefine the level (20), and/or todrawup rating scales (4) and/or todesign taskspecifications(4).TworespondentsstatethattheirtestswerefullybasedontheCEFR,andformostrespondents,theCEFRispartoftheirdailypractice:R2 It’salwaysinthebackground.R15 EverydayIconsidertheCEFR.R28 Everythingwedoisbasedonthelevels.R30 Werefertoitallthetime.TherespondentsmentionthreemaineffectsoftheCEFRonlanguagetestinginEurope.Firstof all, they feel that theCEFRhasbrought standardizationwherethere was disharmony. Secondly, the CEFR has promoted skill-based languagetesting.Thirdly,ithasledlanguageteststoadoptanewlevelstructure,whichisnow so well established that test developers may experience pressure to alignwith it. Seventeen respondents have experienced an external pressure to alignwiththeCEFR,eitherpoliticallyoreconomically.R2 Itwasinfactthedemand[ofthefundingbody]totakeintoaccountthe

CEFR. So because of this, we took the B2 level as a starting point fordevelopingourexam.


50

R10 The CEFR levels are not precise enough, but if you want to sell an

examination or a textbook without indicating the levels, then it will beverydifficultonthemarket,sothisiswhymostoftheinstitutionsusetheCEFRlevelswhentheywanttosellsomething.

R18 TopassourB2examyouneedtobequitegood,butthereareotherexams

whicharen’tascompleteorasstrictintheirmarkingandthenyougetastudentwhoisatalowerlevelbutstillgetsaB2certificateandwouldnotmanagetopassourexam.So,market-wise,it’screatingabitofaproblemhere. [...] The product has to be marketable, you know?

ThereareestablishedproceduresforaligningtestswiththeCEFR,butnotalltestdevelopers have linked their tests to the CEFR using a robust procedure.Respondents of eleven contexts spontaneously mentioned examples ofunfoundedCEFRlinksinuniversityentrancetestsusedintheircontext.Insomecasestherewasnoprocedureatall.R12 Wehadtestsbefore,Ihaveallthepresidentialdecreeshere,andnowthe

announcementisthatitisCEFRcompatibleinalllevels[…]Sothesefourlevelsbecamesixnow,accordingtotheCEFRframework.

I Wheredoes[normativeCEFRuse]comefromyouthink?R30 Cause it’s easy I suspect. “Ooh it’s there, it’s in thebook.Yes!B2 isvery

good,oohthat’sfine,yes!”Boom,jobdone.Ithinkthere’salotofthat.

R18 It’s not the CEFR descriptors which are creating a problem, really. It’sthosewhoare interpreting. [...]Everybodyelse is issuingcertificatesataB2level,butisthereanyguaranteethatthey’reactuallyatthatlevel?

Linking a test score with an external measure such as the CEFR requires astandardthatissomehowfixedoruniformlyinterpretable.EvenifalltestslinktotheCEFR in themost thoroughways, its levels could only serve as a commoncurrencywhenthesame levels roughlymeanthesamething todifferentusers.OnlytworespondentsunequivocallyassumedthattheB2levelisoperationalizeduniformlythroughoutEurope,butotherswerenotsosure.R6 It's very hard to say. I really do hope so. I hope that all tests are testing

different strategies. I hope that we all are testing oral communication in a good way. I hope so, I hope so!


51

DISCUSSION

There is no uniform policy within Europe when it comes to languagerequirementsforforeignL2studentswhowishtostudyinanationallanguage.Inseven of the surveyed contexts there is a national regulation that specifies theofficialuniversity entrance language requirements forL2 students. In theothercasesthereiseithernolanguagerequirementwhatsoever,oreachuniversitysetsitsownlanguagelevelrequirements.Thisdiversificationisreflectedinthetestingpolicy. In twenty contexts multiple language tests are accredited for grantinguniversity access, and in most of these cases centralized tests are acceptedalongside local tests that have been developed by the accepting institutionsthemselves.Quite a few respondents doubted the quality and comparability oftheselocaltests,andifthereisonecommonwishtherespondentssharefortheL2universityentrancepolicyintheircontext,itisincreasedstandardization.

Throughout Europe universities have a lot of autonomy in setting theentrance requirements. Usually they are free to determine the languageproficiency level they require for admission, although in some contexts thegovernmentmayset somegeneral requirements. In sevencontexts,universitieshavenoautonomyindecidingontherequiredlanguagelevelandareobligedtofollow governmental decrees. Irrespective of which body is responsible fordetermining the entrance level however, the reasons for choosing a certainlanguage requirement appear to be quite unfounded. In only one of the 23contexts where university entrance is determined by CEFR-linked languagerequirements, the required level is based on an empirical study. In 22 others,those requirements are not or only partly based on an analysis of the targetlanguageusecontext.Mostoftenthelanguagelevelrequirementsaredeterminedby what other countries or universities do, or by the text of the CEFR itself.Moreover, in about one thirdof the contexts surveyed it is notuncommon forinstitutions to lower the linguistic entrance requirements to attract morestudents,or tomanagetheaccessof studentswith lessdesirableprofiles.Sincemany institutions can decide on the entrance level they require, they are bothpolicymakers and stakeholders at the same time, which leads to situations inwhicheconomicconsiderationsmayoverruleactualstudentneeds.

This study shows that the CEFR has fundamentally impacted universityentrancelanguagetestinginEurope,andthemostinfluentialaspectsoftheCEFRarethesixlevelsandtheillustrativescales.InjustoneofthesurveyedcontextstheuniversityentrancelanguagerequirementsarenotCEFR-related.ThisstudyconfirmsthatthelevelmostcommonlyusedforuniversityadmissioninEuropeisB2,eventhoughtherespondentswerenotconvincedthatB2isoperationalizedin the same way throughout Europe. Half of the respondents did not feelcomfortableconsideringB2asthedefaultstartingrequirementforuniversity.All


52

inall,however,therespondentsofthisstudyareratherpositiveabouttheCEFR,sincetheyfeelitoffersacommon–albeitsometimesvague–standardthathasimprovedtestscorecomparability.

TheresultsofthisstudyshowthatsometimesCEFRlevelsareinterpretedveryrigidlyinaprocessmirroringthereificationFulcher(2004)warnsagainst:B2isusedbecauseitisB2,notbecauseitisthelevelthatbestsuitstheusers’needs.ThisstudyofferslittlesupportfortheclaimthattheCEFRstimulatesdiscussionbetween decisionmakers and language experts (North, 2014a). No respondentmentionedtheCEFRasacatalyst forconversationsbetweenpolicymakersandtestdevelopers,andonlyonerespondentclaimsthattheuniversityentrancelevelwasbasedonaneedsanalysis.IntheshorttermthiskindofCEFRmisusecanbeconsidered unfair – since it does not offer every student equal opportunitiesacrosscontexts–orirresponsible,sinceitignoresuserneedsortargetlanguageuse demands in favor of norm-driven labeling. In the long run it could provepotentiallydestructive for theCEFR,since the levelsmight losecredibility.TheCEFRhasalwaysbeenanopensourcehermeneutic,butinmanycontextsitnowserves as a self-administered seal of quality. It can give university admissionofficersasemi-objectivetooltocontroluniversityentranceanditmayallowtestdevelopers toclaima linktoacertain levelwithouthavingtoofferanykindofproofforthis.

CONCLUSIONThe data presented in Chapter 1 indicate that linguistic university entrancerequirements in Europe do not appear to be based on robust empiricalfoundations.Similarly,theB2requirement,inspiteofitsprevalence,commonlylacks robust backing. This chapter has also shown that only one third of therespondentsconsideredtheB2levelassufficientforinternationalL2studentstofunctionlinguisticallyatthestartofuniversity.OtherrespondentsjudgedthatB2was the absolute minimum, recommended offering B2 students additionallanguage support, or considered B2 users insufficiently proficient to meet thelinguistic demands of academic studies at university. Collecting empiricalevidence regarding the performance of international L2 students in the targetcontextisoneofthegoalsofthefollowingchapter.

Chapter2:Content&levelrepresentativeness

54

CHAPTER2CONTENT&LEVELREPRESENTATIVENESS

The previous chapter investigated the language requirements forinternational L2 students in university entrance policies across Europe,andreportedawidespread,butoftenunsubstantiateduseoftheB2level.In thischapterwe focuson theFlemishcontext,andassessevidence fortheB2requirementbasedondatacollectedfrominternationalL2students(L2P and L2F) and Flemish university staff. This chapter not onlyinvestigates the B2 requirement: substantial attention is also devoted toexamininghowrepresentativetheITNAandSTRTtasksareforlanguageuseinthetargetcontextofFlemishuniversities.

Kane’s (2013) Interpretation/Use Argument (IUA) is central to this chapter.Kane’s thesis is that validity is not a property of a test per se, but of theinterpretations or claims made on the basis of a score. Validation impliesempiricallydeterminingtheextenttowhichreal-worldtestscoreusesorclaimsprovide solutions for specific problems within a specific context (Gorin, 2007;Kane,2013).Kanedemandsstrongunequivocalevidenceforclaims,andwhenthestakes are high, he does not allow for any data that contradict a claim. Kaneassignsgreatimportancetoscoreuse,buthedoesnotabsolvetestdevelopersofallresponsibility.Unlesstestsareusedforpurposesthatclashwiththeintendeduse,testdevelopersareresponsibleforclaimspertainingtotaskselection,rating,andthelike(seeFigure1.2).

ACADEMICLANGUAGEREQUIREMENTSInthecontextoflanguageforacademicpurposes(LAP),asubstantialamountofprimarilyAnglo-Americanresearchhasbeendevotedtoidentifyingwhattypifiesreal-life academic language. There is general agreement that LAP requiresadvanced cognitionandabstraction (Hulstijn, 2011;Taylor&Geranpayeh, 2011):argumentation,logicandanalysisareconsideredcentraltotheLAPconstruct,asis the ability to combine different sources and skills (Cho & Bridgeman, 2012;Cumming, 2013). Furthermore, academic language involves specialized lexis(Snow, 2010; Hulstijn, 2011) and complex syntactical structures, includingnominalizations,conditionalstructuresandembeddedclauses(Gee,2008;Snow,2010; Hulstijn, 2011). Some authors have identified giving presentations,describing graphs, understanding lectures, summarizing texts and building anargument asprototypical LAP tasks (Hyland&Hamp-Lyons, 2002;Lynch, 2011;


55

Cho & Bridgeman, 2012). The threshold language level most associated withacademic language proficiency in Europe is B2, but the debate on whether ahigher level might be more appropriate remains undecided (Taylor &Geranpayeh,2011;Hulstijn,2011;Xietal.,2013).

HelpfulasthedescriptionsofLAPcharacteristicsabovemaybe,theyareperhaps too generic to provide detailed specifications for a university entrancelanguage test.Ageneraldescriptiondoesnot sufficeas thebasis fora test thatwillbeusedwithinaspecificcontextforaspecificpurpose(Lado,1961),andlocalconventions may override general principles (Fløttum et al., 2006). Moreover,since most analyses of academic language proficiency relate to the Anglo-American tradition (Xi et al., 2013), their results will not necessarily apply toother contexts. Furthermore, the LAP characteristics described abovecharacterize the language proficiency of an accomplisheduser of the academicidiom,notnecessarilythelanguageskillsrequiredofstudentsembarkingontheiruniversitystudies.

JUSTICESinceitsearliesttraceableoriginsthepurposeofcentralizedtestinghasbeentoselectindividualswhopossessacertainsetofskillsthataredeemedimportantinthe light of a future role or position (Spolsky, 1995). Bachman (1990)characterizes testing as an impartial way of distributing access to benefits orservices, but for Foucault (1977), examinations have very little to do withimpartiality. In the Foucauldian tradition a test is considered an instrument ofpowerthatallowsanin-grouptoselectmembersfromanout-group.Foucault’sviewshaveinspiredlanguagetesterstocriticallyexaminetheimpactoftestsonpeople’slives,andtoquestionthegatekeepingfunctionstheyoftenperform(seeShohamy, 2001 for a seminal contribution). Recognizing the power imbalanceinherent to testing, language testing organizations have developed a set ofprinciples to ensure that testdevelopersdonot engage in activities inimical tocandidates’best interests (e.g., ILTA,2000).Setagainst thisbackgroundof testethics, this chapter investigates to what extent STRT and ITNA can justifiablyserveasgatekeeperstouniversityentrance.

Mostpost-Messickvaliditytheorists,includingKane,havehighlightedtheimportanceofconsideringthesocialconsequencesofatest.Inthewakeofthis,there has been an increased attention for issues of fairness and justice amonglanguagetesters(Davies,2010;Kane,2010;Kunnan,2010;McNamara&Ryan,2011;Xi,2010).Whilethereisgeneralagreementthatfairnessprimarilyconcernsbiasand impartiality (McNamara & Ryan, 2011), justice has proven more elusive,though some consensus does exist. Contrary to fairness, which alwayspresupposes theexistenceofa test, justicequestions the legitimacyofhavinga


56

testasagatekeeperinthefirstplace(McNamara&Ryan,2011):Insomecasesthevery fact that there is a test may introduce imbalance or inequity in a largerpopulation (Kunnan, 2000). This idea is particularly relevant to the currentchapter,whichrelatestoacontextinwhichonesubpopulation(i.e.,internationalL2 students) is required topassa testbeforegainingentrance toan institutionthatisopentoothers(i.e.,studentswithaFlemishsecondaryschooldegree).

While it is not thepurpose of this chapter to fully conceptualize justicefromaphilosophicalperspective,itaimstocontributetothedebateonjusticeinthe language testing literature by applying insights from the major justicetheoriesand touse themascomplementary to the IUA.Kanehimselfdoesnotexplicitlymentionjustice,butitisalogicalextensionofhissixthmainstatement:“theevaluationof scoreuses requiresanevaluationof theconsequencesof theproposed uses; negative consequences can render a score use unacceptable”(2013,p.1).

Muchofwhathasbeenwrittenaboutjusticeinlanguagetestinghasbeeninfluenced by the writings of John Rawls (Davies, 2010). In Rawlsian politicalphilosophyfairnessprecedesjustice,andthefirstprincipleofjusticestatesthatarulingcannotbejustifthefoundationonwhichitisbasedisunfair.Converselyhowever, fairness offers no guarantees for just rulings. The same applies inlanguage testing:A test canbedemonstrably fairwhile being indefensible as apolicy instrument (McNamara & Ryan, 2011), while the opposite is hard toconceive. Rawls’s second principle permits inequalities insofar as theywork tothe benefit of peoplewho have an unfavorable starting position. Applying thisprinciple to language testing is somewhat more challenging. Clapham (2000)callsforequaltreatmentbyarguingthatL2universityentrancetestsshouldnotincludetasksthatL1speakersarenotexpectedtoperforminthetargetcontext.But,ascanbededucedfromRawls’secondprinciple,unequaltreatmentdoesnotnecessarilyimplyinjustice(Dworkin,2003):Universitiesmayhavesoundreasonsfor demanding that L2 students possess linguistic competences that are notexpected of their L1 colleagues. Consequently, a thorough context analysiswillnotnecessarily yield a just testingpolicy.As amatterof fact,nopreconditionscanoffersuchguarantees,sincejusticemightnotbethatabsolute(Sen,2010).

Rawls’ theory relies on the presumption that true justice exists. Thisidealistic approach obstructs its applicability in the real world. Sen (2010)thereforeproposesanalternativetheoryinwhichjusticeisnotseenasabsolute,but as context-dependent. Sen’s theory of justice is grounded in Rawlsianprinciples,butreliesonequalityoffreedomandontheabsenceofinjustice.Ifasituation isperceivedasunjust,andfreedomisrestrictedwithoutareasonable,rational argument, that situation is unjust. Dworkin (2003, 2013), a Rawlsianproponentofdistributivejustice,alsosupportstheimportanceoffreedominhistheoryofjustice.ToDworkin,anyinstitution,largeorsmall,isunderthemoralobligationtoensureequalityofopportunityforallitsmembers–alsowhenthis


57

implies unequal treatment. Rawls does not offer many practical guidelines forinvestigating justice,buthis andDworkin’sworkofferprinciples againstwhichthe justice of a university entrancepolicy canbe evaluated. Sen’s reason-basedapproachblendswithKane’sviewofvalidationashypothesistesting(Oller,2012).

Based on the available definitions of justice in the language testingliteratureandontheinsightsdrawnfromRawlsianpoliticalphilosophy,itcouldbe argued, with Sen, that justice can be defined as the absence of injustice.Hence,apolicy that relieson tests forgatekeepingpurposescanbeconsideredunjustifitrestrictstesttakers’freedomofaccessongroundsthatdonotstandtoreasonorareunsupportedbyempiricaldata.

TheFlemishuniversityentrancepolicylimitsthefreedomofaccessofL2students based on the assumption that students below a certain languageproficiencylevelcannotsuccessfullyparticipateinacademicstudies.Ifthispolicyis just, peoplewho fail the language testwould not performwell in the targetlanguageuse(TLU)context. If theydid, their freedomofopportunitywouldbeunjustly limited, and the entrancepolicywouldbe indefensible. Irrespective ofthedifferencesbetweenmodern-dayjusticetheories,itisunlikelythatanywoulddispute the injusticeofapolicy thatworks to thedisadvantageofpeople inanalreadydisadvantagedposition,yetlacksrationalorempiricalgrounds.

RESEARCHAIMSThisstudyexaminesAssumptions1and2:A1 B2 is an adequate threshold level to decide on international L2 students’

accesstoaDutch-mediumuniversityinFlanders. Based on the findings of Chapter 1 and on a comparison of the B2

descriptor with the LAP literature review, it was hypothesized that B2learnerswouldstruggletomeetthoselinguisticdemands.

A2 STRTandITNAarerepresentativefortheacademiclanguagerequirements

atFlemishuniversities.

The hypotheses relating to Assumption 2 were that (a) the oralcomponents of STRT and ITNA would be representative of the targetsetting, because they contain typical LAP tasks and that (b) ITNA’scomputer component would be less representative because it lacksproductivewriting-acrucialLAPskill.


58

The third aim of this chapter is to assess the justice of the Flemish universityentrancepolicyforinternationalL2students.Thisrequiresdeterminingwhetherfreedomofaccessisrestrictedongroundsthataresupportedbyempiricaldataorrationalargumentation(Rawls,2001;Sen,2010).

PARTICIPANTS&METHODOLOGYThedatapresented inthischapter includetheperceptionsof24academicstaffmembersandtheexperiencesof31L2studentsregardingthelinguisticdemandsof university studies. It draws on insights from needs analysis (Long, 2005;Gilabert,2005)andmixed-methodresearch(Creswell,2015),andrevolvesaroundthetriangulationofsourcesandmethodsbyusingaconcurrentdesigninwhichquantitativedataaddaninterpretativelayertoqualitativedata.ParticipantsL2students

The study is based on two groups of L2 participants representing the mainresearch traditions (humanities, exact sciencesandsocial sciences)at the threelargestuniversitiesinFlanders.Someattendedclassin1000-seatauditoria,whileothersdidsoinsmallergroupsofabout50students.

Group1(L2P)ElevenL2studentsattendedtheir first semesteratGhentUniversityduringtheacademic year 2012-2013. They were enrolled in a non-obligatory course of L2Dutch for academic purposes, which ended in December 2012. The interviewswereconductedinaseparateroomduringtheseclasses.

The median participant age at the time of data collection was twenty(range:18–45),themedianlengthofL2Dutchinstructionwasfourteenmonths(range:9–48)andmostparticipants(7)werefemale.Threeoftheseparticipantshad entered Ghent University at bachelor level, eight at master level. Theserespondentswererecruitedaftertheyhadregistered,sotheyhadalreadypasseda language test (ITNA = 10, STRT = 1). These participants will be referredindividuallybytheirpseudonym,orcollectivelyasL2P(seeAppendix3).

Group2(L2F)Inthesummerof2014,135non-nativespeakersofDutchwhoplannedtoenrollata Flemish university sat both ITNA and STRT as part of a concurrent validitystudy.Ofthisgroup,68candidatespassedITNAorSTRT,grantingthemaccesstouniversity. Less thanhalf of the group (32)wenton to register for aDutch-


59

medium program at university. Twenty of these registered students agreed toparticipate in this study. Before the start of the academic year, one studentdecided to postpone her studies for financial reasons, so twenty participantsremained (see Appendix 4). It is important to stress that these twentyparticipants all tookboth tests and sevenof them receivedadifferentpass/failoutcomeonSTRTandITNA.Becausethesestudents(exceptStella–seebelow)had passed one of the two tests, they were allowed to register for universitydespitehavingfailedtheotherentrancetest.Quitelikely,thisisthefirststudytobypass the truncated sample problem (Wall, 1994) in such a way. This classicsamplingproblementailsthatstudentswhodonotpassanentrancetestcannotenteruniversity,asaresultofwhichthereisnowayofknowinghowtheywouldhavefared.

Ten participantswere freshmen, tenweremaster students. Six attendedGhentUniversity,sixwereattheUniversityofLeuven,andfourattheUniversityofAntwerp.Leilaattendedaninteruniversityprogram.StellahadfailedSTRTandITNA,buthadbeenable to registerat theUniversityofHasselt,whichacceptscertificatesfromitsownin-houseB2test.Themedianparticipantageatthetimeofdatacollectionwas23(range:19–32),themedianlengthofDutchinstructionwas 11 months (range: 6 – 80), and the majority (n = 17) was female. Theseparticipants will be referred to collectively as L2F, or individually using theirpseudonym.Appendix4providesadditionalinformation.

DatacollectionwascarriedoutbetweenOctober2014andJuly2015.Inanylongitudinalstudy,attritionoccursandbytheendof thedatacollection,sevenstudentshadlefttheproject.OeéanaandClarahaddroppedoutinFebruary2015topursuestudiesintheirL1.Anastasiaquitinthesamemonthbecauseshehadlost all motivation to pursue her studies. Stella (April 2015) and Yazdan(November2015)hadtogiveupbecauseofvisaissues.JessicaandChloélefttheprojectafteronemonthwithoutstatingareason.Universitystaff

In January and February 2014, 24 university staff members (out of 64 invited)eachtookpartinoneofsixfocusgroups.Thefocusgroupsrequiredinformation-richparticipants (Reyboldetal.,2013)whowereable toprovideknowledgeableinsights(Patton,2002)intothelinguisticdemandsthatstudentsareexpectedtomeetatthestartofuniversity.

Purposeful participant selection (Freeman, 2000) was based on threeinclusioncriteria:affiliation,position,andexperience.Theparticipantsrepresentthe major universities (12 Ghent University, 12 KU Leuven) and the mainacademictraditions(6humanities,7exactsciences,and7socialsciences),bothat professorial (15; 7 ofwhomwere alsodirectors of educational affairs) and attutor(6)level.Fourparticipantsworkedatthecentraladministration(2language


60

policyand2educationalaffairs).At the timeofdatacollection, themajorityofthe participants had gained substantial professional experience at university(experienceatuniversity:Md:22years,range:3-35)andinteachingatuniversity(participants notworking in administration, teaching experience:Md: 19 years,range:3-29;experiencewithfirst-yearstudents:Md:16years,range:3-29).TheseparticipantswillbereferredtoasAc1–Ac24(seeAppendix5).

Eventhoughtherewerenodirectprofessionaltiesbetweenparticipantsofthesamefocusgroup,hierarchicdifferencesdoexist,andpowerissuescanmakeindividualschange theirviews tomatchgroupconsensus (Reyboldetal., 2013).Forthatreasoneachfocusgroupbeganbycollectingtheindividualopinionsofeachparticipantinapaper-basedquestionnaire(Kahneman,2011),whichformedthebasisofthegroupdiscussion.Datacollection&analysisL2Interviews&focusgroupsAllinterviewsandfocusgroupswereconductedbytheauthor,whousedaseriesof recurringmust-askquestionsbutwas free toelaborateonsalient subthemesthatemergedduringthe talk.Thedatawereaudiorecordedandtranscribed inDutch,butspecificquotesweretranslatedintoEnglish.

TheL2interviewswereconductedtodeterminewhetherL2studentswhopassedaB2testfeltreadyforthelinguisticdemandsofuniversity(Assumption1)andwhethertheacademiclanguagetaskstheyreceivedinreal lifematchedtheonesoperationalizedinSTRTandITNA(Assumption2).

The interviews of L2P took place in October (the first weeks of theacademicyear)andDecember2012,anddealtwiththeparticipants’experiencesat university, the university’s linguistic demands, the students’ social networkand their perceived linguistic ability. L2F participants were interviewed duringthe academic year 2014-2015. Their perceptions of the linguistic demands ofuniversity and of their own language proficiency in relation to those demandswereavitalpartofeachinterview,whichfocusedonadifferenttopiceverytime:the firstweeks of university (October), classroomexperiences (November), thefirst exams (February) and the students’ social network (March). The AprilinterviewwasreplacedbyaretestofSTRTandtheinterviewsinJulylookedbackonthepastyear.

The purpose of the focus groups with the academic participants was tocome to a cross-disciplinary consensus (Belzile &Öberg, 2012) concerning thelinguisticdemandsthatstudentsareexpectedtomeetatthestartofuniversity,and to assess whether these demandsmatched the B2 target level of the tests(Assumption 1). When a focus group began, participants were asked if theywishedtodifferentiatebetweenlinguisticdemandsforL1andL2students,butall


61

agreedthattheminimallinguisticdemandshadtobethesameforallstudents,irrespectiveoftheirL1.Participantswerethenaskedtoindividuallyestimatetherelative importanceof listening, reading,writingandspeakingskills.Next, theyreceived three sets of four listening, reading, writing or speaking samples (seeTable 3.1),which they rank ordered in terms of difficulty or ability. It isworthnoting that the agreed-upon order for every skill in every focus groupcorresponded with the CEFR levels assigned to the samples. Afterwards, as agroup, theydetermined theminimal proficiency level theybelieved a first-yearstudent should have, using an approach based on the bookmark method, afrequently used standard-setting procedure (Bérešová et al., 2011). Table 3.1presents an overview of the samples used in the focus groups (excluding thespeakingsamples,sincetheywillnotbereferredtospecificallyinthisarticle)andidentifies the source, the topic, the length, the percentage of low-frequency (≥5000)andhigh-frequencywords(≤2000),thedifficulty(asmeasuredbyFlesch-Douma, FD), and theCEFR level of the samples.Word frequencies, readabilityindicesandspeechratewereusedasindicatorsofcomplexitytosupplementtheCEFRlevelassignedtothesamples.

AllL2samples(W1,W3,R1,R2,R4,Li2,Li4)wereselectedfromasamplebankcontainingL2performancesandtasksthatwerelinkedtotheCEFRbyanindependent committee of experts (Nederlandse Taalunie, 2015) following theproceduresoutlinedinFiguerasetal.(2009).TheL1writingsampleswerechosenby academic writing tutors, who were asked to provide a representativeperformance of a first-year (W2) and final-year (W4) student. The authenticreading sample (R3) was taken from the first chapter of a first-year sociologycourse book and was considered representative by the focus group members.SampleLi1was selected froma radiobroadcast inwhichaprofessor explains amathematical problem to a wide audience of non-specialists, while Li3 wasrecordedpurposefullywith aphilosophyprofessor,whowas asked to teachhisintroductoryclass.Giventhedemandsofacademiclistening(Field,2011), itwasunlikely that the threshold levelwouldbeplacedatB1.Consequently, to avoidceilingeffects, twoC2sampleswere included.SamplesW2,W4,R3,Li1andLi3were linkedtotheCEFRby fourexperiencedmembersof theabove-mentionedcommittee.

Thetranscriptionswerecodedaprioriandinductively(Dey,1993;Miles&Huberman, 1994) using NVivo 11 For Mac. The a priori coding schemes werebased on salient themes that emerged from the LAP literature review, on theinterview and focus group scenarios, and on earlier research into L2 students’experiences at Flemish universities (De Bruyn, 2011). During coding, themesemergedthatwerenotforeseenintheapriorischeme,addinganinductivelayerofanalysis (Glaser&Strauss, 1967). Inorder tocheck thecodingconsistency,aresearchassistant recodedone focusgroupandallL2F interviewsconducted in


62

November2014,usingthea-prioricodingscheme(forinter-coderagreement,seeTable1.4).

Table3.1.Focusgroupsamples,arrangedbyCEFRlevel

Code Source Topic Length ≤2000 ≥5000 FD w/mWriting

B1 W3 L2testperformance Law 72 88,2% 8,7% 59 B2 W1 STRTperformance Advertising 186 83,2% 7,3% 52 C1 W2 1styearpaper,L1 Arabicstudies 170 78% 4,7% 61 C2 W4 Dissertation,L1 Engineering 121 75,4% 13,2% 40

Reading

B1 R4 B1test History 163 74,4% 7,3% 81B2 R1 B2test(STRT) Musicology 177 79,1% 13% 55C1 R2 C1test Linguistics 179 79,8% 11,3% 29C2 R3 Coursebook Sociology 159 70,8% 21,1% 4

Listening

B2 Li4 B2test(STRT) Biology 2.14 76,3% 8,8% 147C1 Li2 C1test Physics 1.56 84,6% 10,3% 126C2 Li1 Radiolecture Mathematics 2.03 86% 8,2% 145C2 Li3 Universitylecture Philosophy 2.01 82,9% 10,3% 116

Note.Length:inwords(writingandreading)orminutes(listening)≤2000:highfrequencywords≤5000:lowfrequencywordsFD:Flesch-Doumareadability:100isveryeasy,0isverydifficultw/m:wordsperminute

AcademiclanguageskillquestionnaireTheuniversity staffparticipantswereasked to filloutaquestionnaire inwhichtheyselectedthemostimportantacademiclanguageskillstheybelievedstudentsshouldpossessuponuniversity entrance. Since the viewof academicson thesemattersmaydifferfromtheperceptionofstudents,theL2FinformantsreceivedthesamequestionnaireinFebruary2015.Theviewsoftheacademicparticipantsand theopinionsof theL2F informantswereused to assesshow representativethe task selection in STRT and ITNA is for the actual linguistic demands atFlemishuniversities(Assumption2).

The list of language skills used in the questionnaire was based on theliteraturereviewaboveandonalistofcommonlyoccurringtasktypesinthirteenEuropean tests that grant access to higher education (CELI 3, CELI 4,Studieprøven,Testinorsk–høyerenivå,StaatexamenNT2II,ITNA,PTHO,PAT,IELTS,DALF,TCF,TELCC1Hochschule,TestDAF).Skillsthatfeaturedinatleastsevenofthesethirteentestswereaddedtothelist.Participantswerefreetoaddskills to the list, which happened on one occasion (“accurate expression ofideas”). The categories in the list (Table 3.3 shows the final version) were


63

purposefully broad, because they had to bemeaningful to non-linguists (Long,2005).ComplementarydatasourcesLong(2005)andGilabert(2005)recommendsupplementinginterviewandfocusgroup data with other sources in order to get a complete picture of thephenomenaunderexamination.Forthisstudythefollowingcomplementarydatawerecollected:ClassrecordingsandfieldnotesInNovember2014theresearcherattendedaclassofeachL2Fstudent’schoosing.Eleven lecturers gave permission to have their class audio recorded. Before,duringandaftertheclassestheresearcheralsotookfieldnotes.

These data were used to compare test tasks to real-life language tasks(Assumption 2).Additionally, the first 30minutes of each class recordingweretranscribedandanalyzedforwordfrequency(usingTSTCentrale,alemma-basedcorpus for Dutch) and speed (words/minute) and compared to STRT audioprompts,allowingforacomparisonbetweenthelinguisticdemandsofuniversitylectures and the language demands of STRT audio prompts (Assumption 1).Lastly,thefieldnoteswereanalyzedforinstancesthatshowedwhetherornotaparticipantwasabletocopelinguisticallyduringclass(Assumption1).

Academicscoretranscripts&test/retestscores

InApril2015 theremainingL2Fparticipants (N= 15) took twoSTRTtesttasksagain:writingasummaryofascriptedlectureabout industrializationandgivingaten-minutepresentationaboutpollution,basedonslides.Sincepracticalreasons prevented administering the whole test again, the two tasks thatexplainedmostof theoverall scorevariance in theprevious testadministration(N=913)wereselectedfortheretest(𝑅!"#! =.91,p<.000;summaryβ=.52,p<.000presentationβ=.57,p<.000).

At the end of the second semester (July 2015), the L2F participantsprovidedtheresearcherwithtranscriptsoftheiracademicresults.Basedontheiracademic success, theparticipantsweredivided into twogroups: studentswhohadpassedatleasthalfofthecoursestheyhadtakenup(𝐿2!!,N=8),andthosewhohadnot(𝐿2!!,N=8).The𝐿2!!groupdidnotincludethetwostudentswhohad left university because of immigration problems, or the two studentswhohad left theprojectearly.Theacademicperformancedatawerecombinedwiththeentrance test results inorder toassesswhetheranyacademicallysuccessfulL2 students had failed STRTor ITNA,which implies that theywouldnot havebeen able to register for university if they had taken only that particular test(justice).


64

Giventhesmallnumberofparticipantsandthenon-normaldistribution,non-parametric testswere used to analyze these data.Wilcoxon’s SignedRankTestandeffectsizeswereusedtodeterminewhether𝐿2!!studentshadachievedhigher initialSTRTor ITNAscores, and tomeasure scoregainsonSTRT tasks.Since the tests’ CEFR-based scales may be too broad to measure gains over aperiod of eight months, more detailed analyses were conducted, based on amethodologicalapproachadoptedbySerranoetal.(2012)andLlanesetal.(2012).This analysis relies on comparing measurements of complexity (lexical:type/tokenratio;syntactic:clauses/T-unit),accuracy(written:errors/T-unit;oral:errors/AS-unit) and fluency (written: words/T-unit; oral: prunedsyllables/minute) over time. The results will be referred to below, but theanalysesthemselvesarediscussedindetailinChapter6.AllquantitativeanalyseswereconductedwithR(QuantPsycandcarpackages).

RESULTSAssumption1 B2 is an adequate threshold level to decide on

international L2 students’ access to a Dutch-mediumuniversityinFlanders.

Thedatausedtoexaminethisassumptionare:§ test/retestscores,tomeasuredifferencesinL2proficiencyovertime;§ Focusgroupdiscussiondataaboutthelistening,readingandwritingsamples

(seeTable3.1), todeterminetheminimal levelofcompetencetheacademicstaffmembersexpected.Speakingsampleswerenotpartofthediscussions,since all groups agreed that it was the least important skill for first-yearstudentstomaster;

§ InterviewswithL2participants,tocross-checkthefocusgroupresultsandtoprovideconcreteexamplesofthelinguistichurdlestheyfaced;

§ Field notes, to provide first-hand observations of how L2 participantsexperiencedlectures;

§ AcomparisonofthelexicaldemandsandspeedofactuallecturesandSTRTlistening tasks, to determine whether the participants’ perceptions wereconfirmedbyactualobservations.

ListeningThe focus group participants rank ordered the samples (see Table 3.1) beforedeterminingtheminimallyexpectedlevel.Forlistening,readingandwriting,thefocus group participants’ rank order of the samples aligned with their CEFRlevels.


65

In all focus groups, itwasdecided that samplesLi1 (C2) andLi3 (C2)were themost demanding, but also the most representative because they contained anargumentative component and because they were live recordings of lecturesdeliveredinanaturalway.ThefocusgroupsfurtheragreedthatLi3isabovewhatcan be expected from a student on day one because it relies on prior contentknowledge. Li1 was considered lexically less demanding, but with a highinformationdensityandastraightforward lineof reasoning.ThegroupdecidedtoputthecutoffpointbetweenLi1andLi3.TheB2sample(Li4)waslabeledasidealized, unrealistic and unrepresentative because of its straightforwardstructure,itsmonothematicnatureandits“cleanness”.

Ac8 Noprofessorteacheslikesample4.It’stooclean.[…]Ac5 Iagree.Itwassecondaryschooltalk.Ac6 Likeatelevisionprogramforprimaryschoolchildren.

Confirming the university staff’s intuition, all L2P participants struggled tounderstandthenatural,unpolishedlanguageofuniversitylectures.

The professor speaks too fluently forme and too academic. […] I try tounderstand but it still is hard. I am always in doubt.What did he say?Whatdidhesay,Ialwayswonder.

(Noor,October2012)SomeL2participantsdroppedout(Noor),quitgoingtoclasses(Océane,Clara)orexperiencedlossofmotivation(Hoang,Merveille)primarilybecausetheyhadproblemsunderstanding lectures.MostL2Fparticipants felt unprepared for thelistening demands of university lectures, and of the four participants whoreported no listening problems, three gave up before the end of the year. Themain obstacles to understanding lectures had to do with pronunciation,intonationandpace(11),regionalaccents(9),andjargon,idiomsandjokes(9).

[The professor] has the worst accent, so I don’t understandanything.Nothing.Thankgoodnesswehaveasyllabus.

I Doesithavetodowiththecontentofthecourseisitthelanguage? I don’t know, do I? I just bought the syllabus and Iwill discover

whatitisabout.I Soyoureallydon’tunderstandanything? Seriously.Nothing.

(Océane,October2014)


66

Attheendoftheyear,sevenL2Fparticipantsfeltquitesurethattheyunderstoodclassesbetterthanatthestartoftheyear,althoughunfamiliaraccentsorunclearpronunciationremainedapersistentproblemformost.

Duringthefirstsemesteritwasnoteasytounderstandaprofessor,butthesecondsemesterisbetter.Icanunderstandwellnow.Noteverything,butthemostthings.Icanunderstandotherstudents,butnotpeoplewhodonotarticulatewell.

(Merveille,June2014)Theinterviewsshowedthatlexicalproblemscausedadditionaldifficultiesduringlectures. A comparison (see Table 3.2) between the language used in eightscripted lecturesusedasSTRTpromptsand intwelveactualuniversity lecturesconfirmedthis:authenticlecturescontainedmorelow-frequencywordsthantheprompts. Contrary to the perception of the L2 students, however, the averagepaceofreal-lifelectureswasslowerthanthetestprompts.Table3.2.UniversitylecturesandSTRTlisteningprompts 1K-2K* 5K-7K 7K+ w/m♯Test(N=8) M 5.93 2.33 6.50 148.33 SD 4.43 2.52 2.78 18.04Class(N=12) M 6.67 1.23 10.37 103.86 SD 3.79 1.08 5.82 18.60Note.*%ofwordsusedinfrequencyband

♯meanwords/minuteThe field notes reveal other, more qualitative differences between the testpromptsandin-classexperiences.Allbachelorandmasterclassestheresearcherattendedinthecourseofthisstudy,whethertheywereattendedby50orby500students, were primarily ex cathedra. In some classes, professors asked anoccasional question, but there was never any sustained interaction. In mostclasses,therewasalotofbackgroundnoise:“thereisaconstantbuzzofstudentstalking among each other during class. The professor just talks through thenoise” (Field notes Alexandra, p. 3). In one class, the distractions wereparticularlyintrusive:“studentsaroundusaredrinkingbourbon,there’salotoftalking,screamingandshouting”(FieldnotesAlireza,p.1).ReadingTheacademicparticipantsunanimously considered reading sampleR3 (C2) themostdemanding. Inall focusgroups individualmemberssuggestedputtingthecutoffscoreaboveR3becauseitrepresentstheactuallanguageofsyllabi.Inthe


67

endtheconsensuswasthatitisunrealistictoexpectstudentsenteringuniversityto cope with texts of this level, although it is the kind of language they willencounterearlyonintheirstudies.ThegroupsfinallydecidedtoexpectstudentstomasterR2(C1)atthestartofuniversity,butnotR3,becauseof itsstructuralcomplexity.

IneveryfocusgrouptextR1(B2)andtextR4(B1)wereconsideredbelowthemark.Participantsclaimedthat“studentswhocanonlymastertext1haveaproblem” (Ac13) and that text R4 is “annoyingly transparent” (Ac7). Themainreasons why both texts were considered too easy had to do with their clearstructure, low information density and comparatively simple development ofideas.

For theL2F participants readingpresented a problem, but one they saidthey mostly managed. All L2F participants reported that reading took muchlonger in Dutch than in their L1 because they looked up words, because theyconsultedsources inEnglishor intheirL1 tounderstandconcepts theydidnotgrasp inDutch,orbecause they translatedpartsof theircourses.At least threeparticipantshadtranslatedtheirentirecoursesintotheirL1.

Inallhonesty,I’mabitofamaximalist.[…]Ilosealotoftimebytranslating.

I Doyoutranslateyourcourses?Nearly everything yes: some of thewords overlap. But the rest isdifferent. I can’t study in Dutch, but in Armenian I just need toreaditonceortwiceandIknowit.

(Stella,February2015)As the year progressed, quite a few L2F participants reported a perceivedimprovement in terms of reading comprehension (Gabriela, Emma, Océane,Clara) or speed (Alexandra, Marie, Stella). Other participants (Guadalupe,Hoang)confirmedthattheirreadinghadimproved,butwasnotuptostandardyet.

ThereisonebookaboutstuffFreudwrote–verydifficultlanguage[…]Itrytoreadit,butdonotunderstandit.

(Guadalupe,June2015)WritingThefocusgroupsputthecut-offpointforwritingaboveW1(B2)andbelowW2(C1).PoortextstructureandsyntaxwerethereasonswhythefinalcutoffpointwassetaboveW1,eventhoughuniversitystudentsdohandintextsatthislevel:“[W1] isrepresentativeofwhatmanystudentsdo”(Ac17). Insomegroups,even


68

W3 (B1)wasnot considereduncommon,norwaswritten language at this levelseenasareasontofailastudent–eventhoughitwassubstandard.

Ac21 Ifyouaskmewhetherthispersonmayenteruniversity,I’dsayno.Ifyouaskmewhethersomebodycouldpassmycourseifheorshewrites like this: well, yes. If he or she writes factually correctanswersI’dfeelobligedtopassthisperson.

In line with the views expressed in the focus groups, the L2F participantsgenerally found writing difficult and time-consuming but not necessarilyproblematic.Manystudentsdevelopedeffectivecopingstrategies,suchasaskingforpermission towriteexams inEnglish.Studentswhowere involved ingroupworkfoundthatL1studentsoftencorrectedtheirtexts.Otherstudentshadnotyet received a writing assignment and had only taken multiple-choice exams.Quite a few L2F participants did not assume that their written skills hadimprovedsincethestartofclasses.SomeevenfeltthattheirwrittenDutchhadgottenworse(Anastasia,February2014).SpeakingTherewasoverallconsensusthatforfirst-yearstudents,receptiveskillsaremoreessential than productive skills, and that speaking is of little importance:“Speakingjustdoesnothappeninthefirstyear[…]Firstandforemost,studentsentering university should be able to store information” (Ac 4). The universitystaff respondents were asked to determine a cut off score for the skills theyconsideredimportant.Sincespeakingwasconsideredtheleastimportantskillforstudents tomasterwhentheystartatuniversity,nominimumproficiency levelwasdetermined.

Ac7 Wecankeepourexpectationslow,causetheydon’tneedtospeak

in the beginning, do they? [The B2 sample] is definitely going tosurviveatuniversity.

Ac8 Basically, no faculty has any real demands when it comes tospeaking.

Ac6 True.Maybewecanlowertheexpectedlevelthen.Ac8 Thecutoffpointcouldgobelow[B2]then.Ac6 Ormaybejustabove?Ac7 Look,doweneedtotestitatall?Ac6 Yes,you’reright.

After twomonths at university, four L2F participants reported speakingDutchquite often.Others had rarely used it (5),were afraid to use it (5), or had not


69

spokenDutchyet(4).Likewise,10ofthe11L2Pparticipantsclaimedthey“hardlyever” spokeDutch at university. A few students in this studywere involved ingroupwork,whichtypicallyinvolvesspeaking,yetsomestudentsfoundwaystoavoidspeakingheretoo,byusingchat(Janet)ore-mail (Leila)tocontributetogroupdiscussions.

IdoeverythingIcantopreventameetingwithstudents[…]Ialwayswritelongtextstogivemyopinion,butinameetingallIcansayisyes,noandOK.

(Leila,November2014)Leilahintsattheimportanceofspeakingingainingacceptanceinacommunityof peers and building an identity in a new context (Morita, 2004; Amuzie &Winke, 2009). Identity andacceptanceweremajor recurring themes in theL2Finterviews,buttheyarebeyondthescopeofthischapter.ThesethemeswillbediscussedinChapter6.Test/RetestscoresTheacademicparticipantsexpectedthatL2studentswhopassedthetestswouldnotnecessarilypossesstherequiredproficiencylevel.Neverthelesstheyassumedthat L2 students’ language proficiency would improve as the year progressed.Contrarytotheseexpectations,however,theSTRTretestyieldedonlynegligibleeffect sizes and non-significant gains, as measured by the tests’ CEFR basedratingscale,bothforthewholegroup(Writing:W=31,p=.159,r=-.31;Speaking:W=43.5,p= .824,r=-.052)andfortheacademicallysuccessfulsubpopulation(Writing:W=11.5,p=.331,r=-.280;Speaking:W=16.5,p=.872,r=-.046).Moredetailedanalysesoftheperformances(seeChapter6) indicatedthattherewereno significant gains on either task in terms of lexical or syntactic complexity,accuracy or fluency, with small effect sizes r (-.01 – .17). Since STRT is anintegrated-skills test, it does not directly measure listening and reading, butwhen a salient point from the prompt ismentioned correctly in the candidateperformance, one point is awarded. To the extent that STRT’s integrated tasksmeasurereceptiveskills,nosignificantprogresswasrecorded(writtenW=37,p=.206,r=-0.28;oralW=48.5,p=.505,r=-.156).


70

Assumption2 STRT and ITNA are representative for the academiclanguagerequirementsatFlemishuniversities.

Thedatausedtoexaminethisassumptionare§ TheL2Fparticipants’opinionofthetests’representativeness;§ TheexperiencesofL2PandL2Fparticipants;§ Theresultsoftheacademiclanguageskillquestionnaireinthefocusgroups

andintheL2Finterviews.InOctober2014,whenaskedwhichtesttheypreferred,sixL2FparticipantschoseITNA, ten chose STRT, and five were undecided. Participants who preferredSTRT often did so because they felt that ITNA’s computer component lackedcontentrepresentativeness:fourstudentsdislikedITNA’sselected-responsetasksandsixdisapprovedoftheabsenceofwritingtasksinITNA.ITNA’sleastusefultasks according to seven participants were the vocabulary tasks, because theywereperceivedasunrepresentativeforthevocabularyusedatuniversity.

Inpracticeweneveruseproverbs,butwedosometimeshearthem.Manyof the words I studied for ITNA I have forgotten. When you don’tencounteracertainwordatall,youforgetit.

(Marie,June2015)The L2F participants perceived the ITNA and STRT listening tasks as themostuseful,albeitnotentirelyrepresentative.Theimportanceoflisteningisreflectedintheacademicparticipants’skillrankingresults.

TheinterviewswithL2Fparticipantsandtheuniversitystafffocusgroupsclearly showed that all respondents judged receptive skills to be the mostimportant.Forstudentsenteringuniversity,productiveskillsarelessimportant,andspeakingisararerequirement:

Imainly have to listen, basically […] I actually have the feeling thatmyDutch is getting worse. For my courses I don’t need to write much. Imainlywritedownformulas,butthatdoesn’trequiremuchlanguage,soIdon’tpracticeanymore.

(Heddi,December2012)Havingestablishedtherelativeimportanceofreceptiveandproductiveskills,theuniversitystaffparticipantstookthequestionnairetodecidewhichacademiclanguageskillsaremostimportantforfirstbachelorstudentstomasterwhentheyenrollatuniversity.Table3.3belowindicateshowessentialeachfocusgroupconsideredeachacademiclanguageskill.


71

Table3.3.Academiclanguageskillsselectedinfocusgroups(N=6) # + 2+ 3+ 4+ 5+Expressideasaccurately 6 1 1 1 2 1Understandcoherence&cohesion 5 1 1 3Takeclassnotes 5 1 1 2 1Composealogicalargumentation 3 1 1 1 Grammaticalaccuracy 3 1 2 Summarizelongtext 2 1 1 Understandgeneralacademiclexis 1 1Understandscientifictextindetail 1 1 Understandscientifictextasawhole 1 1 Lookupinformation 1 1 Describegraphs&tables 0 Summarizemultiplesources 0 Understandimplicitmessage 0 Giveapresentation 0 Note.#timesselected

+timesawardedlevelofimportance(5+ismostimportant)The consensus in every focus group was that when it comes to speaking orwriting,usingmeaningfullanguageisthemostimportantlanguageskillforfirst-yearstudents.

Ac20 If the message is correct, it’s ok […] What I understand as

“meaningful” is very basic language: I have to be able to agree ordisagreewithwhatisbeingsaid.

Fortheuniversitystaff,thesecondmostimportantacademiclanguageskillwasunderstand coherence and cohesion, which was defined as being able todistinguish essential from non-essential information (Ac4, Ac6, Ac8, Ac17),receptively, but also productively. Even though the university staff consideredreceptiveskillstobeofprimaryimportance,theirselectionofessentialacademiclanguage skills also included skills that are important for passing writtenexaminations.Writing down answers in ameaningful, accurate and structuredwaymattersagreatdealinthatrespect.

When the L2F participants received the questionnaire in February 2014,theirselectionreaffirmedtheimportanceofreceptiveskills:“understandgeneralacademiclexis”,“understandimplicitmessage”and“understandscientifictextasawhole” were most often identified as important. “Compose a logicalargumentation”and“takeclassnotes”occurredinthetopfiveofbothgroups.


72

Infivefocusgroupstheconsensuswasthatstudentscanstartuniversitystudieswithouthavingacquiredspecificacademiclexisbecauseitisthelecturer’stasktointroduce this. Every L2F informant on the other hand complained of limitedlexical knowledge as a major obstacle to attending classes. In most cases, L2Fparticipantswerenotactuallyreferringtohighlyspecializedterms,buttowordsthat are commonly acquired in the course of Flemish secondary education.Possibly,theuniversitystaffunderestimatedthelexicalcomplexityoftheirownlanguage use, assuming that all students would know frequently used wordswithintheirfield.Itisclearfromtheexcerptbelowthatthisassumptionmaybemisguided. Like other L2 participants involved in this study, Alexandra wasunfamiliarwithbasicmathematicalterminologyatthestartofuniversity.

Belgianstudentsknowthesewordsfromhighschool,frombasicmathsorsomething–it’snotthathard.Butwhenyourvocabularyisnotadjusted,you need to think “infinite, what is infinite?” And you need to think innumbers,andwhenIthinkinnumbers,IthinkinSpanish.

(Alexandra,October2014)“Understand implicitmessages”,wasalsoperceiveddifferentlybyacademicstaffand L2F students. Professors were convinced that “academic language is notsupposedtobeimplicit”(Ac7),butforL2Fstudents, implicit languageincludesirony, jokes and idioms – all of which are important when attending lectures.During these lectures, most L2F participants took notes, a skill consideredimportantbyL2studentsanduniversitystaff.But–asbothgroupsacknowledge– note-taking does not mean writing full pencil-and-paper summaries asoperationalized in STRT. More than two-thirds of the L2F participants wrote“commentsonahand-out”(Ac22)withouttakingactualnotes.

At least as important as the skills theparticipants selected, are theonesthey did not select. All L2F participants and all university staff membersdisregarded“giveapresentation”and“describegraphsand tables”.Nevertheless,delivering an oral presentation is one of the two tasks included in the oralcomponentsofSTRTandITNA,andat leasttwoSTRTtasksrelyoncandidatesbeing able to describe graphic or tabular input. All this shows that the testcontent differs from reality in a number of aspects; the next section of thischapterfocusesnotsomuchoncontent,butonlevelrequirements.


73

ThejusticeofusingITNAandSTRTasgatekeeperstouniversityadmissionBecause of the specific design of this study, in which all L2F participants tookbothtests,candidateswhofailedonetestbutpassedtheothercouldstillregisterfor university. The following sources of data were used to assess matters ofjustice:§ The participants’ perceptions about the justice of the university entrance

policy;§ TheinitialSTRTandITNAoutcomes,presentedinappendix3;§ Indicators of academic success (i.e.,L2!! andL2!!for participants who had

attainedmoreorlessthan50%ofthecreditsintheirprogram).Most participants agreed that the use of a language test as a gatekeeper touniversity entrance was warranted. The consensus among university staff wasthatlowlinguisticentrancerequirementscreatefalseexpectations.TheyfeltthatL2entrancerequirementsneededtobehighsincetherearevirtuallynosupportsystems forL2 students (Ac24), and since they “are in the auditoriawithotherstudents [and] it’sbetter togive these studentsa clearmessage fromthe start”(Ac17). Most L2F participants also supported the use of a university entrancelanguage test,butcontrary to theuniversitystaff, theydidnot feel theneedtoraisetherequiredentrancelevel,becauseitwoulddenytoomanyL2studentsthechance to start.Only oneL2F participant opposedL2university entrance tests:“Somebody can find the language easy, but be super stupid academically. Hewon’tsucceed,buttheoppositecanalsobethecase”(Clara).

The academic results of the L2F participants seem to partially confirmClara’s point: there isno clear linkbetween language test scores andacademicsuccess.𝐿2!!studentsdidnotsignificantlyoutperform𝐿2!!studentsontheinitialSTRTandITNAtests(STRT:W=46,p=.625,r=-.115;ITNA:W=51,p=.599,r=-.120).On theSTRTretest too,𝐿2!!didnotoutperform𝐿2!!(STRTwriting:W=14, p = .741, r = -.104; STRT speaking: W = 16, p = .451, r = -.238). Wheninterpreting these outcomes, is important to note that only thirteen L2Fparticipantstookpartinthefinalexamsofthatyear.

Participantswhowereacademicallyunsuccessfulyetgainedadmissiononthebasis of a language test canbe considered falsepositives, in the sense thatthey gained entrance to university but were, for various reasons, not able tosuccessfully complete the first year, whereas participants who failed STRT orITNA yet belonged to the 𝐿2!! group, are false negatives. Of the sixteenparticipantswhodidnotdropoutor involuntarily leave theproject, STRTandITNArespectivelyassignedsevenandsixfalsepositives.Sincefalsepositivesdonotleadtoexclusionofmembersofaspecificgrouphowever,theydonotqualifyas an injustice. From a justice perspective, false negatives carry considerablymore weight. In the𝐿2!! group, ITNA assigned two (Leila, Guadalupe) false


74

negatives,STRTnone.Leilawasnotaconfident speaker,butpassedherexamswithhonors.Guadalupehadexperiencedadifficultfirstsemester,butpassedallofthesecondsemesterexams.

DISCUSSIONTheuniversitystaffparticipantsagreedthatL2studentswouldinevitablybelessproficientthantheirL1peersattheonsetoftheirstudies,butacommonlyheldassumption was that by attending classes, L2 students would become moreproficient at Dutch. This study did not generate any empirical evidence tosupport this hypothesis. The STRT retest in April 2015 did not yield anysignificant score gains, or gains in terms of L2 complexity, accuracy or fluency(for similar finding, seeKinginger, 2008;Amuzie&Winke, 2009;Dewey et al.,2014).TheassumptionthatL2students’languageproficiencywillincreaseoverasemester simply by attending classes in Dutch thus seems unlikely.Consequently, it could be argued that it is vital for L2 students enteringuniversity to have reached a language proficiency level that matches thelinguistic demands of the TLU context. The results show that – especially forlistening–thisisnotthecase.

This study shows that the real-life demands regarding listening andreading skills are considerably higher than those for writing or speaking. TheuniversitystaffandtheL2FparticipantsreferredtotheB2STRTlisteningpromptasanunrealisticidealization.ThescriptedlectureusedinSTRTdidnotcontainthe regional variations, information density, structural flaws, idiosyncraticaccentsordisruptionsthatmakeithardforL2studentstounderstandauthenticuniversity lectures.Therefore, fewL2participantsfeltpreparedforthe listeningdemands of university. With one or two exceptions, all L2 participantsexperiencedproblemsunderstandingacademic lectures.Thisoutcomeconfirmspreviousresearch,whichfoundthatB2listenersareabletounderstandfarlessofan academic lecture than is usually assumed (Field, 2011; Lynch, 2011; Ranta &Meckelborg,2013).

Thefactthatmostparticipantsreportedlisteningasthemostproblematicskilldoesnotimplythattheywereproficientenoughintheotherskills.Listeningsimplyposedthemostimmediatethreat,andtheirrepertoireofcopingstrategieswasfairlylimited.TheuniversitystaffparticipantsalsoconsideredtheB2readingsamples unrepresentative, and all L2 participants reported problems withreading.Formanystudentsthis impliedthattheyhadtostudytwiceaslongastheywouldintheirL1,orhadtotranslatecourseworktotheirL1.L2Fparticipantsalsoreportedproblemswithwriting,butoftenexperiencedsome leniency fromprofessorsor assistance fromL1peers.Given their reported struggles, it canbesomewhatsurprisingthattheL2studentspreferrednottoraisethe levelof the


75

entrance test. For them, however, raising the level implied giving fewerinternationalstudentsthechancetoregisterforuniversity,whichtiesinwiththejusticediscussionbelow.

This study offers little – if any – data to support the assumption thatstudentswhopass theB2 languagetestareable tocopewithreal-life linguisticdemandsofacademicstudies.AllL2studentsincludedinthisstudyhadpassedITNAorSTRTorboth(exceptforStella–seeabove).Somemanagedremarkablywell,butthemajorityofL2participantswasnotreadytodealwiththelinguisticdemands of academic studies at university (Römhild et al., 2011). Additionally,this study affirmsHulstijn’s (2014) assertion that in academic contexts unevenlanguage proficiency profiles are the rule. The data do not suggest that a B2requirementforeveryskillcorrespondswiththeactuallanguagerequirementsatFlemishuniversities.

The second researchgoal of this studywas to investigateAssumption 2.Here,theresultsshowthattosomeextentSTRTandITNAappearrepresentativeof the communicative demands of academic programs at Flemish universities.STRT takes into account content-related criteria, which corresponds to theimportance the university staff assigned to meaningful rather than correctlanguage. ITNA only considers linguistic correctness, however. Both tests takeintoaccounttheimportanceoflexis;inSTRTandintheoralcomponentofITNAtheuseofappropriatevocabularyisaratingcriterion.ITNAalsotestsvocabularyknowledgeinselected-responsetasks,butthistaskwasmostoftenidentifiedasthe least representative by the L2F participants. The L2F participants and theuniversitystaffagreedontheimportanceofargumentationandnote-taking.Bothareoperationalized inSTRT,but theoperationalizationofnote-takingdoesnottrulytakeintoaccounttrendsinPowerPoint-basedteaching(Lynch,2011).

In some cases, however, the operationalization of STRT and ITNAcontrasts with real-life demands. The university staff participants and the L2students at bachelor and at master level agree that for students at Flemishuniversitiesreceptiveskillsaremoreimportantthanproductiveskills.Oralskillsareconsideredtobeofthe least importance.Strikingly,allL2Fparticipantsandalluniversitystaffmembersdidnotconsidergivingapresentationordescribinggraphs and tables to be important skills. This does not necessarily mean thatproductiveskillsshouldnotbeassessedinuniversityentrancetests,becauseoralskills will likely impact students’ social integration (Morita, 2004; Amuzie &Winke, 2009), andwritten skills are important forpassing examinations.Whattheseobservationsdoimplyisthatproductiveskillsaregenerallylessimportantthan receptive skills for Flemish university students, especially in their firstbachelor year. Consequently, assigning decisive importance to oral proficiencytests(ITNA)orrelyingonproductiveoutputalone(STRT)mightnotcorrespondto real-life demands. The test developers’ approach to academic language doesnot alignwellwith the linguistic reality at Flemishuniversities. It appears that


76

thetestdevelopershavedrawnlargelyontheLAPliterature,whichisprimarilyAnglo-Saxon,withoutnecessarilytakingintoaccountthespecificfeaturesoftheFlemishcontextforarepresentativeselectionoftesttasks.

ApartfromexaminingevidenceregardingAssumptions1and2,thisstudyquestioned the justice of using ITNA and STRT as gatekeepers to universityadmission. Carlsen (2017) distinguishes two kinds of interpretations given touniversity entrance language test scores.The strong interpretation implies thatstudentswhopassatestarereadyforthelinguisticdemandsofuniversity.Thisstudyshowsthatstudentswithhighlanguagetestscoreswerenotguaranteedtomanagewellatuniversity.Sincethereislittleifanyresearchtosuggestotherwise(e.g.,Lee&Greene,2007;Cho&Bridgeman,2012),thestronginterpretationwasnot a hypothesis this study was designed to test. The weak interpretationhowever is at the basis of many university entrance policies. It assumes thatstudentswhodonotpassalanguagetestarenotreadyforthelinguisticdemandsofuniversity,andwillthereforebeunlikelytoachieveacademicsuccess.

Inessence,theideathatstudentswhodonotmeettheminimumlanguagerequirements,will notmanage in real life offers the rationale for restrictingL2students’ freedom of access. Investigating this is difficult however, because itoften is impossibletotracefalsenegatives. Inthedesignofthisstudyhowever,theproblemof truncated samples (Wall, 1994)wasbypassedby tracking sevenL2FparticipantswhohadactuallyfailedSTRTorITNA.ITNAassignedtwofalsenegatives, STRTnone. If Stella –whohad a good academic recordbut left thecountry due to visa problems – is included in the count STRT and ITNArespectively assigned three and one false negatives. False negatives signal anunfounded restrictionof access that applies toone subpopulation alone (Kane,2013).Accordingtoleadingjusticetheorists,thismightsheddoubtonthejusticeofanentrancepolicy(Rawls,1971,2001;Dworkin,2003,2011;Sen,2010).

CONCLUSION:ASSUMPTIONS1AND2Theresultsof this studyreveal thatL2studentswhopassed ITNA,orSTRT,orboth,werenotreadyforthereceptivelinguisticdemandsofacademicstudiesatuniversity (Assumption 1). It is also safe to conclude that the content of theFlemish university entrance tests at points deviate from real-life languagedemands(Assumption2).Theobservationthatanumberofstudentswhowereassessed below B2 actually managed at university, also qualifies as negativeevidenceregardingAssumption1.RequiringanevenB2leveldoesnotappeartobe a very effective way to discriminate between students who are likely tomanagethelinguisticTLUdemands,andthosewhoarelikelytostruggle.

Part2:Selection&discrimination

78

PART2SELECTION&DISCRIMINATION

ThefirstpartofthisdissertationinvestigatedtwoaspectsoftheFlemishuniversityentrancepolicy:theB2levelrequirement(Assumption1),andthe representativeness of STRT and ITNA for the target context(Assumption2).Chapters3–5includedinthissecondpartfocusontestequivalence(Assumption3)andthe languageproficiency levelof first-yearstudentswithaFlemishsecondaryschooldegree(Assumption4).

Chapter3examineswhetherSTRTandITNAcanbeconsideredequallydifficult,and whether comparable tasks measure similar constructs. Next, Chapter 4scrutinizesthepartofSTRTandITNAforwhichdirectone-on-onecomparisonscanbemademostdirectly:thelinguisticcriteriausedtoscoreoralperformances.The outcomes of both chapters indicate that STRT and ITNA cannot beconsideredequivalentintermsofdifficulty,construct,orratingscales.Assumption 4 – on the language proficiency level of Flemish high schoolgraduates–isoneoftheresearchquestionsofChapter5,whichalsoinvestigatesperformancedifferencesbetweentheperformancesoftwogroupsofL2learners.TheresultsquiteclearlyshowthatnotallstudentswhograduatedfromaFlemishsecondaryschoolarelikelytoattaintheB2level.Chapters3,4,and5arebasedon:

Deygers,B.(2017,inpress).Universityentrancelanguagetests:examiningassumedequivalence.InJ.Davis,J.Norris,M.Malone,T.McKay,&YSon(Eds.). Useful Assessment And Evaluation In Language Education.Washington,D.C.:GeorgetownUniversityPress.Deygers,B.,VanGorp,K.,&Demeester,T. (2017, inpress).TheB2 levelandthedreamofacommonstandard.LanguageAssessmentQuarterly.Deygers,B.,VandenBranden,K.,&Peters,E. (2017).Checkingassumedproficiency: Comparing L1 and L2 performance on a university entrancetest.AssessingWriting,32,43–56.

These papers have been edited to fit the structure and approach of this book.Sections pertaining to the research context, the methodology, and theparticipantshavebeenrevisedtoavoidredundancy.

Chapter3:Level&constructequivalence

80

CHAPTER3LEVEL&CONSTRUCTEQUIVALENCE

The first chapter showed that universities often require prospectiveinternational students to pass a language test as a precondition foradmission.InEurope,themostcommonlyrequiredlanguagelevelforthispurpose is B2. Quite often, different tests are accepted as proof of B2proficiency,butusuallywithoutempiricallydetermining the relationshipbetweenthedifferenttests.

This chapter examines whether STRT and ITNA can be considered equivalentmeasures of Dutch language proficiency at B2 level. If this assumption (A3) istrue,orlargelytrue,acceptingeitherSTRTorITNAposesnoimmediateproblem.But, if it is false,andifonetest issubstantiallymoredifficultthantheother, itcouldleadtoanunjustentrancepolicy.IninvestigatingwhetherthetargetlevelandtheconstructsofSTRTandITNAarecomparable,thisstudydrawsonKane’s(2013) Interpretation/Use Argument and on Phillips’s (2007) ideas on policyeffectiveness. The implications of the findings are discussed with regards tojustice(Kunnan,2000;McNamaraandRyan,2011;Rawls,2001;Sen,2010).

The implicationofKane’s logic is thatwhenscoresof twodifferent testscarryequalweight inauniversityentrancepolicy,universityadmissionofficersrely on an assumption that requires validation, since it has important socialconsequencesforthecandidates.InKane’slogic(Kane,2013,p.62,butseealsoBachmanandPalmer,2010),sinceneitherSTRTnorITNAhavemadeanyclaimsregardingtheirequivalence,thisassumptionisforscoreuserstoprove.Todate,however,noempiricalevidencehasbeenofferedinthisregard.

EQUIVALENCEWhenauniversityclaimsthatthecertificatesoftwodifferenttestsareequivalentmeasures of a certain level of language proficiency, and this claim iswrong orunsubstantiated,itmayhaveseriousconsequencesfortheeducationalstandardsofauniversityandon the livesof test takers.First,whenauniversityentrancelanguage policy wrongfully assumes that different language tests measure anequivalent level of language proficiency, this policy may cause unacceptablevariation in the assessment of the language abilities of the admitted studentpopulation, thereby failing to meet its main goal. Second, when two tests areassumedtobeequivalent,testtakersshouldhaveacomparablechanceofpassingeither test.When test takers need tomake a choice between two tests on the


81

basisofunreliable information, the justiceof theuniversityentrancepolicycanbequestioned.

A number of studies have examined the relationship between scores ontwo university entrance tests by calculating correlations. Fulcher (1997),comparingalocaltestwithTOEFLresults,computedanoverallcorrelationofr=.64.Anotherstudy(ETS,2010)reportsa.73correlationbetweenTOEFLiBTandIELTS, and correlations between .44 (writing) and .68 (speaking) for thesubskills.TheETSresearchers further investigatedtherelationshipbetweenthetests using regression-based prediction, equipercentile linking, and conditionalprobability. The report excluded the regression and conditional probabilityanalyses because they were less informative than the equipercentile findings.ZhengandDeJong(2011)comparedthePTEAcademictesttoEnglishlanguageteststhatareusedforuniversityadmissionpurposes,suchasTOEIC(r=.76)andTOEFL iBT(r= .75),andusedregressionanalysis tomapPTEAcademicscoresonto the CEFR (r2 = .5). Lastly, Riazi (2013), building on the aforementionedstudy,foundanoverallcorrelationofr= .82betweenIELTSandPTEAcademicscores,andcorrelationsbetweenr=.66(listening)andr=.72(speaking)forthefour skills. Using t-tests, Riazi further showed that PTEAcademic significantlydifferentiated between candidates who received higher and lower IELTS bandscores.Riazireportedmediumeffectsizesfortheproductiveskills(Speakingη2=.50;Writingη2=.50),andclosetomediumeffectsizesforthereceptiveskillsandtheoverallscore(Listeningη2=.38;Readingη2=.44;Overallη2=.45). Insum,recentstudiesthatexaminedtherelationshipbetweenuniversityentrance language test scoresoften showmedium to strongcorrelations.Allofthestudiesconsultedprovidedcorrelationcoefficients,andsome(e.g.,ETS,2010)offered evidence from other types of analyses, because relying on correlationalevidencealonecanberathermisleading(Kane,2013;Lissitz&Samuelsen,2007;Norris,2016).Usually,thesestudieswerebasedonofficialscoretranscripts(e.g.,ETS2010),sometimescombinedwithself-reporteddata(e.g.,Zheng&DeJong,2011),butlittleornostudieshavebeenpublishedinwhichresearchershavehadaccesstodetailedraterdatafromtwodifferent,high-stakesentrancetests.

Another important aspectof test equivalence is theextent towhich twotestsusedforthesamepurposeinthesamecontextmeasuresimilarconstructs.Iftwoteststhatareassumedtobeequivalentdonotmeasurethesameleveloflanguage proficiency, examining construct equivalence may shed light on thenatureofthismismatch(Lindridge,2015).Clearly,aswasthecaseforexamininglevel equivalence, using correlational evidence to tackle research questionsrelating to construct equivalence is insufficient (Kane, 2013; Norris, 2016;O’Loughlin, 2001; Shohamy, 1994). Correlations may be spurious (Lissitz &Samuelsen, 2007), they may hide underlying discrepancies (Harsch & Martin,2012),andtheydonotofferinformationconcerningthenatureoftherelationshipbetweentwotests.Forthatreason,inthesocialsciences,constructequivalence


82

researchisoftenconductedusinginferentialstatisticssuchasexploratoryfactoranalysis(Welkenhuysen-Gybels&vandeVijver,2001).

In language testing, however, little quantitative research has beenconductedtoexamineconstructequivalence,eventhoughitcancontributetoadeeper understanding of the outcomes of level equivalence research (Wang,Wang,&Hoadley,2007).Forscoreusers,suchasadmissionboards,students,orteachers,itcouldbeinformativetoknowwhytestresultsdifferandwhereteststhat serve the same purpose are actually dissimilar. Universities offering post-entry languagecourses for internationalL2 studentscoulduse this informationfor instructional purposes, for example. Or, programs with strict writingrequirements could be interested in learning how ratings on equivalent testsrelatetoeachother.

JUSTICEPhillips (2007), observing that policy measures are not always founded onempiricaldata,recommendscriticallyexaminingpolicyclaimsbyidentifyingtheproblemthatthepolicywasintendedtosolve,andbyevaluatingtheeffectivenessoftheproposedsolutiononthebasisofevidence.Thedecisionrule(Kane,2013;Phillips,2007)quitesimplystatesthatapolicymeasurecannotbemaintainedifitdoesnotsolvetheproblemitwasmeanttoaddress.Translatedtothecontextofthisstudy,thedecisionruleimpliesthat,iftheuseorinterpretationofatestscorewithinauniversityentrancepolicyisunsupportedbyempiricaldata,theremaybereasontodoubtitseffectiveness.

Kane (2013), referring to Phillips (2007), requires the evidence used tovalidateapolicyclaimbeproportionatetothesocialconsequencesofthatclaim.And,whenthestakesarehigh,alltheevidenceshouldsupporttheclaimsmadeby score users (Kane, 2013). If empirical evidence does not support theway inwhich policy uses test scores (in this case, as equivalent), the policy may beineffective,butitmightalsobeunjust.

Justice, in this case, is concerned with the effect of introducing one ormore language tests on a larger population. In linewithRawlsian (Rawls, 1971,2001) logic, the prerequisite for justice is fairness, that is, freedom from bias(Rawls,2001;Sen,2010).But,evenifalltestsacceptedinauniversityadmissionpolicy are equally fair, the admission policy can be unjust when it causes anindefensibledisequilibrium inapopulation (Kunnan,2000;McNamara&Ryan,2011).Whena candidate ismore likely to get intouniversity simplybecauseofpickingtestAratherthantestB,withoutbeingawareofapossibledifferenceinpass probability, the university entrance policy may be unjust, since it mayrestrict freedom of access to university on grounds that are unsupported byempiricaldata(seeChapter2).


83

RESEARCHQUESTIONS

The two research questions that guide this chapter tackle the assumption ofequivalence(i.e.,Assumption3)fromtwoangles:RQ1 Is there empirical evidence to support the claim that STRT and ITNA

certificates are equivalent measures of Dutch language proficiency in theFlemishuniversityentrancepolicy?

Secondly, this chapter aims to explain the reasons for a possiblemismatch byconsideringconstructequivalence.RQ2 WhatisthenatureoftherelationshipbetweencomparableITNAandSTRT

tasks,constructs,andcriteria?Thestudydescribedherecontributestotheexistingliteraturebyexplainingtheresults of level-equivalence research by means of construct-equivalencemethodologies. In addition, the results are used to explore the ethicalconsequences of the university entrance policy with reference to principles ofjustice.

PARTICIPANTS&METHODOLOGYThe introduction of this dissertation offered further information on STRT andITNA.Appendix1and2provideadetailedoverviewoftheSTRTandITNAtasks.Nevertheless,itmaybeusefultoreiteratethatITNAandSTRTdiffermostinthewrittencomponents,butcontainhighlysimilaroraltasks.Thecomputersectionof ITNAcontains selected-responseorgap-filling tasks,whereasSTRT’swritingtasks are integrated and require a lot of writing. Even though theoperationalizations differ, thewritten components of ITNA and STRT are bothscored for reading, listening, vocabulary, grammar, and cohesion. The oralcomponentsofbothtestsincludeapresentationandanargumentationtask.

PerformancesonITNAarescoredafterthetestbytwotrainedexaminerswhoreachaconsensusscorefor five linguisticcriteriabasedonthecandidate’sperformanceonboth tasks. STRT is centrally scoredby two trained raterswhoscorecontentcriteria inabinaryway(i.e.,whetherthecandidatementionstherequired aspects or not) and linguistic criteria on a four-point scale. Thelinguistic criteria in both tests are based on the same CEFR descriptors(Vocabulary, Grammar, Cohesion, Fluency, and Pronunciation), but the STRT


84

ratingscale includes twoadditionalcriteria (Registerand Initiative).ScoringontheITNAcomputertestisbinaryandautomated.Asintheoralcomponent,thewrittenpartofSTRTisscored forcontentcriteriaand linguisticcriteriabytwoindependent,trainedraters.

STRTcandidateswhoachieve anoverallRaschmeasure at or above 1.42(private communication, 6 January 2016) receive certification. ITNA candidateswho score 54% or more on the computer test, may take part in the oralcomponent.ITNAcandidateswhotaketheoralcomponentandattainanoverallscoreof52.5%ormore,gettheB2certificate.

Below, rating criteria will be printed in italics with a capital letter. Toavoid confusion, Vocabulary, as a criterion will be printed differently thanvocabulary,asalinguisticcompetence(usingtheCEFR’sterminology).DatacollectionInordertodeterminewhetherthesamecandidatesreceivedcomparablescoreson STRT and ITNA, and – if not – why, test performance data of the samecandidatesonbothtestswerecollectedforthepurposeofthisstudy.FromMay2014throughSeptember2014,allITNAcandidates(N=802)wereinvitedtotakeSTRT free of charge. Since ITNA scores are communicated to the candidatewithintwodaysoftakingthetest(whileitmaytakeuptoamonthbeforeSTRTresults are known), it was assumed that successful ITNA candidates would bedisinclinedtostill takeSTRT.Thus,toavoidattrition,allparticipants first tookSTRTnomorethanoneweekpriortotakingITNA.

AllwrittenSTRTtestswereadministeredundertheconditionsprescribedin the examination manual, and in the presence of the researcher. Trainedexaminersconductedtheoralexaminations,followingaprocedurethatishighlycomparable inSTRTand ITNA.Thecandidate receives the speaking tasks,getspreparationtime(typicallytenminutes)andreturnstoperformthetaskinfrontof the examiner. All participants also took the ITNA computer test underprescribed examination conditions. Trained ITNA examiners administered theoralITNAtasks.AllperformanceswerescoredbytrainedSTRTandITNAraters.L2FparticipantsBetweenJune2014andSeptember2014,ITNAcandidateswereinvitedtotaketheSTRT free of charge one week before the ITNA administration, which grantedthem an extra opportunity to gain access to university. The predeterminedstoppingcriterionfordatacollectionwasthestartofthe2014-2015academicyear.Afteromitting incompleteperformances(someparticipantsgaveupduringoneorbothtests),118participantsremainedinthedatasetthatwasusedtocomparethe results of STRTwritten and ITNAcomputer. Since ITNA candidates who score


85

below 54% on the computer test cannot take part in the oral component, thenumber of participants that could bemeaningfully compared for the oral testswasreducedto82.

Theexamswereadministeredat the largestFlemishuniversities (37%attheUniversityofAntwerp,34%atGhentUniversity,and29%attheUniversityofLeuven)bytrainedexaminers.Thefirstauthorwasalwaysonsitetoensuretheconsistency of the test administration. Rating the STRT test takes a fewweeksbecauseallperformancesarescoredcentrally.However, theITNAscores inthecurrent administration were available on the day of testing. The candidatesreceivednofurtherformalizedinstructionbetweenthetests,andgiventheone-week time span between the two administrations, it was assumed that theirlanguage skills remained constant. The performances were rated anonymouslyundernormalratingconditions.Table4.1.Researchpopulationvariablesvs.regularSTRTandITNApopulations L2F ITNA STRT Age Mean(SD) 27(7) 28(8) 26(7) Min-Max 16–50 15–61 14–60 Gender Female 70% 67% 65% Male 30% 33% 35% Educationalgoal 66% 76% 58% L1 French 17% French 16% French 29% Spanish 8% Spanish 11% German 25% Arabic 7% Arabic 9% Papiamento 7% Russian 7% Russian 8% Dutch 6% German 6% German 5% Russian 4%N 138 485 521 The L2F participant population was representative for the population of bothtestsintermsofage(L2F𝑋=27,SD=7;ITNA𝑋=28,SD=8;STRT𝑋=26,SD=7), gender (L2F 70% female; ITNA 67% female; STRT 65% female), andnationality. In terms of L1, the actual STRT population had a slightly differentdistributiondue to the largenumberofcandidates fromBelgium’sneighboringcountries, expat communities, and countries with historical ties to the Dutchlanguage.Table4.1displayskeydemographicvariablesforthefullL2Fpopulation,andthetypicalSTRTandITNApopulations.

T-tests showed that the distribution of scores found in the respondentpopulationcorrespondstothescoredistributionintheoveralltestpopulations.No significant differences were found between the final scores of the L2FpopulationandthoseofthetotalITNApopulationwhotookthetestinthesameperiod(t(485)=-.493,p=.622).ForSTRT,thisstudywasthefirstadministrationofanewtestversion,andapart frompilotdata,nootherscoreswereavailable.


86

Levene’s test forequalityofvariance (Field,Miles,&Field,2012)confirmed thevariancecomparabilityofthefinalscoresofthesamplepopulationtotheregularSTRTpopulationinthelastadministrationofthepreviousSTRTtest(F=0.014,p=.907).DataanalysisLevelequivalenceInordertofacilitatetheinterpretationofthedescriptivestatistics,thetotalrawscores of STRT and ITNA (172 and 125 respectively) were recalculated into apercentilescale.Thesignificanceofthedifferencebetweenoverall,written,andoraltestresultswasdeterminedusingt-tests(Riazi,2013).Cohen’sdwasusedtocalculate the effect size for the difference between test results for the fullpopulation(N=118).Sinceitwasassumedthatbothtestswerelikelytoagreeonthe best and theworst performances, the scoreswithin the interquartile range(i.e., the range excluding the 25% highest and 25% lowest performances)wereexaminedaswell.Wilcoxon’srank-sumtest,anon-parametrictest,wasusedtodeterminethesignificanceofthedifferenceswithintheinterquartilerange,andCohen’sdwasusedtoestimatetheirmagnitude.

Both parametric (r) and non-parametric (τ) correlations were used todescribethestrengthoftherelationshipbetweentheoverallscores,thewrittenscores, and theoral scores. Interquartile correlationswereused tomeasure thestrengthoftheagreementaroundthecut-offpoint.

For university admission in Flanders, only one thing reallymatters, andthatisgettingtheB2certificate.Inordertogetaclearideaofthedisagreementintermsofpass/failjudgments,acrosstabwasconstructedonthebasisofbinaryoverallSTRTandITNAoutcomes.Furthermore,theprobabilityofpassingeithertestwascomputed.McNemar’sbinomial sign test served todeterminewhetherthedifferenceinpass/failjudgmentsbetweenthetwotestswassignificant.

Furthermore, inorder togauge the strengthof the relationshipbetweenoverall test scores, and scores on the written and the oral components,parametric and non-parametric correlation coefficients were computed for thefull population and for the interquartile population. The Pearson correlationcoefficientwasusedtoassessthestrengthoftherelationshipbetweenthetotalscoresandbetweenthescoresonthewrittencomponents.Becauseofthesamplesize(Howell,1997),andbecauseofconsiderationsregardingrestrictionofrange(Field,Miles,&Field,2012),Kendall’sTau(τ)waschosentocorrelatetheoralandtheinterquartiledata.


87

ConstructequivalenceTheITNAandSTRTspeakingcomponentscontainthesametasktypesandmanycorrespondingratingcriteria.BothscalesarebasedonthesameCEFRlevels(A2–C1),butforcertainskills(e.g.,Grammar),theITNAscaledifferentiatesbetweenbasicproficiencylevelsandpluslevels,whereSTRTonlyhasonelevel.WhentheITNAratingscale includedaplus level,but theSTRTscaledidnot,both ITNAlevelsweremerged(i.e., ITNA’sB1andB1+bothbecameB1,andITNA’sB2andB2+ both became B2). All scores were recoded in collaboration with thecoordinatorsofbothteststoensurethatnointerpretativeerrorsweremade.Anyrecodingwasdoneaftertheperformanceswererated,soallratersusedtheirownscalesinthewaytheyweretrainedtousethem.

Since the rating criteria were known to be correlated, oblique promaxrotation was used to run a Principal Component Analysis (PCA) on thestandardized z-scores of the oral component. Since ITNAonly offers scores onboth tasks combined, the PCA was run using the STRT scores for both taskscombined.Afterhavingdetermined that thepreconditions foraPCAweremet(Bartlett’stestofsphericity:X2(4)=358,p<.000,KMO=.82),theinitialanalysisshowed that three factorshad eigenvalues at or above 1, explaining 70%of thescorevariance.Thescreeplotwasused forconfirmatorypurposes,andshowedthatathree-factorsolutionwaswarranted.

Linear regression was used on the written and the oral datasets todeterminehowwellSTRTratingspredictedITNAscores(Zheng&DeJong,2011).Inbothdatasets,thenumberofcaseswithlargeresiduals(written,4%;oral,4%afterremovaloftwooutliers)waswithinlimits,Cook’sdistancewasnever>1,noindividualcasesweremorethanthreetimestheaverageleverage,thecovarianceratio was satisfactory, and the assumptions of independence andmulticollinearitywere not violated (Norris, 2015; Purpura, Brown, & Schoonen,2015). A multiple linear regression analysis was conducted to determine howmuchscorevariance in ITNAcomputerwaspredictedbySTRTwrittenscores.For theoral test component, a regressionmodelwas constructedwith the overall oralITNAscoresasa functionoftheSTRTcriteriascores.Giventhe limitedsamplesize,we couldnot reliably assess the contributionof the predictors in the oralmodel,butwewereabletogeneralizefromtheoverallmodelfit(Field,Miles,&Field,2012).

Regression analysis helps to see how different variables relate to oneanother,buttheydonotyieldinsights intotherelativedifficultyoftasks,tests,andcriteriathatMulti-FacetedRaschanalysis(MFRA)offers(Bond&Fox,2007).MFRAconsidersatestscoretheresultofaninteractionbetweendifferentfacets,suchascandidateabilityandtest,task,orcriteriondifficulty(McNamara,1996).MFRA software, such as FACETS (Linacre, 2015), estimates the interactionbetween different facets that contribute to a test score andmaps them onto a


88

commonlogitscale.Eachfacetiscomposedofdifferentvariablesthatarecalledelements.Thedifficultyofanelementiscalledameasure,whichisexpressedinlogits. Infit Mean Square (MnSq) statistics show how well an element fits theRaschmodel and indicates towhat extent the elements of a facet fit the sameconstruct. The closer the InfitMnSq is to 1, the better the data fit themodel;valuesbelow.5 indicateredundancy,andvaluesabove1.5areseenasmisfittingordisruptivetothemodel(Barkaoui,2014).

MFRA was used in this study to compare the relative difficulty ofITNAcomputerandSTRTwritten(N=118)andtorankthetasksonbothtestsintermsof relative difficulty. For this purpose, all tasks were weighted equally in theRaschmodel.AsecondRaschmodelwasconstructedusingequallyweighedoralcriteria (n=82) inorder todeterminehowthecriteriaonboth tests ranked intermsofdifficulty.Lastly,athirdMFRAusedtherealweightsoftheoralcriteriatocomparetheactualdifficultyofbothoraltestsrelativetoeachother.

Priortotheseanalyses,instancesofcandidatemisfitwereexamined.Onecasewasremovedfromthedataset,asthiscandidatehadprematurelyendedtheoral component of STRT, resulting in an incomplete set of observations (thiscandidatewas removed fromallanalyses included in thecurrent study). In thefifteen remainingcasesof candidatemisfit, it concerneddisagreeing judgmentsonSTRTandITNA,whichwasrelevantinlightoftheresearchquestions.Sincethedatasetcontainedeighteencasesinwhichtherewasa30%differenceinrawscores(convertedtothesamepercentilescale),itwasdecidedtogivepreferenceto InfitMnSqmeasures rather than toOutfitMnSqmeasures,which aremoresensitivetooutliers.

RESULTSLevelequivalenceThe overall STRT-ITNA correlation is strong (r = .767**), as is the relationshipbetween STRTwritten and ITNAcomputer (r = .694**). Other correlations result inmuchlowercoefficients,andconsideringthesimilarityof theoralcomponents,thecorrelationcoefficientofτ= .387** isstriking.Lowinterquartilecorrelations(totalτ=.19*;writtenτ=.06;oralτ=.09)indicatealackofagreementbetweenthetwoassessmentsaroundthecut-offpoint.

The overall correlation could be taken as support for the assumption oflevel equivalence, but all other analyses point towards a different conclusion.First,thedescriptivestatistics(seeTable4.2)indicatethattheITNAmeanscores(standardizedtoapercentilescaleforthepurposeofcomparison)arelowerthanthoseonSTRT,bothoverallandfortheseparatecomponents.


89

Table4.2.Descriptivestatistics(percentilescale)

ITNA STRT

Computer Oral Total Written Oral Total

N 118 82 118 118 82 118

𝑋 59.31 50.97 48.06 66.76 73.16 67.68

SD 15.54 19.69 20.09 12.63 10.33 11.72

Med 58.42 48.75 51.63 67.33 72.54 69.19

SE 1.42 2.17 1.84 1.16 1.14 1.07

T-tests confirmed that the differences between STRT and ITNA are significant(total:t(236)=-9.20,p<0.001;written:t(236)=-4.061,p<0.001;oral:t(162)=-9.036,p<0.001),withmedium(written:d=-0.53)tolarge(total:d=-1.19;orald= -1.41) effect sizes. The differences between the two tests remain significantwithintheinterquartilerange,withlargeeffectsizes(total:W=135,p<.000,d=-2.14;written:W=775,p<.000,d=-1.04;oral:W=190,p<.000,d=-1.16).

Inorder todeterminewhether thedifference inmeanscoresmeant thatfewercandidatespassedITNA,acrosstabofSTRTandITNApass/failjudgmentswas constructed. Table 4.3 confirms that more participants failed ITNA thanSTRT and that 24% of the population received a different pass/fail judgment.McNemar’sbinomialsigntestshowsthatthisdifferenceissignificantatp=.001.Additionally, the pass probability is significantly (W = 6010,p = .02) lower forITNA(𝑃!"#$

!"##=.35)thanforSTRT(𝑃!"#"!"##=.50).

Table4.3.Pass/Failcrosstab

STRT

Fail Pass Total

ITNAFail 53 23 76

Pass 5 37 42

Total 58 60 118


90

ConstructequivalenceInordertoexplainwhymorecandidatesfailedITNAthanSTRT,thewrittenandtheoraltestcomponentswereexamined.WritingcomponentThe first step in explaining the nature of the relationship between thewrittencomponentswas constructing amultiple linear regressionmodel (table 4.4) todeterminetowhatextentSTRTtaskscorespredictedthetotal ITNAscore.Themodelexplains48%oftheITNAscorevariance.Thestrongestpredictorsarethewriting-from-listening summarization task and the writing-from-readingargumentationtask.Table4.4.ITNAcomputerscoresasafunctionofSTRTwrittenscores

STRT

Constant ArgAudio SummAudio ArgRead SummRead

B .373 2.514 3.784 1.845 2.996

SEB 1.034 .781 1.155 .828 7.491

β .031 .322 .280 .206

p-value ns ** *** * ns

Note.TotalR2adjustedis.482(p=<.001).

Ifalltasksareweightedequally,theMFRAreliably(.90)showsthatITNAcomputerismore difficult than STRTwritten,mainly due to the relatively high difficulty ofITNA’s vocabularyandgrammar itemsaswell as the relatively lowdifficultyofthe argumentative STRT tasks. TheMFRAmodel reliably (.93) identified threedistinct levels of difficulty in the tasks: the borders between those levels areindicatedwithadashed line in table4.5.Themostdifficult tasksare the ITNAvocabulary and grammar tasks, and the least challenging ones are the STRTargumentative tasks. The dictation task in ITNAmisfits the Rasch model andSTRT’swriting-from-readingargumentationtask(STRT)isredundant.


91

Table4.5.MFRAwrittentasks(equalweights)

Task Test Measure SE InfitMnSq

Language-in-use:Vocabulary(cloze) ITNA .43 .10 1.42

Language-in-use:Cloze(small) ITNA .40 .07 .71

Language-in-use:Vocabulary(MC) ITNA .38 .07 1.22

Language-in-use:Grammar(gaps) ITNA .33 .07 .72

Language-in-use:cloze(big) ITNA .10 .07 .97

Note-taking(listening) STRT -.01 .08 .88

Writtensummary(reading) STRT -.04 .08 .95

Structuring(drag-drop) ITNA -.07 .08 1.06

Multiplechoicereading ITNA -.08 .09 1.40

Multiplechoicelistening ITNA -.15 .08 1.01

Dictation(fillinthegaps) ITNA -.18 .09 1.71

Argumentativewriting-from-reading STRT -.51 .10 .42

Argumentativewriting-from-listening STRT -.59 .11 .83

Summarystatistics:Candidate:Model,Random(normal):X2(115)=98.4,p=.87Task:Model,Fixed(allsame):X2(12)=164.5,p=.00

SpeakingcomponentInordertoinvestigatethesimilaritiesanddifferencesbetweenSTRTandITNA,amultiple linear regression analysis was run in which all STRT criteria wereregressedontotheITNAscores.Theregressionmodelexplains28%oftheITNAscorevariance(p<.001).ThelowadjustedR2(.28)hintsatapossiblediscrepancyintheratingofSTRTandITNA.Whenhighlysimilartasksandcriteriaareusedtoscorethesamepoolofcandidates,onewouldexpecttheregressionmodeltobeabetterfit.Consequently,inordertodeterminetowhatextentcorrespondingcriteriafitthesameunderlyingconstructs,aPCA(seeTable4.6)wasconductedonthestandardizedz-scoresoftheoraltestcomponents.


92

Table4.6.Promax-rotatedfactorloadings

TC1 TC2 TC3

STRT

Vocabulary .92 … …

Cohesion .85 … …

Grammar .80 … …

Fluency .70 … …

Pronunciation .68 … …

ITNA

Vocabulary … .97 …

Fluency … .81 …

Grammar … .75 …

Cohesion … .58 …

Pronunciation … … .96

Eigenvalue 3.32 2.54 1.16

Proportionexplained

.47 .36 .17

Note.Factorloadings≤.3wereomitted.

The PCA confirmed that the ratings of corresponding criteria do not match;corresponding criteria in STRT and ITNA do not load onto the same factors,which one would expect if they measured the same underlying constructs.Instead, all STRT criteria cluster together, as do all ITNA criteria, except forPronunciation.

Finally,thefirstRaschmodel–inwhichthecriteriawereweightedequallyinordertoallowforaclearcomparison–reliably(.98) identifiessevendistinctdifficulty levels in the STRT and ITNA criteria. In this model, the oralcomponents of both tests cannot be reliably separated in terms of difficulty(reliability.00;X2(1)=.2,ns),butthemeasuresofthecriteriarevealsometellingmismatches. Table 4.7 shows that Pronunciation in ITNA ranks as markedlydifficult, andContent in STRT’s argumentation task is disproportionately easy.TheMFRA further shows that, except forVocabulary, correspondingSTRTandITNA criteria invariably belong to a different difficulty band. Thiswouldmostlikelynotbethecaseifbothtestsinterpretedcorrespondingcriteriainthesameway. As such, the first MFRA confirms the PCA and adds an extra layer of


93

information to it; corresponding criteriaprobablymeasuredifferent constructs,and–exceptforVocabulary–dosoatdistinctlydifferentdifficultylevels.Table4.7.MFRAoralcriteria(equalweights)

Criterion Test Measure SE InfitMnSq

Pronunciation ITNA 1.34 0.20 1.49

Fluency STRT 1.09 0.20 1.06

Coherence STRT 0.65 0.20 1.13

Grammar STRT 0.57 0.20 0.92

Contentpresentation STRT 0.40 0.21 1.02

Pronunciation STRT 0.23 0.21 0.97

Vocabulary ITNA -0.35 0.22 0.91

Vocabulary STRT -0.35 0.22 0.97

Initiative STRT -0.40 0.22 0.70

Coherence ITNA -0.89 0.23 0.80

Fluency ITNA -0.94 0.23 1.04

Grammar ITNA -1.37 0.23 0.45

Register STRT -1.69 0.24 0.80

Contentargumentation STRT -5.52 0.39 1.85


TheMFRA,with equallyweighted criteria, offers a clear picture of the relativedifficultyofthecriteria,butinreality,notallcriteriaareweightedequally.STRTweighs linguistic criteria double to compensate for the amount of contentcriteria,andITNA–havingnocontentcriteria–assignsaweightof2and1.6toGrammar andVocabulary. A Rasch model that uses the actual weights of thecriteriareliably(.88)identifiesITNAasthemostdifficultbyhalfalogit.Applyingactualweightsinsteadofequalonesalsochangestherelativeorderofthecriteria(see Table 4.8). All corresponding criteria now appear in distinctly differentdifficultybands;GrammarinITNAbecomesthemostdifficultcriterionbynearlytwo logits, andContent in the STRT argumentation task is roughly four logitseasier than the second easiest criterion, misfitting the Rasch model, as doesPronunciationinITNA.


94

Table4.8.MFRAoralcriteria(actualweights)


Grammar ITNA 2.89 0.33 1.11


Vocabulary ITNA 0.79 0.21 1.25



Grammar STRT 0.43 0.14 0.79

Pronunciation STRT 0.02 0.15 0.88

Content(presentationtask) STRT -0.13 0.21 1.06


Initiative STRT -0.55 0.15 0.76


Fluency ITNA -1.74 0.16 1.01

Register STRT -1.76 0.16 0.79

Content(argumentationtask) STRT -5.71 0.44 1.64


DISCUSSIONFlemish universities that accept STRT and ITNA certificates as equivalentmeasuresofB2proficiencymakeaclaimaboutlevelequivalence.Validatingsuchahigh-stakes claim requires strong empirical evidence (Kane, 2013),whichhadnot been presented to date. Flanders is not exceptional in this sense; inmanydifferent countries, university entrance policies contain similar claims of testequivalence, often without substantiation (McNamara & Ryan, 2011). For thatreason, oneof thewider aimsof this studywas tohighlight the importanceofempiricallyexaminingunfoundedclaimsoflevelequivalence.

This study showed the correlation (r = .77**) between overall STRT andITNA scores to be rather strong and comparable to coefficients reported inprevious research (e.g., ETS 2010 (r = .73); Zheng & De Jong, 2011 (r = .75)).However, since overall correlations can be misleading (Lissitz & Samuelsen,


95

2007),supplementaryanalyseswereconducted,allofwhichrevealedthatSTRTandITNAdonotmapontoeachotherquiteasseamlessly.

When considering only the interquartile range, the correlation betweenSTRTandITNAscoresbecomesvirtuallyzero,indicatingthattesttakerswithoutadistinctlystrongorweakprofilewereassesseddifferentlybySTRTandITNA.Thishypothesisisconfirmedbyananalysisofthedescriptivedata,whichshowssignificantlylowermeanscoresonITNAthanonSTRT.Perhapsthemosttellingpiece of evidence to discount the claim of level equivalence is the significantdifference in pass-fail judgments; 24% of the participants received a differentoutcome on STRT and ITNA. Inmost cases of disagreement, candidates failedITNA but passed STRT. Accordingly, the probability of passing STRT issignificantlyhigherthantheprobabilityofpassingITNA.

The evidence presented in this study casts doubt on any claim of levelequivalence. Apart from the overall correlation, none of the analyses offeredevidenceinsupportofthepolicyclaim,which,asaresult,cannotbeconsideredvalid (Kane, 2013). Moreover, following Phillips’s decision rule (Kane, 2013;Phillips,2007),theuniversityentrancepolicyisunlikelytosolvetheproblemitwasintendedtofix,thatis,assuringaconsistentminimumlanguagelevelamongtheinternationalstudentpopulation.

These results have implications beyond the immediate research context.First,theyreaffirmthedangerofassumingthatdifferenttestslinkedtothesameleveloftheCEFRareequallydifficult(Green,2017).Policymakersoftenrelyontheselevelswhendetermininglanguagerequirements(Fulcher,2012b),butsinceCEFRlevelsarebroadandleaveroomforinterpretation(Alderson,2007;Fulcher,2004),directcross-testcomparisonofthekindthatwaspresentedinthisstudymightbeasafer,morerobustoption.Thisstudyindicatesthateventestswhichhave been linked to the same CEFR level may differ substantially in pass/failjudgments. Secondly, this study confirms the criticism raised against usingcorrelationaldatainvalidationresearch(Lissitz&Samuelsen,2007;Norris,2016),and underscores the importance of supplementing purely correlational resultswith impactdata (i.e.,pass/faildecisions)andwith informationconcerning thenatureofarelationshipbetweentwovariables(i.e.,constructequivalence).

The construct equivalence analyses indicate substantial differencesbetweencomparableSTRTandITNAtasks.STRTtaskscoresexplain48%ofthescore variance on ITNA’s written component, and STRT’s argumentative taskscontributelesstotheregressionmodelthanthesummarizationtasks.Thisisnotentirely surprising, because argumentative tasks require knowledgetransformation,whereassummarizationtasksrelyonrepetitionandaremoreinline with the multiple-choice items found in ITNA. The MFRA identifies theargumentativeSTRTtasksastheeasiesttasksinbothtests,whichcouldexplainwhytheycontributelittletotheregression.Thisobservation,combinedwiththefact that ITNA’s vocabulary and grammar tasks are nearly half a logit more


96

difficult than the most difficult STRT task, explains why ITNA’s writtencomponentisthemostdifficult.

EventhoughtheoralcomponentsofSTRTandITNAarehighlysimilarinterms of task types and rating criteria, the evidence in favor of constructequivalence is weak. In the multiple linear regression analysis, STRT criteriaexplained just 28% of the ITNA score variance, the PCA showed thatcorrespondingcriteriafrombothtestsdonotloadontothesamefactor,andtheMFRAwithequallyweighedcriteriamappedmostof thecorrespondingcriteriainto different difficulty bands. The analyses all indicate that correspondingcriterialikelymeasuredifferentconstructs.Moreover,anMFRA,usingtheactualweights of the criteria, reliably shows that the oral component of ITNA is thehardestbecause it assigns thegreatestweight tocomparativelydifficult criteria(GrammarandVocabulary),andbecausealargeproportionoftheSTRTscoresisderivedfromrelativelyeasycontentcriteria.Importantly,thedifferencebetweenSTRT and ITNA only becomes substantial when the actual weights areoperationalizedintheRaschmodel.ARaschmodelwithequalweightsforeverycriterion did not reliably differentiate between the difficulty levels of the twotests.

It is important to note that ITNA’s greater level of difficulty does notautomatically imply that it is a better university entrance language test.Answering that question requires a different set of data. The data used in thisdissertation do provide substantial evidence to argue that STRT and ITNAmeasure different constructs, even when the tasks and the criteria are highlysimilar.Additionally,theanalysesshowthattherelativeimportanceassignedtocontent,grammar,andvocabularyinSTRTandITNAisthemostlikelycauseofthedifferingdifficultylevelsintheoralandthewrittencomponents.In terms of justice, the situation in Flanders can be improved, perhaps evenwithout invoking drastic measures. It may suffice to extend to test takers thesame service that customersofotherpaid services receive.Beforepurchasingaservice, people typically request and receive detailed information about it,allowingthemtomakeabalancedchoice.Iftheinformationtheyreceivedpriortopurchasewasmisleading,customershavetherighttocomplainandrevokethecontract.The samecouldapplywhenpeople select ahigh-stakes testbasedoninaccurate information or unfounded assumptions. The need “to provide allpotentialtesttakerswithadequateinformationaboutthepurposesofthetest,theconstruct(orconstructs)thetestisattemptingtomeasureandtheextenttowhichthathasbeenachieved”(ILTA,2007,p.3,emphasisadded)isnotanewconcern,but it is one that deserves renewed attention. It would benefit the justice oftesting policies if test takers received accurate information about the actualdifferences and similarities between tests that are presented as equivalentoptions.


97

CONCLUSION:ASSUMPTION3

The data presented in this chapter do not indicate that the STRT and ITNAscorescanbeconsideredequivalent.Roughlyoneinfourparticipantsreceivedadifferentpass/fail score. ITNAappears tobe theharder test, largelybecauseofthelanguage-in-usetasks,andbecauseoftherolevocabularyandgrammarplayinthetestconstruct.Thenextchapterzoomsinonequivalenceofcorrespondingratingcriteriaintheoraltestcomponents.

Chapter4:Criterionequivalence

100

CHAPTER4CRITERIONEQUIVALENCEThis fourth chapter focuses specifically on equivalence of the STRT andITNA components that are most comparable. It examines howcorresponding CEFR-based criteria that are operationalized in highlycomparablespeakingtasksmapontoeachother.

Prior to 1800, when Henry Maudslay developed the first standardized screwthread, nuts and boltswere not easily interchangeable.Maudslay introduced acommonstandard,andchangedthelifeofeveryplumbertothisday.Standardshelp to achieve transparency, uniformity and interchangeability – at least inhardware.Languageperformance,inallitsidiosyncraticandcontextualvariation,doesnoteasilypermitsuchstandardization.Nevertheless, fairandvalid testinghingesupon score comparability and score transparency (Kane, 2013). The firstconceptimpliesthatscoresonteststhattargetthesameaudienceandsharethesame purpose can be meaningfully compared. The second concept – scoretransparency – entails that test scoreshave clearmeaning tousers.Candidatesand admission officers need to be able to meaningfully interpret test scores(Alderson,1991),andratersneedtohavethesameconceptionofthesamelevel.This makes it all the more striking that, in language testing standards areconstantinonesense:theyvary.Currently,numerouslanguageperformancestandardsexistalongsideeachother.TheACTFLstandardsresultedfromnationalefforts(ACTFL,2012),theSTANAG6001 standards are usedwithin supranational organizations (NATO, 2014), andstillotherstandardshavebeendevelopedbytestingorganizationsintheformofratingscales.Sincedifferentorganizationsusedifferentscales, it isnoteasy fortest takers or test users to interpret scores and compare themwith other tests(Gomez,Noah,Schedl,Wright,&Yolkut,2007).InMessick’s(1989)approachtovalidity,whichconsidersscoreuseessentialtoavalidityargument,alackofscoretransparencypresentsaproblem,whichcouldpotentiallyberesolvedifalltestswere linked to the same universally accepted levels and standards ofperformance. In Europe and beyond, the CEFR (Council of Europe, 2001) iswidely considered as such a standard (see Chapter 1). The CEFR’s uptake hasbeenwidespread,but ithasnotbeenempiricallyvalidatedinevery intendedorunintendedcontextofuse.

ThischapterexaminestheuseandusefulnessoftheCEFRinthecontextof rating scale design. More specifically, it contributes evidence regardingAssumption3byexploringtowhatextentthesameCEFRdescriptorshavebeen


101

similarly operationalized and interpreted in the oral components of twouniversityentrancelanguagetests.

CRITERIONEQUIVALENCEANDTHECEFRSince its publication in 2001, the CEFR (Council of Europe, 2001) has becomewidelyusedandadoptedbytestdevelopers,policymakers,teachers,publishersandcandidatesalike.Ithascometobeseenasacommoncurrencyinlanguageperformance levels (Figueras, 2012), and in Europe it is now the leadingframework in language testing (Figueras, 2012; Little, 2007; Papageorgiou, Xi,Morgan,&So,2015).TheCEFRissoinfluentialthatithasbecomenecessaryfortests to link to it in order to gain recognition within Europe (see Chapter 1).Outside of Europe too,many scoring systems andperformance standards havebeenmappedontotheCEFR(e.g.,Bärenfänger&Tschirner,2012;Baztán,2008;Tschirner, Bärenfänger, &Wisniewski, 2015 for ACTFL; Tannenbaum &Wylie,2008forTOEFLiBT;Swender,2010forSTANAG6001;Zheng&DeJong,2011forPTEAcademic; also seeGreen, this issue). Theoretically, a framework that hasreceived such wide recognition by all parties involved could address the scoretransparencyandcomparabilityconcernsKane(2013)andAlderson(1991)raised.Inpracticehowever,thereareissues.

EventhoughthegoalsoftheCEFRinitscurrentformaredescriptive,notnormative (North,2014a),achievingscorecomparabilityacross testswasoneofthe primary goals of its earliest drafts (van Ek, 1975: 8). Today too, in manyEuropean contexts, theCEFR level descriptors are used in a normativeway, asperformance standards, or as labels to facilitate score transparency (Roever &McNamara, 2006; O’Sullivan & Weir, 2011; Fulcher, 2012). With scoretransparency inmind,many test developers are using CEFR descriptors as thebasis for rating scale development, but even though treating the CEFR as aheuristic is common practice (Weir, 2005b; North, 2014a; 2014b), it is notunproblematic. First, the CEFR offers guidance on essential test developmentmatters,suchastestpurpose,responseformat,timeconstraints,andtopic(Weir,2005b). Two tests could have the same CEFR level, but very differentspecifications,anditwouldbewrongtoconsiderthemequivalentsimplybecausetheyshareaCEFRlabel(Green,2017;Taylor,2004).Secondly,becausetheCEFRiscontextandlanguage-independent,testdevelopersneedtoaddspecificdetailstothedescriptorswhenusingitinaratingcontext(Harsch&Martin,2012).ThisnecessarystepmaycausetwoteststodeviateintheirinterpretationoftheCEFRlevels, resulting in reduced comparability. In fact, CEFR descriptors have beencriticized for theirvaguenessand inconsistencies,bothwithinandacross levels(Alderson,2007;Harsch&Rupp,2011;Papageorgiou,2010)andmaysuffer from“descriptionalinadequacy”(Fulcher,Davidson,&Kemp,2011:8),leavingroomfor


102

dissimilarinterpretations.Sincethereisampleevidencethateventrainedratersinterpret the same test-specific criteria differently (Deygers & Van Gorp, 2015;Lumley,2002;2005)andthatalsotrainedraters’experienceandbackgroundmayinfluence the score that is assigned (Barkaoui, 2011), there canbenoguaranteethat different test developers interpret the sameCEFR descriptors in the sameway.Tothebestofourknowledge,nostudyhasyetcomparedtheratingsoftwohigh-stakestestsusingcorrespondingCEFR-basedcriteria.

Nonetheless,quitea fewstudieshavediscussedratingscaleconstructioninrelationshiptotheCEFR(Galaczi,ffrench,Hubbard,&Green,2011;Harsch&Martin, 2012; Papageorgiou, 2015). These studies typically discuss fitting CEFRdescriptors to rating scale logic by rectifying descriptor vagueness and bystraighteningblurred linesbetweenlevels(Alderson,2007;Papageorgiou,2010).Inaddition,Galaczi et al. (2011)havehighlighted thepositivewordingofCEFRdescriptorsandthebrevityofcertainCEFRscalesasmattersofongoingconcernduring rating scale construction and rater training.Deygers&VanGorp (2015)showedthataCEFR-basedratingscalethatwasiterativelyconstructedtogetherwithratersdidnotguaranteeauniforminterpretationofthedescriptors,inspiteof high inter-rater reliability indices. In this regard, Harsch and Rupp (2011)rightfullystressedtheneedforahighlevelofanalyticdetailinCEFR-basedscalesinordertocompensateforthebroadnessoftheinitialdescriptors.

The abovementioned studies show how individual test developers haveoperationalizedCEFRdescriptorsintheirratingscalestofitthepurposeofatest.OtherdocumentsdescribehowsometestshavebeenalignedwiththeCEFR(e.g.,Tannenbaum&Wylie, 2008; Khalifa & ffrench, 2009; De Jong, Becker, Bolt, &Goodman,2011forTOEFLiBT,IELTSandPearsonPTErespectively).Green(2017)hasscrutinizedsomeofthesereportsinanefforttounderstandthevaryingwaysinwhichthemajorEnglishtestshaveestablishedtheirCEFRlinks.Heuncoveredthat the linking methodologies used by these high-stakes university entrancelanguage testsdivergedso substantially that itwouldbemisguided topresumecomparabilityoftheB2levels.

To date, little if any CEFR research has been comparative. Concurrentanalysesthemselvesarenotnewtothelanguagetestingendeavorhowever,andhave actually been central to test validation (Chapter 3 contains an overview).Nevertheless,nostudies,concurrentorotherwise,haveanalyzedempiricaldatatocomparetheinterpretationandoperationalizationofCEFRdescriptorsacrosstests. Nevertheless, it could be argued that exactly this kind of researchdetermines whether the CEFR can act as a catalyst for increased scoretransparency and score comparability,whichwasoneof its original goals (VanEk,1975).Thecurrentstudythusaddressesanimportantgapintheliteraturebyexamining the potential of the CEFR in facilitating score transparency in teststhat share the same purpose, the same population and employ correspondingratingcriteriathatarebasedonthesameCEFRdescriptors.


103

RESEARCHQUESTION

The study described in this chapter examines to what extent correspondingSTRTandITNAcriteria leadtothesamescoresforthesamecandidates inthesameway.RQ Dothe two testsapply thesameCEFR-based leveldescriptors in thesame

wayforsimilartasktypes?InordertosystematicallyanswerthisRQ,foursubquestionswereidentified:a. HowmuchdotheSTRTandITNAcriteriadeviatefromtheoriginalCEFR

descriptors?b. Can corresponding CEFR-based levels in both tests be considered truly

equivalent?c. Are corresponding CEFR-based criteria likely to measure the same

construct?d. ArecorrespondingCEFR-basedcriteriaequallydifficultinbothtests?IfbothtestsinterpretthesameCEFRdescriptorsinthesameway,highoverallcorrelations should carry through down to the criterion level. If the samecandidates are rated highly dissimilarly on corresponding criteria, using theCEFRasa standard for score transparencymaynotbewarranted, since itmaycreateafalsesenseofuniformity.

PARTICIPANTS&METHODOLOGYThe analyses in this chapter are based on the scored performances of 82 L2FparticipantswhotooktheoralcomponentsofSTRTandITNAwithinthesameweek(seeChapter3).STRT&ITNAratingscalesTypically,theoralSTRTorITNAcomponentsdonottakemorethan25minutes,including preparation time (see Appendix 1 and 2). In both tests, candidatesinteractwithatrainedexaminerduringtheoralcomponent,whichconsistsofapresentation and an argumentation task. The argumentation task invites thecandidatestoweighanumberofalternativesolutionstoaproblem,andtoarguewhytheirchoiceisthebetterone.

Fiveoralratingcriteriaareincludedinbothtests:Vocabulary,Grammar,Coherence, Pronunciation, and Fluency. For scoring these criteria, both tests


104

employanalyticbanddescriptorsthatarebasedontheA2,B1,B2,andC1levelsinthecorrespondingCEFRscales.InbothteststhecutofflevelforeachcriterionisB2, except for Pronunciation andGrammar, where the ITNA uses B1 and B2+respectively.Bothtestsdevelopedtheirratingscalesbydrawingontheoriginaldescriptors, but bothmade choices based on their interpretations of theCEFRdescriptors.ITNAratingscaledesignersoftencopiedtheoriginalCEFRtextandsupplemented it with language-specific examples, identifying typical errors ofusersatagivenlevel.TheSTRTcriteriondescriptorsdeviatedfromtheoriginalwordingmoreoften,inordertomaketheoriginaldescriptorsmoreconcreteandeasiertograspforthenoviceraterstheyoftenemploy(seeDeygers&VanGorp,2015).

TheoralITNAperformancesarescoredimmediatelyafterthetestbytwotrainedraterswhocometoonecompositescoreforeachofthefivecriteria.ITNAexaminersandraterstendtobeexperiencedL2teachersofDutchwhotypicallyattend training at least once a year and score oral tests at different timesthroughouttheyear.TheSTRTperformancesarerecorded,andsubsequentlytwoindependenttrainedraters–whoareusuallynoviceraterswithabackgroundinlinguisticsorcommunication–separatelyscoreeachtask.Theyreceiveatwo-daytraining,andtakepartinatrialratingsessiontoestablishtheirconsistencyandreliability.DataanalysisDeterminingwhetherbothtestsapplythesameCEFR-basedratingcriteriainthesame way required recoding certain scoring categories. The STRT rating scaledistinguishedfourproficiencylevels(A2,B1,B2,andC1),buttheITNAhadsixorseven, since it includes the plus levels of B1, B2 and occasionally A2. AfterconsultingwiththeITNAcoordinators,thesedoublebandsweremergedinordertocometoafour-bandscale,whichfacilitatesdirectcomparisonsacrossscales.Below,theanalysesarediscussedbysubquestion:How much do the STRT and ITNA criteria deviate from the original CEFRdescriptors?The STRT and ITNA rating criteria were compared to each other and to theDutch translation of the CEFR, using the Jaccard similarity index. The indexprovidesaverysimplequantificationofthesimilaritybetweendescriptors.Ithasbeen applied in different forms in the field of information retrieval and textcomparison(Manning,Raghavan,&Schütze,2009).TheJaccardindexexpressesthe similarity between two descriptions as their overlap in terms, morespecificallytheratioofthenumberofuniquetermspresentinbothtextsandthenumber of unique terms in either of the texts. The Jaccard indexbecomes 1 if


105

both texts use the exact same set of words, independent of how often thesetermsarerepeated,anditdecreaseswhenthetermsusedinbothtextsdiverge.

In order tomake the comparison of the descriptors used in this studymore robust, the texts were automatically pre-processed using a standardstemmingalgorithmforDutch.Thismeans thatallwordswerestemmed(e.g.,pluralendingsremoved)andallnon-informativewords(suchas“of”and“with”,asdefinedbythePythonNLTKDutchstopwordlist)wereremoved.

CancorrespondingCEFR-basedlevelsinbothtestsbeconsideredtrulyequivalent?

In order to determine the equivalence of the same levels in correspondingcriteria, frequency distributions of CEFR-based scoreswere supplementedwithprobabilityestimatesofattainingtheB2level.Foreverycriteriontheprobabilityof attaining a score of B2 or higher was estimated. The strength of therelationshipbetweencorrespondingcriteriawascalculatedusingKendall’sTau.To determine the level of agreement between the two tests, linear weightedkappa (Kw)was used. Kw is a variation on Cohen’s kappa,whichmeasures thelevel of agreement betweenordinal data sets (Sim&Wright, 2005),whereby 0indicatesnoagreementexceptonestemming fromchance,andvaluesabove .8canbereadasalmostperfectagreement(Landis&Koch,1977;Vanbelle&Albert,2009).Usually,weightedkappaisusedtodeterminerateragreement,butinthisstudyitservedasanadditionalmetrictodeterminewhethertheSTRTandITNAratersscoredthesamecandidatesinthesamewayforcorrespondingcriteria.

ArecorrespondingCEFR-basedcriterialikelytomeasurethesameconstruct?

TheITNAratingscaleconsistsoffivecriteria:Vocabulary,Grammar,Coherence,Pronunciation,andFluency.ThesecriteriaalsooccurintheSTRTratingscale,inaddition to others (e.g., Content, Register). If the STRT and ITNA ratersinterpreted the same criteria in the sameway, the STRT scores on the sharedcriteria would explain a large proportion of the total ITNA score variance. Inorder to investigate this, three multivariate linear regression models wereconstructed.Eachmodeltookthefollowinggeneralform:

ITNATotal=(b0+b1criterion1i+b2criterion2i+…+bncriterionni)+εi

Threeregressionmodelswererun,andcomparedintermsofR2usinganAnova.ThefirstregressionmodelincludedthefivecriteriafromthetwoSTRTtasksthatsignificantly correlated with the ITNA criteria at τ > .3. The second modelincludedthesevencriteriathatsignificantlycorrelated,regardlessofthestrengthof the correlation. The final model included all the STRT criteria. Prior torunningtheregressionanalyses,theassumptionswerechecked:Theproportion


106

of cases with large residuals was acceptable (4% in the oral component afterremovaloftwooutliers),Cook’sdistancewas<1,nocaseswerelargerthanthreetimes the average leverage, the covariance ratio was satisfactory, and themulticollinearity and independence assumptions were supported (Norris 2015;Purpura,Brown,&Schoonen2015).

Next, to determine the relationship between individual criteria, a linearregressionmodelwas constructed for every ITNAcriterionas a functionof thesameSTRTcriterion.

ArecorrespondingCEFR-basedcriteriaequallydifficultinbothtests?InamultifacetedRasch(MFRA)measurementanalysis,atestscoreisseenastheresult of an interactionbetweendifferent facets, such as test-taker ability, taskdifficulty,raterseverityandcriteriondifficulty(McNamara, 1996).InMFRAtheeffectofallthesevariablesonthescoreistakenintoconsiderationandmappedontothesamelogitscale.Inthisstudy,allcomparableSTRTandITNAratingsforthe same candidates were combined in the sameMFRAmodel and the Facetsprogramwas used to estimate criterion difficulty.Of interest are the difficultymeasures(ahighermeasureindicatesamoredifficultcriterion),thestrataindex(which showswhether differentmeasures also translate into different levels ofdifficulty that can be separated reliably) and the fit statistics (InfitMnSq). Thecloserthevalueofthesefitstatistics isto1, thebettertheobserveddatafit theRaschmodel.Acriterionthathasfitstatisticsintherangebetween.50and1.5isconsidered to have an acceptablemodel fit. Lower values indicate overfit (i.e.,redundancy)andhighervaluesindicatemisfit(Linacre,2012;Barkaoui,2014).

TheanalyseswereconductedusingR(psych,irr,Hmisc,QuantPsyc,car,and ggplot2 packages), Facets (Linacre, 2015), and Python (with the NLTKlibrary).

RESULTSThe Jaccard index (see Table 5.1) shows that on thewhole thewording of theITNAcriteriastaysclosertotheexactwordingoftheCEFRthanthewordingoftheSTRTcriteria.Forexample,thelowestJaccardindexforITNA~CEFRis.27,butthreeoutof fiveITNA~CEFR indicesaresubstantially lowerthanthat(J<.10).Sincethewordingofthedescriptorsinbothtestsdeviatessubstantiallyfromthe CEFR original, it is logical that the overlap between the STRT and ITNAdescriptorsistypicallynottoobig.TheratingscaledescriptorsofPronunciationinbothtestsstayclosest to theCEFRwording,asa resultofwhichtheoverlapbetweentheSTRTandITNAdescriptorsisthelargestforthiscriterion(J=.44).


107

Table5.1.Jaccardindexforratingdescriptorpairs ITNA~CEFR STRT~CEFR ITNA~STRTVocabulary .53 .15 .10Grammar .30 .10 .06Coherence .27 .09 .08Pronunciation .80 .40 .44Fluency .88 .26 .29Forreasonsofconfidentiality,thefullratingscaledescriptorscannotberepeatedin their entirety, but a few examples, taken from confidential STRT and ITNArating scale documents, may serve to illustrate how CEFR descriptors wereparaphrased.

TheB2criterionforVocabularyinITNAaddstotheCEFRdescriptor:“hasa good range of vocabulary for matters connected to his/her field and mostgeneraltopicsanddoesnotonlyusehigh-frequencywords”(myitalics).TheSTRTvocabulary criterion, on the other hand, reads: “the lexical variation in theperformanceissufficienttopreventfrequentrepetitionofwords”.Inbothtests,thedescriptorsforCoherenceincludeadditionstotheCEFRwording.TheITNAfocuses on the sentence level: “sentences are linked logically and appropriateconnectors are usedwhen required”. The STRT raters are required to considerthetextlevelaswell:“Theperformanceisonecoherentwhole(…)connectorsaremostlyusedcorrectlyandsupporttheoverallcoherence”.PronunciationinSTRTrepeatstheoriginalB2descriptor,butitissupplementedwithaB1characteristic:“Thepronunciation is clearandnatural,butwith a foreign accent” (my italics).ThisadditiondoesnotoccurintheITNAratingscale,wherethecutoffpointforPronunciation isB1,notB2.TheFluencydescriptorintheITNAratingscalehasbeen literally copied fromtheCEFR: “canproduce stretchesof languagewithafairly even tempo; although he/she can be hesitant as he/she searches forpatternsandexpressions,therearefewnoticeablylongpauses.”Afewadditionsweremade in theSTRTdescriptor: “canproduce stretchesof languagewithaneven tempo; although he/she can be hesitant as he/she searches for the rightexpression,therearefewnoticeableordistractingpauses”(myitalics).After examining the correspondence in the wording of the rating criteria, thescores were analyzed. It was determined whether the same levels ofcorresponding criteria in the STRT and ITNA rating scales can be consideredequivalent. Table 5.2 shows how often CEFR levels were assigned to the samecriteriaintheITNAandinbothSTRTtasks(coherenceisnotaratingcriterionintheSTRTargumentationtask).InmostcasesthemodecorrespondswiththeB2level.Inotherwords,onmostcriteria,mostcandidatesscoredB2(e.g.53ITNAtest takers scoredB2onVocabulary, and8scoredC1).ForGrammarSTRTpres and


108

PronunciationITNA,B1wasthelevelmostoftenassigned.NocandidatescoredA2for GrammarITNA or CoherenceITNA (ITNA assigned 0 A2, and 16 B1 ratings onCoherence,whileSTRTscored9performancesA2and31B1).Table5.2.FrequenciesofassignedCEFRlevels A2 B1 B2 C1Vocabulary ITNA 5 16 53 8 STRTarg 3 17 40 22 STRTpres 7 20 40 15Grammar ITNA 0 4 67 11 STRTarg 4 24 44 10 STRTpres 5 34 33 10Coherence ITNA 0 16 49 17 STRTpres 9 31 31 11Pronunciation ITNA 11 40 23 8 STRTarg 2 24 45 11 STRTpres 3 32 37 10Fluency ITNA 1 17 50 14 STRTarg 5 27 36 14 STRTpres 12 30 37 3FormostcriteriatheprobabilityofanygivencandidatetoattainascoreofB2ormore was found to be higher on ITNA than on either of the STRT tasks (seeTable5.3).Table5.3.ProbabilityofattainingB2orhigheronSTRTandITNA 𝑃!!

!"#"$!" p# 𝑃!!!"#$ p## 𝑃!!!"#"$%&'

Vocabulary .76 1.0 .74 .362 .67 Grammar .66 .000 .95 .000 .52 Coherence .80 .000 .51 Pronunciation .68 .000 .38 .009 .57 Fluency .61 .013 .78 .000 .49

Note.p#:p-valueforthedifferenceinprobabilitybetweenP!"!"#"$%&andP!"!"#$

p##:p-valueforthedifferenceinprobabilitybetweenP!"!"#"$%&'andP!"!"#$

Thep-valuesinTable5.3refertothedifferenceinprobabilitybetweenthetwoSTRT tasks and the ITNA results. A given candidate has a 38% probability ofattainingaB2pronunciationscoreonITNA.Thesamecandidatemay,however,have a 68% probability of being rated B2 on the same criterion if he or sheperforms the STRT argumentation task. The difference between theseprobabilities is significant. In fact, vocabulary excepted, there is a consistent


109

significantdifferencebetweentheprobabilityofattainingascoreofat leastB2ontheITNAtasksoroneoftheSTRTtasks(p<.05).ThisindicatesthattheB2threshold is interpretedoroperationalizeddifferentlyon theSTRTandon theITNAtest.

The frequencies in Table 5.2 show regular discrepancies in STRT andITNAjudgmentsandtheprobabilitiesinTable5.3indicatethatSTRTandITNAjudgmentsdiffer in severity fromone criterion to thenext (e.g., aB2 scoreoncoherenceissignificantlyhardertoreachonSTRTthanonITNA).Assuch,therelikelyisadifferentdistributionintheITNAandSTRTscoresforcorrespondingcriteria.

Table5.4showstheITNAscoreonbothtaskscombinedinrelationtotheSTRT argumentation task and the STRT presentation task. The relationshipbetweenthecorrespondingSTRTandITNAcriteriaisgenerallymediumtolow(τ < .39**) and the agreement is generally weak (kw < .22). This implies thatcorrespondingSTRTandITNAcriteriamightnotmapontoeachotherwell.Therelationship is weakest for Vocabulary and Pronunciation. The correlationbetween the sumsof these five corresponding criteria ismoderate aswell (τ =.37**).Table5.4.RelationshipbetweencorrespondingSTRTandITNAcriteria ITNA~STRTarg ITNA~STRTpres τ kw τ kw Vocabulary .153 .031 .212* .091 Grammar .336* .208*** .351** .184*** Coherence .386** .216*** Pronunciation .117 .122* .212* .207** Fluency .336** .215** .315** .134* Note. *=p<.05,**=p<.01,***=p<.001

Overallcorrelationforsummedcriteriaτ=.37**Havingdeterminedthatthesamelevelsincorrespondingcriteriaareunlikelytobe equivalent, multivariate linear regression was used to determine to whatextentscoresonthecriteriathattheSTRTshareswiththeITNA,predictedtheITNA scores. Threemodelswere run.The first included the five STRT criteriafrom the two STRT tasks which significantly correlated at τ > .3. This modelexplained26%of the ITNAscorevariance (R2

adj= .2585,p< .000).Thesecondmodel included the seven STRT criteria that correlated significantly with thecorrespondingITNAcriterion,regardlessofthestrengthoftherelationship.Thesecondmodelaccountedfor27%ofthetotalITNAscorevariance(R2

adj=.2706,p<.000),butdidnotsignificantlyimprovethemodelfitofthedata,incomparisonwith the firstmodel (F(2, 74) = 1.6334,p <.000). The thirdmultivariate linearregressionmodel includedallnineSTRTcriteria thathadcorresponding ITNA


110

criteria,andexplained26%of the ITNAscorevariance(R2adj= .2603,p< .001).

Sincethismodelwasnotasignificantlybetterpredictorthanthefirst(F(4,72)=1.0476,p < .39)or the second (F(2, 72)=0.4828,p < .62),Table 5.5 shows theregressionresultsofthefirstmodel.Itindicatesthatonlyonepredictor(Fluency,in the argumentation task) significantly contributes to the model. Given thesample size, the estimates of the individual predictors should be treated withsome caution, but generalizing from the overallmodel fit can be donewith adegree of confidence (Field, Miles, & Field, 2012). All things considered, themultiple regression analyses show that no more than 27% of the ITNA scorevariancecanbeexplainedbyscoresoncorrespondingSTRTcriteria.Table5.5.Multivariatelinearregression:ITNAtotal~STRTarg+pres B SEB β p(Constant) .457 3.811 .905GrammarSTRTarg 1.547 1.851 .132 .406GrammarSTRTpres 2.633 2.056 .241 .204FluencySTRTarg 3.709 1.436 .375 .012*FluencySTRTpres -2.251 1.604 -.222 .165CoherenceSTRTpres 1.204 1.365 .128 .381Note.TotalR2adjustedis.2585(p<.000).When corresponding ITNA and STRT criteria were used in a pairwise linearregression (see Table 5.6), the same trend emerges. Regression models withstatisticalsignificance(p<.05)basedontheSTRTcriterianeverexplainedmorethan17.2%ofthescorevarianceincorrespondingITNAcriteria.


111

Table5.6.Linearregressiononcriterionlevel:ITNAtotal~STRTtotal 𝑟!"#! B SEB β p

Vocabulary .022 .156 (Constant) 2.279 .336 .000 STRTarg -.076 .211 -.081 .719 STRTpres .252 .200 .281 .211Grammar .172 .000 (Constant) 2.306 .189 .000 STRTarg .122 .103 .196 .239 STRTpres .156 .096 .267 .109Coherence .170 .000 (Constant) 2.148 .215 .000 STRTpres .325 .077 .425 .000Pronunciation .067 .017 (Constant) 1.664 .400 .000 STRTarg -.339 .250 -.276 .179 STRTpres .605 .240 .513 .014Fluency .133 .001 (Constant) 1.983 .261 .000 STRTarg .213 .127 .259 .098 STRTpres .135 .130 .160 .305Theresultsoftheregressionanalysesaboveindicatethatcorrespondingcriteriaare unlikely to measure the same construct at the same level. To ascertainwhether the same criteria are equally difficult in the two tests, amultifacetedRaschanalysiswascarriedout,basedonall corresponding ratingcriteria.Thismodel(seeTable5.7)reliablyshowedthatwhenonlythefivecommoncriteriaareused,STRT is themoredifficult test.As theprevious chapter showed, thisdoesnotimplythatSTRTisalsothemostdifficulttestoverall.Table5.7.MFRA:STRTandITNA,arrangedbymeasure Measure SE InfitMnSq STRT .23 .09 1.02 ITNA -.23 .10 .96 Note.Model,Sample:Separation3.28,Strata4.71,Reliability.92Table 5.8 shows the results of the criteriameasurement in the Rasch analysis.Importantly, these results showed that corresponding criteria were neverincluded in the same difficulty bands. This implies that the difficulty level ofevery ITNA criterion is significantly different from its STRT counterpart.Moreover,theRaschoutputgenerallyalignswellwiththeprobabilitiesdisplayed


112

inTable5.3.Forexample,pronunciationinITNAisthemostdifficultcriterionintheRaschtable,andalsohadthelowestprobabilityscore.Theprobabilityscoresforvocabularywerenotsignificantlydifferent,andinthistabletoo,themeasuresof the vocabulary criteria of both tests are mapped closest to each other.Nevertheless,inspiteoftheresultspertainingtothevocabularyscores,thisstudyhasyieldednodata to indicate that correspondingCEFR-basedcriteriaused tomeasure the same candidates in near-identical tasks can be consideredequivalent.Table5.8.MFRASTRTandITNAcriteria,arrangedbymeasure





Grammar STRT -0.07 0.21 0.85

Pronunciation STRT -0.42 0.21 0.98

Vocabulary ITNA -0.6 0.22 0.94



Fluency ITNA -1.21 0.23 0.99

Grammar ITNA -1.64 0.24 0.48


Importantly, theorderof thecorrespondingcriteriamatchestheorderofTable4.7above.

DISCUSSIONEvery analysis in this chapter confirms the trend observed in the previouschapter.Thereislittle,ifany,evidencetoconfirmanassumptionofequivalencebetweenSTRTand ITNA.CorrespondingCEFR-basedcriteria in the ITNAandSTRT rating scales are not equivalent. If theywere, the correlationswould bestronger, the kappa valueswould showmore agreement, the linear regressionmodelwouldexplainmorevarianceandthesamecriteriawouldfallwithinthe


113

sameRaschdifficultybands.Oneexplanationforthedivergencescanbefoundin the rating scale descriptors. Even though both tests started from the sameCEFR descriptors, they diverged in interpretation and operationalization. TheJaccardindexindicatedthatthedescriptorsareindeedquitedissimilar,asseenin some of the operationalizations discussed above. In short, the statisticalanalyses in this study fail to confirm the assumption that the STRT and ITNAdescriptors interpret the same CEFR levels in an equivalent way, and providearguments to the contrary. Both tests have developed rating scales from thesame source and adopted the same level system, but the relationshipbetweenequivalentcriteriaisweak.IftheCEFRlevelsweretrue,unequivocalstandards,thisshouldnotoccur,butgiventhenatureoftheCEFRdescriptors,thefindingsarenotunexpected.

The root of the problem lies not somuch in the CEFR itself as in thereificationofitslevelsasstandards(Fulcher,2004).TheCEFRisoftenreferredtoas a gold standard, because it is so eagerly used by all parties involved inEuropeanlanguagetesting,butthereisoneveryimportantdifference:exactness.Thecollectiveagreementthatexactlyoneounceofgoldwouldtradeforexactly$20.67madethegoldstandardthebackboneoftheglobaleconomicsystemfordecades.TheCEFRintentionallylackssuchexactness,however,whichmakesitunusable as a standard. CEFR levels do not exist outside of theminds of thepractitioners,andB2isnotanentity.Asastandard,itshareslessresemblancetoscrewthreadsormonetarysystemsthantoprimarycolours.Thecolourbluehasamarkedbeginningandanend,butencompassesarangefromlightaquamarineto darknavy; itwouldbewrong to argue that onlyPantone 2736C is the trueblue. Likewise, it is problematic to consider the B2 level in one rating scaleequivalenttothenext,simplybecausebothhavebeenbasedonthesamebroadlevel.

CONCLUSION:ASSUMPTION3TheresultsofChapter4areinlinewiththoseofthepreviouschapter:TherearenodatatosupportthepresumptionthatthelevelortheconstructofSTRTandITNA are equivalent. The findings from this chapter show that STRT actuallyusescorrespondingcriteria ina stricterway.Nevertheless, ITNA is thehardesttestduetotheweightingofGrammarandVocabulary,andduetothefactthatSTRT assigns relatively great importance to content criteria, which arecomparativelyeasy.


116

CHAPTER5COMPARINGL1ANDL2PERFORMANCEThis chapter focuses onAssumption 4; the language proficiency level offirst-year university students with a Flemish secondary school degree.Additionally, the study presented here explores to what extent Flemishand international L2 students perform differently on the same writtenSTRTtasks,andwhethertheL2learnerswholearnedDutchinFlandersorattheirhomeinstitutionperformdifferentlyonthesamewritingtasks.

Beforemovingon,a smallnoteon terminology isneeded.Within thegroupofFlemish students, a distinction is made between L1 users and Generation 1.5students(G1.5).ThetermL1userwillbeusedtorefertoastudentwhosefirstorhome language is the same as the official language of instruction (Gorter &Cenoz, 2012), which in our case is Dutch. Students whose home language isdifferent from Dutch, but who acquired Dutch during all or part of theirschooling, will be called G1.5 students (di Gennaro, 2009, 2013, 2016; Harklau,Losey, & Siegal, 1999). Other research might refer to this group as sequentialbilingual learners (Paradis, 2007; Pérez-Tattam et al., 2013). Under the currentFlemishuniversityentrancepolicy,bothFlemishL1usersandG1.5studentscanregister for university studies without taking a B2 language test if they havegraduatedfromaDutch-mediumsecondaryschool.

Additionally, this chapter also distinguishes between two groups ofinternationalL2students:thosewhostudiedDutchataFlemishlanguageschool(L2F)priortotakingtheuniversityentrancelanguagetest,andthosewhostudiedDutchataninternationallanguageschoolintheircountryoforigin(L2I).

RESEARCHINTOL1ANDL2PERFORMANCEAdministering a language test as a gatekeeping instrument to international L2students alone relies on the unsubstantiated assumption that students with aFlemish secondary school degree will meet the language demands that arerequired of international applicants. If this assumption is true, the universityentrance policy justifiably exempts this group of students from additionallanguage screening. If itwere untrue however, and if not all studentswho areexempt from taking a test pass it, the university entrance policy could beconsideredtoapplyunequalstandardstodifferentpopulations(Hamilton,Lopes,McNamara,&Sheridan,1993)–whichcouldraisejustice-relatedconcerns.


117

NativespeakerperformanceHulstijnposits thatnotallL1userscanbeexpectedtoperformwritingtasksatthesamelevelassomeL2learners(Hulstijn,2015;Hulstijn,2011)andproposestodifferentiate between Basic Language Cognition (BLC) and Higher/ExtendedLanguageCognition(HLC).BLCisrestrictedtotheuseoforalskillsincommon,everydaylanguagesituations.HLCinvolveslexically,syntacticallyandcognitivelymore complex language in both oral and written forms and includes “topicsaddressedinschoolandcolleges”(Hulstijn,2015:22).Whilehedoesnotexplicitlydifferentiate between BLC and HLC in CEFR terms, it can be inferred thatperforming a writing task at the B2 level requires HLC competence (Hulstijn,2011).Hulstijn argues that therewill be substantial differences in how L1 usersperformHLC tasks andhis second corollary predicts a relativelywide rangeofscoreswhenL1userscompletesuchtasks.

Existingresearch intoL1performanceonL2writing taskssupports thesehypotheses.Twosuchstudieswereundertaken inFlanders. In the first, twoL2writing tasks were administered to 176 first-year students of four Flemishuniversitycolleges(VanHoutven&Peters,2010).Theresults indicatedthatnotall respondents reached the threshold level, but since they differed greatly interms of educational background the authors hesitated to draw any firmconclusions.Amorerecentpaper(DeWachteretal.,2013)confirmedthatthereis large variation in the Dutch language proficiency of first-year students at aFlemishuniversity.

AfewstudieshaveconsiderednativespeakertestperformanceonEnglishlanguagetests foruniversityadmission.Hamiltonetal. (1993)analyzedthetestscoresofnativespeakers(N=48,32bilingual,16monolingual)onIELTSwritingtasks (one information transfer task and one argumentation task), and foundsubstantialvariationintheL1performances.NotingthatthemeanwritingscorefromL1testtakerswasaroundIELTS6.5(i.e.,B2),theauthorsconcludedthatnotall L1 test takersmet the threshold demanded of L2 students, and questionedwhat thismeant forequityofaccess touniversity.FocusingontheTOEFL iBT,Stricker (2004)used t-testsandCohen’sd toestablish that 168 “American-bornspeakers of English” (no information on home language use was offered)significantly outperformed L2 test takers on an essay writing task (p < .05).Furthermore, he found that even though the score variance in the Americanpopulationwassignificantlysmaller(p<.05)thantheL2scorervariance,itwasnotunsubstantial,andnotallL1userspassedtheL2threshold.

The above-mentioned studies suggest that not all students who areconsiderednativespeakersmeetthe linguisticrequirementsdemandedfromL2students upon university entrance. The next segment of the literature review


118

focusesnot somuchon thequestion ifL1andL2students’writingskillsdiffer,butonhowtheydiffer.WritingperformancedifferencesComparing L1 and L2 texts using a four-dimension rating scale, Weigle andFriginal(2015)onlyfoundonesignificantdifferenceonthefourdimensionstheyhadidentified,namelyexpressionofopinion:L2learnersusedsignificantlymoreopinion statements than L1 users. Weigle (2002) reported that scores forvocabularyareoften foundtobe thestrongestpredictorofL1background,andotherstudieshavealsofoundvocabularytobethestrongestpredictorofoverallscores in L1 (e.g.Wolfe, Song, & Jiao, 2016) and L2 (Koda, 1993)writing tasks.Findings for grammatical accuracy show that L1 users usually outperform L2learnersinthisrespect(Leki,Cumming,&Silva,2008),whilefindingspertainingtosyntacticcomplexityare lessclear-cut.Huie&Yahya(2003) reportedthatL1userswritemorecomplexsentences,whileLee(2003)foundnosuchdifferences.In this respect, Schoonen et al. (2003) suggested that L1 writing may dependmoreonmetalinguisticandtopicalknowledge,whileL2writingproficiencycouldbebetterexplainedintermsoflinguisticknowledge.

L1/L2 writing research has also explored how L1 and L2 writers engagewithwriting tasks. It is clear thatL1 andL2writingprocessesaredifferent, yetnotentirelydisconnected(Polio,2013;Schoonenetal.,2003).SkilledL2writerstend to be skilled L1 writers too, if they have surpassed a certain thresholdproficiency level (Lekietal.,2008).Still,perhapsbecausewriting ina languagethatisnotone’sL1requiresmentalcapacity(Lekietal.,2008),L2writersappeartobelessflexibleinhowtheyreplytowritingprompts.L2writerstendtoadoptamoresystematicapproach(VanWeijen,VandenBergh,Rijlaarsdam,&Sanders,2008; Victori, 1999), and appear to adhere more rigorously to the taskinstructions.Becausetheytendtostickcloselytoafixedroutineorscenario,L2writershavealsobeenfoundtoadoptasmallerrangeofwritingstrategiesacrossdifferentwritingtasks(Rijlaarsdametal.,2005).

ResearchhasnotonlyfocusedondifferencesbetweenL1andL2users,butalsoondifferenttypesofL2learners.SomestudieshavecomparedL2writersbyproficiencylevel,whileotherscomparedL2learnerswhoacquiredthelanguageathomewiththosewhoattendedastudyabroadprogram.Researchfocusingonthe first dichotomy has found that skilled L2 writers tend to be more fluentwriters(deLarios,Marín,&Murphy,2001),betteratplanning(Victori,1999),andmorefocusedontextlevel,whereaslessskilledL2writerstendtofocusmoreonvocabularyandgrammar(Lekietal.,2008;Victori,1999).

Next, the study abroad (SA) literature offers further insights that arerelevantinthelightofthecurrentstudy.Importantly,theterm“StudyAbroad”can cover a whole range of experiences (Engle & Engle, 2003) from two-week


119

immersionprograms,tostudentspursuingadegreeabroad,wheretheL2isthemedium of instruction. Llanes et al. (2012), focusing on Erasmus students inEurope, found that learning a language in a SA contextdoesnot automaticallyleadtobetterwritingskills,andthatsomeSAexperiencesactuallyleadtolessL2learningthanin-classinstructionathome(Llanesetal.,2012).Also,studyingtheeffectsofanErasmusexperienceonL2development,Serranoetal.(2012)foundthatthepositiveeffectsofSAextendlesstothewrittenmodalitythantotheoraland the lexical domain. One consistent finding in the SA literature is that L2learners who study a language in a target context do not always have manymeaningfulinteractionswithnativespeakers.Thiswasobservedinanumberofstudies, such as Amuzie andWinke, (2009) – who tracked the experiences ofChineseandKoreanstudentsenrolledatNorthAmericanuniversities–andGuandMaley(2008)whofocusedontheexperienceofChinesestudentsatBritishuniversities.

Asfarascouldbedetermined,nostudyabroadresearchhasyetcomparedthe performance of SA (study abroad) and AH (at home) L2 learners oncentralized high-stakes tests. In the broader field of ESL, however, a limitednumber of studies have done so. Gu (2014) found no difference in TOEFL iBTscoresforL2learnerswholearntEnglishinanEnglish-speakingcontextorinanat-homecontext–afindingthatcontradictsanolderstudybyGinther&Stevens(1998).

Another useful categorization within the broader group of L2 learnersdistinguishes internationalL2 learners fromG1.5 learners (Harklauetal., 1999).Theformergroupisenrolledinpost-secondaryeducationafterlearningtheL2inaclassroomintheirhomecountry,whilethelatterattendedsecondaryeducationin the L2 context beforemoving on to post-secondary education (di Gennaro,2013). There is relative agreement in the literature (di Gennaro, 2016 offers anoverview) that both groups score differently on grammatical criteria, but thefindingsregardingvocabularyandothercriteriaarelessuniform.Arecentstudy(di Gennaro, 2016) found no significant differences in scores betweeninternational L2 and G1.5 leaners on the scores for cohesion, rhetorics,sociopragmatics, and content. Differences in grammar score – the hardestcriterion for both groups – did reach significance, however. The current studydoesnotdirectly focusonG1.5 learners,butwill refer to theresearch literaturewhenrelevant.


120

RESEARCHQUESTIONSThe literature review above strongly suggests that it is unlikely that all L1students have obtained the language proficiency level that is required ofinternationalL2studentsatthestartoftheiracademicstudiesatauniversity.Nostudy has yet compared L1 and L2 performances on a centralized high-stakesuniversity entrance test with regard to B2 proficiency. Finally, little is knownabout the performance of different types of L2 users on a high-stakes test.Therefore,thisstudyexploreswhetherallL1studentsattainaB2levelonanL2entrance test (Assumption 4) and goes on to compare L1 students’ and L2learners’ performance on the same L2 entrance test, and to investigate theperformanceofSA(L2F)andAH(L2I)learners.

By comparing the performance of three groups of first-year universitystudents,thislarge-scalebetween-groupstudyaddressesanumberofgapsinL2writingassessment research.Thestudy is relevantbecause it couldcorroborateHulstijn’s second corollary (“Individual differences among adult L1ers will berelatively large in tasks involvingHLCdiscourse, in all fourmodesof languageuse”–Hulstijn,2015,p.25).Furthermore,thisstudyaddsanewdimensiontothestudy abroad literature, which to date has mainly focused on qualitativedifferences between SA and AH students rather than on their scores oncentralizedtests.Finally,byincludingL1performanceonL2tasks,thisstudymayhave implications for the university entrance policy in Flanders. The implicithypothesis underpinning this policy is that a B2 level is a prerequisite forsuccessfulparticipation inacademic life,andthatFlemishstudentspossess thislevelaftergraduatingfromsecondaryschool.Thisstudyaimstoanswertwoprimaryresearchquestions.RQ1isconcernedwithAssumption4:RQ1 DoallstudentswithaFlemishhighschooldegree,whoareexemptfrom

takingauniversityentrancelanguagetest,passtheB2threshold?The hypothesis was that not all students who graduated from a Flemishsecondary school would pass the B2 threshold as measured by STRT. Thishypothesis is in line withHulstijn’s (2015) BLC/HLC theory and with previousstudiesonL1performance(DeWachteretal.,2013;Hamiltonetal.,1993;Stricker,2004;VanHoutven&Peters,2010).


121

RQ2 WhatarethedifferencesbetweenthewritingperformancesofFlemishhighschoolgraduatesandinternationalL2students?

2a. DoFlemishstudentsoutperforminternationalL2candidatesontheSTRT

writingtasks?Basedonresultsfrompreviousresearch(Weigle,2002;Weigle&Friginal,2015)itwas assumed that Flemish students would outperform L2 students, but notnecessarilyoneverycriterion(Huie&Yahya,2003;Lee,2003;Lekietal.,2008).2b ArethereanyperformancedifferencesbetweenL2learnerswholearned

DutchinFlandersandthosewhodidsoatalanguageschooloutsideofFlanders?

Becauseoftheinconclusiveresultsinthestudyabroadliterature,nopredictionsweremaderegardingtheperformanceofeithergroup.

PARTICIPANTS&METHODOLOGYSTRTWritingtasksSince ITNAdoes not include anywriting tasks at the B2 level, it could not beused to compare writing performances, so it was not considered as ameasurementinstrumentinthisstudy.STRTconsistsofsixintegratedtasks(seeAppendix1),whichareintendedtoreproducereal-lifesituationsandactivitiesinthecontextofhighereducationinFlanders,wherewritingbecomesprogressivelymore important during a students’ academic career (De Wachter et al., 2013;Herelixka,2013).TaskselectionBecauseoftimeconstraintsitwasnotfeasibletoadministeralltesttaskstotheL1population.Consequently,thetwoSTRTwritingtasksthataccountedformostof the variance in the writing scores (Table 6.1) were selected. Based on allavailable STRT scores (N =913) a regressionmodelwas constructed to identifythewritingtasksthatexplainedmostscorevarianceonthewrittencomponent.In this model, Task 2 (T2) and Task 4 (T4) turned out to be the strongestpredictorsoftheoverallSTRTscore.AregressionmodelcomposedsolelyofTask2 andTask 4 explained91%of the score variance in thewritten component ofSTRT(R2

Adj=.911(F(115)=599.9,p<.000)T2β=.602,T4β=.451).


122

Table6.1.Multivariatelinearregression:STRTwrittenscore~T1-T4scores B(SE) β (Intercept) -0.015(.01) T1 2.377(.001)*** .245 T2 2.475(.001)*** .392 T3 2.772(.001)*** .254 T4 2.380(.001)*** .328 Note.R2Adjusted=1,p<.000T2,alistening-into-writingtask,requirestesttakerstolistentoascriptednine-minute lecture about industrialization. Candidates listen to the lecture twicewhiletakingnotes.Afterwardstheyhavethirtyminutestowriteasummary. InT4, a reading-into-writing task, test takers have one hour to summarize a textabouttheprosandconsofschoolswithoutgenderdifferentiationintheteachercorps,andtoformulatetheirownopinionaboutthetopic.

Integrated writing-from-reading or writing-from-listening tasks of thiskindhavebeensaidtotapintoanessentialpartofacademiclanguageuse(Chan,Inoue, & Taylor, 2015; Cumming, 2013).Writing-from-reading is considered animportant index of academic achievement (Baba, 2009; Hirvela, 2016), andsummarytasksinparticularoffervaluableinformationinthisrespect.Similarly,note-takinginwriting-from-listeningtasksrepresentsanimportantpartofwhatstudents need to be able to do (Lynch, 2011; Song, 2012). STRT developersconsiderwritingtasksagoodyardstickfordeterminingwhetheracandidatewillbe able to meet the linguistic demands of university (Confidential STRTdocument,2014).Appendix1includesamorethoroughdescriptionofT2andT4,anddescribestheratingcriteriausedtoassesswritingperformances.ScoringAllperformanceswereindependentlydoubleratedby17trainedSTRTraterswhowereunawareofthebackgroundofthecandidates,inordertoavoidL1bias(VanWeijen, 2009;Weigle, 2002). The rater reliability in this studywas satisfactory(exactvs.expectedagreement63.9%-56.8%;InfitMnSqμ=.99;X2(17)=17.9,p= .40).The introductionandChapters3and4containmore informationaboutSTRTratertrainingandselection.Participants&datacollectionThreegroupsofparticipantswereinvolvedinthisstudy.Eachgrouptookthetesttasks under examination conditions. Trained examiners and invigilators were


123

always on site to ensure that the examination conditions complied withinstructionsinthetestmanual.

Thefirstgroupconsistedof159first-yearstudentsofbusinessstudieswhohad graduated from Flemish secondary education. In return for theircollaboration,thesestudentswereawardedcreditsforacurricularcourseatthestartoftheyear.Inthelightofthefirstresearchquestionitwasimportantthatthedemographiccompositionofthisgrouprepresentedthatofatypicalfirst-yearcourse, minus the international students. Since the first research question iswhetherallstudentswhoareexemptfromtakingabindinglanguagetestwouldpass its threshold, themainselectioncriterionforthispopulationwashavingaFlemishsecondaryschooldegree.ElevenpercentoftheFlemishparticipantshadahomelanguagedifferentfromDutch,butforallmembersofthisgroup,Dutchwas their language of schooling. The Flemish population thus consists of twosubpopulations,L1(i.e.,homelanguageandlanguageofschoolingisDutch)andG1.5students(i.e.,thestudents’homelanguageisdifferentfromDutch,buttheygraduatedfromaDutch-mediumsecondaryschoolinFlanders).SincethesizeoftheG1.5populationwasratherlimited(n=18),andsincethisgroupwasnotpartof the research questions guiding this study, performance data of the G1.5population will not be a primary focus of this chapter, but references to therelevantliteraturewillbemade.

The second column of Table 6.2 (“Flemish”) displays the demographicvariables of the Flemish research population, 85% of whom had attended theacademic strand of secondary education (the other 15% had attended thetechnicalstrandofsecondaryeducation).Thedemographicsarerepresentativeoffirst-year university students in Flanders (58% female; median age 18; 10%migration background; 96% no previous university experience; 90% academicstrand of secondary education. See Glorieux, Laurijssen, & Sobczyk, 2015;UniversiteitGent,2013). Table6.2.Demographicdataofparticipants Flemish L2F L2IAge Average(SD) 18(.7) 27(7) 20(5) Min-Max 17-21 16-50 14-55

Gender:Female 50% 70% 63%

L1 89%L1Dutch 34%Italic 57%Italic 3%Italic 20%Balto-Slavic 18%Germanic 3%Balto-Slavic

5%Other

11%Germanic35%Other

6%Balto-Slavic19%Other

Universityexperience 10% 59% ≤32%


124

TheFlemishparticipantsperformedthetwowrittenSTRTtasksatthebeginningofthesecondmonthoftheacademicyear2015-2016.L1respondentsreceivednospecific training or preparation, as the skills required for successful taskperformance are in line with the attainment targets of the academic andtechnical strands of secondary education issued by the Flemish government(OnderwijsVlaanderen,2015).Priortotakingthetest,Flemishrespondentsfilledout a form with basic demographic information (gender, age, L1, educationalbackground).L2Fusers areoneof twogroupsof internationalL2 students considered in thischapter.Throughoutthedissertation,thecodeL2F(i.e.,L2learnersinFlanders)isused to refer to test takerswho learnedDutch inFlanders and took the testthere. In this chapter, the L2F population is somewhat larger, because scorescollectedintheregularFlemishMay2015STRTadministrationwereincluded,inadditiontothetestscoresofL2Fcandidates(N=118)whohadtakenbothSTRTand ITNA used in Chapters 3 and 4. The total number of L2F participantsincludedintheanalysesofthisstudy,is168.TheseL2learnershadnotattendedanyDutchclassespriortoarrivalinFlanders.Nomembersofthispopulationhadattended secondary school in Flanders. The median length of Dutch L2instruction for this group of respondents was eighteen months. Informationregarding prior university experience was available for 118 respondents. Therespondentsinthisgroupreceivednospecificin-classpreparationforSTRT,buttheywerefamiliarwithwritingtasksandtheyhadreceivedalinktosampletasksontheSTRTwebsite.ThisgroupofrespondentsfilledoutthesamedemographicinformationformastheFlemishstudents.

ThethirdgroupofparticipantsinthisstudyconsistsoftheregularSTRTpopulation: L2 learners who had studied Dutch at a language school in theirhomecountry, and took theentrance test there.These candidates received thestandard STRT demographic information sheet (gender, age, L1), but due tomattersofprivacyanddataprotection,itwasimpossibletotracewhichofthesecandidateshadenrolledwhere.Even though theL2Idatadonot includedirectinformationconcerningprioreducation,wecanassumethatatleast68%hadnouniversityexperienceatthetimeofdatacollection,becausetheywereeighteenor youngerwhen they took the test.Candidatesbelonging to this dataset (N =526) will be referred to as L2I (i.e., L2 learners who studied Dutch at aninternationallanguageschooloutsideofFlanders).

Since theL2Ipopulationrepresents the full test-takingpopulationof theMay2015STRTadministration,thisdatasetissubstantiallylargerthantheothertwo.Formethodologicalreasons,itwasdecidednottobalancethesamplesizeshowever. Undersampling – reducing the size of the largest set by randomlyremoving cases – has major drawbacks, the primary one being the loss ofpotentially valuable data (Kotsiantias et al, 2005). Oversampling on the other


125

hand – randomly duplicating cases from the smaller dataset – may entailanalogousrisks,suchasoverfittingdata(Kotsiantiasetal,2005).Consequently,insteadofrandomlyaddingorremovingobservationstothedatasets,onlythoseanalysesthatallowforunbalanceddatasetswereconducted.DataanalysisRQ1 Do all students with a Flemish high school degree, who are exempt from

takingauniversityentrancelanguagetest,passtheB2threshold?Descriptivestatisticsof thecombinedscoresofT2andT4,andaMulti-FacetedRasch analysis (MFRA) were used to determine whether all Flemish studentspassed the STRTwriting tasks. A binomial sign testwas performed to test thenull hypothesis (𝑃!!!"## = 1). For the MFRA four variables were identified:candidate ability, rater severity and itemdifficulty. The variable group (L1, L2I,L2F)wasenteredasadummyfacet.T1andT3scoresweretreatedasmissingdatafor the L1 group. L2 scores on the oral component were available but wereexcluded from the analysis because no spoken L1 performances were availableandoralandwrittenperformancesonLAPtestshavebeenshowntobelong todifferent dimensions (Gu, 2014). In line with the STRT procedure, this studyadoptedaRaschmeasureof1.42asacut-offpointforpass/faildecisions.Thiscutscore was determined in a standard setting procedure (CNaVT, 2014). Alsofollowing STRT procedure, a pass judgment was assigned to candidates whosescorewaswithintherangeofonestandarderrorbelowthecutscore. RQ2a DoFlemishstudentsoutperforminternationalL2candidatesontheselected

STRTwritingtasks? A Wilcoxon signed-rank test was performed to determine whether anydifferencesinmedianscoresoncontentandformcriteriabetweenthedifferentpopulations were significant. The same analysis was conducted within theFlemishsample,tocomparethescoresofL1andG1.5.Giventhesmallproportionof G1.5 students (11%), no further analyses were conductedwithin the Flemishgroup.

Aprincipalcomponentanalysis(PCA)wasconductedonthestandardizedz-scorestodeterminewhichcriteriacouldbecombinedfortheWilcoxonsigned-rank test. The PCA was run on the ten rating criteria using oblique promaxrotation, since the variables were known to be correlated. Bartlett’s test ofsphericityshowedthataPCAwaswarranted(X2(45)=3878,p< .000),andtheKaiser-Meyer-Olkinmeasureshowedthesamplingadequacy(KMO= .87)tobegood (Kaiser, 1974).All individual criteriahadKMOvalues (> .83)above the .5limit(Field,Miles,&Field,2012).Basedontheinitialanalysisandthescreeplot,


126

itwasdecided to run the analysiswith three factors, as three componentshadeigenvaluesatoraboveone,which–takentogether–explained70%ofthescorevariance. Table 6.3 shows the factor loadings of the different criteria andindicatesthatitwaswarrantedtocombinethelinguisticcriteriaofT2andthoseofT4.Thecontentcriteriaofbothtasksloadontothethirdcomponent.Table6.3.Promaxrotatedfactorloadings FormT2 FormT4 ContentT2Vocabulary .86 T2Grammar .83 T2Cohesion .77 T2Spelling .67 T4Vocabulary .9 T4Grammar .78 T4Cohesion .69 T4Spelling .65 T2Content .83T4Content .81Eigenvalues 2.74 2.65 1.59%ofvariance 27 27 16Note.Factorloadings<.3wereomittedfromthetable.In order to better understand how L1 candidates performed on the separatecriteria, a logistic regression model was developed in which L1ness wasoperationalizedasa functionoftheratingcriteria.Allcriteriawere individuallyincluded in the regression, because such a model was a significantly betterpredictorthanonewhichusedtheclusteredcriteria(X2(4)=159,p<.000).RQ2b Are there any performance differences between L2 learners who learned

Dutch in Flanders and those who did so at a language school outside ofFlanders?

Multinomial logistic regression was used to identify how well rating criteriapredictedmembershipofoneof thethreerespondentgroups:L1,L2I (=AH)orL2F (= SA). Logistic and multinomial regression models were also used tomeasure the effect of other background variables (age, gender, L1, educationalbackground,cf.Weigle,2002)onscores.Sincetheseresultsindicatetheextenttowhich performance differences between research populations have beeninfluenced by background variables, but do not directly answer the researchquestions,theyarebrieflyreportedhere.

The effect of educational background was examined using the L2Fpopulationonly,becausestudentswithnoexperience inhighereducationwere


127

overrepresentedintheL1group,andnoeducationalbackgroundinformationwasavailablefortheL2Istudents(fortheotherbiasanalyses,thefullpopulationdatawereused).HavinguniversityexperiencewasasignificantpositivepredictorforL2F scores onCohesion andVocabulary, but a negative predictor forGrammar(CohesionT2(B(SE)=1.23(.43)**;VocabularyT4(B(SE)=1.05(.43)*;GrammarT4(B(SE)=-1.04(.45)*).Giventhesamplesizeavailableforthisbackgroundvariable,generalizations(NagelkerkePseudoR2=.18)shouldbemadecautiously.

Since the dataset included nineteen different language branches,languages were clustered into five groups: “Germanic”, “Italic”, “Balto-Slavic”,“Other Indo-European” and “Other”. An exploratory logistic regression showedthatoverallscorespredictedmembershipoftheGermaniclanguagegroup(B(SE)=0.26(.06)**),so“Germanic”wasusedasthebaselineinthemultinomiallogisticregression. The model with two additional predictors (“Italic” and “Other”)yieldedthemostexplainedvariance(McFaddenR2=.18,X2=290,p<.000)andisreported here.MFRA reliably (.92) determined thatGermanic/Italic candidateshadhighertestscoresthanothertesttakers.Overall,scoresonlinguisticcriteriadidnotsignificantlypredictmembershipoftheGermanic/Italicgroup,butscoresoncontentcriteriadid(T2B(SE)=0.26(.06)***;T4B(SE)=0.23(.06)***).

Additionally, Facets bias analyses on task levelwere conducted for eachbackgroundvariable.IneachMFRAmodel,thebackgroundvariablewasenteredas dummy facet. The regression models and the MRFA both showed thatcandidates aged eighteen and younger significantly outperformed older ones(B(SE) = -.28(.032)***; MFRA reliability .96). Other bias analyses yielded non-significantorminutedifferences.

RESULTSDoallL1studentspasstheB2threshold,asmeasuredbySTRT?The binomial sign test rejected the null hypothesis at the 95% and 99%confidence level (p < .000). Not all Flemish candidates pass the STRT writingtasks.Asagroup,Flemishstudentsperformbest,butnotallmakethegrade.

Table 6.4 shows that themedian andmean scores of theFlemish groupwerehigher than those of bothL2 groups. This doesnot imply that individualFlemishtest takersperformedbestonthetest,however,sincethebestFlemishperformerwasoutperformedby21L2IandL2Fcandidates.Overall,however,theFlemishgroupgainedthehighestscores,andtherangebetweenthehighestandthe lowest score is the smallest in theFlemishgroup.Ona twenty-point scale,therangeofscoresonT2andT4combinedis7.8fortheFlemishgroup,versus15.5and18fortheL2IandtheL2Fgrouprespectively.


128

Table6.4.Descriptivestatistics:groupscores,combinedscoresT2andT4

Flemish L2F L2I

N 159 168 526Mean 15.74 12.49 14.52SD 1.62 3.58 2.09Median 15.71 13.06 14.69Min 10.61 1.22 4.08Max 18.37 19.18 19.59Range 7.76 17.96 15.51Skew -0.56 -0.8 -0.45Kurtosis 0.04 0.47 0.97SE 0.13 0.28 0.09Note.Maxscore=20TheMFRA analysis for the facet “Group” (see Table 6.5) confirmed the trendsemerging from the descriptive statistics: in general, L1 candidates were thestrongestcandidateson theSTRTwriting tasks.TheRaschmodel reliably (.99)separatedFlemish,L2IandL2Fcandidatesintermsofability,implyingthateachgroupperformedatadistinctlydifferent level.TheL1groupoutscoredbothL2groups,butnotallL1candidatesscoredabovetheSTRTcutoffpoint.Table6.5.MFRAforfacet“Group” Measure ModelSE Infit %belowcut-offFlemish 0.57 0.02 1.12 11L2I -0.15 0.01 0.82 30L2F -0.43 0.01 1.29 57ElevenpercentoftheFlemishgroupscoredbelowtherequired1.42thresholdanddidnotpassthewritingtests.Incomparison,30%oftheL2Ipopulationand57%of the L2F population did not obtain the required score. The ability measure(reliability .99)of theFlemishpopulationspannedfour logits,comparedto fivefor L2F and six for L2I candidates, reaffirming that the range of scores for theFlemishparticipantswassmallerthanfortheothergroups.

Thedemographics of the small groupof Flemish candidateswho scoredbelowthecutscoreshowedaslightoverrepresentationofmen,ofG1.5students,andofstudentswhoattendedatechnicalstrandofsecondaryeducation.Oneoutof five G1.5 students failed the writing test, versus one out of ten Flemish L1students.Oneinthreestudentswithatechnicaleducationbackgroundfailedthewritingtest.SixoutoftenFlemishstudentswhofailedthetestweremale.


129

What are the differences between the writing performances of FlemishhighschoolgraduatesandinternationalL2students?Wilcoxonsigned-ranktestresults(seeTable6.6)showedthatFlemishstudentsconsistentlyoutperformedbothL2groupsonthelinguisticcriteria(maxscore=16),buttheeffectsizerwasthelargestforL1versusL2Fcandidates.Thecontentscores(maxscore=17)revealedthatFlemishstudentsperformedsimilarlytoL2Fcandidates, and both groups’ median scores were below the L2I median, withmediumeffectsizes.Table6.6.Wilcoxonsigned-ranktest:Flemish,L2FandL2I Median Flemish&L2F Flemish&L2I L2I&L2F Fl L2F L2I W p r W p r W p rT2Ling 13 9.5 10 23275 <.000 -.64 71833 <.000 -.53 53196 <.000 -.15T4Ling 13.5 10 11 22750 <.000 -.61 67486 <.000 -.45 53684 <.000 -.16Cont 12.5 12 15 14825 .08 -.09 23028 <.000 -.33 62871 <.000 -.32

Wilcoxon signed-rank testswere also carriedoutwithin theFlemish sample todeterminewhether therewereanyperformancedifferencesbetweenFlemishL1candidates and G1.5 students (see Table 6.7). The effect sizes show that thedifferences are rather small, or – in one case –non-significant.ThedifferencesbetweentheL2FandtheG1.5scoresonlinguisticcriteria,however,arelargeandsignificant, and a detailed look at the individual criteria shows that G1.5candidates significantly (p < .05) outperformed the L2F candidates on everylinguisticcriterion.Theeffectsizeswerethe largest forVocabulary (T2r=-.30;T4r=-.30)andGrammar(T2r=-.36;T4r=-.24).Table6.7.Wilcoxonsigned-ranktest:L1,G1.5,andL2F Median L1&G1.5 L2F&G1.5 L1 G1.5 L2F W p r W p rT2Ling 13.5 12.5 9.5 827.5 <.05 -.18 2562.5 <.000 -.35T4Ling 13 12.5 10 916.5 .07 -.14 2369 <.000 -0.29Cont 12.5 11.25 12 794 <.05 -.19 1344 .43 -.05Note.r=effectsizeInordertodeterminewhetherFlemishstudentsoutperformedL2studentsonalllinguisticcriteria,a logisticregressionmodel(seeTable6.8)wasconstructedtopredict membership to the Flemish group from criterion scores (NagelkerkePseudo R2 = .41). In this model, Grammar and Vocabulary were the onlysignificant positive predictors of Flemish group membership, whereas contentscores were significant negative predictors. The Content score of T2 was the


130

strongest (negative) predictor of this model. Cohesion and Spelling did notsignificantlypredictwhetherornotcandidatesbelongedtotheFlemishgroup.Table6.8.Logisticregression:Flemish~criterionscores B(SE) z Oddsratio(Intercept) -7.33 (0.79)*** -9.3 T2Vocabulary 1.018 (0.22)*** 4.62 2.768T2Grammar 0.503 (0.22)* 2.29 1.654T2Cohesion 0.051 (0.18) 0.29 1.053#T2Spelling -0.013 (0.2) -0.06 0.987#T2Content -0.382 (0.07)*** -5.45 0.682T4Vocabulary 1.151 (0.23)*** 4.95 3.162T4Grammar 0.571 (0.23)* 2.43 1.769T4Cohesion -0.271 (0.2) -1.33 0.762#T4Spelling 0.132 (0.18) 0.72 1.141#T4Content -0.102 (0.07) -1.41 0.903#Note.*p<.05;**p<.01;**p<.001

#confidenceintervalcrosses1In order to compare both L2 groups with each other, while allowing forcomparison with the Flemish candidates, the multinomial logistic regressionmodel (McFaddenR2= .37,Likelihoodratio testX2=587.67,p< .000)was runtwice; once with the L2I group as a baseline, and once with the Flemishpopulation.Table6.9displaystheresultsofL2FvsL2I,L2IvsFlemish,andL2FvsFlemish.

ThefirstpartofTable6.9(i.e.,L2FvsL2I)indicatesthattheonlycriteriathatsignificantlypredictwhetheracandidateisL2IorL2FwereVocabularyinT2,cohesion in T2, and content in T2 and T4. Vocabulary scores in T2 can beconsidered a positive predictor of L2F membership, while Cohesion scores onTask 2, andContent scoresonbothT2 andT4 tasks arenegativepredictors.Aone-unit increase inVocabulary scores canbeexpected to increase theoddsofbelongingtoL2FoverL2Iby0.92units.Aone-unitincreaseintheCohesionandContentscoreswilldecreasethoseodds.

WhenconsideringthesecondandthirdsectionofTable6.9, twosimilarprofiles emerge:Vocabulary is a negative predictor of both L2 groups, but theopposite is true ofContent scores. Higher content scores increase the odds ofbelonging to either L2 group, but the trend is most pronounced for L2Icandidates.HigherVocabularyscoresonT2andT4areassociatedwithFlemishcandidates, but for the other linguistic criteria, the results are moreheterogeneous.Grammar is a positive predictor ofmembership to the FlemishgrouponlyinT2,andSpellingonlyinT4.Remarkably,CohesionisapositiveL2-predictor in T4, but a significantly negative one for L2F candidates in T2.


131

Grammar inT4 and spelling inT2 didnot significantly predictmembership ofanyofthethreerespondentgroups.

DISCUSSION

The data presented in this study show that eleven percent of the Flemishparticipantsdonotmeetthewritten languagedemandsthat their internationalL2 peers are required to meet. The Flemish group as a whole was the mostsuccessful, but the best performers on the writing tasks were L2 students. NoFlemishcandidatesappearedinthehighest-scoringpercentile.Eventhoughthescore rangewaswider in the L2 groups, therewas substantial variation in the

Table6.9.M

ultino

miallog

isticregression

L2Fv

s.L2 I

L2Ivs.Flemish

L2Fvs.F

lemish

B(

SE)

Odd

sratio

B(SE

)

Odd

sratio

B(SE

)

Odd

sratio

(intercept)

2.68

(0.55

)***

9.86

(1.2

9)***

12.54(1.32)

***

T2

Voc

abulary

0.92

(0.21)

***

2.50

-2.26(0.33)

***

3.84

-1.34(0.35)

***

0.40

T2

Grammar

-0.30(0.20)

0.74

# -0.62(0.31)

* 2.51

-0.92(0.33)

**

1.35

T2Coh

esion

-0.45(0.19

)*

0.64

-0.26(0.25)

2.03

# -0.71(0.28

)*

1.57

T2Sp

ellin

g0.00

(0.17

)1.0

0#

-0.04(0.30)

1.04#

-0.04(0.32)

1.00#

T2Con

tent

-0.41(0.07

)***

0.67

0.84

(0.10)*

**

0.65

0.43

(0.10

)***

1.50

T4Voc

abulary

0.34

(0.20)

1.40#

-2.35(0.35)

***

7.46

-2.01(0.37)*

**

0.71

T4Grammar

-0.28(0.20)

0.76

# -0.20(0.32)

1.62#

-0.48(0.33)

1.32#

T4Coh

esion

-0.12

(0.19

)0.89

# 1.0

1(0.30

)***

0.41

0.89

(0.32)

**

1.13

T4Sp

ellin

g0.01(0.16)

1.01#

-1.02(0.28)

***

2.75

-1.01(0.30

)***

0.99

T4

Con

tent

-0.25(0.07)

***

0.78

0.55

(0.11)*

**

0.74

0.30

(0.11)*

* 1.2

8 N

ote.*p<.05;**p<.01;

**p<.001

# con

fiden

cein

tervalcrosses1


132

scores of the Flemish candidates (a range of 7.8 on a twenty-point scale). ThegroupofL1studentswhodidnotpassthewritingtasksistoosmalltodrawanyfirmconclusions,but the trends in thegroupcomposition reflect the resultsoflarge-scale research into performance indicators in Flemish post-secondaryeducation (Glorieux et al., 2015). In absolutenumbers,most L1 candidateswhodidnotattain theB2 level,weremonolingualDutchstudents (n= 13)whohadgraduated from the academic strand of secondary education (n = 11), butproportionally, studentswithahome languageother thanDutch, and studentswith a degree from the technical strand of secondary education are slightlyoverrepresented (Glorieux et al., 2015). The overrepresentation ofG1.5 studentsamonglow-scoringFlemishtesttakersechoesFox(2005),whoconcludedthatitwould bewrong to assume that G1.5 studentswill perform equally well in thetargetcontextastheirL1peers.

This study also supports Hulstijn’s claim that some L2 learners willoutperform L1 users on cognitively demanding tasks, and it is in line withprevious findings by Hamilton et al. (1993) and Stricker (2004). The data alsoback Hulstijn’s hypothesis that there will be relatively large differences in theperformancesofL1usersoncognitive tasks (Hulstijn, 2015,p. 53), even thoughtheFlemishpopulationinthisstudywasrelativelyhomogenousintermsofage,homelanguageandsecondaryschooldegree.

ThefactthatmorethanoneoutoftenFlemishparticipantsfailedtomeetthe writing demands of a B2 university entrance test could have considerableimplications. The results indicate that the differential treatment of FlemishstudentsandL2students(withregardtouniversityentrancetests)maybebasedonanassumptionthatisnotsupportedbyempiricaldataorbyrecenttheories:notallstudentswhograduatedfromaFlemishsecondaryschoolpossesstheB2level, as measured by STRT. Additionally, previous research has shown thatallowingG1.5students toenrollwithout takinga languagetest,maynotalwaysbe in their best interest, since these students may be less likely to achieveacademicsuccessthantheirL1peers(Fox,2005).

Even though the best L2I and L2F candidates outperformed the bestFlemishtest takers, theanalysesshowedthat theFlemishgroupasawholedidbetterthantheL2testtakers intermsofoverallscoresandscoresonlinguisticcriteria. The linguistic criteria that significantly predicted membership of theFlemish subpopulation were vocabulary and, to a lesser extent, grammar andspelling.Thescoresoncohesiondidnotofferaclearpicture:ononetask,highcohesionscoreswereapositivepredictorofL2groupsincontrasttotheFlemishgroup, while on the other task high cohesion scores significantly predictedmembershipoftheFlemishpopulationratherthantheL2Fgroup.

AcomparisonofthescoresoftheFlemishandtheL2populationsshowstwomaintrends:Flemishstudents,bothL1andG1.5,scoredsignificantlyhigheron linguistic criteria, andL2 students gainedhigher content scores.The scores


133

for content are perhaps the most surprising, all the more because within theFlemishsample,L1studentssignificantlyoutperformedG1.5studentsoncontent,butnotonthelinguisticcriteriaofbothtasks.

If L1 writing performance depends primarily on content andmetacognition (Schoonenet al., 2003) and if vocabulary is akey component inlistening and reading, onewould expectL1users tohave an advantageoverL2learnerswhenitcomestoscoringcontentpoints.OneexplanationmaybefoundintheacademicbackgroundoftheL2population,butbiasanalysisshowedthatthisvariabledidnothaveasignificantimpactoncontentscores.L2learners’test-taking strategy could provide an alternative explanation for content scores. L2writersprefer to stay closer to the sourcematerial thanL1writers (Keck, 2006,2014; Wu, 2013) and tend to stick closely to a fixed routine or scenario(Rijlaarsdam et al., 2005). The binary content criteria in STRT actually awardcandidateswhostickcloselytothepromptanddirectly–thoughnot literally–reuse elements from the input material. This could explain why the L2 groupperformed better than the Flemish population on STRT content criteria. Theresultsof thecontentcriteriacouldbeacauseofconcern forSTRTdevelopers,andurge them to examine the construct relevance of the operationalization ofcontentitems.

TheFlemishgroupoutperformedbothL2groupsonlinguisticcriteria.ThePCAshowedthat linguisticcriteriacluster togetherona taskbasis, rather thanby criterion. This reflects the data found in Tillema et al. (2013) and mayindirectly confirm Yu's (2013a, 2013b) hypothesis that input material used inintegrative tasks may impact performance more than a candidate’s writingability.ThePCAalsoconfirmsSchoonenetal.(2003)whoarguethatthescoreona writing task should be seen as the interaction between various linguisticfeatures such as grammar, vocabulary and structure. Nevertheless, the datareaffirm that Vocabulary scores are an important predictor of overall writingscores (Koda, 1993; Weigle, 2002; Wolfe et al., 2016), and Flemish candidatesperformedsignificantlybetteronthiscriterion.Reminiscentofpreviousresearch(diGennaro,2016),theG1.5subpopulationoutperformedtheL2Fpopulationonevery linguistic criterion, but the largest effect sizes were measured forVocabulary and Grammar. In di Gennaro’s study too, G1.5 participantsoutperformedinternationalL2studentsintermsofwordchoiceandintermsofgrammaticalcriteria.

Overall, L2I candidates outperformed their L2F peers. StudyingDutch inFlanders (i.e., abroad)doesnot seem to automatically lead tohigher scores onthewrittencomponentofSTRTthanstudyingDutchinthehomecountry.Thevocabulary score on the writing-from-listening task (T2) was the only positivepredictorforL2Fcandidatesinthemultinomialregressionmodel,whichisinlinewithLekietal.(2008),whoobservedthatlessskilledL2writerstendtofocuson


134

lexis.ContentscoresonbothtaskssignificantlypredictedmembershipoftheL2Igroup.Theothercriteriadidnotsignificantlycontributetothemodel.

At first sight itmay seem striking that learningDutch in a context thatoffers great opportunities to become immersed in the target languageenvironmentmaynotpayoffintermsoftestoutcomes,butsimilarresultshavebeen found in previous studies. Research has shown that a study abroadexperiencedoesnotnecessarilybenefitoverallwritingproficiency,while itmaylead to gains in oral fluency or lexical development (for an overview see Sanz,2014).Reminiscentofpreviousstudies, theresults forVocabulary reportedhereshowed that L2F candidates gained higher vocabulary scores than L2I students(Housen et al., 2011; Juan-Garau, Salazar-Noguera, & Prieto-Arranz, 2014). Thisseems to confirm that study abroad is more beneficial to lexical developmentthan learninga language inone’snativecountry.Sincethisstudydidnot focusonmeasuringprogressorgains,furtherpresumptionsorgeneralizationsonthismatterwouldbespeculativeatbest.

L2IstudentsdidbetterthantheirL2Fpeersoncontentcriteria.Thisgroupmayhavestuckmoreclosely to thesource text (seeabove)andmayhavebeenawarded for it in the rating process. Alternatively, even though L2 users as agroup tend to stick closer to source texts than L1 users, L2 candidates with ahigher proficiency incorporate more details from the source text in theirperformance than their lessproficientpeers (Wu,2013).Possibly, theL2Igroupwasmoreproficientandmoreabletoincludemorecontentspecificity.

Afurtherexplanationforthedifferenceinscoresmaylieinthelengthofinstruction.TheL2I candidates in this studyhad typicallybeen learningDutchforafewyears,whilethemedianlengthofDutchinstructionforL2Fcandidateswaseighteenmonths.Possibly,candidateswhohavebeenstudyingDutchlongertendtoscorebetteronSTRT.

CONCLUSION:ASSUMPTION4ThischapterhasfoundnosupportingevidenceforAssumption4.EventhoughasagroupFlemishstudentsoutperformedbothgroupsofL2students,11%didnotmeet the B2 level requirements asmeasured by STRT. Based on these data, itcannot be automatically assumed that Flemish high school graduates who areexemptfromuniversityentrancelanguagetestswillperformuptocriterion.

Part3:Gains&contents

136

PART3GAINS&CONTEXT

Thefirsttwopartsofthisdissertationwereconcernedwithempiricallyinvestigating assumptions that support the university entrance policyfor internationalL2 students.Untilnow, the focuswasonproficiencylevels,onrepresentativeness,andontestequivalence.Inthethirdpart,which consists of one chapter, the attention shifts to what happensaftertheentrancetest:doL2studentsmakegainsintermsofacademicDutchlanguagedevelopment,andifyesorno,why?

The researchgoal examined in thispart (RG5)doesnot relyonan implicit orexplicit assumption present in policy texts. Instead, it examines whether theassumptionsrelatingtolanguagegainsvoicedbysomeindividualpolicymakers(see Chapter 2) hold true. Additionally, since the first part of the dissertationshowed that most international L2 students are not entirely ready for thelinguistic requirements of academic studies at university, and that theB2 leveloffered no guarantees for coping linguistically in the TLU context, post-entrylanguagegainswouldbenefit internationalL2students.Todatehowever, therewasnoinformationavailabletoshowwhetherorwhythesegainsaremade.Chapter6isbasedon:

Deygers, B. (2017, accepted). A year of highs and lows. ConsideringcontextualfactorstoexplainL2gainsatuniversity.TheModernLanguageJournal.

This paper has been edited to avoid redundancy. The method section has been revised.

Chapter6:ExaminingL2gains

138

CHAPTER6EXAMININGL2GAINS

The exponential increase of international L2 students at universitiesworldwide has generated substantial research interest into universityentrance language requirements and university entrance language tests.The issue of L2 gainsmade by international students while studying atuniversity has been the subject of comparatively fewer studies (Knoch,Rouhshad,Oon,&Storch,2015),eventhoughthisisanissueofsubstantialrelevancetotestdevelopersandpolicymakersalike.Itisnotuncommonforpolicymakerstoexpectinternationalstudentstomakelanguagegainsbyvirtueofattendingclass in theL2 (Storch,2009),even thoughrecentresearch has contested the assumption that rich input alone will yieldlanguagegains(e.g.,Gass,2003;butseealsoKrashen,1985)Forlanguagedevelopment to take place, learners need sufficient opportunities to usethelanguageinameaningfulcontext(Ellis,2003).

Offering support to the idea that language gains are not simply a matter ofadequateinputgeneratingadequateoutput,recentstudieshaveshownthatevenstudyabroadprograms,whichweredesignedtostimulate languagegains,yieldlessdemonstrableeffectsthaneducatorsandpolicymakersmaywanttobelieve(Sanz, 2014). Gains in oral fluency and lexical complexity have regularly beenreported, but the findings for pronunciation, spoken accuracy, or writing aremore ambivalent (Collentine & Freed, 2004; Díaz-campos, 2004; Hernández,2010; Llanes, Tragant, & Serrano, 2012; Pérez-Vidal, 2014; Pérez Vidal & Juan-Garau,2014;Serrano,Tragant,&Llanes,2012).

Typically,studyabroadresearchconsiderssituationsinwhichL2learnersgoabroadforthepurposeoflearningalanguage(Engle&Engle,2003).Formostinternationalstudents,however,developingtheirL2proficiencyisnotthegoaloftheir stay; theL2 is simply themainmediumof instruction.Studiesmeasuringlanguagegainsinthiscontextexist,butusuallyfocusonorincludeparticipantswho received additional language support (Elder & O’Loughlin, 2003; Green,2004; Llanes et al., 2012; Serrano et al., 2012). Only a handful of studies haveexamined language progress made by international L2 students who have notreceivedadditionalformalL2instruction,whichisthedefaultsituationinmanycountries(Knochetal.,2015).


139

WhichlanguagegainscanbeexpectedininternationalL2students?Longitudinal research into the language development of international studentswho do not receive additional language support is rather scant, but findingssuggest that the gains are less substantial than commonly held beliefs wouldallow. Focusing on the language gains made by 24 Spanish undergraduatesduring one semester at an English university, Llanes et al. (2012) reportedsignificantgainsandmediumtosmalleffectsizesforwrittenfluency(Words/T-unit:p = .032, r = .24) andoral fluency (Pruned syllables/minute:p = .000, r =.40). Ina similardesign,Serranoetal. (2012) traced languagegainsof fourteenSpanishundergraduatesonanErasmusexchangeprogramintheUK.Statisticallysignificantgainsandmediumtolargeeffectsizeswerereportedfororalfluency(Syllables/minute:p=.004,d=1.08),lexicalrichness(Guiraud’sIndex:p=.46,d=.65)andoralaccuracy(Errors/T-unit:p=.011,d=-1.33).Inthewrittenmodality,gains were reported in terms of fluency (Words/T-unit: p = .009, d = 1.11),syntactic complexity (Clauses/T-unit: p = .046, d = .83), lexical richness(Guiraud’sIndex:p=.41,d=.58),andaccuracy(Errors/T-unit:p=.03,d=-1.15).Importantly,however,inbothstudies,alargeproportionoftheparticipantshadvoluntarilyattendedadditionallanguageclasses(Llanesetal.N=18/24;Serranoetal.N=8/12),soitisimpossibletodeterminewhetherthesegainswouldhaveoccurredhadformallanguagesupportbeenabsent.

Perhaps theonly longitudinal research focusingon languagegainsmadebyinternationalL2studentswhodidnotattendformalL2languagesupportwasconducted at the University of Melbourne, Australia. In a number ofpublications,researchersusedatest-retestdesigntomeasurewritinggainsafterone semester, one year, and three years. Storch (2009) asked 25 Asianinternationalstudentstowritea300-wordargumentativeessayatthebeginningof the semester, and again twelve weeks later. She found no statisticallysignificant gains on fluency, accuracy, or grammatical and lexical complexity.Arguingthatonesemestermightbetooshortatime-frametoobservelanguagegains (see Ortega, 2003), Knoch, Rouhshad, & Storch (2014) then used theDiagnosticEnglishLanguageAssessment(DELA)tomeasurewritinggainsin101participantsoverthecourseofoneacademicyear.Theyfoundnogainsinwritingscores,nogainsintermsofaccuracyandcomplexity.Onlyfluency,asmeasuredbytheamountofwordswritteninthetimeallowed,significantlyincreased,withalargeeffectsize(p<.004,𝜂!!=.216).Lastly,coveringathree-yeardatacollectionperiodthatincluded31participants,Knochetal.(2015)reportedsimilarresults:no gains on the DELA, and no gains in terms of discourse measures, writtenfluency excepted. After three years, participants wrote significantly (p < .003)morewordswithinthe30-minutetimeframe.

Little if any research has investigated spoken language gains made byinternational L2 students who received no additional language instruction.


140

IdentifyingcausesforlimitedgainsOnlyafewstudieshaveexaminedthecausesforlimitedlanguagegainsmadebyinternationalL2students.Inthestudyabroadliterature,authorshavepointedtoindividual and contextual parameters as possible explanatory variables. In thestudy by Llanes et al. (2012), participants with positive attitudes towards theexperience abroad, and participants who interacted more with L1 users mademoregains.Deweyetal.(2014)reportedthatstudentsenrolledinprogramsthatstimulated interaction, mademore gains, and that – in contradiction to someearlier studies – older students made more gains than their younger peers.AccordingtoEngle&Engle(2003)contextualvariablestoomayimpactlanguagegains:longerstays,interactivedidacticapproaches,accommodationsharedwithL1 speakers, and opportunities for interactionwith L1 speakers were argued topositively affect language gains.Here too, however, findings are contradictory.Housing conditions, for example, have been found to correlate with writtenaccuracy(Llanesetal.,2012),butzeroeffectshavealsobeenreported(Magnan&Back,2007).Similarly,incertainstudiesthelengthofastayabroadimpactedthegainsmade (Pérez-Vidal, 2014),while inothers it didnot (Elder&O’Loughlin,2003;Serranoetal.,2012).

Again, research that offers explanations for limited language gains ininternational students is rather scant, but the studies that do exist typicallysuggest thatonewouldhave to consider theamountofmeaningful interactionwith speakers of the target language. A consistent finding in studies on L2socialization,however, is thatmeaningful interactionwith theL1community isproblematic, or rare (Ranta&Meckelborg, 2013).Gu&Maley (2008) identifiedfourmainareasinwhichtheparticipantsexperiencedproblemsadaptingtotheacademic community in the UK: the culture of the host society, the didacticapproaches,thelanguage,andtheirnewsocialrole.Insomecases,thisresultedin boredom, loneliness and alienation. In a study that traced the academicsocialization of six Japanese students at a Canadian university, Morita (2004)argued that international students’ in-class participation does not necessarilydependontheirintellectualabilities,butonthesocialrolestheyassume,orareallowed to assume. Kormos et al., (2014), focusing on interactions betweeninternational students and English students at universities in the UK, showedhowL2learnersperceivednegativereactionsofL1speakerstotheirlanguageuseasthreatening,resultinginreducedwillingnesstoseekoutfurthercontactwithL1peers.AstudybyDuff(2002)inaCanadianhighschoolsettingexplainedhowparticipating in class allows L2 learners to show and develop their academicidentity, yet the participants she described were caught in a dilemma thatobstructed access to and acceptance in the discourse community: fearing thattheywouldberidiculedbecauseoftheirEnglish,theL2learnersinDuff’sstudy


141

wereafraidtospeak,butalsofearedtheotherstudents’contemptwhentheykeptquiet.

Insummary,thereisgeneralconsensusthatinternationalL2studentsmayfind it hard to become legitimate members of a new academic community ofpractice for power-related cultural or linguistic reasons (Kormos et al., 2014;Ranta&Meckelborg,2013;Subtirelu,2014).Additionally,L2learnerswhoremainon the fringesof adesired communityof practicehave feweropportunities formeaningful interaction,which isakeyprerequisite forL2 learning (Hernández,2010;Ortega,2008).ExplanatorytheoriesAfewcurrenttheoriesofL2learningofferframeworkstoexplainL2learningbyreferring to the social environment, including institutional and interpersonaldynamics,asmediating factors (Duff,2002;Kinginger,2004;Lantolf&Genung,2003;Norton,2013;Norton&Toohey,2011,butseealsoHolmes,Marra,&Vine,2011).Inthesetheories,identityandsocialacceptanceareofpivotalimportance.

Influenced by Bourdieu’s poststructuralist writings (Bourdieu, 1991),identitytheorypositsthatidentitiesshapeandareshapedbytheunequalsocialstructuresofthevariouscommunitiesinwhichwelive(Norton&Toohey,2011).Identity,inotherwordsisdefinedbywhoweidentifywith,andbywhowefeeldistanced from (Baynham, 2006). Thus, a sense of familiarity causes people toidentify with one another (Kormos et al., 2014) and may lead to a cyclicalreaffirmationofpowerstructures.Norton(2013:47)definespoweras“thesociallyconstructed relationsamong individuals, institutionsandcommunities throughwhichsymbolicandmaterialresourcesinasocietyareproduced,distributedandvalidated”.Similarly,Lantolf&Genung(2003:178),conceptualizesocialpowerasthe ability to see the world from one’s own perspective, without needing toconsidertheperspectiveofothers.Assuch,sociallypowerfulgroupsdictatetheterms of legitimatemembership to their group or their community of practice(Block, 2007). Becoming part of such a group thus requires identityreconstruction and adjustment to new rules and norms (Lave&Wenger, 1992;Norton & Toohey, 2011; Swain & Deters, 2007). Since language learning isconditional on meaningful interaction (Amuzie & Winke, 2009; Ranta &Meckelborg, 2013),membership to these communities of practice is crucial formakinglanguagegains.

Research in the fieldof psychology supports the importance assigned tosocialbelonging inthese languagesocializationtheories.Psychologistsconsidersocialbelongingtobeabasichumanneed(Baumeister&Leary,1995;Walton&Cohen,2007).Intheirextensivereviewofavailableresearch,MacDonald&Leary(2005)evenargue that thehumanbrainperceives socialexclusion inmuchthesamewayasphysicalpain.Inasimilarvein,CharlesTaylorhasarguedthatour


142

identity is at least partly shaped by recognition from institutions and peoplearoundus,andinathought-provokingessayhearguesthatnon-recognitionbyotherscancauseactualdamage,andmayleadpeopletointernalizedepreciatoryattitudes towards themselves (Taylor, 1992). Psychology has shown how thisdamage can manifest itself: feelings of exclusion may result in withdrawal,loneliness, sadness, or shame (Buckley,Winkel,& Leary, 2004). In a universitycontext this may impact cognitive processing (Baumeister & Leary, 1995) andacademic results (de Beer, Smith, & Jansen, 2009; Maestas, Vaquera, & Zehr,2007;Walton&Cohen,2007).Importantly,membersofminoritygroups,suchasL2 learners, may be disproportionally impacted by events that confirm aperceivedabsenceofsocialconnection(Walton&Cohen,2007).

Recently, the Douglas Fir Group (2016) has proposed a promising andrather comprehensive framework that could advance research into languagegains made by international L2 students. In their framework they combineinterdisciplinary insights to distinguish three interconnected levels that couldexplainwhy language learning isobstructedor facilitated. In thischapter Iwilladdress the micro and meso levels directly (interpersonal, institutional),extrapolate to the macro level (ideological), and explain how the levels areinterconnected.

ThemacroleveloftheDouglasFir(2016) frameworkreferstomattersofideology. As such, it is closely connected to power structures (Subtirelu, 2014)and refers to the collection of ideas and values that speakers of a dominantdiscourse community commonly hold about language and the hierarchiesbetween languages (De Costa, 2010, 2011; Subtirelu, 2014).When the dominantideologyisamonolingualone,negativeattitudestowardsspeakersoflanguagesother than the official L1 may ensue (Subtirelu, 2014; The Douglas Fir Group,2016), especially when that language is perceived to have lower social status(Jordens,2016).

Ideologiesmayremainat the levelof languagebeliefs,where theyshapeand express latent consensus about what constitutes a good member of adiscoursecommunity(Spolsky,2004),buttheymayalsobetranslatedintopolicymeasures – themeso level. Inmany contexts, language ideologies implicitly orexplicitly shape institutional policies (De Costa, 2010; Linton, 2009; Shohamy,2006; Van Splunder, 2015). Studies have shown that a monolingual languageideologyhasbecomepervasive inmanyEuropeannations (Gogolin, 2002), andtheUS(Linton,2009).ThismayputL2learnersatapowerdisadvantage,sinceitrationalizesorreinforcesthebeliefthatsomelanguagesaredominantorsuperior(Subtirelu, 2014).Matters of ideology andpower impact not only thenorms ofinstitutions and societies, but also themicro level of interpersonal interactionswithinthem.TheycaninfluencetheexpectationstowardsL2learners,therolesthey can expect to assume, and the ideas L2 learners have about themselves(DouglasFirGroup,2016).


143

The Douglas Fir framework can be seen as a broad, three-tier interpretationalframework to explain language gains made by international L2 students thattakes into account ideology, institutional dynamics, and interpersonalrelationships.Typically,studiesoninternationalL2learnersfocusoninteractionswith other students (Ranta &Meckelborg, 2013), on contactwith ESL teachers(Lee, 2008), or on the academic culture of the host institution (Braine, 2002;Seloni,2012).Nostudies inthis fieldhaveyetproposedabroadexplanationforlanguage gains by linking institutional and interpersonal variables, and byconsideringtheminrelationtomattersofideology. SummaryoftheexistingresearchItisclearthatanypresumptionthatinternationalL2studentswillmakelanguagegains simplybyvirtueof attendingclass in theL2, isunsupportedby research.ThereiswideconsensusthatmeaningfulinteractionwiththeL2communityisacatalystforlanguagegains,butinmanycontexts,thisinteractionappearsratherlimited.

Longitudinal studies that focused explicitly on language gains byinternationalL2studentswhoreceivednoadditionallanguagesupport,areverylimited in number however, and have so far only considered writing gains.Additionally,noexistingstudieshavecombinedanattentionfororalandwrittenlanguagegainswith thepersonal andacademicexperiencesof theparticipants.Last but not least, apart from the Douglas Fir framework, very fewcomprehensiveexplanatoryframeworksexist.

RESEARCHQUESTIONSThis chapter reports on the experiences of 21 international students at Flemishuniversities over an eight-month period. Crucially,Dutch is not amajorworldlanguage,butacomparativelysmall language,whichisratherexceptional inanEnglish-dominatedresearchfield.Asa languagewithtwenty-twomillionnativespeakers, Dutch is of modest international importance. This makes theeducationalsettinginwhichthisstudyiscarriedoutuniqueinatleasttwoways.One, itmeans thatmost international studentshavehad little exposure to thelanguageasmediumofinstruction,Dutch,beforestudyingit.Secondly,formostinternational studentsDutch is amedium to reach adesiredoutcome,butnottheactualgoaloftheirstayinFlanders.

The first research question considers language gains made by full-timestudents enrolled in five Flemish institutions of higher educationwhohave aninternationalL2background.Gainsweremeasured,perhapsforthefirsttimeinastudyofthiskind,byre-administeringsectionsoftheactualuniversityentrance


144

test theparticipantshad takenuponenrolment andby analyzingnotonly testscores but also measures of complexity, accuracy and fluency of their output.Importantly, theparticipants receivedno formal language supportbetween thetestandtheretest.Theperformancesonthetestandtheretestwere inspectedquantitatively and supplemented with the participants’ perceptions for thepurposeoftriangulationtoanswerthefirstresearchquestion:RQ1 DideightmonthsatuniversityleadtoDutchlanguagegainsamongtheL2

participants?Basedon the resultsofprevious research, itwashypothesized thatparticipantswouldmakeorallanguagegains,butthatthegainsinthewrittenmodalitywouldbelimitedatmost.Thesecondresearchquestion,whichisattheheartofthecurrentstudy,focuseson explaining the outcomes by analyzing the experiences and perspectives ofinternational L2 students during their first year at a Flemish university. Usingqualitative interviewdata, thedynamics that existwithin theuniversitywill beilluminatedattwolevels,theinstitutionalandtheinterpersonal:RQ2a Attheinterpersonallevel,howmightstudent-studentandstudent-teaching

staff interactions at Flemish universities impact the language learningopportunities of international L2 students and hence their chances atmakinglinguisticgains?

RQ2b Attheinstitutionallevel,howmighttheperceivedpositionsofL2learnersat

their university impact international L2 students’ language learningopportunities?

By operationalizing the Douglas Fir framework, this study was designed toidentify interactionsbetween institutional and interpersonal variables thatmayinhibitorpromotelanguagelearning.Additionally, thisstudygenerateddataaboutdrop-out.Duringorat theendofthedata collectionperiod, someparticipants left university.The third researchquestionlinkstheirreasonstodosotoinstitutionalandinterpersonalvariables:RQ3 Towhat extent could institutional variables or interpersonal relationships

explaincertainparticipants’decisiontoleaveuniversity?This study relies on a sequential explanatory design, in which longitudinalqualitative data serve to interpret quantitatively measured language gains


145

(Creswell, 2015). The measurement of language gains draws on a repeated-measurewithin-subjectmethodology (Kinginger, 2008;Knochet al., 2015, 2014;Storch, 2009). The results pertaining to the other research questions primarilydrawonpatternsidentifiedintheanalysisofthecodedinterviewtranscripts.

PARTICIPANTS&METHODOLOGYParticipantsThetwentyL2Fparticipantswhoparticipatedinthisstudywereselectedfromthelarger L2F population. This group was already introduced in Chapter 2, butcertain characteristics relevant to this chapterwill be reiterated or highlightedhere.

To be eligible for participation in the longitudinal leg of the researchproject,anumberofinclusioncriteriahadtobemet.Participantshadtoregisterforuniversity–theacademicyearfollowingthelanguagetest.PhDstudentswerenotconsideredbecausetheyarenotrequiredtoattendclassesandbecausetheydonotneed tomeet language requirements;neitherwere studentswhosignedupforanEnglish-mediumprogram.Outofthirty-twopossibleL2Fparticipants,twenty agreed to be participants. Appendix 4 lists a number of demographicvariablesperparticipant,and includes informationabout their studysuccess inJuly2015.A“+”indicatesthattheparticipantpassedmorethanhalfofthecoursesheorshehadregisteredfor.Thiscut-offpointwasbasedonthestudysuccessofinternationalstudents inFlanders,who(onaverage)attain47.7%of thecreditstheytakeon(Glorieux,Laurijssen,&Sobczyk,2015:15). Themedianagewas23forthetenbachelorstudents(min=19,max=32)and24forthetenmasterstudents(min=19,max=44).TheoverallmeanageinFlanders is 21 for bachelor students, and 24 for masters (Wartenbergh et al.,2009).FrenchandSpanishwerethemostpredominantL1s,andmostparticipantswere from Europe and Latin America. Four participants were from thefrancophonesouthernpartofBelgium,where theregionalgovernmentsetsoutaneducationalpolicy that is independent fromtheFlemishone. These twentyparticipantsregisteredforaprogramwithinoneofthemainacademictraditions:humanities, exact sciences, and social sciences. Importantly, Dutch was theirmedium of education, but Dutch language learning was not a goal in itself –except for Oksana who studied translation studies with Dutch as the targetlanguage.Mostparticipantswereenrolledatoneofthethreelargestuniversitiesin Flanders: Ghent University (6), the University of Leuven (8), and theUniversity of Antwerp (4). Leila attended an interuniversity program thatincluded these three universities. Stella had intended to register at GhentUniversity, but was denied entry because she had not passed either university


146

entrance language test.She thenregisteredat the smallerUniversityCollegeofHasseltafterpassingthelocaltest.

Thirteenparticipantstookpartintheprojectforitsentireduration.Suchattrition is tobeexpected in longitudinalprojects,evenmoreso inviewof thefactthatuniversityeducationinFlandershaslargedropoutrates.Fortypercentof the students drop out prior to the July examinations of their first year atuniversity(Goovaerts,2012). Inthisstudyattritionwasnotnecessarilyregardedasa lossofdatahowever,sincethereasonswhyparticipantsdiscontinuedtheirstudieswashighlyrelevantinthelightoftheresearchquestions.

Theparticipants receivedno remuneration for theirparticipation.WhenaskedinJune2015whytheyhadchosentoremainintheprojectforsolong,thereasonstheystatedwerebeingabletotalktosomebody(e.g.,Gabriela,Oksana),feeling part of a group (e.g., Elena, Alexandra), practicing Dutch (e.g., Marie,Guadalupe),ordoingsomethingthatmatters(e.g.,Merveille).Datacollection:test-retest,monthlyinterviews,andfieldnotesThe data collection for this study lasted from July 2014 until June 2015, andincludedaninitiallanguagetest,semi-structuredinterviews,andaretest,whichincludedalistening-into-writingtaskandanoralpresentationtask.Test/RetestdesignAll participantshad takenSTRT in the summerof 2014. For the retest inApril2015,itwasdecidednottorunSTRTfully,buttoselecttworepresentativetasks.Thiswasdonemainlyforreasonsoftime;STRTtakesuptofourhours,andatatimewhenthe finalexamswerecomingup, itwasnotconsidered in theirbestinterest to ask theparticipants todevote thismuch timeon a retest.Basedondata from the previous full-scale STRT administration that included the sametasks (N = 913) a multiple linear regression was run inR (QuantPsyc and carpackages) to determine the oral and written tasks that explained most scorevariance(seeTable7.1).


147

Table7.1.Multivariatelinearregression:STRTtotalscore~T1-T6scores B(SE) β (Intercept) -0.008(.011) T1 1.396(.001)*** .161 T2 1.451(.001)*** .257 T3 1.628(.001)*** .167 T4 1.397(.001)*** .216 T5 1.860(.002)*** .161 T6 2.268(.001)*** .295 Note.R2Adjusted=1,p<.000Basedon the standardizedbetavalues, theoralpresentation task (β = .52) andthewriting-from-listening summary task (β = .57)were selected for the retest.Together,theyexplained90.8%oftheoverallSTRTscorevariance(𝑅!"#! =.908,p<.000).Appendix1offersmoredetailsconcerningthesetasks.MonthlyinterviewsThe participants in this study were interviewed every month of the academicyear, exceptduring the study-intensivemonths (December, January,May).Theinterviewsweresemi-structuredandtypicallylastedaround45minutes(Median=44,Min=24,Max=97).Everymonththeinterviewshadadifferentfocus,butissuesregardingtheparticipants’sociallife,academicexperiencesandperceivedlanguageprogresswerealwaysontheagenda.Figure1showsthetimelineofthestudyandindicateswhichparticipantswerenolongerinvolvedatwhichpoint.

Inordernottocreateafurtherdisequilibriuminthepowerbalancethatisinherenttointerviews,theparticipantsalwayshadasayindeterminingthetimeand place of the interview (Sin, 2003). Some preferred a café, somewanted tomeet in the university cafeteria, and others felt comfortable to meet at myworkplace, where the seating had been arranged to create a collaborativeatmosphere. The researcher invested energy in developing a sense of mutualunderstandingorcommunalitybysharingpersonaldetailswhenrequested,andby establishing rapport (Manderson, Bennett, & Andajani-Sutjahjo, 2006).Familiarity with the interviewer can generate a sense of comfort and security(Pellegrino Aveni, 2005), and the participants in this study felt free to discusspersonal matters. At the end of the year, some participants (e.g. Alexandra,Guadalupe, Oksana, Gabriela, Julia, Hoang, and Anastasia) acknowledged thattheyhadsharedthoughtsandexperienceswithmethattheyhadnotsharedwithotherpeopletheytrusted.


148

FieldnotesTheresearcherkeptfieldnotesduringthedatacollectionperiod.Inthischapter,onlythenotestakeninNovember2014willbereferredto.Duringthatmonththeresearcheraccompaniedtheparticipantstoaclassoftheirchoosing,takingnoteoftheinteractionsandoftheparticipants’responsestothein-classdynamics.Allclasseswereaudio-recorded,exceptforone,becausetheprofessorhadnotgivenpermissiontodoso.AnalysisTestdataTheperformanceswereanonymouslydouble ratedby two independent trainedSTRT raterswhoused the STRT rating scale. The STRT ratingprocedure takesinto account content criteria and linguistic criteria, which are scored on anordinalfour-pointscale.Duetotheordinalnatureofthescoresandtherelativelylownumberofparticipants,Wilcoxon’sSignedRankTestwasusedtodeterminethesignificanceof scoredifferences (Field,Miles,&Field,2012),anda functionwascreatedinRtocalculatetheeffectsizer.Sinceitispossiblethatbandscoresaretoobroadtocapturelanguagegainsoveraperiodof a fewmonths (Green, 2004), gains in termof complexity, accuracy,and fluency were calculated too, using a methodology based on Llanes et al.(2012) and Serrano et al. (2012). Written and oral fluency were determinedrespectivelyby computing theamountofwordsperT-unit, and theamountofsyllablesperminute(i.e.,prunedtoexcluderepetitions,falsestartsandthelike).Lexical complexity was calculated using Guiraud’s Index. The measure foraccuracy was the amount of errors per T-Unit for the written data and theproportion of errors per AS unit for the oral data (Foster, Tonkyn, &Wigglesworth, 2000: 365). Wilcoxon’s Signed Rank Test and effect sizes wereusedtodeterminethesignificanceandmagnitudeofpossiblegains.


149

InterviewsAll transcriptionswere coded and analyzed inNVivo 11 ForMac. Based on theliteraturereview,anaprioricodingschemewithfixedcodingcategorieswassetup(Miles,Huberman,&Saldaña,2013).Themainbranchesoftheaprioricodingtree were “Background variables” (6 categories), “Academic work” (11subcategories),“Languageuse”(7subcategories),“Languagetests”,and“Identity”(20subcategories). I codedall interviewsusing thesecategoriesbutwas free toaddanopen,inductivelayerofcoding,whensalientthemesemerged(DeCosta,2011).One such codingwas “Key transformational episode”,whichwas used tomark instances or anecdotes that had a profound impact on a participant andrecurred in different interviews with the same person. To check for codingaccuracyatrainedresearchassistantcodedtheNovember2014interviews(54239words)usingtheaprioricategories.Therewasaanexactagreementbetweentheratersof>90%(Landis&Koch,1977).

Afterdataanalysis,allcodingcategoriesrelevanttotheresearchquestionsathandwerecombinedintodatamatrixes(O’Cathain,Murphy,&Nicholl,2008),which included the most prominent indicators of variables affecting languagegainsontheinstitutionalandinterpersonallevel(seeDeweyetal.,2014;Engle&Engle, 2003). On the institutional level, four variables were included: classes,examinations, use of support systems, and language use at university. Theanalysisof the interpersonal level focusedonsocialnetworksatuniversity (i.e.,teaching staff and students). As an addition to manual coding, the mostfrequently occurring content words in the transcriptions of the participants’speechwereidentified.Thesetextsearchesofferacomplementaryperspectiveonthedata.

RESULTSLinguisticgainsaftereightmonthsatuniversityBetweenthefirstSTRTadministrationinthesummerof2014andthesecond,inApril 2015,no significant speakingorwritinggainsweremade, asmeasuredbySTRT.Themedian score (seeTable 7.2) indicates a slightnon-significant scoregainon thewritten summary,withamediumeffect size (r = -.31).Themedianscore for the presentation task has decreased over time, but the effect isnegligible(r=-.05)andthedifferenceisnotsignificant.


150

Table7.2Languagegains(N=13) MedianTest Medianretest p rWrittensummary STRTscore 17.25 19 .159 -.31 Nwords 207 240 .07 -.4 Lexcomplexity 56.5 57.5 .89 -.03 Syntcomplexity .26 .25 .79 -.06 Accuracy .94 .80 .8 -.06 Fluency 9.9 11 .85 -.04Presentation

STRTscore 28 27 .824 -.05 Nwords 424 345 .03 -.5 Lexcomplexity 38 37 .96 -.01 Syntcomplexity .41 .38 .48 -.17 Accuracy .86 .86 .66 -.1 Fluency 12.1 13.7 .73 -.08The only significant (p < .05) difference and the largest effect (r = -.5) is thedecreased amount of words used in the retest of the presentation task. Thesecond largest effect (r = -.4) is the increased number ofwords in thewrittensummary. Word counts convey no information about the quality of aperformance however. All indicators of text quality – syntactic and lexicalcomplexity, accuracy and fluency – showed that no significant or substantiallanguagegainsweremade.Intermsofeffectsize,thelargestdifferenceisfororalsyntacticcomplexity,whichdecreasedslightlybetweenthetestandtheretest(r=-.17).

Thequantitativeresultswereconfirmedbytheoverallintuitiveestimationof the participants. At the end of the year, no participant felt that his or heroverallDutchlanguageabilityhadimproved.Somefeltmoreself-confidentwhenusingDutch(Guadalupe),orbelievedthatoneortwoskillshadimproved:Afewparticipants reported perceived gains in writing (Marie), listening (Emma,Merveille)or lexicalrange(Marie,Ersi).Ontheotherhand,somealsobelievedthattheirspeakingabilityhaddecreased(Ersi),orthatnogainswhatsoeverhadbeenmade(Oksana,Elena,Gabriela).


151

InterpersonalrelationshipsTable 7.3 shows the recurrence of the words “Different” and “Difficult” in theinterviewee transcripts. A detailed analysis of the interviews confirms that allparticipants had a difficult time adjusting their new role in a different reality.Thisremainedtrueatthelevelofallinterpersonalrelations.Table7.3.Reduceddatamatrix:interpersonal

Cod

es

Interviews

Freq

uentw

ords

Timesused/

code

Facultystaff 35 25 Different .8

Dutch .7

Question .6

Students 437 65 People 1

Different .86

Difficult .50

Table 7.4 shows an overview of the perceived attitude of the participants’teachingstaffandtheirL1peers,andindicateswhattheparticipantsconsideredtheir best and theirworst social experienceof the year.The interview analyseswillbediscussedindetailbelow,butthetrendsareclearfromthetable:onlyoneparticipant described the teaching staff as involved, while others mostlydescribed their professors and teaching assistants as distant, with occasionalpositive(respectful,kind)ornegative(demotivating,face-threatening)qualifiers.MostparticipantsalsodescribedtheattitudeoftheirL1peersasclosedordistant.In the first semester only Elif described the attitude of some of her fellowstudentsinapositiveway.Afteroneyear,notmuchhadchanged.TherestillwasnoregularcontactbetweentheparticipantsandtheirL1peers.


152


153

If we inspect the evidence about students’ perceptions of the teaching staff attheiruniversity,therelationshipcanbestbedescribedasdistant.Becauseofthepredominant teaching style and because of the group sizes,which could be aslarge as 500 enrolled students in a classroom, interaction between theparticipants and the professors was mostly non-existent, or limited to anoccasional question. Stella was the only participant who reported regular,meaningful interactionwith her teachers. The nineteen other participantswhowereaskedabouttheirperceptionoftheteachingstaffduringthefirstinterviewdescribedcontactwithteachingstaffaslimited,alsowhentheywereenrolledinsmallerprogramsofabout50students.

For threeparticipants the fewencounterswiththeirprofessorshadbeenpositive experiences, but for others these interactions confirmed the feeling ofhierarchic distance. The only participant who described her professors asinvolved, was Stella. Yazdan, Merveille en Elif described the relationship withtheir professors as distant, because professors had toomany students and toolittletime,butoverallpositive.Otherparticipantsreportednotinteractingwithprofessors (e.g., Emma, Elena), or shared anecdotes of interactions withprofessors that were perceived as condescending, distance-inducing or eventhreatening(Alireza,Leila,Clara,Anastasia).

Marie, a francophone student in Flanders claimed that her professorsreacted ratherpositively toherbeinganL2 learnerofDutch.Others,however,offered frequent anecdotes of negative encounters with professors concerninglanguage-relatedissues.Atleastnineparticipantsreportedfeelinguneasyaboutbeing an L2 student in class, but not all of these participants had concreteanecdotestosubstantiatethisfeeling.Claradiscussedconcreteepisodes,suchasthisone:

Last time theprofessor askedmeaquestionand I got all redand I said“sorry I speakFrench, Ididn’tunderstandeverything”.She lookedatmewithoutasmileandthenshelookedintotheauditoriumandsaid“madamspeaksFrench”.

(Clara,November2014)Likewise,Oksana,aUkranian-bornstudentoftranslationstudies,describedhowoneprofessorhadreactedquitenegativelytoherattendingclassasanL2learner:

So when I needed to discuss my curriculum I emailed one of theprofessors,whosaid“I’msorrybutifyoumakesuchbasicmistakesinane-mail,thisprogramwillnotbeattainableforyou”.

(Oksana,October2014)


154

Onemonthlater,IaccompaniedOksanatoaclasstaughtbythesameprofessorandtooknoteofthefollowinginteraction.

Theprofessor asksher aquestion about aword in thedialect of coastalFlanders.“Youareotherlingual,right?What’syourmothertongue?”“UkranianandRussian”“I bet they don’t say this in Ukranian” [pronounces dialect word,classroomlaughs][…]After class, he comes over to Oksana andme. They have never spokenbefore. He asks her which courses she has and how long she has beenstudyingDutch. “Sixmonths”, shesays. […]He is surprisedathowgoodherDutchis,buttellsherthatshewillprobablynotpass,thatsheshouldconsiderquittingandfindingajob.

(FieldnotesNovember2014,Oksana)Thenegativenatureof this interaction is anexception in the fieldnotes. Inallother classes observed, with the exception of one (Stella), professors wouldtypicallydeliveranexcathedratalk.Somedidnotinteractatall,asinAlexandra’sclass:

There’salotofloudtalkingwhiletheprofessorkeepsontalkingwithouteitheraskingforsilenceorstimulatingdialogue.

(FieldnotesAlexandra,November2014)

Other professorswhowere observed for the current study kept a distance andmadeoccasionalcommentswithoutreallyinteracting,butmostdeliveredatwo-to three-hourmonologue inanauditorium.ThesettingofAnastasia’sclasswasexceptionallyuncomfortable:

“The room is very dark and it’s hard to seemy notes. The benches arehard. Teaching assistant delivers monotonous monologue sitting down,power point projected above her head. […] At one point a studentwhispers something. The TA immediately stops talking and says excuseme,thisisverybadformyfocus”.

(FieldnotesAnastasia,November2014).All participants confirmed that the class I had attended was representative ofotherclassesofthesamecourse.


155

Inshort,mostL2Fparticipantsperceivedtheirrelationshipwiththeteachingstaffasdistant.ThiswasalsothegeneralperceptionoftheirrelationshipwithFlemishL1 classmates.With the exception of Ersi, all participants found it difficult tobefriendFlemishstudents.Fromtheanalysisofthedata,threemainpatternscanbe identified: (a) networks among Flemish students appeared impenetrable tothe participants, (b) many participants perceived a power imbalance betweenthemandtheFlemishL1students,and(c)manyparticipantshadtorenegotiatetheiracademicidentity.InwhatfollowsIdiscusseachinturn.

The themeof closedL1 networkswas prominent during thewhole year.With227instances,Socialdistancefrompeerswasthemostfrequentlyusedcodein all transcriptions combined.Most participants agreed that Flemish studentsmadeaclosedimpression,andallweresurprisedbythedifficultyofbefriendingFlemish students. In the October interview fifteen out of twenty participantsmentioned feeling stupid, incompetentorunlikeable among their L1 peers. Forstudents like Anastasia, the perceived impenetrability of L1 networks and theperceivedindifferenceofherL1peerswerethefinalargumentinherdecisiontodropout.

After the first month it was clear that I would not be happy here. ThestudentspretendlikeI’mnotthere.Idon’tthinktheyshowanyinterest.TheytalkamongeachotherandwhenIsaysomething,theyanswerandstarttalkingtoeachotheragain.

(Anastasia,February2015)

Particpants who registered for master degrees referred to already-formednetworksof studentswhichwerehard topenetrate.Butalso first-yearstudentslikeGabriela,Hoang,andChloédidnotgainaccess toL1 studentnetworks.Totheparticipantsthesesemi-impenetrablegroupsseemedtohavebeenformedinsecondaryeducation,orappearedtoberegionallydetermined.

SomeFlemishpeoplearea littledifficulttobefriendIthink.[…]Italkedabout thiswith other international people and everybody says the samethings:youguyshavegotthesamefriendsfromwhenyou’relikethreetillyou’relikeseventy.It’sveryhardtofindyourwayintoagroupoffriends.

(Gabriela,October2014)

Somebody else toldme “we already have a group of friends, why add afrancophone?” He wasn’t being mean, but it was like “we are fromHasselt”, “we are from Kortrijk” [two small Flemish cities with distinctdialects].

(Chloé,October2014)


156

In June, when asked which aspect of the last year they would have liked tochange,sixoutoftheremainingthirteenparticipantsreferredtothedispositionofFlemishstudents,whichwascharacterizedaspoliteandfriendlybutdistant.For a few participants, the situation improved in the second semester, as theybecameacquaintedwithFlemishstudents.Butevenso,onlyErsiandAlexandrareportedhavingL1friendsinclass.Others(Marie,Merveille,HoangandEmma)didnotgoasfarastocallanyL1classmatestheirfriends.

Mostparticipants remaineddistant from theirL1peers, butnot allwereaffectedbythisinthesameway.Alireza,ratherintrovertbynature,didnotreallymindthelackofcontact.LeilaandGuadalupe,respectively44and30atthetimeofdata collection, also felt distant from their fellow students, butdid seekoutcontact because of the age difference. Nevertheless, irrespective of how theparticipantsrespondedtotheattitudeofFlemishstudents,allofthemperceiveditasclosed.

-Canyoudescribethecontactbetweenyouandyourfellowstudents fromFlanders?Non-existent.Nothing.-Doyoumindthatsituation?Iusedto.Now?No.IhavelotsoffriendsfromtheUkraine,fromRussia,fromeverywhere.IhavefriendsIcangotobarswith.Iamnotalone.

(Elena,April2015)

ElifandErsiweretheonlyparticipantswhohadmadeFlemishL1friendswithintheirstudyprogramduringtheirfirstsemesterasstudents.Ersihadmadeafewfriendsduringaconferencethatsheandherclassmateshadattended.Duringthefirst semester Elif, who was enrolled in a comparatively small and specializedmaster’sprogram,hadoccasionaldinnerswithafewclassmates,someofwhomwereFlemish.

Thatdoesnotmean that all Flemish studentswere closedhowever, andnot all participants were socially isolated. Quite a few had a group of friendsoutsideuniversity,andalthoughmostparticipantsusuallyinteractedwithotherinternational students, some had Flemish friends outside university.Merveille,for example,hadquite a few friends ather student accommodation.Anastasia,OksanaandAlexandrahadFlemishboyfriendsatthetimeofdatacollection,buttheydidnotconsidertheirboyfriends’friendsastheirown.TheirdescriptionsofNewYear’sEveforexamplewerestrikinglysimilar.

Ididn’twanttogooutwithmyboyfriend’sfriendsbecausetheyaren’tmyfriends. […] So I studieduntil 11.30PMand thatwas really hard. But thenextdayIboughtatickettoPeru.

(Alexandra,February2015)


157

Ijustwentouttocelebratewithmyboyfriend’sfriends.ButIdidn’tlikeit.I didn’t feel like celebrating. I’m having a hard time here without myfriends.[…]AndIhaven’tseenmyfamilyintwoyears.

(Oksana,February2015)Loneliness, insum,wasapervasive feelingamongthe internationalstudents inreactiontotheclosedL1networks:

Mybiggestproblemherewasloneliness.Ihavealmostnocontactwithmyclassmates. Sometimes I just have questions or I don’t understandsomething,andthenthereisnobodytohelp.

(Anastasia,February2015)ButfeelingexcludedfromL1friendshipsandcopingwithlonelinesswasnottheonly challenge the international students faced. Throughout the year,participants referred to instances of power imbalance, which in most casesresulted from feeling inferior in terms of language ability, not quiteunderstandingtheculture,feelingexcludedsocially,orfearingridicule.InMarch,allparticipantsansweredthequestion“Howdoyouthinktheotherstudentsseeyou?” None of the answers given (see Table 7.5) indicate that their fellowstudents know their personality. Two participants offered a somewhat positiveresponse,buttheytoofocusedondifferencesratherthancommunalities.Inquitea few cases, participants shared stories of exclusion. Typically, these storiesrevolvedaroundoneparticipantwishingtobeincludedinanetworkofL1peers,but not gaining admittance. In March, Gabriela recounted one anecdote shereturnedtoduringeverysubsequentinterview:

This has happened like two or three times. I’m sitting here, right? Andthere’s twogirlshere [points right]andhere [points left]. I talk to thesegirlssometimes,oroccasionally,orwhatever.Anywaysoonegirlaskstotheother“doyouwanttogetacoffee”?AndI’mrightinthemiddle!Andthen Iwonder can I go toobut then I think I’m in themiddle and theydon’taskme.ThesetypicalthingsIdon’tunderstand.Ithinkit’sterrible.

(Gabriela,March2015)


158

Table7.5.Howdothinkyourclassmatesseeyou?(March) Negative Elena ThatIamstupid Guadalupe ThatIamweird Alireza TheythinkofIraninstereotypes Marie Theyaresurprised.Sometimestheythinkthatwearelazy. Hoang IhopetheycanseethatIamkind.Indifference Alexandra Theyareindifferent Leila Iamisolated Elif Idon’tknow Oksana Theydon’tpayattentiontome Merveille Theyjustsayheyandthat’sit Gabriela Theyaren’tinterestedPositive Ersi TheyappreciatethatIlearnedDutchsoquickly Emma Somearesurprised,somelovethatIamforeignIn some cases, the inhibitions and preconceptions of participants obstructedinteractionwith theirL1peers.ForHoangandLeila, language imbalancemadethem feel like an outsider. Both had been addressed and invited out by L1students,butkepttheirdistancebecausetheyfeltlinguisticallyinferior.

I don’t dare to speak because [the L1 students] talk so quickly and youdon’t understand. But then you try to say something and they don’tunderstand.Itmakesthingsdifficult.Suchshame.IfeelsuchshamewhenIneedtospeaktosomebody.

(Hoang,February2015)

Ihavelittlecontactwiththeotherstudents.Inthisclassforexample,thisis an English course and we are equal.We are on equal terms when itcomestolanguage,andIfeelnoembarrassment,noshame.Ifeellikeanoutsider,buttheyareoutsiderstoo.-Whydoyoufeellikeanoutsider?That’s normal. They have an advantage that I lack. They are Dutchspeakers in a Dutch course. I am francophone and my Dutch is notperfect.Thatisdifferent.SometimesIwantedtoaskaquestion,buttoldmyself“nonono”.

(Leila,July2015)Occasionally too participants asserted power over their L1 peers by stressingdifferences in age andperceivedmaturity in aderogatoryway. For example, atdifferent occasions participants referred to their fellow students as children,babiesorteenagers(Leila,Guadalupe,Anastasia,Hoang,Alexandra),orpointed


159

outdifferencesbetweenthemselvesandtheirL1peersintermsoffinancialmeansorparentalsupport(Anastasia,Hoang,Alexandra).

Bytheendofthesecondsemester,mostparticipantshadbecomemilderintheiropinionabouttheFlemishstudents’behavior.Quiteafew(e.g.,Hoang,Emma and Gabriela, Elena, Guadalupe) had accepted that there was a powerimbalance,andthatinternationalL2studentshadtomakealargerefforttogainsocialacceptancethanFlemishstudents:

Iunderstand.Youlivehereandyougetalltheseforeigners,andyoucloseup. It’snormal. I’ddo thesame. […]Youcan’texpectBelgians togo like“comehere,strangeperson”.

(Guadalupe,March2015)Ascanbeinferredfromtheresultsabove,mostparticipantsfeltsomewhatoutofplace and socially isolated at university. Over time, however, they may havestarted feeling more legitimate as peripheral participants in the world of theuniversity. Halfway through the second semester the participants were askedwhethertheyfeltathomeinclass.Fourparticipantsconfirmedyes,sevenwereindoubt,andtwodidnotfeelathome.Thesetwostudentsmotivatedtheiranswerby referring to being out of place as a non-Flemish student at a Flemishuniversity:

TheclassesareforFlemishstudents[…]Notmuchattentiongoestonon-Flemishstudents

(Hoang,April2015)When discussing why they did or did not feel at home in class, the studentsmentioned three primary reasons: understanding the language of instruction,having friends, and understanding the content of the class. ForMerveille andGuadalupe,forexample,realizingthattheyhadstartedtounderstandtheclasses,markedacleartransformationinhowtheyperceivedofthemselvesasstudents.

Clearly, feeling at home in class was linked to having friends there. Inmost cases the sense of lonelinessmay not have been the product of Flemishstudentswillingly and noticeably underappreciating the international students,butsometimesFlemishstudentswereperceivedasnotvaluingthecontributionsof their international peers, or as sabotaging them academically. For example,OcéanerecallshowoneFlemishstudenttoldherthathisclassnoteswerefreeforhisFlemishfriends,butthatshehadtopay€20.Elenareceivednohelpwhensheasked classmates to check a Dutch text she had written. And the Flemishmembers of one group Alexandra belonged to made agreements withoutinvolvingher:


160

IwasinvolvedingroupworkandweweresupposedtospeakEnglishbutthey always spokeDutch. Sowe speak and yeahwhen I had ideas theyheardandignoredmeinacertainway.I’mthere,Ialwaysgoontime,readeverything–exceptmaybeonce–buttheyignoreme,donotincludeme,andchangethetopic.

(Alexandra,November2014)Even though the situation for Alexandra improved during the year, and shegainedmorepositiveexperienceswithothergroups,thefeelingofbeingexcludedfromtheacademiccommunitypersistedandstillimpactedthewayshefeltaboutbeingpartoftheacademiccommunityattheendoftheyear:

Iwenttothelibrary[…]Isawsomeclassmates,buttheyjustignoredme.[…]ThatwasreallyhardcauseIreallytry.Itrytokeepmydistancefrompeople frommycountry. I try toget integrated,but itdoesn’twork. I’mnot looking for a deep friendship, but at least some eye contact, youknow?[…]Iwasreallycrying.Ididn’twanttogotothelibraryanymore.Ihad expected maybe to see somebody or to talk to somebody or if Idoubtedsomethinginthecourse,tofindasolutionwithsomebody.

(Alexandra,June2015)Lastly, understanding course content appeared to be a precondition for theparticipants’feelingofbeingalegitimatememberofclass.ParticipantswhohadstartedtofeelathomeinclassbythesecondsemesterincludeGuadalupe,whohadstartedtheyearwithrelativelylowexpectations,buthadstartedstudyingonadailybasisafterattaininggoodgradesinJanuary.Allparticipantswhodidnotreallyfeelathomeinclassandmentionedstudy-relatedreasonstosubstantiatethat opinion, stated that they no longer were the student they rememberedthemselves to be. Elena, Alexandra, Guadalupe, Oksana and Anastasia talkedabouthowthefeelingofbeinganaccomplishedstudenthadbeenreplacedwithasenseofdoubt:

HereIsometimesfeelstupid.[…]IhadagoodjobintheUkraine.Therewasacompetitionthat includedprofessorsandeverything.ButIgotthejob. Iworked at the airport as an aviation engineer. […]Here therewasthis situation in classwhen I didn’t understand something as quickly astheothersdid,andsomepeople laughed, like“hahaha,youare fromtheUkraineandyouarestupid”.

(Elena,February2015)


161

Once,wehadaguest lecture inEnglishand itdealtwith linguisticsandthe lecturer askedmany questions and I could answer, because it dealtwithlinguisticsanditwasinEnglish.Ikneweverything!AndforthefirsttimeIfeltathome.NormallyIamjustconfusedallthetime:“WhyamIhere?Idon’tunderstandanything.”

(Oksana,February2015)InstitutionalsupportFeelingathomeintheeducationalsystemTable 7.6 shows that the most frequently used words in the participants’transcriptswere“difficult”and“different”.Theinterviewanalysesalsoshowthatnoparticipantsfoundtheirwayintothesystemwithoutahiccup,butthatmostparticipants became accustomed to the organization of their university in thecourseofthesecondsemester.Table7.6.Reduceddatamatrix:institution

Cod

es

Interviews

Freq

uentw

ords

Timesused/

code

Classes 286 65 Moredifficult .68

Different .66 Everything .43

Examinations 149 43 Difficult .82 Passed .60 Different .60

Supportsystems 107 47 Different .55 Dutch .45 Students .42

Languageuse 277 59 Dutch .76 Difficult .71 Different .48


162

ThreeaspectsoftheinstitutionalorganizationofFlemishuniversitiesturnedouttobehurdlestolearning:thelargeclasssizes,theteacher-andlecture-centeredpedagogicalapproach,andexams.Inwhatfollows,eachwillbebrieflydescribed.

During every interview the participants were asked to characterize thedidacticculturetheyexperiencedatuniversity.Overall,themostcommonlyusedcharacterizations were “not interactive”, “big groups”, “boring”, “solitary”, and“scary”.ThefirsttwolabelsofthislistappearedineveryinterviewconductedinOctober,but as the yearprogresses,participants startedusing them to a lesserextent. InNovember, “boring” isusedmostoften,andafterFebruary, “solitary”becomes the term most often used to characterize the classroom experience.These listsof terms indicate that theparticipantsdidnot enjoy their classes atuniversity,especiallyduringthefirstsemester.

Aftertheirfirstclass,allparticipantsmentionedthelackofinteractionasa striking characteristic of the classroom pedagogy, but it was not necessarilyseenasadisadvantagehowever.Mostparticipantsfeltthatanexcathedrastyleofteaching made classes rather monotonous, but on the other hand, they werehappythatitshieldedthemfromhavingtospeakinsuchbiggroups.Throughouttheyear, fifteenoutof the twentyparticipantsexpressed language-related fearsabout answering questions in class. Group sizes had an impact on theirwillingnesstospeakinclass,butalsomadeiteasiertodisappearinthecrowd.

It’sabitscary,soIdon’tliketospeakinclass,becauseIamafraidtoshowmyself.

(Hoang,November2014)Ihadclassinbigauditoriafor500students[…]atfirstIwasshocked,butnowIhavefriendsit’snoproblem.AtfirstIwastotallyalonethere.Ifeltsosmall.

(Ersi,June2015)

In October, only Stella,Merveille and Jessica shared positive comments abouttheir in-class experience, but as the year progressed, the opinion of someparticipants started to shift. By March, three out of the remaining thirteenparticipantsstilldidnot feelathomeinclass(Hoang,Elena,Oksana),but fourdid (Ersi,Emma,Merveille,Alireza) , and the remaining sixdid so toa certainextent(Gabriela,Alexandra,Marie,Guadaloupe,Elif,Leila).Correspondingly,thenumberofnegativewordsusedtodescribeclassdecreased,andinJuly,at leastsix out of the remaining thirteen participants looked back on their classroomexperiencesinsomewhatpositiveterms.


163

It’sashamethatIdon’tknowmanypeople,butapartfromthatitwasfun.Ilearnedalot,andIlikeditwhenIunderstoodtheprofessor.[…]WhenIlistentotheprofessor,Ireallyfeellikeoneoftheotherstudents.

(Guadalupe,June2015)Thestyleofteachinghadnotchanged,butsomeparticipantshadadaptedtothedidacticculture,whichneverthelessremainedadauntingexperienceforothers:

Inhindsight,sometimesweweretoomanyinoneroom.Wewerelike300ormore.[…]Wehadsomeinteractioninonecourse.Theprofessorwouldpicktwostudentstositinfront;andhewouldaskthemquestions.

(Marie,June2015)Otherparticipantsneverfeltathomeinclassfortheentireyear.

IfI’mbeingtotallyhonest,thisyearhasbeentheworstexperienceIhaveeverhad.[…]Isawtheprofessortalkwithotherstudents,andsometimestheywerelaughing,like“ohwelldone”or“maybepickadifferentanswer”,andwithme itwas always like “no, that’swrong”. Itwasn’t in a sad, orangry,orirritatedway,butIneverheard“welldone”.AtonepointI justwantedtohearthatIhaddonesomethingright.Butnever,andIthought“comeon,Iworkedsohard”,andallfornopositivefeedback.

(Oksana,July2015)AtFlemishuniversitiestherearethreeexaminationperiods.OneinJanuary,afterthefirstsemester,oneinMay,afterthesecondsemester,andoneinSeptember,forstudentstoretakeexamstheyfailedduringtheyear.Typically,allexamsareplannedinatwotothree-weekperiod,whichisprecededbyathree-weekstudybreak.

During the first interview, in October, eight participants spontaneouslybrought up the topic of examinations. In every instance, the participantmentionedfearingtheexaminationsbecausetheyfeltthattheydidnothavethelanguagecompetencerequiredtosuccessfullysitawrittenoranoralexam.

Speaking on an examination is something I can’t imagine myself doingrightnow–allthosedifficultwordsIjustdon’tknowyet.[…]Mostexamsareoral, somearewritten,but that is tricky too.Environmental law is awritten exam. It has written questions you have to provide a writtenanswerto,usingdifficultwordsthatIdon’tevenknowhowtowrite.

(Anastasia,October2014)


164

After the firstexaminationperiodparticipantsdidnotperceive languageas themainproblemtheyencounteredhowever;loneliness,stressandmonotonywerethe most frequently mentioned problems the participants associated with theJanuary examinations. For some (e.g., Alexandra and Elif), the exams were achallenge they were proud to have withstood. For others, it had been ademotivatingexperience.

Youalwaysjusteat,studyandsleep.Thatwasboringforme.Superboring.[…]I thinkIhaven’tseenmanypeople inamonthandwas justaloneathome.OnsomedaysIdidn’tdomuchbecauseitwassoboring.[…]

(Gabriela,February2015)WhentheJanuaryexamsbegan,mostparticipantswereunprepared.Manyweresurprisedbytheamountofstudyingandbythememorizationthatisexpectedinmanycourses.Quiteafewparticipantshadbeenunabletostudyalltherequiredmaterial,andmanyfeltthattheyhadstartedstudyingtoolate,orthattheydidnotknowwhatwasexpectedofthem.

Flemishstudentsknowwhattostudy.“Likeoneofmyclassmatessaidwhyare you studying that? It’s nice to know but not necessary to pass.” […]Theygrewupwiththisimplicitknowledge[ofhowyouneedtostudy].

(Alireza,April2015)Marie,Oksana, Elif andAlexandrawere the only participantswhowere happywith their results on the January examinations and who did not change theirstudyapproachinthesecondsemester.Allotherremainingparticipantsdecidedtostudymore,tostudymoresystematically,andtofocusmoreonmemorization.

ThelevelishighherebutwhenBelgianstudentsstudy,theymemorize.Iexpectedquestionslike“explainthis”,butalltheydoismemorize.

(Elena,June2015)AftertheJulyexaminations,eightoutoftheremainingsixteenparticipantshadpassedatleasthalfofthecoursestheyhadtakenon.PreparednessfortheuniversitylanguagedemandsHere,webriefly revisit theL2Fparticipants’ readiness for the real-life linguisticdemandsofacademicstudiesatuniversitythatwerediscussedinChapter2,butfrom a longitudinal perspective, and with attention to the impact of languageproficiencyonidentity.


165

Atthestartoftheacademicyear,noneoftheparticipantsfeltfullypreparedforthe listening demands of university. They felt unprepared for the variety ofaccentsusedbylecturers,forthevarietyintermsofpronunciations,andforthelexical range they were expected to master. A few participants reported nothaving understood anything of the first classes they attended (e.g., Yazdan,Océane, Leila), and some students never quitemanaged to get over the initialshock they felt when they attended their first class and discovered theyunderstood little or nothing. Emma, for example, described a feeling of panicwhen confronted with Dutch during the first semester. During the secondsemesterthisfeelingfadedaway:

Iwasreallypanickinginthebeginning.Reallypanicking.Ididn’tseeawayout.[…]Itwasjusttoomuch,andIunderstoodsolittle.

(Emma,November2014)

-Can youname three keywords that describe the examinationperiod foryou.Justpanic.[…]Panic.Yes.Becauseofthelanguageproblem.

(Emma,February2015)

[My new study strategy] works better. I won’t panic anymore if I don’tunderstandsomething.

(Emma,March2015)

I have no idea what happened during that first semester. I really don’tknow. Iwasn’t thinking, Iwas just panicking, but actually I don’t thinkthereisanyreasonto.

(Emma,June2015)Butbythen,Emma’schancesofachievingacademicsuccessthatyearhadbeengravelyreduced.Most participants identified listening as their main language-related problem.Readingwasperceivedasdifficult and time-consuming,butnotunsurpassable.AllparticipantsreportedspendingatleasttwiceasmuchtimestudyinginDutchas theywould studying the samematter in their L1. At least two students hadtranslated extensive sections of Dutch coursework (Oksana) or entire syllabi(Stella) into their L1.Other participants (e.g., Gabriela,Océane, Clare) studiedEnglishorFrenchcoursebooksthatweresimilartothecompulsoryDutchones.

Astheyearprogressed,mostparticipantsfeltthattheirreceptivelanguageskillswerebecomingbetter,butmanyperceivedtheirproductivelanguageskillsasstableordeteriorating.Oneofthereasonsforthiswasalackofwritingtasks


166

duringtheacademicyear,andanabsenceofopportunitiesforspeaking.AtleastfourparticipantsreportedavoidingsituationsthatwouldrequirethemtospeakDutch (Elif, Leila, Hoang, Emma). These participants, and others (e.g.,Guadalupe)consideredtheinterviewsarareopportunitytospeakDutchfreely.TheuseofsupportsystemsLastbutnot least, support systemsturnedout tobedesirablebutnot inplace.Throughout the year, the use of and experience with two different kinds ofsupportsystemswasinvestigated:theuniversitypolicytowardsinternationalL2students,andtheuseofstudysupportservice.Ontheuniversitylevel,theredoesnot appear to be a clearly communicated policy for international L2 students.Participants typically felt that they did not really belong to the group of L1students,ortothegroupofErasmusstudents,butthattherewasnoclearpolicyforthegroupofinternationalstudentstheyassociatedwith.

Weareaninvisiblegroup.PeopleseeErasmusstudents,Flemishstudents,but international students are not very visible […] If you want to getintegrated,thelabelofinternationalstudentisnotwhatyouneed.

(Alexandra,March2015)The support systems for international students during classes or examinationsdidnotappeartoberegulatedinasystematicway,butseemedtovaryfromoneprofessortothenext,andseemedtodependontheassertivenessofL2students.Someparticipantswhoaskedtouseadictionary,ortotakeanexaminEnglishwere allowed to do so, while others who had not askedwere not able to takeadvantage of this opportunity. Similarly, someprofessorswould allow someL2studentsaccommodationsduringexaminationssuchasextratimeortheuseofEnglish,whiletheircolleagueswouldnot.

Someprofessors letususeadictionary,whileothersdonot.Someletususe the French version of the same law,while others donot.And somegivealittlemoretimeduringtheexam.

(Marie,March2015)

I asked [thephilosophyprofessor] if I coulddo theexam inEnglishandshe saidyes,but last semester I asked the same toa literatureprofessorandshesaidno[…]AndwhenIaskedthestudentaffairsdepartmenttheysaidno.SoIdon’tknowwhoisright.

(Gabriela,March2015)


167

International students can use the study advisory services that are open to allstudents.Threeparticipantsknewoftheexistenceofthisservice,regularlyusedit,andvaluedthesupporttheyreceivedthere.Othersdidnotknowitexisted,orfelt that the study support did not deal with the specific problems theyencounteredasinternationalstudents.Participantswholeftuniversity,andtheirreasonstodosoThe participants who left university voluntarily did so for a combination ofinterpersonal and institutional factors. Océane and Clara left primarily out ofunhappinesswith institutional factors,butstatedthattheyhadhopedtogettoknowmore Flemish students. Both had underestimated theworkload and hadoverestimatedtheirabilitytoreadcoursematerialinDutchatthesamepaceastheydidinFrench.ForOcéane,herinabilitytounderstandsomeofthelecturers,contributedtoherdecision.

It’sgotnothingtodowithDutch.It’sjusttoomuchwork.[…]YesterdayIread 45 pages and it took almost thirteenhours ofwork […] I’m just sotired.Peopleexpectmetoworklikethiseveryday,butI’msorry,Ican’t.

(Clara,November2014)

InFrench,evenwhenIdon’t listencarefully, Iunderstandwhat isbeingsaid,butherewhen Idon’t listencarefully, Idon’tunderstand. I’smuchmoreexhausting[…]IwantedtocometoFlanders,butIdidn’trealizethatitwouldinvolvethismuchwork,andthat’sthemainproblemforme.

(Océane,November2014)HoangleftuniversityduringtheJuneexaminationsforanumberofreasons,butlanguageandthefeelingofisolationwerethemostimportantones.ForbothhimandEmma, the shockof beingunprepared for the level ofDutchused in classcausedastateofpanic.Hoangperceivedthesituationashopeless,foundoutthathecouldstudyinGermanythenextyear,andgaveupbeforetheendoftheJuneexaminations. In the second semester Emma learned tomanage her language-induced panic, started to study routinely but did not catch up. In June shedecidedto leaveuniversityandpursuehighereducationataFlemishuniversitycollege.

Anastasia’scaseclearlyshowedthatproblemsstemmingfrominstitutionalfactors can be aggravated by issues on the interpersonal level. She found thecombination of not being able to adjust to the didactic culture, and of notconnectingwithherclassmatestoodifficult.Aftershebecameill inJanuaryherclassmates sent her messages to complain about her not handing in an


168

assignmentontimewithoutinquiringaboutherhealth.Atthispointshedecidedtodropout:

Idon’tliketheuniversity.Theweirdwayofteachingandofdealingwithstudents. I don’t like that. […] Andmy personal situation was the finaldrop.[…]Ididn’twanttoquit.Ithinkit’shorrible,becausetheplanwastostayhereasastudent.ButnowI’mnotstudyinganymore,butIcan’tworkeitherbecauseoftheconditionsinmyvisa[…]AfterthefirstmonthitwasclearthatIwouldnotbehappyhere.Iwenttoclassandnobodyspoketome.WhenIaskedaquestion,peoplerepliedpolitely,butthatwasit.Andwhen I went to the student support service they didn’t offer any helpreally.

(Anastasia,February2015)Anastasiareferredtovisarequirementsasacomplicatingfactorinherdecisiontostop studying. Two students, Yazdan and Stella, involuntarily left universitybecauseof immigration issues.Yazdanneededa jobbecausehis allowancewasreduced. The combination of working and studying proved too much inNovember 2014, so he decided to save money and return to university thefollowingyear.Stella,whowasquiteenthusiasticaboutherprofessors fromthefirstinterviewonandwastheonlyparticipanttohavepassedeveryexam,repliedtoaninvitationfortheMarchinterviewbysayingthathervisahadbeenrevoked.At first she assumed that she had an agreement with the university to sit theexams in August and September (Stella, e-mail, March 25 2015), but later itappeared that theuniversity requiredproof of permanent residency,which shedidnothave(Stella,e-mail,May282015).BySeptember itwasclearthatStellawouldnotreturntoBelgium:

Unfortunately,IamforgettingDutchmoreandmoreeveryday.[…]Nearlyall Ihavedone in thepast threeyearshasgonetowaste.Theonlygoodnewsisthat[…]IamnowaqualifiedauditorinArmenia.

(Stella,e-mailextract,September172015)

DISCUSSIONUnavoidably, the results of longitudinal qualitative action research contain amyriadofuncontrollablevariables.Often,thedatastemmingfromsuchresearchcanappearratherdisordered,sincethecomplexityoflifeisnoteasilycapturedinanabstractmodel(Leung,Harris,&Rampton,2004).Nevertheless,inspiteofallthe important peculiarities and idiosyncrasies of every individual participantincludedinthisstudy,somecleartrendsemerge.Utilizingthebroadframework


169

proposed by the Douglas Fir Group (2016) in the analysis of the substantiveamount of data gathered for this study, has facilitated the identification andorganization of patterns that help to the explain the limited language gains asmeasuredbytheSTRTtest.Inthisdiscussionsection,theresearchresultswillbeinterpretedinthelightofpreviousfindings,andthethreelevelsoftheDouglasFirframework.

After eight months at university, the only significant performancedifferencebetweenthetestandtheretestwasadecreaseintheamountofwordsusedintheoralpresentationtask(p=.03,r=-.5).Theamountofwordsusedinthe writing task had increased, however, with a difference that approachedsignificanceandamediumeffectsize(p=.07,r=-.4).Thisoutcomeremindsoffindings by Knoch et al. (2015, 2014), who reported that the only significantdifference inwritingperformanceafteroneyear and three years at anEnglish-medium university was an increased written fluency, which was measured bynumber of words used. The current study adds further support to argumentschallenging the assumption that exposure to L2 input will yield productivelanguage gains (Ellis, 2003; Ortega, 2008). The data presented in this studyfurtherchallenge thebelief that internationalL2studentswillmakeproductivelanguage gains by virtue of attending class in which the L2 is themedium ofinstruction(Knochetal.,2015;Ranta&Meckelborg,2013;Storch,2009).

Educators inFlandersoftenuse themetaphorof the “languagebath”, tosupportthebeliefthatwhenL2studentsaresubmergedinacontextwherethetargetlanguageisthemainoronlylanguageused,theywillmakelanguagegains(Departement Onderwijs en Vorming, 2016). Previous authors have contestedthis metaphor, stating that L2 learners may drown in language baths (VanAvermaet & Slembrouck, 2014) – a statement which is supported by thelongitudinal findings of the current study. Additionally, further eroding thevalidityofthelanguagebathmetaphor,thisstudyshowsthatsimplysittinginalanguagebathisnoteffective.Instead,learnersneedtogetampleopportunitiestomeaningfullyinteractwiththeirL1peers(Elder&O’Loughlin,2003;Serranoetal.,2012).

The results of the longitudinal interviewdata show, however, thatmostparticipantsinthecurrentstudyexperiencedproblemsontheinterpersonalandon the institutional level, resulting in limited opportunities for meaningfulinteraction.

Ontheinterpersonallevel,itisclearfromtheresultsofthecurrentstudythat interactiondidnotoccurswiftly,smoothlyor frequently,even ifL1andL2students attended the same classes, or were involved in the same groupassignments (see Ranta & Meckelborg, 2013). Issues of power and legitimateperipheral participation obstructed L2 students’ access to the community ofFlemishL1students,whichwasperceivedasimpenetrable.Interactionisdialogicby definition however, and both parties share a responsibility for its success


170

(Kinginger,2008).ItislikelythatmanyL1studentswhowereperceivedasclosed,did not have the intention to exclude their international peers, but wereperceivedas suchby theL2Fparticipants,whoexperiencedapower imbalance.Indeed, situations thatmayappear rather innocent tomembersof themajoritygroup could disproportionally impact members of minority groups (Walton &Cohen, 2007). Elena, for example, related an instance in which the classroomlaughedand inferred that theyhaddone sobecause shewas from theUkraineandmusthavethereforebeenstupid.Quitelikely,herL1classmateswouldhaveperceivedthatsamesituationdifferently,butforElenathiswasthehardreality.

Quiteafewparticipantsfacedsimilarissues,resultinginarenegotiationoftheir academic identity (seework byDeCosta, 2011).Others, like Leila, EmmaandHoang,respondedtoaperceivedlanguage-relatedinferioritybywithdrawing(seeDuff,2002).Manyrespondentsatsomepointmentionedbeingimpactedby(perceived)disapprovingattitudesaboutthemwithintheL1community(Taylor,1992).Somereactedtothisbyfurtherdistancingthemselves,categorizingtheirL1peers as young, childish, or spoiled. Similar dynamics have been reported byNorton (2013) andPellegrinoAveni (2005),who observed that L2 learnersmayself-excludefromaninteractionbecauseofperceivedorrealpower imbalances.Thus, even though theL1-L2 interactionsweremarkedpowerasymmetries, theinterview data show that it would be wrong to characterize L2 learners arepassiveobjectswhowait for theirL1peers to let them into theircommunityofpractice. Quite a few respondents made attempts to create a social networkinvolvingL1students.

Thesecondlevel,theoneofinstitutionalpolicies,canbothfacilitateandobstructcontactbetweenL1usersandL2learners(Holmes,Marra,&Vine,2011;DouglasFirGroup,2016).ThisstudyshowedthatthecontextofFlemishuniversitiesdoesnotappeartobeespeciallyconducivetofacilitatingfrequentL1/L2interaction,orto promoting an empowered L2 identity. Especially during the first semester,participantsdidnotexperienceinstitutionalsupport,andtheyfeltoutofplaceinlargeclassroomswithminimalinteraction.

The participants included in this study felt that the university had adistinctpolicyforErasmusstudentsandforFlemishstudents,butnotforthem.The reason for this is clear: inpolicy terms theyarenotadistinctgroupatall.Thecurrentstudydidnotfindanyexamplesofsystematicsupportsystemsthatcater to the needs of international L2 students. Individual professors didsometimes provide accommodations for L2 students’ needs, but there was noclearsysteminplace.

Theinterviewdatasupportthehypothesis(DouglasFirGroup,2016)thatthe interpersonal and institutional levels are interconnected.The studentswhodroppedout voluntarily, did sobecauseof a combinationof reasons related tothemicroandmeso-level:Manyparticipantsdidnothaveasocialnetworktofall


171

backonorastudentnetworktohelpmakesenseoftherulesandexpectationsatuniversity,didnotconsidertheprofessorsapproachable,andlackedthelanguageto fully understand everything that was being said. Likewise, students whodroppedoutinvoluntarilydidnothaveaccesstoapowerfulsocialnetworkortoinstitutionalsupport,tohelpthemappealthedecisionsthatforcedthemtoleaveuniversity,orthecountry.Inthecourseofthisresearchnodirectevidenceofideologicalforces–themacrolevel in the Douglas Fir framework – was collected. It is possible, however, toextrapolateanumberofinsightsconcerningtheideologicalstructurespresentatFlemish universities from the institutional and interpersonal results. Thepredominant language ideology in Flanders has been described as territorialmonolingualism (Blommaert, 2011; Blommaert & Van Avermaet, 2008; VanSplunder,2015).VanSplunder(2015)writesthatthelanguagenorminFlandersisnot simply Dutch however, but native-like Dutch, be it in dialect or in thestandardized variety. He asserts that there is little tolerance towards languagelearnerswhodonotattainthatnorm.Possibly,thefearofridiculethatwithheldmany participants from speaking in class (see also Duff, 2002), partaking inmeetings,orinteractingwithL1peerscouldstemfromnotbeingabletoliveuptothisimplicitnorm.Additionally,nationalregulationsthatgovernlanguageuseatuniversity,limittheuseoflanguagesotherthanDutchinordertoreaffirmtheimportance of Dutch as a medium of instruction at university. Even thoughmaintainingDutchasanacademicallyviablelanguageisavaluablegoal,strictlyregulating the use of a language may foster a Dutch-only attitude amongprofessorsandstudents,whichmayreinforcealreadyexistingpowerasymmetriesbetween L1 and L2 students. Similar dynamics have been observed in Flemishprimary(Jordens,2016;Strobbe,2016)andsecondaryschools(Agirdag,2010).By the second semester, life at university had begun to improve, linguistically,socially and academically, adding support to the idea that international L2students need time to adjust (Chirkov, Vansteenkiste, Tao, & Lynch, 2007;Kinginger,2004;Ortega,2008).Everyinternationalstudentincludedinthisstudyneeded time to negotiate the differences encountered in the new setting(Kinginger, 2010), to discover that their identity had been altered (Kinginger,2004; Swain&Deters, 2007), and to gain acceptance into anewcommunityofpracticethattheyperceivedasclosedorunwillingtoacceptthem.TheseresultshaveanumberofpolicyimplicationsforFlemishuniversities.

First,thisstudyshowsthatthesuddentransitiontouniversity,whichmaybe intimidating for any eighteen-year old, canbe a daunting experience for aninternational L2 student. Consequently, in line with previous research (e.g.,Kinginger,2004),ittookmostparticipantsafewmonthstogetusedtothenewsituation.Consideringthedifficultiestheyfaced,itisquitestrikingthateightout


172

of sixteen participants (i.e., excluding Jessica and Chloé who left the study inNovember, and Stella and Yazdan who gave up because of migration issues)passedatleasthalfofthecoursestheyhadregisteredfor.Proportionally,thatiscomparabletotheaveragestudysuccess inFlanders(Glorieuxetal.,2015). It isnot inconceivablethatparticipants inthisstudywouldhavefaredbetter iftheyhadexperiencedasmoothertransitiontouniversity.Currently,GhentUniversityispilotingaprograminwhichinternationalstudentsmeetregularlyduringtheirfirst semester at university. During this time they also receive needs-basedlanguage support. Projects like this are vital for the increasing internationalstudents’chancesofsuccess. Additionally,universitiesneedtodevelopaclearandtransparentsupportsystem. It should be clear for international students from day one whichaccommodations are available to them. Being able to take an examination inEnglishorbeinggrantedmoretimetofinishanexaminationshouldnotdependontheassertivenessofastudentorthepersonalityofaprofessor. Thirdly, this study reaffirmed that international students may not beimmediately ready for the linguistic demands of university (Field, 2011). It hasalso confirmed that L2 students will not necessarily make language gains byvirtueofattendingDutch-mediumclasses.WithByrnes,Maxim&Norris(2010),Iwould argue that it is the responsibility of the university to have clear L2attainment targets in addition to entry requirements. If a university allowsinternational L2 students when they have insufficient language to understandlectures, that university has a responsibility to provide opportunities forinternationalstudentstodeveloptheirL2.OfferingthelanguageofinstructionasacurricularcourseforL2studentsisonewayofdoingso.

Lastly, international students who enter university may have gainedrelevant insights or expertise that could be put to gooduse in their ownor inother programs. Leila’s personal experiences inHaitiwere certainly relevant toherpoliticalsciencesprogram.Elena’sbackgroundasanairplaneengineercouldhavebeenturnedintoausefulcontributiontoclass,andOksana–L1speakerofRussian – would certainly have been able to contribute to class in herRussian/English program of translation studies. Valorizing the potential ofinternational students could positively impact the quality of the lessons, andwoulddefinitelymakeinternationalstudentsfeelmoreathomeinclass.


173

EPILOGUE

Twoyearsafterconductingthefirstinterviewstheparticipantsreceivedthefirstversion of this chapter (eight respondents replied: Leila, Alexandra, Gabriela,Alireza,Marie, Guadalupe,Oksana, and Elena). Their responses indicated thattheyhadmovedon,andthatlifeatuniversityhadbecomesomewhateasierafterthatfirstyear.Oksanawasabouttograduate,andGabriela,andGuadalupewereprogressingacademically:

TodayIamstartinginthethirdyearandafterreadingthepaperIamextramotivated.Foryourinformation:Ipassedstatistics1and2andI’mtakingonafullstudyprogram,likeanormalstudent.J

(Guadalupe,textmessage,September2017)Otherswere excited about newprofessional opportunities. Leila had become alegalpolicyadvisoratagovernmentagency,andAlexandra–havingfoundajobasanengineer–haddecidedtostayinBelgiumforatleastafewmoreyears:

Ireadyourpaperanditmademecryalittle.Ihadalreadyforgottenjusthowhard ithadbeen forus.Now, for the first time Ihavebeenable toread about the experiences of other international students, and I wasrelievedtoseethatIwasn’tcrazyoralone.

(Alexandra,mail,October2017)

The future belongs not so much to thepurethinkerswhoarecontent–atbest–with optimistic or pessimistic slogans; itis a province, rather, for reflectivepractitioners who are ready to act ontheirideals.Warmheartsalliedwithcoolheads seek a middle way between theextremesofabstracttheoryandpersonalimpulse.

Toulmin,2001,p.214

Part4:Policy,conclusion&implications

176

PART4POLICY,CONCLUSION&IMPLICATIONS

Thefirsttwopartsofthisdissertationwereconcernedwithempiricallyinvestigating assumptions that support the university entrance policyfor internationalL2 students.Untilnow, the focuswasonproficiencylevels,onrepresentativeness,andontestequivalence.Inthethirdpart,which consists of one chapter, the attention shifts to what happensaftertheentrancetest:doL2studentsmakegainsintermsofacademicDutchlanguagedevelopment,andifyesorno,why?

The final part of this dissertation includes three sections.Chapter 7 provides adiscussion of the processes and mechanisms that have led to the Flemishuniversity admission policy. Chapter 8 summarizes the research findings(Chapters1–6),andChapter9combinestheoutcomesoftheempiricalresearchwiththeperspectiveofpolicymakersinordertoformulaterealisticimplicationsandrecommendations.

Chapter7:Thepolicy-makingprocess

178

CHAPTER7THEPOLICY-MAKINGPROCESS

The purpose of this chapter is to bridge the gap that sometimes existsbetweenresearchandpractice.It linkstheoriginalresearchassumptionstothepolicymakers’perceptions,andshowstowhatextentempiricaldatamayormaynotimpactpolicy.

Intheory,policy-makingisastraightforward,linearprocess:aperceivedproblembecomes part of the policy agenda, after which policymeasures are developedandimplemented.Aftersometime,anevaluationofthesemeasuresleadstoanadjustment, alteration or continuation of the existing policy (Howlett & Giest,2013; Wilson, 2006). Laswell’s (1956) policy cycle, briefly summarized here,remainsinfluentialinpolicyanalysisstudies,primarilybecauseitconceptualizesa complicated process in a straightforward way. Like most models, however,Laswell’s is an idealization (Jann&Wegrich, 2007). In reality, policy-making iscyclical nor linear, logical nor rational. It is influenced by a multitude ofvariables,suchasbudgetrestrictions,partisantensions,orlegalconstraints(VandenBosch&Cantillon,2006).Since policy is not easily captured in standard models, it is important tounderstand why and how policy is made in a specific context before relevantrecommendationsorimplicationscanbeformulated(O’Toole,2000;Ross,2008).Consequently,thepurposeofthissectionistotracethemechanismsthatimpactthe Flemish university entrance policy. This chapter relies on interviews withpolicymakers,conductedafter theempiricaldatapresented inthisdissertationhad been analyzed. It addresses a gap in the language testing literature byshowing how and why university entrance language requirements aredetermined.

EXAMININGUNIVERSITYADMISSIONPOLICIESIn the context of higher education, few studies have examined the real-worldconditionsunderwhichadmissionpoliciesareshaped.Whatismore,studiesinthisfieldhavetraditionallyadoptedasomewhatpositivistictechnocraticmodel,which focusesonquantifyingtheextent towhichapolicyreaches itsobjective,withoutnecessarilyconsideringreal-worldlimitations(Fischer,2007;Howlett&Giest,2013;Vedung,2013).Typically,thesestudiesexaminetheeffectivenessandside effects of a policy. In an important publication in this tradition, Wainer


179

(2011) focused on the use of SAT results in the university admission policy ofNorthAmericanuniversities. The quantitative analyses uncovered fundamentalflaws and inconsistencies in the assumptions that supportuniversity admissionpolicies. Wainer did not focus on language tests, but his conclusions were awarningcallaboutthedangersofconductingpolicyonthebasisofmisguidedoruntested assumptions. Similar publications in the field of educational policyoftenfindevidencethatpartlyorcompletelydisprovestheverypremiseonwhichapolicyrelies(e.g.,seeBall,2015;Borg,2006foradiscussion).

Thereareonlyahandfulofstudiesthatfocusspecificallyonthelanguagerequirements in university admission policies. The assumptions behind auniversityentrancepolicyfor internationalL2studentshavenotoftenbeenthetopic of extensive language testing research (McNamara & Ryan, 2011). Studiesthatdotouchuponthis fieldmostly focusonscoreuse.O’Loughlin(2011,2013)showed thatuniversity admissionofficers arenot always awareof themeaningand scope of a test score, and primarily desire clear-cut, straightforwardinformation.Green (Forthcoming)demonstrated that the informationprovidedby language testdevelopers about a test’sCEFR levelwasnot as clear-cut as itseems.HeconcludedthatequivalencebetweenEnglishuniversityadmissionL2tests that share the sameCEFR level cannot be assumed, since the proceduresused to link to the CEFR may differ substantially. Chapters 3 and 4 of thisdissertationofferfurtherbackingtoGreen’spoint.

Importantly, research that examines university admission languagerequirementsor tests typically adopts an empirically-driven researchparadigm,rather than the pragmatic perspective of policymakers (Howlett&Giest, 2013;Jann&Fischer,2007;Wollmann,2007).Aclearexampleofthiscanbefoundinthe language assessment literacy literature. Fueled by evidence of intended orunintendedmisuse of language tests and language test scores (Fulcher, 2012a;O’Loughlin, 2011, 2013; Spolsky, 2008; Taylor, 2009, 2013), this discipline isconcerned with the assessment-related competences required by variousstakeholders in the language testing process. Importantly, however, authors inthisfieldtendtopresumethatthelanguagetesterisattheheartoftheprocess(Malone, 2013;Taylor, 2013).Consequently, theapproach taken in the languageassessmentliteracyliteratureisexplicitlytop-down:“thosewhoneedtodevelopsuchliteracyarelikelytohavelesstimeandenergytospendseekingoutwhatisrelevantanduseful totheirrequirements; theonusofresponsibility formakingkey informationmore accessiblemust surely lie with those who already knowwhereitislocated”(Taylor,2013,p.408).

Recentpolicy evaluation research strongly suggests that academicpolicyevaluation initiatives are not likely to bring about real-world change (Wilson,2006).Tohaveimpact,researchersshouldbeawareoftheexactcontextinwhichapolicyisset,beattentivetothereal-worldconstraints(Ross,2008),andacceptthatpolicy-makingismessybydefault(Ball,2015).Iftheyarenot,policyadvice


180

mightbetoodisconnectedfromrealitytohaveanyimpactatall(Bovens,’tHart,&Kuipers,2006).

RESEARCHQUESTIONSBefore formulating policy recommendations (Chapter 9), this sectionmaps theconstraints and conditions that impact the university admission policy inFlanders. By investigating two explorative research questions, it examines whythe Flemish language-related university admission requirements are what theyare:RQ1 How is the university admission policy for international L2 students at

Flemishuniversitiesmade?RQ2 Whatarethecommonlyheldassumptionsbehindthisadmissionpolicy?

PARTICIPANTS&METHODOLOGYParticipantsToselectparticipants,apurposefulsamplingstrategywasused.Attheendofthedataanalysis,inNovember2016,thevice-deansandeducationaldirectors1ofthefive Flemish universities were asked to identify the members of staff who aredirectlyresponsiblefortheadmissioncriteriaregardinginternationalstudentsattheir university. All participants were senior members of staff, directlyresponsiblefortheadmissionpolicyattheirinstitution.Table8.1.Policymakerrespondentcodes UniversityofLeuven UL1 UL2 GhentUniversity UG1 UG2 UniversityofAntwerp UA1 UA2 UniversityofBrussels UB1 UB2 UniversityofHasselt★ UH1 UH2 UH3 UH4 FlemishGovernment FG1 FG2 FG3 Note.(★)Fourmoremembersofstaffwerepresent,butdidnotpartakeintheinterview.

1UnliketheotherFlemishuniversities,GhentUniversitydoesnothaveavice-deanforeducation.Instead,theofficialtitleisDirectorofEducationalPolicy.


181

Importantly,sincethepoliciesattheFlemishuniversitiesarepartlydeterminedbyaFlemishdecree(VlaamseRegering,2013),threeseniorpolicymakersatthegovernment level were also recruited. These respondents were specificallyresponsibleforguidelinesconcerninguniversityadmissionissuedbytheFlemishgovernment.Allpolicymakersatboth levelsagreed toparticipate,andas suchthe research population (N = 15) represents the full real-world population. Inorder toguaranteeanonymity, all respondentswillbe referred tousinga code,consistingoftheacronymfortheiraffiliation,andanumber(seeTable8.1).Datacollection&analysisAll interviews except for one, whichwas limited to 45minutes because of therespondents’ schedule, took more than one hour (Min: 49 minutes, Max: 91minutes, Median: 74.5 minutes). Every interview was structured, and builtaround three central components. First, the respondentswere asked to explainhow the university entrance policy regarding international students is shaped.Next, every university entrance requirement in place at a given institutionwasdiscussed.Inthisphaseoftheinterviewtheassumptionsthatdrovethisresearchwerecheckedwithpolicymakers.Assumption2(testrepresentativeness)wasnotincludedintheinterviewscenario,becausethiswasnotconsideredaclaimmadebyscoreusers,butbytestdevelopers.Respondentswerepromptedtoexplainthenature,purpose,andperceivedeffectivenessofeachrequirement(seeTable1.2).In the last component of the interview, themain research conclusions of thisresearch were discussed with the respondents. Every interview was audiorecorded and transcribed. The transcriptions were analyzed using an a prioricodingtreeconsistingofthreemainbranches(seeTable8.2).Table8.2.Datacodingcategories Branch Topic Code1 Policy-makingprocess impactingvariables,empiricalfoundation2 Policyenactment goal,effectiveness,post-admittancepolicy3 Policyassumptions B2 level requirement, STRT-ITNA equivalence,

Flemish students’ language proficiency, post-entrylanguagegains,60creditsandlanguageproficiency

The branches of the coding tree correspond to the interview scenario, andprimarily concern factual data. The first two branches focus on how policy ismadeandenacted(relyingonpolicyevaluationliterature,e.g.,Jann&Wegrich,2007)andthethirdbranchinvestigatestowhatextentAssumptions1,3,4,and5areupheldbythepolicymakers.


182

RESULTSHowpolicyismadeatgovernmentlevelThe admission requirements for international L2 students at all Flemishuniversities share threeguidelines,whichoriginate fromaFlemishgovernmentdecree known as “the codex” (Vlaamse Regering, 2013). This document brieflystipulates that universities may use the following documents as sufficientevidencefortheadmissionofinternationalL2students:(1)alanguagetestresult,(2) proof of having successfully completed one year in a Dutch-mediumsecondaryschool,and(3)proofofhavingachieved60creditsinaDutch-mediumhighereducationprogram.Thecodexdoesnotspecifyarequiredlanguagelevel,and neither does it identify which tests are accepted. A more recent decree,which concerns international L2 professors is noticeablymore specific, but thelack of specification in the codex is not necessarily the consequence of adeliberatestrategy.FG2 I think it’s primarily a difference in timing. [The codex] is very old.

Actually,it’salwaysbeenlikethis.Andnow,withthenewregulationsforprofessorstheywentmuchfurther.

Becausethecodexhasbeeninuseforquitesometime,therespondentsdidnotknowwhatitisbasedon,orwherethethreerequirementscamefrom:“Wehavebeenworkinghere forapretty long time,but this rulehasbeenaround longer[…] To us the rules always seemed logical. Actually, they have never beenquestionedinallthoseyears”(FG1).Policymeasuresarenotroutinelyevaluated,butreconsideredwhenuniversitiesorpoliticalstakeholdersraiseconcerns,andtheadmissionrequirementshavethroughtheyears“simplybeenreused”(FG2).When policy texts are revised, the impact of empirical research is minimal,comparedtotheimpactofstakeholders.FG3 Research is often used a little selectively, like when people want

somethingtobecomepolicy.FG1 Ifwewantedtochangepolicywewouldn’tfirstdoororderastudy.FG2 Whatcouldhappenisthatpolicyadviceisbasedonascientificstudy[…]

But there are always political negotiations, and all stakeholders areinvolved.

HowpolicyismadeatuniversitylevelAll respondents defined the goal of the university admission policy in a verysimilarway:Toselectstudentswhohaveasufficientleveloflanguageproficiency


183

tobeabletoattendaDutch-mediumuniversityprogram.Whenaskedaboutthemeasures in place to pursue that policy goal, all respondents referred to thecodex as the key source for the requirements. Since universities are under nolegal obligation to follow the codex, however,most institutionshave identifiedexemptionsforstudentsatthemasterorpostgraduatelevel(seeTable8.3).OnlytheUniversityofBrusselshasexemptionsforbachelorstudents.

Respondentsattwouniversitiesmentionedrelyingontrendsinpassandfail rateswhenmakingpolicydecisions, butnotwhen it came to the languagerequirements. No respondent referred to empirical research as a reason foradjusting the language requirements for specific programs, but not because offundamentalobjections.Table8.3.ExemptionsfromadmissionrequirementsatFlemishuniversities UniversityofLeuven • Faculty or program can drop language requirements for

masterstudentsGhentUniversity • Faculty or program can drop language requirements for

masterstudentsUniversityofAntwerp • Faculty or program can drop language requirements for

postgraduatestudents• Facultyorprogramcanexemptstudentswithpartialstudy

load(so-calledcreditcontracts)UniversityofBrussels • All students from a Belgian French-medium secondary

school• Programdirectors candecide to allow individual student

afterareviewoftheirfileUniversityofHasselt /UL2 Most likelywewillneverbeable to implementverybig changesbutwe

are able to make recommendations supported by research. It will stillremain tobe seen if theproposal ispassed though […] in theend it’s agameofpolitics.

Toa largeextent,policydecisionsatuniversity levelappear influencedbybothinternalstakeholdersandexternalfactors.TheexemptionslistedinTable8.3arethe result of internal pressure from specific faculties or programs. Universityadmissionofficerscannotdesignapolicywithouttakingintoaccounttheviewsandaimsofpowerfulinternalstakeholders:UA1 We have had a lot of debate about [an exemption for certain students]

withprofessorswhowantedtoattractspecificprofiles.


184

UA2 That’s right. Is everybody happy with these exemptions? I have to behonest:no.Butithasbeendecidedthattheywanttocontinuetoallowforthisexemption.

Quite a few respondents referred to admission requirements that werepurposefullyvague.“Ithinktherearesupposedtobesmallflawsinthesystem”,UL2 stated, before referring to instances of professors admitting students whohad not passed an accredited test: “I try to stand my ground then […] andsometimesIcan,sometimesIcan’t”.

External events that are beyond the control of a university may alsoprompt policy changes. For example, after 2012 the University of Brussels haddropped all language requirements for international L2 students. In 2015 thedecisiontoreinstatethemwastakenbecausetheuniversitywassuddenlyfacedwithaninfluxofGermanstudents.Fromoneyeartothenext,morethanhalfofthe freshmen in psychology and biomedical sciences were German students:“These students were coming to our university because of a reform in theirsecondary education system. […] These people were looking for solutions, andthey came here. And then we needed to fix that situation” (UB1). Anotherexternal influence is thepolicyadoptedbyotheruniversities.TheUniversityofHasselt lowered the required entrance level to match the policy of a largeruniversity nearby: “it was because Leuven had B2 as well back then. And alsobecause we thought C1 was rather high, and then we checked the CEFR andthoughtB2wouldbesufficient”(UH2).CommonlyheldassumptionsAt every university the default entrance level is B2, but the reasons given forusing that level do not necessarily refer to empirical research. In spite of theabsence of empirical backing, all respondents considered B2 an adequatethreshold level to differentiate between studentswhose languageproficiency islikelytobeanobstacletoacademicsuccess,andstudentswhoseproficiencywillnot prove to be a hindrance. At the same time, respondents pointed out thattherearemanyothervariablesinfluencingastudent’sacademicsuccess:“It’snotjustlanguage.It’saverycomplicatedprocess.Ithinkeverybodythinks‘wehavetodosomething,solet’sdothis’.”(UH3).

At all universities, both STRT and ITNA are legally equivalent.Respondents at three universities also assumed level equivalence between thetwotests,butwithoutaclearrationale:“weconsiderthemequivalent.Idon’tknowwhy. Probably because both are at the B2 level according to something orsomebody”(UL2).Respondentsatotherinstitutionsnotedthatforthem,thelegalperspectivewastheonlyonethatmattered.


185

UG1 Wedon’t actuallywonder about equivalence […]Whatmatters for us isthatwehaveall thedocumentsweneed for registration […]WeassumethatbothtestsareminimallyB2.

UG2 Legallytheyareequivalent,andthatisourapproach.Most respondents believed that students graduating from Dutch-mediumsecondary education are at the B2 level. At the University of Brussels theparticipantswerenotsosure,becauseofthepredominantlyFrenchpopulationinBrussels. Few respondents felt sure that the requirementofhaving successfullyattendedoneyearataDutch-mediumsecondaryschoolwouldbereliableproofofB2proficiency.Atthesametime,thisrequirementhasnotbeenalteredinanyuniversityadmissionpolicy.UA1 It’salwaysbeeninthere[…]Youcoulddoubtthisrequirement,probably

[…] I have alwayswondered ifwe couldmake itmore strict but I knowtherewouldbealotofinternalresistanceifwewoulddothat.

UA2 No!Weshouldaccept theguidelines in thegovernmentdecree,andnotmake them stricter. Let’s give [students] a chance if we can. Theirlanguageproficiencymayimproveatuniversity.

AsUA2’s last comment indicates, respondents generally expected internationalL2 students to make language gains by virtue of attending Dutch-mediumclasses. Most universities offer academic language classes for students whoexperience language-related problems, but since there is no general post-admittancepolicy for internationalL2students, theseclassesaremostlygearedtowardsthegeneral(i.e.,L1)studentpopulation.

Whenaskedtoassesstheeffectivenessofthepolicymeasurestoreachthepolicygoal,allrespondentswerehesitant,sincetheygenerallylackthemeanstomeasureeffectivenessinapreciseway.Atmostinstitutions,therewerenoclearstatisticsof internationalL2studentswhostudy inDutch.No institutionhadapost-admissionpolicyforthesestudents,soitislargelyimpossibletotrackthem.If universities focus their attention on international students, they tend toconcentrate on those who attend English-medium programs. Orientation daysforinternationalstudents,forexample,donotnormallyfocusonthosestudentswhowillstudyinDutch:“I think thegroupof students is too small andhard toreachandthat’swhyitdoesn’thappen(UL2)”.

Finally, during the third part of the interview, the research resultswerediscussed.Thiscomponentoftheinterviewgeneratedalotofinterest,andafterevery interview participants stated that they would take the data intoconsideration.Participantsateveryuniversitysentfollow-upe-mails,expressinginterest in the research, asking to be kept informed.Nevertheless, it was clearthat this was no guarantee for the findings to find their way to policy: “it all


186

depends onwhat is politically achievable at a givenmoment. […]What happensnext is a whole process of negotiations and talks. And in the end, the originalproposalmaylookentirelydifferently”(FG2).

DISCUSSION

The interview data consistently show that the Flemish university admissionpolicy,likeanyreal-worldpolicy,doesnotrigidlyfollowalinearorcyclicallogic,and it does not rely on routine evaluation (Jann & Wegrich, 2007). Instead,policies are adjusted when problems need to be solved, or when importantstakeholderswantchange.

Thisstudyconfirmedthattheuniversityadmissionpolicyistheresultofaseries of pragmatic rather than empirical or logical decisions (Ball, 2015).Universitiesfollowthecodex,exceptwhentheydonot.Theymaysometimesuseempirical data, but at other times may not. Tellingly, no respondent gaveempiricallyfoundedargumentstoarguewhyspecificlanguagerequirementshadbeen adjusted. In reality, these requirements were used to flexibly controlstudent access to certain programs (see also Chapter 1). At the University ofBrussels, thedecision to firstdismissand later reinstate language requirementswasdrivenentirelybypracticalconsiderationsregardingthestudentpopulation.Similarly, the University of Hasselt lowered the requirement from C1 so theywould not lose students to Leuven. At the University of Antwerp, certainprogramshavenolanguagerequirements,simplybecauseprofessorsdonotwanttolosecertainstudentsbecauseofthem.

If policy is essentially about the use of power, asWilson (2006) argues,then policy making is about making compromises that take into account thediverging interests of different parties. The same dynamic is demonstrablypresentatFlemishuniversities,whereprogramdirectorsareimportantdriversoftheFlemishuniversityadmissionpolicy.Thismayexplainwhysomeadmissionrequirements for international L2 students have remained unchanged andunchallengedforyears:InternationalL2studentsstudyinginDutcharetoosmallagrouptobepowerful,andtoodispersedtobenoticedbyastakeholder.Thecentralassumptionsinthisdissertationweremostlyconfirmed.Amongtherespondents therewasaconsensuswas thatB2 isa satisfactoryminimumlevelfor university admission. Secondly, while policy makers at two universitiesconsidered STRT and ITNA linguistically equivalent, there was unanimousagreement among all respondents that both tests are legally equivalent. Theissuesoflevelequivalenceandconstructequivalencearenotnecessarilyequallyrelevanttopolicymakers,sincetheirframeofreferenceappearstobeprimarilyoriented towards legality. Thirdly, there was general consensus that students


187

with a Flemish high school degree can confidently be assumed to have B2language proficiency. Lastly, even though this is not a language requirement,quite a few respondents remarked that they would expect international L2studentstomakelanguagegainsbyvirtueofstudyingataFlemishuniversity.

CONCLUSION:ADIFFERENTPARADIGM

No respondent believed that the admission system at their university waswatertight. Nevertheless, it was felt that the admission criteria served theirpurpose, because they were an acceptable compromise. The interview datalargelyconfirmthatpolicycomesdowntofixingproblems(Ball,2015),andinthispatchworkofpragmaticcompromisesempiricalresearchisoflittleimportance.

One and the same problem looks different from different angles, andpolicymakersandresearchersmaycomeupwithverydifferentsolutionstothesame issue (Goodin, Rein, & Moran, 2006). Echoing this paradigm schism(Howlett&Giest,2013;Jann&Fischer,2007;Wollmann,2007),thisstudyshowedFlemishuniversityadmissionpolicymakersarepragmatists.Researchers,ontheotherhandmaynotalwaystake intoaccountreal-worldconstraintsorpoliticalstrategies. Language assessment literacy authors, for example,may adhere to aparadigm that is quite distant from the one used by policymakers. Outliningassessmentcompetencyprofilesthatlistthespecifictestingexpertisethatwouldbe required of university admission officers (Taylor, 2013), does not appear tomatch the day-to-day reality of policy makers. Similarly, schooling admissionofficers inmatters of validity, in order to ensure that they canmake informedindividual decisions (O’Loughlin, 2013) may adhere to a somewhat idealisticparadigmthatisquitedistantfromthepragmaticoneusedbypolicymakers.

That does not mean that policy makers should not be informed aboutempiricalresults.Theinformationshould,however,meettheirframeofreferenceand theirday-to-day reality, rather than the researchers’ ideals. Proposals fromresearchers that are premised on academically-oriented paradigms may toodistant from reality to have an impact. Nevertheless, since policy makers arelimited in what they can do by an invisible network of interests and powerpolitics,empiricalresultsmaynothavetheimpactthatresearchersmaywishittohave,evenifresultsarecommunicatedinanappropriateway.Itissafetosaythat the responsibility for a university admission policy may reside with thepolicymaker,butthepowertochangeitdoesnot.

Chapter8:Summary&discussionoftheresearchfindings

190

CHAPTER8SUMMARY&DISCUSSIONOFTHERESEARCHFINDINGS

In Flanders, Belgium, international L2 students are typically required toprove B2 language proficiency in Dutch. Different universities acceptdifferentkindsofproofofB2ability,butallinstitutionsacceptalanguagetest certificate by STRTor ITNA.Additionally, international L2 studentswith any Flemish secondary school degree are allowed to enroll, andstudents who have already obtained sixty credits at a Dutch-mediumFlemishuniversityoruniversitycollegecanregisteratanotheruniversitywithouthavingtoproveB2abilityagain.Thisresearchprojectinvestigatedthe Flemish university entrance policy by addressing five differentresearchgoals.Thefirstfourgoalswerebasedonfourassumptionsdrawnfrom the entrance requirements that all Flemish universities share. Thefifth considered language gainsmade by international L2 students afteradmission.

EXAMINETHEEMPIRICALSUPPORTFORTHEB2LEVELASANENTRANCEREQUIREMENT

SummaryofthefindingsThefirsttwochaptersaddressedthematterofusingtheB2levelastheuniversityentrance threshold level. The first chapter reported on structured interviewsconductedwith 30 informed respondents from28European contexts thathave(quasi)autonomyovereducationalmatters.The findingsshowthat throughoutEurope,B2isthemostcommonlyusedleveltodetermineuniversityentrance.Intwentyofthe22EuropeancontextsthatuseCEFR-relatedlanguagerequirementsto determine university entrance, B2 plays an important role in the universityentrancerequirementsforinternationalL2students.Intenofthosecontexts,itistheonlyrequirement.Intenother,B2isoneoftherequirements,butforsomeprogramsA2 (n = 1), B1 (n = 1), orC1 (n = 8)might be theminimumentrancelevel.At the same time,only three respondentsoutof 30wereassured thatB2users would be able to function linguistically at the start of university.Additionally,therequiredentrancelevelwasbasedonempiricalresearchinonlyoneoutof23contexts thatuse language tests todetermineuniversityentranceforinternationalL2students.Onthewhole,theresultsfromthisstudystronglysuggest that the B2 level has become the default university entrance level inEuropewithoutmuchempiricalbacking.


191

Thesecondchapter investigatedwhethertheB2levelcorrespondstotheminimum language requirementsatFlemishuniversities.The results showthatthe university staff (N = 24) considered the B2 level vastly insufficient forlisteningandreading,butacceptable forwriting.Similarly, the internationalL2students(N=31)consultedforthisstudyallstruggledwiththereal-lifelisteningdemands of university. All respondents reported problems with understandingtheirfirstlectures,andafewhadunderstoodnothingatall–mainlybecausetheywere not prepared for the variation in accents and pronunciation stylesencounteredinreallife.Actually,theC1descriptorsintheCEFRappeartomatchthereal-lifereceptiverequirementsmorethantheB2descriptorsdo.Forreading,the C1 level states “lengthy, complex texts likely to be encountered in social,professionaloracademic life” (CouncilofEurope,2001,p.70).Similarly, theC1descriptor for listening mentions unfamiliar accents, and following “extendedspeech even when it is not clearly structured andwhen relationships are onlyimplied”(CouncilofEurope,2001,p.66).

The internationalL2students reported fewerproblemswithreadingandwriting, primarily because these skills typically allow language learners to dealwithinputattheirownpace.Typically,readinginDutchwasestimatedtotaketwiceaslongascomparedtoreadingintheL1.DiscussionThe data presented in this dissertation have reaffirmed that the CEFR hasfundamentally altered university entrance language testing in Europe (Little,2007). The wide uptake of the CEFR could be considered a good thing, andproponents have pointed out the benefits of using a common language todescribelanguageproficiencylevels(North,2014a,2014b,2016).Withoutwishingto deny the positive impact of the CEFR on teaching and curriculumdevelopment (e.g., the focus on a can-do approach), it is important to remainaware of potential shortcomings in the way the CEFR is used in high-stakestesting.Throughoutthisdissertation,twoimportantCEFR-relatedrisksrecurred:normativeuseofthresholdlevels,andreificationoftheB2profile.Thefirstrisk–normativeuseoflevels–impliesthattheB2levelisrequiredforuniversityadmission,simplybecauseitisB2.Itbecomesthenorm,notbecauseempiricaldatashowittobeadequate,butbecauseitalreadyisanorminothercontexts.

The CEFR itself has been amply criticized for the gaps in its empiricalfoundation(Alderson,2007;Fulcher,2012b),fornotincorporatinginsightsfromempirical SLA research (Little, 2007), and for not relying sufficiently on actuallearnerdata(Hulstijn,2007).ThiscriticismhasbeenpartyacknowledgedbytheCEFR’s authors, who estimated that some ten percent of the illustrative


192

descriptors(i.e.,23C-leveldescriptors,theentireorthographiccontrolscale,tensociolinguistic appropriateness descriptors, and two phonological controldescriptors) had not been empirically validated (North, 2014a). Currently, aninitiativeisunderwaytoaddempiricalsupporttocertaindescriptors,whilealsodevelopingnewscales(e.g.,mediation).TheseflawsintheempiricalfoundationoftheCEFRitselfhavebeenthesubjectofmuchscrutiny,beginningrathersoonafteritspublication(e.g.,Fulcher,2004).Itwouldseemthataframeworkthatiscontestedempiricallywouldbeused forhighstakespurposesonlyaftercarefulanalysis.Yet, intheoverwhelmingmajorityofEuropeancontextssurveyed, thisdoes not appear to happen. The B2 level appears to have achieved a specialappeal in the context of university admission, without a clear scientific orempiricalrationaletobackitsomnipresence.

NormativeuseoftheB2levelcanbeobservedinpolicies,butalsointests.ClearcasesinpointaretheFlemishSTRTandITNAtests,whichweredevelopedwiththeB2levelinmind.Tosomeextentthetypicalfeaturesofthislevelseemto have driven the operationalization of these tests. In its validity argument,ITNAjustifiesmostof its itemtypesbyreferringtowhataB2 learnerofDutchcan do (Interuniversitair Testing Consortium, 2015). In STRT the B2 levelorientation was requested by the funding organization (Nederlandse Taalunie,2013).EspeciallyintheselectionofinputmaterialitbecomesclearthatSTRTmayhave reliedmore on B2 descriptors than on an analysis of target language usedemands.EventhoughtheSTRTlisteningandreadingpromptsmaynotalwaysreflect the challenges of real-life receptive demands, they match the B2descriptorsquitewellindeed.ApartfromnormativeCEFRusethereappearstobenoclearmotivationtocreateauniversityentrancetestthattestseveryskillattheB2level.Actually,oneofthegoalsoftheCEFRwastoofferanalternativetotheideathatonepersonhadonekindofuniformproficiencylevel(Krumm,2007).Ina2011paper,Hulstijnarguedthat demanding an even CEFR profile in an academic context would mostprobablynot correspondwith real-life requirements. The researchdiscussed inChapter2offersempiricalsupporttoHulstijn’sassertion:inFlanders,B2maybeanappropriatethresholdlevelforwritingandpossiblyspeaking,butforlisteningandreadingahigherlevel–oronethatincorporatesmorefeaturesofreal-worldlanguageuse–wouldbeclosertothereal-lifedemands. The observations made in the paragraphs above tie in with the secondrisk: reificationof theCEFRlevels.This threatwascoinedbyFulcher ina2004paper, and remains true thirteen years later. A logical fallacy first termed “thefallacy of misplaced concreteness” (Whitehead, 1925), reification occurs whensomethingabstractistreatedasifitweresomethingconcrete.WhentheB2levelistreatedassomethingthatisanexactlymeasurableentitysuchastemperature


193

orweight (e.g.,De Jong, 2013, 2014), it is being reified, and thismayover timeendangertheCEFR’scredibilityandusefulness(Hulstijn,2015).

InthisresearchnodataregardingthelinkingprocedureofSTRTorITNAwas consulted. Both tests used analogous familiarization-specification-standardizationprocedures(Figueras,North,Takala,Verhelst,&VanAvermaet,2005)wellenoughtopassanALTEaudit.Nevertheless,beinglinkedtothesameCEFRlevelfarfromguaranteesequivalence,asisclearfromChapter3and4.Onthesurface,usingthesameCEFRlevelsacrosstestsmayseemtoincreasescorecomparabilityandscoretransparency,butinpracticethedifficultylevelsoftwoteststhatsharethesameCEFRlevelmaydivergesubstantially.Thefactthattwotests that are linked to the same CEFR levelmay lack equivalence should notsurpriseinformedresearchers:B2coversquitearangeofperformances,andtheCEFR is hardly precise enough to act as a ready-made measurement scale(Galaczi,ffrench,Hubbard,&Green,2011;Harsch&Martin,2012;Weir,2005b).

Thisshouldnotnecessarilybeaproblem.TheproblemonlyariseswhenCEFRlevelsaresomehowbelievedtobe“true”,orwhenstakeholdersassumethattestsareequallydifficultybecausetheyhavethesameCEFRlevel.Usingtestsasif theywere equivalent, or implying that they are, only because they share thesameCEFR level, does not qualify as responsible score use, and policymakersshouldbewarnedagainstit.Measuringtruescoreequivalenceentailsconductingtest-equivalence studies by means of equipercentile ranking (ETS, 2010),regression analysis (Zheng &De Jong, 2011), or Rasch (Fulcher, 1997). Anotherrecommended approach, which does not necessarily reduce the need forequivalence studies, is to develop test specifications based on the actual needsandrequirementsofthetargetcontext,andtolinktheseteststotheCEFRlater.

Overall, the evidence suggests that the empirical basis for using the B2levelinthefirstplace,isratherthin.ItalsoappearsthatusingtheB2levelasanoverall requirement foruniversityentrancedoesnot correspond to the real-lifedemands. In reality, language demands are often unevenly distributed acrossskills (Hulstijn, 2014). Based on these results, it is recommended tomatch thereceptivetesttaskstothereal-worldrequirements.Thiscouldimplythatthatthelevel requirements for receptive skills move closer to C1. As it stands,international L2 students appear quite likely to enter universitywith receptiveskillsthatdonotpreparethemsufficientlyforthereal-lifedemandsinthetargetcontext.


194

COMPAREREAL-LIFELANGUAGEREQUIREMENTSAT

FLEMISHUNIVERSITIESTOSTRTANDITNAOPERATIONALIZATIONS

SummaryofthefindingsThesecondchapterdidnotonlyfocusontheB2levelasanentrancerequirementat Flemish universities – it also compared the operationalization of STRT andITNA to the real-life demands of academia. Speaking, for example, is ofcomparativelyminor importance at university in Flanders. The university staffrankeditastheleastimportantskill,andmostrespondentsspokeDutchrathersparsely in their daily lives. Nevertheless, speaking tasks feature quiteprominently inSTRTand ITNA.At times, thecontentandrequirementsof thetests seem to rely more on general conceptualizations of what language foracademicpurposescomprises,thanonathoroughanalysisofthetargetcontext.Additionally,thestudydidnotgeneratedatatoconvincinglysuggestthatthereisaclearlinkbetweenacademicsuccessandscoresonSTRTandITNA.Thesamplesizeusedtodrawthatconclusionwasrather limited,but theresultsare in linewith other international predictive validity studies that show the weakrelationshipbetweentestscoresandacademicperformance(Cho&Bridgeman,2012;Hill, Storch,&Lynch, 1999;Kerstjens&Nery, 2000;Lee&Greene, 2007).Thesepreviousstudiessolelyconsideredtheacademicperformanceofsuccessfullanguagetestcandidates.Inthecurrentresearch,asmallcohortofstudentswhofailedoneoftwolanguagetestsweretrackedduringtheirfirstyearatuniversity.DiscussionThroughout thisdissertation,differentkindsofdatawerecollected thatcanbeused to investigate the validity argument of STRT and ITNA as universityentrance language tests. It is important to reiterate that test developers arerequiredtosubstantiateavalidityclaiminanycontexttheyexplicitlypromoteorcan reasonably foresee (Kane, 2013). In otherwords: the fact that STRT is alsoused as an entrance test to university colleges and that ITNA is also used as acourse-finalleveltest,doesnotdiminishtheneedfortestdeveloperstoprovideaconvincingvalidityargumentineverysinglecontext. In a forthcoming publication, Kane, Kane, & Clauser (2017) lay out aframework for validating credentialing tests. These tests are used to determinewhether candidates possess the right knowledge, skills, and/or judgment to beallowedaccesstoatargetdomain.Theframeworkprovidesausefulperspectiveon validation, which can be used for the purpose of this dissertation.


195

Interpretation/Use Arguments (IUA) for credentialing tests, Kane et al. (2017)argue,typicallyrelyonminimallyfourinferences:ascoringinference(whichlinksaperformancewithascore),ageneralizationinference(whichtranslatesallitemor task scores to an overall score, often using statistical modeling), anextrapolation inference (which infersreal-lifeperformanceonthebasisofa testperformance),andadecision inference(whichdeterminestheperformancelevelthat marks acceptable competence). These inferences are linked to foursuppositions,whichareparaphrasedanddiscussedbelow.(1) Thetesttasksincorporateskillsthatareessentialforthetargetcontext.Table9.1.STRT&ITNAtasktypes,toimportantskillstomasterwhenenteringuniversity STRT ITNA

Argum

entative

writing

from

aud

io

Summarizelecture

Argum

entative

writing

from

text

Summarizepa

per

Orala

rgum

entation

Presen

tation

Lang

uage

-in-us

e,voc

abulary

Lang

uage

-in-us

e,grammar

Rea

ding

com

preh

ension

Listen

ingco

mpreh

ension

Listen

ingco

mpreh

ension

,dictation

Orala

rgum

entation

Presen

tation

Composealogicalargumentation ★ ★ ★ Takeclassnotes ★ ★ Expressideasaccurately ★ ★ ★ ★ ★ ★ ★Grammaticalaccuracy ★ ★ ★ ★ ★ ★ ★ ★ ★Understandgeneralacademiclexis ★ ★ ★ ★ ★ ★ ★ ★ ★ ★ ★ ★Understandcoherence&cohesion ★ ★ ★ ★ ★ ★ ★ ★ ★ ★Understandimplicitmessage ★ ★ ★ ★ ★ ★Understandscientifictextasawhole

★ ★

Lookupinformation Summarizelongtext ★ Summarizemultiplesources Understandscientifictextindetail ★ Describegraphs&tables ★ ★Giveapresentation ★ ★InChapter2,theuniversitystaffrespondentsandtheL2Fparticipantswereaskedto list the skills they considered essential for students tomaster at the start ofuniversity. Table 9.1 includes those skills in four categories. The first categoryshows the skills thatwere consideredessential bybothuniversity staff andL2F


196

respondents. Below the first dashed line are the skills that were consideredessential by of one of both groups. The third category lists skills that wereselected once or twice in each group, but were not considered important byeither.ThelastcategorycontainstheskillsthatwereneverselectedbyanyL2oruniversity staff respondent. Within each category, the skills are placed in analphabetical and non-hierarchal order. If a given STRT or ITNA task type(Appendix1and2)operationalizesacertainskill,orusesitintheratingcriteria,it is marked with a “★”. The table was constructed by relying on the taskspecifications, on input from theSTRTand ITNA teams, andon the validationreportswrittenbythetestdevelopmentteaminpreparationfortheALTEaudit(CNaVT,2014;InteruniversitairTestingConsortium,2015).

ItisimportanttostressthatTable9.1isaroughoutline,whichprimarilyservestodisplaywhetheratestoperationalizes insomewaytheLAPskillsthatwere introduced in Chapter 2. It offers a binary picture, and does not conveyshades of importance. For example, grammar and vocabulary are pivotal toITNA’sconstructandlargelyaccountforitsdifficultylevel,aswasconcludedinChapter 3 and 4. Judging from the table, however, it may appear that STRTassignsgreaterimportancetogrammarandvocabulary,becausetheyconsistentlyusedasaratingcriterion.WhileTable9.1doesnotconveynuance,itdoesdisplayquireclearlywhetherITNAandSTRTinsomeformconsideracademiclanguageskillsthatareconsideredessentialforthetargetcontext.

TwoessentialskillsarenotoperationalizedinITNA’sB2test,inthesensethattheydonot impacta test-taker’sscore: “compose a logical argumentation”,“take class notes”. At the B2 level, ITNA does not include productive writingtasks, and the rating criteria in the oral component are focused on linguisticquality,ratherthancontent.

Insomeway,theSTRTtasksincorporateeveryessentialskill,butthisdoesnotmean that every skill is operationalizedperfectly. The “scientific texts”, forexample,arepopularizingratherthanacademic,andthelisteningpromptsdiffersubstantially from actual lectures. In ITNA, the reading tasks also rely onpopularizingsources,buttheaudiopromptsaremoreinlinewithnatural–butnotnecessarily academic– languageuse, since theyareactual radio fragments.Because STRT pays considerable attention to content criteria, it consistentlyappealstotheessentialskill“express ideasaccurately”.Nevertheless,thewayinwhichthisskillisoperationalizedisprobablybelowthelevelofreal-lifeacademiclanguage:Chapter 3 showed thatSTRT’swrittenargumentative tasks (inwhichaccurateexpressionofideasisofpivotalimportance)anditscontentcriteriaaredisproportionallyeasy.


197

(2) The absence of certain test tasks would make a substantial difference in

real-worldpracticeeffectiveness.Theprimacyof speaking inSTRTand ITNAcontrasts to somedegreewith themoderateimportanceofspeakingskillsinthetargetcontext.Theprominenceofspeakingtasksinthetestsmayhavetobereconsidered,ortheapproachthatisadopted(i.e.,bothtestscouldfocusmoreonsocialinteraction,orondiscussingatopicchosenby thecandidate). Inanycase, thecurrent ITNAdesign, inwhichaccesstotheoralcomponentisgrantedbasedonthecomputertestscore,seemstogiveanauraofimportancetospeaking,whichdoesnotcorrespondtoreality.

It isalsodoubtfulwhetheromitting thedictation task fromITNAwouldmakeasubstantialdifferenceinreal-worldpracticeeffectiveness,sinceitmisfitstheRaschmodelcomposedofallSTRTandITNA’swrittentasks(InfitMnSq1.71–seeTable4.5)anddoesnotoperationalizeanyoftheessentialLAPskills.Lastly,there are clear indicationsof redundancy in thewrittenSTRTcomponent.TheRasch model does not reliably differentiate between the two summary tasks(Listening,Measure=-.01,SE= .08;Reading,Measure=-.04,SE= .08)andthetwoargumentationtasks(Listening,Measure=-.59,SE=.11;Reading,Measure=-.51, SE = .10), one of which overfits the Raschmodel (Argumentative writing-from-readingInfitMnSq=.41).Inawritingtestthattakesthreehours,itmightbebeneficialtoomittasksthatmaybeconsideredredundant.(3) Thescoringisaccurateandconsistent.Before conducting analyses on the test scores (Chapter 3 and 4), the ratingproceduresofSTRTandITNAwerechecked.TheRaschrateranalysisoftheL2Fperformances on STRT showed only minimal differences in rater severity (.73logits).At .35, the reliabilitywithwhich themodel coulddifferentiatebetweenraters’severitywaslow,which,togetherwiththenon-significantX2(4)=3.5(p=.47), showed that the assumption of rater equivalence could be upheld. TherewerenodirectindicationsthatcastdoubtontheaccuracyorconsistencyofSTRTratings.

ApythonscriptwasusedtodeterminewhichanswersITNA’sautomatedscoringtoolconsideredcorrector incorrect,andtoverifythescoringalgorithmofITNA’scomputercomponent.Sinceonlyone,ratherminor,inconsistencywasfound,itwasdecidedthatthescoringprocedureofITNA’scomputersectionwassatisfactory. The accuracy and consistency of the scoring procedure used inITNA’s oral component is hard to assess, since two raters reach one jointlyagreed-upon score. ITNA’s internal audit report (Interuniversitair TestingConsortium,2015)only includes information regarding thecorrelation (r= .90)inter-rater agreement (k = .73) between two raters. The report also mentions


198

researchregardingtheconsistencyacrosstestcenters.AcomparisonofthetotalscoresassignedinthedifferentITNAtestcentersyieldnon-significantresultsonthe Kruskal-Wallis test (p = .39). The report takes this as evidence of ratingconsistency,butitispossiblethatothervariables(e.g.,strongercandidateswouldmaskstricterraters)haveinfluencedthisoutcome.Consequently,itisimpossibleto make any reliable claims regarding the accuracy of ITNA’s oral scoringprocedure.

Chapter4offeredadditionaldataregardingtheconsistencyofthescoringproceduresacross testsandshowedthat thesamecriteriawereoperationalizedverydifferentlyinSTRTandITNA.Suchacomparisondoesnotyieldinformationregarding internal scoring consistency, but it does underscore the uncertainnatureofdeterminingwhat“appropriate”scoringmayentail.(4) Thepassingscoreisappropriate.Ifanentrancetest isconsideredameasureofnecessary (i.e.,what isminimallyrequired) rather than effective performance (i.e., what should ideally bemastered),candidateswhostrugglewithtesttaskscanbeexpectedtoexperienceproblemsinthetargetcontext.Ontheotherhand,notallcandidateswhopasswill be expected to thrive (Kane et al., 2017). This line of reasoning appearsgenerallytrueforSTRTandITNA,thoughthesamplesizeusedtoassessitinthisresearchwastoosmalltomakeanydefinitiveclaims.Table9.2.STRT&ITNAresultvs.academicsuccess Academicsuccess <50% >50%

STRTFail 1 0 Pass 7 8

ITNAFail 2 2 Pass 6 6

Corroborating previous research (e.g., Cho & Bridgeman, 2012; Lee & Greene,2007), no significant relationship was found between language test scores andacademicsuccess.Theeffectsizesforbothtestswerecomparable(STRT,r=-.115;ITNA,r=-.120).Acrosstab(seeTable9.2)showstherelationshipbetweenSTRTand ITNA pass/fail outcomes and academic success (defined by having passedmore than 50% of the credits). Based on the sixteen respondentswho did notleavetheprojectprematurelybecauseofvisaissuesorwithoutstatingareason,ITNAassignedtwofalsenegativesandsixfalsepositives,whereasSTRTassignedzero and seven respectively. If we include the L2F respondent Stella, who hadachieved good academic results in January but had to leave the country some


199

time later, STRT would have assigned one false negative, and ITNA three.Deducing any sort of trend from the candidates who failed the language testwould be careless. The sample sizes are simply too small. Nevertheless, falsenegativesshouldbeavoided–ifonlyfromasocialjusticeperspective.

It is likely that simply raising the overall level of the tests to a C1 levelcouldalsoraisetheleveloffalsenegatives.IfonlycandidateswithaC1levelonSTRT’slinguisticcriteriahadbeenallowedtoregisterforuniversitySTRTwouldhaveassignedsevenfalsenegatives.Thishypothesisisbasedonscoresforwrittenandspokenproduction.Itremainstobeseenwhattheeffectwouldbeofusingmoreauthenticlisteningandreadingprompts.Nevertheless,giventheavailableinformation, there is no immediate reason to assume that the cut off score ofSTRT or ITNA would be inappropriate or substantially misguided. Furtherresearchcouldaimtofleshouttheappropriatenessofthepass/failboundary.EMPIRICALLYESTABLISHTOWHATEXTENTSTRTANDITNA

SCORESCANBECONSIDEREDEQUIVALENTSummaryofthefindingsRelying on the scores of 118 participantswho took STRT and ITNAwithin thesameweek,thestudydiscussedinChapter3showedthatthecorrelationbetweentheSTRTandITNAoverallandwritingscoreswasmoderatelyhigh(overallr=.767**;writing r = .694**).Theagreementbetween the scoreson theoral testswas much lower however (τ = .387**). Additional analyses revealed furtherdiscrepancies. T-tests confirmed that the differences between STRT and ITNAmeanscoresweresignificant(p<0.001),witheffectsizesrangingfromd=-0.53(writing components) tod = -1.41 (speakingcomponent).Additionally, thepassprobabilitywasfoundtobesignificantly(p= .02) largerforSTRT(.50)thanforITNA (.35). Linear regression andMulti-Faceted Rasch analyses indicated thatITNA’s reliance on comparatively difficult language-in-use (i.e., grammar andvocabulary) tasks, combined with the inclusion of two relatively easyargumentative tasks in STRT may explain the discrepancy in the writtenmodality.Additionally,RaschanalysisshowedthatITNA’sspokencomponentismoredifficult thanSTRT’s–againbecausetherelativeweightofcomparativelydifficult linguistic criteria (grammar and vocabulary) in the former, and therelativeimportanceofcomparativelyeasycriteria(e.g.,contentcriteria).

The following chapter considered the scores on the oral components ofSTRT and ITNA. These components are very similar in terms of task type andratingcriteria.Both tests include fivecriteria thatarebasedon thesameCEFRdescriptors.LinearandmultipleregressionandMulti-FacetedRaschshowedthatfor every CEFR-based criterion ITNA and STRT interpreted the B2 level in a


200

different way. Weighted kappa coefficients were low for every correspondingcriterion (kw≤ .216), andcorrespondingcriteriawerenever includedwithin thesamedifficultybandofthemulti-facetedRaschanalysis.DiscussionThisstudydidnotfindevidencetosupportthatSTRTandITNAoperationalizecorrespondingCEFR-basedcriteriaincomparabletasksinacomparableway.Thedivergingoutcomes can likelybe explained, at least inpart,by considering thetests’constructs,aswellastheirdifferinginterpretationsofCEFRcriteria.ITNAprioritizes lexis and grammar. STRT assigns substantial importance to non-linguisticcontentcriteria,whichconsiderablyimpactsSTRTscores.ThefactthatthereisasignificantdifferencebetweenSTRTandITNAinpassprobability,andthat one in four candidates receive a different pass/fail judgment should notnecessarilybeseenasaproblem.As longasbothtestshavebeen linkedto theCEFR inanappropriateway (whichappears tobe thecase),andas longas thevalidityargumentforbothtestsisconvincingforthetargetcontext(whichdoesnotfullyappeartobethecase),testsdonotneedtohavethesamecutofflevel.AslongasbothtestsreliablymeasureatsomepointwithintheB2range,thereisnocauseforconcernfromtheperspectiveofthetestdevelopers.

From the perspective of the university accepting both tests in a legallyequivalent way, a substantial difference in pass/fail judgments, combinedwithevidenceofdifferentinterpretationsofhomonymousconstructsandcriteriamaybecauseforconcern.ItseemsopportuneforuniversityadmissionofficerstobeawareofthedifferencesbetweenSTRTandITNA.Possibly,theuniversitycouldconsider informingprospectivestudentsof thesedifferences too.As there isnoreasontoassumethatanytestisabetterpredictorofacademicsuccess,students’accesstouniversityshouldnothingeuponthetesttheychose.

DETERMINEWHETHERALLSTUDENTSWHOENTERUNIVERSITYWITHAFLEMISHHIGHSCHOOLDEGREEPASS

THEB2THRESHOLDSummaryofthefindingsToexaminewhetherallstudentswithaFlemishhighschooldegreehaveattainedthe B2 level in Dutch, 159 first-year Flemish L1 students sat twowritten STRTtasks during their first month of university education. Using non-parametricstatisticsandMulti-FacetedRaschanalysis,theL1scoreswerecomparedagainsttheperformanceoftwogroupsofL2candidates:L2studentswhohadstudied


201

Dutchattheirhomeinstitution(N=629),andL2studentswhohaddonesoinFlanders (N = 116). The results showed that L1 students significantly (p < .000)outperformed both groups of L2 students – both overall and on the linguisticcriteriaofbothtasks(whenusingaconglomeratescoreforall linguisticcriteriausedinonetask),withmediumeffectsizes.L2studentswhohadstudiedDutchabroad achieved significantly (p < .000) higher scores on content criteria.Importantly, however, eleven percent of the L1 students did not attain the B2levelasmeasuredbytheSTRTwritingtasks.Logisticregressionshowedthatoutof all linguistic criteria, scores on Grammar and Vocabulary were the bestpredictorsofmembershiptothegroupofFlemishstudents.DiscussionThe fourth chapter did not offer empirical support to the assumption that allFlemishhighschoolgraduatespossesstheB2level.Logically,italsounderminestheassumptionthatpeoplewhohavespentoneyearataFlemishhighschool–without necessarily graduating from one – meet the B2 requirement. From aresearch perspective, this is not entirely surprising. It helps confirmHulstijn’shypothesis thatL2 learnersmayoutperformL1usersoncognitivelydemandingtasks (Hulstijn, 2015; but see also Stricker, 2004), and corroborates previousfindingsconcerningtheimportanceofvocabulary(Weigle,2002;Wolfe,Song,&Jiao, 2016) and grammar (di Gennaro, 2016) in distinguishing L1 and L2performance.

Thisstudy–perhapsforthefirsttime–comparedL1users’performanceon a centralizedL2 testwith theperformanceof L2 learners. The results showquite clearly that graduating from secondary school does not automaticallyguarantee B2 ability in the language of instruction. Additionally, the results ofthis study indicate that studying a language in the target context does notautomatically lead to higher test scores compared to studying a language in ahomecontext.TheimplicationsofbothfindingswillbediscussedinChapter9.


202

TRACKANDEXPLAINLANGUAGEGAINSMADEBY

INTERNATIONALL2STUDENTSDURINGTHEIRFIRSTYEARSummaryofthefindingsIn order to measure the language gains and document the experiences ofinternational L2 students at Flemish universities, twenty respondents wereregularly interviewed during their first academic year at a Flemish university.After eightmonths, the respondents who had not left university (n = 13) tookSTRT writing and speaking tasks again (the combination of these tasks waspredictivefortheoverallscoreat𝑅!"#! =.908,p<.000).

Theresultsshowedthattherespondentshadmadenosignificantgainsintermsoftestscore,or intermsofmeasuresofcomplexity,accuracy,orfluency.Theonlysignificant(p=.03)differencewasadecreasedamountofwordsusedintheoralpresentationtask.Theinterviewdatashowedthatnearlyallrespondentshad experienced some degree of social and academic isolation, and reported aperceived lack of institutional support. Likely, an important reason why therespondentshadmadelimitedorzerogains,waslimitedopportunitiestoengageinmeaningfulinteractionwithL1speakers.DiscussionBy operationalizing the Douglas Fir framework, this study has found thatidentifying the dynamic interactions between institutional mechanisms andinterpersonal relationships can help to pinpoint patterns that affect languagelearning. While it is important to recognize that these findings stem fromobservations at Flemish universities, it is likely that similar dynamics occur inothercontextsaswell, inviewof the fact thatsomeof theresultsof this studyhave been confirmed by previous research. Limited language gains byinternationalstudentshavebeenreported inAustralia(Knoch,Rouhshad,Oon,& Storch, 2015; Knoch, Rouhshad, & Storch, 2014; Storch, 2009); internationalstudents’frustrationswithdidactictraditionswereobservedintheUS,inCanadaand in the UK (Amuzie &Winke, 2009;Morita, 2004; Gu, 2005; Gu &Maley,2008); feelings of alienation, distance, and limited interaction with the L1communityhavebeen reported in theUS,Russia, France,Canada, and theUK(Amuzie &Winke, 2009; Gu &Maley, 2008; Kinginger, 2004; Kinginger, 2008;Morita,2004;PellegrinoAveni,2005;Ranta&Meckelborg,2013).Thisstudythusconfirmedpreviouswork,yetalsoaddstothe literaturebyconnectingthedotsallthewayfromlanguagegainstoideology.Whattheresultsofthisstudyimplyintermsofpolicywillbediscussedbelow.

203

Chapter9:Limitations,implications&recommendations

204

CHAPTER9LIMITATIONS,IMPLICATIONS&RECOMMENDATIONSThislastsectionisconcernedwiththeshortcomingsandstrengthsoftheresearch, and with its real-world implications. It brings together theempirical findings to reassess the original assumptions that drove theresearch questions, and considers how the findings could impact theuniversityadmissionpolicy,giventhepragmaticrealityofpolicymaking.

AFEWWORDSONSTRENGTHSANDLIMITATIONSContextThisstudywasconductedinFlanders–anatypicalsettingintheAnglo-Americandominated research domain of university entrance language testing. As is thecase with much contextualized research, generalizing the results beyond theoriginal context should be done with utmost care. Even within Flanders, it isquite likely that the resultswould have been somewhat different had the databeencollectedatuniversitycollegesratherthanatuniversities.

On the other hand, conducting research in the comparatively small andsparsely researched context of Flemish universities has a number of benefits.First,giventheeducationallandscapeinFlanders,itwaspossibletotalkwiththeright policymakers at every university and within the Flemish Department ofEducation. Additionally, as was discussed in Chapter 6, the modest status ofDutch as an international language ensured thatmost L2 respondents hadnotbeenexposedto itbeforetheyhadstartedstudying it. Inrelationtothispoint,theL2 respondents inFlanderswere less likely touseDutchas a lingua francawhencommunicatingwithspeakersofadifferentL1thantheywouldwithatrulyinternationallanguagesuchasEnglish.Real-worldsettingTheratherpracticalnatureoftheresearchgoalsmeantthatresearchdatahadtobecollected ina real-worldcontext (Horii, 2015).This isbotha strengthandaweaknessofthecurrentresearch.Ontheonehand,collectingdatainthetargetcontext eliminates the need to make the kinds of inferences that laboratory-based researchwouldneed tomake.L2 respondentswho tookSTRTand ITNAbore the real-world consequences of their performance, and tracing those wasone of the primary research goals. On the other hand, in the real world,


205

unforeseenoruncontrollablesituationsoccur,whichimpactsthedatacollectionin various ways. Some of these factors contributed to the richness of the data(e.g., respondents dropping out offered an insight into their reasons for doingso), but others not somuch (e.g., being tied to theMay - October period forcollectingSTRTand ITNA testdata impacted the sample size). It is alsoworthnotingthat, incontrasttotheL2participants,thetestscoresdidnothavereal-worldconsequencesfortheFlemishstudentswhotooktwoSTRTtasks.Thismayhaveaffectedthewaybothgroupsapproachedthetests,andmayhaveimpactedtheirperformance.

Lastly, the policy makers, language testers and academic staff wereinterviewedabouta stateofaffairs thatwas trueat the timeofdatacollection.The results presented here were accurate at the time of data collection butpoliciesmaychangefromyeartoyear.Therefore,thefinalinterviewswithpolicymakerswereorganizedattheendofthewritingprocess.Assuchwecanassumethattheirtestimoniesremainfactuallyaccurateforatleasttheremainderofthe2016-2017academicyear.PopulationandsamplesizeThesamplesizesusedinthisresearchfluctuatedquiteabit.Thelanguagegainswere measured among a relatively small population of thirteen respondents,whiletheL2Ipopulationconsistedof526candidates.Althoughcarewastakentointerpret the results in the light of sample sizes, and results were backed byreferencing other studies, it is important to keep in mind sample sizes whenconsideringscoregains,groupdifferences,andeffectsizes.

Insomecases,populationsizesweredictatedbypracticalconstraints–aconsequenceofthereal-lifesettingofthisresearch.TherecruitingamongtheL2Fpopulation could not extend beyond the predetermined natural stoppingcriterion; thestartof theacademicyear. Inothercases, largergroupsizeswereimpossible given the design of the study. In Chapter 2 and Chapter 6,quantitativeresultswereusedtoaddan interpretative layer toqualitativedata.Thenumberofparticipantsinvolvedinthesestudies(N=55inChapter2,N=20inChapter 6)was rather small for quantitative analyses, but rather substantialgiventhenatureofthequalitativedatagatheredinthelongitudinaldesign.

Even though the demographics of the sample populations arerepresentativefortheactualtestpopulation,thesamplingmethodologyusedinthis study could qualify as convenience sampling. This links in with the real-world nature of the data collected, andmay impact the generalizability of theresultsinChapters3and4.BecauseITNAregulationsonlyallowcandidateswhopassedthewrittencomponenttotaketheoralexam,thenumberofrespondentsthat could meaningfully be compared for the oral component was reduced.Althoughallstatisticalassumptionswerecheckedpriortotheanalyses(Purpura,


206

Brown,&Schoonen,2015),rangerestrictionmayhavehadaneffectonthedata,weakeningthecorrelations.TheactofconductingresearchTheveryactof conducting research influences reality.Undoubtedly, frequentlyinterviewing the L2F participants during their first academic year at a Flemishuniversitychangedthewaytheyexperiencedthatyear.Duringtheretrospectiveinterview, various participants explained why being part of this study was astrengthening experience. Even though the researcher never arranged for theparticipants tomeet, someacknowledged thatbeing in the study andknowingthat therewereotherpeople in thesamesituationoffered themsomecomfort.Otherstudentsfeltsupportedbybeingabletosharetheirstory.

EveryL2Fparticipantinthelongitudinalstudyhadaverydifferentstorytoshare,andthis resulted ina largedataset. In thisdissertationIhave looked forpatterns and correspondences to link their stories (Chapters 2 and 6).While Ihave taken care to include important nuances and highlight salient deviationsfromotherwiseconsistentobservations,noresearchpapercandojusticetoeachindividual story. Inorder tomake sure thatnoparticipant feltmisrepresented,theywereallgiveninsightintotheresearchresultsbasedontheinterviews,withthe explicit invitation to contactme if theywanted a section tobe revised.Allparticipants but onewere happywith the rendition of the interviews. The oneparticipantwhoaskedforaneditrequestedthatsomequoteswereomitted,sincethey were too personal. Needless to say, these quotes were removed from thetext.

IMPLICATIONSThediscussionofthepolicyimplicationswillrelyonFischer’sapproachtopolicyevaluation(Fischer,2003,2007).DrawingonToulmin’sargumentativestructure,Fischerassignssubstantialimportancetodatafrommixedmethodsresearch,andto a dialogue with policy makers. The first two policy evaluation levels areconcernedwithpolicy itself.Thetworemaining levels focusonthe impactofapolicyonawidersociety.Below,thelevelsarediscussedinpairs.PolicyimplicationsProgramverificationinvolvesdeterminingtowhatextentthepolicymeasuresareeffectiveinachievingthepolicygoal.Situationalvalidationisconcernedwiththeassumptionsbehindthepolicy,andaskswhethereverypolicymeasureisequallyrelevanttosolvingtheperceivedproblem.


207

Thepolicygoal,asvoicedbythepolicymakers is toselectstudentswhohave enough language proficiency to be able to attend a Dutch-mediumuniversity program. The perceived problem is students attending classwithoutsufficient language proficiency for doing so successfully. By the policymakers’ownassessment,notallpolicymeasuresareequallyeffectiveinlightofthepolicygoal, but they do represent a workable compromise. Based on the researchfindings discussed above, Toulmin’s argumentative structure can be used torevisetheoriginalassumptions.Assumption1Thefirstassumptionguidingtheresearchwas“B2isanadequatethresholdleveltodetermineinternationalL2students’accesstoaDutch-mediumuniversity inFlanders”.Basedon(data):

- TheinterviewswithEuropeantestdevelopers(Chapter1),whichstressthegenerallyweakempiricalfoundationforusingtheB2levelasauniversityentrancelevel;

- Theacademicrespondents(Chapter2),whodoubtedtheadequacyoftheB2levelforreceptiveskills;

- TheL2Frespondents(Chapter2),whopassedSTRTand/orSTRTattheB2level,yetstruggledwiththelinguisticdemandsofuniversity;

- The very weak relationship between passing a B2 test and achievingacademicsuccess(Chapter2);

- Theinterviewswithpolicymakers(Chapter7);Assumption1canberevisedinto:B2may(qualifier)functionasaminimumthresholdforuniversityadmission

if(rebuttal1) the admission officer is aware of the wideperformancerangeinherentintheB2level,

eventhough(rebuttal2) differentiated language level requirementscorrespond more closely with reality andresearchconsensus.

Thewarrantconnectingthedata to theassumptionreliesonthe interpretationandanalysisofqualitativedata,ontheinterpretationofnon-parametricstatisticsand on the principles of mixed methods data triangulation. Backing for thiswarrantcanbefoundintheneedsanalysis(e.g.,Gilabert,2005;Long,2005)andpredictivevalidity(e.g.,Cho&Bridgeman,2012;Lee&Greene,2007)literature.


208

Assumption2The second assumption (“STRT and ITNA are representative of the academiclanguage requirements at Flemish universities”) pertains to test developmentratherthantopolicy.Eventhoughthisisnotanassumptionforpolicymakerstojustify,itisonetheyshouldbeinformedabout.Basedon(data):

- The interviews with the academic respondents and the international L2students(Chapter2);

- Thefieldnotes,includingtheclasstranscriptions(Chapter2);- The comparison between essential skills and test task requirements

(Conclusion,p.195);Assumption2canberevisedinto:Toanextent(qualifier),STRTandITNAincludeessentiallanguageskillsforthetargetcontext

but(rebuttal1) theysometimesappeartoalignmorewiththetargetlevelthanwiththetargetcontext,

and(rebuttal2) constructover/underrepresentationdoesoccur.The warrant combines principles of mixed methods data triangulation andprinciples of TLU analysis, which have been used in the LSP/LAP literature(Douglas, 2000; Hyland, 2001 Weigle & Malone, 2016), and in the validationliterature(Kane,2017).Assumption3Theoriginal formulationof the third assumptionwas: “STRTand ITNAcanbeconsideredequivalentmeasuresofDutchlanguageproficiencyattheB2level”.Basedon(data):

- Thelevelequivalencedata(i.e.,ttest,McNemar’stest,MFRA,Chapter3);- Theconstructequivalencedata(i.e.,linearregression,MFRA,Chapter3);- Thecriterionequivalencedata(i.e.,linearregression,MFRA,Chapter4);- Theinterviewswithpolicymakers(Chapter6);

Assumption3canberevisedinto:STRTand ITNAwill likely (qualifier)assignacorrespondingpass/fail judgmenttomostcandidates


209

but(rebuttal1) they operationalize homonymous constructs andcriteriadifferently,

and(rebuttal2) may yield a different pass/fail judgment for asubstantial proportion of the population around thecut-offscore.

ThewarrantreliesontheinterpretationofinferentialstatisticsandMulti-FacetedRasch analysis, which have been used in equivalence research (e.g., ETS, 2010;Riazi, 2013…), concurrent validation (e.g., Lissitz & Samuelsen, 2007…), CEFR-basedratingscaledesign,(e.g.,Galaczietal.,2011;Harsch&Martin2011…).Assumption4Theassumptionthat“studentswithaFlemishhighschooldegreehaveobtainedDutch language proficiency at B2 level” can be rephrased based on the STRTwritingtestdata(Chapter5):Most Dutch-medium high school graduates (typically from the academically-orientedstrand)willlikely(qualifier)meettheB2thresholdforwriting

but(rebuttal1) most probably this claim cannot be maintained forstudents with a different home language or anatypicalSEprogram.

Thewarrant reliesonthe interpretationofnon-parametricstatisticsandMulti-FacetedRaschanalysis,whichhavebeenusedin–amongothers–L1/L2writingresearch(e.g.,Lekietal.,2008;Polio,2013;Weigle&Friginal,2015).Assumption5Based on the research data, the original formulation of Assumption 5(“International L2 studentswillmake language gainsby virtueof studying at aFlemishuniversity”)cannotbemaintained.Based on the analysis of the STRT test/retest data and on the longitudinalinterviews, it is clear that: “International L2 students in Flanders who attendlarge-scale study programsmust not be assumed tomake productive languagegainsifthereisnoappropriatepost-admittancepolicyinplace”.Since this is not an assumption but a statement, there are no rebuttals. Thewarrant linking the data to the statement relies on non-parametric statistics(usedtomeasuredifferencesintermsofscore,linguisticcomplexity,accuracy,or


210

fluency) and on principles of mixed-method data triangulation, previouslyemployedintheL2languagegains,andthestudyabroadliterature(e.g.,Storch,2009;Knoch, 2015; Serranoet al., 2012).Assumption5doesnotdirectly impacttheadmission requirements,butdoeshave implications for thepost-admissionpolicy,whichwillbediscussedbelow.In sum, the Flemish university admission policy does not appear to be fullyeffective tomeet the policy goal:Many studentswho enter universitywill stillexperience linguisticproblems.Additionally, it hasunwanted side-effects (suchasfalsenegatives),itincludesempiricallyunjustifiedexemptions,anditreliesonpartially unfounded assumptions.At the same time these policy characteristicsdo not appear to deviate substantially from the average European universityadmission regulations. Moreover, the limited effectiveness of the universityadmission policy in Flanders does not contrast with the policy makers’assessment, who are fully aware that some admission requirements areconcessionstoorcompromiseswithimportantstakeholders. Therevisedassumptionsaretheresultofacademicratherthanpragmaticreasoning,andassuchtheymaybetoonuancedtobeuseableforpolicymakers.They have, however been used as the foundation for concrete policyrecommendations,attheendofthischapter.SocietalimplicationsThe societal layerof Fischer’spolicy evaluationmodel incorporates the level ofsocietal vindication, which examines whether a policy contributes positively tothewider context inwhich it operates, and the level of social choice,which isconcernedwiththeideologyonwhichapolicy isbased,andquestionswhetherthat ideologywould be conducive to building a society that embraces equalityand freedom.The basic concerns of societal vindication and social choice alignwellwiththedefinitionofjusticeoperationalizedinChapter2.

Ideologically, the university language regulations show a tendency forlinguistic protectionism (see Introduction and Chapter 5): There are strictlanguage quota, and legally binding language requirements for internationalprofessors. Likewise, every university has chosen to implement languagerequirements for international students.Aswasargued in the introduction, thelanguage ideology inFlanderscouldbedescribedas territorialmonolingualism.Central to this idea is the conception that Flemish L1 users of Dutch are thenorm, and international students or migrants are expected to adjust to thatnorm,downtothelevelofpronunciation(Blommaert,2011;VanSplunder,2016).In such a context, introducing language tests for university admission is notsurprising. It is far from unique however: University entrance is one of theprimaryfieldsofhigh-stakes languagetestingworldwide.Atthesametime,the


211

justice of having tests determine access to education is rarely questioned(McNamara&Ryan,2011).Justice authors have argued that the introduction of requirements andregulations that limit the access of one specific subgroup from a largerpopulation may cause inequities (Kunnan, 2000, 2004). In Chapter 2 it wasargued,relyingonSen(2010),thatapolicyislikelytobeunjustifitrestrictstesttakers’ freedomof accesswithout empirical or reasonablemotivation. A policymay thus limit the access of certain people to certain goods, services orcapabilities,onlyifitreliesoncarefulexaminationofsolidevidence.Absenceofevidence, or the presence of negative evidence implies that a policy is unjust(Dworkin, 2013). There is of course always a chance that tests or policiesdiscriminate adequately by accident, but as Sen argues, this is insufficient.Heuses the analogy of a clock todemonstratehis point (Sen, 2010: 40):Abrokenclockisexactlyrighttwiceaday,butthatdoesnotmakeitmorereliablethanawatchwhichrunsalittlebehind.Senthusprefersapolicythatreliesonreasonedscrutiny,notbecausethisyieldsaperfectlyjustsystem,butbecauseitimpliesaconcernforanevidence-basedpolicy.

Ifweapplythislogictothecurrentresearch,itisclearthatthepurposeofthe Flemish admission policy (i.e., ensuring that incoming international L2students possess the language proficiency required to attend university) isethically defensible. Given the number of false positives and false negatives,however, the system in place appears rather ineffective. Additionally, itdiscriminates among international students (but notall international students,since the entrance requirements donot apply to certain programs) based on acriterionthatnotallFlemishstudentsmeet.All inall,theempiricalfoundationsupporting the admission system appears rather thin, and most of the policymakers’ assumptions are unsupported by empirical data. Even within theparadigm of territorialmonolingualism, this dissertation strongly suggests thatthe Flemish university admission policy could be revised to meet higherstandards in terms of justice and empirical foundation. Evidence from othercountriesindicatedthatFlandersisnotexceptionalinthisregard(e.g.,Carlsen,2017)–itissimplythefirstcontextinwhichtheprimaryassumptionshavebeensystematicallyexaminedfrommultipleperspectivesinonestudy.

RECOMMENDATIONSRecommendationsforpracticeThereareanumberofpossibleways inwhichthisdissertationcouldhavereal-worldimpact.Onepossible,albeitundesirable,scenarioiszeroimpact:nothinghappens.Byincludingpolicymakersinthisstudy,andtalkingtothemaboutthis


212

study,thisoutcomehashopefullybeenprevented,butChapter7didshowquiteclearlythatempiricaldatadonotalwaysleadtopolicychanges.However,ifthereis willingness to improve current practice – and this seems to be the case –relatively straightforward adjustments by different stakeholders could yieldimportantimprovementsinthefield.

Languagetestdeveloperscouldconsideraligningthetestconstructsmorewith the real-world requirements.Listening tasks could featuremore authenticprompts,whichmayfuelpositivewashback.Theweightingoforaltaskscouldbereduced in order to align better with the actual importance of speaking atuniversity. These recommendations should not be rushed, of course, but theyseemworthy of investigation. In linewith this, the admission requirements ofuniversities couldbe re-examined.Conductingneeds analyses inorder todrawup language requirements that meet the actual language needs of prospectivestudents would be a major step forward, but given the reality of stakeholdersinfluencing admission requirements for the purpose of controlling studentnumbers,thiswilllikelynothappensoon.

Perhaps themost important recommendationconcernsnot theentrancerequirements,butthepost-admittancepolicy.AfterinternationalL2studentsareregistered for Dutch-medium programs, they become part of the generalpopulation.Nospecificaccommodationsareinplaceforthisgroup.Torecognizethepresenceof this group (e.g., bywelcoming them specifically), to create thecircumstances that would help them build a network, and to address theirlanguage-relatedneeds(e.g.,byprovidingclassrecordings),wouldbeabigstepforward.

Finally, as Byrnes et al. (2010) argued, the responsibility of a universitydoesnot stop at admission – that iswhen it begins.Consequently, universitiescouldconsider settingattainment targets inaddition toentrancerequirements.Universities could help international L2 students make language gains byproviding curricular language classes that offer language support throughouttheiracademictrajectory.ImplicationsforresearchEach chapter lists the research gaps addressed in this dissertation. Brieflysummarized,thisdissertationhasprovideddataregardingtheuseoftheCEFRinuniversityadmission testing,hasexaminedcontext representativeness from theperspectiveof justice,hasscrutinizedtestequivalencebyconsideringitem-levelscoresandratingscaledescriptors,hascomparedL1andL2testperformance,hasoffered an explanation for limited spoken and written language gains byconsideringthesocialcontext,andhastracedtheoriginsoftheadmissionpolicybyconsultingpolicymakers.Thisis,tothebestofmyknowledge,thefirststudy


213

to systematically examine the main assumptions behind an admission policyfromavarietyofperspectives.Methodologically, thisdissertationhasproposedanumberof innovations,suchas bypassing truncated sample issues, operationalizing level and constructequivalence, estimating rating scale descriptor similarity by using the JaccardIndex, and utilizing the Douglas Fir framework as a method of qualitativeanalysis.Undoubtedly, thereare limitations to this research(seeabove)and itsoperationalization,but futureresearchcoulduse theseapproachesasastartingpointforfurtherexamination.

The following three implications could serve as an inspiration foradditional research. First, this dissertation demonstrates the need to keepexamining and validating important test-related assumptions – nomatter howwidely-held they are (Connolly, Arkes, &Hammond, 1999;McNamara& Ryan,2011; Phillips, 2007). Second, this research echoes that test validation can andshouldgobeyondthetestitself,andconsidertheimpactonsociety(Kane,2013;Messick, 1989),notonlyfromarational,butalsofromareasonableperspective(Toulmin,2001).Thirdly,relatedtotheideaofreasonableness,itisimportantforlanguagetesterstoengagewithreal-worldpractice. Ifpolicymakersweremoreactively involved in research, or if they were more actively informed aboutresearch, maybe our research would have more of an impact, and maybe ourrecommendationswouldhavemoreresonance.


214

“THENEEDFORVALIDATION,BABYGONECOMPLETELYBERSERK”Quite likely,NickCave, inthemottoof thisdissertation,referredtoadifferenttypeofvalidationthanMessickorKane.Butwasheright,anyway?Iwouldarguethattheneedforvalidationisperhapsmorepressingnowthanithaseverbeen.

In an age of unprecedented migration flows, more people – refugees,students, academics – are affected by the use of language requirements asgatekeeperstovaluedstatuses,goods,orservices.Asthisdissertationhasshown,the motivation for these requirements may reside in empirically unsoundassumptions or unverified claims. Logically, as the number of people impactedrises, so does the extent of the impact ofmisguided assumptions and populistclaims.

Thisevolutionemphasizesthesocialresponsibilityoflanguagetesters.Ifitis the purpose of science to find truth, and the purpose of policy to advancejustice, then certainly there is a role for research to hold policy claims to thelight,anddistinguishbetweentruthanduntruth.

References

216

REFERENCESACTFL.(2012).ACTFLProficiencyGuidelines2012.Alexandria:AmericanCouncil

onTheTeachingofForeignLanguages.ACTFL.(2016).AssigningCEFRRatingstoACTFLAssessments.American

CouncilOnTheTeachingOfForeignLanguages.RetrievedOctober20,2015,fromwww.actfl.org.

Agirdag,O.(2010).Exploringbilingualisminamonolingualschoolsystem:insightsfromTurkishandnativestudentsfromBelgianschools.BritishJournalofSociologyofEducation,31(3),307–321.

Alderson,C.(1991).Bandsandscores.InC.Alderson&B.North(Eds.),Languagetestinginthe1990s(pp.71–86).London:Macmillan.

Alderson,C.(2007).TheCEFRandtheNeedforMoreResearch.TheModernLanguageJournal,91(4),659–663.

Alderson,C.,Clapham,C.,&Wall,D.(1995).LanguageTestConstructionandEvaluation.Cambridge:CambridgeUniversityPress.

Alderson,J.,Figueras,N.,Kuijper,H.,Nold,G.,Takala,S.&Tardieu,C.(2006).AnalysingTestsofReadingandListeninginRelationtotheCommonEuropeanFrameworkofReference:TheExperienceofTheDutchCEFRConstructProject.LanguageAssessmentQuarterly:AnInternationalJournal,3(1),3-30.

ALTE.(2001).PrinciplesofgoodpracticeforALTEexaminations.RetrievedFebruary13,2014,fromwww.alte.org.

Amkreutz,R.(2013,September24).Amper40procentvandeeerstejaarsstudentenslaagt.RetrievedMay25,2016,fromwww.demorgen.be.

Amuzie,G.,&Winke,P.(2009).Changesinlanguagelearningbeliefsasaresultofstudyabroad.System,37(3),366–379.

Baba,K.(2009).Aspectsoflexicalproficiencyinwritingsummariesinaforeignlanguage.JournalofSecondLanguageWriting,18(3),191–208.

Bachman,L,&Palmer,A.(2010).LanguageAssessmentinPractice.NewYork:OxfordUniversityPress.

Bachman,L.(1985).Performanceonclozetestswithfixed-ratioandrationaldeletions.TESOLQuarterly,19(3),535–556.

Bachman,L.(1990).FundamentalConsiderationsinLanguageTesting.Oxford:OxfordUniversityPress.

Bachman,L.(2002).Somereflectionsontask-basedlanguageperformanceassessment.LanguageTesting,19(4),453–476.

Bachman,L.,Davidson,F.,&Ryan,K.(1995).AnInvestigationIntotheComparabilityofTwoTestsofEnglishasaForeignLanguage.Cambridge:CambridgeUniversityPress.

References

217

Ball,S.(2015).Whatispolicy?21yearslater:reflectionsonthepossibilitiesofpolicyresearch.Discourse:StudiesintheCulturalPoliticsofEducation,36(3),306–313.

Ball,S.,Maguire,M.,&Braun,A.(2012).HowSchoolsDoPolicy:PolicyEnactmentsinSecondarySchools.NewYork:Routledge.

Bärenfänger,O.,&Tschirner,E.(2012).AssessingEvidenceofValidityofAssigningCEFRRatingstotheACTFLOralProficiencyInterview(OPI)andtheOralProficiencyInterviewbycomputer(OPIc).Leipzig:InstituteforTestResearchandTestDevelopment.

Barkaoui,K.(2011).EffectsofmarkingmethodandraterexperienceonESLessayscoresandraterperformance.AssessmentinEducation:Principles,Policy&Practice,18(3),279–93.

Barkaoui,K.(2014).MultifacetedRaschAnalysisforTestEvaluation.InA.Kunnan(Ed.)TheCompaniontoLanguageAssessment(pp.1301–1322.).NewJersey:JohnWiley&Sons,Inc.

Baumeister,R.F.,&Leary,M.R.(1995).Theneedtobelong:desireforinterpersonalattachmentsasafundamentalhumanmotivation.PsychologicalBulletin,117(3),497–529.

Baynham,M.(2006).Performingself,familyandcommunityinMoroccannarrativesofmigrationandsettlement.InA.DaFina,D.Schiffrin,&M.Bamberg(Eds.),DiscourseandIdentity(pp.376–397).Cambridge:CambridgeUniversityPress.

Baztán,A.(2008).Laevaluaciónoral:unaequivalenciaentrelasguidelinesdeACTFLyalgunasescalasdelMCER.Granada:UniversidaddeGranada.

BeleidscelDiversiteitenGender.(2016).DiversiteitaandeUGent:deinstroomvankansengroepenincijfers.GhentUniversity,unpublishedpolicydocument.

Belzile,J.,&Öberg,G.(2012).Wheretobegin?Grapplingwithhowtouseparticipantinteractioninfocusgroupdesign.QualitativeResearch,12(4),459–472.

Bérešová,J.,Breton,G.,Noijons,J.,&Szabó,G.(2011).RelatinglanguageexaminationstotheCommonEuropeanFrameworkofReferenceforLanguages:Learning,teaching,assessment(CEFR).HighlightsfromtheManual.Strasbourg:CouncilofEuropePublishing.

Block,D.(2007).TheRiseofIdentityinSLAResearch,PostFirthandWagner(1997).TheModernLanguageJournal,91,863–876.

Blommaert,J.(2011).Thelonglanguage-ideologicaldebateinBelgium.JournalofMulticulturalDiscourses,6(3),241–256.

Blommaert,J.,&VanAvermaet,P.(2008).Taal,onderwijsendesamenleving.Deklooftussenbeleidenrealiteit.Berchem:Epo.

Bond,T.,&Fox,C.(2007).ApplyingtheRaschModel:FundamentalMeasurementintheHumanSciences.Mahwah,NJ:LawrenceErlbaum.

References

218

Borg,S.(2006).TeacherCognitionandLanguageEducation:ResearchandPractice.London:Continuum.

Borsboom,D.,&Markus,K.A.(2013).TruthandEvidenceinValidityTheory.JournalofEducationalMeasurement,50(1),110–114.

Borsboom,D.,Mellenbergh,G.,&vanHeerden,J.(2004).Theconceptofvalidity.PsychologicalReview,111,1061–1071.

Bouma,K.(2016,August26).MeerdandehelftvandestudiesvollediginhetEngels.DeVolkskrant.RetrievedonNovember14,2016,fromwww.volkskrant.nl.

Bourdieu,P.(1991).LanguageandSymbolicPower.Cambridge,Mass:HarvardUniversityPress.

Bovens,M.,’tHart,P.,&Kuipers,S.(2006).Thepoliticsofpolicyevaluation.InM.Moran,M.Rein,&R.E.Goodin(Eds.),TheOxfordhandbookofpublicpolicy(pp.319–336).Oxford:OxfordUniversityPress.

Braine,G.(2002).Academicliteracyandthenonnativespeakergraduatestudent.JournalofEnglishforAcademicPurposes,1(1),59–68.

Buckley,K.,Winkel,R.,&Leary,M.(2004).Reactionstoacceptanceandrejection:Effectsoflevelandsequenceofrelationalevaluation.JournalofExperimentalSocialPsychology,40(1),14–28.

Byrnes,H.(2007).Perspectives.TheModernLanguageJournal.91:641–5.Byrnes,H.,Maxim,H.,&Norris,J.(2010).RealizingAdvancedForeign

LanguageWriting.TheModernLanguageJournal,94,1–202.Cai,H.(2013).PartialdictationasameasureofEFLlisteningproficiency:

Evidencefromconfirmatoryfactoranalysis.LanguageTesting,30(2),177–199.Carlsen,C.H.(2017,inpress).TheadequacyoftheB2-levelasuniversityentrance

requirement.LanguageAssessmentQuarterly.Carlsen,C.H.(2014).HowvalidistheCEFRasaconstructforlanguagetests?

PresentedattheALTE5thInternationalConference,Paris,France,10-11April2014.

Cave,N.(2004).AbattoirBlues.AbattoirBlues/TheLyreofOrpheus.London:MuteRecords.

Chan,S.,Inoue,C.,&Taylor,L.(2015).Developingrubricstoassessthereading-into-writingskills:Acasestudy.AssessingWriting,26,20–37.

Chirkov,V.,Vansteenkiste,M.,Tao,R.,&Lynch,M.(2007).Theroleofself-determinedmotivationandgoalsforstudyabroadintheadaptationofinternationalstudents.InternationalJournalofInterculturalRelations,31,199–222.

Cho,Y.,&Bridgeman,B.(2012).RelationshipofTOEFLiBTscorestoacademicperformance:SomeevidencefromAmericanuniversities.LanguageTesting,29(3),421–442.

Clapham,C.(2000).Assessmentforacademicpurposes:wherenext?System,28(4),511–521.

References

219

CNaVT.(2013).Jaarverslag01/09/2011-31/08/2012.CNaVT/NederlandseTaalunie,unpublishedpolicydocument.

CNaVT.(2014).STRTValidityArgument.CNaVT/NederlandseTaalunie,unpublishedpolicydocument.

CNaVT.(2016a).EducatiefStartbekwaam.RetrievedOctober25,2016,fromwww.cnavt.org

CNaVT.(2016b).Jaarverslag2015.CNaVT/NederlandseTaalunie,unpublishedpolicydocument.

Cohen,J.(1988).Statisticalpoweranalysisforthebehavioralsciences.Hillsdale,NJ:LawrenceEarlbaumAssociates.

Collentine,J.,&Freed,B.(2004).Learningcontextanditseffectsonsecondlanguageacquisition.StudiesinSecondLanguageAcquisition,26,153–171.

Connolly,T.,Arkes,H.,&Hammond,K.(1999).JudgmentandDecisionMaking:AnInterdisciplinaryReader.Cambridge:CambridgeUniversityPress.

CouncilofEurope.(2001).CommonEuropeanFrameworkofReferenceforLanguages:Learning,Teaching,Assessment.Strasbourg:CouncilofEurope.

Creswell,J.(2015).Aconciseintroductiontomixedmethodsresearch.LosAngeles:Sage.

Cumming,A.(2013).AssessingIntegratedWritingTasksforAcademicPurposes:PromisesandPerils.LanguageAssessmentQuarterly,10(1),1–8.

Curtis,A.(2015).DiscussantattheILTA-AAALjointevent:RevisitingtheInterfacesbetweenSLAandLanguageAssessmentResearch.Toronto,Canada,21-24March2015.

Davies,A.(1984).ValidatingthreetestsofEnglishlanguageproficiency.LanguageTesting,1(1),50–69.

Davies,A.(2010).Testfairness:aresponse.LanguageTesting,27(2),171–176.DeBeer,J.,Smith,U.,&Jansen,C.(2009).“Situated”inaseparatedcampus–

Students’senseofbelongingandacademicperformance:Acasestudyoftheexperiencesofstudentsduringahighereducationmerger.EducationasChange,13(1),167–194.

DeBruyn,K.(2011).Dewetvandesterksten?UniversiteitGent:BeleidscelDiversiteitenGender.

DeCosta,P.I.(2010).LanguageideologiesandstandardEnglishlanguagepolicyinSingapore:responsesofa“designerimmigrant”student.LanguagePolicy,9(3),217–239.

DeGeest,A.,Steemans,S.,&Verguts,C.(2015,March).ITNAenITACE:tweehigh-stakestaaltoetsen.Presentedat50JaarILT,Leuven.

DeJong,J.(2013).HowtosavetheCommonEuropeanFramework.PresentedatLanguageTestinginEurope:TimeforanewFramework?Antwerp.

DeJong,J.(2014).Standards&Scaling.PresentedatLTRC,Amsterdam.

References

220

DeJong,J.,Becker,K.,Bolt,D,&Goodman,J.(2014).AligningPTEAcademicTestScorestotheCommonEuropeanFrameworkofReferenceforLanguages.RetrievedonJuly8,2015,fromhttp://pearsonpte.com.

deLarios,J.R.,Marín,J.,&Murphy,L.(2001).ATemporalAnalysisofFormulationProcessesinL1andL2Writing.LanguageLearning,51(3),497–538.

DeStandaard.(2013).Selecteerstudentennaeerstesemester.RetrievedonJuly3,2014,fromwww.standaard.be.

DeWachter,L.&Heeren,J.(2011).Taalvaardigaandestart.EenbehoefteanalyserondtaalproblemenenremediëringvaneerstejaarsstudentenaandeKULeuven.Leuven:ILT.

DeWachter,L.,Heeren,J.,Marx,S.,&Huyghe,S.(2013).Taal:noodzakelijke,maarnietenigevoorwaardetotstudiesucces.CorrelatietussenresultatenvaneentaalvaardigheidstoetsenslaagcijfersbijeerstejaarsstudentenaandeKULeuven.LevendeTalenTijdschrift,14(4),28–36.

DeWit,K.,VanPetegem,P.,&DeMaeyer,S.(2000).GelijkekanseninhetVlaamseonderwijs:hetbeleidinzakekansengelijkheid.Leuven/Apeldoorn:Garant.

Dehandschutter,W.(2016).Eenappomanoniemdeprofessorwatmeeruitlegtevragen.DeStandaard.RetrievedonMarch6,2016,fromwww.standaard.be.

DepartementOnderwijsenVorming.(2015).Taalverslagacademiejaar2013-2014.DepartementOnderwijsenVorming.AfdelingHogerOnderwijsenVolwassenenonderwijs.RetrievedfromonJanuary4,2016,fromwww.vlaanderen.be.

DepartementOnderwijsenVorming.(2016).Screeningniveauonderwijstaal,taaltrajectentaalbadinhetgewoonlageronderwijs.BaO/2014/01.RetrievedonMay24th,2016,fromhttp://data-onderwijs.vlaanderen.be.

Dewey,D.P.,Bown,J.,Baker,W.,Martinsen,R.A.,Gold,C.,&Eggett,D.(2014).LanguageUseinSixStudyAbroadPrograms:AnExploratoryAnalysisofPossiblePredictors.LanguageLearning,64(1),36–71.

Dey,I.(1993).QualitativeDataAnalysis.London:Routledge.Deygers,B.,&Luyten,L.(2012).VerslagITNA/PTHO---onderzoeksoverleg2011-

2012.Unpublishedpolicydocument.Deygers,B.,DeWachter,L.,VanGorp,K.,&Joos,S.(2013).Examiningthe

concurrentvalidityofatask-basedandanindirectacademiclanguagetest.PresentedatTBLT,Banff,Canada.

Deygers,B.,VanGorp,K.,Luyten,L.,&Joos,S.(2013).Ratingscaledesign:acomparativestudyoftwoanalyticratingscalesinatask-basedtest.InE.Galaczi&C.Weir(Eds.),ExploringLanguageFrameworks.ProceedingsfromtheALTEKrakówConference.(Vol.36,pp.273–289).Cambridge:CambridgeUniversityPress.

References

221

diGennaro,K.(2009).InvestigatingdifferencesinthewritingperformanceofinternationalandGeneration1.5students.LanguageTesting,26(4),533–559.

diGennaro,K.(2013).Howdifferentarethey?AcomparisonofGeneration1.5andinternationalL2learners’writingability.AssessingWriting,18(2),154–172.

diGennaro,K.(2016).Searchingfordifferencesanddiscoveringsimilarities:Whyinternationalandresidentsecond-languagelearners’grammaticalerrorscannotserveasaproxyforplacementintowritingcourses.AssessingWriting,29,1–14.

Díaz-campos,M.(2004).ContextofLearningintheAcquisitionofSpanishSecondLanguagePhonology.StudiesinSecondLanguageAcquisition,26(2),249–273.

Duff,P.A.(2002).TheDiscursiveCo-constructionofKnowledge,Identity,andDifference:AnEthnographyofCommunicationintheHighSchoolMainstream.AppliedLinguistics,23(3),289–322.

Dworkin,R.(2003).Equality,LuckandHierarchy.Philosophy&PublicAffairs,31(2),190–198.

Dworkin,R.(2013).JusticeforHedgehogs(Reprintedition).Cambridge,Mass:BelknapPress.

EACEA(2010).FocusonHigherEducationinEurope.Brussels:Education,AudiovisualandCultureExecutiveAgency.

EALTA.(2000).EALTAGuidelinesforgoodpracticeinlanguagetestingandassessment.RetrievedonApril16,2014,fromwww.ealta.eu.org.

EEA.(2001).Reportingonenvironmentalmeasures:Arewebeingeffective?Copenhagen:EuropeanEnvironmentAgency.

Elder,C.,&O’Loughlin,K.(2003).InvestigatingtheRelationshipbetweenIntensiveEnglishLanguageStudyandBandScoreGainonIELTS.IELTSResearchReports,4,207–254.

Ellis,R.(2003).Task-basedlanguagelearningandteaching.Oxford:OxfordUniversityPress.

Engle,L.,&Engle,J.(2003).Studyabroadlevels:Towardaclassificationofprogramtypes.Frontiers:TheInterdisciplinaryJournalofStudyAbroad,9,1-20.

ETS.2010.LinkingTOEFLiBTTMScorestoIELTS®Scores—AResearchReport.Princeton:EducationalTestingService.

Field,A,Miles,J.,&Field,Z.(2012).DiscoveringStatisticsUsingR.London:Sage.Field,J.(2011).Intothemindoftheacademiclistener.JournalofEnglishfor

AcademicPurposes,10(2),102–112.Figueras,N.(2012).TheimpactoftheCEFR.ELTJournal,66(4),477–485.Figueras,N.,North,B.,Takala,S.,VanAvermaet,P.,Verhelst,N.(2009).Relating

LanguageExaminationstotheCommonEuropeanFrameworkofReferenceforLanguages:Learning,Teaching,Assessment(CEFR).AManual.Stasbourg:CouncilofEurope.

References

222

Figueras,N.,North,B.,Takala,S.,Verhelst,N.,&VanAvermaet,P.(2005).RelatingexaminationstotheCommonEuropeanFramework:amanual.LanguageTesting,22(3),261–279.

Fischer,F.(2007).Deliberativepolicyanalysisaspracticalreason:integratingempiricalandnormativearguments.InF.Fischer&G.J.Miller(Eds.),HandbookofPublicPolicyAnalysis:Theory,Politics,andMethods(pp.223–236).BocaRaton:CRCPress.

Fløttum,K.,Gedde-Dahl,T.,&Kinn,T.(2006).AcademicVoices:AcrossLanguagesandDisciplines.Amsterdam:JohnBenjaminsPublishing.

Foster,P.,Tonkyn,A.,&Wigglesworth,G.(2000).Measuringspokenlanguage:aunitforallreasons.AppliedLinguistics,21(3),354–375.

Foucault,M.(1977).Disciplineandpunish.Thebirthoftheprison.London:Penguin.

Fox,J.(2005).Rethinkingsecondlanguageadmissionrequirements:problemswithlanguage-residencycriteriaandtheneedforlanguageassessmentandsupport.LanguageAssessmentQuarterly,2(2),85–115.

Freeman,M.(2000)Knockingondoors:onconstructingculture.QualitativeInquiry,6,59–369.

Fulcher,G.(1997).AnEnglishLanguagePlacementTest:IssuesinReliabilityandValidity.LanguageTesting14,113–39

Fulcher,G.(2004).DeludedbyArtifices?TheCommonEuropeanFrameworkandHarmonization.LanguageAssessmentQuarterly,1(4),253–266.

Fulcher,G.(2012a).AssessmentLiteracyfortheLanguageClassroom.LanguageAssessmentQuarterly,9(2),113–132.

Fulcher,G.(2012b).Scoringperformancetests.InG.Fulcher&F.Davidson(Eds.),TheRoutledgeHandbookofLanguageTesting(pp.378–392).LondonandNewYork:Routledge.

Fulcher,G.,Davidson,F.,&Kemp,J.(2011).Effectiveratingscaledevelopmentforspeakingtests:Performancedecisiontrees.LanguageTesting,28(1),5–29.

Galaczi,E.,Ffrench,A.,Hubbard,C.&Green,A.(2011).Developingassessmentscalesforlarge-scalespeakingtests:amultiple-methodapproach.AssessmentinEducation:Principles,Policy&Practice,18(3),217–237.

Gass,S.(2003).Inputandinteraction.InC.Doughty&M.Long(Eds.),Handbookofsecondlanguageacquisition(pp.224–250).Oxford:Blackwell.

Gilabert,R.(2005).Evaluatingtheuseofmultiplesourcesandmethodsinneedsanalysis:AcasestudyofjournalistsintheAutonomousCommunityofCatalonia(Spain).InM.Long(Ed.),SecondLanguageNeedsAnalysis(pp.182–200).Cambridge:CambridgeUniversityPress.

Ginther,A.,&Stevens,J.(1998).Languagebackground,ethnicity,andtheinternalconstructvalidityoftheAdvancedPlacementSpanishLanguageExamination.InA.J.Kunnan(Ed.),Validationinlanguageassessment.(pp.169–194).Mahwah,NJ:LawrenceErlbaum.

References

223

Glaser,B.&Strauss,A.(1967).TheDiscoveryofGroundedTheory:StrategiesforQualitativeResearch.NewYork:AldinedeGruyter.

Glorieux,I.,Laurijssen,I.,&Sobczyk.(2015).StudiesuccesinheteerstejaarhogeronderwijsinVlaanderen.Eenanalysevandeimpactvankenmerkenvanstudentenenvanopleidingen.Leuven:SteunpuntSSL.

Gogolin,I.(2002).LinguisticandCulturalDiversityinEurope:AChallengeforEducationalResearchandPractice.EuropeanEducationalResearchJournal,1(1),123–138.

Gomez,P.,Noah,A.,Schedl,M.,Wright,C.,&Yolkut,A.(2007).Proficiencydescriptorsbasedonascale-anchoringstudyofthenewTOEFLiBTreadingtest.LanguageTesting,24(3),417–444.

Goodin,R.E.,Rein,M.,&Moran,M.(2006).Thepublicanditspolicies.InM.Moran,M.Rein,&R.E.Goodin(Eds.),TheOxfordhandbookofpublicpolicy(pp.3–39).Oxford:OxfordUniversityPress.

Goovaerts,M.(2012).Wieoverleeftheteerstebachelorjaarniet?Eenonderzoeknaardrop-outinhethogeronderwijs.Brussel:VlaamsVerbondVanKatholiekeHogescholen.

Gorin,J.(2007).ReconsideringIssuesinValidityTheory.EducationalResearcher,36(8),456–462.

Gorter,D.,&Cenoz,J.(2012).Regionalminorities,educationandlanguagerevitalization.InM.Martin-Jones,A.Blackledge,&A.Creese(Eds.),TheRoutledgeHandbookofMultilingualism(pp.184–198).NewYork:Routledge.

Green,A.(2004).Makingthegrade:scoregainsontheIELTSWritingtest.CambridgeESOLResearchNotes,16,9–13.

Green,A.(2017,inpress).LinkingTestsofEnglishforAcademicPurposestotheCEFR:TheScoreUser’sPerspective.LanguageAssessmentQuarterly.Grin,F.(2000).EvaluatingpolicymeasuresforminoritylanguagesinEurope:

towardseffective,cost-effectiveanddemocraticimplementation.Flensburg:EuropeanCentreforMinorityIssues.

Grin,F.(2003).LanguagepolicyevaluationandtheEuropeancharterforregionalorminoritylanguages.NewYork:PalgraveMacmillan.

Gu,L.(2014).Attheinterfacebetweenlanguagetestingandsecondlanguageacquisition:Languageabilityandcontextoflearning.LanguageTesting,31(1),111–133.

Gu,Q.,&Maley,A.(2008).ChangingPlaces:AStudyofChineseStudentsintheUK.LanguageandInterculturalCommunication,8(4),224–245.

Gysen,S.,&Avermaet,P.V.(2005).IssuesinFunctionalLanguagePerformanceAssessment:TheCaseoftheCertificateDutchasaForeignLanguage.LanguageAssessmentQuarterly,2(1),51–68.

Hamilton,J.,Lopes,M.,McNamara,T.,&Sheridan,E.(1993).RatingscalesandnativespeakerperformanceonacommunicativelyorientedEAPtest.LanguageTesting,10(3),337–353.

References

224

Harklau,L.,Losey,K.M.,&Siegal,M.(1999).Generation1.5MeetsCollegeComposition:IssuesintheTeachingofWritingToU.S.-EducatedLearnersofESL.Mahwah,N.J:Routledge.

Harsch,C.,&Martin,G.(2012).AdaptingCEF-descriptorsforratingpurposes:Validationbyacombinedratertrainingandscalerevisionapproach.AssessingWriting,17(4),228–250.

Harsch,C.,&Rupp,A.(2011).DesigningandScalingLevel-SpecificWritingTasksinAlignmentWiththeCEFR:ATest-CenteredApproach.LanguageAssessmentQuarterly,8(1),1–33.

Hechanova-Alampay,R.,Beehr,T.A.,Christiansen,N.D.,&VanHorn,R.(2002).AdjustmentandStrainamongDomesticandInternationalStudentSojourners:ALongitudinalStudy.SchoolPsychologyInternational,23(4),458–74.

Herelixka,C.(2013).Academischtaalvaardig:vanstarttotfinishDeinvloedvanmeertaligheidopdeTaalVaST-toets.Leuven,unpublishedmasterthesis.

Hernández,T.A.(2010).PromotingSpeakingProficiencythroughMotivationandInteraction:TheStudyAbroadandClassroomLearningContexts.ForeignLanguageAnnals,43(4),650–670.

Hill,K.,Storch,N.,&Lynch,B.(1999).AComparisonofIELTSandTOEFLasPredictorsofAcademicSuccess(Vol.2).RetrievedonSeptember4,2012,fromwww.ielts.org.

Hirvela,A.(2016).Academicreadingintowriting.InK.Hyland&P.Shaw(Eds.),TheRoutledgeHandbookofEnglishforAcademicPurposes(pp.127–139).LondonandNewYork:Routledge.

Holmes,J.,Marra,M.,&Vine,B.(2011).Leadership,discourse,andethnicity.Oxford:OxfordUniversityPress.

Horii,S.Y.(2015).Secondlanguageacquisitionandlanguageteachereducation.InM.Bigelow&J.Ennser-Kananen(Eds.),TheRoutledgehandbookofeducationallinguistics(pp.313–324).NewYork:Routledge.

Housen,A.,Schoonjans,E.,Janssens,S.,Welcomme,A.,Schoonheere,E.,&Pierrard,M.(2011).ConceptualizingandmeasuringtheimpactofcontextualfactorsininstructedSLA-theroleoflanguageprominence.IRAL:InternationalReviewofAppliedLinguisticsinLanguageTeaching,49(2),83–112.

Howell,D.(1997).Statisticalmethodsforpsychology.Belmont,CA:Duxbury.Howlett,M.,&Giest,S.(2013).Thepolicy-makingprocess.InE.J.Aral,S.Fritzen,

M.Howlett,M.Ramesh,&X.Wu(Eds.),Routledgehandbookofpublicpolicy(pp.17–28).London&NewYork:Routledge.

Huie,K.,&Yahya,N.(2003).LearningtoWriteinthePrimaryGrades:ExperiencesofEnglishLanguageLearnersandMainstreamStudents.TESOLJournal,12(1),25–31.

References

225

Hulstijn,J.(2007).TheShakyGroundBeneaththeCEFR:QuantitativeandQualitativeDimensionsofLanguageProficiency.TheModernLanguageJournal,91(4),663–667.

Hulstijn,J.(2011).LanguageProficiencyinNativeandNonnativeSpeakers:AnAgendaforResearchandSuggestionsforSecond-LanguageAssessment.LanguageAssessmentQuarterly,8(3),229–249.

Hulstijn,J.(2014).TheCommonEuropeanFrameworkofReferenceforLanguages.Achallengeforappliedlinguistics.InternationalJournalofAppliedLinguistics,165(1),3–18.

Hulstijn,J.(2015).LanguageProficiencyinNativeandNon-nativeSpeakers:Theoryandresearch.Amsterdam ;Philadelphia:JohnBenjaminsPublishingCompany.

Hume,D.(1978,1738),ATreatiseofHumanNature.Oxford:ClarendonPress.Hyland,K.(2016).GeneralandspecificEAP.InK.Hyland&P.Shaw(Eds.),The

RoutledgeHandbookofEnglishforAcademicPurposes(pp.17–30).LondonandNewYork:Routledge.

Hyland,K.,&Hamp-Lyons,L.(2002).EAP:issuesanddirections.JournalofEnglishforAcademicPurposes,1(1),1–12.

ILTA.(2000).ILTACodeofEthics.Retrievedfromwww.iltaonline.com,8March2014.

ILTA.(2007).GuidelinesforPractice.InternationalLanguageTestingAssociation.RetrievedonApril28,2016,fromwww.iltaonline.com.

InteruniversitairTestingConsortium.(2015).InteruniversitaireTaaltestNederlandsvoorAnderstaligen.Zelfevaluatierapport.Unpublishedpolicydocument.

ITNA.(2016).Testprincipes.RetrievedOctober25,2016,fromwww.itna.be.Jann,W.,&Fischer,F.(2007).Deliberativepolicyanalysisaspracticalreason:

integratingempiricalandnormativearguments.InF.Fischer&G.J.Miller(Eds.),HandbookofPublicPolicyAnalysis:Theory,Politics,andMethods(pp.223–236).BocaRaton:CRCPress.

Jann,W.,&Wegrich,K.(2007).TheoriesofthePolicyCycle.InF.Fischer&G.J.Miller(Eds.),HandbookofPublicPolicyAnalysis:Theory,Politics,andMethods(pp.43–62).BocaRaton:CRCPress.

Johnson,R.B.,Onwuegbuzie,A.J.,&Turner,L.A.(2007).Towardadefinitionofmixedmethodsresearch.JournalofMixedMethodsResearch,1(2),112–133.

Jordens,K.(2016).Turkishisnotforlearning,Miss.Valorizinglinguisticdiversityinprimaryeducation.Leuven:KULeuven.

Juan-Garau,M.,Salazar-Noguera,J.,&Prieto-Arranz,J.I.(2014).L2Englishlearners’lexico-grammaticalandmotivationaldevelopmentathomeandabroad.InC.Pérez-Vidal(Ed.),LanguageAcquisitioninStudyAbroadandFormalInstructionContexts(pp.235–259).Amsterdam:JohnBenjamins.

References

226

Kahneman,D.(2011).Thinking,fastandslow.NewYork:Farrar,Straus&Giroux.Kaiser,H.F.(1974).Anindexoffactorialsimplicity.Psychometrika,39(1),31–36.Kane,M.T.(2001).CurrentConcernsinValidityTheory.JournalofEducational

Measurement,38(4),319–342.Kane,M.(2010).Validityandfairness.LanguageTesting,27(2),177–182.Kane,M.T.(2012).Articulatingavalidityargument.InG.Fulcher&F.Davidson

(Eds.),TheRoutledgeHandbookofLanguageTesting(pp.34–48).LondonandNewYork:Routledge.

Kane,M.(2013).ValidatingtheInterpretationsandUsesofTestScores.JournalofEducationalMeasurement,50(1),1–73.

Kane,M.,Kane,J.,&Clauser,B.(2017).Avalidationframeworkforcredentialingtests.InC.Buckendahl&S.Davis-Becker(Eds.),TestingintheProfessions :CredentialingPolicesandPractice(pp.20–41).Routledge.

Keck,C.(2006).Theuseofparaphraseinsummarywriting:AcomparisonofL1andL2writers.JournalofSecondLanguageWriting,15(4),261–278.

Keck,C.(2014).Copying,paraphrasing,andacademicwritingdevelopment:Are-examinationofL1andL2summarizationpractices.JournalofSecondLanguageWriting,25,4–22.

Kerstjens,M.,&Nery,C.(2000).PredictivevalidityintheIELTStest:AstudyoftherelationshipbetweenIELTSscoresandstudents’subsequentacademicperformance.IELTSResearchReports,3,85–108.

Khalifa,H.,&ffrench,A.(2009).AligningCambridgeESOLexaminationstotheCEFR:issuesandpractice.CambridgeResearchNotes37,10-15.

Khan,K.,&McNamara,T.(2017,inpress).Citizenship,immigrationlaws,andlanguage.InS.Canagarajah(Ed.),TheRoutledgeHandbookofMigrationandLanguage(pp.451–467).London&NewYork:Routledge.

Kinginger,C.(2004).Alicedoesn’tlivehereanymore:Foreignlanguagelearningandidentityconstruction.InA.Pavlenko&A.Blackledge(Eds.),Negotiationofidentitiesinmultilingualcontexts(pp.219–242).Clevedon:MultilingualMatters.

Kinginger,C.(2008).LanguageLearninginStudyAbroad:CaseStudiesofAmericansinFrance.TheModernLanguageJournal,92,1–124.

Kinginger,C.(2010).Americanstudentsabroad:Negotiationofdifference?LanguageTeaching,43(2),216–227.

Knoch,U.,Rouhshad,A.,&Storch,N.(2014).DoesthewritingofundergraduateESLstudentsdevelopafteroneyearofstudyinanEnglish-mediumuniversity?AssessingWriting,21,1–17.

Knoch,U.,Rouhshad,A.,Oon,S.P.,&Storch,N.(2015).WhathappenstoESLstudents’writingafterthreeyearsofstudyatanEnglishmediumuniversity?JournalofSecondLanguageWriting,28,39–52.

Koda,K.(1993).Task-InducedVariabilityinFLComposition:Language-SpecificPerspectives.ForeignLanguageAnnals,26(3),332–346.

References

227

Kormos,J.,Csizér,K.,&Iwaniec,J.(2014).Amixed-methodstudyoflanguage-learningmotivationandinterculturalcontactofinternationalstudents.JournalofMultilingualandMulticulturalDevelopment,35(2),151–166.

Krashen,S.(1985).TheInputHypothesis:IssuesandImplications.London,NewYork:Longman.

Krumm,H.-J.(2007).ProfilesInsteadofLevels:TheCEFRandIts(Ab)UsesintheContextofMigration.TheModernLanguageJournal,91(4),667–669.KULeuven.(2016,April28).Onderwijs-enexamenreglement2016-2017.

RetrievedSeptember30,2016,fromwww.kuleuven.be.Kunnan,A.(2000).Fairnessandjusticeforall.InA.J.Kunnan(Ed.),Fairnessand

validationinlanguageassessment(pp.1–14).Cambridge:CambridgeUniversityPress.

Kunnan,A.(2004).Testfairness.InM.Milanovic&C.Weir(Eds.),Europeanlanguagetestinginaglobalcontext(pp.27–48).Cambridge:CambridgeUniversityPress.

Kunnan,A.(2007).Introduction:Testfairness,testbiasandDIF.LanguageAssessmentQuarterly,4(2),109–112.

Kunnan,A.(2010).TestfairnessandToulmin’sargumentstructure.LanguageTesting,27(2),183–189.

Lado,R.(1961).Languagetesting.London:Longman.Landis,J.&Koch,G.(1977).Themeasurementofobserveragreementfor

categoricaldata.Biometrics,33(1),159–174.Lantolf,J.P.,&Genung,P.B.(2003).“I’dratherswitchthanfight”:Anactivity-

theoreticstudyofpower,success,andfailureinaforeignlanguageclassroom.InC.Kramsch(Ed.),LanguageAcquisitionandLanguageSocialization:EcologicalPerspectives(pp.175–197).London&NewYork:BloomsburyAcademic.

Lave,J.,&Wenger,E.(1992).Situatedlearning:Legitimateperipheralparticipation.NewYork:CambridgeUniversityPress.

Lee,E.(2008).The“other(ing)”costsofESL:ACanadiancasestudy.JournalofAsianPacificCommunication,18(1),91–108.

Lee,M.Y.P.(2003).DiscoursestructureandrhetoricofEnglishnarratives:DifferencesbetweennativeEnglishandChinesenon-nativeEnglishwriters.Text,23(3),347.

Lee,Y.,&Greene,J.(2007).ThePredictiveValidityofanESLPlacementTestAMixedMethodsApproach.JournalofMixedMethodsResearch,1(4),366–389.

Lee,Y.,&Greene,J.(2007).ThePredictiveValidityofanESLPlacementTestAMixedMethodsApproach.JournalofMixedMethodsResearch,1(4),366–389.

Leki,I.,Cumming,A.,&Silva,T.(2008).ASynthesisofResearchonSecondLanguageWritinginEnglish.NewYork:Routledge.

References

228

Leliaert,S.(2011).België,hetbeloofdestudieland?RetrievedJune15,2016,fromhttp://www.dewereldmorgen.be.

Leung,C.,Harris,R.,&Rampton,B.(2004).Livingwithineleganceinqualitativeresearchontask-basedlearning.InK.Toohey&B.Norton(Eds.),Criticalpedagogiesandlanguagelearning(pp.242–267).Cambridge:CambridgeUniversityPress.

Lievens,S.(2016).DiversiteitaandeUGent:deinstroomvankansengroepenincijfers.GhentUniversity,unpublishedpolicydocument.

Linacre,J.(2012).Auser’sguidetoFACETSRasch-modelcomputerprograms.Retrieved,November20,2013,fromwww.winsteps.com.

Linacre,J.(2015).FACETS(Version3.71.4).Beaverton,Oregon:Winsteps.com.Lindridge,A.(2015).ConstructEquivalence.InC.Cooper(Ed.),Wiley

EncyclopediaofManagement9,1–3.NewJersey:Wiley.Linton,A.(2009).LanguagepoliticsandpolicyintheUnitedStates:implications

fortheimmigrationdebate.InternationalJournaloftheSociologyofLanguage,(199),9–37.

Lissitz,R.,&Samuelsen,K.(2007).ASuggestedChangeinTerminologyandEmphasisRegardingValidityandEducation.EducationalResearcher,36(8),437–448.

Little,D.(2007).TheCommonEuropeanFrameworkofReferenceforLanguages:PerspectivesontheMakingofSupranationalLanguageEducationPolicy.TheModernLanguageJournal,91(4),645–655.

Llanes,À.,Tragant,E.,&Serrano,R.(2012).Theroleofindividualdifferencesinastudyabroadexperience:thecaseofErasmusstudents.InternationalJournalofMultilingualism,9(3),318–342.

Long,M.(2005).Methodologicalissuesinlearnerneedsanalysis.InM.Long(Ed.),SecondLanguageNeedsAnalysis(pp.19–79).Cambridge:CambridgeUniversityPress.

Lumley,T.(2002).Assessmentcriteriainalarge-scalewritingtest:whatdotheyreallymeantotheraters?LanguageTesting,19(3),246–276.

Lynch,T.(2011).Academiclisteninginthe21stcentury:Reviewingadecadeofresearch.JournalofEnglishforAcademicPurposes,10(2),79–88.

MacDonald,G.,&Leary,M.R.(2005).Whydoessocialexclusionhurt?Therelationshipbetweensocialandphysicalpain.PsychologicalBulletin,131(2),202–223.

Maes,C.(2016).HetCNaVTvernieuwt.Retrievedon2November,2016,fromwww.tijdschriftles.nl.

Maestas,R.,Vaquera,G.S.,&Zehr,L.M.(2007).FactorsImpactingSenseofBelongingataHispanic-ServingInstitution.JournalofHispanicHigherEducation,6(3),237–256.

Magnan,S.,&Back,M.(2007).SocialInteractionandLinguisticGainDuringStudyAbroad.ForeignLanguageAnnals,40(1),43–61.

References

229

Malone,M.(2013).Theessentialsofassessmentliteracy:Contrastsbetweentestersandusers.LanguageTesting,30(3),329–344.

Manderson,L.,Bennett,E.,&Andajani-Sutjahjo,S.(2006).TheSocialDynamicsoftheInterview:Age,Class,andGender.QualitativeHealthResearch,16(10),1317–1334.

Manning,C.,Raghavan,P.,&Schütze,H.(2009).AnIntroductiontoInformationRetrieval.Cambridge:CambridgeUniversityPress.

Martyniuk,W.(2010).StudiesinLanguageTesting33:AligningTestswiththeCEFR.ReflectionsonusingtheCouncilofEurope’sdraftManual.Cambridge:CambridgeUniversityPress.

McNamara,T.(1996).Measuringsecondlanguageperformance.London:Longman.

McNamara,T.(2001).Languageassessmentassocialpractice:challengesforresearch.LanguageTesting,18(4),333–349.

McNamara,T.(2007).TheCEFRinEuropeandBeyond.Paperpresentedatthe4thAnnualEALTAConference,Sitges,Spain.

McNamara,T.(2012).LanguageAssessmentsasShibboleths:APoststructuralistPerspective.AppliedLinguistics,33(5),564–581.

McNamara,T.,&Ryan,K.(2011).FairnessVersusJusticeinLanguageTesting:ThePlaceofEnglishLiteracyintheAustralianCitizenshipTest.LanguageAssessmentQuarterly,8(2),161–178.

Messick,S.(1989).Validity.InR.Linn(Ed.).EducationalMeasurement,(pp.13–103).Washington,DC:AmericanCouncilonEducation/Macmillan.

Miles,M.&HubermanA.(1994).QualitativeDataAnalysis.BaverlyHills:Sage.Miles,M.B.,Huberman,A.M.,&Saldaña,J.(2013).QualitativeDataAnalysis:A

MethodsSourcebook.ThousandOaks,CA:Sage.Morita,N.(2004).NegotiatingParticipationandIdentityinSecondLanguage

AcademicCommunities.TESOLQuarterly,38(4),573–603.Moss,P.(2007).ReconstructingValidity.EducationalResearcher,36(8),470–476.Nagel,S.(2002).Handbookofpublicpolicyevaluation.ThousandOaks,CA:Sage.NATO.(2014).StandardizationAgreementSTANAG6001LanguageProficiency

Levels.Brussels:BureauforInternationalLanguageCoordination.NederlandseTaalunie.(2013).Aankondigingvaneenopdracht-Diensten.Den

Haag:NederlandseTaalunie.Negishi,M.,Takada,T.&Tono,Y.(2013)Aprogressreportonthedevelopment

oftheCEFR-J.InE.Galaczi&C.Weir(Eds.),ExploringLanguageFrameworks.ProceedingsfromtheALTEKrakówConference.Cambridge:CambridgeUniversityPress.

Norris,J.M.(2002).Interpretations,intendedusesanddesignsintask-basedlanguageassessment.LanguageTesting,19(4),337–346.

Norris,J.M.(2008).ValidityEvaluationinLanguageAssessment.NewYork:PeterLang.

References

230

Norris,J.M.(2015).StatisticalSignificanceTestinginSecondLanguageResearch:BasicProblemsandSuggestionsforReform.LanguageLearning,65,97–126.

Norris,J.M.(2016).ReframingtheSLA-AssessmentInterface:‘Constructive’DeliberationsattheNexusofInterpretations,Contexts,andConsequences.PaperpresentedattheLanguageTestingResearchColloquium,Palermo,Italy.

North,B.(2007).TheCEFRIllustrativeDescriptorScales.TheModernLanguageJournal,91(4),656-659.

North,B.(2014a).EnglishProfileStudies.TheCEFRinPractice.Cambridge:CambridgeUniversityPress.

North,B.(2014b).PuttingtheCommonEuropeanFrameworkofReferencetogooduse.LanguageTeaching,47(2),228–249.

North,B.(2016).UseandmisueoftheCEFRinteachingandassessment.PresentedattheALTE48thConferenceDay,Stockholm.

Norton,B.(2013).IdentityandLanguageLearning:ExtendingtheConversation.Bristol:MultilingualMatters.

Norton,B.,&Toohey,K.(2011).Identity,languagelearning,andsocialchange.LanguageTeaching,44(04),412–446.

Nussbaum,M.(2002).CapabilitiesandSocialJustice.InternationalStudiesReview,4(2),123–135.

O’Cathain,A.,Murphy,E.,&Nicholl,J.(2008).Multidisciplinary,Interdisciplinary,orDysfunctional?TeamWorkinginMixed-MethodsResearch.QualitativeHealthResearch,18(11),1574–1585.

O’Loughlin,K.(2001).TheEquivalenceofDirectandSemi-DirectSpeakingTests.Cambridge:CambridgeUniversityPress.

O’Loughlin,K.(2011).TheInterpretationandUseofProficiencyTestScoresinUniversitySelection:HowValidandEthicalAreThey?LanguageAssessmentQuarterly,8(2),146–160.

O’Loughlin,K.(2013).Developingtheassessmentliteracyofuniversityproficiencytestusers.LanguageTesting,30(3),363–380.

O’Sullivan,B.(2016).AStorytoTell,aLessontoLearn:TheTestingIndustryandValidation.PresentedattheALTE48thConference.Stockholm.

O’Sullivan,B.,&Weir,C(2011).TestingandValidation,inB.O’Sullivan(ed.)LanguageTesting:TheoryandPractice(pp.13–32).Oxford:Palgrave.

O’Toole,L.(2000).ResearchonPolicyImplementation:AssessmentandProspects.JournalofPublicAdministrationResearch&Theory,10(2),263–288.

Oller,J.(2012).Groundingtheargument-basedframeworkforvalidatingscoreinterpretationsanduses.LanguageTesting,29(1),29–36.

OnderwijsVlaanderen.(2015).Curriculum:Eindtermen,ontwikkelingsdoelen,basiscompetentiesendoelenberoepsgerichtevorming.RetrievedApril11,2016,fromwww.ond.vlaanderen.be.

References

231

Ongenaert,D.(2015).BuitenlandsestudenteninVlaanderen:wiezijnzeenwatdoenze?RetrievedSeptember22,2015,fromhttp://mo.be.

Ortega,L.(2003).SyntacticComplexityMeasuresandtheirRelationshiptoL2Proficiency:AResearchSynthesisofCollege-levelL2Writing.AppliedLinguistics,24(4),492–518.

Ortega,L.(2008).UnderstandingSecondLanguageAcquisition.London:Routledge.

Papageorgiou,S.(2010).Investigatingthedecision-makingprocessofstandardsettingparticipants.LanguageTesting,27(2),261–282.

Papageorgiou,S.,Xi,X.,Morgan,R.,&So,Y.(2015).DevelopingandValidatingBandLevelsandDescriptorsforReportingOverallExamineePerformance.LanguageAssessmentQuarterly,12(2),153–177.

Paradis.(2007).Earlybilingualandmultilingualacquisition.InP.Auer&L.Wei(Eds.),HandbookofMultilingualismandMultilingualCommunication(pp.15–45).NewYork:MoutondeGruyter.

Patton,M.(2002).QualitativeEvaluationandResearchMethods.NewburyPark,CA:Sage.

PellegrinoAveni,V.A.(2005).StudyAbroadandSecondLanguageUse:ConstructingtheSelf.Cambridge:CambridgeUniversityPress.

PérezVidal,C.,&Juan-Garau,M.(2014).Theeffectofcontextandinputconditionsonoralandwrittendevelopment:AStudyAbroadperspective.InC.Pérez-Vidal(Ed.),LanguageAcquisitioninStudyAbroadandFormalInstructionContexts(pp.157–185).Amsterdam:JohnBenjaminsPublishingCompany.

Pérez-Tattam,MuellerGathercole,Yavaz,&Stadthagen-González.(2013).Measuringgrammaticalknowledgeandabilitiesinbilinguals:implicationsforassessmentandtesting.InV.C.M.MuellerGathercole(Ed.),IssuesintheAssessmentofBilinguals(pp.111–129).Bristol:MultilingualMatters.

Pérez-Vidal,C.(2014).LanguageAcquisitioninStudyAbroadandFormalInstructionContexts.Amsterdam:JohnBenjaminsPublishingCompany.

Peters,E.,&VanHoutven,T.(2010).Taalbeleidinhethogeronderwijs:dehypevoorbij?Leuven:Acco.

Phillips,D.(2007).AddingComplexity:PhilosophicalPerspectivesontheRelationshipBetweenEvidenceandPolicy.YearbookoftheNationalSocietyfortheStudyofEducation,106(1),376–402.

Polio,C.(2013).Theacquisitionofsecondlanguagewriting.InS.M.Gass&A.Mackey(Eds.),TheRoutledgeHandbookofSecondLanguageAcquisition(pp.319–334).London;NewYork:Routledge.

Porto,M.(2012).AcademicperspectivesfromArgentina.InM.ByramandL.Parmenter(Eds.),TheCommonEuropeanFrameworkofReference:TheGlobalisationofLanguageEducationPolicy(pp.129–38).Bristol:MultilingualMatters.

References

232

Purpura,J.,E.,Brown,J.D.,&Schoonen,R.(2015).Improvingthevalidityofquantitativemeasuresinappliedlanguageresearch.LanguageLearning,65(1),36-73.

Ranta,L.,&Meckelborg,A.(2013).HowMuchExposuretoEnglishDoInternationalGraduateStudentsReallyGet?MeasuringLanguageUseinaNaturalisticSetting.TheCanadianModernLanguageReview/LaRevueCanadienneDesLanguesVivantes,69(1),1–33.

Rawls,J.(1971).ATheoryofJustice.Cambridge,MA:HarvardUniversityPress.Rawls,J.(2001).JusticeasFairness:ARestatement.Cambridge,MA:Belknap

Press.Reybold,L.,Lammert,J.,&Stribling,S.(2013).Participantselectionasa

consciousresearchmethod:thinkingforwardandthedeliberationof“Emergent”findings.QualitativeResearch,13(6),699–716.

Riazi,M.(2013).ConcurrentandpredictivevalidityofPearsonTestofEnglishAcademic(PTEAcademic).PapersinLanguageTestingandAssessment,2(2),1–27.

Rijlaarsdam,Braaksma,Couzijn,Janssen,Kieft,&vandenBergh.(2005).Psychologyandtheteachingofwritingin8000andsomewords.BritishJournalofEducationalPsychologyMonographSeriesII:PsychologicalAspectsofEducation-CurrentTrends,3,127–153.

Roever,C.,&McNamara,T.(2006).Languagetesting:thesocialdimension.InternationalJournalofAppliedLinguistics,16(2),242–258.

Ross,S.J.(2008).LanguagetestinginAsia:Evolution,innovation,andpolicychallenges.LanguageTesting,25(1),5–13.

Salamoura,A.&Saville,N.(2009).CriterialfeaturesacrosstheCEFRlevels:EvidencefromtheEnglishProfileProgramme.ResearchNotes,37,34–40.

Sanz,C.(2014).ContributionsofstudyabroadresearchtoourunderstandingofSLAprocessesandoutcomes:TheSALAProject,anappraisal.InC.PérezVidal(Ed.),LanguageAcquisitioninStudyAbroadandFormalInstructionContexts(pp.1–17).Amsterdam:JohnBenjaminsPublishing.

Sasayama,S.(2016).Isa“Complex”TaskReallyComplex?ValidatingtheAssumptionofCognitiveTaskComplexity.TheModernLanguageJournal,100(1),231–254.

Schoonen,R.,Gelderen,A.van,Glopper,K.de,Hulstijn,J.,Simis,A.,Snellings,P.,&Stevenson,M.(2003).FirstLanguageandSecondLanguageWriting:TheRoleofLinguisticKnowledge,SpeedofProcessing,andMetacognitiveKnowledge.LanguageLearning,53(1),165–202.

Schwartz,N.,Knäuper,B.,Oyersman,D.,&Stich,C.(2008).Thepsychologyofaskingquestions.InE.deLeeuw,J.Hox,&D.Dillman(Eds.),InternationalHandbookofSurveyMethodology(pp.18–35).NewYork:Routledge.

References

233

Seloni,L.(2012).AcademicliteracysocializationoffirstyeardoctoralstudentsinUS:Amicro-ethnographicperspective.EnglishforSpecificPurposes,31(1),47–59.

Sen,A.(2010).TheIdeaofJustice.London:Penguin.Serrano,R.,Tragant,E.,&Llanes,À.(2012).ALongitudinalAnalysisoftheEffects

ofOneYearAbroad.CanadianModernLanguageReview,68(2),138–163.Shaw,S.,&Imam,H.(2013).AssessmentofInternationalStudentsThroughthe

MediumofEnglish:EnsuringValidityandFairnessinContent-BasedExaminations.LanguageAssessmentQuarterly,10(4),452–475.

Shohamy,E.(1994).TheValidityofDirectVersusSemi-directOralTests.LanguageTesting,11,99–123.

Shohamy,E.(2001).Thepoweroftests:Acriticalperspectiveontheusesoflanguagetests.Harlow,England:Longman.

Shohamy,E.(2006).LanguagePolicy:Hiddenagendasandnewapproaches.London&NewYork:Routledge.

Shohamy,E.(2011).AnengagementwiththeCEFR.Acriticalviewandtime.PaperpresentedattheALTE4thInternationalConference,Krakow,Poland.

Sin,C.(2003).Interviewingin“place”:thesocio-spatialconstructionofinterviewdata.Area,35(3),305–312.

Snow,C.(2010).AcademicLanguageandtheChallengeofReadingforLearningAboutScience.Science,328(5977),450–452.

Song,M.-Y.(2012).Note-takingqualityandperformanceonanL2academiclisteningtest.LanguageTesting,29(1),67–89.

Spolsky,B.(1995).MeasuredWords:TheDevelopmentofObjectiveLanguageTesting.Oxford:OxfordUniversityPress.

Spolsky,B.(2004).LanguagePolicy.Cambridge:CambridgeUniversityPress.Spolsky,B.(2008).Introduction:LanguageTestingat25:Maturityand

responsibility?LanguageTesting,25(3),297–305.Stæhr,L.(2008).Vocabularysizeandtheskillsoflistening,readingandwriting.

LanguageLearningJournal,36(2),139–152.Steemans,S.,&Vlasselaers,A.(2013).B2orNotB2?MappingtheOralPartofthe

ITNAtotheCEFR.InJ.Colpaert,M.Simons,A.Aerts,&M.Oberhofer(Eds.),LanguageTestinginEurope:TimeforaNewFramework?(pp.101–102).Antwerp:UniversityofAntwerp.

Storch,N.(2009).Theimpactofstudyinginasecondlanguage(L2)mediumuniversityonthedevelopmentofL2writing.JournalofSecondLanguageWriting,18(2),103–118.

Stricker,L.J.(2004).TheperformanceofnativespeakersofEnglishandESLspeakersonthecomputer-basedTOEFLandGREGeneralTest.LanguageTesting,21(2),146–173.

References

234

Strobbe,L.(2016).Taalbeleidoftalenbeleid?Deplaatsvanmeertaligheidopschool.InL.VanPraag,S.Sierens,O.Agirdag,P.Lambert,S.Slembrouck,P.VanAvermaet,…M.VanHoutte(Eds.),Haalmeeruitmeertaligheid.Omgaanmettaligediversiteitinhetbasisonderwijs(pp.117–130).Leuven:Acco.

Subtirelu,N.(2014).Alanguageideologicalperspectiveonwillingnesstocommunicate.System,42,120–132.

Swain,M.,&Deters,P.(2007).“New”MainstreamSLATheory:ExpandedandEnriched.TheModernLanguageJournal,91,820–836.

Swender,E.(2010).ATaleofTwoTests.STANAGandCEFR.ComparingtheResultsofside-by-sidetestingofreadingproficiency.PaperpresentedatBILC,Istanbul.

Tannenbaum,R.J.,&Wylie,C.E.(2008).LinkingEnglish-LanguageTestScoresontotheCommonEuropeanFrameworkofReference:AnApplicationofStandard-SettingMethodology.Princeton,NJ:ETS.

Taylor,C.(1992).ThePoliticsofrecognition.InA.Gutmann(Ed.),Multiculturalismand“thepoliticsofrecognition”(pp.25–75).Princeton,N.J:PrincetonUniversityPress.

Taylor,L.(2004).Issuesoftestcomparability.CambridgeResearchNotes,15,2-5.Taylor,L.(2009).Developingassessmentliteracy.AnnualReviewofApplied

Linguistics,29,21–36.Taylor,L.(2013).Communicatingthetheory,practiceandprinciplesoflanguage

testingtoteststakeholders:Somereflections.LanguageTesting,30(3),403–412.

Taylor,L.,&Geranpayeh,A.(2011).Assessinglisteningforacademicpurposes:Definingandoperationalisingthetestconstruct.JournalofEnglishforAcademicPurposes,10(2),89–101.

TheDouglasFirGroup.(2016).ATransdisciplinaryFrameworkforSLAinaMultilingualWorld.TheModernLanguageJournal,100(S1),19–47.

Tillema,M.,Bergh,H.vanden,Rijlaarsdam,G.,&Sanders,T.(2013).QuantifyingthequalitydifferencebetweenL1andL2essays:AratingprocedurewithbilingualratersandL1andL2benchmarkessays.LanguageTesting,30(1),71–97.

Toulmin,S.(2001).ReturntoReason.Cambridge,Mass.:HarvardUniversityPress.

Toulmin,S.(2003).TheUsesofArgument.Cambridge,U.K. ;NewYork:CambridgeUniversityPress.

Truyts,J.,&Torfs,M.(2015,January22).BijNederlandseoverrompelingwordthetingewikkeld.RetrievedSeptember22,2016,fromhttp://deredactie.be.

Tschirner,E.,Bärenfänger,O.,&Wisniewski,K.(2015).AssessingEvidenceofValidityoftheACTFLCEFRListeningandReadingProficiencyTests(LPTandRPT)UsingaStandard-SettingApproach.(TechnicalReport2015-EU-PUB-2).Leipzig:InstituteforTestResearchandTestDevelopment.

References

235

Turner,C.(2014).Mixedmethodsresearch.InA.J.Kunnan(Ed.),ThecompaniontoLanguageAssessment(pp.1-15).NewYorkCity:JohnWiley&Sons.

UNGeneralAssembly.(1948).UniversalDeclarationofHumanRights.UNGeneralAssembly.Retrievedfromhttp://www.ohchr.org.

UniversiteitAntwerpen.(2016).PROCEDUREPROC/ADOND/001.1.RetrievedonApril12,2016,fromwww.uantwerpen.be.

UniversiteitGent.(2013).RegistratievankansengroepenaandeUGent.RetrievedonApril1,2016,fromwww.ugent.be.

UniversiteitGent.(2015).Onderwijs-enexamenreglement2015-2016.Retrievedon12January,2015,fromwww.ugent.be.

UniversiteitGent.(2016).EducationandExaminationCodeacademicyear2016-2017.RetrievedSeptember30,2016,fromwww.ugent.be.

UniversiteitHasselt.(2015)Toelatingsvoorwaarden.Retrievedon29january,2015,fromwww.uhasselt.be.

UniversiteitHasselt.(2016).Taalvoorwaarden.Retrievedon29january,2015,fromwww.uhasselt.be.

UniversityofCambridge,ESOLExaminations.(2011).UsingtheCEFR:PrinciplesofGoodPractice.Cambridge:UniversityofCambridge,ESOLExaminations.

VanAvermaet,P.&Pulinx,R.(2013).LanguagetestingforimmigrationtoEurope.InA.J.Kunnan(Ed.),Thecompaniontolanguageassessment(pp.376–389).Malden,MA,USA:JohnWiley&Sons,Inc.

VanAvermaet,P.,&Gysen,S.(2006).Fromneedstotasks:Languagelearningneedsinatask-basedapproach.InK.vandenBranden(Ed.),Task-BasedLanguageEducation:FromTheorytoPractice(pp.17–46).Cambridge:CambridgeUniversityPress.

VanAvermaet,P.,&Slembrouck,S.(2014).Ineentaalbadverzuipje.RetrievedonMay17,2016,fromwww.standaard.be.

VandenBosch,K.,&Cantillon,B.(2006).Policyimpact.InM.Moran,M.Rein,&R.E.Goodin(Eds.),TheOxfordhandbookofpublicpolicy(pp.296–319).Oxford:OxfordUniversityPress.

VanEk,J.(1975).SystemsDevelopmentinAdultLanguageLearning:TheThresholdLevelinaEuropean-Unit/CreditSystemforModernLanguageLearningbyAdults.Strasbourg:CouncilofEurope.

VanEk,J.&Trim,J.(1991a).Threshold1990:Strasbourg:CouncilofEurope.VanEk,J.&Trim,J.(1991b).Waystage1990.Strasbourg:CouncilofEurope.VanEk,J.&Trim,J.(2001).Vantage.Cambridge:CambridgeUniversityPress.VanGorp,K.,Luyten,L.,DeWachter,L.,&Steemans,S.(2014).Aconcurrent

validitystudyoftwoacademicplacementtests.PresentedattheALTE5thInternationalConference,Paris.

VanHoutvenTine,PetersElke(2010).Dewegnaarmateriaalontwikkelingisgeplaveidmetbehoeftes.In:PetersE.,VanHoutvenT.(Eds.),Taalbeleidinhethogeronderwijs:dehypevoorbij?,(pp.69-85).Leuven:Acco.

References

236

VanHoutven,T.,Peters,E.,&ElMorabit,Z.(2010).Hoestaathetmetdetaalvanstudenten?ExploratievestudienaarbegrijpendlezenensamenvattenbijinstromendestudenteninhetVlaamsehogeronderwijs.LevendeTalenTijdschrift,11(3),29–44.

VanSplunder,F.(2015).Taalstrijd.Overrelatiestussentalenindewereld,Europa,NederlandenVlaanderen.Brussels:ASP.

VanWeijen,D.(2009).Writingprocesses,textquality,andtaskeffects:Empiricalstudiesinfirstandsecondlanguagewriting.Utrecht:LOTDissertationSeries.

VanWeijen,D.,VandenBergh,B.,Rijlaarsdam,G.,&Sanders,T.(2008).Differencesinprocessandprocess-productrelationsinL2writing.ITLInternationalJournalofAppliedLinguistics,156,203–226.

Vanbelle,S.,&Albert,A.(2009).Anoteonthelinearlyweightedkappacoefficientforordinalscales.StatisticalMethodology,6(2),157–163.

Vedung,E.(2013).Sixmodelsofevaluation.InE.J.Aral,S.Fritzen,M.Howlett,M.Ramesh,&X.Wu(Eds.),Routledgehandbookofpublicpolicy(pp.387–400).London&NewYork:Routledge.

Victori,M.(1999).AnanalysisofwritingknowledgeinEFLcomposing:acasestudyoftwoeffectiveandtwolesseffectivewriters.System,27(4),537–555.

VlaamseRegering.(2013).BesluitvandeVlaamseRegeringtotcodificatievandedecretalebepalingenbetreffendehethogeronderwijs,Pub.L.No.B.S.27/02/2014,§Deel2.Hoofdstuk8.RetrievedonDecember11,2016,fromhttp://data-onderwijs.vlaanderen.be.

VrijeUniversiteitBrussel.(2014).Onderwijs-enexamenreglement2014-2015.RetrievedonOctober25,2016,fromhttp://www.vub.ac.be.

Wainer,H.(2011).Uneducatedguesses:Usingevidencetouncovermisguidededucationpolicies.Princeton,N.J:PrincetonUniversityPress.

Wall,D.,Clapham,C.,&Alderson,J.C.(1994).Evaluatingaplacementtest.LanguageTesting,11(3),321–344.

Walton,G.M.,&Cohen,G.L.(2007).AQuestionofBelonging:Race,SocialFit,andAchievement.JournalofPersonalityandSocialPsychology,92(1),82–96.

Wang,S.,Wang,N.,&Hoadley,D.(2007).ConstructEquivalenceofaNationalCertificationExaminationthatUsesDualLanguagesandAudioAssistance.InternationalJournalofTesting,7:255–68.

Wartenbergh,F.,Brukx,D.,vandenBroek,A.,Jacobs,J.,Pass,J.,Hogeling,L.,&vanKlingeren,M.(2009).StudentenmonitorVlaanderen2009.DepartementOnderwijsenVorming.

Weigle,S.C.(2002).AssessingWriting.Cambridge:CambridgeUniversityPress.Weigle,S.C.,&Friginal,E.(2015).Linguisticdimensionsofimpromptutest

essayscomparedwithsuccessfulstudentdisciplinarywriting:Effectsoflanguagebackground,topic,andL2proficiency.JournalofEnglishforAcademicPurposes,18,25–39.

References

237

Weigle,S.C.,&Malone,M.M.(2016).AssessmentofEnglishforacademicpurposes.InK.Hyland&P.Shaw(Eds.),TheRoutledgeHandbookofEnglishforAcademicPurposes(pp.165–177).LondonandNewYork:Routledge.

Weir,C.(2005a).LanguageTestingandValidation.NewYork:Palgrave.Weir,C.(2005b).LimitationsoftheCommonEuropeanFrameworkfor

developingcomparableexaminationsandtests.LanguageTesting,22(3),281–300.

Welkenhuysen-Gybels,J.,&vandeVijver,F.(2001).AComparisonofMethodsfortheEvaluationofConstructEquivalenceinaMultigroupSetting.InProceedingsAmericanStatisticalAssociation(CD-ROM),Atlanta.

Welsh,E.(2002).DealingwithData:UsingNVivointheQualitativeDataAnalysisProcess.Forum:QualitativeSocialResearch,3(2).RetrievedonJuly30,2015,fromhttp://www.qualitative-research.net.

Wetophethogeronderwijsenwetenschappelijkonderzoek.(1992).Pub.L.No.BWBR0005682,§7.2.

Whitehead,A.N.(1925).Scienceandthemodernworld.NewYork:PelicanBooks.Wilson,R.(2006).Policyanalysisaspolicyadvice.InM.Moran,M.Rein,&R.E.

Goodin(Eds.),TheOxfordhandbookofpublicpolicy(pp.152–169).Oxford:OxfordUniversityPress.

Wolfe,E.W.,Song,T.,&Jiao,H.(2016).Featuresofdifficult-to-scoreessays.AssessingWriting,27,1–10.

Wollmann,H.(2007).Policyevaluationandevaluationresearch.InF.Fischer&G.J.Miller(Eds.),HandbookofPublicPolicyAnalysis:Theory,Politics,andMethods(pp.393–402).BocaRaton:CRCPress.

Wu,R.-J.(2013).Nativeandnon-nativestudents’interactionwithatext-basedprompt.AssessingWriting,18(3),202–217.

Xi,X.(2010).Howdowegoaboutinvestigatingtestfairness?LanguageTesting,27(2),147–170.

Xi,X.,Bridgeman,B.&Wendler,C.(2014).TestsofEnglishforAcademicPurposesinUniversityAdmissions.InA.J.Kunnan(Ed.),TheCompaniontoLanguageAssessment(pp.318–33).Malden,MA:JohnWiley&Sons,Inc.

Yu,G.(2013a).FromIntegrativetoIntegratedLanguageAssessment:AreWeThereYet?LanguageAssessmentQuarterly,10(1),110–114.

Yu,G.(2013b).TheUseofSummarizationTasks:SomeLexicalandConceptualAnalyses.LanguageAssessmentQuarterly,10(1),96–109.

Zheng,Y,&DeJong,J.H.A.L.(2011).ResearchNote:EstablishingConstructandConcurrentValidityofPearsonTestofEnglishAcademic.RetrievedonSeptember29,2016,fromhttp://pearsonpte.com.

AcademicoutputrelatedtothisPhD

239

ACADEMICOUTPUTRELATEDTOTHISPHDPeerreviewedjournalarticlesDeygers,B.(2017,accepted).Ayearofhighsandlows.Consideringcontextual

factorstoexplainL2gainsatuniversity.TheModernLanguageJournal.Deygers,B.,VandenBranden,K.,&Peters,E.(2017).Checkingassumed

proficiency:ComparingL1andL2performanceonauniversityentrancetest.AssessingWriting,32,43–56.

Deygers,B.,VandenBranden,K.,VanGorp,K.(2017,inpress).Universityentrancelanguagetests:amatterofjustice.LanguageTesting.

Deygers,B.,VanGorp,K.,&Demeester,T.(2017,inpress).TheB2levelandthedreamofacommonstandard.LanguageAssessmentQuarterly.

Deygers,B.,Zeidler,D.,Vilcu,D.,&CarlsenC.H.(2017,inpress).Oneframeworktounitethemall?TheuseoftheCEFRinEuropeanuniversityentrancepolicies.LanguageAssessmentQuarterly.

Deygers,B.&VanGorp,K.(2015).Determiningthescoringvalidityofaco-constructedCEFR-basedratingscale.LanguageTesting,32(4),521–541.

PeerreviewedbookchaptersDeygers,B.(2017,inpress).Universityentrancelanguagetests:examining

assumedequivalence.InJ.Davis,J.Norris,M.Malone,T.McKay,&YSon(Eds.).UsefulAssessmentAndEvaluationInLanguageEducation.Washington,D.C.:GeorgetownUniversityPress.

PresentationsPlenaryDeygers,B.(2016).Nadetoets.Deervaringenvananderstaligestudentenaan

Vlaamseuniversiteiten.ForumdagTaalbeleidHogerOnderwijs.UniversiteitAntwerpen.

Deygers,B.(December,2015).Aftertheentrancetest:AlongitudinalstudyofL2students’experiencesatuniversity.AdvancedResearchinMultilingualismTalkSeries#9.GeorgetownUniversity,Washington,D.C.

Deygers,B.(2015).Languagetestsforuniversityadmission:amismatchbetweentestandpractice?EALTA,Copenhagen.

Deygers,B.&Zeidler,B.(2015).TheCEFRanduniversityentrancetests–astateofaffairsinEurope.ALTE,Bergen.

AcademicoutputrelatedtothisPhD

240

Deygers,B.&VanGorp,K.(2014).KeepingitReal:Task-Basedlanguage

assessment.ALTE,London.Deygers,B.(2014).Creatingmeaningoutofvagueness?Developingaratingscale

fromgeneraldescriptors.NATO/BILCConference,Bruges.Invitedplenary.Parallel(selection)Deygers,B.(2016).AlongitudinalstudyofL2students’experiencesatuniversity.

LTRC,Palermo.Deygers,B.(2016).Universityentrancelanguagetests:Justiceatthecoreofthe

construct.GURT.GeorgetownUniversity,Washington,D.C.Deygers,B.(2015).Concurrentvalidityinuniversityentrancetests:discrepancies

belowthesurfacecorrelation.ECOLT,Washington,D.C.Deygers,B.(2015).Justiceandvalidityinuniversityentrancetesttasks:Thecase

ofFlanders.TBLT,Leuven.Deygers,B.(2015).Theconcurrentandpredictivevalidityofuniversityentrance

tests.LTRC,Toronto.Deygers,B.&VanGorp;K.(2014).Creatingmeaningoutofvagueness?Usingthe

CEFRasthefoundationforanacademicratingscale.LTRC,Amsterdam.Deygers,B.&Carlsen,C.H.(2014).TheB2levelanditsapplicabilityinuniversity

Placementtests.ALTE5thInternationalConference,Paris.Deygers,B.&VanGorp,K.(2013).Examiningtheconcurrentvalidityofatask-

basedandanindirectacademiclanguagetest.TBLT.Banff.

SummaryinDutch

242

SAMENVATTINGInternationale L2 studenten kunnen zich pas inschrijven aan een Vlaamseuniversiteitwanneerzevoldoenaande taligeeisen.Aanelkeuniversiteit ishetbasisniveau dat van deze studenten verwacht wordt B2 op het ERK (CommonEuropean Framework of Reference for Languages – Council of Europe, 2001).Studenten kunnen op verschillendemanieren bewijzen dat ze voldoen aan detaaleisen.Zekunneneentaaltoetsafleggen,maarzekunnenookdadelijkstartenwanneer ze minstens één jaar in het Nederlandstalige secundair of hogeronderwijssuccesvolhebbenafgerond.

DetweeB2testsdieaanelkeuniversiteitaanvaardworden,zijnITNAenSTRT.Deze testshebbenhetzelfdedoelenhetzelfdeERKniveau,maarkennenenkele substantiële verschillen in operationalisering. ITNA heeft eencomputergestuurde schriftelijke component en bestaat uit gesloten vraagtypesdie vooral beroep doen op receptieve vaardigheden en woordenschat- engrammaticakennis. De schriftelijke component van STRT daarentegen istaakgerichtengeïntegreerd.DemondelingesectiesvanSTRTenITNAlijkenwelsterk op elkaar; ze bestaan uit een presentatie- en een argumentatietaak, enhebben vijf overlappende beoordelingscriteria die gebaseerd zijn op dezelfdeERK-descriptoren.

Ditonderzoeksprojectonderzochtdevoornaamsteassumptiesdieaandebasisliggenvanhettoelatingsbeleid,vanuitdrieperspectievenomnategaanhoeeffectief het toelatingsbeleid van Vlaamse universiteiten is ten aanzien vaninternationaleL2studenten:constructenenniveaus(metendetoetsendezakendiebelangrijkzijnophet juisteniveau),selectie(selecterendetoetsendezelfdekandidaten),entaalevolutienadetoets.Constructen&niveausDe eerste studie onderzocht het toelatingsbeleid voor internationale L2studentenuniversiteitenin28Europeseregio’s.UitdezebevragingbleekdatB2veruit het meest gevraagde toelatingsniveau is, en dat in de meeste regio’sverschillendetestsnaastelkaaraanvaardworden.IndiezinishetVlaamsebeleidrepresentatiefvoorwaterdoorgaansbinnenEuropagebeurt.

Hoofdstuk 2 vergeleek de operationalisering van STRT en ITNAmet dereëletaligeeisenaandeuniversiteit,enonderzochtinhoeverreslagenvoorSTRTenITNAookimpliceertdatmenklaar isvoordetaligeuitdagendievolgen.Destudie combineerde de meningen en ervaringen van 24 universitairemedewerkersen31internationaleL2studenten,vanwieertwintiglongitudinaalwerdengevolgdnadatzeSTRTenITNAafgelegdhadden.UitderesultatenbleekdatdewerkelijketaaleisenaandeVlaamseuniversiteitensomscruciaalafwijken

SummaryinDutch

243

vandeinhoudvanbeidetests;datL2studentendiegeslaagdwarenvoorITNAofSTRT, of beide, lang niet klaar waren voor de receptieve eisen van deacademischewereld;endatviervandezevenstudentendienietgeslaagdwarenvoor STRT of ITNA toch goed presteerden aan de universiteit. Uit deze studiebleektevensdathetB2niveaualstoelatingsniveauonvoldoendegarantiesbiedtdat instromende internationaleL2studentenzullenvoldoenaande taligeeisenvandeuniversiteit.Selectie&equivalentieIntweestudieswerdnagegaanofSTRTenITNAequivalenteB2toetsenkunnenzijn.Deeerstestudiewasgerichtopequivalentiequaniveauenquaconstruct,enhet tweede onderzoek richtte zich specifiek op de gelijkwaardigheid vanovereenkomstige ERK-gebaseerde criteria. Op basis van de scores van 118deelnemersdieSTRTenITNAbinnendezelfdeweekaflegden,toondedeeerstestudieaandatdetotalecorrelatietussenSTRTenITNAscoresmatighoogwas(r=0,767**),netzoalsdecorrelatie tussendeschriftelijkeonderdelen(r=0,694**).Deovereenkomsttussendescoresophetmondelingexamenwasechterveellager (τ = 0,387 **).Uit de aanvullende analyses kwamenverderediscrepantiesnaarvoren.AllereestbleekdekansomteslagenvoorSTRT(50%)significant(p=0,02)groterdandeITNA-slaagkans(35%).TentweedetoondenlineaireregressieenRaschanalysesaandaterbelangrijkeverschillenbestaantussendeconstructsvan STRT en ITNA. De woordenschat- en grammaticataken van ITNA zijnmoeilijker dan alle andere schriftelijke ITNA of STRT taken, terwijl deargumentatieve taken van STRT de makkelijkste geschreven taken bleken.Bovendien duidde de Rasch analyse betrouwbaar (0,88) aan dat de gesprokencomponentvanITNAmoeilijkerisdandievanSTRT.Ookhierwasderedenvoorde discrepantie de relatieve moeilijkheid van de taalkundige criteria bij ITNAversusderelatievemildheidvaninhoudelijkecriteriabijSTRT.

Detweedestudievergeleekscoresopcorresponderendecriteriabinnendemondelinge onderdelen van STRT en ITNA. Deze componenten bevatten zeervergelijkbaretaaktypesencriteria:beidetestsbevattenvijfcriteriadiegebaseerdzijn op dezelfde ERK descriptoren. De analyses (lineaire en meervoudigeregressie en Rasch) toonden dat ITNA en STRT elk ERK-gebaseerd criteriumanders interpreteerden. Voor alle corresponderende criteriawaren de gewogenkappacoëfficiëntenlaag(Kw≤0,216)enwasdecorrelatielaag.BovendiengafhetRasch model betrouwbaar aan dat overeenkomstige criteria nooit dezelfdemoeilijkheidsgraadhadden. HoofdstukvijfverifieerdedeveronderstellingdatVlaamsestudenten,diegeentaaltestmoetenafleggenomtoelatingtekrijgentotdeuniversiteit,hetB2niveau de facto hebben bij instroom. Indien niet alle Vlaamseeerstejaarsstudentenditniveaubereiken,slaagthettoelatingsbeleidernietinom

SummaryinDutch

244

hetB2minimumniveauonderdeeerstejaarsstudententegaranderen.Omditteonderzoeken, legden 159 Vlaamse eerstejaarsstudenten twee schriftelijke STRTtakenaftijdensdeeerstemaandvanhetuniversitaironderwijs.Metbehulpvanniet-parametrische statistiek en Rasch analyse werden de L1 scores vergelekentwee groepen van L2 kandidaten. De resultaten toonden aan dat L1 studentenover het algemeen hoger scoorden dan L2 kandidaten, maar ook dat L2kandidaten hogere scores haalden op de inhoudelijke criteria. Dat Vlaamsekandidatenhetbeterdedenopvlakvanformelecriteriaenopvlakvanalgemenescores impliceert echter niet dat alle Vlaamse studenten slaagden: 11% van deVlaamsestudentenhaaldehetB2niveauniet.TaalevolutienadetoetsTot op heden is er weinig onderzoek gedaan naar hoe internationale L2studenten zich talig redden in de doelcontext tijdens de maanden na detoelatingsproef. Nog minder studies hebben dit punt onderzocht vanuit eenkwalitatief en longitudinaal perspectief. In dit onderzoek werden 20internationale L2 studenten gevolgd tijdens hun eerste jaar aan een Vlaamseuniversiteit.Tijdensdieperiodewerdenzijmaandelijksgeïnterviewden legdenzenaachtmaandenopnieuwtweeSTRTtakenaf.Deresultatentoondenaandatde respondenten geen significante vooruitgang hadden gemaakt in termen vanSTRT-score,ofintermenvancourantematenvancomplexiteit,nauwkeurigheid,of vlotheid. Het enige significante verschil was een verminderde hoeveelheidwoorden tijdens de mondelinge presenteeropdracht. Uit de analyses bleek datbijna alle respondenten een sociaal, institutioneel en academisch isolementervaren hadden. De internationale L2 studenten hadden dus slechts beperktemogelijkheden gehad om te interageren met Vlaamse sprekers, wat zeerwaarschijnlijkheeftbijgedragentotdebeperktepositievetaalevolutie.ConclusieInhet vierdeen laatstedeel vanditonderzoekwerdbekekenhoehetVlaamsetoelatingsbeleid tot standkomt.Beleidsmakers opVlaamsniveau enopniveauvan de vijf universiteiten werden bevraagd. Uit de analyse van de interviewskwam naar voren dat het Vlaamse beleid niet in de eerste plaats stoelt opempirisch onderzoek, maar eerder resulteert uit de belangen van belangrijkestakeholdersenuithetuitwerkenvanoplossingenvoorad-hocproblemen.De resultaten van het onderzoek bieden slechts beperkte argumenten om deeffectiviteit vanhet toelatingsbeleid te ondersteunen.Het lijkt onwaarschijnlijkdathethuidigetoelatingsbeleidinstaatisomeenB2taalniveaubinnendegehelestudentenpopulatie te waarborgen, aangezien de gebruikte tests niet als

SummaryinDutch

245

gelijkwaardig kunnen worden beschouwd, en aangezienmeer dan een op tienVlaamse studenten niet slaagde voor een schriftelijke B2 toets. Bovendien isgebleken dat het B2-niveau lager ligt dan de reële receptieve taaleisen aan deuniversiteit en dat de taken binnen de taaltoetsen niet steeds inovereenstemming zijn met de werkelijke taalopdrachten. Ten slotte makeninternationale L2 studenten tijdens hun eerste jaar vermoedelijk weinig taligevooruitgang,enlijkendetaaltoetsenweinigtotgeenpositiefeffecttehebbenopdemaatschappelijkeintegratievaninternationaleL2studenten.

App

endix

24

7

App

endix1(

1/3).S

TRTPa

rt1:Listening

-into-writing

Ta

sk1

Task2

Taskgoa

lW

riteanargu

men

tative

textbased

onradiofrag

men

tListen

toale

cturean

dtosum

marizeitfo

rfello

wstude

nts

Instruction

§ W

riteasum

maryan

dan

argum

ent

§ To

taltim

e:29minutes

§ W

riteasum

mary

§ To

taltim

e:50minutes

Recep

tive

de

man

ds

Four-m

inutescripted

rad

iofrag

men

t§ Listen

onc

e§ To

pic:using

laptop

sve

rsus

pen

and

pap

erto

take

classnotes

§ Ev

enpace(152

words

/minute)

§ Clearpronu

nciation

Nine-minutescripted

lecture

§ Listen

twice

§ To

pic:in

dustrialization

§ Ev

enpace(126

words

/minute)

§ Clearpronu

nciation

Prod

uctive

de

man

ds

Write18

0+w

ords

Th

epe

rforman

cesho

uld

§ be

und

erstan

dableforpe

opleunfam

iliarw

iththebroa

dcast;

§ co

ntainan

introd

uction

,bod

y,and

acon

clus

ion;

§ prov

idefourargum

ents.

Write16

0+w

ords

Th

epe

rforman

cesho

uld

§ be

helpfulfo

rpe

opleunfam

iliarw

iththetopic;

§ includ

ethemainpo

intsofthe

lecture;

§ follo

wapresets

truc

ture.

Criteria

Con

tent

9bina

rycriteria(e.g.sou

ndargum

ents)

Con

tent

9bina

rycriteria(e.g.c

ondition

sforindu

strialization)

Voc

abulary

Grammar

Coh

esion

Mecha

nics

CEF

R-linke

d4-ba

ndscale

4≥C1

3=B2

2=B1

1≤A2

Voc

abulary

Grammar

Coh

esion

Mecha

nics

CEF

R-linke

d4-ba

ndscale

4≥C1

3=B2

2=B1

1≤A2

Appendix

248

Appen

dix1(2/3).STRT,Part2:R

eading-into-writing

Task3Task4

TaskgoalW

riteaformallettertoyouruniversity’sexam

inationcommittee

Readapopularizingpaper,sum

marizeitandform

ulateanargument.

Instruction

§ Readthedescriptionoffourstudyprogram

sandapplyfortwo

§ Totaltime:45m

inutes

§ Summarizethearticleandexplainow

nviewpoint

§ Totaltime:60m

inutes

Receptive

demands

Fourtextsdescribingstudyprograms

§ 311wordscom

bined§ FleshR

eadingEase:30§ Estim

atedgradelevel:11

Articleconcerninggenderdifferentiationineducation

§ 910words

§ FleshReadingEase:49

§ Estimatedgradelevel:9

Productivedem

andsW

rite130+words

Theperformanceshould

§ includeaquestionforthecommittee;

§ describechosenstudyprograms;

§ includeareasonforchoosingeachprogram.

Write250+w

ordsTheperform

anceshould§ explaintheissuediscussed;§ m

entionthemainresearchresults;

§ providetwoargum

entstosubstantiateone’sopinion;§ includeaconclusion.

Criteria

Content

8binarycriteria(e.g.programdescription)

Content

8binarycriteria(e.g.researchresults)

Vocabulary

Gram

mar

Cohesion

Mechanics

Register

CEFR

-linked4-bandscale

4≥C1

3=B22=B11≤A

2

Vocabulary

Gram

mar

Cohesion

Mechanics

Register

CEFR

-linked4-bandscale

4≥C1

3=B22=B11≤A

2

App

endix

24

9

App

endix1(

3/3).S

TRT,Part3

:Spe

aking

Task5

Task6

Taskgoa

lMotivateach

osen

intern

shipposition

Giveapresen

tation

abo

utocean

pollution

Instruction

§ Rea

dinform

ationab

outt

wointern

shippositions

§ Cho

oseon

e,and

motivatech

oice

§ To

taltim

e:10

minutes(5

minutesprepa

ration

)

§ Gothroug

htheslides,rea

dtheba

ckgrou

ndarticle

§ Giveapresen

tation

and

ans

werth

equ

estion

s§ To

taltim

e:15

minutes(10minutesprepa

ration

)

Recep

tive

de

man

ds

Briefw

ritten

description

ofinterns

hippo

sition

so

tablelayo

ut,b

ulletp

oints

o 187words

§ 7slidesw

ithbu

lletp

oints,graph

san

dtables

§ Po

pularizing

article

o 30

5words

o

FleshRea

ding

Ease:49

o Grade

leve

l:10

Pr

oduc

tive

de

man

ds

Five

-minuteco

nversation

Th

epe

rforman

cesho

uld

§ includ

ethreeargu

men

ts;

§ prov

idead

equa

teans

werstoth

eex

aminer’squ

estion

s.

Five

-minutepresen

tation

Th

epe

rforman

cesho

uld

§ discus

sallslid

es;

§ ex

plainagrap

h;

§ includ

epo

ssiblesolutions

toth

eprob

lempresented

;§ prov

idead

equa

teans

werstoth

eex

aminer’squ

estion

s.

Criteria

Con

tent

9bina

rycriteria(e.g.c

ondition

sforindu

strialization)

Con

tent

15binarycriteria(e

.g.informationab

oute

veryslid

e)

Voc

abulary

Grammar

Reg

ister

Pron

unciation

Flue

ncy

Initiative

CEF

R-linke

d4-ba

ndscale

4≥C1

3=B2

2=B1

1≤A2

Voc

abulary

Grammar

Coh

esion

Pron

unciation

Flue

ncy

Initiative

CEF

R-linke

d4-ba

ndscale

4≥C1

3=B2

2=B1

1≤A2

Appendix

250

Appen

dix2(1/2).ITNA:com

putertest

Format

Operationalization

Language-in-use

Vocabulary

Multiplechoice

Pickoneoffouroptionstocompleteasentence

Multiplechoice

Selectoneoffourexpressionswhichbestfitsagivendescription

Shortopenanswer

Wordtransform

ationsoentrymatchesagivensentence

Gram

mar

Shortopenanswer

Wordtransform

ationsoentrymatchesagivensentence

Cloze

ShortclozetextsThreetextsw

ithfivegapseach(selectedwordsareom

itted,ratherthaneverynthw

ord):popularizingscientifictext

LongclozetextOnetextw

ithtengaps(selectedwordsareom

itted,ratherthaneverynthw

ord):popularizingscientifictextReading

Com

prehensionMultiplechoice

Fiveshortnewspaperarticles(around300w

ords),withtw

ocomprehensionquestionseach

Structure

Drag-and-drop

Firstandlastsentencearegiven,candidatesrestructurefivejumbledsentences

Listening

Com

prehensionMultiplechoice

Threefour-minuteradioextracts,w

ithfourcomprehensionquestionseach

Dictation

Shortopenanswer/

dictationW

ritedowneightw

ordsasmentionedinnaturalspeech(new

sbroadcast)

App

endix

25

1

App

endix2(2/2).IT

NA:S

peak

ingtest

Argum

entation

Pr

esen

tation

Ta

skgoa

lFo

rmulatean

argum

ent

Giveapresen

tation

abo

utagen

eralto

pic(i.e.,sm

oking)

Instruction

§ Rea

dinform

ationab

outt

wopo

ssibleto

pics

§ Cho

oseon

etopic,and

form

ulatean

argum

ent

§ To

taltim

e:10

minutes(15minutesprepa

ration

forbo

thta

sks)

§ Gothroug

htheslides

§ Giveapresen

tation

and

ans

werth

equ

estion

s§ To

taltim

e:5m

inutes

Recep

tive

de

man

ds

Briefw

ritten

description

oftop

ics

§ 5slidesw

ithbu

lletp

oints,graph

san

d/orta

bles

Prod

uctive

de

man

ds

Five

-minuteco

nversation

Th

epe

rforman

cesho

uld

§ includ

ethreeargu

men

ts;

§ prov

idead

equa

teans

werstoth

eex

aminer’squ

estion

s.

Three-minutepresen

tation

Th

epe

rforman

cesho

uld

§ discus

sallslid

es;

§ ex

plaintw

ograp

hsand

/ortables;

§ prov

idead

equa

teans

werstoth

eex

aminer’squ

estion

s.

Criteria

Voc

abulary

Grammar

Pron

unciation

Flue

ncy

Coh

esion

CEF

R-linke

d4-ba

nd

scale

4≥C1

3=B2

2=B1

1≤A2

Appendix

252

Appen

dix3.L2P participants

M/F

*L1

Nation

alityU★

B/M

°Faculty

L2♯

Test +

Heddi

FGerm

anSw

itzerlandG

BEngineering

24STR

T

Noor

FFarsi

IranG

M

Medicine

17ITN

A

Hassan

M

TurkishTurkey

G

M

Law

9ITN

A

Vincent

M

FrenchFrance

G

M

Sciences(Geology)

12ITN

A

Marion

FGreek

Greece

G

M

Politicalsciences48

ITNA

Sandrine

FFrench

Belgium

G

M

Law

12ITN

A

David

M

EnglishSouthA

fricaG

BSocialsciences

36ITN

A

Henna

FFinnish

FinlandG

M

Psychology14

ITNA

Abdullah

M

Arabic

Morocco

G

M

Sciences(Chem

istry)12

ITNA

Hatice

FTurkish

TurkeyG

M

Politicalsciences36

ITNA

Janet

FEnglish

Cam

eroonG

BMedicine

12ITN

A

Note

.

*Male/Fem

ale°Bachelor/M

aster♯m

onthsofDutchL2instruction

+universityentrancetesttaken

App

endix

25

3

App

endix4.L2 Fparticipa

nts

M/F

* Age

L1

Nationality

U★

B/M

° Fa

culty

L2♯

STRT

ITNA

+/-+

Elen

aF

23

Ukrainian

Ukraine

G

M

Engine

ering

18

11

+

Alexa

ndra

F24

Sp

anish

Peru

G

M

Econ

omics

71

1+

Marie

F19

Fren

ch

Belgium

G

M

Law

120

11

+

Leila

F

44

Haitian

Haiti

IM

Politicalscien

ces

20

10

+

Gua

dalupe

F

30

Span

ish

ElSalva

dor

G

BPs

ycho

logy

12

10

+

Elif

F24

Tu

rkish

Turkey

L

M

History

12

11

+

Oksan

aF

21

Ukrainian

Ukraine

L

BLing

uistics

91

1+

Er

si

F19

Alban

ian

Alban

ia

A

BLa

w

81

1+

Alirez

aM

24

Farsi

Iran

G

BEn

gine

ering

60

1-

Merve

ille

F27

Fren

ch

Con

go

LM

Biom

edical

10

10

-

Gab

riela

F23

Span

ish

CostaRica

LB

Ling

uistics

11

10

-

Emma

F21

German

German

yL

BPs

ycho

logy

10

11

-

Hoa

ng

M

24

Vietnam

ese

German

y/Vietnam

L

BPs

ycho

logy

12

11

-

Ana

stasia

F26

Rus

sian

Rus

sia

LM

Engine

ering

22

11

-

Océan

eF

20

Fren

ch

Belgium

A

M

Law

72

11

-

Clara

F21

Fren

ch

Belgium

A

M

Law

72

11

-

Stella

F32

Arm

enian

Arm

enia

H

BEc

onom

ics

70

0+/V

Ya

zdan

M

24

Pash

to

Afgha

nistan

A

BEc

onom

ics

14

10

V

Jessica

F27

Sp

anish

Chile

G

M

Med

icine

24

10

?

Chloé

F

21

Fren

ch

Belgium

LB

Econ

omics

72

11

?

Note.

*Male/Fe

male

★ AntwerpUnive

rsity/G

hentU

nive

rsity/Interunive

rsity/Unive

rsityCollege

ofH

asselt/U

nive

rsityofLeu

ven

°Bache

lor/Master

♯ mon

thsofD

utch

L2instruction

+more(+)/less(-)th

an50%

ofc

oursespassedinJu

ly2015/Visaprob

lems/reason

forattritionun

know

n(?)

Appendix

254

Appendix5.UniversitystaffFaculty/Department Position IDCentraladministration Didacticspolicymanager Ac2

Universitydirectorofeducationalaffairs Ac12Languagepolicymanager Ac5Languagepolicymanager Ac10

Humanities Professor Ac1Tutor Ac16Professor Ac17Facultydirectorofeducationalaffairs Ac22Tutor Ac15Professor Ac7

Engineering Tutor Ac11Professor Ac6Facultydirectorofeducationalaffairs Ac23

Medicine Facultydirectorofeducationalaffairs Ac14Sciences Professor Ac13

Facultydirectorofeducationalaffairs Ac18Professor Ac20

Economics Tutor Ac3Facultydirectorofeducationalaffairs Ac21

Law Professor Ac8Tutor Ac19

Psychology Facultydirectorofeducationalaffairs Ac9Social&PoliticalSciences Professor Ac4

Tutor Ac24

255

Documents

ASSESSING HIGH-STAKES ASSUMPTIONS · 2017. 10. 17. · ASSESSING HIGH-STAKES ASSUMPTIONS A longitudinal mixed-methods study of university entrance language tests, and of the policy