The KIMEP English Entrance Test (KEET) Validity Argument‐Based Validation for the KIMEP English Entrance Test (KEET) Marina Gerassimenko, University of Leicester Kazakhstan Institute of Management, ... Background KIMEP: Kazakhstan Institute of Management ...

  • Published on
    27-Apr-2018

  • View
    218

  • Download
    5

Transcript

<ul><li><p>AnArgumentBasedValidationfor theKIMEPEnglishEntranceTest (KEET)</p><p>MarinaGerassimenko,University</p><p> of</p><p> Leicester</p><p>KazakhstanInstituteofManagement, EconomicsandStrategicResearch(KIMEP)</p></li><li><p>Outline</p><p>BackgroundValidityasanargument</p><p>ToulminsstructureofanargumentKane(1992,2006)andChapelleetal.</p><p> (2008)TheKEETsvalidityargumentSummaryoftheKEETsvalidityargumentLessonslearned</p></li><li><p>Background </p><p>KIMEP:KazakhstanInstituteofManagement,Economicsand StrategicResearch,Almaty,Kazakhstan,CentralAsia</p><p> AnEnglishmediumuniversity</p><p> Degreesineconomics,management,finance</p><p> 87%undergraduatestudents;13%graduate;3.6%international</p><p> 30003500applicantsannually</p></li><li><p>TheKEET: paperbased 100items</p><p> multiplechoice threesections Grammar,VocabularyinUse,Reading</p><p> Comprehension parallelforms</p></li><li><p>Avalidityargument</p><p>Avalidityargumentshouldpresentand integrateevidenceandrationalesfromwhich</p><p> validityconclusioncanbedrawnpertainingto particularscorebasedinferencesandusesof</p><p> atest.Chapelle(1999:263)</p></li><li><p>AnArgumentStructure(Toulmin, 1958,2003):</p><p>WARRANT</p><p>Backing1</p><p>Backing2</p><p>Assumption</p><p> 1</p><p>Assumption</p><p> 2 Assumption 3</p><p>Backing3</p><p>Grounds1</p><p>Claim(Grounds2)</p><p>Rebuttals</p></li><li><p>Kane(1992,2006): </p><p>TwoStagesof Arguments</p><p> Aninterpretativeargument </p><p>specifiesthe proposedinterpretationsandusesoftest resultsbylayingoutthenetworkofinferences</p><p> andassumptionsleadingfromtheobserved performancestotheconclusionsanddecisions</p><p> basedontheperformances Thevalidityargument</p><p> providesanevaluation</p><p> oftheinterpretativeargument</p></li><li><p>Chapelleetal.(2008)Testuse</p><p>Construct</p><p>Target</p><p> score</p><p>Expected</p><p> score</p><p>Observation</p><p>Observed</p><p> score</p><p>Target</p><p> domain</p><p>UTILIZATION(6)</p><p>EXPLANATION(4)</p><p>EXTRAPOLATION </p><p>(5)</p><p>GENERALIZATION(3)</p><p>EVALUATION(2)</p><p>DOMAIN DESCRIPTION(1)</p></li><li><p>Step1:TheDomainDescription Inference</p><p>WARRANT</p><p> 1:TheKEETisareliableandvalidtestfor</p><p> admissionpurposes;thetestrevealsimportant Englishlanguageskillsandabilities</p><p>B1.1Tasksare</p><p> selectedas</p><p> representative</p><p> ofGeneral</p><p> Englishdomain</p><p>B1.2Important</p><p> skillswere</p><p> selectedbased</p><p> onformatand</p><p> administration</p><p> constraints</p><p>A1.1Importanttasks areidentified</p><p>A1.2Performanceonthe</p><p> testrevealsabilitiesand</p><p> skillsrelevantatEnglish</p><p> mediumuniversity</p><p>A1.3Scoresarenot</p><p> affectedbythetestformat</p><p> (multiplechoice)</p><p>B1.3Scoringis</p><p> objective</p><p>Target</p><p> Domain</p><p>Observations</p></li><li><p>Step2:TheevaluationInferenceWARRANT2:TheKEETsobservedscores</p><p> reflecttargetedlanguageabilities</p><p>A2.1Scoring</p><p> rubricsare appropriate asevidence</p><p>A2.2Task</p><p> administration conditionsare appropriateevidence</p><p> fortargetedabilities A2.3TheKEEThas appropriate</p><p> psychometric properties</p><p>Observations</p><p>Observed</p><p> score</p></li><li><p>B2.1</p><p>Rubrics: grammar,use</p><p> ofEnglish, reading</p><p> comprehensionprovide</p><p> evidencefor targeted</p><p> abilities</p><p>B2.2Task administration</p><p> conditionsare standardized</p><p> byAdmission Office</p><p> protocol</p><p>B2.3Acceptable forstandardized tests</p><p> coefficients Alfa:</p><p>formA.935; formB.922</p><p>Bothformshave distribution</p><p> closetonormal</p></li><li><p>Backing3.3:TheKEETpsychometricdataforformsAandB:</p><p>Scale FormA FormB</p><p>Nofitems 100 100</p><p>Samplesize 232 241</p><p>Mean 58.375 54.095</p><p>Variance 279.467 232.584</p><p>Median 58.000 55.000</p><p>Std.Dev. 16.717 15.251</p><p>SEM 4.235 4.273</p><p>Meanp 0.584 0.541</p><p>Alfa 0.935 0.922</p><p>ITEMANsoftware ResearchQuestions:Howsignificantisthedifference?Whydoparallelformshavedifferentmeans,variances</p><p> andSD?Doparallelformshaveitemsofsimilardifficultylevel?</p></li><li><p>Howparallelisparallel?</p><p>Researchmethods:(1)paralleltesttheoryaspartofClassicalTestTheory(Kline,2005):</p><p>ModesMeansMediansVariances(2)ttest:howsignificantisthedifference?(3)Cohendtest:whatistheeffectsize?</p><p>Findings:(2)T=2.9108,df(degreesoffreedom)is471,2tailconfidenceat99.62%thedifferenceinmeans</p><p> issignificant;</p><p>(3)Cohensd=0.27Cohen(1988) notsignificant;Wolf(1986) </p><p>practicallysignificant</p></li><li><p>Step3:TheGeneralizationInference</p><p>WARRANT3:Observedscoresareindicative</p><p> ofexpectedscoresoverparallelversionsof tasksandtestforms</p><p>B3.1Secondary</p><p> schoolEFL</p><p> programme</p><p>B3.2TheKEETs</p><p> specifications:Threefixedrubrics;100freeitems</p><p>A3.1Testdomainis</p><p> sampledfrom</p><p> generalEnglish</p><p> proficiencyarea:</p><p> Problematic</p><p> GrammarStructures,</p><p> vocabulary,reading</p><p>A3.2Test</p><p> specificationsas testtemplate</p><p>Observedscore</p><p>Expectedscore</p></li><li><p>R3.2TestSpecifications:fixedand</p><p> freeelements</p><p>Backingforassumption3.2TestSpecifications</p><p>METHODOLOGY:</p><p>couldfreeelements</p><p> intestspecifications causetestSD,</p><p> meansvary?</p><p>RESEARCH</p><p> QUESTION:</p><p>pvaluescompared</p><p> acrosssectionsof grammar,vocabularyin</p><p> contextandreading comprehensionand acrosstwoparallelforms</p></li><li><p>FINDINGS:</p><p>Grammar FormA FormB</p><p>Meanpvalue .66 .59</p><p>Pvaluerange .11</p><p> .91 .18</p><p> .97</p><p>Vocabulary FormA FormB</p><p>Meanpvalue .49 .57</p><p>Pvaluerange .09</p><p> .87 .17</p><p> .89</p><p>Reading</p><p> comprehensionFormA FormB</p><p>Meanpvalues .52 .34</p><p>Pvaluesrange .38</p><p> .76 .12</p><p> .60</p></li><li><p>TheKEETexample</p><p>FormA</p><p>14.Thistimenextweek,we.....thechemistryexam.A.</p><p> hadfinished</p><p>B. </p><p>havebeenfinishingC.</p><p> willhavefinished</p><p>D. </p><p>willhavebeenfinishing</p><p>Prop. Disc.Point</p><p>CorrectIndexBiser.0.610.310.24</p><p>FormB</p><p>14.Dontworry I.....writingmy reportbyFridaynextweek.Ill giveittoyoubythen.</p><p>A. </p><p>finishB.</p><p> willbefinished</p><p>C. </p><p>amfinishingD.</p><p> willhavefinished</p><p>Prop.Disc.PointCorrectIndexBiser.0.720.560.50</p></li><li><p>ReadingComprehensionFormA FormB</p><p>Length 600 700</p><p>Topics Generalinterest Specificinterest</p><p>Discoursefeatures</p><p>Narrative Narrative</p><p> imbeddedwith descriptions</p><p>Vocabulary General Specific</p><p>Questiontypes Askingforexplicit</p><p> informationAskingfor</p><p> inferences</p></li><li><p>SummaryoftheclaimsabouttheKEET</p><p>1stclaim:thetestpurpose</p><p>2ndclaim:thetestconstruct</p><p>3dclaim:thetestspecifications</p><p>4thclaim:theitemanalysis</p><p>5thclaim:theuseofthetestscores</p></li><li><p>Conclusion:LessonsLearned</p><p> TheKEETsconstruct Thetestspecifications Analysisofparallelforms Itemanalysis</p></li><li><p>Thankyouforyourattention!</p><p>An Argument-Based Validation forthe KIMEP English Entrance Test (KEET)OutlineBackgroundKIMEP: Kazakhstan Institute of Management, Economics and Strategic Research, Almaty, Kazakhstan, Central AsiaAn English-medium universityDegrees in economics, management, finance87% undergraduate students; 13% graduate; 3.6% international3000-3500 applicants annuallyThe KEET:paper-based100 itemsmultiple-choicethree sections Grammar, Vocabulary in Use, Reading Comprehensionparallel formsA validity argumentAn Argument Structure (Toulmin, 1958, 2003) :Kane (1992, 2006): Two Stages of ArgumentsChapelle et al. (2008)Step 1: The Domain Description InferenceStep 2: The evaluation InferenceBildnummer 11Bildnummer 12How parallel is parallel?Step 3: The Generalization InferenceBildnummer 15FINDINGS:The KEET exampleReading ComprehensionSummary of the claims about the KEETConclusion: Lessons LearnedBildnummer 21</p></li></ul>

Recommended

View more >