View
28
Download
0
Category
Preview:
Citation preview
CentreforInternationalResearchonEducationSystems
StephenLamb
JenniferJackson
CentreforInternationalResearchonEducationSystems,VictoriaUniversity
RussellRumberger
UniversityofCaliforniaSantaBarbara
ISCY Technical Paper:
Measuring 21st Century Skills in ISCY
International Study of City Youth
1 CentreforInternationalResearchonEducationSystems
Measuring21stCenturySkillsinISCY
StephenLamb,VictoriaUniversity
JenniferJackson,VictoriaUniversity
RussellRumberger,UniversityofCaliforniaSantaBarbara
Exploring the development of non‐cognitive, or “21st century” skills, is animportantgoaloftheInternationalStudyofCityYouth(ISCY).Thispapersetsoutaframeworkformeasuring21stcenturyskillsinISCYalongsideselectedmeasuresofstudentengagementanddispositionstowardsschool,toinformanalysisoftheskills across the 12 cities participating in the ISCY project. The proposedframework draws on contemporary literature, as well as conceptual andquantitativeanalysisoftheISCYStudentSurveydata,todeveloprobustscalesformeasuringtheskillsinaninternationalcontext.
IntroductionIncreasing attention is being given in contemporary educational research to non‐cognitiveor21stcenturyskills1,orcapabilitiesbelievedtomeetthedemandsof“21stcentury work” (Silva 2008, p. 1). While test scores have been centre stage ininternational comparisons, there has been growing recognition of the effects thateducation has on the development of interpersonal and intrapersonal skills andcapabilities that affect the success of students in school and success in the labourmarket(e.g.Levin,2012).Somestudiespointtolastingeffectsofnon‐cognitiveskillsonstudent’s lives including impactsoneducational, careerandhealthoutcomes (see, forexample, Kautz et al., 2014). Emphasising these skills, alongside core literacy andnumeracy, arguably creates “more powerful learning experiences that lead to deeperunderstanding andmore useful knowledge in tune with our times” (Trilling & Fadel2009, p. 172). Assessing such skills may also help make visible the achievements ofstudentswhodonotperformwellinacademictests,andthecontributionschoolsmaketo their learning (Jordan 2010). Capturing how well these skills are developed isthereforeanimportantgoaloftheInternationalStudyofCityYouth(ISCY).This paper sets out to establish a common framework for the measurement of 21stcentury skills, to be used by all cities involved in the ISCY project. ISCY is aninternational longitudinal study of 10th Grade students being conducted in 12 citiesaroundtheworld,tofindoutmoreaboutstudentjourneysthroughschoolintofurtherstudy,workandlifebeyondschool. Aswellascognitiveskills,studentplans,attitudestoschool,andengagement,thestudyaimstomeasurearangeof21stCenturyskillsandthe extent to which they affect student progress and later outcomes. Establishing acommon framework formeasuring21st centuryskills in ISCY isnotaneasy task.Thedevelopmentof the ISCYStudent Surveydrewonmany international instruments formeasuringstudentskills,attitudesandbehaviours(PISAstudentsurvey,ESLSstudentsurvey, the Gallup student poll, Education Longitudinal Study of 2002, High SchoolSurvey of Student Engagement, the University of Chicago Consortium for School
1 The skills discussed in this paper have been variously referred to as “21
st century” or “non‐cognitive” skills in
international literature. The term “21st century skills” has been adopted for this paper.
International Study of City Youth
2 CentreforInternationalResearchonEducationSystems
Research student survey, the big five questionnaire, as examples). Items from theseinstrumentswereusedselectivelyandjudiciously,toenablethesurveytocoverabroadrangeof topicswhile remainingat amanageable length for students.Translation intodifferent languages also necessitated adjustments to some items, to maximiseinternational consistency. Some entirely new items were also created, based on theinterestsandcontemporarycontextsoftheparticipatingcities.Forthesereasons, it isnot possible to simply map ISCY survey items to established instruments, to easilydeterminewhichconstructstheyaremeasuring.
Previousworkon21stCenturySkills
Since the ISCY Student Survey was created, new research in this field has emerged,includingvaluablesynthesesofpriorinternationalliterature(e.g.Farringtonetal2012;Gutman & Schoon 2013). The literature reflects a “confusing, overlapping array ofconceptsandterms”(Farringtonetal2012),withnumerousscholarsproposingvarioustaxonomies of 21st century skills, each comprising adifferent set of constructs.Whiletheremaybegeneralagreementintheliteraturethat21stcenturyskillsareimportant,there is far less agreement as to what these skills are; whether they are malleable;whether they have any effect on other outcomes; and how they might be measured(Gutman&Schoon2013).ThispaperseekstochartacoursethroughthisliteraturethatwillfacilitateanalysisforISCYresearchers.Table1summarisesthemeasurability,malleability,effectandstrengthofevidenceforeightkeygroupsof21stcenturyskillsidentifiedintheliterature.Table1 Gutman & Schoon’s summary of findings on non‐cognitive (21st
century)skills
Qualityofmeasurement Malleability
Effectonotheroutcomes
Strengthofevidence
1. Self‐perceptions
Self‐conceptofability High Medium Notavailable MediumSelf‐efficacy High High High Medium
2. Motivation
Achievementgoaltheory High Medium Lowtomedium MediumIntrinsicmotivation High Medium Lowtomedium HighExpectancy‐valuetheory Medium Notavailable Mediumtohigh Medium
3. Perseverance
Engagement Medium Notavailable Notavailable LowGrit Medium Noevidence Noevidence Low
4. Self‐control Medium Lowtomedium Low Medium
5. Meta‐cognition Medium Mediumtohigh Mediumtohigh High
6. Socialcompetencies
Leadershipskills Low Notavailable Noevidence LowSocialskills Medium Mediumtohigh Lowtomedium High
7. Resilience&coping Medium High Low Medium
8. Creativity Medium Notavailable Noevidence Low
Source:Gutman&Schoon2013,p.2.
International Study of City Youth
3 CentreforInternationalResearchonEducationSystems
As Table 1 shows, current research on 21st century skills is characterised by a highvolume of activity, but relatively limited evidence regarding clearly measurableconstructsthatareknowntohaveanimpactonlearning.Thetaskofdevelopinga21stcentury skills framework for ISCY therefore involves some degree of professionaljudgement,inidentifyingliteraturethatbestoffersapathwaythroughthiscomplicatedfield.TwoframeworkswereidentifiedasparticularlyrelevanttotheISCYproject.
1. BridgingthegapbetweenskillsandpersonalityThe first framework used in developing the ISCY 21st century skills framework isPellegrino and Hilton’s taxonomy (2012) (Figure 1), developed for the US NationalAcademyofSciences,andsubsequentlyadoptedinarecentUSDepartmentofEducationreport(Schectmanetal2013).Thistaxonomyidentifieseight“clusters”of21stcenturyskills from the literature, and maps these against the “Big 5” personality traits(openness;conscientiousness;emotionalstability;agreeableness;andextraversion),aswellasthreecorecognitiveabilitiesfromCarroll’s(1993)CognitiveAbilitiesTaxonomy(cognitiveprocessesandstrategies;knowledge;andcreativity).This taxonomy is valuable in bridging the gap between personality traits and 21stcentury skills. While these may sometimes be conflated, personality traits are morestableandlessmalleablethanskills,andalsomoreambiguousintheirdesirability.Forexample, thepersonality traitofopennessmaybe lessdesirable thanconservatism insomecircumstances(suchasfollowingarecipe);andeventhetraitofneuroticismhasbeenfoundtohavesomepositiveeffects(Turianoetal2013).Thesetraitsarethereforeunlikelytobeusefulmeasuresofwhatschoolsshouldaimtocultivateintheirstudents.An example from the ISCY Student Survey is the item “I prefer to work alone”. Thisarguably can be viewed as a measure demonstrating the personality trait ofextroversion(inreverse).Butwhatdoesthistellusabouttheindividualasalearner?Apreferencetoworkalonedoesnotnecessarilyequatetoaninabilitytoworkwellwithothers; indeed, preliminary analysis suggests that the correlationbetween agreementwiththisstatement,anddisagreementwiththestatement“Iworkwellingroups”,isnotespeciallystrong.Skills are a far more useful measure, as they are malleable and therefore open toinfluence and development by schools. Skills are also more clearly desirable thanpersonality traits, as theymeasure growth inwhat an individual can do, rather thantheir intrinsic qualities.While individualswith certainpersonality traitsmaydevelopsomeskillsmoreeasilythanothers,schoolsmustseektoensurethatallstudentshaveopportunities tocultivate theskills thatmatter for lifeand learning.The taxonomyatFigure1liststhemanyskillsthatschoolsmayseektocultivate,toequiptheirstudentsforfuturesuccess.
International Study of City Youth
4 CentreforInternationalResearchonEducationSystems
Figure1 21st century skills clusters, with associated cognitive abilities andpersonalitytraits
Source:Pellegrino&Hilton2012,pp.2‐12–2‐14(adapted).
2. Connecting21stcenturyskillstoeducationalsuccessThe taxonomy at Figure 1 provides a useful conceptual foundation for differentiatingbetweenmalleable skills and non‐malleable traits, but it does not address how theseskills operatewithin the education system. This is a key concern of the ISCY project,
International Study of City Youth
5 CentreforInternationalResearchonEducationSystems
especially the role of 21st century skills in contributing to student achievement andreducingtheimpactofeducationalfailure.The University of Chicago Consortium on Chicago School Research (CCSR) hasdeveloped a hypothetical framework, showing how 21st century skills might supportstudent achievement. A key feature of this framework is the notion of academicmindsets,whichCCSRdefinesasfollows:
Ibelonginthisacademiccommunity(senseofbelonging) Myabilityandcompetencegrowwitheffort(implicittheoriesofability) Icansucceedatthis(self‐efficacy) Thisworkhasvalueforme(expectancy‐valuetheory)
CCSR suggests that educators focus their efforts on cultivating thesemindsets, ratherthanattemptingtofosterskillsthatare“notdirectlymalleableanddependconsiderablyoncontext”(Nagaokaetal.2013,p.48).Byfosteringacademicmindsets,educatorscandrive their students’ academic perseverance (or persistence with academic tasks), aswellasassistinginthedevelopmentofsocialskillsandlearningstrategies.Inturn,thesefactors lead to improved academic behaviours (such as positive classroom behaviourand study habits), culminating in improved academic performance. As performanceimproves,positiveacademicmindsetsarereinforced,creatingaself‐perpetuatingcycleofimprovement(Figure2).Figure2 CCSR’shypotheticalmodeloftherelationshipbetweennon‐cognitive
factors(21stcenturyskills)andacademicperformance
Source:Farringtonetal2012,p.12.
This model is an attractive tool for organising the various skills, attitudes andbehaviours measured in the ISCY Student Survey. Of course, ISCY cannot claim to
International Study of City Youth
6 CentreforInternationalResearchonEducationSystems
measurecomprehensivelyallofthecomplexconstructsinthismodel;forexample,ISCYdoesnotcapturedetailedinformationrelatedtospecificlearningstrategiesadoptedbystudents, which are captured in leading international measures of this construct(Pintrichetal.1993).Nevertheless,themodelprovidesaguideastohowISCYsurveyitems can be organised usefully into a series of scales that reflect currentunderstandingsof21stcenturyskills,andhowtheyoperate ininterconnectedwaystoimprovestudentlearning.
DevelopingtheISCYFramework
TheISCYFrameworkfor21stcenturyskillsandengagementwasdevelopedinductivelyanddeductively,usingquantitativeanalysisofdatafrombaselineISCYStudentSurvey,supportedbyareviewofcurrentliteratureon21stcenturyskillsandengagement,andusingthevariousexistingtaxonomiesormodelsofskillsasguides.Twelvescalesweredeveloped formeasuring the constructs of 21st century skills, academic dispositionsandstudentengagement,withtwofurtherscalesformeasuringcognitiveskills(mathsandreading).ThescaleswillbeusedincomparingdatabetweenISCYcities,aswellascomparingdatafordifferentstudentgroupswithinandacrosscities.TheprocessfordevelopingtheISCYFrameworkwasasfollows:
1. Identifying ISCY Student Survey items likely tomeasure constructs relevant toskills,engagementanddispositions;
2. ConductingaprincipalcomponentanalysiswithbaselineStudentSurveydata2,toidentifyitemsthatgroupintoscales,andconfirmingthevalidityofscales;
3. Using the related literature on models of relationships between factors andoutcomes,toorganisethescalesintoaframeworkforanalysis.
Each of these stages of development is detailed below. Once the framework wasdeveloped,ascoreforeachISCYstudentwascalculated.
1. Identifyingitemslikelytomeasurerelevantconstructs
The ISCY student survey questionnaires included numerous items on studentcharacteristics,studentfamilybackground,studentperceptionsanddispositions,viewson school, engagement in school, educational and career plans, civic participation,politicalandsocialvalues,and21stCenturyskills.ManyoftheitemsincludedintheISCYstudent surveywere based on items used in the OECD Programme for InternationalStudent Assessment (PISA) in order to promote consistency in internationalcomparisons. This included items on engagement and dispositions aswell as studentbackgroundandcharacteristics.Manyofthenon‐cognitiveskillitemsweremodelledonexistingsurveysandscales,suchastheBigFivemeasuringopenness,conscientiousness,agreeableness,extraversion,andemotionalstability(seeBorghansetal.,2008)andtheUniversity of Chicago School Consortium on School Research My Voice, My Schoolstudent questionnaire measuring academic behaviours, relations with peers andteachers,dispositions,perseverance,andsocialskills. 2 Cities with baseline data available were Melbourne, Ghent, Montreal, Hong Kong, Bergen, Barcelona, Sacramento, Santa Barbara, and Wroclaw.
International Study of City Youth
7 CentreforInternationalResearchonEducationSystems
Themajorityofmeasuresof21stcenturyskillsintheISCYStudentSurveyareitemsinwhichstudentsindicatetheiropiniononaLikertscale3.Someoftheseitemsprovideadirect self‐report of the student’s skill level (such as “I am good at leading others”),whereas others provide indirect indicators of certain dispositions (such as “There islittletopreventmefromreachingmygoals”).Studentself‐reportisawell‐establishedmethodofmeasuring21stcenturyskills(ChildTrends,2014).Indeed,someresearchershavearguedthatnon‐cognitiveskillscanonlyreallybeassessedthroughself‐reportedquestionnairesthatelicitdimensionssuchastheBigFive(seeJohn&Benet‐Martínez,2000).Other types of measures can also be valuable for inclusion such as measures ofbehaviouralengagementthatcallonfrequenciesofbehaviour.Thestrongestmeasuresof behavioural engagement can be items relating to the frequency with which thestudent engages in behaviours, for example, those that are often associated with orpredictive of disengagement from school, including skipping classes, coming late toschool, or getting into trouble frequentlywith teachersdue to theirbehaviour. In theISCY survey, four such itemshad comparabledata available for all cities.Other itemswere noted as potential measures of positive behavioural engagement, such asinvolvement in extra‐curricular activities, but these are ratherdependent on city andschool policies linked to provision of such activities and therefore may not facilitatereliableinternationalcomparisons.Theseitemsmayneverthelessbevaluableforfutureanalysis alongside the scales developed for ISCY, especially in measuring studentengagement.Other ISCY student survey items excluded from consideration for the current scaledevelopmentworkincluded: Items relating to the student’s perception of their own school, as they relate
specifically to the student’s own school, rather than more general academicdispositions.
Itemsrelateddirectlytoastudent’swellbeingoutsideofschool(suchashappinesswith lifeathome),as the frameworkconcernsstudentengagementwithschoolingandeducation.
Items related to civic engagement. It is nevertheless recognised that civicengagementisanimportantskillforschoolstocultivate,bothforitsintrinsicvalueanditsflow‐oneffectsforlearningandengagement(Cress2012).
Wherepossible,itemswereselectedthatwereavailableforallcities.MostLikert‐scaleditems compliedwith this criterion, except for four items not available forMontreal.4
3 It is acknowledged that the use of Likert scales for this purpose rests on certain assumptions: The distance between values within each Likert scale is considered to be equivalent. For example, the
distance between “Disagree” and “Agree” is assumed to be equal to the distance between “Agree” and “Strongly agree”.
It is assumed that scales using a similar structure are comparable. For example, the scale “Strongly disagree/ Disagree/ Agree/ Strongly agree” is assumed to be comparable to “Very unhappy/ Unhappy/ Happy/ Very happy”.
4 The following items were excluded from the Montreal Student Survey: Right now I see myself as being pretty successful as a student; There is little that can prevent me from reaching my goals;
International Study of City Youth
8 CentreforInternationalResearchonEducationSystems
Theseitemswerestillretainedintheframework,astheymeasureimportantconstructs.ValueswereimputedfortheseitemsforthemajorityofMontrealstudents.2. Usingprincipalcomponentanalysistoidentifyscalecomponents
Principal componentanalysis (PCA)was conductedusinga largenumberofpotentialsurveyitems,toexplorehowtheitemstendedtogrouporcluster.PCAisacommonlyusedtechniqueforcreatingindicesasitsmainpurposeistoreducethedimensionalityindatawithoutlosingrelevantinformation.Theideaofthemethodistoconvertasetofobservations of correlated variables into a set of values of linearly uncorrelatedvariables called principal components which account for as much as possible of thevarianceoftheincludeddataitems(see,Abdi&Williams,2010).PCA was conducted to explore how the identified potential skill, disposition andengagement variables grouped into principal components. The analysis produced 11principalcomponents,withthreetoeightitemsloadingmoststronglyagainsteachone.Twoadjustmentsweremadeatthisstage: Onecomponentcomprisingeightitemswasseparatedintotwoscales.Theseitems
concerned the level of student emotional engagement and sense of belonging atschool,suchas“Ilikebeingatschool”and“IgetafeelingofsatisfactionfromwhatIdo in class”. There is a clear relationship between these items, and similar itemshavebeengroupedtogetherinpriorresearch(seeRotermund2010).Nevertheless,other literature supports a separation between a senseof belonging in the schoolenvironment, which has more of a social quality, and a sense of intellectualsatisfaction in schoolwork (Taylor&Parsons2011,p. 20).While these scalesarecloselyrelated,itseemsplausiblethatastudentmayscorehighlyononebutnottheother.
Onecomponent involvesonlytwoitems,“I liketothinkofnewwaystodothings”and“Iamgoodatcomingupwithnewideas”andrelatestotheconceptofcreativity.The use of two items to measure creativity, while not ideal, is supported by anumberofresearcherswhohavedevelopedshorterinventoriesformeasuringnon‐cognitive skills (see for example the discussion by Gosling et al, 2003, on thedevelopment and application of a ten‐item inventory for measuring the Big Fivedomains).Allotherscaleshavethreeormoreitems.
ThehighestloadingsforeachcomponentarereportedinTable1withCronbachAlphareliabilityestimatesattheendofthetable.
I can think of many ways to reach my current goals; There are lots of ways around any problem that I am facing now.
Table1 ResultsofprincipalcomponentanalysisforISCYStudentSurveywithconstructlabels
Purpose
Behvio
ural
engage
ment
Cognitive
engage
ment
Emotio
nal
engage
ment
Conscie
ntio
usness
Belongin
g
Hope
Self‐efficacy
Collab
oratio
n
Communicatio
n
Creativity
Self‐M
anage
men
t
Working hard in school matters for success in the workforce 0.88
What we learn in class is necessary for success in the future 0.80
School teaches me valuable skills 0.74
My classes give me useful preparation for what I plan to do in life 0.70
Skipped a class without permission 0.85
Been absent from school for a day without permission 0.82
Been in trouble with a teacher because of your behaviour 0.47
Arrived late at school 0.66
I get into trouble frequently at school 0.27
In class, I try to work as hard as possible 0.83
In class, I put in my best effort 0.81
In class, I keep working even if the material is difficult 0.79
School is often a waste of time 0.31
I get a feeling of satisfaction from what I do in class ‐0.39
High level of interest in school work ‐0.31
I find most school work boring 0.56
Hours of homework 0.45
I always try to do my best 0.88
I always get work in on time 0.80
I persevere with a job until it is done 0.74
I am a hard working student 0.70
I feel safe at school ‐0.64
International Study of City Youth
I will leave this school with good memories ‐0.61
Happy with life at school ‐0.66
I like being at school ‐0.65
I am confident of finding a good job when I finish my studies ‐0.50
Happy with future ‐0.53
There is little that can prevent me from reaching my goals ‐0.68
Right now I see myself as being pretty successful as a student ‐0.52
I can think of many ways to reach my current goals ‐0.75
There are lots of ways around any problem that I am facing now ‐0.79
I am confident of doing well in school ‐0.53
I understand how others are feeling 0.66
I get along well with others 0.66
I work well in groups 0.65
I treat others fairly 0.55
I take time to help others 0.49
I express ideas clearly in oral presentations ‐0.71
I express ideas clearly in written text ‐0.61
I am good at getting ideas across in discussions ‐0.74
I am good at leading others ‐0.60
I like to think of new ways to do things ‐0.67
I am good at coming up with new ideas ‐0.63
I am easily distracted in class 0.68
I tend to be lazy 0.77
I tend to leave things to the last minute 0.78
Cronbach’s Alpha 0.81 0.70 0.82 0.71 0.69 0.74 0.71 0.78 0.77 0.74 0.68 0.70
International Study of City Youth
11 CentreforInternationalResearchonEducationSystems
ItisimportanttonotethatthegoalherewasnottousethePCAtovalidatetheexistingconstructs (such as the Big Five) of 21st Century skills and student engagement anddispositions,butrathertoevaluatethedegreetowhichsetsof itemsusedintheISCYstudentsurveyanddrawnfromlargerinventoriesassessthesameconstructsasthoseusedinlonger,establishedmeasures.Constructvaliditycanbedefinedintermsofthedegreetowhichaconstructshowstheoreticallypredictedpatternsofcorrelationswithother related and unrelated constructs, and the results of the PCA are broadlyconsistent. The reliabilities of the twelve scales were tested using Cronbach’s Alpha,thoughitshouldbestatedthatgoodfitindicesforitemswhicharedesignedtomeasurebroad domainsmay not always generate high alphas. Some researchers point to themisleading place of alphaswhen calculated on scaleswith smaller numbers of items(e.g.Kline,2000;Wood&Hampson,2005).Thissaid,nearlyallscalesreturnedagoodalphaestimate,withmostreturningstrongvalues.Thetestwasalsoconductedforeachscaleforeachoftheindividualcities,withgoodorstrongalphasreturnedforeachscale.
3. Methodforderivingscalescores
OncethetwelvescaleshadbeenidentifiedfromthePCAandconceptualmapping,themethod used to construct the scales from the identified items was the same as thatemployedinthescalingoftheAmericanHumanDevelopmentIndex(seeSocialScienceResearchCouncil,2014).TheHumanDevelopmentIndex(HDI)isasummarymeasureofkeydimensionsofhumandevelopmentand issetonascaleof0 to10. Oneof theadvantagesofusingthisapproachtoscalingisthattheresultscanbecomparedusingaconsistentscalewhichprovidesa simplemeans forcommunicatingresults,aswellasprovidingrobustmeasuresforanalysis.TocalculateeachscaleusingtheHDImethod,theminimumandmaximumvalueswereidentified foreachunderlyingscale item.Performanceineachdimensionisexpressedasavaluebetween0and10byapplyingthefollowinggeneralformula:
Itemvalue= actualscore– minimumvalue X10maximumvalue– minimumvalue
Each scale is calculated by taking the average of the component items transformedusing theabove formula.Sinceall componentsrange from0 to10,eachderivedscalealsovariesfrom0to10,with10representingthehighestlevelofskill,engagementordisposition. An example is provided based on the items classified as part of the‘conscientiousness’scale.Example:calculatingtheConscientiousnessIndexTheConscientiousnessIndexismadeupoffiveitems:
1. Hoursofhomework2. Ialwaystrytodomybest3. Ialwaysgetworkinontime4. Iperseverewithajobuntilitisdone5. Iamahardworkingstudent
International Study of City Youth
12 CentreforInternationalResearchonEducationSystems
The Index isderivedby calculating thevalues for each scale itemusing the followingmethodwhereastudenthasrespondedwiththevaluesof3,2,3,2,and3tothe5items:
Item1= actualscore– minimumvalue 3– 1 X10 =6.67maximumvalue– minimumvalue 4– 1
Item2= actualscore– minimumvalue 2– 1 X10 =2.50maximumvalue– minimumvalue 5 – 1
Item3= actualscore– minimumvalue 3– 1 X10 =5.00maximumvalue– minimumvalue 5 – 1
Item4= actualscore– minimumvalue 2– 1 X10 =3.33maximumvalue– minimumvalue 4– 1
Item5= actualscore– minimumvalue 3– 1 X10 =6.67maximumvalue– minimumvalue 4– 1
TheIndexiscalculatedbytakingthesimpleaverageofthefiveitemscores.Sinceallfivecomponents range from 0 to 10, the Index itself also varies from 0 to 10, with 10representingthehighestlevelofself‐efficacy.Usingtheexampletheresultwouldbe:Mean(item1,item2,item3,item4,item5)=Mean(6.67+2.50+5.00+3.33+6.67)=4.83ThefinalsetofconstructswiththeirmeansandstandarddeviationsforallstudentsarereportedinTable2.Table2 Finallistofscaleconstructswithmeansandstandarddeviations
Scale N Minimum Maximum Mean Standard Deviation
Behavioural Engagement 26720 0 10 7.9 2.0
Belonging 26772 0 10 6.6 1.8
Collaboration 25866 0 10 7.1 1.5
Creativity 25766 0 10 6.3 2.0
Effort 26399 0 10 6.0 2.5
Interest 26769 0 10 5.7 1.8
Hope 26529 0 10 6.6 1.9
Purpose 26667 0 10 6.6 2.0
Self‐efficacy 25860 0 10 6.4 1.9
Self‐control 25817 0 10 4.1 2.1
Communication 25843 0 10 6.2 1.8
Conscientiousness 26757 0 10 6.3 1.7
International Study of City Youth
13 CentreforInternationalResearchonEducationSystems
4. MissingvaluesForeachofthescales,amajorityofstudentshaddataforallthesub‐itemsasshowninTable3.
Table3 Percentageofstudentswithresponsestoallitems,byscale(%) %studentswith
allitems %studentswith
allitemsBehavioural Engagement 93.1 Hope 92.4Belonging 93.2 Purpose 92.9Collaboration 90.1 Self‐efficacy 90.1Creativity 89.7 Self‐control 89.9Effort 91.9 Communication 90.0Interest 93.2 Conscientiousness 93.2Cases thathadmissingvalues forall49 scale itemswereremoved fromanalysis.Thenumberofcasesremovedforeachcity,andtheadjustednumberofcases,arelistedinTable4.Table4 Numberofmissingcasesforallscaleitems
City Originaln Missingall % Adjustedn
Melbourne 4,781 123 2.6% 4,658
Bergen 2,329 140 6.0% 2,189
Montreal 4,119 323 7.8% 3,796
Ghent 2,354 48 2.0% 2,306
HongKong 6,315 371 5.9% 5,944
Barcelona 2,128 78 3.7% 2,050
Wroclaw 2,556 134 5.2% 2,422
SantaBarbara 792 52 6.6% 740
Sacramento 2544 239 9.4% 2,305
Total 24,582 1217 5.0% 23,365
Where a student was missing a single item on a scale, a score was imputed for themissingitem.Thisvaluerepresentedthemedianscoreonthatitemforstudentswhosescoreontheremainingitemsforthatscalewasequivalenttothescoreforthestudentforwhomtheitemwasmissing.Wheremorethanoneitemwasmissingforascale,thestudentwasassignedamissingvalueforthatscale.
International Study of City Youth
14 CentreforInternationalResearchonEducationSystems
TheISCYFrameworkof21stCenturyskills,dispositionsandengagementThetwelvescalesandtheircomponentitemswereanalysedconceptuallytoassesstheirface validity as measures of relevant constructs. Most of the twelve scales and theircomponent items aligned with skills and mindsets identified in other surveys andmodels including PISA (sense of belonging, behavioural engagement), the EducationLongitudinalStudyof2002(behaviouralengagement,cognitiveengagement,emotionalengagement),theGallupStudentPoll(hope,belonging),theBigFive(conscientiousness,openness,perseverance)andtheUCCCSRstudentsurvey(belonging,self‐efficacy,hopepurpose).Where thealignmentof ISCY componentswithexisting scaleswasnot self‐evident,thetaxonomyof21stcenturyskills(Figure1)andotherliteraturewasusedtoidentify the constructs measured by ISCY. These constructs were then frameddiagrammatically.Figure3presentsanoutlineoftheISCYframeworkfor21stCenturyskills,dispositionsandengagement,basedonthisconceptualandempiricalmapping.ISCYaimstoexplorehow system‐level factors in each city interact with school‐level and student‐levelfactors, to shape educational outcomes in each of the 12 cities. The city context caninfluencestudentoutcomesdirectlyandindirectly,includingthroughmediatingfactorsat the school and student level. The framework shows how the various 21st centuryskills, dispositions and engagement measured in ISCY may be situated as potentialmediating factors between contextual and system factors, and student performance.Analysis of these ISCY measures must therefore be situated within contextual andsystemfactors,andlinkedtotheoutcomesforstudentsthatsystemsaimtoachieve.Shapedand influencedby the city context (socio‐cultural, economic andpolitical, andeducation system), are fourdispositions:belonging (I belonghere), self‐efficacy (I cansucceed),hope(Iwill findaway)andpurpose(Ivaluewhat Ido).Thesedispositionsarepositionedasinfluencesonengagement(cognitive,behaviouralandemotional),aswellasontheskills(non‐cognitiveinterpersonalandintra‐personal,andcognitive).Theskills in conjunction with dispositions and engagement mediate the effects of citycontextandultimatelyinfluenceacademicperformance.The ISCY frameworkrepresentsahypothetical framework forhow21st centuryskills,dispositions and engagement influence educational success. Further analysis on therelationship between the twelve ISCY scaleswill test these assumptions, and identifyhow the skills, and measures of engagement and dispositions interact in shapingacademic performance. This could be done with SEM that not only measures theconstructs,butidentifiesthestructuralrelationshipamongthemandtheirrelationshiptoacademicperformance.AnalysisacrossISCYcitieswillhelpidentifytheeducationsystemsinwhichtheseskillsarebestsupported,andthetypesoflearnerswhoaremostlikelytodevelopthem.
15 CentreforInternationalResearchonEducationSystems
Figure3 ISCY Framework for 21st century skills, dispositions and engagement, showing relationship to city context and academicperformance
16
Acknowledgements
This paper uses data from the International Study of City Youth (ISCY). ISCY is aninternational collaborativeprojectdesigned and implementedby various researchpartnersfromacrosstheworldandledbytheCentreforInternationalResearchonEducationSystems(CIRES) at Victoria University, Australia. ISCY has received funding from the AustralianResearch Council, the Victorian Department of Education and Training and CIRES. Weacknowledge the support and contribution made by all students, teachers and schoolsparticipatingintheproject.
References
Abdi, H & Williams, L. (2010). Principal component analysis. Wiley Interdisciplinary Reviews:ComputationalStatistics.Volume2,Issue4,Articlefirstpublishedonline:30JUN2010.
Borghans, L, Duckworth, A, Heckman, J, & ter Weel, B (2008) The Economics and Psychology of
PersonalityTraits.NBERWorkingPaperNo.13810IssuedinFebruary2008.Downloadedfrom:http://www.nber.org/papers/w13810
Child Trends (2014).Measuring Elementary School Students’ Social and Emotional Skills: Providing
EducatorswithToolstoMeasureandMonitorSocialandEmotionalSkillsthatLeadtoAcademicSuccess. Retrieved 15 January 2015, www.childtrends.org/?publications=measuring‐elementary‐school‐students‐social‐and‐emotional‐skills‐providing‐educators‐with‐tools‐to‐measure‐and‐monitor‐social‐and‐emotional‐skills‐that‐lead‐to‐academic‐success
Cress, C.M. (2012). Civic Engagement and Student Success: Leveraging Multiple Degrees of
Achievement.Diversity&Democracy,15(3),pp.2–4.Farrington,C.A.,Roderick,M.,Allensworth,E.,Nagaoka,J.,Keyes,T.S.,Johnson,D.W.,&Beechum,N.O.
(2012). Teaching adolescents to become learners. The role of noncognitive factors in shapingschool performance: A critical literature review. Chicago: University of ChicagoConsortiumonChicagoSchoolResearch.
Gosling, S, Rentfrow, P., & Swann, W. A very brief measure of the Big‐Five personality
domains.JournalofResearchinPersonality37(2003)504–528.Gutman,L.M.andSchoon, I.(2013).The impactofnon‐cognitiveskillsonoutcomes foryoung
people:Literaturereview.London:InstituteofEducation.John,O.P.,&Benet‐Martínez,V.(2000).Measurement,scaleconstruction,andreliability.InH.
T. Reis & C. M. Judd (Eds.), Handbook of researchmethods in social and personalitypsychology(pp.339–369).NewYork:CambridgeUniversityPress.
Jordan,W.J. (2010), ‘DefiningEquity:MultiplePerspectives toAnalyzing thePerformanceof
DiverseLearners’,ReviewofResearchinEducation,Vol.34,pp.142–178.Kautz, T, Heckman, J, Diris, R., ter Weel, B, & Borghans, L. Fostering andMeasuring Skills:
ImprovingCognitiveandNon‐CognitiveSkillstoPromoteLifetimeSuccess.,NBERWorkingPaper No. 20749 Issued in December 2014. Downloaded from:http://www.nber.org/papers/w20749.
17
Kline,P.(2000).HandbookofPsychologicalTesting.London,Routledge.Levin, H. (2012)More than just test scores. Prospects Quarterly Review of Comparative Education,
Interantional Bureau of Education. Published Online 2012:https://roundtheinkwell.files.wordpress.com/2012/09/more‐than‐just‐test‐scores‐sept2012‐2.pdf
Pellegrino, J.W. andHilton,M.L. (Eds.) (2012).Education forLifeandWork:DevelopingTransferable
Knowledge and Skills in the 21st Century. National Research Council. Washington, D.C., TheNationalAcademiesPress.
Pintrich,p.R.,Smith,D.A.,Garcia,T.,&McKeachie,W.J.(1993).Reliabilityandpredictivevalidityof
the motivated strategies for learning questionnaire (MSLQ). Educational and PsychologicalMeasurement,(3).801.
Rotermund,R.(2010)Theroleofpsychologicalantecedentsandstudentengagementinaprocessmodel
ofhighschooldropout.Doctoraldissertation.UCSB.Seligman,M.P.(2012).Flourish.NorthSydney(NSW):WilliamHeinemannAustralia.Shectman,N.,DeBarger,A.H.,Dornsife,C.,Rosier,S.andYarnall,L. (2013).PromotingGrit,Tenacity,
and Perseverance: Critical Factors for Success in the 21st Century. Draft report for the USDepartment of Education. Retrieved 15 January 2015, http://pgbovine.net/OET‐Draft‐Grit‐Report‐2‐17‐13.pdf
Silva, E. (2008),Measuring skills for the21st century,WashingtonDC,Education Sector. Retrieved 6
December2013,www.educationsector.org/usr_doc/MeasuringSkills.pdfSocialScienceResearchCouncil(2013)MethodologicalNote:TheAmericanHumanDevelopmentIndex.
Accompanying Measure of America 2013‐2014. Downloaded fromhttp://www.measureofamerica.org/Measure_of_America2013‐2014MethodNote.pdf
Taylor, L. & Parsons, J. (2011). Improving Student Engagement. Current Issues in Education, 14(1).
Retrievedfromhttp://cie.asu.edu/Trilling,B. andFadel,C. (2009).21stCenturySkills:Learning for life inour times. SanFrancisco,CA,
JohnWiley&Sons.Turiano, N.A., Mroczek, D.K., Moynihan, J. and Chapman, B.P. (2013). Big 5 personality traits and
interleukin‐6: Evidence for “healthy Neuroticism” in a US population sample.Brain,Behavior,andImmunity,28,PP.83–89.
Wood,S.A.,&Hampson,S.E.(2005).MeasuringtheBigFivewithsingleitemsusingabipolarresponse
scale.EuropeanJournalofPersonality,19,373‐390.
Recommended