Measuring and Comparing Achievements of Learning Outcomes in HE in Europe

RobertWagenaarOverallCoordinatorCALOHEE

DirectorInternationalTuningAcademy

ECAworkshopandAnnualMeetingHCERES– Paris22-23June2017

CALOHEE:WHY?

Preposition: If academic experts can agree on the set of learning outcomes, they should also be able to measure performance in comparative perspective in (inter)national contexts!

THE PROOF IS IN THE EATING OF THE PUDDING !

Do students enrolled in higher education around Europe develop thecompetences they need? Are study programmes delivering theirpromises? Can we learn from comparing students’ achievements inan (inter)national context in a meaningful way?

Measuring and Comparing Achievements of Learning Outcomes in HE in Europe

WHY? HOW?WHAT?

WHY?ØMeetingthedemandformorereliableinformationabouttheoutcomesoflearninginHE

ØOfferingadriveforquality,takingintoaccountfullytheneedsofsociety,inparticularthefourmajorstakeholders:HEstudents,HEstaffandmanagement,employersandemployees,andcivilsociety

ØAnattempttocreate(inthelongerrun)amoreeffective,lessbureaucraticandmorereliableinstrumentforqualityassurance!

CALOHEE:MeasuringandComparingAchievementsofLearningOutcomesinHigherEducationinEurope

WHY? HOW? WHAT?

HOW?Ø Showingwhatasubjectareadoesrepresentafterconsultationwithstakeholders,intermsofcorecompetencesandlearningoutcomesaccordingtothediscipline

ØDevelopinginstrumentsthatacknowledgethedifferentmissions,profilesandculturalcontextsofHigherEducationinstitutions

CALOHEE:MeasuringandComparingAchievementsofLearningOutcomesinHigherEducationinEurope

WHY? HOW?WHAT?

WHAT?ØOfferingsubjectareabasedagreedframeworksofdescriptorswhichallowforcomparingandmeasuringqualityofperformanceandoutcomessetinthecontextoftheinstitution

ØOfferingadetailedsubjectareabasedagreedAssessmentFrameworkwhichalignsLearningOutcomes,ActivitiesandAssessments

ØOfferinganinstrumentwhichallowsfordiagnosticassessmentaboutmeetingagreedqualityrequirements,societalneedsandtheparticularmissionoftheprogrammeinvolved

CALOHEE:MeasuringandComparingAchievementsofLearningOutcomesinHigherEducationinEurope

CALOHEEProjectaims

Ø BENCHMARK1:Updatedtransnationalsectoral/subjectarea frameworks(finalisedafterConsultationProcess)

Ø BENCHMARK2:

Multi-dimensionalinstrumenttomeasureandcomparelevelsoflearningdoingjusticetothedifferentmissions,orientationsandprofilesofHEinstitutions

Ø BENCHMARK3:

AssessmentFrameworks forfiveacademicdomainsandfiverelateddisciplines(CivilEngineering,Nursing,History,EducationandPhysics)(finalisedafterConsultationProcess)

Theseinstrumentsarethefoundationsfordevelopinganassessmentinstrumentthatshouldallowfordiagnosticassessments(servingteachingstaff,students,managementandsociety)

OverallobjectiveofTUNING: HighQualityandRelevantHigherEducation

Engineering/CivilEngineering

SocialSciences/Education

Humanities/History

CoreCompetences(SubjectSpecificandGeneric)/LearningOutcomesDescriptors

Comparisonoflearning(gain)intrans-institutionalandtrans-nationalperspective

FiveAssessmentFrameworks,covering:

Irreduciblecore:1.Knowledge:theory+methodology2.Applicationofknowledgeandskills3.Employability4.Civic,SocialandCulturalEngagement

HealthCare/Nursing

NaturalSciences/Physics

CoreCompetencesetc.

CoreCompetences,etc.

CoreCompetences,etc.

CoreCompetences,etc.

CALOHEEFLOWCHART

DiagnosticAssessment

CommonFramework:

Acknowledgedifferentmissions,profilesandculturalcontextsofHEinstitutions

Itoffersagreedindicatorsof:Ø LevelØ ContentØ Relevance

EQF / QF for EHEA

Descriptors

TUNING Sectoral Reference Points

TUNING Subject Specific Frameworks: Reference Points

Tuning Sector / Subject Area Based Assessment Frameworks

CALOHEEisbasedonamergeroftwoEuropeanQualificationsFrameworks

Recognisingdifferentmissions,profilesandculturalcontextsofHEinstitutions

Approachallowsfor(trans)national/institutionalcomparisonfortheachievementoflearning(gain)

Profiles of individual degree programmes

Flowchart2

July12,2017 Slide10

Knowledge SkillsWiderCompetencesAnExample

Dimen

sions

KeyDescriptorsofasingleacademicfieldononepage(basedonamergeroftheEQFforLLLandtheQFfortheEHEA)

Formulatedescriptorsaslearningoutcomes:useactiveverbsinaconsistentway,butalsodefinelevelintermsofscopeandcomplexity(averbisaninsufficientindicatorforidentifyingthelevel)

PROGRESSIONOFLEARNING:knowledge– skills–application- contribution

KNOWLEDGE SKILLS WIDERCOMPETENCES

LEVEL/GROWINGCOMPLEXITY

Common body of knowledge, skills and wider competences for the subject area

Employability Civic, social and cultural engagement

Knowledge: theory and

methodology

Application knowledge and skills

PARAMETERS / CATEGORIESEQF:Knowledge Skill WiderCompetences

1

2

QFforEHEA

DIM

ENSI

ONS

34

ETC.

CALOHEE ASSESSMENT MODEL

Core Competences (Subject Specific and Generic) / Learning Outcomes

DIAGNOSTICASSESSMENTS

Assessment Frameworks based on parameters and dimensions

Frameworkacknowledgesdifferentmissions,profilesandculturalcontextsofHigherEducationinstitutions

A dimension indicates a constructive key element which defines a subject area; each subject area is based on a multiple of dimensions

DIM

ENSI

ONS

Diagnosticassessmentsinformstakeholdersaboutmeetingagreedqualityrequirementsandthemissionoftheprogramme

Flowchart3

LinkingLearning,TeachingandAssessmenttosetsofbroken-downSubjectAreaCompetences/LearningOutcomes

Thetotalityofdimensionsreflectsadetaileddescriptionofwhataprogrammecancontain,leadingtolearningalignment:LearningOutcomes,ActivitiesandAssessments

Adimensionindicatesaconstructivekeyelementwhichdefinesasubjectarea;eachsubjectareaisbasedonamultipleofdimensions

Flowchart4

Anysuggestionsforimprovement/simplification?

CALOHEEframeworkforcivic,socialandculturalengagement

Fourparameters(basedonanalysisofpresentdevelopmentsandrecentliterature):

• SocietiesandCultures:Interculturalism andconflictmanagement

• Processesofinformationandcommunication• Processesofgovernanceanddecisionmaking• Ethics,norms,valuesandprofessionalstandards

CALOHEEframeworkforCivic,socialandculturalengagementKnowledge Skill Widercompetences

1. Demonstratecriticalunderstandingofdifferencesinandbetweensocietiesandcultures(frames)

Identify,describeandanalyseissuesinandbetweensocietiesandcultures

Demonstrateengagementbydevelopingscenariosandalternativesforidentifyingbestpracticesandinterventionsinthecaseoftensionsandconflicts

2. Demonstratecriticalunderstandingoftheprocessesofinformationandcommunication

Reviewandjudge(mis)useofsources,data,evidence,qualities,intentionsandtransparencyandexpertopinions

Activecontributiontosocietaldebatesusingreliabledataandinformationsourcesandinformedjudgements

3. Demonstratecriticalunderstandingoftheprocessesofgovernanceanddecisionmaking

Applyandsupportagreedgoverningprinciples,normsandvaluesregardingfairness,transparency,accountability,democracyandrelevanceinpolicymakingprocesses

Activecontributiontoandwithlocaland(inter)nationalcommunities,communitygroups,(political)organisationsandpressuregroupsrespectingagreedprinciples,normsandvalues

4. Demonstratecriticalunderstandingofgeneralethicalprinciples,normsandvaluesandprofessionalstandards

Understandandapplytheprocessesofdecisionmakingandtheconsequencesofactionstakingintoaccountprinciples,norms,valuesandstandardsbothfromapersonalanda

Activecontributiontoupholding,promotinganddefendinggeneralethicalprinciples,norms,valuesandprofessionalstandardsingovernance,communicationandculturalinteraction.

CALOHEEframeworkforcivic,socialandculturalengagement

Nextsteps:Ø Addingsub-dimensionstofourdimensionstooffermoreprecisedescriptorsØ IntegratingframeworkdescriptorsinsubjectareaconceptualframeworksØ IdentifyinggapsØ Updatingandfinalising CALOHEEframeworkforcivic,socialandcultural

engagement asareferenceforallacademicfields

Next steps

Frameworksoffermeasurableindicatorsfordevelopingassessmentitems/assessmentbanksfordiagnosticassessments

Conditionsforsuccessfultrans-institutionalandtransnationalassessments:

ØShoulddojusticetoindividualacademicdomains/subjectareasØShouldallowforassessingunderstandingandapplicationØShouldbemulti-dimensional:coveringall4parameters/categoriesoftheassessmentmodel,aswellastheidentifieddimensionsforeachsubjectareainvolved

ØShouldallowfordifferentiationintheweightingofsub-scoresfordifferenttypesofdegreeprograms(researchdriven/applied,etc.)followingthemulti-dimensionalapproach;accordingtotransparentandagreedcriteria

ØShouldbemachinebased,butgo(far)beyondmultiplechoice(makinguseofalgorithms?)

Ø AllowfortestingprofoundknowledgeandunderstandingandhighlevelskillsØ Allowfortestingcriticalawarenessandthinking,analyzing(textinterpretation)and

composing

CALOHEE-WEBSITE:https://www.calohee.eu