Embed Size (px)
Citation preview
CORECOMPETENCIES(CCs)FORLEADERSINBIOMEDICALRESEARCH
SampleRubric: Fullrubriccopiesavailable,onpaper.
2016RATINGS
2015RATINGS
ASSESSMENTMUSTBEBASEDONOUTCOMES:• Asstudentsareintegratedintoscience,iterativeandoutcome-
orientedfeedbackisimperative.² Formativeandsummativeassessmentsareneeded² Skill-basedperformanceassessmentdifferentiatestudent
progressiontodegree² Suchassessmentsvalidatehabitsthatarerootedinthe
behaviorsofsuccessfulstudentsandshouldbetransparenttoallstudents.
FACULTYDEVELOPMENTNEEDS:• Facultymustbetrainedinitially,andthendevelopedtoassess
appropriatelyandconsistently.
• Involvementoffacultyindevelopinginstitution-specificCoreCompetenciesiscriticalforunderstandingandcommitment.
STUDYDESIGN:• Thistypeofoutcomes-orientedsurveysinrepeatedmanner
withanalysisbenefitsdesign(andrevision)ofeffectiveprograms.
• Longitudinaldataarerequired,withrepeatedmeasuresduetothecomplexityofadmissions,studentvariability,andinter-raterreliability.
Studentsbecomingprofessionals:transparentassessmentusingCoreCompetenciesJulieDavisTurner,PattyFarrell-Cole,andStevenJ.Triezenberg
VanAndelInstituteGraduateSchool,333BostwickAve.,NE,GrandRapids,Michigan49503
Background&Rationale
REFERENCESAppreciationtotheseresourcesaswedevelopedtheframeworkforCoreCompetencies:1. Walker,G.E.,C.M.Golde,L.Jones,A.C.Bueschel,P.Hutchings.(2008)TheFormationofScholars(San
Francisco:Jossey-Bass).2. Lovitts,B.(2007)Makingtheimplicitexplicit:creatingperformanceexpectationsforthedissertation.
(Sterling,VA:StylusPress).3. NRC.(2001)Knowingwhatstudentsknow.(cited1/20/2017)
http://www.nap.edu/openbook.php?isbn=03090727274. CouncilofGraduateSchools,Assessment&ReviewofGraduateProg,1990,2005,2011.
http://cgsnet.org/assessment-and-review-graduate-programs
Evidenceofprogressionacrosscohorts• DomainsofResearch &Ethic/ProfessionalPracticeshowthelargest
increaseindevelopmentacrosscohorts.
• DomainsofKnowledge andCommunication demonstratelessevenlevelsofmastery,possiblyduetoindividualcohortattributes.
EDUCATIONALPHILOSOPHY• Interactive,problem-basedcurriculumgivesstudentsresponsibilityforlearning.• Proposal-drivenproblemsmeasurecontentmastery,creativity,andscientific
communication.• Likescience,optimallearningrequirescollaborationamongstudentsandfaculty.
WHATWELEARNED
MISSIONThemissionofVAIGSistotrainPhDscientiststobeleadingscholarsincellular,molecular,andgeneticbiologyofhumandisease.
Claim1:CCassessmentidentifiesadevelopmentalprogression.
Claim2:CCsdefinemasterybetweenthesisadviserandstudent.
Claim3:EvidencefromannualCCassessingdrivesprogramchanges.
CONCLUSIONS
• CoreCompetenciesarticulateastrongsetofexplicitexpectationsforstudentlearningoutcomesatVAIGS.
• Therubric,asadevelopmentalframework,aidsthesisadvisersinmonitoringandunderstandingtheirstudents’progressandfacilitatesstudentaccountabilitywithcommonexpectations.
• Therubricprovidesfacultywithlanguagetoconveyclearexpectationsandgraduatedratingsforobjectiverationale.
VAIGScohortsincreaseinmasteryoffourCoreCompetencyDomains(Knowledge,Research,Communications,Ethics&ProfessionalPractice),assessedbyThesisAdviser(2016averages;cohort=2-4students,n=18total)
Exceptional
Heightened
Advancing
Intermed.
Beginning
Exceptional
Heightened
Advancing
Intermed.
Beginning
cohort
Basedon“expectedoutcomes”(setpercohort):• By6th Year,allstudentshaveachievedHeightenedto
Exceptional(asanticipated– showninWHITE).• IsolatedareasofClinicalContext(Knowledge),Scientific
Ethics,orAuthorshipshowrepeatedlags.
ComprehensiveProgramReviewrecommendations(2016)employedthesepreliminarydatatoaddresslags:• Curricularmodificationsinwrittencommunicationand
grantsmanship• Revitalizedcoursein“translationalresearch”• Refreshercourseforscientificethics
Futurefacultycommitteemandateswillbeinformedby2017-18studydata.
Takentogether,CoreCompetencyoutcomesprovidesareal-time“dashboard”forprioritizingprogramchanges.
VAIGSCORECOMPETENCIESv2Thestudentcananddoes:
©VanAndelInstituteGraduateSchool2015,version2 CoreCompetencies-122915v2.docx
KNOWLEDGE Beginning Intermediate Advancing Heightened ExceptionalDescribekeyconceptsinbiomedicalscience
Describegeneralconceptswithgapsanderrors
Describegeneralconceptscorrectlyandclearly
Explainconceptswithexpandedhistoricalandcontextualdetail
Recognizeandapplykeyconceptsastheyappearinnewsettings
Evaluatenewchallengesorcontroversiesrelevanttoconcepts
Placecoreconceptsintherelevantclinicalcontext
Describethesisresearchanddiseaseassociationingeneralandlimitedterms
Describeclinicalcontextofmolecularmechanismsaspresentedbyothers
Connectclinicalissuesandmolecularmechanismsaccuratelyandbroadly
Recognizeandapplykeyconceptsastheyappearinnewclinicalsettings
Useclinicalinformationtoelicitgapsincollectiveknowledge
Knowscientificliteraturerelevanttotheresearcharea
Understandbiomedicaltextbookdescriptionsofkeyconcepts
Understandreviewarticledescriptionsofemergingconcepts
Understandandevaluateaprimaryresearcharticle
Comparecontradictorypublicationsinthedissertationresearcharea
Evaluateliteraturebeyondthedissertationresearcharea
RESEARCH Definesoundrationale/identifygapinknowledge
Performexperimentsasinstructedwithoutconsideringrationale
Understandrationaleprovidedbyothers
Userationaletojustifyprioritiesfordailywork
Buildexpandedrationaleforprojectandexperiments
Effectivelydefendgivenrationaleagainstobjectionsoralternatives
Frameappropriatehypothesis
Accepthypothesisfromothers Contributetoformationofhypotheses
Generateindependentbutincompletehypotheses
Independentlygeneratetestablehypothesescoupledwithspecificaims
Recognizealternativeandtestablehypotheses
Applycreativeandappropriateexperimentaldesign
Executeexperimentsdesignedbyothers
Collectandemployappropriateapproaches
Seekexpertadvice&gleannovelapproachesfromliterature
Criticallyevaluateandmodifyapproaches
Inventandapplyalternativestrategiesandtroubleshootspotentialoutcomes
Usecontrolsappropriately Performcontrolsasinstructedbyothers Usecontrolsformostapproaches Incorporatebothpositiveand
negativecontrolsconsistently
Applyrigorouscontrolsanduseoutcomestoredirectexperimentalpath
Analyzeunexpectedresultsandallownewdiscoveriestoemerge
Executeexperimentswithtechnicalskill
Receiveinstructionandpracticenewskillswithsupervision
Exhibitappropriateexperimentaltechniqueandseekadvicewhenexperimentsfail
Anticipateresultsandobtainreproducibleresults
Mastertechnicalunderstandingandtroubleshootexperimentalfailuresindependently
Generatehightechnicalproductivity,publication-qualityfigures;directothersintechnique
Demonstratecriticalanalysisandthinking
Acceptcurrentmodelsasaccurateandsufficient
Describeexperimentsthatgeneratedcurrentmodel
Observelimitsofcurrentmodels;identifygapsinknowledge
Comparecontraryresults,yieldingmultiplenewhypotheses
Defendpositionorhypothesis-drivenexperimentstoexpandthefieldofstudy
Integrateresultsintorelevantmodels
Acceptonlythoseresultsconsistentwithhypothesis.
Comparenewdatawithpriorresultsfromself,samelab,otherlabs
Elucidatelogicalflawsincurrentmodelsbasedonexperimentaldata
Reviseextantmodelsbasedonnewdatafromselforothers.
Createnewmodelsthatintegrateexperimentalfindingswithexternaldata;Critiquedataandanalysisofothers
COMMUNICATION Beginning Intermediate Advancing Heightened Exceptional
SpeakeffectivelyPlanandpresentshortoralsummariesinlabmeetingsormentormeetings
Planandpresentjournalcluborresearch-in-progresstalksforlaborcenterusingsimpleorpre-setfigures
Presentclear,well-organizedresearchandjournalclubtalksorposterswitheffectiveuseofschematicanddatafigures/conclusions
Presenteffectiveposterorplat-formpresentationsatconferences;handlequestionsconciselyandeffectively;managelogisticsanddisruptions
Elegantcommunicationinmultiplesettings(conferences,seminars,classroom,massmedia,adhoc).
Writeeffectively Writesentences,paragraphs,and Employstandardstructuresof Buildcoherentandlogicalargument Presentcomplexmodelsorrationale Presentcomplexmodelsorproposals
Cohortratingsdeterminecurricular“gaps”.• ThesisAdviserratingswereaveragedandcomparedto
“expectedoutcomes”foreachcohort• Thosecompetenciesinwhichratingslagareshown:
ValueofcomparingStudenttoThesisAdviserratings• Clarifiesmentorexpectationsiniterativemannerwithconsistentlanguage• Identifiesmiscommunicationsinthementor-studentrelationship• Providestangible,consistentbehaviorsthatallstudentscanworktowardcoordinately
Concordanceequals:average [(faculty scorex ) – (student scorex )]
across domainaverage [(faculty scorex ) – (student scorex )]
across domain
HistoryofCoreCompetenciesatVAIGS
18students 18students
VAIGSCoreCompetencies–Two-yearstudy
page2
©VanAndelInstituteGraduateSchool2017
Claim2.CoreCompetenciesdefinemasteryinformativemannerbetweenthesisadviserandstudent.
InordertodeterminetheeffectCore
Competencieshaveontheunderstandingof
masteryinscience,wecomparedthe
concordanceofscoresfromfacultytostudent.
Concordanceequals:
average[(facultyscorex)–(studentscorex)]acrossdomain
• Concordanceatzeroindicatesthatthestudentandfacultywereinagreement.
• Positiveconcordanceindicatesthatmean
facultyratingofstudentmasterywashigher
thanmeanstudetrating.
Figure2showshighestconcordanceoverallfourdomainsfor6thyearor2ndyearcohorts.
Discordanceishighestamong4thand5thyear
studentswherestudentratingsareconsistently
higherthanmentorratings.
Claim3.EvidencefromCoreCompetenciesannualassessmentdriveVAIGSprogramchanges.
ToidentifyCoreCompetencieswhere
VAIGS’programmaynotadequately
preparestudents,thesisadviserratings
wereusedtodeterminewhetherVAIGS
studentsexcelorlagindevelopmentof
specificcorecompetencies.
Thedataindicatethatby6thyear,all
studentshaveachievedHeightenedtoExceptionalratingsfortheResearchdomain.
However,furtherprofessionaldevelopment
isneededinEthicsandProfessionalPractice.
CONCLUSIONSCoreCompetenciesarticulateastrongsetof
explicitexpectationsforstudentlearning
outcomesinourgraduateprogram.The
rubric,asadevelopmentalframework,aids
thethesisadviserinmonitoringand
understandingtheirstudent’sprogressandfacilitatesaccountabilityforthestudent.
CONTACTSJulieDavisTurner,PhD PattyFarrell-Cole,PhD SteveTriezenberg,PhD
Co-organizer;AssociateDean Co-organizer;EvaluationSpecialist Dean,VAIGS234-5596;[email protected] 234-5363;[email protected] 234-5708;[email protected]
Figure2.DisparitybetweenFacultyandStudentRatings.ConcordancewascalculatedforeachCoreCompetencywithineachdomainand
averagedoverthecohort(keyshownatright).
Table1.UsingCoreCompetenciesforProgramReviewbyevidenceofLags
Domain/CoreCompetencies 2ndyr
3rdyr
4thyr
5thyr
6thyr
Knowledge(3totalcompetencies)Describekeyconceptsinbiomedicalscience
Knowscientificliteraturerelevanttoresearcharea
Placecoreconceptsintherelevantclinicalcontext
Research(7totalcompetencies)Applycreative/appropriateexperimentalcontrols
Usecontrolsappropriately
Integrateresultsintorelevantmodels
Communications(3totalcompetencies)Speakseffectively
Writeseffectively
Ethics&ProfessionalPractice(6competencies)Managedatawithscientificintegrity
Engageinbestauthorshippractices
Addressethicalproblemsinscientificresearch
Complywithsafety®ulatorystandards
Displayappropriatelabcitizenship
Workcollegially&effectivelyasa
team/collaborator
LEGEND:LAGSin2015LAGSin2016 Lagsinboth2015and2016
KnowledgeResearch CommunicationEthics/
(3competencies)(7competencies)(3competencies)Prof.Practice
(6competencies)
Method:deidentified datacollection(groupedbycohort)• UsingSurveyMonkey© anduniqueidentifer codesforstudentsandmentors,VAIGSgathersanddeidentifies ratings,maintainingmentor-studentassociation.
Higherm
entor
Higherstud
ent
-1.2
-0.9
-0.6
-0.3
0.0
0.3
0.6
0.9
1.26thYear(n=2)
5thYear(n=4)
4thYear(n=3)
3rdYear(n=4)
2ndYear(n=4)
CommunicationsResearch Ethics&ProfessionalPracticeKnowledge
Concordance
POSTER#11
Whileearlyinthestudy,preliminarydatashow:• Lackofconcordance(in2015)necessitatedrelianceonmentorratings.• Cohort-specificattributesmayskewconcordance.• Improvedconcordanceoccurredin2016relativeto2015.
– Thisimprovementmaybeduetoenhancedrubricfamiliarity.
Growthovertime (limiteddisplay:only2longitudinaldatasets)
• Formostofdomains,mentorratingsincreasefrom2015to2016.• Ratingsarecomplicatedbyissueswithtimingandconsistency
5th –6th year
2nd –3rd year
-1.2
-0.9
-0.6
-0.3
0.0
0.3
0.6
0.9
1.26thYear(n=3)
5thYear(n=3)
4thYear(n=4)
3rdYear(n=4)
2ndYear(n=4)
Research(7Competencies)
Communica/ons(3Competencies)
Knowledge(3Competencies)
Ethics&ProfessionalPrac/ce(6Competencies)
Concordance
Higherm
entor
Higherstud
ent
Communicationü SpeakeffectivelyüWriteeffectivelyü Communicatetodiverseaudiences
Ethical&ProfessionalPractice
üManagedatawithscientificintegrityü Engageinbestauthorshippracticesü Addressethicalproblemsinscientificresearchü Complywithsafetyandregulatorystandardsinlabü DisplayappropriatelabcitizenshipüWorkcollegiallyandeffectivelyasateamor
collaborator
Summer2015
2010- 2014Intermittentusebyfaculty/students
February2016
December2016
RevisionofCCrubricbyfacultysubcommittee
1st CoreCompetencyAssessment
(100%compliance)
2nd CoreCompetencyAssessment
(100%compliance)
Knowledgeü Describekeyconceptsinbiomedicalscienceü Placecoreconceptsintherelevantclinicalcontextü Knowscientificliteraturerelevanttotheresearcharea
Researchskillsü Definesoundrationale/identifygapinknowledgeü Frameappropriatehypothesesü Applycreativeandappropriateexperimentaldesignü Usecontrolsappropriatelyü Executeexperimentswithtechnicalskillü Demonstratecriticalanalysisandthinkingü Integrateresultsintorelevantmodels
