• Home

  • Documents

  • Learning Outcomes A: Best Practices in …Design assessment approaches that generate actionable...
46
Assessing Student Learning Outcomes A: Best Practices in Assessment and Accreditation NATASHA JANKOWSKI, PHD DIRECTOR, NATIONAL INSTITUTE FOR LEARNING OUTCOMES ASSESSMENT

Learning Outcomes A: Best Practices in …Design assessment approaches that generate actionable evidence about student learning that key stakeholders can understand and use to improve

  • Upload
    others

  • View
    2

  • Download
    0

Embed Size (px)

Citation preview

Page 1: Learning Outcomes A: Best Practices in …Design assessment approaches that generate actionable evidence about student learning that key stakeholders can understand and use to improve

AssessingStudentLearningOutcomesA:

BestPracticesinAssessmentandAccreditation

NATASHAJANKOWSKI, PHD

DIRECTOR,NATIONALINSTITUTEFORLEARNINGOUTCOMESASSESSMENT

Page 2: Learning Outcomes A: Best Practices in …Design assessment approaches that generate actionable evidence about student learning that key stakeholders can understand and use to improve

NILOANILOA’smissionistodiscoveranddisseminateeffectiveuseofassessmentdatatostrengthenundergraduateeducationand

supportinstitutionsintheirassessmentefforts.

● SURVEYS ● WEB SCANS ● CASE STUDIES ● FOCUS GROUPS● OCCASIONAL PAPERS ● WEBSITE ● RESOURCES ● NEWSLETTER ●

PRESENTATIONS ● TRANSPARENCY FRAMEWORK ● FEATURED WEBSITES ●ACCREDITATION RESOURCES ● ASSESSMENT EVENT CALENDAR ● ASSESSMENT

NEWS ● MEASURING QUALITY INVENTORY ● POLICY ANALYSIS ●ENVIRONMENTAL SCAN ● DEGREE QUALIFICATIONS PROFILE ● TUNING ●

LEARNING SYSTEM

www.learningoutcomesassessment.org

Page 3: Learning Outcomes A: Best Practices in …Design assessment approaches that generate actionable evidence about student learning that key stakeholders can understand and use to improve
Page 4: Learning Outcomes A: Best Practices in …Design assessment approaches that generate actionable evidence about student learning that key stakeholders can understand and use to improve

Whyassessstudentlearning?Accreditationstandards

Questionswehaveregardingourpracticesandcurriculum

Arestudentslearningwhattheyneedinordertobesuccessfulinlifeandcareers?

Areourschoolsandcollegesworththecost?(ROI)

Howdoweknowstudentsreceiveahigh-qualityeducation?

Whatdoourcredentialsrepresentintermsoflearning?

Page 5: Learning Outcomes A: Best Practices in …Design assessment approaches that generate actionable evidence about student learning that key stakeholders can understand and use to improve

Yourquestions

Whatquestionsdoyouhaveaboutassessingstudentlearning– whatwouldyouliketowalkawayfromthismorningknowingmore

about?

Page 6: Learning Outcomes A: Best Practices in …Design assessment approaches that generate actionable evidence about student learning that key stakeholders can understand and use to improve

Overview

Focusonhowschoolscanbettercommunicatetoavarietyofaudiencestheirassessmentprocesses

Shareresourcesavailabletosupportandassistcollegesandschoolsundertakingassessmentofstudentlearning

Page 7: Learning Outcomes A: Best Practices in …Design assessment approaches that generate actionable evidence about student learning that key stakeholders can understand and use to improve

AssessmentDefinedAssessmentofstudentlearningisthesystematiccollection,review,anduseofinformationabouteducationalprogramsundertakenforthepurposeofimprovingstudentlearninganddevelopment.

Aboutstudentsandlearning,notinstitutions

Page 8: Learning Outcomes A: Best Practices in …Design assessment approaches that generate actionable evidence about student learning that key stakeholders can understand and use to improve

AssessmentCubeofMisunderstandings

DefinitionsPurposes/Value

Uses/Questions

Levels/Focus

Page 9: Learning Outcomes A: Best Practices in …Design assessment approaches that generate actionable evidence about student learning that key stakeholders can understand and use to improve

AddedLayersBehindeachsideofthecubethereare:1. Theoriesabouthowstudentslearn2. Beliefsaroundwhatcanbeassessed3. The“best”or“proper”meanstoassessstudentlearning4. Howtowarrantargumentsaboutwhatstudentsknowandcan

doasaresultofeducation

Allwithnoagreements,leadingtodisputesbuiltaroundunclearassumptionsthatimpactpracticewithpeopledismissingother

sides– rememberassessmentisafieldandit’safieldthatmoves.

Page 10: Learning Outcomes A: Best Practices in …Design assessment approaches that generate actionable evidence about student learning that key stakeholders can understand and use to improve

2017Pre-releasedata

0% 10% 20% 30% 40% 50% 60% 70% 80%

Portfolios

GeneralKnowledgeMeasures

Externally-SituatedPerformanceAssessments

EmployerFeedback

LocalGeneralKnowledgeMeasures

PlacementExams

LocalSurveys

AlumniFeedback

CapstoneProjects

Classroom-basedPerformanceAssessments

Rubrics

NationalStudentSurvey

Page 11: Learning Outcomes A: Best Practices in …Design assessment approaches that generate actionable evidence about student learning that key stakeholders can understand and use to improve

ThreeSchoolsofThoughtMeasurement

Compliance(Reporting)

TeachingandLearning(Improvement)

Page 12: Learning Outcomes A: Best Practices in …Design assessment approaches that generate actionable evidence about student learning that key stakeholders can understand and use to improve

MeasurementBuiltuponscientificprinciplesorempiricalresearch,objective,rational,validity,andreliability

TheMulti-StateCollaborative:APreliminaryExaminationofConvergentValidationEvidence~MarkNicholas,JohnHathcoat,&BrittanyBrown

◦ Testingandstandardization◦ Mustbemeasureable◦ Arguenarrowingofcurriculum◦ Goaldriven◦ Focusedonprocess◦ Interventions◦ Pre/post◦ Comparisons

Page 13: Learning Outcomes A: Best Practices in …Design assessment approaches that generate actionable evidence about student learning that key stakeholders can understand and use to improve

VALUEreport

Page 14: Learning Outcomes A: Best Practices in …Design assessment approaches that generate actionable evidence about student learning that key stakeholders can understand and use to improve

ComplianceDocumentinginstitutionalqualityassurancethroughreportingframeworks

Isassessmentdestroyingtheliberalarts?~KarinBrown

◦ Bureaucractic◦ Laborious◦ Timeconsuming◦ Separatedfromteachingandlearning◦ Addon◦ Accountabilityandqualityassurance◦ Reportingandarchive◦ Lotsofdatacollection,minimaluse

Page 15: Learning Outcomes A: Best Practices in …Design assessment approaches that generate actionable evidence about student learning that key stakeholders can understand and use to improve

Write Outcomes

Identify Assessments

Gather Results

Package Results

Submit Reports

ACCREDITATION/ PROGRAM

REVIEW

Page 16: Learning Outcomes A: Best Practices in …Design assessment approaches that generate actionable evidence about student learning that key stakeholders can understand and use to improve
Page 17: Learning Outcomes A: Best Practices in …Design assessment approaches that generate actionable evidence about student learning that key stakeholders can understand and use to improve

Butwherearethestudents…?

Page 18: Learning Outcomes A: Best Practices in …Design assessment approaches that generate actionable evidence about student learning that key stakeholders can understand and use to improve

TeachingandLearningFocusonpedagogy,understandingofstudentexperience,informingprogramimprovement,embeddedincurriculardesignandfeedback,buildsstudentagencyDoescontinuousassessmentinhighereducationsupportstudentlearning?~RosarioHernandez

◦ Drivenbyfacultyquestionsregardingtheirpraxis– iswhatIamdoingworkingformystudents?

◦ Improvementoriented◦ Focusonindividualstudents◦ Studentsasactiveparticipants– notsomethingdonetothem◦ Formative◦ Feedback◦ Collaborative◦ Assessmentforlearning◦ Adaptiveandembedded

Page 19: Learning Outcomes A: Best Practices in …Design assessment approaches that generate actionable evidence about student learning that key stakeholders can understand and use to improve

Name Expectations for Learning

Communicate Expectations to Learners

Collect Student Work

Determine Extent of Learning

Strategize New Student

Success Plans

IMPROVEMENT

Page 20: Learning Outcomes A: Best Practices in …Design assessment approaches that generate actionable evidence about student learning that key stakeholders can understand and use to improve

Whatdoesgoodassessmentlooklikeforus?Whydowethinkthatwhatwearedoing,forthesestudents,willleadtoenhancedlearning,atthistime?

Page 21: Learning Outcomes A: Best Practices in …Design assessment approaches that generate actionable evidence about student learning that key stakeholders can understand and use to improve

PrinciplesofLocalPracticeDevelopspecific,actionablelearningoutcomesstatements.

Connectlearningoutcomeswithactualstudentdemonstrationsoftheirlearning.

Collaboratewithrelevantstakeholders,beginningwiththefaculty.

Designassessmentapproachesthatgenerateactionableevidenceaboutstudentlearningthatkeystakeholderscanunderstandandusetoimprovestudentandinstitutionalperformance.

Focusonimprovementandcompliancewilltakecareofitself.

Page 22: Learning Outcomes A: Best Practices in …Design assessment approaches that generate actionable evidence about student learning that key stakeholders can understand and use to improve

Evidenceofstudentlearningisusedinsupportofclaimsorargumentsaboutimprovementandaccountabilitytoldthroughstories topersuadeaspecific

audience.

Needtotellourstoryandhelpstudentstelltheirs.

CommunicatingourAssessment

Page 23: Learning Outcomes A: Best Practices in …Design assessment approaches that generate actionable evidence about student learning that key stakeholders can understand and use to improve

Toulmin (2003)

Evidence Claim

Warrant

Warrants

Arguments

But…

Page 24: Learning Outcomes A: Best Practices in …Design assessment approaches that generate actionable evidence about student learning that key stakeholders can understand and use to improve

Whydowethinkthechangeswemakewillleadtobetteroutcomes?

Whatisassumedinthechangesweselectasitrelatestohowstudentsunderstandandnavigatehighereducation?

TheoriesofChange

Page 25: Learning Outcomes A: Best Practices in …Design assessment approaches that generate actionable evidence about student learning that key stakeholders can understand and use to improve

Coverageandcontent

Opportunitiesandsupport

Intentional,coherent,alignedpathways

Withineachoftheseisthebeliefaboutrootcauses– whystudentswerenotlearningornotmeetingtheoutcomeandthemechanismbywhichtheinstitutioncanhelpthemsucceed

Forinstance…

Page 26: Learning Outcomes A: Best Practices in …Design assessment approaches that generate actionable evidence about student learning that key stakeholders can understand and use to improve

Nationalrecognitionprogramforcampusassessmentleadersattwolevels

EvaluationbasedontheNationalInstituteforLearningOutcomesAssessment(NILOA)TransparencyFramework

Focusoncampus-wideassessment– includingstudentaffairs&externalstakeholders

JointprojectoftheVSA,NILOA,andtheAssociationofAmericanColleges&Universities(AAC&U)

ExcellenceinAssessmentDesignations

Page 27: Learning Outcomes A: Best Practices in …Design assessment approaches that generate actionable evidence about student learning that key stakeholders can understand and use to improve

WhydidwecreatetheEIADesignations?

Faculty&instructional

staff

StudentAffairsstaff

Accreditors,governingboards

Employers,subsequentinstitutions

Students&Alumni

Page 28: Learning Outcomes A: Best Practices in …Design assessment approaches that generate actionable evidence about student learning that key stakeholders can understand and use to improve

Whatisworkingonyourcampus?Whatisworkingwell?Wherearetherepressurepoints?Whichstakeholdersareinvolved?

Page 29: Learning Outcomes A: Best Practices in …Design assessment approaches that generate actionable evidence about student learning that key stakeholders can understand and use to improve

Sowhatishappeningnow?

Page 30: Learning Outcomes A: Best Practices in …Design assessment approaches that generate actionable evidence about student learning that key stakeholders can understand and use to improve

AssignmentCharrettes

Page 31: Learning Outcomes A: Best Practices in …Design assessment approaches that generate actionable evidence about student learning that key stakeholders can understand and use to improve

25minutesegmentsperperson/assignment

ØPresentassignmentbriefly—whichoutcomes,whatcourse,whichstudents,howdoesitwork….

ØDiscussionwithgroup(focusonquestionsonfeedbackform)

ØSave5minutesforwrittenfeedback

ØBreath

ØStartagain…

ØDebriefatend

SmallGroupPeerReviewProcess

Page 32: Learning Outcomes A: Best Practices in …Design assessment approaches that generate actionable evidence about student learning that key stakeholders can understand and use to improve

AssignmentDesignResources

https://www.assignmentlibrary.org/

Page 33: Learning Outcomes A: Best Practices in …Design assessment approaches that generate actionable evidence about student learning that key stakeholders can understand and use to improve

Alignmentwithincourses

Assignment

ScaffoldingLearning

EvaluativeCriteria

LearningOutcomes

Page 34: Learning Outcomes A: Best Practices in …Design assessment approaches that generate actionable evidence about student learning that key stakeholders can understand and use to improve

Assignment

ScaffoldingLearning

EvaluativeCriteria

LearningOutcomes

Assignment

ScaffoldingLearning

EvaluativeCriteria

LearningOutcomes

Assignment

ScaffoldingLearning

EvaluativeCriteria

LearningOutcomes

Howassignmentsconnect

Page 35: Learning Outcomes A: Best Practices in …Design assessment approaches that generate actionable evidence about student learning that key stakeholders can understand and use to improve

AlignmentHowdocoursesbuildtowardsmasterythroughrepetitionandincreasingexpectationsforparticularoutcomes?

Howdoassignmentsandactivitieselicitstudentdemonstrationsofaspecificlearningoutcome?

Howdoindividualfacultycontributetothiscollectiveworkintheircourses?

Page 36: Learning Outcomes A: Best Practices in …Design assessment approaches that generate actionable evidence about student learning that key stakeholders can understand and use to improve

Connections

Assignment

ScaffoldingLearning

EvaluativeCriteria

LearningOutcomes

Employers

Co-curricular

Implicationsfortranscripts,careerdevelopment,andpathways

Page 37: Learning Outcomes A: Best Practices in …Design assessment approaches that generate actionable evidence about student learning that key stakeholders can understand and use to improve

Butwehavetomaketheconnectionsclear!

AwarenessofLearningOutcomeStatements

Page 38: Learning Outcomes A: Best Practices in …Design assessment approaches that generate actionable evidence about student learning that key stakeholders can understand and use to improve

TransparencyinAssignmentsTransparencyinTeachingandLearning:https://www.unlv.edu/provost/teachingandlearning

Purpose

Skillsyou’llpracticebydoingthisassignment

Contentknowledgeyou’llgainfromdoingthisassignment

Howyoucanusetheseinyourlifebeyondthecontextofthiscourse,inandbeyondcollege

Task

Whattodo

Howtodoit(Arethererecommendedsteps?Whatroadblocks/mistakesshouldyouavoid?)

Criteria

(Areyouontherighttrack?Howtoknowyou’redoingwhat’sexpected?)

Annotatedexamplesofsuccessfulwork

(What’sgoodabouttheseexamples?Usethechecklisttoidentifythesuccessfulparts.)

Page 39: Learning Outcomes A: Best Practices in …Design assessment approaches that generate actionable evidence about student learning that key stakeholders can understand and use to improve

RubricsDoweshareourrubricsorcriteriawithstudentsandactivelyengagetheminthereviewprocess?Rubric Criteria StudentEvaluation FacultyFeedback

RubricContent Stipulate whygavescoredid

Facultystipulatewhygave scoredid

Stipulate whattheyneedtodotoadvance

Targetedfeedbacktoimprove

Page 40: Learning Outcomes A: Best Practices in …Design assessment approaches that generate actionable evidence about student learning that key stakeholders can understand and use to improve

TimingandPlacementofAssignmentsDoweconsiderstudentmovementthroughcurriculumintermsoftheplacementofsignatureassignments?Whendowegivestudentsfeedbackanddotheyhaveachancetodoanythingwithit?

Page 41: Learning Outcomes A: Best Practices in …Design assessment approaches that generate actionable evidence about student learning that key stakeholders can understand and use to improve
Page 42: Learning Outcomes A: Best Practices in …Design assessment approaches that generate actionable evidence about student learning that key stakeholders can understand and use to improve

ConnectionPoints

Toensurestudentsuccess,it’showallofthepiecesconnecttogethertosupportcollectivedevelopmentof

activeandengagedlearners.

Page 43: Learning Outcomes A: Best Practices in …Design assessment approaches that generate actionable evidence about student learning that key stakeholders can understand and use to improve
Page 44: Learning Outcomes A: Best Practices in …Design assessment approaches that generate actionable evidence about student learning that key stakeholders can understand and use to improve

MoreResources

Page 45: Learning Outcomes A: Best Practices in …Design assessment approaches that generate actionable evidence about student learning that key stakeholders can understand and use to improve

TheLearningSystemsParadigm

Page 46: Learning Outcomes A: Best Practices in …Design assessment approaches that generate actionable evidence about student learning that key stakeholders can understand and use to improve

DiscussiontimeEmail:[email protected]

http://www.learningoutcomesassessment.org

www.assignmentlibrary.org

www.degreeprofile.org


Improving Student Outcomes through Actionable Data and ...Improving Student Outcomes through Actionable Data and Collaboration Emily Lawrence ... learning what works for improving

Improving Student Outcomes through Actionable Data and ...Improving Student Outcomes through Actionable Data and Collaboration Emily Lawrence ... learning what works for improving

Documents
Actionable Learning: A Handbook for Capacity Building ... · Capacity Building as Actionable Learning This Handbook addresses case based learning in the context of developing capacity

Actionable Learning: A Handbook for Capacity Building ... · Capacity Building as Actionable Learning This Handbook addresses case based learning in the context of developing capacity

Documents
ACTIONABLE NETWORK INTELLIGENCE - infoblox.com · With Infoblox Actionable Network Intelligence, you gain a nimble platform that provides the capabilities you need to more easily

ACTIONABLE NETWORK INTELLIGENCE - infoblox.com · With Infoblox Actionable Network Intelligence, you gain a nimble platform that provides the capabilities you need to more easily

Documents
Delivering actionable insights that drive optimized ...€¦ · actionable insights to drive optimized operational performance. Unlike other approaches that effectively exclude organizations

Delivering actionable insights that drive optimized ...€¦ · actionable insights to drive optimized operational performance. Unlike other approaches that effectively exclude organizations

Documents
UKSG webinar: Learning analytics - gaining good actionable insight with Martin Hawksey, Association for Learning Technology

UKSG webinar: Learning analytics - gaining good actionable insight with Martin Hawksey, Association for Learning Technology

Education
Learning dashboards for actionable feedback · learning analytics design and build analytics dashboards, dashboards that go beyond identifying at-risk students, allowing actionable

Learning dashboards for actionable feedback · learning analytics design and build analytics dashboards, dashboards that go beyond identifying at-risk students, allowing actionable

Documents
20180516 Learning Dashboards Tinne De Laet...learning analytics design and build analytics dashboards, dashboards that go beyond identifying at-risk students, allowing actionable feedback

20180516 Learning Dashboards Tinne De Laet...learning analytics design and build analytics dashboards, dashboards that go beyond identifying at-risk students, allowing actionable feedback

Documents
Making Competency Based Learning Real for Educators Competency Based...and leaders have an actionable, data-based, real-time map for modeling educator competency-based learning that

Making Competency Based Learning Real for Educators Competency Based...and leaders have an actionable, data-based, real-time map for modeling educator competency-based learning that

Documents
Contextual Codifiable, explicit Easily transferable Insight, creates meaning, judgmental, actionable Human, tacit, transfer requires learning Data

Contextual Codifiable, explicit Easily transferable Insight, creates meaning, judgmental, actionable Human, tacit, transfer requires learning Data

Documents
Actionable Learning A Handbook for Capacity Building Through Case Based Learning

Actionable Learning A Handbook for Capacity Building Through Case Based Learning

Documents
Actionable intelligence - Tyco Security · 2020-04-06 · Actionable intelligence - Autonomous solutions Within automation we have Tyco AI, an ever-evolving, deep learning solution

Actionable intelligence - Tyco Security · 2020-04-06 · Actionable intelligence - Autonomous solutions Within automation we have Tyco AI, an ever-evolving, deep learning solution

Documents
Making intelligence actionable - IQVIAus.imshealth.com/.../BI/IMS_eBook_Making-Intelligence-Actionable.pdf · Making intelligence actionable: ... actionable information that meets

Making intelligence actionable - IQVIAus.imshealth.com/.../BI/IMS_eBook_Making-Intelligence-Actionable.pdf · Making intelligence actionable: ... actionable information that meets

Documents
DBI326. PhraseGoal “Data Mining”Inform actionable decisions “Machine Learning”Determine best performing algorithm

DBI326. PhraseGoal “Data Mining”Inform actionable decisions “Machine Learning”Determine best performing algorithm

Documents
Problem Solving Strategies in Case Based Learning158.132.155.107/posh97/private/learning/actionable-learning-07.pdfPART 7 PROBLEM SOLVING STRATEGIES IN CASE BASED LEARNING 153 tor

Problem Solving Strategies in Case Based Learning158.132.155.107/posh97/private/learning/actionable-learning-07.pdfPART 7 PROBLEM SOLVING STRATEGIES IN CASE BASED LEARNING 153 tor

Documents
Actionable Architecture

Actionable Architecture

Technology
Actionable Models: Unsupervised Offline Reinforcement ...proceedings.mlr.press/v139/chebotar21a/chebotar21a.pdfActionable Models: Unsupervised Offline Reinforcement Learning of Robotic

Actionable Models: Unsupervised Offline Reinforcement ...proceedings.mlr.press/v139/chebotar21a/chebotar21a.pdfActionable Models: Unsupervised Offline Reinforcement Learning of Robotic

Documents
Likable Content: 33 Actionable Tips to Create Content that gets Clicks

Likable Content: 33 Actionable Tips to Create Content that gets Clicks

Business
Actionable Learning Analytics: Practical Lessons for Teaching and Learning

Actionable Learning Analytics: Practical Lessons for Teaching and Learning

Education
Machine Learning Transform data into actionable insights

Machine Learning Transform data into actionable insights

Documents
Uncover actionable insights that drive operational improvement...Uncover actionable insights that drive operational improvement Operational challenges Devices/Equip-ment Predictive

Uncover actionable insights that drive operational improvement...Uncover actionable insights that drive operational improvement Operational challenges Devices/Equip-ment Predictive

Documents
Nk.pl transforms Big Data into Actionable Insights that Drive Business Success

Nk.pl transforms Big Data into Actionable Insights that Drive Business Success

Technology
Actionable Dreams

Actionable Dreams

Presentations & Public Speaking
Actionable Science @ DOI SE Climate Science Ctr ... · SECSC vision for science that “adds up” – In general: Actionable science provides data, analyses, projections, tools,

Actionable Science @ DOI SE Climate Science Ctr ... · SECSC vision for science that “adds up” – In general: Actionable science provides data, analyses, projections, tools,

Documents
Actionable Software Metrics: An Industrial Perspectivesmartinez/wp-content/papercite-data/pdf/ram20… · contribute any actionable information [25]. Essentially, a metric that can

Actionable Software Metrics: An Industrial Perspectivesmartinez/wp-content/papercite-data/pdf/ram20… · contribute any actionable information [25]. Essentially, a metric that can

Documents
2016 Implementation Strategy Report - Kaiser Permanente...KFH San José consulted with Actionable Insights, LLC. Actionable Insights is a consulting firm that helps organizations discover

2016 Implementation Strategy Report - Kaiser Permanente...KFH San José consulted with Actionable Insights, LLC. Actionable Insights is a consulting firm that helps organizations discover

Documents
Creating actionable data using Alma Analytics dashboard · Libraries and Learning Services TeTumu Herenga Beyond the numbers: Creating actionable data using Alma Analytics dashboard

Creating actionable data using Alma Analytics dashboard · Libraries and Learning Services TeTumu Herenga Beyond the numbers: Creating actionable data using Alma Analytics dashboard

Documents
Achieving ROI from EHRs: Actionable insights that can

Achieving ROI from EHRs: Actionable insights that can

Documents
Making Project-Based Learning Actionable with Ambitious

Making Project-Based Learning Actionable with Ambitious

Documents
Towards Actionable Learning Outcomes

Towards Actionable Learning Outcomes

Documents
Special Reports | Finding Actionable Insights Through ... · Special Reports Finding Actionable Insights through AdWords Reporting 5 Here’s a helpful example for an account that

Special Reports | Finding Actionable Insights Through ... · Special Reports Finding Actionable Insights through AdWords Reporting 5 Here’s a helpful example for an account that

Documents