Embed Size (px)
DESCRIPTION
Efficiency of assessment: initial thinking from the ITEAM project. Mark Russell, Liz Gormley-Fleming and Karen Robins The ITEAM project. University of Hertfordshire. The ITEAM project. Strand A project A (reasonably) tightly defined focus EVS QMP On line assessment and feedback system - PowerPoint PPT Presentation
Citation preview
Efficiency of assessment: initial thinking from the ITEAM project
Mark Russell, Liz Gormley-Fleming and Karen Robins
The ITEAM project.University of Hertfordshire
The ITEAM project
• Strand A project
• A (reasonably) tightly defined focus– EVS– QMP– On line assessment and feedback system– Dashboard
A matrix
Educationally effective
Reso
urce
effi
cien
cy o
r (1
/t)
A matrix
Educationally effective
Reso
urce
effi
cien
cy o
r (1
/t)
What about you?
Think about one of your assessments …
1. Where would you locate it on the grid?
2. How might you nudge it to be more educationally effective and more resource efficient?
…Now, next year, the year after …Thoughts in the chat box - please
The ESCAPE assessment for learning themes …
Good assessment for learning:…Engages students with assessment criteria…Supports personalised learning…Focuses on student development…Ensures feedback leads to improvement…Stimulates dialogue
The ESCAPE assessment for learning themes …
Good assessment for learning:…Engages students with assessment criteria…Supports personalised learning…Focuses on student development…Ensures feedback leads to improvement…Stimulates dialogue…Considers staff and student effort
What about some sample assessments?
Essay - Summative (2000 words)
Students
Activity Number
Activities Time (minutes) 20 60 120
1 Module aims and learning outcomes 10 10 10 102 Writing assignment brief & marking
criteria50 50 50 50
3 Essay title. 30 30 30 304 Team agreement 20 20 20 205 External examiner comments 20 20 20 206 Re drafting assignment. 10 10 10 107 Student preparation (specific) 60 60 60 608 Tutorial support-drafts & feedback* 20 per student. 400 1200 24009 Submission-setting up site, guidance 20 20 20 2010 Marking* 60 per 3 students 400 1200 240011 Feedback* 30 per 3 students. 200 600 1200
12 Module report 30 30 30 3013 Moderation* 60 per 3 students. 400 1200 2400
14 External examiner 30 30 30 3015 Assessment office*-inputting results 2 per student. 40 120 24016 Exam Board 10 10 10 1017 Student tutorial post assessment
process*15 per student. 300 900 1800
Total time in minutes 2030 5510 10730 Time in minutes per student 101.5 91.8 89.4
EVS test - 1 hour MCQs, 25 questions Students
Activity Number Activity Time (minutes) 20 60 120
1 Module aims and LO’s 10 10 10 102 Create and type up 25MCQ questions 15 per question 375 375 3753 Internal Moderation 20 20 20 204 External moderation 20 20 20 205 Update questions 20 20 20 206 Create practice test 375 375 375 3757 Student preparation 120 120 120 1208 Preparing environment for assessment* 15 15 15 159 Run MCQ test 60 60 60 6010 Marking - creating EVS report 10 10 10 1011 Assessment office and inputting of grades* 1 per student 20 60 12012 Providing feedback* 15 15 15 1513 Module report 30 30 30 3014 External examiner moderation 30 30 30 3015 Module board 20 20 20 2016 Post assessment process tutorial support* 15 per student 300 900 180017 Student tutorial post assessment process* 120 120 120 120 Total in minutes 1560 2200 3160 Time in minutes per student 78 37 26
OSCE-summative.
Students
Activity Number
Activities Time (minutes) 20 60 120
1 Module aims and LO’s 5 5 5 52 Case scenario’s 120 / scenario 120 120 1203 Team meeting 120 120 120 1204 External examiner 20 20 20 205 Uploading assessment information 30 30 30 306 Student preparation 120 120 120 1207 Environment* 450 450 450 4508 OSCE* (role of Academic) - 6hrs*10
academics3600 3600 10800 21600
9 Marking & feedback* 10 per student 200 600 120010 Moderation* 5 per student 100 300 60011 Module report 30 30 30 3012 External examiner 20 20 20 2013 Assessment office* 5 per student. 100 300 60014 Module board 10 10 10 1015 Post assessment process feedback* 15 per student. 300 900 1800
Total time in minutes 5225 13825 26725
Time in minutes per student
261.3 230.4 222.7
Essay OSCE Group Presentation Exam EVS 0
50
100
150
200
250
300
Assessment time per student (mins)
20 students60 students120 students
Essay OSCE Group Presentation Exam EVS 0
50
100
150
200
250
300
Assessment time per student (mins)
20 students60 students120 students
12% change
66% change34% change
And you …?What sort of things are you doing / (need to do) to help your staff make informed
decisions about choosing an assessment type for effectiveness and
efficiency?Thoughts in the chat box - please
Developing a tool
Plotting the data
Numbers of students
Tim
e (m
inut
es)
Diverging time scales
Numbers of students
Tim
e (m
inut
es)
Assessm
ent type A
Assessment type B
Overlapping time scales
Not prescriptive!
Not prescriptive!
And so …In what ways might these tools (and approaches) help or hinder your work?
Thoughts in the chat box - please
Does (say) 20 hours of our effort yield the same learning payoff?
Does (say) 20 hours of our effort yield the same learning payoff?
An example with EVS
The ESACPE assessment for learning themes …
Good assessment for learning:…Engages students with assessment criteria…Supports personalised learning…Focuses on student development…Ensures feedback leads to improvement…Stimulates dialogue…Considers staff and student effort
So let’s always use the least time consuming?
So let’s always use the least time consuming?
Alas, it’s
a little (a lot) m
ore complex and
messy th
an that.
31
The purposes of assessment are important too!
SummativeHigh StakesMeasure learning
FormativeNo / Low StakesMotivate learning
Educative Auditing
32
The purposes of assessment are important too!
SummativeHigh StakesMeasure learning
FormativeNo / Low StakesMotivate learning
Feedback rich?Linked (in-process) tasks?Regular tasks?Error does not lead to punishment
Need for feedback is questionable (MBR)One-off tasksEnd of process tasks?
Educative Auditing
Assessment patterns
Beyond the single assessment task
Some alternatives…
And many more …
Programme view of assessment
Programme view of assessment
Programme view of assessment
Far reaching implications• Our work (as teachers) extends beyond our
classroom. How might we measure this?
• Performance on singular assessment?• Success in Module?• Success in Year?• Success in Programme?
• Institutional reputation?
Conclusion – to date
• Numbers ‘might’ be countable• Numbers ‘might’ be meaningful (process
improvement, stimulating dialogue, moving in the ‘right direction’)
A possible conclusion – to date
• Numbers ‘might’ be countable• Numbers ‘might’ be meaningful (process
improvement, stimulating dialogue, moving in the ‘right direction’)
But…
A possible conclusion – to date
• Numbers ‘might’ be countable• Numbers ‘might’ be meaningful (process
improvement, stimulating dialogue, moving in the ‘right direction’)
But…
Some questions for you to ponder…What is the difference in learning payoff between a 4000 word and a 2000 word essay ?
Is a single 4000 word essay more beneficial than (say) two 1500 word essays?
What are the resource implications of the above variants?
What are the various processes of assessment doing?
Where might you best place your efforts to reduce the resource burden while keeping the effectiveness of the assessment?
In what ways are you facilitating meaningful discussions about both resource efficiency and educational effectiveness?
