Embed Size (px)
Citation preview
TESTA: an evidence – informed approach to improving programme assessment
Dr Tansy JessopTESTA Project Leader & Senior FellowKingston University L&T Conference20 April 2012
What is TESTA? “Transforming the experience of Students through
Assessment”
HEA funded research project (2009-12)
Seven programmes in four partner universities
Mapping programme assessment
Listening to student voice
Engaging with Quality Assurance processes
Diagnosis - Intervention - Evaluation
TESTA ‘Cathedrals Group’ Universities
Two Paradigms
Transmission model
Expert to novicePlanned & ‘delivered’Feedback by expertsFeedback to novicesPrivatisedMonologueEmphasis on measuringCompetitionMetaphor - machine
Social constructivist model
Participatory, democraticMessy and process-orientedPeer reviewSelf-evaluationSocial processDialogueEmphasis on learningCollaborationMetaphor - the journey
1. Programme assessment patterns2. Robust research methods (Gibbs & Dunbar-
Goddet 2007,2009)3. Educational principles (Gibbs and Simpson
2004).4. Changing concepts and discourse (Nicol
2012)5. Participatory social process
TESTA principles of transformation
Programme Audit
• How much summative assessment• How much formative (reqd, formal,
feedback)• How many varieties of assessment• Proportion exams to coursework• Word count of written feedback• How much ‘formal’ oral feedback• Criteria, learning outcomes, course docs
Assessment Experience Questionnaire
• 28 questions• 5 point Likert scale where 5 = strongly
agree• 9 scales & an overall satisfaction question• Scales link to conditions of learning• Examples:
• quantity and distribution of effort; • use of feedback; • quantity and quality of feedback; • clear goals and standards
Focus groups
• Different kinds of assessment• How assessment influences study
behaviour• Whether and how students know what
quality work looks like • The quality and timing of feedback • The usefulness of feedback
Research Methodology
ASSESSMENT EXPERIENCEQUESTIONNAIRE (AEQ n= 1200+)
FOCUS GROUPS (n=50 with 301 students)
PROGRAMME AUDIT (n =22)
Programme Team
Meeting
Case Study
TESTA Case Study 1: what’s going on?
• Lots of coursework, of very varied forms • Very few exams• Masses of written feedback on assignments• Learning outcomes and criteria clearly specified….looks like a ‘model’ assessment environment
But students:• Don’t put in a lot of effort and distribute their effort across few
topics• Don’t think there is a lot of feedback or that it very useful, and
don’t make use of it• Don’t think it is at all clear what the goals and standards are
TESTA Case Study 2: what’s going on?
• 35 summative assessments• No formative assessment specified in documents• Learning outcomes and criteria wordy and woolly• Marking by global, tacit, professional judgements• Teaching staff mainly part-time and hourly paid….looks like a problematic assessment environment
But students:• Put in a lot of effort and distribute their effort across topics• Have a very clear idea of goals and standards • Are self-regulating and have a good idea of how to close the
gap
Case Study 1:
• Staff do loads of work, and it doesn’t really work for students.
• Students are unable to evaluate their own performance.
• Students don’t take control of their own learning.
• Summative assessment drives effort but not necessarily engagement & learning.
Case Study 2:
• Students do loads of work, and it works well as a programme.
• Students are continually engaged in evaluating their own and others’ performance.
• Students control and manage their own learning.
• Formative assessment drives effort, engagement and learning.
Assessment trends on TESTA
• High summative, low formative (36:11?) • High variety (ave 11; range 7-17)• Written feedback (ave 7,153; range 2,869-
15,412 )• Oral feedback (ave 6 hours, range 37 mins
to 30 hrs)• Watertight documents, tacit standards• Huge institutional and programme variations
Gibbs and Simpson’s effort principles (2004)
1. Assessed tasks need to capture sufficient student time and effort
2. These tasks should distribute effort evenly across topics and weeks
TESTA evidence on effort• Audit: Mean 36 summative assessments• AEQ: Students from only 1/7 programmes agreed
that assessment encouraged regular effort• Focus groups:
We could do with more assessments over the course of the year to make sure that people are actually doing stuff.
The more you write the better you become at it… and if we’ve only written 40 pieces over three years that’s not a lot.
• So you could have a great time doing nothing until like a month before Christmas and you’d suddenly panic. I prefer steady deadlines, there’s a gradual move forward, rather than bam!
• If it’s not going to be in the exam, and it is quite difficult, I wouldn’t bother with that.
• We get too much of this end or half way through the term essay type things. Continual assessments would be so much better.
Effort ideas1. Multi-stage assessment2. Social pressure – public work3. Spread and co-ordinate hand in dates4. Formative course requirements5. Peer marking in class6. Sampling7. Set first year expectations8. Frequent, authentic, innovative
Clear and high standards principles
3. Assessment communicates clear and high expectations to students (Gibbs et al, 2004)
4. Students should: (a)know what is expected, (b)know how this relates to their actual
performance, and (c)have some information about how to close the
gap (Sadler, 1989).
TESTA evidence on clarity
Audit: clear written statements mapped to outcomes; low formative feedback; huge variety of tasksAEQ: Students on 0/7 programmes agreed.Focus groups:
There are criteria, but I find them really strange. There’s “writing coherently, making sure the argument that you present is backed up with evidence” but that isn’t enough.
I’m not a marker so I can’t really think like them... I don’t have any idea of why it got that mark.
They have different criteria, build up their own criteria. Some of them will mark more interested in how you word things.
It’s easier to know if you’ve done a good essay than creative piece... it’s very subjective.
You know who are going to give crap marks and who are going to give decent marks.
Ideas for internalising standards1. Marking workshops (Aske CETL, Oxford Brookes) 2. Students rewriting criteria in own words 3. Criteria-based peer review 4. Self-evaluation against criteria5. Observation, imitation, participation, dialogue
(Bloxham and Campbell, 2010).6. Contextualised Feedback – standards, criteria,
outcomes
Feedback principles (Gibbs & Simpson 2004)
5) Sufficient feedback is provided, both often enough and in enough detail
6) It is quick enough for students to act on it7) It focuses on learning rather than marks8) It is understandable9) Students attend to feedback and act on it to
improve their learning
TESTA evidence on feedback• Audit: high volumes of written, low oral, delivered slowly• AEQ: Students from only 1/7 programmes agreed that it was useful
I read it and think “Well that’s fine, but I’ve already handed it in now and got the mark. It’s too late”.
It was about nine weeks before we got it back. I’d forgotten what I’d written.
You know that twenty other people have got the same sort of comment.
Once the deadline comes up to just look on the Internet and say ‘Right, that’s my mark. I don’t need to know too much about why I got it’.
I only apply feedback to that module because I have this fear that if I transfer it to other modules it’s not going to transfer smoothly.
You can’t carry forward most of the comments because you might have an essay first & your next assignment might be a poster.
Getting feedback from other students in my class, I can relate more to what they’re saying and take it on board. I’d just shut down if I was getting constant feedback from my lecturer.
TESTA: patterns of changesChanges through TESTA
Types of changes
• Reducing summative• Increasing formative tasks • Streamlining variety• Student workload expectations• Sequencing and linking tasks• Practice based changes• Changed conceptions of A&F
References:Gibbs, G. & Simpson, C. (2004) Conditions under which assessment supports students' learning. Learning and Teaching in Higher Education. 1(1): 3-31.Gibbs, G. & Dunbar-Goddet, H. (2009). Characterising programme-level assessment environments that support learning. Assessment & Evaluation in Higher Education. 34,4: 481-489.Jessop, T, McNab, N and Gubby, L. (2012) Mind the gap: An analysis of how quality assurance processes influence programme assessment patterns. Active Learning in Higher Education. 13(3).Nicol, D. (2010) From monologue to dialogue: improving written feedback processes in mass higher education, Assessment & Evaluation in Higher Education, 35: 5, 501 – 517Nicol, D. (2012) Assessment Principles Webinar on JISC.Sadler, D.R. (1989) Formative assessment and the design of instructional systems, Instructional Science, 18, 119-144.Sambell, K (2011) Rethinking Feedback in Higher Education. Higher Education Academy Escalate Subject Centre Publication.
