Embed Size (px)
Citation preview
1. Pressure on staff as HE financing changes
2. Student expectations (esp. under new fee regime)
Context
Key question
What’s ‘the deal’ (going to be)?
Assessors have to be able to sustain the expectations laid upon them.
What volume of good assessment practice can they realistically undertake as the economic screw tightens? (And does ‘the management’ know?)
A ‘sustainability’ consideration for staff
1. Curricula for autonomy
2. Formative assessment, feedback and feedforward
3. Summative assessment (especially grading)
Agenda (rather loosely organised)
Acquiescence Autonomy
Kohlberg 1964Perry 1970 (reprinted 1998)King and Kitchener 1994Kuhn and Weinstock 2002Baxter Magolda 2009
Becoming a graduateOught to involve young people (and perhaps some older people) in a significant transformation
Guiding learners through the transformation from authority dependence to self-authorship is a primary challenge for twenty-first century higher educationBaxter Magolda (2009, p.144)
Students often take time to ‘get it’
[School study] habits can continue well into the first year of university.(Leckey and Cook, 1999)
[N]ew undergraduates often see the tutor as the 'expert' who can (and perhaps should) give them 'the information'. By contrast university history teachers emphasise the need for student autonomy and independent judgement.(Booth, 2005)
There may not be a ‘truth’ that can be handed down
Despite help, students did not grasp what was required of them as regards essay-writing.(McCune, 2004)
Students need help in order to develop as autonomous learners.(Fazey and Fazey, 2001)
At the beginning you have no idea what constitutes a pass as you have no frame of reference. Need feedback on earlier work before progressing to next assessment. (Student, in Johnston and Kochanowska, 2009)
Students often take time to ‘get it’
But what if the next assessment task is very different?
A pedagogy for autonomy
Students should be encouraged to develop their capacity to self-assess over a lifetime (Boud, 2000).
Royce Sadler is insistent that higher education should develop in students an appreciation of standards and how their work measures up (e.g. Sadler, 2009).
Assessment and feedback are of critical importance
A ‘patient curriculum’? In which ...
Formative assessment, feedback and feedforward
Two equations (after Maddalena Taras)
Grade + Feedback + Feedforward = Formative assessment
Formative assessment + Engagement = Learning
What surveys say
Feedback has relatively low ratings from ‘graduates’ (NSS; CEQ)
‘Your First College Year’ survey in the US doesn’t ask about it
In first-year experience surveys that did:
UK FYE 2007
57% found
feedback helpful
AustraliaFYE 2005
33% found
feedback helpful
AustraliaFYE 2010
33% found
feedback helpful
Good assessment and feedback should…1. Clarify goals, criteria, standards2. Encourage time and effort on challenging tasks3. Give good feedback that helps self-correction4. Provide opportunities to act on feedback5. Ensure summative assessment assists learning6. Encourage dialogue about learning7. Facilitate self-assessment and reflection8. Encourage motivation and self-esteem
Adapted from Nicol, 2009
9. Give students choice regarding assessment10. Involve students in policy and practice re assessment11. Support development of learning groupings12. Inform teachers about their teaching
Sadler: Students should
internalise standards
So what about students’ expectations and experience?
Perceptions of feedbackInforming students about feedback is important since what staff consider to be feedback is not always appreciated as such by students
Staff often or very often gave it
Students often or very often received it
0 20 40 60 80 100
80.4
38.6
AUSSE 2008
Have the methods by which your work has been assessed been as you had expected? N
Per cent
No 55 8.1Partly 226 33.1Yes 402 58.9
Total respondents 683 100.0
Match of expectations and experience: assessment methods (A&D)
Match of expectations and experience: assessment methods (A&D)
No Partly Yes0%
10%
20%
30%
40%
50%
60%
70%
Low prior infHigh prior inf
Expectations regarding feedback
Issue (abbreviated) 2006 2007 2010
N=979 N=1774 N=3091
% Agree % Agree % Agree
Ready access to staff outside f2f important 87 87 87
I expect teachers to read drafts 52
Feedback on drafts important to learning 92 95
Crisp et al (2009); Scutter et al (2011)
Acceptable time-interval for return of work
1 week 2-3 weeks 4-6 weeks
2006 % 34 57 4
2007 % 38 55 3
2010 % 19 60
Expectations re feedback ... and experience
Item no.
Item theme (varies with group studied)
At orientation
% Agree
End year 1
% Agree
Year 2
% Agree
Teachers
% Agree
7 ‘Ready’ access tolecturers, tutors 88 89
61 50
73 57
64 94
8 Feedback on submitted work 97 97 66 37 74 48 100 59
9 Feedback on DRAFTS of work
94 91 20 7 21 26 0 22
Colour code: Humanities Science
Note: phrasing adjusted to fit circumstances
Brinkworth et al (2009)
Inadequate understanding of the task (Glover & Brown 2006)
Student-staff differences in perception (Maclellan 2001)
Feedback not understood (Chanock 2000; Weaver 2006)
Tutors’ intentions re feedback not understood (Higgins et al 2002)
‘Being told’ is not engaging (Crisp 2007)
Disjuncts in the feedback loop (Hounsell et al 2006)
Transferability not perceived (Carless 2006; Duncan 2007) ...... especially where assessment demands are diverse (Gibbs, ongoing)
Timeliness (Brinkworth et al 2009)
The grade’s the thing (Snyder 1971)
Espoused theory & practice out of sync (Orrell 2006; Orsmond et al 2011)
Issues relating to the effectiveness of feedback
Where might enhancement efforts most usefully be targeted?
Worthwhile feedback? The feedback on my assignments comes back so slowly that we are already on the topic after next and I’ve already submitted the next assignment. It’s water under the bridge, really.
I just look at the mark and bin it.
Collected by Graham Gibbs
I found that I did not learn anything from my mistakes as I was never told what they were.
Misko and Priest (2009, p.15)
Re a diagnostic essay:It took ages … about two months … by that time I guess you had forgotten … we had already wrote (sic) another essay. Murphy and Cornell (2010, p.46)
I don’t like these shorthand comments, and we get too many of them— ‘good analysis’—what does that mean? Where? And words like ‘weak’, ‘good’ and ‘strong’—I mean, is the whole thing weak … and blank, pointless comments like ‘wrong’ or ‘irrelevant’ or random question marks, what use is that to anybody? Brown (2007, p.40)
Terms like ‘check spellings’ and ‘more depth’ are unhelpful, just as are vague phrases like ‘this is unclear’.Orsmond et al (2005, p.377)
What some students said
... I got told that a piece of work was more like an essay than a literature review. This is not helpful as it does not tell me what should be contained in a literature review or how it should be presented. Weaver (2006, p.388)
When it (feedback) was written I didn’t understand it, but obviously you make the appointment to follow through on that and she clarifies what she means …
I didn’t know where I’d gone wrong, what she meant, and she explained it, broke it down …
Murphy and Cornell (2010, p.47)
When writing is not enough
Some comments from a recent survey of first-year students in Art & Design
The assessment criteria are thoroughly explained at numerous points throughout the module so you know exactly what they are looking for.
To be honest, I'm not entirely clear how my work is assessed. It's never been explained in simple terms, instead, paragraphs of descriptions, I'm sure it must be simpler!
We get a copy of the assessment criteria with our brief so we always have it to refer to so we understand what exactly is expected.
Criteria
The written feedback was most helpful but was given too late at the end of the projects meaning there was no opportunity to improve.
My work is marked at several intervals throughout the year, rather than everything being marked at the end of the year, so I have several deadlines instead of one big final deadline. This reduces stress and helps me to keep on top of my work.
Interim verbal and written feedback half way through an assignment has been the most useful to me as it has allowed me time to take on board what has been said and use (or not) the comments made.
Timing
You get given a piece of paper at the end where they have ticked some boxes and given you a percentage. That is not an assessment, it’s disgusting for the amount of work you do.
Being given areas for improvement or development is the most useful [aspect of assessment] and that the fact that the formal aspect of this is written and you can reflect upon what is written is very useful.
Outcomes
Don’t forget ‘the personal’
Pedagogical approaches at tertiary level must motivate students to learn if deep, effective and engaged learning is to take place.
(Kift and Field, 2010, p.4, original emphasis)
Motivation
With striking consistency, studies show that innovative, active, collaborative, and constructivist instructional approaches shape learning more powerfully, in some forms by substantial margins, than do conventional lecture-discussion and text-based approaches.
(Pascarella and Terenzini, 2005, p.646)
Low Low
Moderate
Motivationlevel
Engagementlevel
Teaching approach
High High
Motivation and engagement
Students observed that feedback was given in such a way that they did not feel it was rejecting or discouraging . . .
[and] that feedback procedures assisted them in forming accurate perceptions of their abilities and establishing internal standards with which to evaluate their own work
Mentkowski and Associates (2000, p.82), emphasis added
Boud and Sadler (who argue for students’ internalisation of standards) would approve
Encouraging motivation
I was really, really scared because it was my first report. I had to hand it in just to see where I’m at. Where am I standing? (McGinty, 2011)
Anxiety
Psychological pain
They are writing all over my work and it is like mangled upand most of the lecturers use red pen and I don’t know it kind of gets to me if I open it up and it’s covered with red crosses and marks and it’s horrible. It’s like my work is bleeding. (McGinty, 2011)
Week 4 Week 10Lack of confidence 23% 40%
Expected ActualDistinction + 7% ~ 7%High pass 80% 57%Fail 1% 17%
Cameron (2008)
Self-confidence
For [first-year] students feedback goes beyond providing information on how to improve assessment marks. The ‘effective feedback’ for these students is that which provides emotional support and facilitates integration into university.(Poulos and Mahony, 2008)
Acknowledging emotion
To what extent can ‘the personal’ be accommodated in contemporary HE?
Message not quite clear?
The numbers and language used in assessment are opaque and need to be demystified
Standards
Normative (even when expressed in criterion-referenced terms).Discipline-driven, sometimes with external input.
Difficult to define (like QAA Subject Benchmarks?).But ‘We know them when we see them’ – or we ought to).
Or assessors may think they know standards, but notions such as • ‘graduateness’• ‘employability’• ‘wicked competences’• ‘soft skills’stretch assessors’ confidence to (beyond?) the limit.
Learning outcomes are a kind of proxy, but are inherently fuzzy.
Assessment criteria are likewise fuzzy.
A/1st: Able to recognise consistency and reconcile inconsistency between information using cognitive and hypothesising skills.
B+/2.1: Consistent understanding demonstrated in a logical and lucid manner.
B/2.2: Demonstrate understanding in a style which is mostly logical, consistent and flowing.
C/3rd: Attempts to demonstrate a logical and coherent understanding of the subject area but aspects become confused or are underdeveloped.
Refer/Fail: Understanding of the assignment not apparent, or lacks a logical and coherent framework, or the subject is confused or underdeveloped.
Price & Rust (1999, p.134)
The problem with criteria: ‘Conceptualisation’
How do you convey the meaning of all this to students?
Learning outcome Fully achieved Partly achieved Not achieved
A
B
C
...
Dealing with multiple learning outcomes: a hypothetical example
Intended learning outcomes [ILOs] might be categorised as primary or secondary, thereby influencing the overall assessment.Developed from Sadler (2005)
Do students appreciate the relative importance of ILOs?
Formal statements of expectations are fairly meaningless without exemplification, even for staff (e.g. Wolf, 1995).
Students’ perceptions of what assessment is seeking may not align with what staff think they ought to be perceiving (Maclellan, 2001).
35 of 76 history and politics students did not understand themeaning of ‘more analysis, less description’ (Chanock, 2000).
Staff themselves did not agree on what was meant by ‘analysis’ and ‘evaluation’ (Webster et al, 2000).
Whilst published assessment criteria are used, not all of them may be applied – and some extraneous criteria may be invoked(Webster et al, 2000, regarding u/g dissertations).
The potential for confusion: some examples
How well do grades inform?
The [honours degree] classifications are too general, there’s no way of showing if you were close to the grade above.
A straight percentage mark would be fairer.
‘Maria’
The Guardian RISE supplement, 18 December 2004
Not so. For a start, there’s variation between subject areas
L1 Module means from 13 Owning Organisational Units
CMP 5 LSS 31 ENR 8 PBS 19 NSP 29 ECL 75 BUE 16 LSA 26 SPS 11 HSS 37 HEA 16 LBS 34 LAW 1030.0
40.0
50.0
60.0
70.0
80.0
So why the variation?
Lots of variables may have exerted influence, including:
• Student calibre (entry qualifications; commitment)• Nature of the subject (hard/soft; pure/applied)• Curriculum design• Pedagogic quality• Resourcing• Expected standards (intended learning outcomes)• Mode of assessment• Nature of the assessment demand• Marker variability (in some cases, due to differences in School)
What is the relationship between mode of assessment and marks awarded?
65+
60-64.99
55-59.99
50-54.99
<50
Mean CW = <35% CW = 35-75% CW = >75%
BUE HSS LBS LSA LSS NSP
Civil engineering surveying 1, CW=30%
LJMU L1 module mean scores (N=317), by amount of coursework
LJMU uses ‘percentages’. If you are marking work out of 100, how do you arrive at the ‘percentage’ mark? (If marking work out of a smaller total, the same general question applies.)
• You might tot up the marks from components of the student’s performance.• You might take a broad view of the overall performance (e.g. ‘this is of 2.1 standard’), and then decide where in the relevant band the actual mark should lie.• You may have some other approach.
What approach do you use?
And what assumptions underlie your marking methodology?
‘Fudges’? (e.g. Baume et al 2004; Bloxham et al 2011)
A question to ponder ...
Should you finely grade work-based achievements, employability and ‘graduateness’?
If ‘yes’, then how?
Too complex an issue for this session, but a starting point is
Yorke M (2011) Assessing the complexity of professional achievementhttp://learningtobeprofessional.pbworks.com/f/CHAPTER+A10+FINAL.pdf
What assumptions are being made?
(a)By staff(b)By students(c)And by others
0
5
10
15
20
25
30
35
40
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17
0
5
10
15
20
25
30
35
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17
The psychometric tradition would give
Using learning outcomes would give
But it’s rarely as cut and dried as that
Four modules with varied mark profiles
1-5 6-10
11-15
16-20
21-25
26-30
31-35
36-40
41-45
46-50
51-55
56-60
61-65
66-70
71-75
76-80
81-85
86-90
91-95
96-100
0
10
20
30
40
50
1-5 6-10
11-15
16-20
21-25
26-30
31-35
36-40
41-45
46-50
51-55
56-60
61-65
66-70
71-75
76-80
81-85
86-90
91-95
96-100
0
10
20
30
40
50
1-5 6-10
11-15
16-20
21-25
26-30
31-35
36-40
41-45
46-50
51-55
56-60
61-65
66-70
71-75
76-80
81-85
86-90
91-95
96-100
0
10
20
30
40
50
1-5 6-10
11-15
16-20
21-25
26-30
31-35
36-40
41-45
46-50
51-55
56-60
61-65
66-70
71-75
76-80
81-85
86-90
91-95
96-100
0
10
20
30
40
50
0 100
0 100
0 100
0 100
Perf Arts N=77CW 30+70%
(Essay+Portfolio)
Perf Arts N=48CW 30+70%
(Perf Proj+Cont Ass)
Acc&Fin N=192CW 10+30%; Ex 60%(CW=Diag Test+rep)
Acc&Fin N=193CW 100%
(2 reports + log)
Grades are not transparent
They can demoralise if they are awarded on a basis that differs (negatively) from students’ previous experience.
Grading at A-level, or in other countries’ systems, may not align with UK HE practices.
What does a mark of 64% mean?
And what does the ‘missing’ 36% relate to?
How different is 53% from 54%?
And so on ...
Grade only on assignment and/or feedback boxes ticked (or not)
Mark out of sync with comment
Feedback comment on matters outside stated criteria
Students not taught how to use feedback
The difference between being positive and being constructive
The need to provide emotional support (esp for 1st year students)
Some other issues
Rounding up:
Some ‘big picture’ issues
In a nutshell ...
1. Don’t assume that students will understand ‘assessment language’. Give explanations and, particularly, examples.
2. Provide opportunities for students to develop their capacity to self-assess, without undue risk.
3. Find ways to make feedback (and feedforward) effective.
4. Clarify what grades signify, in the subject context.
1 and 4 are the responsibility of teachers, and are fairly straightforward.
2 and 3 are much more challenging, in that they may require a measure of innovation – creativity even – on the part of teachers and certainly the active engagement of the students.
Some references
Assessment and Evaluation in Higher Education is a particularly useful source of relevant articles, e.g.• Maclellan, 2001• Orsmond et al, 2005• Weaver, 2006• Webster et al, 2000
Bloxham S and Boyd P (2007) Developing effective assessment in higher education: a practical guide. Maidenhead: Open University Press.
Boud D & Falchikov N, eds (2007) Rethinking assessment in higher education: learning for the longer term. London: Routledge.
Nicol D (2009) Transforming assessment and feedback: enhancing integration and empowerment in the first year. http://www.enhancementthemes.ac.uk/documents/firstyear/First_Year_Transforming_Assess.pdf
Yorke M (2008) Assessing student achievement in higher education: signals and shortcomings. Abingdon: Routledge (Contains a wide range of references.)
Yorke M (2011) Assessing the complexity of professional achievementhttp://learningtobeprofessional.pbworks.com/f/CHAPTER+A10+FINAL.pdf
Yorke M (2011) Assessment and feedback in the first year: the professional and the personal. At www.fyhe.com.au/past_papers/papers11/FYHE-2011/content/pdf/MantzYorke_abstract.pdf
