Upload
kenji-lamb
View
107
Download
0
Tags:
Embed Size (px)
DESCRIPTION
Presentation delivered by Peter Hartley, University of Bradford, at the 2011 e-Assessment Scotland conference.
Citation preview
Assessment Assessment Futures: role for Futures: role for e-Assessment?e-Assessment?
Peter Hartley, University of [email protected]
http://www.brad.ac.uk/educational-development/aboutus/team/Full_details_27414_en.php
This sessionThis session
o Why bother? Why worry?o Assessment trends and futureso What can e-assessment contribute?
And the argumentAnd the argument
o Assessment is a problem which will ‘get worse’.
o Assessment futures: trends re strategy and environment.
o Significant potential for ‘e’: some examples.
o And so the role for e-assessment must be part of a broader strategic initiative.
Assessment is Assessment is a problem, #1 a problem, #1
• Ask a student of X:what makes a ‘First Class X?
Assessment is Assessment is a problem, #1 a problem, #1
• Ask a student of X:what makes a ‘First Class X?’o How would they respond?o Can they give you a coherent summary of the
main programme outcomes (or just the algorithm that determines the marks)?
Assessment is Assessment is a problem, #1 a problem, #1
• See useful summaries of major issues:o the the PASS Project Issues Paper
Please comment/feedback and use.• http://www.pebblepad.co.uk/bradford/viewasset.aspx?oid=260486&type
=file
o Recent article:• Price, M., Carroll, J., O'Donovan, B. and Rust, C. (2011)
''If I was going there I wouldn't start from here: a critical commentary on current assessment practices', Assessment & Evaluation in Higher Education. Vol. 36, No. 4, 479-492.
Assessment is Assessment is a problem, #2 a problem, #2
• From PASS, would highlight:o Assessment ‘drives and channels’o What/why are we measuring: the‘slowly
learnt’problem.o Limitations of grading (e.g. marks are not
numbers).o Implications for course structures/regulations.
Problem #3: multi-Problem #3: multi-purpose purpose
& multi-audience& multi-audience
Problem #4: Problem #4: The meaning of The meaning of
feedbackfeedback• Cannot we ‘recapture’ the ‘original’ meaning of
feedback: enabling self-correcting behaviour towards a known goal.
• This means rediscovering the ‘feedback loop’ whereby information must be ‘fed back’ so that it:• relates to the goal.• is received.• is correctly interpreted.• enables corrective action
cf. the work of Royce Sadler in Higher Education, e.g. http://www.northumbria.ac.uk/sd/central/ar/academy/cetl_afl/earli2010/themes/rsadler/
Problem #5 Problem #5 • More
demanding ‘consumers’:
• E.g. the NUS Charter
Assessment futures:Assessment futures:strategy and strategy and environmentenvironment
Environment:Environment:TESTA project
• NTFS group project with 4 partners: ‘aims to improve the quality of student learning through addressing programme-level assessment.’
• starting from audit of current practice on nine programmes: osurveyed students using focus groups and AEQ – Assessment Experience Questionnaire – Graham Gibbs et aloalso using tool to identify programme level ‘assessment environments’ (Gibbs)
Consistent practice?Consistent practice?
Characterising programme-level assessment environments that support learning by Graham Gibbs and Harriet Dunbar-Goddet Published in: Assessment & Evaluation in Higher Education, Volume 34, Issue 4 August 2009 , pages 481 - 489
Data from TESTAData from TESTA
Assessment Assessment environment and environment and
impactimpact• Interim findings from TESTA
o variety of assessments can cause problemso Issues over understanding assessment criteria,
marker variation, and feedbacko variation across programmeso QA ‘myths and traditions’ can get in the way
The need for strategyThe need for strategy• An example finding from Gibbs
o ‘greater explicitness of goals and standards was not associated with students experiencing the goals and standards to be clearer’
• And what did make a difference?
The need for strategyThe need for strategy• An example finding from Gibbs
o ‘greater explicitness of goals and standards was not associated with students experiencing the goals and standards to be clearer’
• And what did make a difference?o Formative-only assessmento More oral feedbacko Students ‘came to understand standards
through many cycles of practice and feedback’
Programme-based Programme-based assessment: assessment: PASS
• NTFS group project over 3 yearso development and investigation leading to
pilots and implementation
• Consortiumo Led by Bradfordo 2 CETLs – ASKE and AfLo Plus Exeter, Plymouth and Leeds Met.
What are we What are we investigating?investigating?
How to design an effective, efficient, inclusive and sustainable assessment strategy that delivers the key course/programme outcomes.
Also look out for:Also look out for:
And also …And also …• New JISC Programme
on Assessment and Feedbacko See the Strand A projects.
Typical student concerns Typical student concerns (based on PASS)(based on PASS)
• perceptions of ‘the course’ variable.• assessment experienced as ‘fragmented’.• anxieties re move to more integrated
assessment – perceived risk in terms of performance.
• Concerns about feedback and timing.
Searching for typesSearching for types
Searching for typesSearching for types
An example from PASS: An example from PASS: Peninsula Medical Peninsula Medical
SchoolSchoolIncludes:•four assessment modules that run through the 5 year undergraduate medical programme and are not linked directly to specific areas of teaching•focus on high-quality learning (Mattick and Knight, 2007)
Further case studies Further case studies being exploredbeing explored
• Brunelo New regulations which separate study and
assessment blocks.o Visit last week.
• Liverpool Hopeo New regulations which ‘abandon modules’ in
all undergraduate programmes.o ‘Key Honours Assessment’.o Visit now arranged
Brunel: the regsBrunel: the regs• 120 credits per year of study.• Course/programme can include mix of
study, assessment and modular blocks.• Option blocks must be modular.• Blocks must be in multiples of 5 credits• Maximum assessment block is 40
points
Examples from BrunelExamples from Brunel• Biomedical Sciences
o Study and assessment blocks in all years.o Cut assessment load by 2/3rds; generated
more time for class contact.o Synoptic exam in all three years.
• Mathematicso Conventional modules in final year only.o Improved understanding and ‘carry-over’ of
‘the basics’ into year 2.
Do you PASS?Do you PASS?
What can What can e-assessment e-assessment
achieve?achieve?
1. Processes and 1. Processes and systemssystems
Examples from the Examples from the Curriculum Design Curriculum Design
Prog.Prog.• eBioLabs (Bristol)
o By combining interactive media with formative self-evaluation assessments students learn the methods and techniques they will use in the lab, without risking valuable time, equipment or materials. Because students first experiment on-line there is a reduced chance of cognitive overload during the practical and they are more able to concentrate on the wider aims of the experiment, rather than blindly following the lab instructions.
o Because eBiolabs includes tools that automatically mark student assignments, and tools that allow academics to easily track student attendance and achievement, the marking and administrative burden associated with running practicals is very significantly reduced.
Curriculum Design Curriculum Design exampleexample
• CASCADE project (Oxford)o Online assessment submission
• Students can now submit assignments much more easily at any time from anywhere in the world. It is also possible to predict significant efficiencies in assignment handling time for the Registry staff who deal with student submissions for approximately 260 course assignments across 48 course cohorts a year: a saving of 30 minutes per assignment or more soon cumulates savings in the order of half a day per week. Other advantages of the new online system are the reduction in paper handling and photocopying, as well as better auditing and control. Reduction in paper storage is a further advantage both in terms of less physical space being required and also in terms of less staff time being required to retrieve data from the archive.’
Curriculum Design Curriculum Design exampleexample
• ESCAPE project (Hertfordshire)o Effectiveness vs efficiency. (watch the video)
o Next two slides from Mark Russell, Deputy Director of the Blended Learning Unit, University of Hertfordshire
ESCAPE Themes …ESCAPE Themes …
Good assessment for learning:…Engages students with assessment
criteria…Supports personalised learning…Focuses on student development…Ensures feedback leads to improvement…Stimulates dialogue…Considers staff and student effort
Numerous legacy resources
2. Feedback 2. Feedback and self-evaluationand self-evaluation
Example 2.1: audioExample 2.1: audio• The ASEL project
o led by Bradford with Kingston as partner.o various uses of audio, including feedback, in
different disciplines.
o See the intro by Will Stewart as part of the recent Leeds University Building Capacity project:
• Jorum resource • Direct weblink
ASEL noted:ASEL noted:• Technology now easy and accessible.• Positive student reactions.• Different tutor styles and
approaches.• Serendipity – e.g. feedback
stimulated podcasts.And• A different form of communication?
ASEL main ASEL main conclusionsconclusions
o … audio is a powerful tool, providing opportunities for personalising learning, promoting greater student engagement, and encouraging creativity. In introducing audio into their practice, lecturers were required to rethink their pedagogical approaches and learning design, adopting new and innovative ways to enable students to be more actively involved in the learning process.
ASEL main ASEL main conclusionsconclusions
o … (audio) allowed lecturers to provide more personal and richer feedback to students, and increased the level of interaction and dialogue amongst students and between students and lecturers.
(Will Stewart and Chris Dearnley)
Example 2.2:Example 2.2:audio and videoaudio and video
• Growing number of examples.o ALT/Epigeum Awards 2010: see the
ALT Open Access Repository
• See the winning entry by Read and Brown from Southampton:o Organic Chemistry.o Use of tablets to show solutions and working.o Focus on self-assessment.
Example 2.3:Example 2.3:clickers are comingclickers are coming
• Student Response Systems at the moment?o They work … they can change staff and
student behaviour and performance.
• Buto can be cumbersome and fiddly.o setup time.o need strong commitment and support (e.g. see
experience at Exeter Business School).
Example 2.3:Example 2.3:clickers are comingclickers are coming
• Student Response Systems in the future?o They will radically change staff and student
behaviour.o They will be flexible and easy to use.o They will be on the student’s own device!
• e.g. experimentation at University of Bradford arising from our JISC Building Capacity project – work with TxtTools by John Fairhall et al)
Example 2.4:Example 2.4:adaptive systemsadaptive systems
• PBL with consequences – you get immediate feedback on the consequences of your decisions.o e.g. The G4 project at St George’s
• http://www.generation4.co.uk/ • iEthics case online
• Adaptive assessmento e.g. the work of Trevor Barker
3. Integrating systems3. Integrating systems
Example 3.1:Example 3.1:integrating systems: integrating systems:
CAACAA• ITS4SEA project at Bradford
o 100-seater facility, now plus break-out 30 seats.
o Thin client technology.o QMP as University standard for summative
assessment.o Procedures agreed with Exam Office.o Design of room (available as cluster outside
assessment times)o Teaching potential.
The main The main CAA room at BradfordCAA room at Bradford
And the growth …And the growth …
And recent changes …And recent changes …• growth in ‘hybrid exams’:
o mix of automatic marking (QMP) and open ended response items (e.g. short answer questions).
o short answers to spreadsheet & marked by human.o likely use of word-processing in future.o Impact on teaching - new flexibility.
• Example of impact: o ‘reduced my marking load for this module from 5
days to one day, whilst still enabling assessment of higher order cognitive skills.’
Example 3.2:Example 3.2:integrating integrating applicationsapplications
• Use of mobile technologyo e.g. CampusM at Bradford:
• http://www.campusm.com/• http://www.techrepublic.com/software/university-of-bradford-about-
uob-10-mobile/2194295?tag=content;selector-1
And finally … And finally … back to the roleback to the role
• E-assessment can play an important role re:o Assessment authenticity and diversityo Feedback quantity and qualityo Profiling and ‘mindset’.
BUT ONLY IF• Embedded in meaningful
course/programme strategy/environment
And what is an ‘effective And what is an ‘effective assessment strategy’?assessment strategy’?
• Will it explain to staff, students and external agencies:o How the course/programme assesses the main
learning outcomes?o How assessment and teaching are linked?o How assessment both supports ‘high-quality
learning’ and develops it over the course?
And finally …And finally …assessment/identity assessment/identity
interfaceinterfaceStudents as ‘conscientious consumers’
(Higgins et al, 2002).But:• personal identity as ‘mediator’.
o e.g. apprentice (‘feedback is useful tool’) cf. victim (‘feedback is another burden’).
So need to change the mindsets of some students.
And finally finally …And finally finally …some other contactssome other contacts
PASSo Project Manager: Ruth Whitfield
ASELo Project Manager: Will Stewart
• CAA (building on ITS4SEA)o Project Manager: John Dermo
And maybe students And maybe students will never again see …will never again see …
59% Excellent.
This was the only tutor comment on a student assignment. How do you think the student reacted and felt?
And some other And some other interesting stuffinteresting stuff
• Challenging students to search Wikipedia creatively: C-Link
• http://www.conceptlinkage/clink /
• Helping students review and evaluate their interview performance
• http://www.gowerpublishing.com/isbn/9781409411369
• Helping research students to prepare for their viva: Interviewer VivaContact me for further info/demo.
Thank you for listeningThank you for listening
Peter [email protected]