Assessing scenario-based learning

John Mitchell

Electronic and Electrical Engineering

With thanks to: Sally Day, Rosalind Duhs, Lewis Elton, Tony Kenyon, Oleg Mitrofanov, Cyril Renaud, Miguel Rio, Ed Romans, Seb Savory, Jan Smith, Benn Thomsen.

SYMPOSIA ONASSESSMENT FOR LEARNING

Developing scenario-based learning forformative and summative assessment• It is not in fact teaching but assessment that plays the

largest part in guiding students’ learning. 1

• ‘it can be argued that giving learners feedback is just about the most important dimension of the work of teachers in post-compulsory education … but perhaps all told, formative feedback is the vital dimension, as given at the right time and in the best possible way it can lead learners steadily towards successful achievement in summative assessment contexts. 2

1 Snyder B.R, (1971) The Hidden Curriculum. New York; Alfred A Knopf2 Race P. (2005) “Making Learning Happen”, Chapter 5, Learning through Feedback, London, Sage

Proforma

• Very useful for general feedback

• Common areas of mistakes• Less easy to use for more

‘open’ task

Scenarios

• Follow the model developed by Civil Engineering of one week, self contained, group work scenarios (problems)

• Aim of supporting learning rather than teaching new material

• Pilot process of 3 first year scenarios and two second year scenarios just completed

Scenario Assessment

• A mix of instruments used– Milestones– Presentations Individual technical reports

• Traditional reports• Critical Assessments of other teams solutions

– Group technical report• User manual• Due diligence document

– Individual Narratives

History of Student-Centred Learning in E&EE

• 2004 – Problem Based Learning Modules Begin• 2008 – First Scenario Weeks Piloted

STUDENT–CENTRED LEARNING: WHAT DOES IT MEAN FOR STUDENTS AND LECTURERS? Geraldine O’Neill and Tim McMahonIn: Emerging Issues in the Practice of University Learning and Teaching

Problem Based Learning

• 4 Problems over 12 Weeks (Term 1)• 6 Students per group, groups stay the same, • Students expected to spend 10 hours per week

total (inc private study) on a problem• No lectures!• Reports and Presentations

Rational for assessment developments

• Students didn’t perceive that they were receiving feedback

• Highlighted in the national student survey• Feedback on my work has been prompt.

• Agree – 41%• I have received detailed comments on my work.

• Agree – 49%• Feedback on my work has helped me clarify

things I did not understand.• Agree – 53%

“The NSS has highlighted that students are notably less positive about assessment and feedback on their assignments than about other aspects of their learning experience.” Exploring the National Student Survey, The HEA

NSS - Engineering and Technology, UCL

perceive

Problem Based Learning

• Iterative Assessment Cycle– Report submitted at end of problem for summative

assessment– Opportunity to use feedback to improve report– At End portfolio of reports submitted for formative

assessment

• Narrative Commentary introduced• Peer Assessment

Re-submit -

Formative

Summative Feedback

Submit

BSummative Feedback

• Referencing:– Referencing of figure that have been taken from sources is good.

However, there are few references in the text. This must be improved.

• Other Comments:– Good report which is well presented. There are some parts that are

better than others, and although some attempts have been made to pull things together summary of the specifications would have been useful.

• Grade: B

Narratives

• The following are suggested questions to guide you in developing this piece of writing:– How did my group go about tackling this problem?– Was this a successful approach?– If so, what made it successful and how can I use that in future?– If not, what were the major difficulties, and how would I avoid them in future?– How good was our solution to the problem?– If I could tackle this problem again, how could I improve what we did?– How have I learned to cope with all the information I had to find for myself?– Has the group shared this information effectively?– How have we handled difficult issues in our group?– Did I do my best to ensure that the team worked well?– Do I recognise the importance of deadlines?– Did I and/or my group stick to the deadlines?– What else do I feel is of importance to include?

Narratives – advice given

• Merely documenting the steps you have gone through in the problem solving process will not earn you a very high mark – you will need to demonstrate what you have learned from the process.

But often ignored!

Narratives

• “Oh yeah, the narrative [wry chuckle]. The narrative, I didn’t like the way the narrative was 50% of the marks because we spent a lot of time on those reports, a lot of time doing research into suitable system diagrams

and everything else and I didn’t like the way half the marks just

went to the narrative…”

Opportunities for Feedback in Scenario Assessment

Formative

Summative

Encouraging Brainstorming

Milestones

Presentations

• Presentations mid-way as guides rather than later as assessment

• Student dedicate significant time to presentations• Too much focus on

‘Information’ not enough

on ‘message’

• Opportunity to learn from

other groups and provide

mid-point feedback

Competitions

Reports

• Attempted to get away from ‘usual’ reports– User Manuals/Documentation– Very specific guidance given

Marking Grid

Marking Proforma 1st Grade /5 2nd Grade /5Problem Definition

This could have been improved, You need a lot more details

2

Design description

It is not clear why some choices have been made. And a bullet point time line does not give any idea on some of the design. Your design is based on an error in your data-collection.

2

Theoretical Analysis

The theory is adequate but you have not used it to its full potential.

3

Results and Conclusion

There is almost no analysis in the conclusion, and you do not compare your results with the theory.

1

Individual input

You claim to have done your part of the work over 1 day. You should have spend more time trying to understand the rest of the issues.

2

Reflective commentary

More comments on what you have learned are needed

3

Total Mark 2 0

Written Feedback

Student Debrief and Feedback Session

• Not yet fully utilised• General comments to the whole group• Citing good and bad practice

Assessment Weighting

• Course marking scheme (1-5 range)• Currently a low percentage of final grade• All aspects rewarded but none too heavily

• Group vs Individual?• Peer Assessment?

Conclusions

• Student always appreciate feedback• Even if they then go on to ignore it!• Use assessment to guide learning rather than just

to judge learning• Linking feedback to further assessment seems

vital

Scenario Brief

Milestones

Feedback

Assessment

Summative Feedback

X