SOLSTICE Conference 2015

4th & 5th June 2015

Technology enhanced formative assessment: Participant experiences, relative efficiencies and tutor learning curves

Dr W. Rod Cullen, Manchester Metropolitan University

Dr W. Rod Cullen Susan GregoryDr Neil Ringan

Mark Roche

Formative assessment...“... is designed to provide learners with feedback on progress and inform development, but does not contribute to the overall assessment.”

QAA 2000

“I can’t get my students to do

work that doesn’t get a mark”

“My students don’t use the

feedback I give them – sometimes

they don’t even pick it up”

“That all looks very impressive,

but how long is it going to take me

to do that?”

Session outlineNarrative account of evolution of “Enhancing Learning Teaching and Assessment with Technology” unit (ELTAT)•Delivery model•Feedback methods•Feedback efficiencies•Participant experiences•Conclusions and recommendations

A brief history of ELTATDate Unit Title Credit Weeks Tutor(s)

2007-09 Designing Courses for VLEs (DCVLEs)

10 4 WRC

2009-10 Designing Courses for VLEs (DCVLEs)

10 4 WRC, SG

2011-13 Design Effective Blended and Online Learning (DEBOL)

10 4 WRC, SG with guest appearances from NR and MR

2014 Enhancing Learning Teaching and Assessment with Technology (ELTAT)

30 12 WRC, SG, NR, MR

Learner Profile• Range of backgrounds

– New and experienced academic staff– Academic support staff– Technical support staff

• Majority working full-time but some part-time• Differing assessment requirements

– PG Cert Academic Practice– MA Academic Practice– CPD for credits– CPD non-credits

• Just like real students!

Formative delivery model

Online Preparation

In ClassSession 1

Onlineformativeactivity

Online Preparation

In ClassSession 2

Onlineformativeactivity

Week 1 Week 2

Feedback

Mini (e)Portfolio

Feedback

Providing feedback – Text based

Audio feedbackMS Word file including annotations

MP3 audio file

Screen capture feedback

Data CollectionTutor Activity – since 20071. Kept a detailed

feedback activity log2. Worked out time spent

engaged in each feedback activity.

End-to-end process and each stage: Download, Annotate, Write/ Record, Publish, Upload

Participant experience – 2007-2014

1. Interviewed participants about experiences

2. End of unit survey3. Submission data4. Attendance5. Informal anecdotal

evidence

Ongoing Opportunistic Action Research

Early Experiences DCVLEs

Feedback methods (DCVLEs)Methods DetailsWritten Annotated comments on Submission MS

Word + Written Comments

Audio Annotated comments on Submission MS Word + Audio recording

Video Screen Capture

Annotated comments on Submission MS Word + Video Screen Capture

Producing formative feedback1. Getting up to speed with new technology (Pebblepad)

2. Technical problems with codecs and files sizes (common with web-based video)

3. Initially uncomfortable with recording process

Issues for Susan

4. Got up to speed with audio format more quickly

Cullen W. R. (2011)

Per

cen

tag

e

Attendance and engagement with formative tasks 2007-10

1. Reasonably high overall attendance

2. Generally high (70% +) submission rates for formative tasks

4. Consistently lower submission rates in Week 4

MASummative Assessment

3. Use of the feedback is embedded in the online prep and F2F activities

Ongoing development2011 Programme Review:Designing Effective Online and Blended Learning (DEBOL)

• Change in unit title• Neil and Mark – contributions as specialist lecturers

2013 Major Programme Review:Enhancing Learning Teaching and Assessment with Technology (ELTAT)

• 30 Credits • Much longer delivery – 12 weeks• More time to complete tasks (and provide feedback)• Team approach to formative feedback – Tutors allocated group

of student to work with throughout the course

Recent experience (ELTAT)

Same delivery and assessment model

Feedback methodsFormative Activities

Learning outcomes review

Complete LTA matrix review

Revise LTA matrix

Assessment strategy

Revised LTA matrix 2

Monitoring and evaluation plan

Timeline

ELTAT - 2014Feedback provision

Moodle Assignment; MS Word Comments +Audio recording

Moodle Assignment; MS Word Comments +Video Screen capture

Moodle Assignment; MS Word Comments + Written Proforma

Turnitin Assignment; Quickmarks, General Comment + Audio recording

Moodle Assignment; MS Word Comments +Audio recording

Moodle Assignment; MS Word Comments +Audio recording

Moodle Assignment; MS Word Comments +Audio recording

Producing formative feedback1. Rod + Susan very similar times

2. Neil + Mark learning the ropes

3. Pronounced for video and audio

About the task as much as the technology

4. Similar times for written feedback – new task

5. Similar times for TII (Written + Audio)

Team overview1. For team as a whole consistent ≈ 55 mins per submissions (end-to-end)

2. Variation in what the time is required for

3. Download times minimal

4. Upload of audio and video = minimal time but 2-3 mins more than written.

5. No down/upload for TII

6. Additional “publish” stage for audio

7. Reading through and annotating the submissions is most time consuming activity

Efficiency of productionWritten feedback:

•Word count tool in MS Word

•Divided by the total time (end-to-end) to produce the feedback

Screen capture and audio feedback:

•Counted spoken words in 1 minute of recorded feedback = est. speaking rate.

•Length audio and video recording X speaking rate = estimate of the total number of words in each piece of feedback.

•Divided by the total time (end-to-end) to produce the feedback

Turnitin: •Pasted the general text comments into a word document – Word count tool.

•Length of audio feedback X tutors speaking rate = estimate of number of word spoken.

•Work count + Words spoken = estimate of the total number of words in each piece of feedback.

•Divided by the total time (end-to-end) to produce the feedback

Annotations (comments in MS word and Quickmarks in Turnitin) not included

Estimated Feedback words/per minute1. Audio and Video screen capture – learning curve?

Task as well as tech

2. Written feedback - Personal preferences and expertise is an influence

3. TII least efficient – multi-media + suspect we made more comments in the text (Quickmarks)

5. Comparing efficiencies across technologies is more complex than anticipated

4. Mark has felt need to reflect on his feedback on all tasks

Student perspectives

Prefer audio and video to written feedback

Personal and Engaging

Understandable

Annotations important

50:50 split between audio and video

Utilised differently

Video more impact

Audio more reflective

• Learning curves associated with – “Technical skills”– Feelings of self-consciousness– Familiarity with requirements of the formative task

• Technical aspects not as time consuming as cognitive• Audio feedback can be produced quickly and simply• Model of delivery can influence engagement – need to add

value to completing formative tasks• Participants indicated a preference for audio and video screen

captures– Linked to characteristics of the task– Possible novelty aspect

ConclusionsTake home messages

To embed formative assessment into a curriculum •Design it into delivery model – make the value of the feedback clear•Align with summative assessment and signpost this to students•Select feedback technology best matched to the type of tasks•Be prepared for the learning curve

Just in case section

Feedback methodsOnline Formative Activities

Learning outcomes review

Complete LTA matrix

Revise LTA matrix

Activity Design Plan

Monitoring and evaluation plan

Timeline

2011-14Feedback provision

Moodle Assignment; Track Changes/annotated script MS Word

Moodle Assignment; Track Changes/annotated script MS Word

Moodle Assignment; Annotated MS Word doc + Video screen capture

Moodle Assignment; Annotated MS Word doc + Audio recording

Moodle Assignment; Annotated MS Word doc +Audio recording

Moodle Assignment; Annotated MS Word doc +Audio recording

2008Feedback provision

Online discussion forum; Track changes/ annotated script MS Word

Annotated MS Word doc + Written Comments

Annotated MS Word doc + Video screen capture

Annotated MS Word doc + Video screen

Annotated MS Word doc + Video screen

Annotated MS Word doc + Video screen

2009 and 2010Feedback provision

Written proformas via Pebblepad

Written proformas via Pebblepad

Annotated MS Word doc + Video screen capture

Annotated MS Word doc + Audio recording

Annotated MS Word doc +Audio recording

Annotated MS Word doc +Audio recording

Data Collection• All tutors kept details feedback activity logs relevant to

the technologies being usedTrack Changes + Audio recording

Track Changes + Screen Capture

Track Changes + Written Proforma

Turnitin (Quickmarks, Comments, Audio)

Download Download Download

Annotation (Track Changes)

Annotation (Track Changes)

Annotation (Track Changes)

Annotation (Quickmarks)

General Text Comment

Recording Recording Recording

Publishing

Upload Upload Upload

Logistical considerations• DCVLEs & DEBOL versions

Core Option 10 credits – Nominal 100 hrs• Very limited F2F contact (4 x 3 hours sessions - 12 hours

in total)

• ELTAT• 30 Credits – Nominal 300 hours• More F2F • Over a long period

Design considerations• Relevant• Purposeful