Electronic management of assessment - Jisc Digital Media 2015

Electronic management of assessment

Maximising the benefits technology can offer

» Background and context

» Headline findings from EMA landscape review

» Consulting on priorities

» From challenges to solutions

» Solution scoping activity

Overview

10/03/2015 Jisc Digital Festival, 9-10 March 2015, ICC Birmingham 3


Background and context

bit.ly/jisc-assess and http://bit.ly/jiscdsaf

» Background and context

› Recently completed – a 3 year programme exploring technology-enhanced assessment and feedback

» Four main themes:

› Feedback and feed forward

› Enhancing student employability

› Strategies for change

› Electronic management of assessment (EMA)

http://bit.ly/jisc-assess

http://bit.ly/jiscdsaf


» EMA - benefits and drivers are clear

» Challenges surfacing relating to process, practice and technology

» Working in collaboration with the Heads of e-Learning Forum (HeLF)and UCISA to maximise the benefits technology can offer in the management of assessment

Electronic management of assessment project


» Aims:

› To gather examples of practices, policies & process as well as approaches to implementation

› To identify the challenges around EMA, in terms of pedagogy/culture, process or technology and to identify existing solutions where they can be found

› To provide interim guidance to institutions on the state of play and effective practice with approaches to implementation

› To make recommendations to institutions, systems providers and Jisc on priorities to address

The EMA study

» HeLF EMA surveys

» Jisc assessment and feedback programme: 30+ institutions

» Online questionnaire: 70 institutions

» Think Tank: 32 institutions

» Other Jisc learning and teaching related projects

» Literature review

» Interviews with learning providers

Information sources


» Nuanced responses but broad trends visible

» Localised initiatives but beginning to scale up

» System integration is a key problem area

» EMA exposes variability in business processes

Headline findings


Loose system integration


Variation in business processes



What are the pain points in EMA?

Pain points

» Anonymity: both ends of spectrum motivated by student pressure

» Resistance: many stakeholders take narrow view; little joined-up thinking

» Scaling up: may expose technology issues & ‘harden’ staff attitudes

» Feedback: storage & access issues; good practice can be hard to scale

» Practice: may be simple but process workarounds cause frustration and resentment

» Technology: limited range of core products how hard can it really be?

» Formative approaches: less well developed than summative

What causes the pain?


A lifecycle model


Priorities 1


Top 10 EMA challenges

Priorities 2


Top 20 EMA challenges


» Review of the EMA landscape across UK HE and FE completed

» Identified and prioritised the challenges around EMA

» Workshops in Dec 2014 and Jan 2015 explored possible solutionsto these prioritised challenges

Where are we now?

ema.jiscinvolve.org/

http://ema.jiscinvolve.org/

Where next?


Phase 2.1Oct – Aug 2015

Second phase of consultation, refining and prioritising challenges (speaking as a ‘single voice’)

Moving from challenges to solutions (Oct – Jan)

Solution development (Feb – July)

Moving from challenge to solutions –

4 projects going forward

Ongoing capture of resources –‘EMA toolkit’

Landscape review –

challenges prioritised

» EMA Requirements Map

» Feedback hub

» Reliable submissions

» EMA systems integration web resource

» Assessment & feedback toolkit

Four projects



» Project to identify, validate and specify key EMA requirements and workflows, through development of generic models that represent the diversity of practice

» Providing clarity and transparency on workflows (particularly from submissionto return of grades). Helps systems suppliers better support good pedagogic practice and helps institutions review their own practice

EMA requirements map


» Develop a visual way of presenting workflows; map current systems to workflows; and engage with suppliers and developers to fill gaps

EMA requirements map


» System independent tool for aggregating, organising and presenting feedback at a programme level, for both staff and students. Also enables interaction around feedback between staff and students

Feedback hub


» Students benefit from aggregatedview of their feedback to support self-reflection on progress; lecturers see a more holistic view of students’ progress and better understand an individual’s progression to identify where intervention or support is needed

Feedback hub


» Resource to address lack of interoperability between marking systems and student records systems and subsequent need for ‘workarounds’ by staff

» Helps institutions find solutions to EMA systems integrations issues via improved workflows and by sharing existing integrations (use cases and code)

Systems integration


» Gaps can be surfaced and prioritised for potential development

» A community of practice might support this resource

Systems integration


» Web-based toolkit of resources: practices, processes, policies, technical integrations etc based around the assessment and feedback lifecycle

» Examples of solutions , enabled by technology, based on pedagogy and underpinned by research, that can be repurposed for local contexts

EMA toolkit


» Addresses the question of ‘what does good assessment design look like?’ and enhances assessment literacies of staff and students

» Navigation routes through the toolkit e.g. by identified workflow and stakeholder groups (academic staff, administrative staff, student)

EMA toolkit

» Choose one project area to work on

» The aim is to help to shape the detail of these projects and your facilitator will provide guidance. Discuss:

› What would you like to see from this project and its outputs?

› What form should the outputs take?

› How might you use the outputs within your context?

› What do you have that you can contribute?

› How can we best reflect the needs of HE and FE within this project?

Activity



» Landscape report available from: bit.ly/jisc-ema

» Join the conversation on the blog: ema.jiscinvolve.org/and on twitter #jiscassess

» Join the mailing list:

www.jiscmail.ac.uk/tech-enhanced-assessment

» Guides, case studies, videos and other resources on assessment and feedback: bit.ly/jisc-assess

Find out more

http://bit.ly/jisc-ema

http://ema.jiscinvolve.org/

http://www.jiscmail.ac.uk/tech-enhanced-assessment

http://bit.ly/jisc-assess

Find out more…

Contact…

Lisa GraySenior Co-Design manager

[email protected]

Gill Ferrell, [email protected]

Education

Electronic management of assessment - Jisc Digital Media 2015