DMW 2006 The Open University's Institute for Educational Technology Forward Planning: Findings from a Roadmap for e-Assessment Denise Whitelock Institute

DMW 2006 The Open University's Institute for Educational Technology

Forward Planning: Findings from a Roadmap for e-Assessment

Denise Whitelock

Institute of Educational Technology

The Open University,e-mail: [email protected]


Background

• e-Assessment is of strategic importance to the UK

• Major global growth industry

• Recognised by JISC’s e-Learning Programme

• Funded by JISC to produce a Roadmap


What is a roadmap?

• Obtain a consensus view or vision of the future landscape of e-assessment to decision makers

• Kostoff & Schaller (2001)

• roadmapping provides a way to identify and evaluate and select strategic alternatives for the desired objective


Road mapping

• Technology push

• Requirements pull

• Layers

• Know why – envisioning activities

• Know what – policy and theoretical developments

• Know how - technological

• Time Dimension

• Adapted from Phaal, Farrukh et al, 2004


Adapting the Delphi Method

• Delphi questionnaire often used http://www.is.njit.edu/pubs/delphibook/index.html#toc

• Consensus

• Reduced to lowest common denominator

• No priorities emerge

• This consultation will identify core and outlying issues – avoid polarities


Application of the Delphi Method

Select panel members based on the expertise required

Develop Questionnaire

Distribute questionnaire

Analyse responses

Determine whether a consensus has been reached

If no consensus seek further information


Building the e-Assessment Roadmap

• Map current terrain

• Review current policy and initiatives

• Identify current practice

• Recognise practitioners’ concerns

• Check whether rhetoric matches reality

• Gauge stakeholders’ visions

• Recommendations for keeping on track


Roadmap framework


Sourcing the information for Stage 1

• Superinstitutional

• Policymakers : HEFCE, SQA etc.

• Institutional

• HE, FE etc.

• Personal

• Tutors, students, developers, commercial companies


Superinstitutional: Policies and Practices (1)

• All student centred

• Personalised learning/e-portfolios

• Instant feedback

• Learner control

• More choice


Superinstitutional: Policies and Practices (2)

• e-Assessment will be rolled out in post-16 education by 2009

• e-Assessment will make a significant contribution to reducing the assessment burden and improving the quality of assessment

• e-Assessment field trials should be taking place in at least two subjects per awarding body during 2005

• 75% of key and basic skills tests will be delivered on screen by 2005

• N.B. 2009 deadline made explicit by QCA but not SQA


Institutional Drivers

• Increase in student retention

• Enhanced quality of feedback

• Flexibility for distance learning

• Coping with large student numbers

• Objectivity in marks

• More effective use of VLE


Institutional Vision

• More learner control

• Feedback

• Flexibility

• Embedding the e-learning within the institutional tools


Consulting Experts with a Survey

• Survey designed to probe the following:

• Timings of policy implementation

• How e-Assessment can cut the burden of assessment

• Ways in which the quality of e-Assessment will be improved

• Implications of the vision of the policy documents

• Visions for the future

• Barriers to the visions


Experts

• Superinstitutional

• Representatives from SQA, HEFCE, BECTA etc.

• Institutional

• Directors of e-Learning Units and policy makers in HE and FE

• Personal

• Champions of e-Assessment


Survey Findings (1)

• Predicted timings:

• Agree with 2010 deadline

• ICT accepted in all aspects of student experience within 2/4 years

• Tutors will have tools to assist with course design and providing electronic feedback to students within 2/4 years


Survey Findings (2)

• Cutting the burden

• Technological change means revamp of old practice

• Questions need to be more interactive

• Efficiencies is in results processing, transparency in grading, develop criterion referenced exams

• Introduce formative assessment

• Faster feedback

• Better plagiarism detection

• Production of essay marking tools

• Accessibility issues require higher profile


Survey Findings (3)

• Improving the quality of e-Assessment

• Offer more realistic assessments

• Cover wide range of curriculum

• Forces assessors to produce clearly defined objectives that enhance student learning

• Better regular feedback will assist students who usually under-perform

• More improvement seen in vocational sphere


Survey Findings (4)

• Implications for the vision set by the policy documents

• e-Assessment sites will be open 24/7

• More students will study University courses while at school

• Parents will over-pressurise students to take exams too early

• Over-use of on-demand testing does not always increase grades and produces lack of confidence in standards

• e-Assessment to play key role in personalised learning agenda


Policy implications

• 50% agreed students studying at own pace will result in higher grades and a lack of confidence in exam style

• Agreed resits should be regulated

• Agreed more breadth and depth of learning will be achieved through e-assessment

• 16 – 18 year olds will be studying some university courses while at school


Paperless examinations as recommended by CCEA and Edexcel

• Authors need training

• Authors do not need same technical skills as technology partner

• Research into different question types sorely needed

• Good piloting essential


Further Expert Opinions

• Adaptation of existing standards regarding security (BS7799), quality (ISO9000), test delivery (BS7988) and item production (QTI). Commercial developers will need to subscribe to these standards

• Strong encouragement of open-source software licensing models

• An agreed level of interoperability based on national standards.


Visions: Superorganisational

• Large-scale testing sites

• Test sites will deliver some paper and pencil components

• Prevalent from primary through to university and vocational qualifications

• High quality question banks

• No longer referred to as e-assessment


Visions: Organisational

• e-Portfolios increase

• Formative, self assessment and e-portfolios main tools

• Random internal testing

• Confidence declared in systems by teacher, awarding bodies, students and general public

• More summative assessment in HE

• Flexible entry and examination


Visions: Personal

• e-Portfolios exchanged maybe as microchips in business cards

• All e-assessment except for practicals

• e-Assessment seamlessly integrated into day to day practice

• Peer e-assessment

• Assessment for Learning

• More use of diagnostics


Barriers: Superorganisational

• Customer attitudes

• Lack of public confidence

• Negative stories in the media

• Lack of integration of policies

• Key antenna of quality, accessibility, reliability and security are evaluated with pilot activities


Barriers: Organisational

• Lack funding to encourage pilot and R & D

• Lack of recognition in RAE

• Not engaging with in proven technology

• Lack infrastructure to pilot

• Lack of item banks

• VLE can push out champions


Barriers: Personal

• Staff attitude

• Availability of resources

• Lack of IT skills in a pilot project’s target student population


Barriers to adoption

• Reformation of methods

• Training and development

• Resources

• Technical

• Customer attitude

• Staff attitude

• Learner attitude

• VLE


Recommendations: Superorganisational

• Clear policy directives

• Government agencies to support directions

• Standard system not acceptable to awarding bodies

• Standards for systems spelt out by QCA

• Initiatives to share expertise

• Recognition of training

• Encourage open source licensing models

• Co-ordinate dissemination at BECTA/JISC levels


Recommendations: Organisational

• Tool development

• Examine role of VLEs on progress

• Follow SQA’s lead COLA and SOLAR

• Find self and peer assessments

• Item banks

• Plagiarism

• Research into automatic text recognition

• Guidelines for best practice

• Accessibility


Recommendations: Personal

• Training

• Recognition of effort/time

• Assist champions

• Fund studies of changes to job allocation and staff culture

• Appoint mentors

• Free text entry needed for student assessments


Research and Development

• VLE

• Evidence needed:

• Learning objectives match e-assessments

• e-Assessment promoting learning control and reflective learning

• Pilot on demand testing

• Gauging emotional pressures as well as cognitive gains


Priorities

• Sort out authentication and plagiarism issues

• Train staff to devise e-assessments

• Build good test item banks following the lead of SQA

• Pass on good practice tips from current cohort of students to new learners using e-assessment for the first time

• Make more testing sites for summative assessment available and pass on good practice

• Support more research into automatic marking of e-assessment

• Address accessibility issues directly through superorganisational directives

• Public acceptance of high stake assessments needs to be achieved through dissemination of key findings using press releases, TV and press editorials and PR techniques



Summary

• Policy pressure main driver in England and Wales• Affects FE more than HE• More reliance than maybe anticipated on the

commercial sector • More use of formative e-Assessment• More research funding into text recognition becoming

acknowledged• Andragogy rather than pedagogy?

Roadmap final report http://www.jisc.ac.uk/elp_assessment.html

Documents

DMW 2006 The Open University's Institute for Educational Technology Forward Planning: Findings from a Roadmap for e-Assessment Denise Whitelock Institute