ICT in Assessment and Learning

SCROLLA Symposium 6 Feb 2002 1

ICT in Assessment and Learning

Developments from the Enigma Project

Robert HardingNick Raikes

ITAL UnitInteractive Technologies in Assessment and Learning


Introduction

ICT-led innovation:Exciting image……but assessment is Cinderella!

Holistic nature of ‘The Learning System’ :Assessment is a goal for teachers and learners……but valid assessment must be rooted in learningHow will new methods fit with existing practice?The “Three-legged race”.


Outline The Enigma Project trials

Scope User interface

Evaluation: ‘Traditional’ style questions (conceptual) ‘New interactive’ style questions (analytical) Administrative issues

Developments triggered as a result Brief resumé of some resulting work Closer look at one aspect:

on-screen marking Conclusion.


The Enigma Project- Scope Two years, 3+2 schools, about 170+120 pupils Pilot 1: O-Level Physics examination in paper and CBT forms

A few graphics manipulation items Pilot 2: Selection of O-Level Science in CBT form

Conceptual ‘objective’ items Analytical simulation+free text answer items

Marking: Objective/multiple choice: automatic Open answers: printed out, marked by hand

Comparative evaluation of pupils performance.


User Interface

Navigation

Time count-down

Progress indicator

Candidate bookmark

Audio feature for questions.


User Interface

Navigation

Time count-down

Progress indicator

Candidate bookmark

Audio feature for questions.


Evaluation - ‘traditional’

Pilot 1: the evaluators concluded: For MCQ’s, no obvious differences between groups (3%) For open-ended, those on paper theory Q’s better than CBT

(11%)

Pilot 2: poor correlation between pilot and real examinations: Was computer aptitude responsible? Was it mismatch of trial vs real examination?.


Evaluation - ‘analytical’ Objective - how do students react to ICT oriented

question styles? Use of simulations 6 Q’s: 2 each Physics, Chemistry, Biology Panel of examiners Typical example follows …note:

access to simulation, free text response boxes ability to record student actions.


Evaluation - ‘analytical’ Student responses:

Only 1 in 6 failed to understand what was required of them… …but 40% said ‘occasions when unsure what to do’.

(but how true of most of us in many situations!)

Poor correlations between this test and ‘real’ examination

BUT … 70% said: “computer based tests are as fair as conventional tests” Computer literacy cited as most common reason for bias.


Evaluation - overall 40% students said the computer slowed them down:

Interpretation: they are unfamiliar with computers in schoolwork Conclude that learning and assessment must be integrated

40% said computer stopped them answering as wanted Most common complaint - not enough space to write Did not like not being allowed to go back and change answer

70% thought CBT’s to be as fair as paper-based Reasons why not so fair: typing speed and literacy (lack of).


Administrative issues Schools’ normal IT systems can interact with

assessment software Individual system crashes - incremental backup

essential High level of expert attention was needed Is printing needed? Security - visibility of screens At least one candidate observed using email! Issues of accessibility to other software and data Interference with normal working must be minimised.


What developed? - Analysis We have not seen a surge of CBT use in schools:

Material circumstances and resources Holistic nature of changes needed

Cast of influences

Can you ‘box off’ summative assessment?

Traditional links between assessment and teaching

Examiner-teachers set questions, not Boards

Feedback loops: Learning process geared to passing examinations Examinations are rooted in ‘educational ambience’.


What we did - ‘ambience’ Teacher support for using ICT - TEEM

http://www.teem.org.uk/

Syllabus support e-Lists and electronic communities Electronic learning resources:

e.g. Enigma simulations on the www http://www.cie.org.uk/learning/science_practicals/ .


What we did - assessment

CALM / CUE / EQL collaborationMEI A-Level Maths examinationsIMS standards and ‘QTI’ workgroup

Question test interoperability

Standards for Examination conductOn-screen marking.


On-screen marking

Scan scripts, or capture electronic scriptsWhat do we want it for?

Faster, more flexible marking, management More efficient quality control Better feedback to Centres Transition to on-line assessment.


Study - ‘proof of concept’Actual, live scripts::

O-Level: Maths (350) A-Level: Geography (900), Eng Lit (300)

5 examiners per subjectConventional and screen markedWhole script, marking by questionDownload scripts via Internet.


Features of the software

Ticks and crossesAnchored commentsDisplay control - e.g. zoom, script scrollView marking guideNavigation between question or scriptProgress monitor and referal.


Examiners’ impressions

Generally managed the downloadingScripts at least as legible as on paperMost felt they gave same marks:

Exceptions in English Lit & Geography ?’s on trial nature, annotation, whole script Maths points re marking by question

All would mark on-screen again.


Analysis of marksMaths component - consistencyGeography - mostly satisfactory:

one examiner more severe on screen one consistent on paper but not on screen

English: 2 more severe on screen All more consistent on paper than screen Possible link with ‘whole script’ judgement?.


Conclusions

Holistic nature of system ‘Three-legged race’ - or more?

Central role of teachersChallenge:

Integrate, make the technology invisible

Way forward - ‘open source’?.


Some URL’s and email ITAL Unit

http://ital.ucles-red.cam.ac.uk/ Teacher support for using ICT - TEEM

http://www.teem.org.uk/ Enigma simulations on the www

http://www.cie.org.uk/learning/science_practicals/

Robert Harding <[email protected]> Nick Raikes <[email protected]> .

Documents

ICT in Assessment and Learning