‘At the Coal Face’

Experiences of Computer-based Exams

John Winkley, BTL

Paul Humphries, Edexcel

Dorit Reppert, CCEA

July 8th 2003

Evaluating Computer-based Exams

Key Stakeholders:

• candidates• test centres• examiners• Awarding Bodies

Key Stakeholders:

• candidates• test centres• examiners• Awarding Bodies

Key Success Factors:

• engagement• accessibility• technical usability• system reliability• security• test validity, reliability• data analysis

Key Success Factors:

• engagement• accessibility• technical usability• system reliability• security• test validity, reliability• data analysis

Project Background

PEP 2 Project

Schools:

• 19 schools• GCSE style exams• over 1,000 candidates• approx. 1,400 exams

learndirect & Army Test Centres:

• 13 test centres• real National Tests• over 300 candidates• approx. 300 tests

The ExamBase System

ExamBase Client

ExamBase Server

MicroBoard website

Test Centre

Assessment Producer

exam

exam

question

marked question

data

ExamBase Client

ExamBase Client

marked ‘papers’

Standard PCsNot limit use of PCs at other timesSecure, ReliableEasy-to-use, Engaging, Accessible

Standard PCsNot limit use of PCs at other timesSecure, ReliableEasy-to-use, Engaging, Accessible

On-screen marking

MicroBoard

Test Centre

MarkerBase Server

exam

Examiner

Marking Centre

partially marked ‘papers’

partially marked ‘papers’

fully marked ‘papers’

Examiner

Exam Security

Web services

MicroBoard

Web interface

Web service consumer

ExamBase Server

Test Centre

ExamBase Client

Secure internet (https/ssl) Encrypted file transfer

Secure internet (https/ssl) Encrypted file transfer

Encrypted file transfer (htps/ssl/proprietary)

Encrypted file transfer (htps/ssl/proprietary)

MicroBoard

MicroBoard

MicroBoard

ExamBase Server

(red) (red) Downloaded but lockedDownloaded but lockedThis exam has been downloaded but has not been scheduled to run

(orange) (orange) Downloaded and ScheduledDownloaded and ScheduledThis exam has been scheduled but is not accessible by candidates until the scheduled start time.

(green) (green) Unlocked and ready to runUnlocked and ready to runThe four hour scheduling window has begun and the exam is ready to be accessed by candidates.

(yellow) (yellow) PausedPausedThe exam is currently paused by the administrator.

(blue) (blue) FinishedFinishedThe exam has been scheduled, should have been used and the four hour window for access has now finished or the exam has been closed (see 3.5 Finish exam).

(red) (red) Downloaded but lockedDownloaded but lockedThis exam has been downloaded but has not been scheduled to run

(orange) (orange) Downloaded and ScheduledDownloaded and ScheduledThis exam has been scheduled but is not accessible by candidates until the scheduled start time.

(green) (green) Unlocked and ready to runUnlocked and ready to runThe four hour scheduling window has begun and the exam is ready to be accessed by candidates.

(yellow) (yellow) PausedPausedThe exam is currently paused by the administrator.

(blue) (blue) FinishedFinishedThe exam has been scheduled, should have been used and the four hour window for access has now finished or the exam has been closed (see 3.5 Finish exam).

ExamBase Server

Example On-ScreenExample On-ScreenAssessment QuestionsAssessment Questions

The Test Centre Experience

Priority 1 – a reliable technical infrastructure:

includes trained support staff unreliability may interrupt an exam funding, management processes are key factors

Priority 1 – a reliable technical infrastructure:

includes trained support staff unreliability may interrupt an exam funding, management processes are key factors

Priority 2 – an appropriate exam environment:

setting-up takes longer can be difficult to provide adequate privacy

Priority 2 – an appropriate exam environment:

setting-up takes longer can be difficult to provide adequate privacy

The Test Centre Experience

Schools:

computer-based exams entirely new to schools

transfer of responsibility from administrators to technical staff

48% found software installation easy or very easy

54% had no problems with candidate registration

all schools ran exams successfully

Schools:

computer-based exams entirely new to schools

transfer of responsibility from administrators to technical staff

48% found software installation easy or very easy

54% had no problems with candidate registration

all schools ran exams successfully

The Test Centre Experience

learndirect centres :

technical infrastructure already exists for e-learning and formative assessment

used to maintaining ‘business critical’ IT systems

95% of 29 Skills for Life project centres found the installation easy or very easy

all centres found registration of candidates using MicroBoard easy

learndirect centres :

technical infrastructure already exists for e-learning and formative assessment

used to maintaining ‘business critical’ IT systems

95% of 29 Skills for Life project centres found the installation easy or very easy

all centres found registration of candidates using MicroBoard easy

The Candidate Experience

School Pupils:

‘digital natives’ at ease with IT over 94% use a computer at home at least once a week exams are mandatory

School Pupils:

‘digital natives’ at ease with IT over 94% use a computer at home at least once a week exams are mandatory

Adult Learners:

great disparity of IT experience some with no IT experience have opted for National Tests

Adult Learners:

great disparity of IT experience some with no IT experience have opted for National Tests

The Candidate Experience

School Pupils:

Overwhelmingly positive

92% said they enjoyed the tests

School Pupils:

Overwhelmingly positive

92% said they enjoyed the tests

Adult Learners:

computer-based exams seen as new and positive

“almost unanimously they reported preferring to take an online test rather than a paper test” - Ufi

Adult Learners:

computer-based exams seen as new and positive

“almost unanimously they reported preferring to take an online test rather than a paper test” - Ufi

“It didn’t feel like an exam”

“more enjoyable

than writing”

The Candidate Experience

Common concerns:

technology failures:

• rare but disconcerting• mainly due to PC system problems• candidates can continue on a spare PC without losing work

the exam environment

• keyboard noise in particular

Common concerns:

technology failures:

• rare but disconcerting• mainly due to PC system problems• candidates can continue on a spare PC without losing work

the exam environment

• keyboard noise in particular

MarkerBase

The Examiner Experience

General responses:

9 out of the 10 examiners found the marking either very enjoyable or enjoyable

all found the software either very easy or easy to use

most would prefer to work at home

liked choice of being able to mark ‘by candidate’, ‘by question’ or both

General responses:

9 out of the 10 examiners found the marking either very enjoyable or enjoyable

all found the software either very easy or easy to use

most would prefer to work at home

liked choice of being able to mark ‘by candidate’, ‘by question’ or both

Potential for more rapid marking :

marking speed increased with experience

marking ‘by question’ increases efficiency

Potential for more rapid marking :

marking speed increased with experience

marking ‘by question’ increases efficiency

The Examiner Experience

More accurate tallying:

automatic tallying of marks

error-prone task removed

More accurate tallying:

automatic tallying of marks

error-prone task removed

The Awarding Body Experience

Engaging learners:

closer to day-to-day experiences for many

exam stress reduced

positive impact on self-esteem

more engaging than paper-based exams

Engaging learners:

closer to day-to-day experiences for many

exam stress reduced

positive impact on self-esteem

more engaging than paper-based exams

Accessibility:

Much greater flexibility:• can register candidates up to the exam start• ‘on demand testing’ will be possible

‘on screen’ approach allows tests to be taken:• in mobile situations (e.g. touring bus)• in remote locations (e.g. using laptops)

Accessibility:

Much greater flexibility:• can register candidates up to the exam start• ‘on demand testing’ will be possible

‘on screen’ approach allows tests to be taken:• in mobile situations (e.g. touring bus)• in remote locations (e.g. using laptops)

The Awarding Body Experience

The Awarding Body Experience

Quality assurance:

Less opportunity for human error

Quality checks have to be refocused

Quality assurance:

Less opportunity for human error

Quality checks have to be refocused

Security:

less vulnerable to lost or mishandled papers

encryption technologies increase security

Security:

less vulnerable to lost or mishandled papers

encryption technologies increase security

The Awarding Body Experience

Exam reliability:

100% reliability for ‘closed’ question types

‘open’ question types:• marking remains subjective• MarkerBase helps examiners to be more consistent

Exam reliability:

100% reliability for ‘closed’ question types

‘open’ question types:• marking remains subjective• MarkerBase helps examiners to be more consistent

The Awarding Body Experience

Exam validity:

Good correlation with paper-based National Tests

GCSEs, using wider range of question types, will require additional validation effort

Exam validity:

Good correlation with paper-based National Tests

GCSEs, using wider range of question types, will require additional validation effort

The Awarding Body Experience

Results generation:

possibility of more rapid feedback to candidates

provide results to test centre rather than to candidates

Results generation:

possibility of more rapid feedback to candidates

provide results to test centre rather than to candidates

Data analysis:

richer feedback to candidates and test centres

easier to detect anomalies in candidate responses

Data analysis:

richer feedback to candidates and test centres

easier to detect anomalies in candidate responses

Conclusions

very popular with candidates - more engaging and interactive, but additional validation needed

greater security and reduced risk of human error

improved accessibility and flexibility – highly desirable for Awarding Bodies

quality of technical infrastructure is vital

may initially place a burden on test centres, but with the promise of efficiency gains over time

very popular with candidates - more engaging and interactive, but additional validation needed

greater security and reduced risk of human error

improved accessibility and flexibility – highly desirable for Awarding Bodies

quality of technical infrastructure is vital

may initially place a burden on test centres, but with the promise of efficiency gains over time