1

Test Evaluation

Computerised Tests Overview

June 2010

'Developments in Automated Testing'

Garry CleereQ-Validus

Dublin

Overview

1. Background

2. Plan

3. Develop

4. Item Formats (samples)

5. Administer

6. Score & Report

7. Analyze

8. Communicate

9. Security

10. Accessibility

11. Final Comments

2

Background

1. Background

Significant Evolution in Automated Test Evaluation Systems (ATES)

Changes overtime: platforms / construct / content / process enhancement

Impact of a range of new technologies on ATES

Implications of greater bandwidth for delivery and format

Configuration options

Impact of greater processing power

Impacts of Legislation, Laws and Guidelines

Increasing security concerns

3

Plan

2. Plan

‘From the ground up’ Validity and quality Expert Group (SME's) Formal Syllabus Definition Test Blueprinting ‘Soft’ Characterisation? CTT (Characterisation Test Template) Linear / sequential Item banking Accessibility considerations ‘Universal Design’

4

Develop

3. Test Development

New formality ; process: Item writers / scripting process Building to a formal test blueprint Item writing workshops Item Development using studio tools Graphics and designers Readability considerations Accessibility considerations Test Banking Posting to the ATS engine Pilot testing, results analysis Test form release Ongoing validation / maintenance

5

Item Formats

4. Test Item Formats

From the ‘humble MCQ’ … Through to the Image supported MCQ Further range of engaging theoretical formats:

- Drag n’ Drop; FITB; Matching; Re-ordering

Hotspot item formats Simulations type items Light simulation item formats Rich simulation item formats In-applications testing (providing a ‘real’ or authentic test experience) High Face Validity “Form and Function” and item format congruency with Syllabus

6

Sample Item Formats

7

Test Administration

5. Tests Administration

Candidate management Self registration Candidate group import facilities Global programmes administration features Different territories, countries, languages Clocks, alarms, reminders … Much enhanced breadth of reporting New registrations management features Automatic LMS imports Integration with LMS

8

Score and Report

6. Scoring and Reporting

Formal Test Blueprint Pass / Fail reporting Percentage reporting Summative v. formative assessment Diagnostic results Final state comparison Spelling accommodations Primer tests / mock tests Range of reports Depth / detail in tests reports Psychometric analysis

9

Psychometric Analysis

7. Psychometric Analysis

'P' Value Discrimination value Point Bi-serial indicators Measuring the item’s performance Granular item analysis looking to the:

‘anticipated outcomes for borderline or just capable students’ Cheating and analysis outputs Test validity Comparable / parallel test forms Weakly versus strongly parallel test forms

10

Communications

8. Communications

Sensitivity / responsibility / validity? Information about the test Impact of collateral materials Branding and programme message Value of Support materials Preparation for the tests Value of Tests primers, mock tests Purpose of test instruments Centre communications Holistic experience Centre Support Valuable resources

11

Security

9. Security

Increasing threats Tests Stakes involved High / moderate / low stakes Appropriateness (Risk / Reputation) ‘50 ways to cheat’ - firm in the US who specialize Tests for sale on the Internet Learning from test analysis Invigilation procedures Login validation Biometrics Cameras Range of security measures

12

Accessibility

10. Accessibility

The BCS (British Computer Society) consider the following categories for ATES accessibilty: Sensory (hearing and visual) ;

Cognitive (dyslexia and other specific learning difficulties);

Physical (mobility);

Behavioral (emotional or social).

New Drivers: Assistive technologies; Plain language ; Visual ; Hearing impaired.

13

Final Comments

1. Final Comments

Bandwidth and delivery

In-applications

Imaginative item formats

Multi-lingual

Global Administration

Feature rich systems

Exciting Item formats

Process bound development

Continuing evolution

14

15

Garry CleereManaging DirectorQ-ValidusDublinwww.q-validus.comgarry.cleere@q-validus.comTel. 00 353 1 716 3740