Download ppt - Redesigning the Computer-assisted Language Exams for Federal Government Employees

Bert WylineAssessment Scotland 2010

Redesigning the computer assisted language examens for federal government employees

Redesigning the computer assisted language exams for federal government employees: a didactic, methodological and technological challenge







0. Context

1.Didactic challenge

2.Methodological challenge

3.Technological challenge

4.Results & future developments



0. Context

ı SELOR: Belgian governmental selection bureauin a trilingual country, multilingualism is crucial for state servants-> language testing is crucial part of assessment & selection procedures

ı Since the 90s:‖ ATLAS =

electronic language testing system for Dutch and French.‖ Yearly thousands of candidates‖ Wide variety of governmental jobs (policeman as well as diplomats)



ATLAS = state-of-the art at its creation,but needed a complete overhaul in three domains

1. didactical component:ı strongly focused on language knowledgeı weak integration of skills-based view on language competence

(Common European Framework of Reference didn’t exist at that time)

2. Methodological componentı Level structure without psychometric underpinning:

4 levelsı No evaluation of the reliability and validity of the ATLAS tests



3. Technological component

ı ATLAS operated on the SELOR network, not onlineı Closed, non-adaptable and non-updatable systemı Off-line accompanying training module on cd-romı No itembanking ı No integration into Selor admin



ı Constraints:‖ Legal constraints:

e.g. vocabulary & grammar should be tested separatelye.g. 4 levels (1 to 4) should be distinguished

‖ Practical constraints:

e.g. each examen takes maximum 120 minutese.g. SELOR wanted us to reuse the existing items as much as possiblee.g. whole operation had to be realised within one year



ı Research team: partners

Didactic componentFrench: Piet Desmet (K.U.Leuven Campus Kortrijk)Dutch: Guy Deville (FUNDP Namur)

Methodological componentSara Gysen (K.U.Leuven)

Technological componentBert Wylin (Televic Education)

CoordinationPiet Desmet & Sara Gysen



0. Context







I. Didactic challenge

ı I.I. Construct definition

ı I.2. Item revision and new item writing

ı 1.3. Metadata



1.1. Construct definition

ı From 9 modules to 4 components:‖ 2 knowledge-oriented: vocabulary & grammar

2 skills-oriented: listening & reading



I.2. Item Revision & New item writingı Revision of existing items:

‖ Uniformity(e.g. same type MCQ; only one gap in all cloze exercises)

‖ Transparancy for test candidates(e.g. dichotomous rating for all items)

ı New item writing‖ As authentic as possible: real audiofragments, scanned

articles, letters etc‖ A spectrum of different item types, not only multiple

choice, in order to test as direct as possible the different tasks specified in the construct



ı New items writing

‖ New items were developed for the new categories within the listening and reading component

‖ As authentic as possible: real audiofragments, scanned articles, letters etc: same look and feel, same distribution of images as in real-life tasks

‖ A spectrum of different item types, not only multiple choice, in order to test as direct as possible the different tasks specified in the construct

‖ Standard choice of technical item type for each part of the construct



1.3. Metadata

ı Item tags and features of 3 types:‖ Content metadata (automatic and manual)‖ Psychometric metadata (cf. 2)‖ Dynamic metadata (evolving through use of system)

ı Important for itembanking:‖ Control of item selection in examen versions‖ Monitoring of item quality

(cf. psychometric data and dynamic metadata)



ı Metadata for each item of the database:‖ Content metadata

ш Identification numberш Question formatш Excluded when other item presentш Linked to other itemш Date of creationш Date of adaptationш Adapted for candidates with special needsш Ratingш In training environmentш Inactiveш Assets (multimedia)ш Length audio/videoш Length text ш Example item

‖ Dynamic metadataш Popularity of itemш Average answer time

‖ Psychometric metadataш Logit-valueш P-value



0. Context







2. Methodological challenge

ı 2.I. Screening and calibration of existing database

ı 2.2. Development of an IRT-based item database

ı 2.3. Standard setting & selection rules



2.1. Screening of the existing database

ı Screening based on test data from 1995-2000

‖ Elimination of items based onш their p-value (percentage correct answers provided by the test candidates)

lower than 0.10 (extremely difficult)higher than 0.95 (extremely easy)

ш and their occurence in test versions:at least 100 times in test version

ш Results: 218 French items and 849 Dutch items were eliminated



2.2. Development of an IRT-based database

ı Submitting the items to a psychometric analysis based onItem Respons Theory (IRT) which allows to place items on a scale

that orders items in function of their intrinsic difficulty level (logit value)

&that orders examinees in terms of their ability



ı Example of a measurement scale in IRT model

ı Probabilistic model :Person B has great potential to answer items c & b correctly but far less to answer items d or f correctly, which will normally be solved correctly by person A.The chance that person B will be able to answer item e is almost non existing

Meetschaal

Kandidaten

vaardigheid

Items

moeilijkheidsgraad

Item g

Item e

Item a

Item b

Item c

Item d

Item f

Item h

Item i

Item j

Beheersing in termen van kansen

Meetschaal

Kandidaten

vaardigheid

Items

moeilijkheidsgraad

Item g

Item e

Item a

Item b

Item c

Item d

Item f

Item h

Item i

Item j

Item g

Item e

Item a

Item b

Item c

Item d

Item f

Item h

Item i

Item j

Item e

Item a

Item b

Item c

Item d

Item f

Item h

Item i

Item j

Item a

Item b

Item c

Item d

Item f

Item h

Item i

Item j

Beheersing in termen van kansen

A

B



ı Eight different scales:one per target language and per component

e.g. Logit distribution of candidates and items (French) for component 4 - Reading

Metadataset your own metadata

Metadataset your own metadata

Use system metadata

Use system metadata



1. Test management by candidate him/herself:- Candidate decides when to start up the next component

(but fixed time limit of 120min for the whole exam) - Possibility to review within component- Overview screen and ‘brainteaser’ tag- No restriction on playing audio and video input,

but limited time allocation is mentioned in instruction- Possibility of not answering an itemBut restrictions:- Time limit: fixed per component (time interval on screen)- Fixed order of components: C1 C2 C3 C4

2. “Resuming” possible in case of problems

3. Equal share of components in overall score

2.3. Standard setting and selection rules



Exam version1 General introduction to test No time

allocation

2 Vocabulary component

Instruction + 1 example item

20 test items 15 min

3 Grammar component

Instruction + 5 example items


4 Listening component Instruction + 2 example items


5 Reading component Instruction + 3 example items




0. Context







3. Technological challenge

ı 3.I. E-testing: Edumatic technology

ı 3.2. Exam & preparatory learning environment

ı 3.3. Itembanking & Selor Test Administration System



3.1. E-testing:Edumatic based environment

ı Edumatic is an authoring system for exercises and testsfor both online and offline assessments(online server based,with export button to offline SCORM-compliant)

ı xml-based data in a flash user interface(ims-qti and scorm compliant)

ı 20+ question typessupports multimedia in all question types

Visit our booth!



3.2. Exam & preparatory learning environmentı Customization of Edumatic environment

‖ Selor-skin

‖ Single Sign-On with Selor Admin(SSO: login once, get access to multiple applications)









ı Exam configuration‖ Secure browser

Full classroom control system (AvidaNet Exam)

‖ Resume

‖ Strict time allocation

‖ Sequencing



ı Online learning environment:available at http://www.selor.be

‖ Preparatory learning environment

‖ Login for free via “My Selor”

‖ Components 1 & 2 (vocabulary & grammar) : access to the entire database+ 35.000 items

‖ Components 3 & 4 (listening & reading):only model items













• Stats: user stats



• Stats: user stats

• Stats: package stats



0. Context







4. Results & further developmentsı Fully operational

‖ Exam version: in use since October 2007many thousands of candidates every year

‖ Online learning environment: online since September 2007ı Edumatic now official electronic test environment

‖ For all Selor domains (including social skills, law, informatics, mathematics, accountancy)

ı Development of decentralized exam facilities ı Added question types

‖ Including voice recording (with scoring by “human”)‖ Including open questions (with scoring by “human”)‖ Including open questions with automatic feedback

(not used in real exams)



Please visit

[email protected]

[email protected]

http://www.edumaticonline.com

Edumatic Exam

Edumatic Mobile