- 24

Extending Computerized Adaptive Testing to Multiple Objectives:Envisioned on a Case from the

Health Care.Christian Weber, Réka Vas

Egovis 2014

- 24

Background: The eduworks network

2023.04.12. Corvinno Technology Transfer Center 2

University of Salamanca

- 24

For a better educational testing

2023.04.12. Corvinno Technology Transfer Center 3

Cognitive Type

Know-ledge

CultureMatching

Labour

Education

Human

profile

Capturing the user

- 24

The health perspective

2023.04.12. Corvinno Technology Transfer Center 4

- 24

Need for ongoing training in the health care

• Immense pressure on doctors within their daily practice and as well in terms of an (expected) ongoing trainingE.g. obligation for training in Germany since 2005

• More and more tasks are transferred to nurses or other caretakerE.g. they have a major role in Alzheimer early detection

(World Alzheimer Report 2011)

• But .. a focus for ongoing training for nurses is missing need for support in educationBut the need comes with special demands..

2023.04.12. Corvinno Technology Transfer Center 5

- 24

The demands of medical training

• Time is restrictedFor education (but also for the support of domain experts)

• The learning has an educational context• Must-know knowledge

regulations & mandatory knowledge

• Should-know knowledgenew insights & specializations

• The learning has a cultural contextCare taker and nurses have different cultural backgrounds with

different perception to learning. patients

2023.04.12. Corvinno Technology Transfer Center 6

- 24

Designing a new test

2023.04.12. Corvinno Technology Transfer Center 7

Learning goal

Know-ledge

Restricted time

CultureInfluenceNurses

Labour

Education

Human

profile

Scaling education to the user

Time to learn

Time to test

- 24

Computerized Adaptive Testing - CAT

2023.04.12. Corvinno Technology Transfer Center 8

pass

pass

fail

passpass

passfail

fail

pass

Te s t I t e m s

Ite

m D

iffi

cu

lty A b i l i t y L e v e l

100

- 242023.04.12. Corvinno Technology Transfer Center 9

Item response function (3PL)as the probability that a person with ability “theta”

answers an item

• b – difficulty, (half way between c(min) and 1)• a – discrimination, (slope at b, differentiating the users)• c – chance for guessing

(building on a logistic function)

- 24

Limitations of question organization

2023.04.12. Corvinno Technology Transfer Center 10

Cognitive Type

Know-ledge

Culture

Q40

Q60

Q60

Q60

Q70

Q60

Q40

Q65

Q70

Q50

- 24

Add Meta Information• Cultural context• Learning type• Obligation yes/no•...

Cognitive Type

Know-ledge

Culture

Q40

Q60

Q60

Q60

Q70

Q60

Q40

Q65

Q70

Q50

Limitations of question organization

2023.04.12. Corvinno Technology Transfer Center 11

- 24

Add Meta Information• Cultural context• Learning type• Obligation yes/no•...

Cognitive Type

Know-ledge

Culture

Q40

Q60

Q60

Q60

Q70

Q60

Q40

Q65

Q70

Q50

Exploring the user on the education

User domain

Educational domain

Limitations of question organization

2023.04.12. Corvinno Technology Transfer Center 12

- 24

Overcoming limitations

• How to grasp the user knowledge in the best possible way?Framing question with no final answer.

• How to add more context to it?Extending the structure and indicators.

• How to detect also his/her understanding of an area?Using knowledge, structured and in relation.

2023.04.12. Corvinno Technology Transfer Center 13

- 24

The demands of medical training - revisited

• Time is restrictedFor education (but also for the support of domain experts)

• The learning has an educational context• Must-know knowledge as meta data

regulations & mandatory knowledge

• Should-know knowledge as meta datanew insights & specializations

• The learning has a cultural contextCare taker and nurses have different cultural backgrounds with

different perception to learning. through modelling

2023.04.12. Corvinno Technology Transfer Center 14

- 24

Differential Item Functioning - DIF

• Occurs when people of different groups have a different probability to answer correct, independent of the same underlying ability.Groups in terms of culture, educational character or learning

style

• Detection normally through statistical analysis of the performance on the test.The designer view: Detection only if a

grouping context is known..

2023.04.12. Corvinno Technology Transfer Center 15

- 24

DIF context through virtualization

• Instead of removing DIF items, redundant item modelling(Makransky and Glas)

• DIF biasingHarder or easier

(difficult level)

• Alternatively:Redundant question

and real structure

But how could a structure look like?..2023.04.12. Corvinno Technology Transfer Center 16

Item Context

12

n

12

nVirtualized Item

- 24

Structure - A new concept of layers

2023.04.12. Corvinno Technology Transfer Center 17

Q11Q12

Q13 Q21Q22

Q31Q32

Q33

Qxy+1

Qxy

Q41Q42

1

2

3

4

y

“meta-parameter”

question ite

ms

sub-ontology grouping

ontology

e.g. item difficulty, cognitive type..

complexity / relations

educational domain

difficulty new ?= f(diff Q xy , compl N x)

++ meta-information

- 24

Event

the system behavior as a stream of data

Agent

Agent

Consumer

Consumer

EventConsumer

PatternDetection

Trans-for-mation

Filter

Agent

Data

Data

Eventproducer

Testing strategy: Thinking about a new frame with (Complex) Event Processing (CEP)

2023.04.12. Corvinno Technology Transfer Center 18

- 24

DomainOntology

A new frame for CAT with context

2023.04.12. Corvinno Technology Transfer Center 19

Meta-

Strategies

TestingStrategies

Data

the system behavior as a stream of data + access to the ontologyData

Eventproducer

IntelligentTest Engine

Event

Q11

Q12

Q13

Q21

Q22

Q31

Q32

Q33

Qxy+1

Qxy

Q41

Q42

1

2

3

4

y

“addon-parameter”

concept group level

ontology

question level

ContextAgent

ContextAgent

ContextAgent

change/modify

e.g right/wrong~perform welltime for answer

DataEvent

e.g learning typecompatible to job xfast tester

EventConsumer

- 24

Exploration, yes - but where (how) to go..

2023.04.12. Corvinno Technology Transfer Center 20

Q QQ

Q

?

?

?

Meta-Strategies

TestingStrategies

IntelligentTest Engine

- 24

Next level questions..

• How are the rules to switch and modify the testing strategy and how to manage it on the fly?..depth/breadth first, related concepts, maintain main test

• What is the best choice to model a context parameters for the strategic exploration?

• What is the best fusion of parameters?.. psychometrics and graph/ontology based complexity

measures

2023.04.12. Corvinno Technology Transfer Center 21

- 24

Data Sources

2023.04.12. Corvinno Technology Transfer Center 22

• Now:• Educational knowledge within the domain ontology as classes

and relations• Logging feedback from the actual testing

• Future:• Target-oriented performance feedback from the user To:

• grouping of users after special criteria• Feedback on the content – e.g. Improving meta-data on the nodes

- 24

Summary

• Need for a crisp testing to detect training potential in knowledge intense professions.Especially in the health care sector

• CAT is a suitable solution for educational testing....but there is room for improvement for a context aware testing

• A new frame and thinking could enable a next generation of context sensitive tests, with..• structured backing knowledge (ontology)• a novel testing behaviour (modifying test behaviour online)

2023.04.12. Corvinno Technology Transfer Center 23

- 24

Thank you for your attention!

Corvinno Technology Transfer CenterH-1093 Budapest, Közraktár utca 12/a.

Tel: +36 1 210 80 62 http://www.corvinno.com