Upload
giulio
View
33
Download
0
Embed Size (px)
DESCRIPTION
Making Sense of Data from Complex Assessments. Robert J. Mislevy University of Maryland Linda S. Steinberg & Russell G. Almond Educational Testing Service FERA November 6, 2001. Buzz Hunt, 1986:. How much can testing gain from modern cognitive psychology? - PowerPoint PPT Presentation
Citation preview
FERA 2001 Slide 1November 6, 2001
Making Sense of Data fromComplex Assessments
Robert J. MislevyUniversity of Maryland
Linda S. Steinberg & Russell G. AlmondEducational Testing Service
FERA
November 6, 2001
FERA 2001 Slide 2November 6, 2001
How much can testing gain from modern cognitive psychology?
So long as testing is viewed as something that takes place in a few hours, out of the context of instruction, and for the purpose of predicting a vaguely stated criterion, then the gains to be
made are minimal.
Buzz Hunt, 1986:
FERA 2001 Slide 3November 6, 2001
Opportunities for Impact Informal / local use Conceptual design frameworks
E.g., Grant Wiggins, CRESST Toolkits & building blocks
E.g., Assessment Wizard, IMMEX Building structures into products
E.g., HYDRIVE, Mavis Beacon Building structures into programs
E.g., AP Studio Art, DISC
FERA 2001 Slide 4November 6, 2001
For further information, see...
www.education.umd.edu/EDMS/mislevy/
FERA 2001 Slide 5November 6, 2001
Don Melnick, NBME:
“It is amazing to me how many complex ‘testing’ simulation systems have been developed in the
last decade, each without a scoring system.
“The NBME has consistently found the challenges in the development of innovative testing methods to lie primarily in the scoring arena.”
FERA 2001 Slide 6November 6, 2001
The DISC Project
The Dental Interactive Simulations Corporation (DISC)
The DISC Simulator The DISC Scoring Engine Evidence-Centered Assessment Design The Cognitive Task Analysis (CTA)
FERA 2001 Slide 7November 6, 2001
Evidence-centered assessment design
The three basic models
Evidence Model(s) Task Model(s)
1. xxxxxxxx 2. xxxxxxxx3. xxxxxxxx 4. xxxxxxxx5. xxxxxxxx 6. xxxxxxxx
Student Model Stat model Evidence
rules
FERA 2001 Slide 8November 6, 2001
What complex of knowledge, skills, or other attributes should be assessed?
(Messick, 1992)
Evidence Model(s) Task Model(s)
1. xxxxxxxx 2. xxxxxxxx3. xxxxxxxx 4. xxxxxxxx5. xxxxxxxx 6. xxxxxxxx
Student Model Stat model Evidence
rules
Evidence-centered assessment design
FERA 2001 Slide 9November 6, 2001
What complex of knowledge, skills, or other attributes should be assessed?
(Messick, 1992)
e Model(s) Task Model(s)
1. xxxxxxxx 2. xxxxxxxx3. xxxxxxxx 4. xxxxxxxx5. xxxxxxxx 6. xxxxxxxx
Student Model Stat model Evidence
rules
Student ModelVariables
Evidence-centered assessment design
FERA 2001 Slide 10November 6, 2001
What behaviors or performances should reveal those constructs?
Evidence-centered assessment design
Evidence Model(s) Task Model(s)
1. xxxxxxxx 2. xxxxxxxx3. xxxxxxxx 4. xxxxxxxx5. xxxxxxxx 6. xxxxxxxx
Student Model Stat model Evidence
rules
FERA 2001 Slide 11November 6, 2001
Evidence Model(s) Task Model(s)
1. xxxxxxxx 2. xxxxxxxx3. xxxxxxxx 4. xxxxxxxx5. xxxxxxxx 6. xxxxxxxx
Student Model Stat model Evidence
rules
What behaviors or performances should reveal those constructs?
Work product
Evidence-centered assessment design
FERA 2001 Slide 12November 6, 2001
Evidence Model(s) Task Model(s)
1. xxxxxxxx 2. xxxxxxxx3. xxxxxxxx 4. xxxxxxxx5. xxxxxxxx 6. xxxxxxxx
Student Model Stat model Evidence
rules
What behaviors or performances should reveal those constructs?
Work productObservable variables
Evidence-centered assessment design
FERA 2001 Slide 13November 6, 2001
Evidence Model(s) Task Model(s)
1. xxxxxxxx 2. xxxxxxxx3. xxxxxxxx 4. xxxxxxxx5. xxxxxxxx 6. xxxxxxxx
Student Model Stat model Evidence
rules
What behaviors or performances should reveal those constructs?
Observable variables
Evidence-centered assessment design
FERA 2001 Slide 14November 6, 2001
Evidence Model(s) Task Model(s)
1. xxxxxxxx 2. xxxxxxxx3. xxxxxxxx 4. xxxxxxxx5. xxxxxxxx 6. xxxxxxxx
Student Model Stat model Evidence
rules
What behaviors or performances should reveal those constructs?
Observable variables
Evidence-centered assessment design
Student ModelVariables
FERA 2001 Slide 15November 6, 2001
What tasks or situations should elicit those behaviors?
Evidence-centered assessment design
Evidence Model(s) Task Model(s)
1. xxxxxxxx 2. xxxxxxxx3. xxxxxxxx 4. xxxxxxxx5. xxxxxxxx 6. xxxxxxxx
Student Model Stat model Evidence
rules
FERA 2001 Slide 16November 6, 2001
Evidence Model(s) Task Model(s)
1. xxxxxxxx 2. xxxxxxxx3. xxxxxxxx 4. xxxxxxxx5. xxxxxxxx 6. xxxxxxxx
Student Model Stat model Evidence
rules
What tasks or situations should elicit those behaviors?
Stimulus Specifications
Evidence-centered assessment design
FERA 2001 Slide 17November 6, 2001
Evidence Model(s) Task Model(s)
1. xxxxxxxx 2. xxxxxxxx3. xxxxxxxx 4. xxxxxxxx5. xxxxxxxx 6. xxxxxxxx
Student Model Stat model Evidence
rules
What tasks or situations should elicit those behaviors?
Work Product Specifications
Evidence-centered assessment design
FERA 2001 Slide 18November 6, 2001
Implications for Student Model
SM variables should be consistent with …
The results of the CTA.
The purpose of assessment:
What aspects of skill and knowledge should be used to accumulate evidence across tasks, for pass/fail reporting and finer-grained feedback?
FERA 2001 Slide 19November 6, 2001
Simplified Version of the
DISC Student Model
Communality
Information gathering/Usage
Assessment
Evaluation
Treatment Planning
Medical Knowledge
Ethics/Legal
Student Model 2
9/3/99,rjm
Simplified version of DISCstudent model
FERA 2001 Slide 20November 6, 2001
Implications for Evidence Models
The CTA produced ‘performance features’ that
characterize recurring patterns of behavior and
differentiate levels of expertise.
These features ground generally-defined, re-usable
‘observed variables’ in evidence models.
We defined re-usable evidence models for recurring
scenarios for use with many tasks.
FERA 2001 Slide 21November 6, 2001
Information gathering/Usage
Assessment
Adapting to situational constraints
Addressing the chief complaint
Adequacy of examination procedures
Adequacy of history procedures
Collection of essential information
Context
InfoGathAss simplified
9/3/99,rjm
A simplified version of the EMfor InformationGathering
Procedures in the context ofAssessment
An Evidence Model
FERA 2001 Slide 22November 6, 2001
Evidence Models: Statistical Submodel
What’s constant across cases that use the EM» Student-model parents.
» Identification of observable variables.» Structure of conditional probability relationships between
SM parents and observable children.
What’s tailored to particular cases» Values of conditional probabilities» Specific meaning of observables.
FERA 2001 Slide 23November 6, 2001
Evidence Models: Evaluation Submodel
What’s constant across cases» Identification and formal definition of observable variables.» Generally-stated “proto-rules” for evaluating their values.
What’s tailored to particular cases» Case-specific rules for evaluating values of observables--
Instantiations of proto-rules tailored to the specifics of case.
FERA 2001 Slide 24November 6, 2001
“Docking” an Evidence Model
Evidence ModelStudent Model
Communality
Information gathering/Usage
Assessment
Evaluation
Treatment Planning
Medical Knowledge
Ethics/Legal
Information gathering/Usage
Assessment
Adapting to situational constraints
Addressing the chief complaint
Adequacy of examination procedures
Adequacy of history procedures
Collection of essential information
Context
FERA 2001 Slide 25November 6, 2001
“Docking” an Evidence Model
Evidence ModelStudent Model
Communality
Information gathering/Usage
Assessment
Evaluation
Treatment Planning
Medical Knowledge
Ethics/Legal
Information gathering/Usage
Assessment
Adapting to situational constraints
Addressing the chief complaint
Adequacy of examination procedures
Adequacy of history procedures
Collection of essential information
Context
FERA 2001 Slide 26November 6, 2001
Initial Status
Expert .28Competent .43Novice .28
All .33Some .33None .33
FERA 2001 Slide 27November 6, 2001
Expert .39Competent .51Novice .11
All 1.00Some .00None .00
Status after four ‘good’ findings
FERA 2001 Slide 28November 6, 2001
Expert .15Competent .54Novice .30
All .00Some .00None 1.00
Status after one ‘good’ and three ‘bad’ findings
FERA 2001 Slide 29November 6, 2001
“Docking” another Evidence Model
Evidence ModelStudent Model
T rea tm en t P lan n in g
M ed ica l K n o w led g e
C o n tex t
A d eq u acy o f trea tm en t p ro ced u res
In d iv id u a liz a tio n o f p ro ced u res
E ffec t o f trea tm en t o n p a tien t
P e rfo rm an ce o f ex tran eo u s trea tm en t
Communality
Information gathering/Usage
Assessment
Evaluation
Treatment Planning
Medical Knowledge
Ethics/Legal
FERA 2001 Slide 30November 6, 2001
“Docking” another Evidence Model
Evidence ModelStudent Model
T rea tm en t P lan n in g
M ed ica l K n o w led g e
C o n tex t
A d eq u acy o f trea tm en t p ro ced u res
In d iv id u a liz a tio n o f p ro ced u res
E ffec t o f trea tm en t o n p a tien t
P e rfo rm an ce o f ex tran eo u s trea tm en t
Communality
Information gathering/Usage
Assessment
Evaluation
Treatment Planning
Medical Knowledge
Ethics/Legal
FERA 2001 Slide 31November 6, 2001
Implications for Task Models
Task models are schemas for phases of cases,
constructed around key features that ... the simulator needs for its virtual-patient data base,
characterize features we need to evoke specified aspects of skill/knowledge,
characterize features of tasks that affect difficulty,
characterize features we need to assemble tasks into tests.
FERA 2001 Slide 32November 6, 2001
Implications for Simulator
Once we’ve determined the kind of evidence we need as evidence about targeted knowledge, how must we construct the simulator to provide the data we need?
Nature of problems» Distinguish phases in the patient interaction cycle.» Use typical forms of information & control availability.
» Dynamic patient condition & cross time cases. Nature of affordances
» Examinees must be able to seek and gather data,» indicate hypotheses,» justify hypotheses with respect to cues, » justify actions with respect to hypotheses.
FERA 2001 Slide 33November 6, 2001
Payoff
Re-usable student-model
» Can project to overall score for licensing
» Supports mid-level feedback as well
Re-usable evidence and task models» Can write indefinitely many unique cases using schemas
» Framework for writing case-specific evaluation rules
Machinery can generalize to other uses & domains
FERA 2001 Slide 34November 6, 2001
Two ways to “score” complex assessments
THE HARD WAY:
Ask ‘how do you score it?’ after you’ve built the assessment and scripted the tasks or scenarios.
A DIFFERENT HARD, BUT MORE LIKELY TO WORK, WAY:
Design the assessment and the tasks/scenarios around what you want to make inferences about, what you need to see to ground them, and the structure of the interrelationships.
Part 2 Conclusion
FERA 2001 Slide 35November 6, 2001
We can attack new assessment challenges by working from generative principles:Principles from measurement and evidentiary
reasoning, coordinated with... inferences framed in terms of current and
continually evolving psychology, using current and continually evolving technologies
to help gather and evaluate data in that light, in a coherent assessment design framework.
Grand Conclusion
FERA 2001 Slide 36November 6, 2001