Upload
debra-mcintosh
View
26
Download
1
Embed Size (px)
DESCRIPTION
Some Implications of Expertise Research for Educational Assessment. Robert J. Mislevy, PhD Professor of Measurement & Statistics University of Maryland Keynote address at the 34th International Association for Educational Assessment (IAEA) Conference, Cambridge University, September 8, 2008. - PowerPoint PPT Presentation
Citation preview
IAEA 2008 Slide 1September 8, 2008
Some Implications of Expertise Research for Educational Assessment
Robert J. Mislevy, PhDProfessor of Measurement & Statistics
University of Maryland
Keynote address at the 34th International Association for Educational Assessment (IAEA) Conference, Cambridge University, September 8, 2008.
IAEA 2008 Slide 2September 8, 2008
Introduction These are exciting times in
assessment. Developments in psychology and
technology. Insights from expertise research with
implications for assessment design. Let’s start with a quiz.
IAEA 2008 Slide 3September 8, 2008
What is this a picture of? (http://www.optillusions.com)
IAEA 2008 Slide 4September 8, 2008
Which cards need to be turned over?
E 4 T 7
Each card has a letter on one side and a number on the other. Consider the rule “If there is a vowel on one side, there is an even number on the other.” Which cards do you need to turn over to make sure the rule is not violated? (Wason,1966)
IAEA 2008 Slide 5September 8, 2008
A Little Story
I will read a little story and ask you some questions about it.
IAEA 2008 Slide 6September 8, 2008
Was this sentence in the story?
1. Mary gave Ed a stuffed hippo.
2. Harold received ninety gifts.
3. Ralph gave Tina a toy car.
IAEA 2008 Slide 7September 8, 2008
There were five sentences in the story.
In order, what were the second-to-last words in each of the sentences?
IAEA 2008 Slide 8September 8, 2008
What is this a picture of? (http://www.optillusions.com)
IAEA 2008 Slide 9September 8, 2008
Which cards need to be turned over?
E 4 T 7
Each card has a letter on one side and a number on the other. Consider the rule “If there is a vowel on one side, there is an even number on the other.” Which cards do you need to turn over to make sure the rule is not violated? (Wason,1966)
Whose IDs do we need to check?
Each person has an age and a beverage. Consider the rule “If you are under 21, your beverage must be non-alcoholic.” Which people do you need to check to make sure the rule is not violated?
15Year old
Water drinke
r
30Year old
Wine drinke
r
IAEA 2008 Slide 10September 8, 2008
Was this sentence in the story?
1. Mary gave Ed a stuffed hippo.
2. Harold received ninety gifts.
3. Ralph gave Tina a toy car.
YES
NO
NO
Ralph gave a toy car to Tina.
IAEA 2008 Slide 11September 8, 2008
There were five sentences in the story.
In order, what were the second-to-last words in each of the sentences?
One, stuffed, to, four, holiday.
IAEA 2008 Slide 12September 8, 2008
Limitations and DifficultiesProcessing limitations Limited attention Limited working memory
Knowledge limitations Not knowing what information is relevant Don’t know how to integrate information Not knowing what to expect Not knowing what to do and when to do it Lack of production proficiency
IAEA 2008 Slide 13September 8, 2008
Capabilities Reasoning in terms of patterns Many patterns simultaneously (language) Can make perception, procedures, strategies
automatic with practice Can think about our thinking (metacognition) Benefit from procedures, methods, tools, external
knowledge representations
Expertise as the circumvention of human processing limitations (Salthouse, 1991)
IAEA 2008 Slide 14September 8, 2008
A Closer Look at Cognition Knowledge as patterns, at many levels… Assembled to understand, to interact with, and to
create particular situations in the world Developed, strengthened, modified by use Associations of all kinds, including applicability,
affordances, procedures, strategies, affect
“The user’s knowledge of the language rules is
interlocked with his knowledge of when, where, and
with whom to use them” (R. Ellis, 1985)
IAEA 2008 Slide 15September 8, 2008
Walter Kintsch’s CI Theory of Reading Comprehension
More focused research areas within cognitive psychology today differ as to their foci, methods, and levels of explanation. They include perception and attention, language and communication, development of expertise, situated and sociocultural psychology, and neurological bases of cognition.
Text Text base Situation ModelContext
Context1
LTM
•E.g., reading tasks in Occupational English Test (OET; McNamara, 1996) call upon patterns re language, but also genre, medical knowledge, use of information in clinical settings
•E.g., reading tasks in Occupational English Test (OET; McNamara, 1996) call upon patterns re language, but also genre, medical knowledge, use of information in clinical settings
A relevant pattern from LTM may be activated in contexts but not others (e.g., physics models, use of conditionals).If a pattern hasn’t been learned, it won’t be activated (although it may get constructed in the interaction).
A relevant pattern from LTM may be activated in contexts but not others (e.g., physics models, use of conditionals).If a pattern hasn’t been learned, it won’t be activated (although it may get constructed in the interaction).
IAEA 2008 Slide 16September 8, 2008
Walter Kintsch’s CI Theory of Reading Comprehension
More focused research areas within cognitive psychology today differ as to their foci, methods, and levels of explanation. They include perception and attention, language and communication, development of expertise, situated and sociocultural psychology, and neurological bases of cognition.
Text Text base LTM Situation Model ActionContext
Context1
Context2
IAEA 2008 Slide 17September 8, 2008
Walter Kintsch’s CI Theory of Reading Comprehension
More focused research areas within cognitive psychology today differ as to their foci, methods, and levels of explanation. They include perception and attention, language and communication, development of expertise, situated and sociocultural psychology, and neurological bases of cognition.
Text Text base LTM Situation Model ActionContext
Context1
Context2
IAEA 2008 Slide 18September 8, 2008
Walter Kintsch’s CI Theory of Reading Comprehension
More focused research areas within cognitive psychology today differ as to their foci, methods, and levels of explanation. They include perception and attention, language and communication, development of expertise, situated and sociocultural psychology, and neurological bases of cognition.
Text Text base LTM Situation Model ActionContext
Context2
Context3
IAEA 2008 Slide 19September 8, 2008
Expertise Research
Cognitive task analysis (e.g., Simon & Chase) Compare experts & novices in replicable conditions
What knowledge is needed? How is it represented? How is it used? What makes tasks hard?
Ethnographic research (e.g., Lave) Expertise in situ “Critical incident” studies (NBME)
Replication possible in simulations Flight simulators, football kick in video situations
IAEA 2008 Slide 20September 8, 2008
Expertise ResearchExperts organize their knowledge effectively
Perceive / understand / act in terms of fundamental principles rather than surface features (Chi, Feltovich, & Glaser)
Importance of interaction with situationCycles of “perceive / understand / act”
External knowledge representations (KRs)Nexus of info-processing & sociocultural POVSupported cognition / distributed cognition
IAEA 2008 Slide 21September 8, 2008
So…
How do you use this improved understanding of the nature and acquisition of expertise to design and conduct assessments?
IAEA 2008 Slide 22September 8, 2008
Assessment Arguments
What complex of knowledge, skills, or other attributes should be assessed?
What behaviors or performances should reveal those constructs?
What tasks or situations should elicit those behaviors?
(Messick, 1994)
IAEA 2008 Slide 23September 8, 2008
Examples
The Architectural Registration Examination (ARE)
Architectural design; CAD-like environment
DISC simulator
Simulations for problem-solving in dental hygiene
NetPASS (Cisco)
Computer network design & troubleshooting
IAEA 2008 Slide 24September 8, 2008
ARE Example (Irv Katz, ETS) To replace 10-hour hand-drawn design
problem Reflects changing of profession to CAD Premium on thinking, not drawing Planning the firestation site
IAEA 2008 Slide 25September 8, 2008
An Example of a Task Prompt for the ARE
IAEA 2008 Slide 26September 8, 2008An Illustrative Base Diagram for an ARE Task
AB
D
IAEA 2008 Slide 27September 8, 2008 A Sample Solution to an ARE Task
IAEA 2008 Slide 28September 8, 2008
NCARB Example (Irv Katz, ETS)
Differences between novices and experts• Success rate: 98% vs. 88%
• Planning time & execution time
• Patterns of revision involving rework
Implications for task design: Constraints• Number
• Variation in importance/difficulty
• Degree of conflict
• Implicit constraints? (c.f. writing expertise)
IAEA 2008 Slide 29September 8, 2008
Design Patterns
“Design under constraints” is common to many domains: e.g., engineering, assessment design, wedding planning, apparel design
Could define a Design Pattern with these as foci What are characteristics of performances (i.e.,
observables) that evidence these knowledge / capabilities / attunements?
What are characteristics of situations that elicit these observables?
IAEA 2008 Slide 30September 8, 2008
Other design pattern possibilities
Troubleshooting finite systems Medical diagnosis Inquiry cycle Model-based reasoning
IAEA 2008 Slide 31September 8, 2008
DISC Example
The Dental Interactive Simulations Corporation (DISC)
The DISC Simulator & Scoring Engine Cognitive Task Analysis to support
design rationale
IAEA 2008 Slide 32September 8, 2008
Goals of the DISC CTA
What are the kinds of knowledge and skills that hygienists call upon to solve problems and make decisions in dental hygiene?
What do they say and do that gives you evidence about their use of this knowledge?
What kinds of situations call upon this knowledge?
How do you make re-usable schemas to evoke evidence and construct ‘stories’ around these recurring patterns?
IAEA 2008 Slide 33September 8, 2008
Using Disparate Sources of Information
Expert Tries to use all sources of information all the time. Constructs a model of the patient, with each source of information an imperfect and incomplete window on some aspect of the total situation. Exhibits movement back and forth between sources (resources and personal knowledge/experience), trying to fit the partial clues together into a unified whole.
Novice Uses single information sources in isolation. Compartmentalized use of information and failure to integrate information across sources.
IAEA 2008 Slide 34September 8, 2008
Formulating Problems & Hypotheses Expert Forms problems and generates hypotheses
using efficient, focused, and targeted action. Uses forward and deductive reasoning in formulating problems. Thinks strategically and functions within the problem space. Efficiently generates and prunes search trees.
Novice Uses forward reasoning but scope and depth of knowledge to support forward reasoning is limited. Generates a search tree but possesses limited tools for pruning the search tree.
IAEA 2008 Slide 35September 8, 2008
Vocabulary and Language Usage Expert Retrieves and uses appropriate, clear,
sophisticated, accurate, and precise terminology. Uses spontaneous declaration and can retrieve without effort. Creates messages that are easily understood by the target audience.
Novice Strained, unreliable, and effortful retrieval of terminology. Hesitant, delayed, and labored responses due to time required to process and lack of knowledge. Uses terminology unsystematically. Prone to canned speeches and explanations without awareness of individual patients’ needs.
IAEA 2008 Slide 36September 8, 2008
An example with the Cisco Learning Institute: The NetPass Prototype
Create on-line performance assessment of networking skills
Focus on learner feedback rather than high-stakes testing
Expert/Novice studies ground design
IAEA 2008 Slide 37September 8, 2008
The task starts with a scenario and description of goals
IAEA 2008 Slide 38September 8, 2008
To capture their mental model of the network, we ask them to draw the network with a diagramming tool
IAEA 2008 Slide 39September 8, 2008
The diagram is created by dragging and dropping icons
IAEA 2008 Slide 40September 8, 2008
Configuring the devices
IAEA 2008 Slide 41September 8, 2008
When students are done, they press Submit”…
IAEA 2008 Slide 42September 8, 2008
And the graphical representation is converted into a text representation in XML format
IAEA 2008 Slide 43September 8, 2008
The text file is scored following detailed rules, which result in characterizations of the work
IAEA 2008 Slide 44September 8, 2008
And feedback is created for the student…both diagnostic and a summary profile.
IAEA 2008 Slide 45September 8, 2008
Conclusion Insights from expertise research can improve
the practice of assessment, and support deeper learning.
Doing so requires a deeper understanding of assessment design.» More explicit arguments and representations
» Generativity, re-usability, and inter-operability
“Too many notes” ? (Emperor Joseph II) Suitable conceptual frameworks, tools, and
exemplars are now beginning to appear.