Embed Size (px)
Citation preview
ADL Slide 1December 15, 2009
Evidence-Centered Design and Cisco’s Packet Tracer Simulation-Based Assessment
Robert J. MislevyProfessor, Measurement & Statistics
University of Maryland
with
John T. Behrens & Dennis FrezzoCisco Systems, Inc.
December 15, 2009
ADL, Alexandria, VA
ADL Slide 2December 15, 2009
Simulation-based & Game-based Assessment
Motivation: Cog psych & technology» Complex combinations of knowledge &
skills» Complex situations» Interactive, evolving in time, constructive» Challenge of technology-based
environments
ADL Slide 3December 15, 2009
Outline
ECD Packet Tracer Packet Tracer & ECD
ADL Slide 4December 15, 2009
ECD
ADL Slide 5December 15, 2009
Messick’s (1994) guiding questions: What complex of knowledge, skills, or other
attributes should be assessed? What behaviors or performances should
reveal those constructs? What tasks or situations should elicit those
behaviors?
Evidence-Centered Assessment Design
ADL Slide 6December 15, 2009
Evidence-Centered Assessment Design
Principled framework for designing, producing, and delivering assessments
Process model, object model, design tools Explicates the connections among
assessment designs, inferences regarding students, and the processes needed to create and deliver these assessments.
Particularly useful for new / complex assessments.
From Mislevy & Riconscente, in press
Assessment DeliveryAssessment DeliveryStudents interact with tasks, performances evaluated, feedback created. Four-process delivery architecture.
Students interact with tasks, performances evaluated, feedback created. Four-process delivery architecture.
Assessment Implementation
Assessment Implementation
Conceptual Assessment Framework
Conceptual Assessment Framework
Domain ModelingDomain Modeling
Domain AnalysisDomain Analysis What is important about this domain?What work and situations are central in this domain?What KRs are central to this domain?
What is important about this domain?What work and situations are central in this domain?What KRs are central to this domain?
How do we represent key aspects of the domain in terms of assessment argument. Conceptualization.
How do we represent key aspects of the domain in terms of assessment argument. Conceptualization.
Design structures: Student, evidence, and task models. Generativity.
Design structures: Student, evidence, and task models. Generativity.
Manufacturing “nuts & bolts”: authoring tasks, automated scoring details, statistical models. Reusability.
Manufacturing “nuts & bolts”: authoring tasks, automated scoring details, statistical models. Reusability.
Layers in the assessment enterpriseLayers in the assessment enterprise
ADL Slide 8December 15, 2009
Packet Tracer
ADL Slide 9December 15, 2009
Cisco’s Packet Tracer
Online tool used in Cisco Networking Academies Create, edit, configure, run, troubleshoot networks Multiple representations in the logical layer Inspection tool links to a deeper physical world Simulation mode
» Detailed visualization and data presentation Standard support for world authoring
» Library of elements» Simulation of relevant deep structure» Copy, paste, save, edit, annotate, lock
ADL Slide 10December 15, 2009
Instructors and students can author their own activities
ADL Slide 11December 15, 2009
Instructors and students can author their own activities
Instructors and students can author their own activities
Explanation
Experimentation
ADL Slide 15December 15, 2009
Packet Tracer & ECD
From Mislevy & Riconscente, in press
Assessment DeliveryAssessment DeliveryStudents interact with tasks, performances evaluated, feedback created. Four-process delivery architecture.
Students interact with tasks, performances evaluated, feedback created. Four-process delivery architecture.
Assessment Implementation
Assessment Implementation
Conceptual Assessment Framework
Conceptual Assessment Framework
Domain ModelingDomain Modeling
Domain AnalysisDomain Analysis What is important about this domain?What work and situations are central in this domain?What KRs are central to this domain?
What is important about this domain?What work and situations are central in this domain?What KRs are central to this domain?
How do we represent key aspects of the domain in terms of assessment argument. Conceptualization.
How do we represent key aspects of the domain in terms of assessment argument. Conceptualization.
Design structures: Student, evidence, and task models. Generativity.
Design structures: Student, evidence, and task models. Generativity.
Manufacturing “nuts & bolts”: authoring tasks, automated scoring details, statistical models. Reusability.
Manufacturing “nuts & bolts”: authoring tasks, automated scoring details, statistical models. Reusability.
Layers in the assessment enterpriseLayers in the assessment enterprise
From Mislevy & Riconscente, in press
Assessment DeliveryAssessment DeliveryStudents interact with tasks, performances evaluated, feedback created. Four-process delivery architecture.
Students interact with tasks, performances evaluated, feedback created. Four-process delivery architecture.
Assessment Implementation
Assessment Implementation
Conceptual Assessment Framework
Conceptual Assessment Framework
Domain ModelingDomain Modeling
Domain AnalysisDomain Analysis What is important about this domain?What work and situations are central in this domain?What KRs are central to this domain?
What is important about this domain?What work and situations are central in this domain?What KRs are central to this domain?
How do we represent key aspects of the domain in terms of assessment argument. Conceptualization.
How do we represent key aspects of the domain in terms of assessment argument. Conceptualization.
Design structures: Student, evidence, and task models. Generativity.
Design structures: Student, evidence, and task models. Generativity.
Manufacturing “nuts & bolts”: authoring tasks, automated scoring details, statistical models. Reusability.
Manufacturing “nuts & bolts”: authoring tasks, automated scoring details, statistical models. Reusability.
From Mislevy & Riconscente, in press
Assessment DeliveryAssessment DeliveryStudents interact with tasks, performances evaluated, feedback created. Four-process delivery architecture.
Students interact with tasks, performances evaluated, feedback created. Four-process delivery architecture.
Assessment Implementation
Assessment Implementation
Conceptual Assessment Framework
Conceptual Assessment Framework
Domain ModelingDomain Modeling
Domain AnalysisDomain Analysis What is important about this domain?What work and situations are central in this domain?What KRs are central to this domain?
What is important about this domain?What work and situations are central in this domain?What KRs are central to this domain?
How do we represent key aspects of the domain in terms of assessment argument. Conceptualization.
How do we represent key aspects of the domain in terms of assessment argument. Conceptualization.
Design structures: Student, evidence, and task models. Generativity.
Design structures: Student, evidence, and task models. Generativity.
Manufacturing “nuts & bolts”: authoring tasks, automated scoring details, statistical models. Reusability.
Manufacturing “nuts & bolts”: authoring tasks, automated scoring details, statistical models. Reusability.
• Assessment argument structures• Design Patterns
From Mislevy & Riconscente, in press
Assessment DeliveryAssessment DeliveryStudents interact with tasks, performances evaluated, feedback created. Four-process delivery architecture.
Students interact with tasks, performances evaluated, feedback created. Four-process delivery architecture.
Assessment Implementation
Assessment Implementation
Conceptual Assessment Framework
Conceptual Assessment Framework
Domain ModelingDomain Modeling
Domain AnalysisDomain Analysis What is important about this domain?What work and situations are central in this domain?What KRs are central to this domain?
What is important about this domain?What work and situations are central in this domain?What KRs are central to this domain?
How do we represent key aspects of the domain in terms of assessment argument. Conceptualization.
How do we represent key aspects of the domain in terms of assessment argument. Conceptualization.
Design structures: Student, evidence, and task models. Generativity.
Design structures: Student, evidence, and task models. Generativity.
Manufacturing “nuts & bolts”: authoring tasks, automated scoring details, statistical models. Reusability.
Manufacturing “nuts & bolts”: authoring tasks, automated scoring details, statistical models. Reusability.
Application to familiar assessments
Upfront design of features of task
situation – static, implicit in task, not
usually tagged
Upfront design of
features of response classes
Fixed competency variables
From Mislevy & Riconscente, in press
Assessment DeliveryAssessment DeliveryStudents interact with tasks, performances evaluated, feedback created. Four-process delivery architecture.
Students interact with tasks, performances evaluated, feedback created. Four-process delivery architecture.
Assessment Implementation
Assessment Implementation
Conceptual Assessment Framework
Conceptual Assessment Framework
Domain ModelingDomain Modeling
Domain AnalysisDomain Analysis What is important about this domain?What work and situations are central in this domain?What KRs are central to this domain?
What is important about this domain?What work and situations are central in this domain?What KRs are central to this domain?
How do we represent key aspects of the domain in terms of assessment argument. Conceptualization.
How do we represent key aspects of the domain in terms of assessment argument. Conceptualization.
Design structures: Student, evidence, and task models. Generativity.
Design structures: Student, evidence, and task models. Generativity.
Manufacturing “nuts & bolts”: authoring tasks, automated scoring details, statistical models. Reusability.
Manufacturing “nuts & bolts”: authoring tasks, automated scoring details, statistical models. Reusability.
Application to complex assessments
More complex
evaluations of
features of task
situation
Multiple, perhaps
configured-on-the-fly
competency variables
From Mislevy & Riconscente, in press
Assessment DeliveryAssessment DeliveryStudents interact with tasks, performances evaluated, feedback created. Four-process delivery architecture.
Students interact with tasks, performances evaluated, feedback created. Four-process delivery architecture.
Assessment Implementation
Assessment Implementation
Conceptual Assessment Framework
Conceptual Assessment Framework
Domain ModelingDomain Modeling
Domain AnalysisDomain Analysis What is important about this domain?What work and situations are central in this domain?What KRs are central to this domain?
What is important about this domain?What work and situations are central in this domain?What KRs are central to this domain?
How do we represent key aspects of the domain in terms of assessment argument. Conceptualization.
How do we represent key aspects of the domain in terms of assessment argument. Conceptualization.
Design structures: Student, evidence, and task models. Generativity.
Design structures: Student, evidence, and task models. Generativity.
Manufacturing “nuts & bolts”: authoring tasks, automated scoring details, statistical models. Reusability.
Manufacturing “nuts & bolts”: authoring tasks, automated scoring details, statistical models. Reusability.
Application to familiar assessmentsApplication to complex assessments
Macro features of performance
Micro features of performance
Unfolding situated
performance
Micro features of situation
Macro features of situation
TimeSome up front
design of features of
task situation; others
recognized (e.g., agents)
Some up front design of
features of performance
or effects; others
recognized (e.g., agents)
Evolving, interactive, situation
From Mislevy & Riconscente, in press
Assessment DeliveryAssessment DeliveryStudents interact with tasks, performances evaluated, feedback created. Four-process delivery architecture.
Students interact with tasks, performances evaluated, feedback created. Four-process delivery architecture.
Assessment Implementation
Assessment Implementation
Conceptual Assessment Framework
Conceptual Assessment Framework
Domain ModelingDomain Modeling
Domain AnalysisDomain Analysis What is important about this domain?What work and situations are central in this domain?What KRs are central to this domain?
What is important about this domain?What work and situations are central in this domain?What KRs are central to this domain?
How do we represent key aspects of the domain in terms of assessment argument. Conceptualization.
How do we represent key aspects of the domain in terms of assessment argument. Conceptualization.
Design structures: Student, evidence, and task models. Generativity.
Design structures: Student, evidence, and task models. Generativity.
Manufacturing “nuts & bolts”: authoring tasks, automated scoring details, statistical models. Reusability.
Manufacturing “nuts & bolts”: authoring tasks, automated scoring details, statistical models. Reusability. Object models for representing …
• Psychometric modelsIncludes “competences/proficiencies
•Simulation environments• Task templates• Automated scoring
Student Model
Evidence Model(s)
Measurement Model
Scoring Model
X1
Task Model(s)
1 . x x x x x x x x 2 . x x x x x x x x
3 . x x x x x x x x 4 . x x x x x x x x 5 . x x x x x x x x 6 . x x x x x x x x 7 . x x x x x x x x 8 . x x x x x x x x
X2
X1
X2
From Mislevy & Riconscente, in press
Assessment DeliveryAssessment DeliveryStudents interact with tasks, performances evaluated, feedback created. Four-process delivery architecture.
Students interact with tasks, performances evaluated, feedback created. Four-process delivery architecture.
Assessment Implementation
Assessment Implementation
Conceptual Assessment Framework
Conceptual Assessment Framework
Domain ModelingDomain Modeling
Domain AnalysisDomain Analysis What is important about this domain?What work and situations are central in this domain?What KRs are central to this domain?
What is important about this domain?What work and situations are central in this domain?What KRs are central to this domain?
How do we represent key aspects of the domain in terms of assessment argument. Conceptualization.
How do we represent key aspects of the domain in terms of assessment argument. Conceptualization.
Design structures: Student, evidence, and task models. Generativity.
Design structures: Student, evidence, and task models. Generativity.
Manufacturing “nuts & bolts”: authoring tasks, automated scoring details, statistical models. Reusability.
Manufacturing “nuts & bolts”: authoring tasks, automated scoring details, statistical models. Reusability.
PADI object model for task/evidence models
From Mislevy & Riconscente, in press
Assessment DeliveryAssessment DeliveryStudents interact with tasks, performances evaluated, feedback created. Four-process delivery architecture.
Students interact with tasks, performances evaluated, feedback created. Four-process delivery architecture.
Assessment Implementation
Assessment Implementation
Conceptual Assessment Framework
Conceptual Assessment Framework
Domain ModelingDomain Modeling
Domain AnalysisDomain Analysis What is important about this domain?What work and situations are central in this domain?What KRs are central to this domain?
What is important about this domain?What work and situations are central in this domain?What KRs are central to this domain?
How do we represent key aspects of the domain in terms of assessment argument. Conceptualization.
How do we represent key aspects of the domain in terms of assessment argument. Conceptualization.
Design structures: Student, evidence, and task models. Generativity.
Design structures: Student, evidence, and task models. Generativity.
Manufacturing “nuts & bolts”: authoring tasks, automated scoring details, statistical models. Reusability.
Manufacturing “nuts & bolts”: authoring tasks, automated scoring details, statistical models. Reusability.
Graphical representation of network & configuration is expressable as a text representation in XML format, for presentation & work product, to support automated scoring.
From Mislevy & Riconscente, in press
Assessment DeliveryAssessment DeliveryStudents interact with tasks, performances evaluated, feedback created. Four-process delivery architecture.
Students interact with tasks, performances evaluated, feedback created. Four-process delivery architecture.
Assessment Implementation
Assessment Implementation
Conceptual Assessment Framework
Conceptual Assessment Framework
Domain ModelingDomain Modeling
Domain AnalysisDomain Analysis What is important about this domain?What work and situations are central in this domain?What KRs are central to this domain?
What is important about this domain?What work and situations are central in this domain?What KRs are central to this domain?
How do we represent key aspects of the domain in terms of assessment argument. Conceptualization.
How do we represent key aspects of the domain in terms of assessment argument. Conceptualization.
Design structures: Student, evidence, and task models. Generativity.
Design structures: Student, evidence, and task models. Generativity.
Manufacturing “nuts & bolts”: authoring tasks, automated scoring details, statistical models. Reusability.
Manufacturing “nuts & bolts”: authoring tasks, automated scoring details, statistical models. Reusability.
• Authoring interfaces• Simulation environments• Re-usable platforms & elements• Standard data structures• IMS/QTI, SCORM
In Packet Tracer, Answer Network Serves as base pattern for work product evaluation
Dynamic task models - Variable Assignment: Initial Network
Similar to the Answer Network Tree.When the activity starts, instead of using the initial network as the starting values, the activity will configure the network with the contents of the variables.
From Mislevy & Riconscente, in press
Assessment DeliveryAssessment DeliveryStudents interact with tasks, performances evaluated, feedback created. Four-process delivery architecture.
Students interact with tasks, performances evaluated, feedback created. Four-process delivery architecture.
Assessment Implementation
Assessment Implementation
Conceptual Assessment Framework
Conceptual Assessment Framework
Domain ModelingDomain Modeling
Domain AnalysisDomain Analysis What is important about this domain?What work and situations are central in this domain?What KRs are central to this domain?
What is important about this domain?What work and situations are central in this domain?What KRs are central to this domain?
How do we represent key aspects of the domain in terms of assessment argument. Conceptualization.
How do we represent key aspects of the domain in terms of assessment argument. Conceptualization.
Design structures: Student, evidence, and task models. Generativity.
Design structures: Student, evidence, and task models. Generativity.
Manufacturing “nuts & bolts”: authoring tasks, automated scoring details, statistical models. Reusability.
Manufacturing “nuts & bolts”: authoring tasks, automated scoring details, statistical models. Reusability.
From Mislevy & Riconscente, in press
Assessment DeliveryAssessment DeliveryStudents interact with tasks, performances evaluated, feedback created. Four-process delivery architecture.
Students interact with tasks, performances evaluated, feedback created. Four-process delivery architecture.
Assessment Implementation
Assessment Implementation
Conceptual Assessment Framework
Conceptual Assessment Framework
Domain ModelingDomain Modeling
Domain AnalysisDomain Analysis What is important about this domain?What work and situations are central in this domain?What KRs are central to this domain?
What is important about this domain?What work and situations are central in this domain?What KRs are central to this domain?
How do we represent key aspects of the domain in terms of assessment argument. Conceptualization.
How do we represent key aspects of the domain in terms of assessment argument. Conceptualization.
Design structures: Student, evidence, and task models. Generativity.
Design structures: Student, evidence, and task models. Generativity.
Manufacturing “nuts & bolts”: authoring tasks, automated scoring details, statistical models. Reusability.
Manufacturing “nuts & bolts”: authoring tasks, automated scoring details, statistical models. Reusability.
• Interoperable elements• IMS/QTI, SCORM• Feedback / instruction / reporting
ADL Slide 30December 15, 2009
Conclusion
Behavior in learning environments builds connections for performance environments.
Assessment tasks & features strongly related to instruction/learning objects & features.
Re-use concepts and code in assessment,» via arguments, schemas, data structures that are
consonant with instructional objects.
Use data structures that are » share-able, extensible,
» consistent with delivery processes and design models.
ADL Slide 31December 15, 2009
Further information
Bob Mislevy home page» http://www.education.umd.edu/EDMS/mislevy/» Links to papers on ECD» Cisco NetPASS» Cisco Packet Tracer
PADI: Principled Assessment Design for Inquiry » NSF project, collaboration with SRI et al. » http://padi.sri.com
