Upload
liz
View
35
Download
0
Embed Size (px)
DESCRIPTION
Enhanced Assessment Grant. Integrating Simulation-Based Science Assessments into Balanced State Science Assessment Systems: Findings and Implications. Richard Vineyard Nevada Department of Education Edys Quellmalz and Matt Silberglitt WestEd - PowerPoint PPT Presentation
Citation preview
COPYRIGHT WESTED, 2011
Integrating Simulation-Based Science Assessments into Balanced State Science
Assessment Systems:Findings and Implications
Richard VineyardNevada Department of Education
Edys Quellmalz and Matt SilberglittWestEd
Presented to the Council of State Science Supervisors, San Francisco, CA, March 9, 2011
Enhanced Assessment Grant
COPYRIGHT WESTED, 2011
Enhanced Assessment Grant State Partners
• Nevada (Lead State) • Utah • North Carolina • Massachusetts • Connecticut • Vermont
COPYRIGHT WESTED, 2011
Advisors
• Rebecca Kopriva, Wisconsin Center for Education Research–Assessments for English Learners
• Martha Thurlow, National Center on Education Outcomes–Assessments for Students with
Disabilities
COPYRIGHT WESTED, 2011
Subcontractors
• WestEd STEM (SimScientists)– Project Management– Content Development
• Redhill Studios– Interactive (Flash) Programming
• UCLA/CRESST– External Evaluation
COPYRIGHT WESTED, 2011
EAG Goals
• Establish the technical quality of simulation-based assessments – for two topics, ecosystems and force and motion
• Study the feasibility and utility of simulation-based assessments in classrooms from 3 states
COPYRIGHT WESTED, 2011
EAG Goals
• Study the measurement of science knowledge and inquiry by simulation-based curriculum-embedded assessments and unit benchmark assessments in comparison to traditional items for:– All students– ELLs– Students with disabilities
• Propose alternative models for integrating simulation-based assessments into state assessment systems
COPYRIGHT WESTED, 2011
Relevance to Current Assessment Programs
• Need innovative, technology-enhanced assessments that align with new frameworks that focus on – fewer, deeper, more integrated core knowledge
targets • E.g., the new Framework for Science Education and
next generation national science standards • Models as structures for understanding and studying
science systems (model-based learning)• Science practices for using knowledge and inquiry in
significant, recurring authentic tasks
COPYRIGHT WESTED, 2011
Relevance to Current Assessment Programs
Need innovative, technology-enhanced assessments that• Target 21st century skills within STEM• Use technology to engage students in use of 21st
century “tools of the trade” • Provide evidence supporting one type of technology-
enhanced performance assessment– For summative and formative purposes – Called for in the RTT Consortia
COPYRIGHT WESTED, 2011
EAG Design
• Align to national science standards and to science standards of the pilot states– Ecosystems and force & motion– At middle school level– Nevada, North Carolina, Utah
• Accommodations– Extended time– Audio– Zoom
COPYRIGHT WESTED, 2011
Ecosystem Target Model
System Model Levels
Model LevelDescriptions
Content Targetsby Model Level
Inquiry Targets
Components What are the components of the system and their rules of behavior?
Every ecosystem has a similar pattern of organization with respect to the roles (producers, consumers, and decomposers) that organisms play in the movement of energy and matter through the system.
Use principles to identify role of organisms.
Interactions How do the the individual components interact?
Matter and energy flow through the ecosystem as individual organisms participate in feeding relationships within an ecosystem.
Observe interactions among organisms.
Emergent Behaviors What is the overall behavior or property of the system that results from many interactions following specific rules?
Interactions among organisms and among organisms and the ecosystem’s nonliving features cause the populations of the different organisms to change over time.
predict observe explain investigate
COPYRIGHT WESTED, 2011
Multiple Modes of RepresentationActive Inquiry
Organism box shows icons that appear and disappear
Graph shows size of population over time.
Data inspector shows population for one point on graph.
Table shows population at start, end, and point selected by data inspector.
COPYRIGHT WESTED, 2011
EMBEDDED IN CLASSROOM INSTRUCTION
Assessment Module Components for Each Topic/System
Online assessment without feedback
Teacher scores constructed responses
Bayes Net
Proficiency report
Benchmark Summative Unit Assessments
Online assessment with feedback and coaching
Follow up Classroom Reflection ActivityProgress report
Embedded Assessments for Formative Purposes (2 to 4)
COPYRIGHT WESTED, 2011
Ecosystem Embedded Assessment: Feedback,
Coaching, and Self-AssessmentDemo
COPYRIGHT WESTED, 2011
Progress Reports to Students
COPYRIGHT WESTED, 2011
Progress Reports to Teachers
NH = needs help P = making progress OT = on track
COPYRIGHT WESTED, 2011
Grouping Recommendations for Classroom Reflection Activity
COPYRIGHT WESTED, 2011
Classroom Reflection Activity
• Formative use of assessment results– Students assigned to teams based on embedded results
• Transfer to different, more complex system• Jigsaw structure
– Allows differentiated instruction via tasks of varying difficulty– Promotes small and large group discourse and collaboration
• Guidance for teacher– Teacher review of key points in simulation– What to look for during group work and questions to pose in response– Presentations– Evaluation of presentations
COPYRIGHT WESTED, 2011
Benchmark Assessment:Assess Transfer to New Context
Demo
COPYRIGHT WESTED, 2011
Benchmark Report: Class Summary
COPYRIGHT WESTED, 2011
Benchmark Report: Student Details
COPYRIGHT WESTED, 2011
Samples
• 28 think aloud sessions• Pilot testing in 4 states
• 60 participating teachers• ~5,800 students• 40 schools• 28 districts– Response data on the assessments, post tests and
classroom observations
COPYRIGHT WESTED, 2011
Research Evidence• Quality of the assessments (external review)
– Alignment with standards and curriculum– Coherence– Science content and inquiry– Item design
• Implementation– Feasibility in schools and classrooms– Utility for students and teachers
COPYRIGHT WESTED, 2011
Research Evidence
• Technical Quality (Validity & Reliability)– Alignment with science standards– Cognitive labs– Analyses of responses
• Comparison of student responses on simulation-based and traditional items– Cognitive labs �– IRT model fits�
COPYRIGHT WESTED, 2011
Reliability of Benchmark and Post Tests
• Ecosystems– Benchmark (n=3844)
• Mean score was 28.14 (max 39)• Standard Deviation 6.64• Coefficient Alpha: 0.85
– Post Test (n=3422)• Mean score: 16.52 (max 30) • Standard deviation: 6.68• Coefficient Alpha (KR 20): 0.88
COPYRIGHT WESTED, 2011
Reliability of Benchmark and Post Tests
Force and Motion– Benchmark (n=1504)
• Mean score: 28.15 (max 40) • Standard Deviation: 5.51• Coefficient Alpha: 0.79
– Post Test (n=2166)• Mean score: 14.87 (max 28) • Standard Deviation: 5.73• Coefficient Alpha (KR 20): 0.82
COPYRIGHT WESTED, 2011
Correlation of Benchmark to Posttest
Ecosystems(n=2924)
Force and Motion(n=1496)
ContentAbility
estimate
InquiryAbility
estimate
ContentAbility
estimate
InquiryAbility
estimateCorrelation with ability estimates on posttest
.51** .24** .40** .45**
** Correlation is significant at the 0.01 level (2-tailed)
COPYRIGHT WESTED, 2011
CRESST Evaluation Questions
• How effective was SimScientists implementation?
• Was SimScientists feasible and useful in middle school classrooms?
• How effective were SimScientists’ accommodations?
COPYRIGHT WESTED, 2011
CRESST Evaluation Methods
• Convenience sample of teachers from all three pilot states, both topics– Eight teachers with 3-4 classes– Five schools
• Observation and interview protocols– Assessments– Reflection Activities
• Teacher survey
COPYRIGHT WESTED, 2011
Teacher’s Role (Assessment)
COPYRIGHT WESTED, 2011
Class Organization (RA)
COPYRIGHT WESTED, 2011
CRESST Interviews
“I think that [students] are way more engaged … using this kind of interactive assessment … it was more engaging them simply, “Here is a pencil, here is a paper, answer questions” .”
COPYRIGHT WESTED, 2011
CRESST Interviews• “…I like that kind of feedback when it doesn’t just go
to the next page and they don’t know whether they did it right or not.”
• “…Yes, the science content is really being tested. Students are asked to conduct experiments, investigate, and draw conclusions and to use scientific skills. Students are not able to guess on the multiple choice questions because it probes them until they choose the right answer. Students are also taught about food webs in one biome and they are tested on another biome.”
COPYRIGHT WESTED, 2011
CRESST Evaluation Conclusions
• Observations showed that students were actively engaged most of the time during assessments.
• Both teachers and students generally believed that the SimScientists program was beneficial to learning.
• Teachers found the automatically scored, immediate feedback—especially the reports generated by the questions—helpful to students. The instant reports allowed teachers to easily see which questions students had the most difficulty with so that they could tailor their lessons accordingly.
COPYRIGHT WESTED, 2011
CRESST Evaluation Conclusions
• Teachers collectively agreed that the simulation assessments had greater benefits than traditional paper-and-pencil tests because of the simulation’s instant feedback, interaction, and visuals.
• Teachers agreed that the assessments would be useful in measuring their individual state standards.
COPYRIGHT WESTED, 2011
Multilevel Balanced Assessment System Models
• Reporting benchmark results alongside district and state data
• Matrix sampling of short “signature” tasks from different topics
COPYRIGHT WESTED, 2011
COPYRIGHT WESTED, 2011
Matrix Sampling Short “Signature Tasks” Across Topics
COPYRIGHT WESTED, 2011
Policy Brief
• Summary of alternative models for integrating simulation-based assessments into balanced state assessment reporting systems
• Summary of study findings on feasibility, utility, and technical quality
• Implications for further study
COPYRIGHT WESTED, 2011
Related Activities• Increased use of technology in classroom instruction and
assessment• Continued (and growing) frustration around measuring
student’s knowledge and skills above conceptual level• Growth in costs associated with hand scoring of CR items on
science assessments at grades 5 and 8• Development of electronic and on-line resources for
instruction, assessment, and professional development for math and science – MINES, Science Builder, Pathways to Nevada’s Future
COPYRIGHT WESTED, 2011
Other Projects• Race to the Top• Assessment consortium related to CCSS in
math and reading• Multilevel Assessments of Science Standards• Science Assessments with UDL • EAG proposal for 2011?
– Study series of assessment suites within a grade
COPYRIGHT WESTED, 2011
Contact information
Email: [email protected]@[email protected]
Website: www.simscientists.org