Upload
ryo
View
45
Download
0
Tags:
Embed Size (px)
DESCRIPTION
MCAS Standards Validation: High School Introductory Physics. Sheraton Hotel Braintree, MA September 17-18, 2007. Overview of August Standards Setting. Independent standard-setting for: Biology Chemistry Introductory Physics Technology/Engineering Cut scores recommended for: - PowerPoint PPT Presentation
Citation preview
MCAS Standards Validation:MCAS Standards Validation:High School Introductory High School Introductory
Physics Physics
Sheraton HotelBraintree, MA
September 17-18, 2007
Overview of August Overview of August Standards SettingStandards SettingIndependent standard-setting for:
BiologyChemistryIntroductory PhysicsTechnology/Engineering
Cut scores recommended for:Failing/Needs ImprovementNeeds Improvement/ProficientProficient/Advanced
Each panel composed of:• high school science teachers and administrators• university representatives• community representatives
Post Standard-Setting Post Standard-Setting AnalysisAnalysis
• Analyzed each of the recommended cut scores (n=12) across all four content areas
• Validated 10 of 12 recommended cut scores
• Issues related to 2 of the recommended cuts:– Raw score ranges associated with two cuts for
F/NI and NI/P for Introductory Physics were extreme outliers
– Some Introductory Physics panelists expressed concern about standard-setting process
Plenary SessionPlenary Session Welcome Overview of August Standard Setting
Meeting Post Standard Setting Analysis Standard Setting versus Standards Validation STE tests: efforts to establish “equivalence”
and uses of test results Recap of Body of Work (BOW) method and
role of performance level descriptors Questions and Answers
Additional Analyses Additional Analyses Conducted Conducted
Relationship of projected cut scores to psychometric properties of test (TCCs and TIFs)
Relationship of previous student performance in science with that of IP test based on recommended cut scores
Relationship of student work classified as Needs Improvement and Proficient to:Introductory Physics performance level descriptorsstudent work classified as Needs Improvement and
Proficient for Biology, Chemistry, and T/E
Conclusions Post Standard-Conclusions Post Standard-SettingSetting
• Interpretation of Needs Improvement and Proficient diverged significantly from interpretation made in the other content areas
• Considered statistical adjustment as remedy but determined Standards Validation preferable given high stakes use of results
• Delay in posting of scaled score/raw score conversion tables for all four content areas until standards validation completed
Remember, we knew this Remember, we knew this was going to be was going to be challenging…challenging…
Standard setting for multiple tests–-where roughly “comparable” standards across all tests is the goal—is technically challenging
Other approaches to standard setting considered:
Overlapping content panelsOne panel composed of all representatives for
all four content areasIndependent panels
General Phases of Standard General Phases of Standard SettingSetting
Data-collection phase
Policy-making/decision-making phase
Standard Setting vs.Standard Setting vs.Standards ValidationStandards Validation
Standard setting– Process of establishing original cut
scores– Panelists are not provided initial cut
points or focused cut point range
Standards validation – Process of validating cut scores– Panelists are provided initial cut points
or a narrowed cut point range
Introductory Physics Introductory Physics Standards ValidationStandards Validation
Warning
Needs Improvement
Proficient
Cut score
needed
Cut score
established
Cut score
needed
Advanced
Standards Validation Standards Validation Bodies of work to be classified represent a narrowed
range of student work based upon:
Standard setting panelists’ recommendations (August)Psychometric and statistical analysesAnalysis of classification of student work for each of the 12
recommended cut scores (conducted by DOE and Measured Progress content experts)
Pinpointing cut score forFailing/Needs ImprovementNeeds Improvement/Proficient
Massachusetts Massachusetts High High School School Competency Competency Determination Determination (CD)(CD)Requirements: Requirements: Class of 2010Class of 2010
Meet local graduation requirements Attain scaled score of 220 or higher
on high school MCAS tests in ELA, mathematics, and STE (or equivalent on MCAS Alternate Assessment)
To earn diploma, a student must:
Efforts to Establish Efforts to Establish “Equivalence” “Equivalence”
of High School STE Testsof High School STE TestsThe High School STE tests:
are based on parallel test design and development processes
are based on performance level descriptors that are comparable in scope and rigor
have been designed to have similar psychometric properties
External Validation of Efforts External Validation of Efforts to Establish “Equivalence” of to Establish “Equivalence” of
STE TestsSTE TestsPerformance level descriptors used in
standard setting have been externally validated by Massachusetts teachers
Technical/psychometric properties analyzed by Dr. Ronald Hambleton, University of Massachusetts, Amherst
Test design and plans for standard setting endorsed by the MCAS National Technical Advisory Committee (TAC)
What is the Body of Work What is the Body of Work Procedure?Procedure?
Standard Setting method where panelists:
examine student work (actual responses to test questions) and performance level descriptors
make a judgment regarding the performance level to which the student work most closely corresponds.
Body of Work Body of Work FundamentalsFundamentals
Examine the student’s responses to multiple-choice questions
Examine the student’s responses to open-response questions
Judge the student’s knowledge and skills demonstrated relative to the PLDs
Panelists do not need to reach consensus on the classifications
Materials Used During Materials Used During Standards ValidationStandards Validation
Performance Level Descriptors• General• Content specific
Bodies of Student Work• Responses to constructed-response
questions• Multiple-choice summary sheet
Rating Forms
General MCAS General MCAS Performance Level Performance Level
DescriptorsDescriptorsNeeds Improvement Students at this level demonstrate partial understanding of subject matter and solve simple problems.
Proficient Students at this level demonstrate a solid understanding of challenging subject matter and solve a wide variety of problems.
AdvancedStudents at this level demonstrate a comprehensive and in-depth understanding of rigorous subject matter, and provide sophisticated solutions to complex problems.
Needs ImprovementOn MCAS, a student at this level
ProficientOn MCAS, a student at this level
AdvancedOn MCAS, a student at this level
Conceptual Understanding and Factual Knowledge
Demonstrates a partial understanding of some facts, concepts, principles, and theories
Uses basic scientific terms
Demonstrates a solid understanding of many facts, concepts, principles, and theories
Uses appropriate scientific terms
Demonstrates a comprehensive, in-depth understanding of many facts, concepts, principles, and theories
Applies scientific terms in an appropriate context
Scientific Process and Skills
Interprets simple data and creates generalized questions without necessarily using the scientific method
Identifies a problem to be solved
Uses most of the steps of the scientific method to design and interpret experiments
Finds solutions to a variety of problems
Designs and evaluates scientific experiments and generates full interpretations of data
Finds solutions to complex problems
Application/Synthesis
Makes simple predictions about a specific topic
Makes predictions based on information given
Justifies predictions in a general sense
Makes sophisticated predictions
Synthesizes a wide array of information from multiple sources
Applies knowledge to abstract or novel situations
GeneralGeneral STE Performance Level STE Performance Level DescriptorsDescriptors
What Next?What Next? Take the Introductory Physics test Discuss the Performance Level
Descriptors Complete the Item Map Complete training round Complete an evaluation form Complete individual ratings Receive feedback from first round of
ratings Discuss feedback and provide final ratings Complete final evaluation form
Ground RulesGround Rules Role of facilitator is to “facilitate” and keep
process on track Process solely focused on recommending
performance standards (cut scores) for MCAS MCAS performance level descriptors are
integral to process but are not up for debate Panelists’ recommendations are vital; however,
final cut scores determined by the MDOE Each panelist must be in attendance for the
duration of the process for his/her judgments to be considered
Each panelist must complete evaluation form at the end of the event
Cell phones off, please!
Agenda Agenda Monday, September 17Breakfast 8:00 am – 9:00 amPlenary 9:00 am – 10:30 am
Break 10:30 am – 10:45 amWork session 10:45 am –12:00 pmLunch 12:00 pm – 1:00 pmWork session 1:00 pm – 5:00 pm
Tuesday, September 18Breakfast 8:00 am – 9:00 amWork session 9:00 am – 12:00 pmLunch 12:00 pm – 12:45 pmWork session 12:45 pm – until completion
Questions?Questions?