Upload
piers-norton
View
219
Download
0
Embed Size (px)
Citation preview
Florida Assessments for Instruction in Reading (FAIR) – Using Scores
for Growth and Instruction
Dr. Barbara FoormanFlorida Center for Reading Research (FCRR)
Stuart GreenbergFlorida Department of Education
What is FAIR?
A K-2 assessment system administered to individual students 3 times a year, with electronic scoring, Adobe AIR version, and PMRN reports linked to instructional resources.
A 3-12 computer-based system where students take the assessments 3 times a year. Several tasks are adaptive. PMRN reports are available, linked to instructional resources. Printed toolkit available.
© 2011 Florida Department of Education 2
© 2009 Florida Department of Education 3
The K-2 “Big Picture” Map
Broad Screen/Progress Monitoring Tool (BS/PMT) “All” students
• Letter Naming & Sounds• Phonemic Awareness • Word Reading
Broad Diagnostic Inventory (BDI) “All” students“Some” students for vocabulary
• Listening Comprehension• Reading Comprehension• Vocabulary • Spelling (2nd grade only)
Targeted Diagnostic Inventory(TDI) “Some” students; some tasks
• K = 9 tasks• 1st = 8 tasks• 2nd = 6 tasks
Ongoing Progress Monitoring(OPM)“Some” students
•K – 2 = TDI tasks•1 – 2 = ORF
3
K-2 Targeted Diagnostic Inventory (TDI) MapKindergarten • Print Awareness
• Letter name and sound knowledge• Phoneme Blending• Phoneme Deletion Word Parts/Initial• Letter Sound Connection Initial• Letter Sound Connection Final• Word Building –Initial Consonants• Word Building –Final Consonants• Word Building –Medial Vowels
First Grade • Letter Sound Knowledge • Phoneme Blending• Phoneme Deletion Initial• Phoneme Deletion Final• Word Building –Consonants• Word Building –Vowels• Word Building –CVC /CVCe• Word Building –Blends
Second Grade • Phoneme Deletion Initial• Phoneme Deletion Final• Word Building –Consonants• Word Building –CVC /CVCe •Word Building –Blends & Vowels• Multisyllabic Word Reading
The K – 2 “Score” Map
BS/PMT PRS = Probability of Reading Success
BDI LC = Listening Comprehension Total questions correct (implicit/explicit)
RC = Reading Comprehension Total questions correct (implicit/explicit), Fluency, Percent Accuracy Target Passage
VOC = Vocabulary Percentile Rank
SPL = Spelling Percentile Rank
TDI ME = Meets Expectations
BE = Below Expectations
OPM ORF = Adjusted Fluency
OPM TDI Tasks = ME or BE and Raw Score5
6
Target RC Passages for Grades 1 and 2 (BDI)
Florida Center for Reading Research
Grade 1 PRS Chart (2010-2011)AP1 PRS AP2 PRS AP3 PRS
0 0.11 0 0.01 0 .011 0.14 1 0.02 1 .022 0.20 2 0.03 2 .033 0.27 3 0.05 3 .064 0.36 4 0.09 4 .125 0.46 5 0.17 5 .206 0.56 6 0.28 6 .337 0.66 7 0.44 7 .498 0.75 8 0.61 8 .659 0.82 9 0.76 9 .79
10 0.86 10 0.86 10 .88
Probability of Reading Success
Look for trends within grade levels, within teachers, and across grades.
Let’s look across grades in our sample data.
What do you notice when you compare across grades?
85% & above
(Green)
16%-84%
(Yellow)
15% & below
(Red)
K 28% 61% 11%
1st 36% 56% 8%
2nd 25% 70% 5%
Instructional Information from FAIR
In grades K-2:
1.Are my students in the green zone (>.85) on the Broad Screen? At or above 50%ile in vocabulary? On the target LC or RC passage?
2.If not, look at performance on the targeted diagnostic inventory tasks to see which skills students have mastered (>80%). Teach to those they haven’t mastered.
© 2011 Florida Department of Education 9
Student Score Detail Box (K-2)
©2011 Florida Center for Reading Research
Excellent report to include in a student’s cumulative folder
Questions to answer using the School Grade Summary Report
What is the distribution of scores in a particular grade level? The School Status told us the Median Score for
Vocabulary was the 24th percentile, but now I see the full distribution and can see that the majority of my students fell within the 11th – 30th percentile range.
This tells me that overall, the First Grade group is below average in their general vocabulary knowledge
I would want to see if this is true in Kindergarten and Second Grade as well (on the School Status Report)
Kindergarten Median Vocabulary = 29th percentile Second Grade Median Vocabulary = 41st
I might look to see how the Core Reading Program addresses vocabulary and if we have any additional programs addressing vocabulary (maybe we are using something different or additional at second grade?)
© 2011 Florida Department of Education 12
Broad Screen/Progress Monitoring Tool
Reading Comprehension Task
(3 Times a Year)
Targeted Diagnostic Inventory
Maze & Word Analysis TasksDiagnostic
Toolkit(As Needed)
OngoingProgress
Monitoring(As Needed)
If necessary
Grades 3-12 Assessments Model
RC Screen Helps us identify students who may not be able to meet the
grade level literacy standards at the end of the year as assessed by the FCAT without additional targeted literacy instruction.
Mazes Helps us determine whether a student has more
fundamental problems in the area of text reading efficiency and low level reading comprehension. Relevant for students below a 6th grade reading level.
Word Analysis Helps us learn more about a student's fundamental literacy
skills--particularly those required to decode unfamiliar words and read and write accurately.
Purpose of Each 3-12 AssessmentPurpose of Each 3-12 Assessment
How is the student placed into the first passage/item?
Task Placement Rules
Reading Comprehension - Adaptive
•For AP 1, the first passage the student receives is determined by: • Grade level and prior year FCAT (if available) • If no FCAT, students placed into a specific grade-level passage
• All 3rd grade students are placed into the same initial passage
•For AP 2 and 3, the first passage is based on students’ final ability score from the prior Assessment Period (AP).
Maze – Not adaptive
Two predetermined passages based on grade level and assessment period (AP).
WA - Adaptive • AP 1-3 starts with predetermined set of 5 words based on grade level. Student performance on this first set of 5 words determines the next words the student receives.
• 5-30 words given at each assessment period based on ability.
How is the student placed into subsequent passages? Based on the difficulty of the questions the student
answers correctly on the first passage, the student will then be given a harder or easier passage for their next passage. Difficulty of an item is determined using Item Response
Theory (IRT).
Because of this, the raw score of 7/9 for Student A and 7/9 for Student B, when reading the same passage, does not mean they will have the same converted scores.
16
The 3-12 “Big Picture” Map
Type of Assessment Name of Assessment
Broad Screen/Progress Monitoring Tool (BS/PMT) – Appropriate for ‘All’ students
• Reading Comprehension (RC)
Targeted Diagnostic Inventory(TDI) – “Some” students
• Maze• Word Analysis (WA)
Ongoing Progress Monitoring (OPM) – “Some” students
• Maze• ORF• RC
Informal Diagnostic Toolkit(Toolkit) – “Some” students
• Phonics Inventory• Academic Word Inventory•Scaffolded Discussion Templates
17
The 3-12 “Score” Map
Reading Comprehension - BS/PMT
FCAT Success Probability (FSP) Color- coded
Percentile Standard ScoreAbility Score and Ability Range FCAT Reporting Categories
Maze - TDI Percentile Standard Score Adjusted Maze Score
Word Analysis - TDI Percentile Standard Score Ability Score (WAAS)
OPM RC – Ability Score, Ability Range, Reporting Categories Maze – Adjusted Maze Score ORF (3rd – 5th) Adjusted Fluency Score
Questions to answer using the School Grade Summary Report
What is the distribution of the ‘yellow’ zone (16-84%)? Say that 38% of Seventh Graders fall in the yellow
zone. Looking at a particular distribution, are more of them at the high end or the low end of the zone?
Say that it is a relatively even distribution across the zone with between 10-12 students falling in each section of the zone until the last two sections, 65-74% chance has 26 students and 75%-84% chance has 45 students.
We want to see the skew fall to the right with more students closer to the high success zone of 85% chance or higher.
Assessment/Curriculum Decision Tree for Reading Improvement Using FAIR’s RC’s FSP and Maze and Word Analysis Percentile Ranks.
Common Questions
What score types should be used to measure growth? For RTI decision-making?
Are scores dropping from AP1 to AP2 this year? Is FAIR norm-referenced? Criterion-referenced? Why does FAIR have adaptive tests (RC and
WA) when we want grade-level information? Why doesn’t FAIR report at the benchmark level? How do we get instructional information out of
FAIR?
© 2011 Florida Department of Education 20
Measuring Growth on FAIR
In K-2, use PRS descriptively : E.g., establish % of students to be in green zone at each AP.
In 3-12, use the reading comprehension ability score (RCAS) because it has the same metric across time (like using FCAT’s DSS rather than SS for growth). FAIR’s RCAS has a mean of 500; SD=100; range=150-1000.
In 3-12, do not use FSP to measure growth in reading comprehension ability because FSP includes prior FCAT but RCAS may change over time. For students without a matched FCAT score, FSP is based solely on RCAS. When a student’s FCAT score becomes available in the PMRN, the FSP may look unexpectedly high or low compared to the previous AP’s FSP.
Why we use Ability Scores
Tom Brady Raw Standard Percentile AbilityAP1 12 100 50th 500AP2 17 100 50th 550AP3 20 100 50th 600
Reading Comprehension
Mazes Word Analysis
AP Score PM score AP Score PM score AP Score PM score
student score
student score
%ile & SS
RCAS
Percentilerank
Percentilerank
FSP
Adj. Maze
SS SS
WAAS
PM = progress monitoring; SS = standard score
3-12 Instructional Toolkit
Phonics screener and academic word list can be used to monitor progress.
© 2009 Florida Department of Education 24
Progress Monitoring in K-2
G2 Broad Screen in G2 is same across each AP--# words read correctly in 45 sec. Thus, monitor progress in timed decoding.
Equated OPM passages in G1-G5 allow for ORF progress monitoring (with 40th %ile at each grade provided at www.fcrr.org as criterion.
Percent of G1 & G2 students who read target passage with comprehension.
Percent of students achieving 80% mastery on TDI tasks.
© 2011 Florida Department of Education 25
AP1 AP2 Scores
Mean FSP and RCAS increased slightly in grades 3-8 and stayed the same in grades 9-10 at the state level.
Standard deviations are fairly large, explaining why some are reporting drops.
In grades 3-8 the majority of students tested were at FCAT level 3 or above. This reflects FAIR norms for FCAT 2.0. Slightly lower growth from 09-10SY to 10-11SY likely due to demands of FCAT 2.0.
© 2011 Florida Department of Education 26
Norm- vs. Criterion-Referenced
The K-2 Broad Screen predicts to a nationally normed test; percentiles for Vocabulary and G2 Spelling are based on FL grade-level norms.
The K-2 diagnostic inventory is criterion-referenced (80% of skills mastered).
The 3-12 Broad Screen predicts to a criterion—passing FCAT. (i.e., the FSP)
Percentiles for the 3-12 Broad Screen, Maze, and WA are based on FL grade-level norms.
© 2011 Florida Department of Education 27
Value of Computer-Adaptive Tests
Provides more reliable & quicker assessment of student ability than a traditional test, because it creates a unique test tailored to the individual student’s ability.
Provide more reliable assessments particularly for students at the extremes of ability (extremely low ability or extremely high ability).
Grade-level percentiles are currently provided; Grade Equivalent scores will be provided next year.
© 2011 Florida Department of Education 28
Benchmark Conundrum
Benchmark tests rarely have enough items to be reliable at the benchmark level. Besides, teaching to benchmarks (e.g., “the student will use context clues to determine meanings of unfamiliar words”) results in fragmented skills. Teach to the standard(s) (e.g., “The student uses multiple strategies to develop grade appropriate vocabulary). Assess at aggregate levels (e.g., Reporting Categories), if CFA show categories are valid.
© 2011 Florida Department of Education 29
FCAT 2.0 Reporting Categories
Reporting Category 1: Vocabulary
Reporting Category 2: Reading Application
Reporting Category 3: Literary Analysis- Fiction/Nonfiction
Reporting Category 4: Informational Text/ Research
Process
FCAT 2.0: Benchmarks x Grade
Category 1 Category 2 Category 3 Category 4 Total
Grade 3 4 6 3 1 144 4 6 3 1 145 4 6 3 2 15 6 4 5 3 2 147 4 5 3 2 148 4 5 3 2 14 9/10 4 5 3 2 14
© 2011 Florida Department of Education 31
Possible Benchmark Solutions
Stop-gap: start each students with grade-level passage. Provide % correct on Reporting Categories. Then continue to current adaptive system to obtain reliable, valid FSP and RCAS.
For the future: Align FAIR to the Common Core. Develop grade-level
CAT that is item adaptive. Challenges: Dimensionality; multi-dimensional IRT;
testlet effects.
© 2011 Florida Department of Education 32
Scarborough (2002)
Dimensions of Word Knowledge Knowledge of word’s spoken form (pronunciation) Written form (spelling) Grammatical behavior (syntactic/morphological
features) Co-locational behavior (occurs with other words) Frequency (orally and in print) Stylistic register (e.g., academic language;
informal) Conceptual meaning (antonyms, synonyms) Association with other words (inter-relatedness)
Nation (1990) in Nagy & Scott (2000)
35
Vocabulary Knowledge Task
Word Meanings
Text
• FCRR has developed a 3- min. adaptive vocabulary sentence task that strongly predicts RC.
• Students pick 1 of 3 morphologically related words to best complete a sentence.
• Teachers can use percentiles, standard scores, & ability scores to drive vocabulary instruction
New Vocabulary Knowledge Task
© 2011 Florida Department of Education 37
In Support of FAIR
Other Resources: PMRN Help Desk and Just Read, Florida! staff
available 8:00-5:00 Monday through Friday FAIR technical tips, Users’ Guides, and frequently
asked questions are available at the FCRR website: www.fcrr.org/fair/ and the Just Read, Florida! website: www.justreadflorida.com/instrreading.asp
LEaRN (Literacy Essentials and Reading Network) www.justreadflorida.com/LEaRN: FAIR resources for teachers, coaches, and principals which include training videos and clips of master trainers administering FAIR to students.
For more detailed information regarding the PMRN: The PMRN User’s Guides located at:
www.fcrr.org/pmrn/userguides.shtm There is a section that provides an Overview of Reports and a
section that provides an Explanation of Score Types
Complete Power Points can be found on the PMRN website: www.fcrr.org/pmrn/ Topics include:
K-2 Data Entry K-2 Electronic Scoring Tool 3-12 Web-based Assessment Module (WAM) School Level Users 1, 2, 3
Phone: 850-644-0931 or 866-471-5019 Email: [email protected]
In summary, remember…
FAIR was designed to inform instruction. FAIR data are just one part of the puzzle. Teachers
bring information from daily instruction that provides additional, important information.
As FCRR gains more experience with the data and collects data statewide, we will be able to provide guidelines on what are ‘typical’ scores at each grade level or what is a ‘good’ percentage to have in the high success zone (green zone).
We will make comparisons to the State Mean scores and percentages at the end of each AP
New 3-12 Tech Manual available at end of summer, 2011.
41
Thank you for your time and attention!
Any Questions?