Academic Rigor: Where Are We Now
Looking at Student Reading Achievement and Increasing Rigor
Using Grade 4 NAEP Item Maps and Percentile Graphs
Prepared by Jeanne Foy
Alaska State NAEP Coordinator
NAEP: the Common Measurement of Student Achievement among States
The National Assessment of Educational Progress is called the “gold standard of assessment”
Created in 1969; the No Child Left Behind Act (NCLB) mandated state participation in NAEP reading and math every other year
Alaska has NAEP data for 2003, 2005, 2007, 2009, and 2011
Purpose of This Presentation
NAEP reading item maps and percentile graphs can help to frame the question of where Alaska is now in terms of student achievement and what would need to be done to raise student achievement.
Alaska is revising reading, writing, and math standards against the background of a national push to raise the rigor of standards.
Alaska’s Revised Standards Many states have adopted the Common
Core standards for English language arts and mathematics. Alaska has developed proposed standards to equal the Common Core standards in rigor: they will be posted at http://www.eed.state.ak.us/regs/
The next slide shows the rigor intended to be reached by the new standards
CC Standards Designed to Be Internationally Competitive The standards have been informed by the best
available evidence and the highest state standards across the country and globe . . .
These standards are designed to ensure that students graduating from high school are prepared to go to college or enter the workforce and that parents, teachers, and students have a clear understanding of what is expected of them.
The standards are benchmarked to international standards to guarantee that our students are competitive in the emerging global marketplace.
http://www.corestandards.org/frequently-asked-questions
NAEP Resources Set Stage for Discussions about Higher Rigor
NAEP percentile box graphs show current academic achievement gaps by ethnic groups
NAEP item maps show what students are achieving now
A Disclaimer! The issues of increasing academic
performance and closing gaps are complicated
This presentation is designed to show current student achievement in Alaska and serve as a springboard for discussion in considering the challenges of increasing academic rigor and closing achievement gaps among ethnic groups
How Percentiles Represent Student Achievmeent
A percentile is a value on a scale of one hundred that indicates the percent of a distribution that is below it.
For example, if a student scores at the 84th percentile on a test, he scored higher than 84% of the students who took the test
Understanding Box Plots
Box plots allow data to be explored by showing the median value of a set of data and the 25th and 75th percentiles
The middle line in the box shows the median value, the reading score at which half the students performed below and half the students performed above
Students who scored at the 75th percentile performed better than 75% of students who took the NAEP; students who scored at the 25th percentile performed better than 25% of students who took the NAEP
The range of the distribution is shown on the box plot by showing a minimum and maximum value (the 10th and 90th percentiles for the NAEP box plots)
Alaska Gr. 4 Reading Percentile Box Plots for 2011 NAEP
Note where the median (the dot in the middle of the box) is for each group in relation to the other groups
Note where the other data points are in relation to the other groups
What are your observations? What questions or issues do
your observations raise?
NAEP Item Maps Illustrate What a Reading Score Represents
Item maps help to illustrate the knowledge and skills demonstrated by students performing at different scale scores; item maps show questions students answered correctly at different scale scores
Item maps combined with percentile box charts help to illustrate the knowledge and skills demonstrated by students performing at different scale scores
For instance, the item or question description at 211 on the scale represents the types of items grade 4 students who scored a 211 on NAEP answered correctly; item descriptions in blue represent items that are available to the public
Looking at the actual question for score 211 helps show what students can do in a concrete way.
The score of 211 is close to the 75th percentile for American Indian/Alaska Native students. American Indian/Alaska Native students scoring at the 75th percentile would probably have answered this question correctly. This question is shown on the next slide.
211 Recognize specific information in expository event—Locate/recall
Question*:What is the main purpose of the article?A. To describe the variety of bees and what bees doB. To explain the ways pollen is used by beesC. To show the way bees communicate with each otherD. To show what different species of bees look like
Nationally, 76% of students chose A (correct answer). In Alaska, 66% chose A, 17% chose B, 11% chose D.
*All questions in this presentation are over the nonfiction reading passage, What’s the Buzz. The proposed English/Language Arts standards emphasize the importance of reading nonfiction.
Each NAEP Question Has a Cognitive Target There are three targets--locate/recall,
integrate/interpret, and critique/evaluate.
Locate/recall description: Identify textually explicit information and
make simple inferences within and across texts, such as:
Definitions Facts Supporting details
What Skills Are Required to Answer the Question What is text complexity of the passage for
fourth grade students? What skills do students demonstrate when
they answer the question correctly? Why would students choose the wrong
answer? Consider the percentage who chose the wrong
answers. What might lead students to choose those
answers?
Consider Another Locate/Recall Question
The question at 239 is between the 50th and 75th percentile for White students. The question is on the next slide.
239 Recognize specific detail of supporting information—locate/recall
QuestionAccording to the article, what can animals of the same species do?A Travel in groups over long distancesB Live together in homes such as hivesC Mate with each other and give birthD Find food for their young
Nationally, 63% of students chose C (correct answer). In Alaska, 57% chose C, 10% chose A, 21% chose B, and 12% chose D.
How Do the Two Locate/Recall Questions Differ
What makes one question easier than the other? What skills are needed for students to answer the more difficult question?
What Does Item at Upper End of Item Map Show
The question at 309 is located above the 90th percentile for any ethnic group. The question is shown on the next slide.
309 Use specific information to describe and explain a process—integrate/interpret
QuestionUsing information from the article,
explain what pollination is and how it happens.
Note: the score of 309 is for students who scored at the full comprehension level according to the scoring guide, which is presented in the following slides.
Integrate/Interpret
Make complex inferences within and across texts to describe problem and solution, cause and effect: Compare or connect ideas, problems, or
situations. Determine unstated assumptions in an
argument. Describe how an author uses literary
devices and text features.
Item: Use specific information to describe and explain a process—integrate/interpret
Consider text complexity of reading passages.
How difficult do you think this item is?
How do you think grade 4 students should perform on this item?
Scoring Guide for Full Comprehension
Responses at this level use information from the article to explain what pollination is and how it happens.• Pollination is the first step of making seeds. This happens when the pollen from the stamen reaches the pistil.• Pollination is the way that plants reproduce. The way it happens is that the bees carry the pollen and it drops on the female part of the flower and it reproduces.
Scoring Guide for Partial Comprehension
a) Responses at this level use information from the article to explain what pollination is, but they do not explain how it happens.• It reproduces and makes new flowers.• Pollination is the first step in making seeds and it happens right beside the pistil.
OR
b) Responses use information from the article to explain how pollination happens, but they do not explain what pollination is. • When bees get pollen and they go to another plant and that gets the pollen from the other plant so that's what pollination is.• Pollination is the movement of pollen from one flower to another. Pollination happens when one bee rubs itself against a flower and moves to another one.
Scoring Guide for Little or No Comprehension
Responses at this level provide incorrect information, irrelevant details, or personal opinions. Responses may simply repeat the question.• First the bees take pollen to the hive, then turn it into honey, then the bears eat it. • Bees get pollen on them.The word "pollen" can be taken to mean "pollination."The words "seeds" and "nectar" are not given credit for "pollen."
What Skills Do Students Need to Demonstrate Full Comprehension?
Examples of full comprehension
What Is Going on with Little Comprehension?
Examples of little or no comprehension
What Does Data Suggest about Student Skills?
In Alaska, 11% got full comprehension, 32% got partial, 50% had little or no comprehension.
National performance data
Questions to Consider
What does data show about the current achievement gaps?
What are the challenges for increasing academic rigor for all students?
What skills do students need to increase their achievement as shown by questions on the NAEP item maps?
Additional Resources
The Alaska-created “NAEP Assessment Toolbox for Teachers” show how NAEP questions can be used instructionally
http://www.eed.state.ak.us/tls/assessment/naep.html
NAEP questions for a variety of subjects can be found at
http://nces.ed.gov/nationsreportcard/itmrlsx/default.aspx