Academic Rigor: Where Are We Now

Looking at Student Reading Achievement and Increasing Rigor

Using Grade 4 NAEP Item Maps and Percentile Graphs

Prepared by Jeanne Foy

Alaska State NAEP Coordinator

NAEP: the Common Measurement of Student Achievement among States

The National Assessment of Educational Progress is called the “gold standard of assessment”

Created in 1969; the No Child Left Behind Act (NCLB) mandated state participation in NAEP reading and math every other year

Alaska has NAEP data for 2003, 2005, 2007, 2009, and 2011

Purpose of This Presentation

NAEP reading item maps and percentile graphs can help to frame the question of where Alaska is now in terms of student achievement and what would need to be done to raise student achievement.

Alaska is revising reading, writing, and math standards against the background of a national push to raise the rigor of standards.

Alaska’s Revised Standards Many states have adopted the Common

Core standards for English language arts and mathematics. Alaska has developed proposed standards to equal the Common Core standards in rigor: they will be posted at http://www.eed.state.ak.us/regs/

The next slide shows the rigor intended to be reached by the new standards

CC Standards Designed to Be Internationally Competitive The standards have been informed by the best

available evidence and the highest state standards across the country and globe . . .

These standards are designed to ensure that students graduating from high school are prepared to go to college or enter the workforce and that parents, teachers, and students have a clear understanding of what is expected of them.

The standards are benchmarked to international standards to guarantee that our students are competitive in the emerging global marketplace.

http://www.corestandards.org/frequently-asked-questions

NAEP Resources Set Stage for Discussions about Higher Rigor

NAEP percentile box graphs show current academic achievement gaps by ethnic groups

NAEP item maps show what students are achieving now

A Disclaimer! The issues of increasing academic

performance and closing gaps are complicated

This presentation is designed to show current student achievement in Alaska and serve as a springboard for discussion in considering the challenges of increasing academic rigor and closing achievement gaps among ethnic groups

How Percentiles Represent Student Achievmeent

A percentile is a value on a scale of one hundred that indicates the percent of a distribution that is below it.

For example, if a student scores at the 84th percentile on a test, he scored higher than 84% of the students who took the test

Understanding Box Plots

Box plots allow data to be explored by showing the median value of a set of data and the 25th and 75th percentiles

The middle line in the box shows the median value, the reading score at which half the students performed below and half the students performed above

Students who scored at the 75th percentile performed better than 75% of students who took the NAEP; students who scored at the 25th percentile performed better than 25% of students who took the NAEP

The range of the distribution is shown on the box plot by showing a minimum and maximum value (the 10th and 90th percentiles for the NAEP box plots)

Alaska Gr. 4 Reading Percentile Box Plots for 2011 NAEP

Note where the median (the dot in the middle of the box) is for each group in relation to the other groups

Note where the other data points are in relation to the other groups

What are your observations? What questions or issues do

your observations raise?

NAEP Item Maps Illustrate What a Reading Score Represents

Item maps help to illustrate the knowledge and skills demonstrated by students performing at different scale scores; item maps show questions students answered correctly at different scale scores

Item maps combined with percentile box charts help to illustrate the knowledge and skills demonstrated by students performing at different scale scores

For instance, the item or question description at 211 on the scale represents the types of items grade 4 students who scored a 211 on NAEP answered correctly; item descriptions in blue represent items that are available to the public

Looking at the actual question for score 211 helps show what students can do in a concrete way.

The score of 211 is close to the 75th percentile for American Indian/Alaska Native students. American Indian/Alaska Native students scoring at the 75th percentile would probably have answered this question correctly. This question is shown on the next slide.

211 Recognize specific information in expository event—Locate/recall

Question*:What is the main purpose of the article?A. To describe the variety of bees and what bees doB. To explain the ways pollen is used by beesC. To show the way bees communicate with each otherD. To show what different species of bees look like

Nationally, 76% of students chose A (correct answer). In Alaska, 66% chose A, 17% chose B, 11% chose D.

*All questions in this presentation are over the nonfiction reading passage, What’s the Buzz. The proposed English/Language Arts standards emphasize the importance of reading nonfiction.

Each NAEP Question Has a Cognitive Target There are three targets--locate/recall,

integrate/interpret, and critique/evaluate.

Locate/recall description: Identify textually explicit information and

make simple inferences within and across texts, such as:

Definitions Facts Supporting details

What Skills Are Required to Answer the Question What is text complexity of the passage for

fourth grade students? What skills do students demonstrate when

they answer the question correctly? Why would students choose the wrong

answer? Consider the percentage who chose the wrong

answers. What might lead students to choose those

answers?

Consider Another Locate/Recall Question

The question at 239 is between the 50th and 75th percentile for White students. The question is on the next slide.

239 Recognize specific detail of supporting information—locate/recall

QuestionAccording to the article, what can animals of the same species do?A Travel in groups over long distancesB Live together in homes such as hivesC Mate with each other and give birthD Find food for their young

Nationally, 63% of students chose C (correct answer). In Alaska, 57% chose C, 10% chose A, 21% chose B, and 12% chose D.

How Do the Two Locate/Recall Questions Differ

What makes one question easier than the other? What skills are needed for students to answer the more difficult question?

What Does Item at Upper End of Item Map Show

The question at 309 is located above the 90th percentile for any ethnic group. The question is shown on the next slide.

309 Use specific information to describe and explain a process—integrate/interpret

QuestionUsing information from the article,

explain what pollination is and how it happens.

Note: the score of 309 is for students who scored at the full comprehension level according to the scoring guide, which is presented in the following slides.

Integrate/Interpret

Make complex inferences within and across texts to describe problem and solution, cause and effect: Compare or connect ideas, problems, or

situations. Determine unstated assumptions in an

argument. Describe how an author uses literary

devices and text features.

Item: Use specific information to describe and explain a process—integrate/interpret

Consider text complexity of reading passages.

How difficult do you think this item is?

How do you think grade 4 students should perform on this item?

Scoring Guide for Full Comprehension

Responses at this level use information from the article to explain what pollination is and how it happens.•  Pollination is the first step of making seeds. This happens when the pollen from the stamen reaches the pistil.•  Pollination is the way that plants reproduce. The way it happens is that the bees carry the pollen and it drops on the female part of the flower and it reproduces.

Scoring Guide for Partial Comprehension

a) Responses at this level use information from the article to explain what pollination is, but they do not explain how it happens.•  It reproduces and makes new flowers.•  Pollination is the first step in making seeds and it happens right beside the pistil.

OR

b) Responses use information from the article to explain how pollination happens, but they do not explain what pollination is. •  When bees get pollen and they go to another plant and that gets the pollen from the other plant so that's what pollination is.•  Pollination is the movement of pollen from one flower to another. Pollination happens when one bee rubs itself against a flower and moves to another one.

Scoring Guide for Little or No Comprehension

Responses at this level provide incorrect information, irrelevant details, or personal opinions. Responses may simply repeat the question.•  First the bees take pollen to the hive, then turn it into honey, then the bears eat it. •  Bees get pollen on them.The word "pollen" can be taken to mean "pollination."The words "seeds" and "nectar" are not given credit for "pollen."

What Skills Do Students Need to Demonstrate Full Comprehension?

Examples of full comprehension

What Is Going on with Little Comprehension?

Examples of little or no comprehension

What Does Data Suggest about Student Skills?

In Alaska, 11% got full comprehension, 32% got partial, 50% had little or no comprehension.

National performance data

Questions to Consider

What does data show about the current achievement gaps?

What are the challenges for increasing academic rigor for all students?

What skills do students need to increase their achievement as shown by questions on the NAEP item maps?

Additional Resources

The Alaska-created “NAEP Assessment Toolbox for Teachers” show how NAEP questions can be used instructionally

http://www.eed.state.ak.us/tls/assessment/naep.html

NAEP questions for a variety of subjects can be found at

http://nces.ed.gov/nationsreportcard/itmrlsx/default.aspx