Key Education Indicators (KEI): Making Sense of NAEP Contextual Variables

Alan Ginsburg and Marshall S. SmithDecember 6, 2013

General goals

• Improve utility and coherence of the NAEP background variables.

• Examine current structure and list of variables – suggest interim steps to improve list.

• Suggest theoretically and empirically based framework for indicators. Purpose: create a stable, flexible approach to monitor and better understand trends in NAEP data.

• Create indicators that combine policy and/or practice related combinations of variables that deserve tracking over time.

2

Why key education indicators (KEI’s)?• KEI’s are statistics that regularly measure the health of

important conditions that may influence the outcomes of the education system over time.

• KEI’s are part of a balance- scorecard approach focused on fairness and quality of education to complement NAEP assessment-based outcome indicators.

• KEI’s make sense of dozens of NAEP contextual questions: bring together currently diverse context variables around research-based education performance criteria.

3

Why KEI’s for NAEP? We already have Condition of Education• NAEP is unique

– Gets inside schools with combination of perspectives (students, teachers, principals).

– Links schools with reports of out-of-school conditions affecting learning.

– Questions embedded in a national assessment – respondents may take questions more seriously and readers better understand NAEP results.

– Supports disaggregated information: • Student SES and race/ethnicity, classroom and school variation.• State and urban district breakouts.

– Makes possible two or three views (4th, 8th, 12th) of students, teachers, principals.

– Provides cross-sectional and possible cohort changes for indicators as well as achievement scores. 4

Current contextual variables• 1441 8th grade math variables. Vast majority not been used in past two

years. See list of variables mentioning computers for 4th grade reading. (examples on following slides)

• Now unused variables might be important to use (teacher perceptions of other teachers = measure of school mindset).

• Need some core variables fixed or improved -- e.g. current measures of school SES are weak and compromised. Individual SES not clearly defined.

• Full list cumbersome, discourages some researchers - suggest that researchers be able to select from a core list of currently live background variables to enable measuring differences over time. Full list of variables would be accessed separately.

• KEI’s would provide a framework to help identify, organize, and consistently measure a set of important context variables.

5

Avoid data overload: Focus on currently available data on computers and instruction

6

From variables to indicators

• NRC developing national indicators --very few for each sector. • NSF developing STEM indicators – focused on part of

education sector.• NAGB unique opportunity to link indicators regularly over

time with students, teachers, classrooms, schools, achievement results.

• Propose, as an example, 4 evidence-based composite indicators formed from a total of 5 student, 6 teacher and 3 school variables.

• Eventually expert teams would need to develop indicators – also SES team recommendation.

7

Example: evidence-based KEI’s: student factors • Student Factors: (basic variables: ethnicity, gender, ELL,

disability) + (5 indicators)– SES (Have expert team create SES KEI’s as recommended

by study committee)– Home Educational Climate (parent support, place to study,

talks with, not to child etc.)– Preschool Experiences (one, two years of preschool,

parent reads to child, sets boundaries)– Student motivation (effort, hard work more important

than luck, engage with school, likes and goes to school)– After-school education opportunities (formal and informal)

8

Example of evidence based KEI’s: school quality factors• School Quality (basic variables: place, size, type, composition) + 6 indicators

(KEI)– Teacher quality (student view of quality, education degree in field,

experience, dispositions &mindset)– Teacher professionalism (Seeks help to improve, supports other

teachers, seeks growth year after year, enjoys work, engaged in professional networks.)

– School climate (excessive student absenteeism, school safety, teacher expectations, teachers support each other, principal trusted, mindset)

– Effectively implement strong standards and curriculum. – School effectively uses technology to teach (access, use, quality)– Continuous improvement throughout (use formative assessment, pd

focused on improving, admin processes [e.g. HR], etc.)

9

Examples of ways to create KEI’s• Indicators can be single or combinations of multiple

variables. Some basic information such as ELL would probably be a single variable. Most indicators would require combining multiple variables.

• Variables should be combined based on theory –the SES report, for example.

• Possibly by using regression / IRT scaling to create weights for each of a set of variables or weights may be reached through theory or judgment.

• Or by creating composite categories indicating, for example, highly qualified teacher, as measured by having achieved four of six measured attributes.

• Or in other ways. 10

Examples of possible KEI’s

• Teacher Quality• School Climate for Learning• Student Motivation• Technology• Socio-Economic Status (SES)

11

Three-component composite indicator: school climate for learning

School Climate for Learning School Climate for Learning

12

Student Excessive Absenteeism

•More favorable•Less favorable

Student Excessive Absenteeism

•More favorable•Less favorable

Teacher Student Expectations

•More favorable •Less favorable

Teacher Student Expectations

•More favorable •Less favorable

Student Misbehavior

•More favorable •Less favorable

Student Misbehavior

•More favorable •Less favorable

Composite Composite

Composite•All more favorable

Composite•All more favorable

Composite•Mixed favorable

Composite•Mixed favorable

Composite•All less favorable

Composite•All less favorable

School-climate composite: student attendance (sub-indicator 1)

13

School-climate composite: teacher expectations (sub-indicator 2)

14

School-climate composite: student misbehavior (sub-indicator 3)

15

School climate: two variable composite indicator of highly favorable and highly unfavorable school climate, 2003

16

School climate: three-variable composite indicator of very high school climate

• Three-variable composite indicator– 39% of students in 2003 had 0-2 days absent the

prior month, in schools with no more than minor discipline problems, and a math teacher with very positive expectations for students.

Note: The three-variable composite indicator of teacher quality would ideally be displayed as above by school-poverty concentrations, but this is beyond the Data Explorer capacity.

17

Student motivation indicator: ex. science engagement variables

18

• Key recommendations– 1. Family income, other indicators of home

possessions and resources, parental education and occupational status components of SES measure.

– 2. Neighborhood and school SES could be used to construct an expanded SES measure.

– 3. Composite measures have many advantages. – 4. Recommendation: should form expert panel to

develop an SES composite measure.

19

SES indicator: expert panel report

Summary: Proposed steps for NAEP1. Make immediate improvements to NAEP Data

Explorer: reduce number of variables dramatically. Recent, useful variables in a prominent file. Old, redundant, useless, in another file.

2. Develop indicator framework and initiate expert teams in several areas to develop individual and composite key education indicators and accompanying NAEP contextual items.

20

More detail on next steps for KEI’S 1. Review, revise, improve proposed KEI’s framework and

indicators.2. Create SES indicator – based on SES expert team.3. Get expertise in each field to put variables and current

and proposed questions into new KEI’s framework -- use psychometricians to build composite indicators. Start with a couple of areas.

4. Field test the new variables and indicators. 5. Build repository of articles, publications, etc. that used

the NAEP variables and indicators for scholars to use.

21