MAPP as an Accountability Assessment Brent Bridgeman Educational Testing Service Center for Validity Research

MAPP as an Accountability Assessment

Brent Bridgeman

Educational Testing Service

Center for Validity Research

Confidential and Proprietary. Copyright ©2008 Educational Testing Service.

2

Overview

• What is MAPP

• Content validity of MAPP

• MAPP norms

• Reliability and relationships among MAPP scores

• Relationship of MAPP to other measures

• Sensitivity to instruction

• MAPP in Europe


3

Measure of Academic Proficiency and Progress (MAPP™) and the Academic Profile

• MAPP was introduced in 2006 as a successor to the Academic Profile– MAPP was created to the same specifications as the Academic Profile

and assesses the same skills (reading, critical thinking, writing, and mathematics) and contexts (humanities, social sciences, and natural sciences)

– Scores on MAPP are equated to Academic Profile

– Two versions

• Standard form—108 questions in 2 hours—ok for individual scores

• Abbreviated form—36 questions in 40 minutes—matrix sample for group scores only

– Two formats—paper-and-pencil and online (proctored or non-proctored)

– Optional essay and option to add locally authored questions


4

Validity of the Measure of Academic Proficiency and Progress as an Accountability

Measure• Tests cannot be validated, but rather “the extent to which

the proposed uses and interpretations are plausible and appropriate” (Kane)

• MAPP has several uses– Counseling tool to identify students with skill deficits

– Track growth of individual students

– Track trends over time in skills of freshmen and seniors

– Curriculum improvement

– Serve as one component of a value added assessment, assessing general education core curriculum skills in critical thinking and writing


5

Content Validity--What Does MAPP Measure?

• “Academic skills developed through general education courses,” and NOT subject matter specifically taught in those courses

• No attempt to measure such areas as:– Graphic artist thinking critically about use of negative

space

– Structural engineer thinking critically about how to maximize both safety and efficiency in a bridge design


6



7



8

MAPP Content Validity

• Ultimately, content validity determination should be made locally--from the MAPP web site:

While the MAPP test is developed to measure a common core set of skills, judging the appropriateness of a test for a particular department or program is an important first step in considering the use of an outcomes assessment. Faculty members should review the test content and skill area coverage to determine whether it is consistent with your institution’s expectations.

• Hundreds of institutions have determined that MAPP assesses relevant general education outcomes


9

Does MAPP have Meaningful Norms?

• Banta is rightly concerned that norms not be based on small samples

• MAPP norms are based on over 375,000 students from more than 380 institutions

• MAPP norms are provided separately by Carnegie Classification, and institution may customize a peer group for comparison


10

Sample Norms


11

…


12

Construct Validity—Are the General Academic Skills in MAPP a Single Construct?

• Reliabilities of all four skills in the .80’s

• True correlations among skills:

Critical Thinking

Reading

Writing

Reading 1.0 Writing .86 .92 Math .76 .75 .75


13

Importance of the Skills Assessed by MAPP

• Core academic skills in critical reading (thinking), quantitative reasoning, and writing are strongly related to high levels of success in college and beyond

14

Relationship of Reading and Quantitative Reasoning Skills to College Success

• Define success as 3.5 or higher GPA over 4 years

(Bridgeman, B., Burton, N., & Pollack, J. [2008]. Predicting grades in college courses: A comparison of multiple regression and percent succeeding approaches. The Journal of College Admission, 199, 19-25)

0

10

20

30

40

50

60

70

80

90

100

1 2 3 4 5

SAT Level

% 3

.5 o

r H

igh

er

ENGLISH

S/M/E

SOC. SCI.

EDUC.

Note.-- SAT Levels: 1 = 200-800; 2 = 810-1000; 3 = 1010-1200; 4 = 1210-1400; 5 = 1410-1600 For students high school GPAs of 3.7+; each data point at least 150 students


15

Are the Critical Thinking Skills Assessed by MAPP Similar to SAT and ACT Scores?

• For 100 students from a “large Midwestern university” who had completed most of their general education requirements, correlation of Academic Profile (short form) with:– PSAT-V .61

– PSAT-M .46

– ACT-Composite .87

– ACT-COMP .62(Hendel, D. D. [1991]. Evidence of convergent and discriminant validity in three measures of college outcomes.

Educational and Psychological Measurement, 51, 351-357.)

• For 810 seniors from the University of Tennessee, correlation of Academic Profile with ACT-Composite: .71

(Pike, G. R. [1988]. A comparison of the College Outcome Measures Program [COMP] exam and the ETS Academic Profile. University of Tennessee technical report, ERIC ED 348 369).


16

Are the Critical Thinking Skills Assessed by MAPP Similar to SAT and ACT Scores? (con’t)

• For 1,823 seniors from 23 colleges

Correlations MAPP with SAT V+M:

Reading .72

Math .63

Writing .54

Critical thinking .59

TOTAL .77


17

Do High Correlations with SAT/ACT Scores Invalidate Value-Added Estimates?

• High within-college correlations are irrelevant for judgments of value added

• High across-college correlations (with college as the unit of analysis) indicate most, BUT NOT ALL, of the variance among colleges in outcomes can be predicted from pre-admission tests– With college as unit of analysis, MAPP correlation with SAT V+M

is .93

– Good news or bad news?

College A College B Freshman Senior Freshman Senior

30 50 30 35 20 40 20 25 10 30 10 15


18

Does a High Correlation of MAPP Scores and SAT/ACT Scores Imply that MAPP Could Not

be used for a Value-Added Assessment?

• If MAPP scores and SAT/ACT scores measure essentially the same construct, and if that construct is a fixed, inherited capability, then MAPP could not measure value added by any instructional program

• The second part of the above statement is clearly not true

– SAT scores increase substantially with increased education

• Scores of seniors substantially above scores of 7th and 8th grade students

• “the verbal and mathematical reasoning skills measured by the SAT may be enhanced to a measurable degree by long-term and intensive instruction” (Messick, S., & Jungeblut, A. [1981]. Time and method in coaching for the SAT. Psychological Bulletin, 89, 191-216).


19

Are Academic Profile Scores Related to College-Defined Success (Grades)?

• Mean Critical Thinking scores:– For students with GPA below 2.50 = 110 (n = 1145)

– For students with GPA above 3.49 = 117 (n = 837)

– SD = 6, so d = 1.2

• Similar findings for Reading, Writing, and Mathematics

(Marr, D. [1995]. Validity of the Academic Profile. Princeton, NJ: Educational Testing Service)


20

Are Academic Profile Scores Sensitive to Instruction?

• Mean Critical Thinking score:– for students starting core curriculum = 109 (n=214)

– for students completing core curriculum = 114 (n=1739)

– SD = 6.0, so d = .8

(Marr, D. [1995]. Validity of the Academic Profile. Princeton, NJ: Educational Testing Service)

• BUT this result could simply be the result of less able students dropping out


21

Are Academic Profile Scores Sensitive to Instruction (continued)

• Compare Academic Profile scores of a longitudinal sample of freshman and seniors

• Sample is 112 students at a small Midwest university– Freshman mean = 448

– Senior mean = 453

– SD = 17, so d = .4

• Must assume equal motivation of freshmen and seniors, so may be an underestimate

(Weimer, D. [1999]. Dealing with messy data: Analyzing pre- and post-test assessment results in the real world. Paper presented at the Annual Forum of the Association for Institutional Research, Seattle)


22

Are MAPP Scores Sensitive to Instruction?

• Freshmen and Seniors from 23 colleges; college is unit of analysis

Freshmen (N = 4373) Seniors (N = 1823) Mean SD Mean SD d MAPP Total 438 7.3 448 9.3 1.2 Critical Thinking 110 1.9 113 2.3 1.4 Writing 113 1.7 115 1.8 1.1 SAT 1001 72 1049 94 0.57


23

What are Threats to Validity of Value Added Scores?

• Construct irrelevant variance– Motivation

– Non-representative sample

• Construct underrepresentation– Banta and Pike (2006) argue, “testing students’

generic as well as discipline specific knowledge and skills in their major fields of study provides a much more promising avenue for assessing growth and development in college than do today’s most commonly discussed tests of generic skills alone”


24

How to Address the Motivation Problem

• Screen unmotivated examinees– Self-reports

– Timing

• Monetary incentives for performance

• Appeal to value to students of having recognized high-performing school

• Despite these concerns, high correlations with SAT scores suggest motivation not currently a major problem


25

How to Address Sampling Problem

• Random sample--avoid volunteer sample

• Test all seniors– Avoids self-selection problem

– Permits comparisons among different majors/programs


26

How to Address Construct Underrepresentation

• In addition to a general skills test, such as MAPP, assess knowledge and skills that are specific to particular majors (e.g. Major Field Tests)

• Field-specific tests are likely to show much stronger growth than general skills test


27

Issues in Using MAPP in Other Countries

• Institution must evaluate extent to which skills assessed in English are relevant to the institutional mission

• Norm groups not comparable

• Translating words will not necessarily create a test in another language that assesses the same construct as MAPP


28

Review and Next Steps

• MAPP is:– Content valid

– Reliable

– Correlated with related constructs

– Sensitive to instruction

• Future research– FIPSE study relating scores on CLA, CAAP, and MAAP currently

underway

– Initial value added study of 23 current MAPP users will be expanded as more institutions provide relevant data

– ETS will work with individual institutions on local validity studies

Documents

MAPP as an Accountability Assessment Brent Bridgeman Educational Testing Service Center for Validity Research