This article was downloaded by: [University of Illinois Chicago]On: 28 October 2014, At: 17:01Publisher: RoutledgeInforma Ltd Registered in England and Wales Registered Number: 1072954Registered office: Mortimer House, 37-41 Mortimer Street, London W1T 3JH, UK

The Journal of Economic EducationPublication details, including instructions for authors andsubscription information:http://www.tandfonline.com/loi/vece20

Comparing AP and College StudentLearning of EconomicsClaire Melican , Fekru Debebe & Rick MorganPublished online: 25 Mar 2010.

To cite this article: Claire Melican , Fekru Debebe & Rick Morgan (1997) Comparing AP andCollege Student Learning of Economics, The Journal of Economic Education, 28:2, 135-142

To link to this article: http://dx.doi.org/10.1080/00220489709595914

PLEASE SCROLL DOWN FOR ARTICLE

Taylor & Francis makes every effort to ensure the accuracy of all the information(the “Content”) contained in the publications on our platform. However, Taylor& Francis, our agents, and our licensors make no representations or warrantieswhatsoever as to the accuracy, completeness, or suitability for any purpose of theContent. Any opinions and views expressed in this publication are the opinions andviews of the authors, and are not the views of or endorsed by Taylor & Francis. Theaccuracy of the Content should not be relied upon and should be independentlyverified with primary sources of information. Taylor and Francis shall not be liablefor any losses, actions, claims, proceedings, demands, costs, expenses, damages,and other liabilities whatsoever or howsoever caused arising directly or indirectly inconnection with, in relation to or arising out of the use of the Content.

This article may be used for research, teaching, and private study purposes. Anysubstantial or systematic reproduction, redistribution, reselling, loan, sub-licensing,systematic supply, or distribution in any form to anyone is expressly forbidden.Terms & Conditions of access and use can be found at http://www.tandfonline.com/page/terms-and-conditions

Comparing AP and College Student Learning of Economics

Claire Melican, Fekru Debebe, and Rick Morgan

The Advanced Placement (AP) program offers talented high school students the opportunity to take college-level courses. To receive college credit andor placement for the courses, students are required to take the AP exams prepared by the Educational Testing Service (ETS) that are administered annually in May and consist of a multiple-choice section and a free-response section. The free-re- sponse questions are scored by AP teachers and college professors at an annual reading in June. The results of the multiple-choice section and free-response sec- tion are combined to determine a composite score and then converted to a 5-point scale:’ 5 is extremely well qualified; 4, well qualified; 3, qualified; 2, possibly qualified; and 1, no recommendation.

The College Board offers two AP economics exams-microeconomics and macroeconomics-designed to test students’ knowledge and understanding of the content covered in a typical first-year one-semester college course. In 1994, 8,253 high school students took the micro exam, and 9,840 took the macro exam. Each exam is two hours in length and consists of a 70-minute multiple-choice section containing 60 items and a 50-minute free-response section containing three questions. The multiple-choice section accounts for two-thirds of the stu- dent’s total score, and the free-response section for one-third.

An important element of the AP program is a comparison of high school stu- dents’ performance on AP exams with that of college students who have com- pleted the requisite college course. To this end, the College Board regularly con- ducts comparability studies in which college professors are asked to administer the AP exams to their students and grade them using their own grading standards. Professors are asked to administer the AP exams under motivated conditions. That is, the AP exams should be part of the final exam or, alternatively, count as a significant part of the course grade to ensure that their students are sufficiently motivated. Because no college professor indicated that the test was given to stu- dents under unmotivated conditions, all colleges were included in the analysis. The college students’ free-response sections are then scored concurrently with the AP exams taken by high school students at the annual AP reading.

Chire Melican, Fekru Debebe, arid Rick Morgan are with the Educutional Testing Service, Prince- ton, N.J. The authors uish to thank Kathy Lynch arid Lee Jones of Educational Testing Svrvice ,for their editorial comments and helpful suggestions.

Spring 1997 135

Dow

nloa

ded

by [

Uni

vers

ity o

f Il

linoi

s C

hica

go]

at 1

7:01

28

Oct

ober

201

4

In this article, we outline the design of the comparability study and report on our analysis of the data generated by the study to assess the validity of the AP exams. Specifically, we use the performance of these two groups of students on the exams to (1) evaluate the validity of the cutpoints currently used in deter- mining the AP grades of 1-5 for the high school students, ( 2 ) compare the gen- eral level of AP candidates’ exam performance with that of college students enrolled in courses for which credit and advanced placement are typically sought, (3) compare the grading of the AP exam (i.e., the conversion of compos- ite scores to grades) with the grading of exam performance in the colleges; and (4) assess the validity of AP grades for identifying students capable of doing col- lege-level work in the course for which credit or advanced placement is sought.

DESIGN OF THE STUDY

In spring 1993, department chairs at the 200 colleges that received the most AP grades in economics and college professors who serve as faculty consultants at the annual AP Reading were contacted and asked to participate in the study. Partici- pating professors were asked to administer the entire 1994 AP micro or macro exam to their students. Students at 11 universities took the micro exam, and stu- dents at 7 universities took the macro exam. (See Table 1 for a list of participating colleges.) For both exams, one college professor administered only the multiple- choice section. After administering the study exams, instructors were asked to as- sign a letter grade of A, B, C, D, or F based on the student’s combined performance on the multiple-choice and free-response sections. Professors were free to assign weights to the different sections of the exams in any manner they chose.

In addition to assigning grades to each student’s exam, professors were asked to provide the final course grade earned by each student. Professors determined the percentage of the final course grade accounted for by the study exam. Final- ly, each professor was requested to provide an estimate of the course grade

TABLE 1 Participating Colleges and Number of Students Tested, by Subject

Microeconomics No. of

students Macroeconomics No. of

students

Colby College Indiana-Purdue University

Mary Washington College Northem Kentucky University Northern Illinois University SUNY-New Paltz SUNY-Westbury Troy State University University of Alabama-Birmingham Washington University Wichita State University

at Indianapolis

84 26

38 87 23 30 26 22 51 64 68

Indiana-Purdue University 43

Mary Washington College 97

Northern Kentucky University 33 St. Lawrence University 47

at Indianapolis

Nebraska Wesleyan University 56

University of Missouri at Rolla 18 University of North Texas 124

136 JOURNAL OF ECONOMIC EDUCATION

Dow

nloa

ded

by [

Uni

vers

ity o

f Il

linoi

s C

hica

go]

at 1

7:01

28

Oct

ober

201

4

obtained by each student excluding the score on the AP exam. This estimated grade was to serve as an independent criterion against which to evaluate the validity of the study exam grades.

After completing their grading, professors returned all materials to ETS. The multiple-choice answer sheets were scanned and scored by computer. Covers identical to those used for the operational AP exam were put on the free-response booklets. Each booklet was then assigned an AP number and scored along with the free-response booklets of actual AP candidates during the AP reading. The graders were unaware of whether the free-response questions were written by high school students or college students. The multiple-choice and free-response scores were combined to compute a composite score that was then converted into the AP 5-point grading scale.

RESULTS

We offer here a brief analysis and interpretation of the statistical information resulting from the study (Tables 2-5). We caution the reader about making any generalization based on the results. Because the study was based on the relative- ly small number of institutions that responded to our invitation, it may not entire- ly reflect the types of colleges to which AP economics candidates apply.

Summary statistics on the performance of AP candidates and college students for both the micro and macro exams are reported in Table 2.? The means of the average scores for each college are presented in the table. Each college provided equal weight in determining the overall means. This procedure was applied to all the tables.3

On average, the AP candidates obtained higher multiple-choice, free-response, and composite scores than did the group of college students (Table 2). The aver- age micro exam score for the AP students was higher than the average score of the college students at 9 of the 11 colleges. For macro, the AP students had a higher total score than did students at 4 of the 7 colleges. The comparative mean differences were larger for the micro exam than for the macro exam. At least part of this difference in comparative exam performance could result from the differ- ential composition of the two groups of colleges that administered the exams.

TABLE 2 Multiple-choice (MC), Free-Response (FR), and Composite

Means (CM) as Percentage of Maximum Score for AP Candidates and College Students

MC FR CM

Microeconomics AP candidates 53.8 5 I .4 53.0 College students 40.0 31.1 36.0

Macroeconomics AP candidates 54.9 46.3 52.0 College students 49.7 31.2 44.2

Spring 1997 137

Dow

nloa

ded

by [

Uni

vers

ity o

f Il

linoi

s C

hica

go]

at 1

7:01

28

Oct

ober

201

4

We present the composite score means for the groups of AP candidates receiv- ing each AP grade and for the groups of college students receiving each final course grade in Table 3. Four of the schools participating in the micro study and one in the macro study did not provide any grades and were excluded from these tables. The data show that for both exams, AP candidates with grades of 5 out- performed the college students receiving course grades of A. AP candidates receiving grades of 4 also outperformed the college students receiving course grades of B and had mean scores most similar to those of the college students receiving grades of A. For both exams, the AP candidates receiving grades of 3 outperformed the college students receiving course grades of B.

In Table 4, we report composite score means for the groups of AP candidates receiving each of the AP grades and the groups of college students receiving each instructor-assigned exam grade. For both exams, the average composite score of AP candidates with a grade of 5 is much higher than the average obtained by col- lege students with an exam grade of A. The average composite score of AP micro

TABLE 3 Means as Percentages of Maximum Composite Score for AP Candidates Receiving Each AP

Grade and College Students Receiving Each Final Course Grade

AP candidates AP Percentage of

grade composite score

College students Instructor Percentage of

grade composite score

5 (14%)

4 (26%)

3 (32%)

2 (14%)

1 (15%)

Microeconomics

82.0 A

66.7 B

50.6 C

(I 1%)

(24%)

(39%) 36.8

21.7

D&F (26%)

59.9

43.2

31.4

23.9

5 (1 6%)

4 (29%)

3 (25%)

2 ( I 6%)

1 (14%)

81.3

64.4

48.6

34.2

17.6

Macroeconomics

A (14%)

B (34%)

C (32%) D&F (20%)

65.5

46.3

36.5

25.8

138 JOURNAL OF ECONOMIC EDUCATION

Dow

nloa

ded

by [

Uni

vers

ity o

f Il

linoi

s C

hica

go]

at 1

7:01

28

Oct

ober

201

4

TABLE 4 Means as Percentages of Maximum Possible of Composite Scorns for AP Candidates Receiving

Each Grade and for College Students Receiving Each Instructor-Assigned Exam Grade

AP candidates College students AP Percentage of Instructor Percentage of

grade composite score exam grade composite score

5 ( 14%)

4 (26%)

3 (32%)

2 (14%)

I ( I 5%)

82.0

66.7

50.6

36.8

21.7

Microeconomics

A (6%) B

(16%)

C (22%)

D&F (56%)

62.8

52.1

42.4

25.4

5 (16%)

4 (29%)

3 (25%)

2 (16%)

1 (14%)

Macroeconomics

81.3 A

64.4 B

48.6 C ( 2 9 9 )

34.2 D&F (44%)

(10%)

(17%)

17.6

60.7

57.4

43.3

28.0

candidates with a grade of 4 is slightly higher than the average composite score obtained by college students with an exam grade of A, whereas the average com- posite score of AP macro candidates with a grade of 4 was slightly lower than the average composite score obtained by college students with an exam grade of A. The two composite score averages of AP candidates with a grade of .1 were between the averages obtained by college students receiving exam grades of B and C. The two composite score averages of AP candidates with a grade of 2 were both lower than the averages obtained by college students receiving exam grades of C and were higher than the combined D and F group.

As noted earlier, an estimate of the final course grade each student would have received acluding the results of the AP exam was sought to obtain indirect evi- dence of the validity of AP grades for decisions concerning the granting of cred- it or advanced placement. The estimated course grade was used as an indepen- dent criterion in evaluating the concurrent validity of AP grades. Correlations of the estimated course grades excluding study exam results with composite scores

Spring 1997 139

Dow

nloa

ded

by [

Uni

vers

ity o

f Il

linoi

s C

hica

go]

at 1

7:01

28

Oct

ober

201

4

by students in the study were calculated and ranged from .I5 to .90, with a medi- an correlation of .61. Given the less-than-perfect reliability of both the study composite scores and instructor-assigned grades, the correlation data suggest some agreement between college grading and AP scoring.

Table 5 contains a cross-classification of college students’ AP grades for micro and macro with their estimated course grades. The rows in Table 5 refer to AP grades, and the columns refer to estimated course grades. The first number in each cell is the total number of students per 1,000 students receiving each com- bination of AP grade and estimated course grade. The number below that refers to the percentage of college students receiving that row’s AP grade as well as that column’s estimated course grade. For example, of the 25 percent of college stu- dents receiving an AP grade of 3 in micro, 11 percent earned an A from their instructor, 47 percent a B, 36 percent a C, 5 percent a D, and less than 1 percent an F. The bottom number refers to the percentage of college students who received that column’s estimated course grade as well as that row’s AP grade. That is, 23 percent of college students with an estimated grade of A received an AP grade of 3.

Of the 401 per 1,000 students who would have received AP grades of 3 or bet- ter in micro, only 4 percent were judged by their instructor as meriting a D or an F, and 68 percent were doing at least B work in their course (Table 5). Of the 616 per 1,000 students who received estimated AP grades of 3 or better in macro, only 5 percent were judged by their professor as meriting a D or F, and 71 per- cent were doing at least B work in their course. In general, students who were judged by their instructors as doing poorly in their course also did poorly on the study exam and those who were doing well in their course did well on the study exam. However, 40 percent of the micro students and 19 percent of the macro students who were judged as doing B or better work in the college course received an AP grade below 3.

CAVEATS

In interpreting the results of the study, we needed to consider a number of issues. AP candidates are generally considered to be highly motivated because their individual performance on this one exam will determine whether or not they receive college credit andor placement for work completed in high school. College students, on the other hand, know that other exam scores, homework assignments, and additional factors will be used in determining their final course grade. If motivational differences between the high school AP students and the college students exist, some of the apparent rigor of the AP grading standards, relative to those used by the college professors, could be attributed to this phenomenon.

A second concern is the relative appropriateness of the AP exam as a measure of the instructional objectives of the AP and college courses. It is likely that the AP high school courses are very similar to one another in that they probably fol- low the outline in the AP course description for the designated course. Although the outline, in general, represents the content covered at most colleges and uni-

140 JOURNAL OF ECONOMIC EDUCATION

Dow

nloa

ded

by [

Uni

vers

ity o

f Il

linoi

s C

hica

go]

at 1

7:01

28

Oct

ober

201

4

versities, individual college professors may place more or less emphasis on any one topic covered in the outline. In addition, high school teachers have access to the free-response questions used on AP exams in previous years, and to multiple- choice questions used in the 1990 exams. Consequently, the AP exam may be

TABLE 5 Cross-Classification of College Student's AP Exam Grades with Their Estimated Course

Grade, Excluding Exam Results" (per 1,000 Students)

Estimated course grade, excluding exam results AP exam grade A B C D F

Microecoiiomrcs

5 26 7 (3%) (79%) (21%)

(22%) (2%)

4 34 59 24 4 (12%) (28%) (49%) (20%) (3%)

(28%) ( 18%) (7%) (3%)

3 28 I I7 88 13 1 (25%) ( 1 1 % ) (47%) (36%) (5%) (0%)

(23%) (35%) (24%) ( 1 1 % ) (2%)

(20%) (7%) (29%) (45%) ( 1 1 % ) (8%) 2 14 59 92 22 16

(12%) (18%) (25%) (18%) (25%)

I 18 91 I59 80 48 (40%) (15%) (23%) (40%) (20%) (12%)

(27%) (44%) (67%) (74%)

( 1 2%) (33%) (36%) ( I 2%) (7%)

4 (21%)

3 (32%)

2 (21%)

I ( I 8%)

70 (81%) (36%)

70 (34%) (36%)

38 (12%) (19%)

10 (5%) ( 5 % )

7 (4%) (4%)

(19%)

Mucroeconornics

16 (19%)

( S % )

109 26 (53%) (13%) (32%) (9%)

I34 121 (41%) (37%) (39%) (40%)

70 88 (34%) (43%) (20%) (29%)

14 67 (8%) (38%) (4%) (22%)

(34%) (30%)

I (0%) ( 1 % )

31 (10%) (27%)

28 10 ( 14%) (5%) (24%) (22%)

55 35 (31%) (20%)

(12%) (5%)

(48%) (78%)

?he top number in each row ot the grade columns refers t o the number of student\ per 1.ooO students.

Spring 1997 141

Dow

nloa

ded

by [

Uni

vers

ity o

f Il

linoi

s C

hica

go]

at 1

7:01

28

Oct

ober

201

4

less appropriate as a measure of achievement for students at one particular col- lege than it is for the high school students.

Many of the professors who participated in the study were familiar with the AP program. Some had previously administered multiple-choice pretest ques- tions, others were faculty consultants who have participated in the scoring of the free-response questions. Professors who were unfamiliar with the program were sent course descriptions to help them understand the content of the exam. Pre- sumably, professors who thought the content of the AP exams was inappropriate for their students declined to participate in the study.

As noted earlier, although well over 200 schools were contacted to participate in the study, fewer than 20 professors at 14 schools actually participated. Thus, the study sample cannot be considered a random sample of colleges that accept AP grades. Furthermore, the colleges in the study do not entirely reflect all the colleges that AP economics students attend. Therefore, some of the difference in performance between AP students and college students may result from the sam- ple of participating colleges. It is suggested that colleges conduct their own stud- ies or participate in future comparability studies to determine the relevance of the current results to situations at their institutions.

CONCLUSION

With the foregoing caveats in mind, the results of the study indicate that AP candidates as a whole performed better than did the college students with respect to the skills measured by both the multiple-choice and the free-response sections. In general, AP students who received a grade of 3 displayed an average exam per- formance superior to that obtained by college students who received a final course grade of C. The exam performance of AP candidates with grades of 4 was superior to the level displayed by college students with grades of B and close to the average performance of the college students earning grades of A. The aver- age exam performance of AP candidates with a grade of 5 was much higher than that achieved by college students with a final grade of A. This suggests that the AP grading standards for these exams is more stringent than those used by the college instructors in the study.

NOTES

1 . The cutpoints are determined after the scoring of the exams and are based on information provid- ed by the statistical analysis staff at ETS. For a detailed discussion of the AP grading scale, con- sult Grading, Interpreting, and Using Advanced Placement Exams (The College Board Advanced Placement Program 1994).

2. The multiple-choice section is scored by formula with a correction for guessing: one-fourth of the number of incorrect answers is subtracted from the number of correct answers. Negative scores are set to zero. No points are deducted for omitted items.

3. Throughout the remainder of the article, percentages of the maximum score will be referred to as points.

142 JOURNAL OF ECONOMIC EDUCATION

Dow

nloa

ded

by [

Uni

vers

ity o

f Il

linoi

s C

hica

go]

at 1

7:01

28

Oct

ober

201

4