Transcript
Page 1: The impact of framing effect on student preferences for university grading systems

Studies in Educational Evaluation 35 (2009) 160–167

The impact of framing effect on student preferences for universitygrading systems

Jeffrey K. Smith *, Lisa F. Smith

University of Otago, New Zealand

A R T I C L E I N F O

Article history:

Received 4 April 2009

Received in revised form 4 November 2009

Accepted 6 November 2009

Keywords:

Framing theory

Grading systems

Assessment

Motivation

A B S T R A C T

Kahneman and Tversky’s (1979, 2000; Tversky & Kahneman, 1981) work in decision-making was applied

to student preferences for grading practices. Undergraduate psychology students (n = 240) were randomly

assigned to 1 of 3 framing conditions related to how a university course might be graded: a 100 point

system, a percentage system, and an open point system of assessment. The assignments in each condition

were the same, and the relative weights assigned to each were the same. The only difference was in the

approach to how many total points were involved and how the final grades were calculated. Thus, the same

performance would yield the same final grade in each condition. Participants reviewed the grading system

they were assigned to and then were asked to rate a series of possible assessments to be used in the course

with regard to their motivation, confidence, effort, accuracy of assessment, and how well the student might

do using those assessments. Multivariate and univariate analyses of variance indicated that the framing

conditions significantly affected motivation, confidence, and effort, but not the perception of how well the

student would do in the course or whether the grading system provides an accurate assessment of

performance. Results are discussed in terms of application to college grading, the relationship to framing

effect theory, and the potential for decision theory to inform assessment practice.

� 2009 Elsevier Ltd. All rights reserved.

Contents lists available at ScienceDirect

Studies in Educational Evaluation

journa l homepage: www.e lsev ier .com/stueduc

Almost all courses at the tertiary level involve grading, andtherefore involve a scheme for assigning grades. Universities varysubstantially in the policies they have for such grading practices. Atmost universities, the options for grades to be assigned (e.g., A, B, C,D, F, and then plusses and/or minuses) are determined for theuniversity as a whole; however, determining which assessmentsand assignments are counted for grades, and how those arecombined to form grades is usually left to the discretion of theindividual faculty member. In the United States, it is most often thecase that assessments and assignments are given weights of sometype, and then graded according to a scale, after which the scoresand the weights are combined and reduced to one score on a 0–100scale, with the following general conversions to letter grades (seee.g., Dixon, 2004):

� A 90–100� B 80–90� C 70–80� D 60–70� F Below 60

* Corresponding author at: University of Otago, Educational Studies and

Professional Practice, PO Box 56, Corner of George and Union, Dunedin, Otago

9001, New Zealand. Tel.: +64 3 479 5467; fax: +64 3 479 7550.

E-mail address: [email protected] (J.K. Smith).

0191-491X/$ – see front matter � 2009 Elsevier Ltd. All rights reserved.

doi:10.1016/j.stueduc.2009.11.001

Usually the top 2 or 3 points in each range merit a plus, and thebottom 2 or 3 points merit a minus. Some faculty operate solelythrough averaging letter grades, but most use a number systemand then convert numerical averages to letters (Dombach & Smith,2004). The number systems used vary widely, but tend to fall intothree broad categories, discussed below.

Perhaps the most common approach is to assign weights toeach of the assessments/assignments to be used for grading, withthe weights summing to 100%. Each assessment is graded on a 0–100 scale, and then multiplied by the weight to determine itscontribution to the total grade.

A second approach assigns a point value to each assessment/assignment with the total of these points summing to 100. Thissystem is functionally identical to the first system, but theweight of the assessment/assignment has been turned into thetotal point value. This obviates the need to multiply the weighttimes the score received on the 0–100 score scale whencalculating a final grade.

A third approach is a variation of the previous twoapproaches. It assigns each assessment/assignment a ‘‘natural’’number of points based on how much the faculty member thinkseach aspect of each assessment/assignment should be worth,without worrying what the total number of points will sum to atthe end of the semester. To establish how the number of pointsearned will translate to final grades, the total number of possiblepoints is determined, and this total is multiplied by .90 to get

Page 2: The impact of framing effect on student preferences for university grading systems

Fig. 1. Example of approaches to assessment.

J.K. Smith, L.F. Smith / Studies in Educational Evaluation 35 (2009) 160–167 161

the A/B cut, by .80 to get the B/C cut, etc. We consider this to bean ‘‘open point’’ system, where the number of total points in thecourse can vary, but within any particular course, would bespecified. Each of these three approaches is summarized inFig. 1, along with the way the scores would work out for animaginary student.

It can be seen that these three approaches are functionallyidentical, varying only in terms of when the mathematics is appliedto the grading system, and when scores are rounded off intopercentages. Note that in the open point system (which we haveinstantiated to be 600 points in this research), finer gradations ofscores on assessments/assignments can be made, and then thepoints are basically rounded into the grade categories. In the 100point system, some of this rounding is done within eachassessment/assignment (there is no equivalent of a score of 63in the 600 point system when using the 100 point system, unlessthe faculty allows for scores such as 10.5). The percent systemactually allows for even finer gradations, especially on assess-ments/assignments that carry a small weight. These are fairlynuanced points in an overall approach to grading; fundamentallythe three systems are the same.

Assessment Percent

system

100 point

system

600 point

system

Student’s scores

Percent 100 pts 600 pts

Homework 20% 20 pts 120 pts 80% 16 pts 96 pts

Midterm exam 20% 20 pts 120 pts 90% 18 pts 108 pts

Class presentation 10% 10 pts 60 pts 60% 6 pts 36 pts

Term paper 25% 25 pts 150 pts 88% 22 pts 132 pts

Final exam 25% 25 pts 150 pts 92% 23 pts 138 pts

aTotal 100% 100 pts 600 pts 85% 85 pts 510 pts

a Grade calculation

Percent : 80%� 20%þ 90%� 20%þ 60%� 10%þ 88%� 25%

þ 92%� 25%

¼ 85%

The student’s final grade is 85%, which would be a B.

100 pts : 16þ 18þ 6þ 22þ 23 ¼ 85

The student’s final grade is 85 points, which would be a B.

Open point ð600 ptsÞ : 96þ 108þ 36þ 132þ 138 ¼ 510

Page 3: The impact of framing effect on student preferences for university grading systems

J.K. Smith, L.F. Smith / Studies in Educational Evaluation 35 (2009) 160–167162

The student earned 510 points on a 600 point scale.

90%� 600 ¼ 540 or above ¼ A

80%� 600 ¼ 480; 480� 539 ¼ B

Therefore, the student’s final grade is a B.Alternatively, 510/600 = 85, which would be a B.

It can be seen that the three approaches to grading are the samefrom a quantitative perspective. But do students perceive them asbeing the same? This question arose from our practical experienceas university faculty. One of us consistently uses the 100 pointsystem under the argument that it eliminates the need to engage incalculations to determine a final grade and thus is easier forstudents to understand where they stand in a course at all times.But, in a discussion of the system in an assessment course, it wasdiscovered that the students did not like that approach. It seemedto them that they were, in the words of one student, ‘‘starting thecourse with 100 points and losing points away from that with eachnew assessment/assignment that was not perfect.’’ That is, theysaw the grading system as one in which each grade on anassignment or assessment took them farther away from the 100points that they had to start with, or the neutral point (Tversky &Kahneman, 1974) where they had started.

Although the grading system may seem to be little more than anecessary evil, for many students it is the payoff for the work donein the course (Brookhart, 1993b, 2004). Students are sensitive tothe grading system used, because the grades they receivesummarize their efforts and reflect upon their achievement inthe course (Brookhart, 1993a). Thus, careful consideration of theassessment system, and the procedures used for tallying grades is afactor that needs to be considered carefully in instructionalsettings (Harlen, 2007). It is natural that some students wouldprefer examinations taken in the course while others prefer termpapers and still others prefer projects done in groups (Smith,2009). Students will prefer whatever they think they are best atand what fits with their course work, study style, and timeallocated for the course. Some prefer whatever they perceive as theeasiest road to the highest grade. But, why should they prefer onemethod of tallying grades over another if the various systems arefunctionally identical?

To look for an answer to this question, we turn to thedecision-making literature, and in particular to the work ofKahneman and Tversky’s (1979, 2000; Tversky & Kahneman,1981). There are many aspects to the life work of thesescientists; in this research we focus on the concept of framing

effects and how people behave in situations where they perceivea potential loss and a potential gain (Tversky & Kahneman,1986). Tversky and Kahneman (1986, p. 258) presented thefollowing simple experiment to illustrate this:

Potential gain setting: assume yourself richer by $300 than youare today. Choose between: Option K: a sure gain of $100 orOption L: a 50% chance to gain $200 and a 50% chance to gainnothing.Potential loss setting: assume yourself richer by $500 than youare today. Choose between: Option K0: a sure loss of $100 orOption L0: a 50% chance to lose $200 and a 50% chance to losenothing.Note that in both of the K situations, you end up with a sure$400, and in both of the L situations, you have a 50% chance toend up with $500 and a 50% chance to end up with $300.

In Tversky and Kahneman’s (1986) study, they found that in thepotential gain setting, 72% of respondents took the $100 sure gain(putting them at an assured $400). In the potential loss setting, 36%of the subjects took the $100 sure loss (putting them at an assured

$400). It appears to be the case that individuals would rather take arisk to avoid a perceived loss than to anticipate a potential gain (inthis case, they were twice as likely). This finding, and ones similarto it, has been replicated in a variety of studies (Johnson, Hershey,Meszaros, & Kunreuther, 1993; Takemura, 1992; Yamagishi, 2002).The prospect of loss appears to be viewed more negatively byindividuals than the prospect of gain is viewed positively.

Now, in real life, individuals are usually not presented with suchclear-cut alternatives and options. Often, they are not presentedwith alternatives at all, but go about their lives navigatingsituations that are not entirely under their control. But they doreact to these situations in fashions that are predictable from howthe situations are perceived (Chong & Druckman, 2007). Withregard to our original set of course grading schemes describedabove, it appears to be the case that students perceive the 100point system as one of successive losses from the perfect gradethey had to begin with, as opposed to the open point system andthe percentage system, which they perceive to be more of abuilding or gaining toward a grade rather than losing points awayfrom it. In part, we believe that this is due to the 100 point systemengendering a sense that the neutral point (Tversky & Kahneman,1974) is 100; in the percent and 600 point system, the neutral pointis either zero, or perhaps simply not well-defined in the student’smind. One possibility in the latter two settings is that the studentsuse their anticipated grade in the course as a neutral point. Thepurpose of this study was to see if, in fact, our observations fromteaching would be confirmed in an experimental setting, and if theframing effect described by Kahneman and Tversky offers a helpfulexplanation of the phenomenon.

The study was designed to allow for an evaluation the followinghypothesis: the 100 point grading system is perceived by studentsas a ‘‘loss scenario’’ in comparison to an open point system and apercentage system. This would, in turn, engender a more negativeattitude toward the course as a whole than either of the other twoapproaches. The negative attitude resulting from this framingeffect would influence students’ response to aspects of the coursethat were logically independent of how the grades were tallied. Toinvestigate this hypothesis, we set up a study to explore the ideathat the framing of the grading scheme influenced not just thestudent response to the scheme directly, but had a broader effecton how students perceived aspects of the evaluation system notdirectly related to how the points were tallied.

Method

Participants were provided with a course scenario (describedbelow) involving the 100 point system, the percent system, or theopen point system (referred to as the 600 point system hereafter),and then asked about different kinds of approaches to assessmentthat might be involved in such a system. That is, we did not directlyask how much they reacted to the grading system, but how theywould react to taking multiple choice tests, essay tests, shortanswer tests, and the like under the scenario to which they wereassigned. How the assessments are combined into grades shouldnot be related to the assessment methods used in the course unlessthe framing of the course grading system affects how the studentsview the course more generally.

Participants

The participants for the study were 240 undergraduate studentsat a public university in New Jersey. The participants were enrolledin psychology classes and received credit for their participation.Forty-seven were male and 193 were female; 198 were betweenthe ages of 18–24, with 52 being 25 or older. One hundred twenty-eight self-identified as Caucasian, 41 were African American, 34

Page 4: The impact of framing effect on student preferences for university grading systems

Table 1Means, standard deviations, and reliabilities for variables.

Variable N Mean Standard deviation Reliability

Effort 240 24.27 3.64 .86

Motivation 240 21.95 3.68 .77

Demonstration 240 21.55 3.34 .69

Usefulness 240 20.68 3.21 .69

Confidence 240 20.47 3.11 .68

Self-efficacy 240 20.45 2.97 .58

Preference 240 18.08 3.94 .66

Anxiety 240 14.18 4.57 .85

J.K. Smith, L.F. Smith / Studies in Educational Evaluation 35 (2009) 160–167 163

were Hispanic, 19 were Asian or Pacific Islanders, 2 were NativeAmericans, and 16 listed their race as mixed. Two hundred twenty-three of the participants were full time students and 17 were parttime.

Materials/measures

Materials included a consent form, a debriefing form, and ademographic questionnaire for each participant. There were threeversions of the questionnaire for the study itself that formed theframing of the grading system: a 100 point system, a percentsystem, and a 600 point system of assessment. In each system, thegrading scheme was designed to produce exactly the same grades.The grading system scenarios are included in Appendix.

After reading the scenario of the grading system, participantsresponded to a series of questions about how they would react todifferent types of assessments in the course (e.g., multiple choiceformat, essays, computer-based testing, etc.). For each type ofassessment, participants were asked to rate each of the followingon a 1–5 (strongly disagree to strongly agree) Likert-type scale:

� My motivation is high (motivation).� My anxiety is high (anxiety).� My confidence is high (confidence).� I apply maximum effort (effort).� This shows what I’ve learned (demonstration).� I do well with this (self-efficacy).� This is a useful technique (useful).� I prefer this in a class (preference).

Procedure

Institutional Review Board approval was received prior tobeginning the study. Each student completed a consent form andwas given a copy of the consent form. At the beginning of the study,each participant completed the demographic questionnaire.Participants were then randomly assigned to one of the threescenarios. After reading their scenarios, they responded to thequestionnaire concerning their reactions to the various approachesto assessing their performance in the course. After handing in theirquestionnaires, participants received debriefing forms and weregiven the opportunity to comment, ask questions, and discuss thestudy. Additionally, discussions about the study were held in theclasses from which the sample was drawn to get student reactionsto the experiment and to help with the interpretation of the results.

Results

A set of eight scales was developed using the responses to theassessment format items listed above. That is, the motivationresponses were summed, then the confidence responses weresummed, etc. Each of the scales summed the items for:

� multiple choice testing;� essay testing;� short answer testing;� paper and pencil testing;� computer-based testing;� open book testing.

Means, standard deviations, and reliabilities for the scales arepresented in Table 1. The scales were constructed to examinestudents’ overall levels of affect and cognition with regard to thescenario in which they were placed. We were not interested inwhether the students preferred multiple choice assessment toessay assessment, etc. Nor were we interested in whether the

motivation scores overall were higher than the usefulness scores.We were, instead, interested in whether motivation or usefulness,etc. (summed over the various formats) varied according to thegrading system scenario the student read. The argument here isthat the ‘‘frame’’ set up by the different grading/scoring schemeswould cause participants to view the assessment systemdifferently, even though the formats used for assessment arenot related to the tallying system, nor are the different schemesfundamentally different. Although several of the reliabilities werelower than might have been hoped for, they are all reasonable forresearch purposes, and fairly good overall for scales based on sixitems.

The data analysis evaluated the basic hypothesis that thedifferent approaches to tallying points into a final grade (theframes) would affect how students viewed the various assessmentoptions (multiple choice, etc.) in the course. We developed fouroutcome measures that were directly related to affect (motivation,confidence, effort, and anxiety), and four that were more cognitive,or evaluative, in nature (demonstration, usefulness, self-efficacy,and preference). Note that although some might argue that Self-efficacy has both affective and cognitive components, in thissetting, participants were simply indicating whether they felt theydid well with this type of format (more a cognitive than affectiveassessment). This categorization scheme would argue for twomultivariate analyses of variance to be conducted, one with theaffective variables, and one with the cognitive variables.

We took a slightly different route in the data analysis, however.We analysed the cognitive/evaluative variables together in onemultivariate anova. However, we analysed the Anxiety variableseparately from the other affective variables, for two reasons. First,the variable is inherently negative in valence, and thus we wouldanticipate that the means would operate in a different fashion fromthe positive affective measures. Second, in other work we haveconducted, we have seen that anxiety toward assessment isfundamentally less influenced by the particulars of a setting thanother affective variables (Smith & Smith, 2002; Wolf, Smith, &Birnbaum, 1995). Thus, we felt that Anxiety was not fundamentallysimilar to motivation, confidence, and effort, and should beanalysed separately from these other variables, which werepositive in valence, and more likely to indicate a response to aframing effect.

Each analysis used the grading system that participants wererandomly assigned to, and a self-rating of college grade pointaverage category (2.00–2.99, 3.00–3.49, and 3.50–4.00) asindependent variables in a 3 � 3 analysis. This allowed us to firsttest the hypothesis across all students, and to see if students ofdiffering achievement levels reacted differently to the variousscenarios. Our interest was in the main effect for the differentframes, and then in the interaction term. We were not interested inthe main effect for grade point average here. Following eachmultivariate anova, we examined the univariate anovas for eachvariable.

It was hypothesized that students would perceive the 100 pointsystem as what Kahneman and Tversky (2000) would see as a

Page 5: The impact of framing effect on student preferences for university grading systems

J.K. Smith, L.F. Smith / Studies in Educational Evaluation 35 (2009) 160–167164

potential loss setting, with the percent system and the 600 pointsystem being viewed as potential gain settings. Recall that eachsystem was functionally identical, and that the student waspresented with being in exactly the same grade situation in eachscenario. The 100 point system was believed to be perceived as‘‘losing’’ points from a starting point of 100 points, whereas theother two systems were believed to be more perceived as buildingpoints toward a final grade. Although we did not formally specifythe 600 point system as being the one most likely to be viewed as apotential gain setting, it seemed to us that students might see it inthis fashion, as there seemed to be many more points to be gainedas compared to the percent system or the 100 point system.

The first analysis used the affective responses (motivation,confidence, effort) as the set of dependent variables. The multivari-ate F (using Wilk’s Lambda for all analyses) was F (6, 458) = 2.28,p = .035 for grading system differences; F (6, 458) = 2.21, p = .041 fordifferences by GPA; and, F (12, 606) = 1.20, p = .282 for theinteraction term.

To examine the nature of the differences among the groups, aset of univariate analyses was conducted on the three dependentvariables. Each of these variables showed statistically significant

Table 2Means and standard deviations for effort, motivation, and confidence by group and GP

Variable GPA Group

Motivation 2–3 100 points

Percent

600 points

Total

3–3.5 100 points

Percent

600 points

Total

3.5–4 100 points

Percent

600 points

Total

Total 100 points

Percent

600 points

Total

Confidence 2–3 100 points

Percent

600 points

Total

3–3.5 100 points

Percent

600 points

Total

3.5–4 100 points

Percent

600 points

Total

Total 100 points

Percent

600 points

Total

Effort 2–3 100 points

Percent

600 points

Total

3–3.5 100 points

Percent

600 points

Total

3.5–4 100 points

Percent

600 points

Total

Total 100 points

Percent

600 points

Total

differences for the main effect of grading system. For Motivation,the result was F (2, 231) = 4.10, p = .018. For Confidence, the resultwas F (2, 231) = 5.03, p = .007. For Effort, the result was F (2,231) = 3.38, p = .036. Tukey’s HSD (alpha = .05) was used as a posthoc procedure. The results indicated that for each of the dependentvariables, the 100 point model was significantly different from the600 point model, but the percent model was not significantlydifferent from either of the other two approaches. The descriptivestatistics for this analysis are presented in Table 2, and are depictedin Fig. 2 through 4. Effect sizes (Cohen’s d) were calculated for thedifference between the 100 point system and the 600 point system.For Motivation, the effect size was .52; for Confidence, it was .51;and, for Effort, it was .49. These effect sizes are remarkablyconsistent, and in the moderate range in terms of magnitude.

Grade point average was only used to look for interactions withgrading system assignment. It was not a fundamental aspect of thestudy, and is not explored further here. As mentioned, there wereno significant interactions.

The second analysis looked at the more cognitively orienteddependent variables of demonstration, self-efficacy, preference,and usefulness. The same multivariate procedure as described

A.

Mean Standard deviation N

20.52 2.40 29

20.17 3.00 18

22.53 3.73 17

20.95 3.07 64

20.69 3.21 36

22.41 3.59 37

22.61 4.27 41

21.94 3.80 114

22.67 3.20 15

22.60 4.03 25

23.73 3.87 22

23.02 3.77 62

21.00 3.01 80

21.96 3.70 80

22.90 4.04 80

21.95 3.68 240

19.00 2.54 29

19.44 3.03 18

21.47 3.61 17

19.78 3.12 64

19.75 2.75 36

20.92 2.66 37

20.76 3.10 41

20.49 2.88 114

20.47 3.80 15

20.88 2.96 25

21.86 3.63 22

21.13 3.41 62

19.61 2.91 80

20.58 2.87 80

21.21 3.36 80

20.47 3.11 240

21.69 3.57 29

22.89 3.27 18

24.94 4.42 17

22.89 3.91 64

23.83 2.75 36

24.38 3.31 37

25.15 3.33 41

24.48 3.17 114

24.87 3.38 15

25.92 3.64 25

24.86 4.25 22

25.29 3.78 62

23.25 3.38 80

24.53 3.54 80

25.03 3.79 80

24.27 3.64 240

Page 6: The impact of framing effect on student preferences for university grading systems

Fig. 2. Level of motivation by grading system and GPA. Fig. 4. Level of effort by grading system and GPA.

J.K. Smith, L.F. Smith / Studies in Educational Evaluation 35 (2009) 160–167 165

above was employed. In this analysis, the main effect for GPAwas significant, but neither the main effect for group nor theinteraction term was significant. For grading system differences,the results were F (8, 456) = 1.108, p = .357; for GPA, the resultswere F (8, 456) = 2.074, p = .037; and for the interaction, the resultswere F (16, 697) = 1.128, p = .324. Thus, it appears to be the casethat the different grading systems influenced an affective response,but not a cognitive response. The differences in GPA found heresimply show that students with higher GPAs believed that thevarious assessment techniques are useful and demonstrate theirabilities to a greater degree than students with lower GPAs. This isnot unexpected, and is not the focus of the issues underconsideration here.

The third analysis examined the remaining variable, anxiety.There were no significant differences for either main effect or

Fig. 3. Level of confidence by grading system and GPA.

for the interaction. For the grading system differences, theresults were F (2, 231) = .251, p = .778; for GPA, the results wereF (2, 231) = 1.065, p = .347; and for the interaction, the results wereF (4, 231) = .784, p = .537.

In discussing the results with the students in their classes,several interesting issues were raised. To begin, there was a generalsense among students that the 100 point system did indeed make itseem like they were losing points instead of gaining them. As onestudent put it, ‘‘I had a class like that. Every time I got a grade back, Ifelt like I was falling further off of a cliff.’’ On the other hand,students in the 600 point system felt as if they had lots of points leftover in the system and ample opportunity to improve their grade.This was particularly intriguing as, in reality, they had no moreopportunity than did students in the other two conditions, but theyfelt like they did. Students agreed that the percent system was theone that they encountered most frequently, and that they feltgenerally comfortable with that approach to tallying grades. The100 point system was the next most frequently used system,according to the students.

Discussion

To summarize the findings, there were significant differencesfor motivation, effort, and confidence between the 100 pointsystem and the 600 point system, with the percent system inbetween. The results for the grading system differences were whatwere anticipated for these dependent variables. The 100 pointmodel generated the least motivation, confidence, and anticipatedeffort. The percent model obtained the middle ratings on each ofthese variables, and the 600 point model had the highest ratings.For GPA, there were significant differences along the lines of whatmight be expected of students of different achievement levels. Theinteraction term was not significant, indicating that the systemworked roughly in the same fashion for students of all levels ofachievement. The analysis for the cognitively oriented variableswas only significant for GPA (again, in a predictable fashion). Theanalysis for anxiety was not significant for any of the terms.

It seems then, for this sample, the framing of the grading systemhad an effect on how motivated and confident students were abouttheir performance, and how much effort they anticipated putting

Page 7: The impact of framing effect on student preferences for university grading systems

J.K. Smith, L.F. Smith / Studies in Educational Evaluation 35 (2009) 160–167166

forth in the class. On the other hand, the different grading framesdo not seem to have affected participants’ perceptions of how well

they would do or whether the system would provide an accurateassessment of how well they would do. It appears from these datathat the desire to avoid loss is more of an emotional reaction than acognitive one. The setting does not seem to affect an objectiveassessment of whether the approach will accurately assess studentachievement fairly, or how well the student perceives he or shewill do, but does affect motivation, confidence in performance, andperceived effort that will be put forward. These effects are seen tobe in the moderate range in this study, and are variables known tobe related to achievement. Thus, the framing effect described byKahneman and Tversky (2000) appears to be operating with regardto how students perceive common alternatives to tallying grades atthe university level.

The results confirmed our suspicions that in the eyes of thestudents, not all systems are created equal, even when theyactually are. It is interesting to note that the 600 point system(instantiation of the open point system) was the most positivelyreceived by the participants, even though it can be argued that itis more similar to the 100 point system than the percent system(in that it is simply the 100 point system multiplied by 6 in thisinstance). Although there may be other interpretations, ourfindings are consistent with our hypothesis that in both the 600point system and the percent system, students perceive thatthere are sufficient opportunities to ‘‘catch up’’ to an A grade,whereas in the 100 point system, they focus more on the currentlosses away from the 100 maximum. The percent and 600 point(open point) systems appear to be viewed more as potential gainsettings, while the 100 point system is viewed more as apotential loss system.

The research presented here is limited by the nature of thesample studied, and the measures used in the study. It does,however, raise the issue of considering the perception of studentsof the choices we take as professors in offering our courses. Theframing effect work of Kahneman and Tversky (2000), as well asother advances of decision theory, may hold substantial potentialfor understanding some of the ‘‘how and why’’ of studentperformance in university level coursework. Student performanceinvolves a strong element of effort and involvement as well asacademic ability (Perlman, McCann, & Prust, 2007). The effort thata student puts into a course during the semester depends upon aseries of choices that the student makes. The influence of affectivevariables (motivation, self-efficacy, self-regulation) on achieve-ment at the college level has been traditionally examined throughthe use of models of motivation (Pintrich, 1989; Wolf & Smith,1995). The results here suggest that decision theory might be auseful adjunct to such approaches.

Uncited reference

Yamagishi (2003).

Appendix A. The scenarios presented to participants

A.1. Scenario 1: Percent system

Imagine that you are taking a required course in your major. In this

course, each assignment is based on 100 points and then is given a

certain weight. So, for example, an exam may be worth 15% of your

final grade.

Here are the assignments in this course with their weights:First exam 20%

Second exam 20%

Third exam 20%

Five homework

Assignments 10%

Term paper 25%

Presentation 5%

Your final grade will be based on the following:A 93–100

A� 90–92.99

B+ 87–89.99

B 83–86.99

B� 80–82.99

C+ 77–79.99

C 70–76.99

D 60–69.99

F 59.99 and below

It is now more than halfway through the semester. You have

completed the first exam, the second exam, and three of the

homework assignments. You still have one exam, two homework

assignments, your term paper, and your presentation to do.

At this point in the semester, your average is an 87.

A.2. Scenario 2: 100 point system

Imagine that you are taking a required course in your major. In this

course, each assignment is based on a certain number of points that

add up to 100 for your final grade. So, for example, a quiz might be

worth a total of five points.

Here are the assignments in this course and how many points each

assignment is worth:First exam 20

Second exam 20

Third exam 20

Five homework

Assignments 10

Term paper 25

Presentation 5

Total = 100 points.

Your final grade will be based on the following:A 93–100

A� 90–92.99

B+ 87–89.99

B 83–86.99

B� 80–82.99

C+ 77–79.99

C 70–76.99

D 60–69.99

F 59.99 and below

It is now more than halfway through the semester. You have

completed the first exam, the second exam, and three of the

homework assignments. You still have one exam, two homework

assignments, your term paper, and your presentation to do.

At this point in the semester, you have 40 points out of a possible

46 points.

A.3. Scenario 3: 600 point system

Imagine that you are taking a required course in your major. In

this course, your final grade is based on 600 points. So, for example,

an exam could be worth 150 points and a quiz could be worth 50

points.

Here are the assignments in this course with the maximum

number of points each assignment is worth:First exam 120

Second exam 120

Third exam 120

Five homework

Assignments 60

Page 8: The impact of framing effect on student preferences for university grading systems

J.K. Smith, L.F. Smith / Studies in Educational Evaluation 35 (2009) 160–167 167

Term paper 150

Presentation 30

Total = 600 points.

Your final grade will be based on the following:A 558–600

A� 540–557

B+ 522–539

B 498–521

B� 480–497

C+ 462–479

C 420–461

D 360–419

F 359 and below

It is now more than halfway through the semester. You have

completed the first exam, the second exam, and three of the

homework assignments. You still have one exam, two homework

assignments, your term paper, and your presentation to do.

At this point in the semester, you have 240 points out of a

maximum of 276 points.

References

Brookhart, S. M. (1993a). Teachers’ grading practices: Meaning and values. Journal ofEducational Measurement, 30, 123–142.

Brookhart, S. M. (1993b). Grading and classroom management: What does it mean toearn a grade? Paper presented at the annual meeting of the National Council onMeasurement in Education.

Brookhart, S. M. (2004). Grading. Columbus, OH: Pearson Education.Chong, D., & Druckman, (2007). Framing theory. Annual Review of Political Science, 10,

103–126.Dixon, C. (2004). Plus/minus grading: If given a choice. College Student Journal, 38,

280–284.

Dombach, A., & Smith, J. K. (2004, July). What course syllabi reveal about collegeinstructors’ epistemological worldviews. Poster session presented at the AmericanPsychological Association 112th Annual Convention.

Harlen, W. (2007). Criteria for evaluating systems for student assessment. Studies inEducational Evaluation, 33, 15–28.

Johnson, E. J., Hershey, J., Meszaros, J., & Kunreuther, H. (1993). Framing, probabilitydistortions, and insurance decisions. Journal of Risk and Uncertainty, 7, 35–51.

Kahneman, D., & Tversky, A. (1979). Prospect theory: An analysis of decision under risk.Econometrica, 47, 263–292.

Kahneman, D., & Tversky, A. (2000). Choices, values, and frames. Cambridge, UK:Cambridge University Press.

Perlman, B., McCann, L. I., & Prust, A. (2007). Students’ grades and ratings of perceivedeffectiveness of behaviors influencing academic performance. Teaching of Psychol-ogy, 34, 236–240.

Pintrich, P. R. (1989). The dynamic interplay of student motivation and cognition in thecollege classroom. In Ames, C., & Maehr, M. (Eds.), Advances in achievement andmotivation. vol. 6 (pp.117–160). Greenwich, CT: JAI Press.

Smith, L. F. (2009). How university students see assessment. San Diego, CA: Poster at theannual meeting of The American Educational Research Association.

Smith, L. F., & Smith, J. K. (2002). The relationship of test-specific motivation andanxiety to test performance. Psychological Reports, 91, 1011–1021.

Takemura, K. (1992). Effect of decision time on framing of decision: A case of riskychoice behavior. Psychologia, 35, 180–185.

Tversky, A., & Kahneman, D. (1974). Judgment under uncertainty: Heuristics andbiases. Science, 185, 1124–1131.

Tversky, A., & Kahneman, D. (1981). The framing of decisions and the psychology ofchoice. Science, 211, 453–458.

Tversky, A., & Kahneman, D. (1986). Rational choice and the framing of decisions.Journal of Business, 59(4), 251–278.

Wolf, L. F., & Smith, J. K. (1995). The consequence of consequence: Motivation, anxiety,and test performance. Applied Measurement in Education, 8, 227–242.

Wolf, L. F., Smith, J. K., & Birnbaum, M. E. (1995). Consequence of performance, testmotivation and mentally taxing items. Applied Measurement in Education, 8,341–352.

Yamagishi, K. (2002). Proximity, compatibility, and noncomplementarity in subjectiveprobability. Organizational Behavior and Human Decision Processes, 87, 136–155.

Yamagishi, K. (2003). Effects of valence and framing in decision making II: Estimatingsubjective weighting. Japanese Psychological Research, 45, 173–187.


Recommended