DOCUMENT RESUME

ED 333 042 TM 016 582

AUTHOR Natriello, Gary; And OthersTITLE Creating More Responsive Student Evaluation Systems

for Disadvantaged Students. Report No. 15.INSTITUTION Center for Resew:ch on Effective Schooling for

Disadvantaged Students, Baltimore, MD.SPONS AGENCY Office of Educational Research and Improvement (ED),

Washington, DC.PUB DATE Apr 91CONTRACT RI17R90002NOTE 18p.

PUB TYPE Reports - Evaluative/Feasibility (142)

EDRS PRICE MF01/PC01 Plus Postage.DESCRIPTORS *Academic Achievement; *Disadvantaged; *Educational

Assessment; Elementary School Students; ElementarySecondary Eduration; Evaluation Problems; Guidelines;Secondary 3chLJ1 Students; *Student Evaluation;Testing Problems

IDENTIFIERS *Performance Based Evaluation; *ResponsiveEvaluation

ABSTRACT

This paper examines the evaluation of studentperformance (SP) in schools in order to develop a set of strategiesfor creating more responsive systems for its evaluation, focusing ondisadvantaged students. Problems involved in the evaluation of theperformance of disadvantaged students are examined, including:evaluation purposes (direction, moti ation, certification, andselection) that are in conflict, task assignments that are overlyroutine, criteria that are unclear and narrow, standards that denydisadvantaged students the feeling of competence, testing practicesthat provide limited information on SP, appraisals that limit teacherdiscretion, feedback that is too limited, and the absence of planningfor the improvement of SP. Elements of a more responsive evaluationsystem for at-risk students include: attention to the purposes ofdirection and motivation; consideration of ta..ks prior to developingthe evaluation process; broad criteria tapping multiple abilities,processes, and non-academic domains; standards that are challengingyet attainable; a broader range of information and more variedcollection techniques; more efficient methods of collectinginformation on SP; appraisals that involve teacher discretion andbroader teacher participation; more detailed feedback; new ways ofpresenting certification/selection information to students and newways of presenting direction/motivation to others; and improvementplans that offer opportunities and resources. A 59-item list ofreferences is included. (RLC)

***********************************************************************Reproductions supplied by EDRS are the best that can be made

from the original document.***********************************************************************

U S. DEPARTMENT or EDUCATSONoe.ce or Educational Remarch end IrnDrOvOrnen1

EDUCATIONAL RESOURCES INFORMATIONCENTER IERICI

liThis document has en reproduced asreceived horn the pereCin Or OrginizatiOnoriginating it

[' Minor changes have Peen made to improvereproduction quality

Points ot view or opinions stated in this dOCu-ment do net necessarily represent otticis0OF RI position Or policy

THE JOHNS HOPKINS UNIVERSITY

Creating More Responsive

Student Evaluation Systems

for Disadvantaged Students

Gary Natriello, Aaron M. Pallas

and

Carolyn Riehl

Report No. 15

April 1991

BEST COPY AVAILABLE

National Ad. isory Panel

3eatriz Arias. College of Education, Arizona State University

Mary Frances Berry, Department of History, University of Pennsylvania

Anthony S. Bryk, Department of Education. University of Chicago

Michael Charleston, Educational Policy Studies. Pennsylvania State

University

Constance E. Clayton, Superintendent. Philadclphia Public Schools

Edmund Gordon (Chair), Department of Psychology, Yale University

Ronald a Henderson, National Education Association

inetta Jones, Dean, Education and Urban Affairs, Morgan State University

Arturo Madrid, !Director, Tomas Rivera Center

Hernan LaFontaine, Superintendent, Hartford Public Schools

Peter Stanley, Dircetor. Education and Culture Program, Ford Foundation

William Julius Wilson, Department of Sociology. University of Chicago

Cc nter Liaison

Harold Himmelfarh. Office of Educational Research and Improvement

Creating More Responsive Student Evaluation Systems

for Disadvantaged Students

Gary Natriello, Aaron M. Pallas and Carolyn Riehl

Teachers College, Columbia University

Grant No. R117 R90002

Report No. 15

March 1991

Published by the Center for Research on Effective Schooling for Disadvantaged Students,supported as a national research and development center by funds from the Office of EducationalResearch and Improvement, U.S. Department of Education. The opinions expressed in thispublication do not necessarily reflect the position or policy of the OERI, and no officialendorsement should be inferred.

Center for Research on Effective Schooling for Disadvantaged StudentsThe Johns Hopkins University

3505 North Charles StreetBaltimore, Maryland 21218

The Center

The mission of the Center for Research on Effective Schooling for Disadvantaged Students(CDS) is to significantly improve the education of disadvantaged students at each level ofschooling through new knowledge and practices produced by thorough scientific study andevaluation. The Center conducts its research in four program areas: The Early and ElementaryEducation Program, The Middle Grades and High Schools Program, the Language MinorityProgram, and the School, Family, and Community Connections Program.

The Early and Elementary Education Program

This program is working to develop, evaluate, and disseminate instructional programscapable of bringing disadvantaged students to high levels of achievement, particularly in thefundamental areas of reading, writing, and mathematics. The goal is to expand the range ofeffective alternatives which schools may use under Chapter 1 and other compensatory educationfunding and to study issues of direct relevance to federal, state, and local policy on education ofdisadvantaged students.

The Middle Grades and High Schools Program

This program is conducting research syntheses, survey analyses, and field studies in middleand high schools. The three types of projects move from basic research to useful practice.Syntheses compile and analyze existing knowledge about effective education of disadvantagedstudents. Survey analyses identify and describe current programs, practices, and trends in middleand high schools, and allow studies of their effects. Field studies are conducted in collaborationwith school staffs to develop and evaluate effective programs and practices.

The Language Minority Program

This program represents a collaborative effort. The University of California at SantaBarbara is focusing on the education of Mexican-American students in California and Texas;studies of dropout among children of recent immigrants are being conducted in San Diego andMiami by Johns Hopkins, and evaluations of learning strategies in schools serving Navajo,Cherokee, and Lumbee Indians are being conducted by the University of Northern Arizona. Thegoal of the program is to identify, develop, and evaluate effective programs for disadvantagedHispanic, American Indian, Southeast Asian, and other language minority children.

The School, Family, and Community Connections Program

This program is focusing on the key connections between schools and families and betweenschools and communities to build better educational programs for disadvantaged children andyouth. Initial work is seeking to provide a research base concerning the most effective ways forschools to interact with and assist parents of disadvantaged students and interact with thecommunity to produce effective commnnity involvement.

5

Abstract

This paper examines the evaluation of student performance in schools in order to develop a

set of strategies for creating more responsive systems for the evaluation of student performance.

The paper examines problems involved in evaluation of the performance of disadvantaged

students, including problems in the purposes of evaluation, task assignment, criteria, standards,

testing, appraisal, feedback, and planning for improvement. A set of guidelines is derived for

creating evaluation systems that would be more responsive to disadvantaged students.

111

Introduction

This paper examines the evaluation of student per-formance in schools in order to develop a set ofstrategies for creating more responsive systems forthe evaluation of student performance. The conceptof responsive evaluation systems derives from amore general concept of responsive schools. Re-sponsive schools are those in which student per-formance and participation are met with timely andappropriate reactions from adults (Natriello,Mc Dill, and Pallas, 1990; McPartland and Mc Dill,1977). In responsive schools, staff recognize andreward good student performance and identify andpenalize poor student performance. Moreover,school staff identify the larger causes of poor stu-dent performance and develop remedies, both inter-personal and structural, to create conditions underwhich students can succeed.

Evaluation processes are key elements in schools(Natriello, 1987). Student performance and behav -ior are routinely monitored and evaluated. Indeed,many of the activities of teachers, from formaltesting to informal observations and comments, canbe seen as evaluation of student performance in thecontcxt of the instructional and extracurricular pro-

gram of the school. It is through evaluations thatthe school attempts to influence the development ofstudent.s. Thus, evaluations and student evaluationsystems are a central element in any attempt to create more responsive school environments.

The process of evaluating student performance inschools and classrooms is complex and involves avariety of activities. Each of these elements maypresent problems for disadvantaged students.Evaluation systems are often characterized by (1)purposes that are in conflict, (2) tasks that tend tobe overly routine, (3) criteria that are unclear andnarrow, (4) standards that deny disadvantagedstudents the feeling of competence, (5) tests thatprovide limited information on student perfor-mance, (6) appraisals that limit teacher discretion,(7) feedback that is too limited, and (8) the absenceof plans for the improvemunt of student perfor-mance. We consider these and other problems inevaluation systems as a prelude to developing aseries of guidelines for creating evaluation systemsthat would be more responsive to disadvantagedstudents.

Problems in Evaluation Systems for Disadvantaged Students

Purposes In Conflict

There are at least four purposes for student evalua-tion systems: direction, motivation, certification,and selection. Direction refers to the goal of pro -viding students with clear and specific informationon their performance and how to modify it alongthe lines desired in the school. Motivation involvessecuring the cooperation and commitment of thosebeing evaluated to the tasks at hand. Both of thesepurposes are compatible with the development ofresponsive evaluation systems. Evaluation activities oriented to these purposes can lead to schooland classroom practices that respond to the needsand performance problems of all students, particu-larly disadvantaged students, who often lack accessto external sources of motivation and direction.

However, the other two purposes for evaluationsystems, certification and selection, do not neces-sarily lead to activities that respond to studentneeds. Certification and selection respond to theneeds of others, not to the needs of students. Cer-tification provides assurance to others that a student

has attained a certain level of accomplishment ormastery. Selection is the identification of somestudents for future educational or occupational op-portunities. Although certification and selection of-fer certain benefits to those students who arc certifled and/or selected, they lead to practices thatcontribute little to the development of students.

The tension between the purposes of certificationand selection and those of direction and motivationare only apparent in the case of the evaluation ofyoungsters who have trouble succeeding in school.Students with reasonable chances of success profitfrom the certification and selection functions ofevaluations as well as from the direction and moti -vation functions. For disadvantaged students, whoare unlikely to experience success as a result 01those practices set in motion by the certification andselection functions of evaluation, the dircction andmotivation functions become even more important.However, the basic tension between direction andmotivation, on the one hand, and certification andselection, on the other, creates problems in design-

ing all aspects of evaluation processes fordisadvantaged students.

Assigning Tasks: Between Boredom andAnxiety

Evaluation processes set up situations in which allstudents are at risk of failure. The level of riskvaries with the nature of the tasks on which stu -dents are being evaluated. Dornbusch and Scott(1975:80) have suggested that tasks differ in pre-dictability, that is, "the extent to which the per-former has knowledge of which path is most likelyto lead to success." Furthermore, they demon-strated that when tasks are predictable, performersprefer that they be given the specific procedures tobe followed to execute the tasks, but that Wientasks are less predictable, performers prefer theautonomy to make their own decisions about howto perform the task.

Predictable tasks accompanied by specificprocedures to be followed carry relatively less riskthan less predictable tasks. In an analysis ofclassroom tasks Doyle (1983) classified opiniontasks and certain memory tasks (i.e., thoseinvolving the reproduction of small amounts ofmaterial) and certain routine tasks (i.e., those re-quiring relatively simple algorithms) as low in risk,and understanding tasks and other memory tasks(i.e., those involving the reproduction of largeamounts of material) and other routine tasks (i.e.,those involving complicated procedures) as high inrisk.

Research in classrooms has revealed two forces atwork to reduce the risk associated with the evalua-tion process. First, Davis and McNight (1976) andWilson (1976) showed that students resist attemptsto shift from low risk routine or procedural tasks tohigh risk understanding tasks in classrooms.Moreover, students also attempt to renegotiate as-signed tasks so that they are more predictable andentail less risk.

Second, there appears to be a tendency to structureevaluation activities as if the tasks being evaluatedare predictable. For example, Armbruster,Stevens, and Rosenshine (1977), in a study ofthree reading curricula, found that although thetexts emphasized comprehension and interpretationskills, the tests solicited factual information fromstudents based on the ability to locate informationin the text. Bailey, Brazee, Chi avaroli, Herbeck,Lechner, Lewis, McKittrick, Redwine, Reid,Robinson, and Spear (1988) found that despite anintention to view reading and writing as meaning-centered processes, the evaluation system tended toencourage practice in isolated drill and skill

2

exercises. To counteract this it was necessary tocreate evaluation and reporting instruments whichmonitored the extent to which students took risks asreaders and writers to engage in learning tasksconsistent with the meaning-centered approach.

Studies of typical classrooms that reveal a tendencyto avoid less predictable tasks and utilize routinetasks to minimize the risk associated with evalua-tion processes suggest that these tendencies may beeven more severe in classes serving disadvantagedyoungsters. Such youngsters are even more likelyto experience failure under traditional evaluationprocesses and so face even greater risks when confronting less routine tasks. However, the retreatfnam unpredictable tasks to minimize risk and anx-iety may leave disadvantaged students in class-rooms where the instructional tasks are insuffi-ciently interesting to hold their attention (Oakes,1985).

Criteria: Focusing Attention or NarrowingOpportunity

Criteria are those properties of tasks that are takeninto account when making an evaluation of perfor-mance. For example, in assessing student perfor-mance in writing, teachers may apply criteria re-garding creativity and imagination as well as criteriapertaining to correct grammar and proper usage.

Several issues surrounding the development anduse of performance criteria have important implica -tions for the evaluation of disadvantaged students.Studies of the impact of evaluation processes onstudents have revealed that criteria are often notclearly communicated to students (Natriello, 1982)and that criteria are often not linked to the instruc-tional program (Linn, 1983; Rudman, Kelly,Wanous, Mehrens, Clark, and Porter, 1980). Eachof these problems presents difficulties for all stu-dents, but the difficulties may be worse for disad-vantaged students, who lack family resources tohelp them understand the demands of their teachersand direct their efforts accordingly.

Efforts to clearly articulate criteria and link them tothe instructional process have the potential forimproving the evaluation process for disadvantagedyouth by focusing attention on the most importantaspects of the curriculum. If these efforts at greaterarticulation and alignment result in a narrowing ofthe criteria and a specification of tasks to the pointwhere routine tasks dominate the classroom,however, they also have the potential to createadditional problems for disadvantaged youth. Thenarrowing of criteria for the evaluation ofperformance results in a reduction of opportunitiesto experience success in the classroom. For

8

disadvantaged students, who typically have dif-ficulty with academic work, a reduction in oppor-tunities to succeed may have serious negative ef-fects on motivation. Moreover, if classroom tasksbecome overly routinized, students may perceivethem as boring and uninteresting. There are prac-tices and special efforts to broaden the base of criteria for evaluations. As Thomdike ( 1967) notes:

In practice, certainly many otherconsiderations than that of pure competence doenter into marks. Such factors enter in as ( 1 )industry and effort -- i.e., completing allassigned work and even doing optional workfor "extra credit" (a kind of educational bribe);(2) frequent and active participation in classdiscussion: (3) neatness in written work andmechanical correctness in such areas asspelling and grammar; and (4) personalagreeableness, attracdveness, cleanliness, anddocility. To some extent and by someinstructors, certain of these factors would beendorsed as legitimate influences on a mark.Others would more uniformly be accepted asextraneous influences to be minimized as faras possible. (p. 762)

Holmes ( 1978) notes that other criteria which makeit possible for an organization to operate oftencovertly enter into the evaluation process. Theseinclude criteria related to behavior and effort suchas politeness, conformity, and perseverance. Salganik (1982) reports on a system in which studentswoe evaluated on three criteria -- achievement, ef-fort, and conduct. In a national survey of sec-ondary school teachers, Natriello and McPartland(1987) found that student effort was "very impor-tant" or "extremely important" in the evaluationprocess of over 70% of the teachers.

Many of the strategies for broadening the set ofcriteria upon which evaluations are based involveincluding criteria of a more formative nature (i.e.,those factors, such as effort and perseverance, thatcontribute to ultimate achievement) as well as sum -mative factors. Rudman (1987) discusses this as astrategy of linking assessment with teaching andBroadfoot (1986) sees it as attending to the curricu-lum function of reporting systems as well as to thecommunication funcdons. Parkay (1987) providessome initial evidence that such approaches may beparticularly effective for disadvantaged students.He presents evidence indicating that students in re-medial English classes in an urban high schooldemonstrated greater knowledge and more favor-able attitudes toward the subject matter after beinggraded for a 3-week period on just the completionof assignments instead of traditional grading proce-

3

dures which take into account the quality of thework completed.

Broadening the base of criteria upon which studentsare evaluated off rs increased opportunities for dis -advantaged youngsters to succeed in schools. Ofcourse, one strategy is simply expanding the rangeof the academic performance criteria. One approachto providing such opportunities for success for stu-dents who would otherwise experience consistentfailure in the classroom involves a restructuring ofthe tasks of the classroom so that they draw on awider range of ability dimensions. Such multipieability classrooms (Cohen, 1986; Rosenholtz,1977) attempt to move beyond the narrow range ofacademic tasks and criteria, most of which relyupon reading skills, so that every student can expe-rience some success. In the multiple-ability class-room, the intention is for all students to find sometask at which they can experience a sense ofcompetence.

Standards: Between A Rock and A HardPlace

Criteria are those dimensions of tasks upon whichevaluations will be based. Standards are the levelsof performance expected on the criteria. Researchand commentary on appropriate standards for theeviluation of student performance have focused onthree types of standards: those set in reference tothe criterion level of a group. those set in referenceto some absolute criterion level, and those set inreference to the previous criterion level of an indi-vidual (Wise and Newman, 1975; Rheinberg,1983; Thorndike, 1969). Each of these types ofstandards has different implications for the evalua-tion of disadvantaged students.

Terwilliger (1977) links norm-referenced standardsto what he terms the pragmatic philosophy, a view-point primarily concerned with practical choices andthe consequences of such choices. An evaluationsystem which differentiates among individual stu -dents is optimal for identifying the choices availableto those who make decisions about admission tofuture educational and employment opportunities.Thus, norm-referenced standards serve the purposeof selection identified earlier.

Norm-referenced standards have been criticized inmany quarters. Bresee (1976) lists a series ofproblems with such standards:

(1) the conflict between the necessity of producinga normal distribution of grades and the goal ofhaving teachers produce improvement in all students in a class;

(2) the distortion of the curriculum as teachers seekto diversify instructional objectives to produce arange of achievement in a class;

(3) the diversion of student attention from the taskat hand to the performance of other students; and

(4) the introduction of false competition becauseachievement is not really in limited supply.

To these Deutsch (1979) adds: (1) the distortion ofthe testing process so that tests take the form ofcontests in which all performers participate underuniform conditions; (2) the lack of rewards createdby the artificial scarcity of good grades that is likelyto impede the development of students' sense oftheir own value; and (3) the encouragement ofcompetition which may be counterproductive fortasks requiring cooperation and communication.Much of the cridcism of norm-referenced standardshas to do with their being ill-suited to the purposesof certification (Glaser, 1963), direction and moti -vation (Beady and Slavin, 1981).

Absolute or criterion-referenced standards haveenjoyed increased support in recent years. They arebased on some externally determined level of per-formance. Terwilliger (1977) associates such stan-dards with the behaviorist perspective. The actualstrategy for setting such absolute standards hasbeen challenged by Glass (1978) and Burton(1978), who conclude that they must be set arbi-trarily, and by Shepard (1976), who concludes thatall current methods of setting absolute standards re-duce to some form of norm-referenced standards.Criterion-referenced standards may serve the pur-pose of certification, but they often fail to providemotivation and direction to students, particularlywhen decisions about standards are removed fromthe informed professional opinion of teachers(Burton, 1978).

Individually-referenced or self-referenced standardsare based on comparing a student's current perfor-mance with some other feature of the student.Terwilliger (1977) points out that current perfor -mance may be compared with earlier performanceor with a student's ability. Terwilliger ( 1977) as-sociates self-referenced standards with the humanistview of education, and sees them as a strategy torecognize individual differences, reward effort, andfoster motivation.

Disadvantaged students are likely to have limitedopportunities to experience success under noun -referenced or criterion-referenced standards.Systems based at least in part on self-referencedstandards would offer disadvantaged students somesense of competence. The dilemma, of course, is

4

that systems that rely solely upon self-referencedstandards deny students information on how theirperformance may be viewed by a broader audience.Such systems may fail to challenge students andleave them with misconceptions about their ownperformance (Natriello and Dornbusch, 1984;McDill, Natriello, and Pallas, 1985).

Collecting Information: Tests and Contests

Collecting information on student performance forthe purposes of evaluation is a demanding and time-consuming task. Because it is impractical if notimpossible to collect complete information on stu-dent performance, the collection of such informa-tion involves sampling decisions. The challenge isto insure that the information collected provides avalid and reliable estimate of performance appro-priate to the purposes, tasks, criteria, and standardsof the evaluation process. A number of analystshave contributed important observations about therelationship between the dominant method of col-lecting performance information -- testing -- and thepurposes, tasks, criteria, and standards for evalua-tion. Deutsch (1979) argues that the structure ofmost testing situations is a reflection of the prevail-ing purpose of evaluation (selection) and the typesof standards utilized (nonn-referenced). He notesthat:

The social context of most educationalmeasurement is that of a contest in whichstudents are measured primarily in comparisonwith one another rather than in terms ofobjective accomplishment. If educationalmeasurement is not mainly in the form of acontest, why are students often asked to revealtheir knowledge and skills in carefullyregulated test situations designed to be asuniform as possible in dme, atmosphere, andconditions for all students? Individuals varyenormously in terms of the amount of timethey need and the kind of atmosphere andcircumstances thLi facilitate or hinder theirexpression of their knowledge and skills; it isonly the comparison of students with oneanother that requires measures of educationalachievement that take the form of contests(Deutsch, 1979: 394).

As long as testing is constructed as a contest with alimited number of winners, disadvantaged studentsare likely to turn out the losers. Deutsch goes on todescribe the damaging effects of norm-referencedstandards for individual students and advocates anevaluation system that would provide individual-ized, particularistic feedback to students to fostertheir development. Thus, his objection to the typi -

1 0

cal testing situation is rooted in a rejection of evalu-ation for the purpose of selection and of standardsthat are norm-referenced. Of course, both selectionand norm-referenced standards are likely to lead topractices inimical to disadvantaged students.

Although much of what has been written on thccollection of information on student perfonnancehas centered on tests and testing, there is a range ofalternative methods of collecting such information.For example, Gaston (1976) suggests monitoringstudents' unassigned reading in the library and lis-tening to student conversations as they leave theclassroom as a way to collect information on stu-dent attitudes and behavior. Heller ( 1978) sug-gests alternatives to standardized reading tests suchas the use of reading materials from popularmagazines, fables, and poems. Solo ( 1977) ex-plains how alternatives such as anecdotal recordsand collections of students' daily work may be usedto provide insight into student performance. Her-man and Dorr- Bremme ( 1984) note a variety oftechniques used by teachers to collect informationon student performance, including routine class andhomework assignments, classroom interactionduring question and answer sessions, recitations,discussions, oral reading, problem-solving at thechalkboard, special projects, presentations, and re-ports.

Such alternative methods of collecting informationon performance may provide disadvantagedstudents with a wider variety of ways todemonstrate their competence. At the very least,they provide those who work with disadvantagedstudents with a broader base of information forunderstanding their academic performance andneeds.

Appraisals: Sensitivity versus Bias

Appraisal involves comparing the information col-lected on student performance on assigned taskswith the criteria and standards previously estab-lished for those tasks. But appraisal is a morecomplex process than this description would sug-gest. As Dombusch and Scott (1975) observe:

The application of standards in specificsituations is rarely a simple or straightforwardprocedure. It requires judgment with rtspectto the comparability of the performancesituation and the situations for which thestandards are considered applicable ...appraisal is seldom a mechanical procedure.In short, appraisal entails deciding how tointerpret a low or high performance score.Accurately appraising a task performancerequires knowledge of extenuating

5

circumstances, whether it be the inexperienceof the task performer, thc lack of facilities, orassistance received from a more skilled co-worker. Such information is of criticalimportance in determining what, if any,message is to be communi cated to theperformer concerning the quality of his or hertask performance (p. 143).

But the exercise of discretion that is so critical toproducing an appropriate appraisal is viewed asdangerous in the case of teachers appraising theperformance of students (Archer and McCarthy,1988), particularly disadvantaged students. 'Thefear, of course, is that teachers will be biasedagainst disadvantaged students either by failing torecognize their strengths (e.g., Rosenthal and Ja-cobson, 1968; Rist, 1970), or failing to challengethem to perform to their capacity (Natriello andDornbusch, 1984), or both.

This fear has led teachers and those who supervisethem to seek to reduce teacher discretion in theevaluation process. This can be done throughincreased used of standardized tests and throughteacher grading practices that are more formulaicand allow less room for judgment. These practicesare seen as easier to defend, if they art challenged(Natriello, 1982). However, the dilemma is thatsuch practices limit the exercise of teacherdiscretion which may be important to developingevaluations that provide sufficient direction andmotivation to disadvantaged students. Teachers,those individuals in the best position to bc moresensitive to students, their performance, and theircondition, may be overly constrained by fears ofpossible bias. Of course, nei ther teacher bias norconstrained teacher discretion is beneficial todisadvantaged students.

Full teacher discretion without bias represents thebest condition for appraisal of the work of all stu-dents, including disadvantaged students. One suat-egy for achieving such conditions might be tobroaden the production of and the audience forteacher appraisals. Currently, most teacher ap-praisals of student performance are the product of asingle teacher and are meant for a single student andperhaps his or her parents. But it is possible to in-volve more than one teacher in the appraisal processor for teachers to share their appraisals with eachother. These processes would prescrve and per-haps even increase teacher discretion while provid-ing checks and balances on the possible biases ofany single teacher. To encourage teachers to worktogether on the appraisal of student performance itwill be necessary to overcome teachers' habits ofnot relying on information from other teachers(Dorr-Bitmme, 1983).

1 1

Feedback: The Economies of ProvidingInformation

Providing feedback on performance evaluations isan important component of the evaluation process.If the results of evaluations are never communi -cated, there is little reason to expect them to have aneffect. The results of evaluations may be commu-nicated to multiple audiences, including the student,parents, school officials, and potential employers(Ahmann and Glock, 1967).

The feedback process can be quite varied and con-tribute to the multiple purposes of evaluations.Slavin (1978) distinguishes three types of feedbackregarding student performance: feedback that tellsstudents where they stand compared to other stu -dents, based on norm-referenced standards; feed -back that provides students with information ontheir on-going performance, based on criterion-ref-erenced standards; and feedback that enhances stu-dent motivation, based on timely assessments ontasks that arc neither too difficult nor too easy.Lissman and Paetziod ( 1983) distinguish betweeninformative feedback and motivational feedback.

Feedback creates several dilemmas for the evalua-tion of disadvantaged students. First, differentkinds of feedback will often result in quite differenteffects on students. For example, disadvantagedstudents may be achieving quite poorly in terms ofnorms for the broader population of students, butthey may be achieving quite well in terms of theirprevious performance. If students receive informa-tion on their performance in relation to the nomthey may be disheartened. If they receive informa -don on their performance in relation to their previ-ous performance, they may be misled about theiropportunities in the broader society. If they receiveboth kinds of feedback they may be quite confusedas to where they actually stand. Thus, providingthe full range of feedback to disadvantaged studentsis a complex undertaking.

Second, providing feedback is a time-consumingprocess that is often made manageable by the re-duction of information on student performance tocommon symbols such as vades. Although thesesymbols may serve the nwk of external audiencesquite well (e.g., prospecdve employers), they oftendo not serve the needs of students for detailed in-formation on performance problems to shape futureefforts to achieve. Moreover, shorthand symbolsoften do not serve the needs of school officials,who must match student needs with availableschool resources. A grade of "D" given to 2student.% may result from very different academicproblems requiring different remedies, yet thesingle grade fails to communicate a clear and

6

detailed meaning tied to a range of possibleresponses.

Detailed feedback seems to increase students'motivation and performance. Butler and Nisan(1986) found that sixth graders scored higher onboth a quantitative task and a qualitative (divergentthinking) task when they niceived task-relatedcomments on their performance. In contrast,students receiving grades scored high on thequantitative task and low on the qualitative task,and students receiving no feedback scored low onboth tasks. Moreover, students receiving writtencomments expressed greater interest in the tasksthan students in the other two groups.

There may be particular advantages to using de-tailed feedback with disadvantaged students. Maasde Brouwer ( 1986) reports on several studies indi-cating that initially poor performing students profitmore from detailed feedback and criterion-centeredfeedback, and that these effects occur in the aca -demic, social, and affective domains.

The dilemma is that providing detailed feedback re-quires a great deal of time, yet detailed feedbackprovided shortly after the performance being evalu-ated is what students, particularly disadvantagedstudents, require to plan future efforts accordingly.A number of investigators are developing alterna-tives to traditional feedback processes. Rudman(1976) advocates the use of reporting devicc !! suchas checklists that are more closely related to themechanisms for recording student performance.Such mechanisms may allow greater efficiency inthe feedback process. Ediger (1975) suggests morefrequent and more varied mechanisms for reportingstudents' performance, including telephone andface-to-face conferences with parents. Giannangeloand Lee (1974) and Giannangelo (1975) describe asystem of computer-assisted reporting that providesmore anecdotal information on studentperformance. Holtz ( 1976) presents a reportingmethod for students' performance in elementaryscience more clearly related to evaluation criteria.

Natriello (1982) reports on non-traditional methodssuch as the use of cassette tapes of teachers'comments, open gradebooks that students canexamine at any time, and procedures for involvingstudents in tallying their own cumulative scores atvarious points in the grading period. Walling (1975) discusses five broad categories of n.poningtechniques -- traditional grades, percentage ratings,checklists of objectives, narrative evaluations, andconferences. Stewart (1975) describes a multi-dimensional reporting system for use in elementaryschools.

1 2

Investigators and practitioners are also consideringthe use of student profiles or portfolios, compositesof various kinds of information on student perfor-mance (Bank & Williams, 1987; Mayfield, ;983;Phoenix Union High School Disnict, 1981). With-ers (1986) presents a typology of the various kindsof profiles and Broadfoot ( 1986) describes onesuch profile, the record of achievement in use inEngland and Wales.

Planning for Improvement

In terms of improving student perfonnance, evalua-tions of student performance that end with thecommunication of the evaluation may have little im -pact, though such evaluation processes may servethe purposes of certification and selection. Simplycommunicating an evaluation is unlikely to provideadequate direction and motivation for students.Thus, it is important to link the evaluation processwith efforts to plan and deliver instructional orother services that can help students, especially dis-advantaged students, to succeed in school.

In particular, evaluative information should enableschools to identify student needs more accuratelyand to assign school resources -- instruction, coun-

seling, books and materials, and so on -- where andwhen they are needed by students. Unfortunately,most schools are not equipped to respond quicklyand appropriately on the basis of information aboutstudent performance. &Went assessment informa-tion is used in school plann'ng efforts only in Ciemost rudimentary ways -- for example, ininfrequent decisions about student courseplacement. Even teachers, who work most directlywith students, base their planning decisions onfactors other than student assessments (Clark &Peterson, 1986; Shavelson & Stern, 1981;Stiggins, Conklin. & Bridgeford, 1986).

Often, school staff plan on the basis of trends in theassessments of previous cohorts of students,because they cannot obtain and process currentassessment information quickly enough. Whenteachers and other school staff do not act on thebasis of student assessment information, studentsand their parents must assume the major burden ofdeveloping appropriate responses to evaluativeinformation. This is especially problematic fordisadvantaged students who lack the personal,family, and community resources to interpretevaluations and react appropriately to them.

Elements of a More Responsive Evaluation System for At-Risk Students

The literature on the evaluation of students inschools and classtooms provides a number of in-sights for the design of evaluation systems for at-risk students. The same systems that are designedwith disadvantaged students in mind are also likelyto be appropriate for all students. These systemswould incorporate the following elements.

1. Attention to the Purposes of Directionand Motivation

Because evaluation systems typically encompassmultiple purposes, it is useful to consider each ofthe pumoses in the design of the system. Becauseevaluation systems have most often neglected thepurposes of direction and motivation, it is particu-larly important to consider these purposes whendesigning evaluation systems for disadvantagedstudents. This can be done by considering each ofthe elements of the system in terms of these pur-poses.

2. Consideration of Tasks Prior to Devel-oping the Evaluation Process

Existing evidence indicates that evaluation pro-cesses often lead to the development or interprets-

7

tion of tasks in ways that minimize the risk experi-enced by those being evaluated by resorting to rou -tinization. Such routinization, while offering someprotection to students from evaluation processesunder conditions of uncertainty, also leads to lessmotivating school experiences. These trends areeven more pronounced in classes and programsserving disadvantaged students.

To avoid such a trend it is necessary to consider thenature of the tasks assigned to students to determinehow best to include less routine and more engagingtasks, interesting tasks that carry reasonable risks.Considering tasks prior to developing the evalua-tion process should prevent the evaluation processfrom driving the selection and definition of tasks.In this way educators can assure an adequate repre-sentation of non-routine tasks in the school experi -ences of disadvantaged students. Some proportionof routine tasks might be included to offer studentsgreater security in terms of being able to securegood evaluations for following the specifiedcourses of action, particularly when such tasks areintegral to mastering a certain body of knowledge.

1 3

3. Broad Criteria Tapping Multiple Abili-ties, Processes, and Non-academic Do-mains

Current evaluation systems have been criticized forfocusing on overly narrow conceptions of what islegitimate to learn in school and what kinds ofcompetencies should be assessed. Educators ofdisadvantaged students should develop broadercriteria for the evaluation of students. Such broad-ening of evaluation criteria can take place in threeways.

First, criteria for academic task performance can bebroadened to encompass more areas of com-petence. It might be particularly important to de-velop criteria that are not closely related to readingability. Second, criteria can be enlarged to includethe processes that lead to successful task comple-tion as well as the final results of tasks. Devotingattention to process criteria will have both directingand motivating effects as students receive both bet-ter directions on how to approach tasks and rein-forcement for devoting the effort to specified pro-cesses. Third, evaluation criteria can include non-academic areas of student performance and devel-opment. Social and affective dimensions of studentperformance often play a role in their ability to suc -ceed in school, and an evaluation system can bothdirect and motivate students to appropriate non-aca-demic behavior in ultimate support of their am -demic performance.

4. Standards that are Challenging yetAt tains ble

Developing appropriate standards for disadvantagedstudents is not as simple as the recent school reformmovement would suggest. It is not just a questionof raising standards to challenge students to putforth greater effort and realize greater achievement.There are at least three kinds of standards that mustbe considered in evaluating students: absolute stan -dards, relative or norm-referenced standards, andindividual or self-referenced standards. Evaluationsystems that primarily serve the purpose of selec-tion place emphasis on norm-referenced standards,while those that primarily serve the purpose of cer-tification place emphasis on absolute standards.

Systems of evaluation designed to direct and moti-vate students should include self-referenced stan-dards which allow students to gain a sense of theirown progress. In reality, most evaluation systemscontain elements of each of these three types ofstandards. The difficulty is in deciding how tocombine different kinds of standards to develop anevaluation of students. A key to any solution tothis situation is to understand clearly which type of

8

standard is being applied to produce any givenassessment.

S. A Broader Range of Information andMore Varied Collection Techniques

The dominant techniques for collecting informationon student performance are limited in scope and inthe conditions under which they are employed.Moreover, techniques such as formal testing arebetter suited to the selection and certification pur-poses of evaluation than to direction and motiva-tion. Techniques of gathering information on per-formance under varying conditions must be devel-oped to serve the direction and motivation purposesof evaluation. Particular attention must be devotedto makirib data collection experiences informativeand engaging for disadvantaged students.

One strategy would be to model information-gatheiing activities after real-life experiences of thestudents, i.e., collect information under conditionssimilar to those in which students are likely to findthemselves in the community or in the workplace.One implication of such a strategy would be toeliminate assessment situations that force studentsto perform within artificial time limits. Moreover,such information gathering should extend over abroader domain of performance if we hope tounderstand the nuances of the performancedifficulties experienced by disadvantaged students.

6. More Efficient Methods of CollectingInformation on Student Performance

Because collecting information on student perfor-mance is labor intensive, any attempt to broaden therange of data and the range of conditions underwhich they are collected must be accompanied bycareful thinking about how to make the collection ofsuch performance information more efficient.Teachers have limited time for evaluation activities,and they must be helped to develop less labor-in-tensive strategies for assembling information onperformance. Another approach would be to in -volve teacher aides in the collection of the broaderrange of information on performance and leave thedevelopment of the appraisal to the teacher.

7. Appraisals that Involve Teacher Dis-cretion and Broader Teacher Participation

The dominant ways of evaluating students leave toolittle discretion for teacher participation. This isonce again due to the selection and certificationpurposes of evaluation. If evaluations are to bccompared across students or against a commonstandard, it ,s necessary to severely constrain theteachers' role in developing and executing the eval-

1 4

uation activities. However, teachers are in the bestposition to structure evaluation activities to rein-force the instructional program and contribute to thedirection and motivation pwposes of evaluation.Teachers should be encouraged and taught how todevelop evaluation processes that support learningand student development Teachers should be ableto "invent" evaluation processes that match theneeds of their students.

8. More Detailed Feedback

It is clear that disadvantaged students need andprofit from more detailed feedback on their perfor-mance. Detailed feedback that is related to thelearning tasks serves not only to communicate theresult of one performance assessment cycle but alsoto begin the neat round of instruction. Providingstudents with detailed feedback requires a great dealof time. Teachers should be assisted to developfeedback strategies that result in the most detailedfeedback within the time constraints of their posi-tions. However, serious attention to detailed feed-back is likely to require greater resources in termsof teacher time, perhaps through alterations in theschool schedule.

9. New Ways to Present Certification/Selection Information to Students and NewWays to Present Direction/Motivation toOthers

Evaluation systems often present summary infor-mation on student performance to students and out-siders. For disadvantaged students, such informa-tion is often a communication of failure. For out-siders such information is often a communication ofthe failure of disadvantaged students relative to lessdisadvantaged students. In each case the communi-cation presents pmblems. Disadvantaged studentsneed to understand how they compare to ahem andto established standards, but they need to understand this in the context of their own development.

9

Disadvantaged students can be making progressand still receive consistently negative informationon their standing in relation to others or establishedstandards. They often lose sight of the former inthe presentation of the latter.

Teachers, administrators, counselors, and parentsalso need to receive information on studentperformance in broader terms than just the relativecomparisons if they are to move beyond thestereotypical assessment of disadvantaged studentsto developing creative solutions that take advantageof more complex patterns of student development.

10. Improvement Plans that Offer Oppor-tunities and Resources

It does little good to communicate the results of anevaluation to students if we are not prepared tosupply them with the resources they need to improve their performance. This is particularly truefor disadvantaged students, for whom theevaluative information is often negative. Perhapswe should adopt a rule or thumb that the onlyevaluative information that may be communicated isthat which is accompanied by a plan forhnprovemera complete with the appropriate learningresources. Evaluations would conclude not withassessments of performance but withassessmenVopportunity packages that are conveyedto students.

Incorporating these ten elements into a system forevaluating student performance would produceevaluation systems that differ dramatically fromthose presently operating in most schools. Theseelements constitute a strategy for taking more seri -ously the direction and motivation functions ofevaluation systems. As such, they should lead tomore responsive schools for all students.

1 5

References

Ahmann, J., & Glock, M. D. (1967). Evaluating pupil growth: Principles of tests and measurements. (4thedition). Boston: Allyn and Bacon.

Archer, J. & McCarthy, B. (1988). Personal biases in student assessment. EducationalResearch, 30, 142-145.

Armbruster, B. B., Stevens, R. J., & Rosenshine, B. (1977). Analyzing content coverage and emphasis: Astudy of three curricula and two tests. (Tech. Rep. No. 26). Urbana: University Illinois, Center for theStudy of Reading.

Bailey, J., Brazee, P. E., Chiavaroli, S., Herbeck, J. , Lechner, T., Lewis, D., McKittrick, A., Redwine,L., Reid, K,, Robinson, B. & Spear, H. (1988). Problem solving our way to alternative evaluation pro-cedures. Language Arts , 65, 364-373.

Bank, A., & Williams, R. C. (1987). Infornwtion systems and school improvement: Inventing the future.New York: Teachers College Press.

Beady, C., & Slavin, R. (1981). Making success available to all students in desegregated schools: An exper-iment. Paper presented at the annual meeting of the American Educational Research Association, Los An-geles.

Bresee, C. W. (1976). On "Grading on the Curve.".. The Clearing House, 5 , 108-110.

Bmadfoot, P. (1986). Records of achievement: Achieving a record? Studies in Educational Evaluation, 12,

313-323.

Burton, N.W. (1978). Societal standards. Journal of Educational Measurement, 15, 263-271.

Butler, R. & Nisan, M. (1986). Effects of no feedback, task- related comments and grades on intrinsic mo-tivation and performance. Journal of Educational Psychology, 78, 210-216.

Clark, C. M., & Peterson, P. L. (1986). Teachers' thought processes. In M. C. Wittrock (Ed.), Handbookof research on teaching (3rd ed.). (pp. 255- 296). New York: Macmillan Publishing Company.

Cohen, E. G. (1986). Designing groupwork: Strategies for the heterogeneous classroom. New York:Teachers College Press.

Davis, R. B., & McNight, C. ( 1976). Conceptual, heuristic, and S-algorithmic approaches in mathematicsteaching. Journal of Children's Mathematical Behavior, 1 (Supplement 1), 271-286.

Deutsch, M. (1979). Education and distributive justice: Some reflections on grading systems. AmericanPsychologist, 34, 391-401.

Dornbusch, S. M., & Scott, W. R. ( 1975). Evaluation and the exercise of authority. San Francisco:Jossey-Bass.

Dorr-Bremme, D. W. (1983). Assessing students: Teachers' routine practices and reasoning. EvaluationComment, 6(4), 1-12. (ERIC Document Reproduction Service No. ED 242 742).

Doyle, W. (1983). Academic work. Review of Educational Research, 53, 159-199.

Ediger, M. (1975). Reporting pupil progress: Alternative to grading. Educational Leadership, 32, 265-267.

Gaston, N. (1976). Evaluation in the affective domain. Journal of Business Education, 52, 134-136.

1 0 16

Giannangelo, D. M. (1975). Make repon cards meaningful. The F.,ducational Forwn, 39, 409-415.

Giannangelo, D. M., & Lee, K. Y. (1974). At last: Meaningful report cards. Phi Delta Kappan, 55, 630-631.

Glaser, R. (1963). Instructional technology and the measurement of learning outcomes. American Psychol-ogist, 18, 519-521.

Glass, C.V. (1978). Standards and criteria. Journal of Educational Measurement, 15, 237-261.

Heller, L. (1978). Assessing the process and the product: An alternative to grading. English Journal , 6, 66 -69.

Herman, J., & Dorr-Bremme, D. W. (1984). Testing and assessment in American public schools: Currentpractices and directions for improvement. Los Angeles: University of California at Los Angeles, Centerfor the Study of Evaluation.

Holmes, M. (1978). Evaluating students in the affective domain. School Guidance Worker, 33, 50-58.

Holtz, R. E. (1976). More than a letter grade. Social Education, 14 , 23-24.

Ingenkamp, K. (1986). The possible effects of various reporting methods on learning outcomes. Studies inEducational Evaluation, 12, 341-350.

Linn, R. L. (1983). Testing and instruction: Links and distinctions. Journal of Educational Measurement20, 179-189.

Lissman, U. & Paetziod, B. (1983). Achievement, feedback, and its effects on pupils -- a quasi-experimen-tal and longitudinal study of two kinds of differential feedback, nom-referenced and criterion-referencedfeedback. Studies in Educational Evaluation,9, 209-222.

Maxfield, B. (1983). Computer aided profiling: A case study at the North Warwickshire College of Tech-nology and Art. (ERIC Document Reproduction Service No. ED 237 651).

McPartland, J. M. & McDill, E. L. (1977). Research on crime in schools. In J. M. McPartland & E. L.McDill (Eds.), Violence in schools: Perspectives, programs, and positions (pp. 3-33). Lexington, MA :D.C. Heath.

McDill, E. M., Natriello, G. N., & Pallas, A. M. (1985). Raising standards and retaining students. Reviewof Educational Research, 55, 415-434.

Natriello, G. (1982). Organizational evaluation systems and student disengagement in secondary schools .

St. Louis, MO: Washington University, Final Report to the National Institute of Education.

Natriello, G. (1987). The impact of evaluation processes on students. Educational Psychologist, 22, 155 -

175

Natriello, G., & Dornbusch, S. M. (1984). Teacher evaluative standards and student effort. New York:Longman.

Natrielio, G., McDill, E. L., & Pallas, A. M. (1990). Schooling disadvantaged children: Racing againstcatastrophe. New York: Teachers College Press.

Natriello, G., & McPartland, J. (1987, April). Adjustments in high school teacher? grading criteria: Ac-commodation or motivation? Paper presented at the annual meeting of the American Educational ResearchAssociation. Washington, DC.

Oakes, J. (1985). Keeping track: How schools striscaire inequality. New Haven, Cf: Yale UniversityPress.

1 1

7

Parkay, F.W. (1987). The success-oriented cuniculum. The Clearing House, 61, 66 - 68.

Phoenix Union High School District. (1981). Instructional manual: English/writing. E.S.E.A. Title One.(ERIC Document Reproduction Service No. ED 216 069).

Rheinberg, F. (1983). Achievement evaluation: A fundamental difference and its motivational consequences.Studies in Educational Evaluation, 9 , 185-194.

Rist, R. C. ( 1970). Student social class and teacher expectations: The Self- fulfilling prophecy in ghetto ed-ucation. Harvard Educational Review, 40, 411-451.

Rosenholtz, S. B. (1977). The multiple ability classroom: An intervention against the se4e-fulfillingprophecy. Unpublished doctoral dissertation, Stanford University.

Rosenthal, R., & Jacobson, L. (1968). Pygmalion in the classroom. New York: Holt, Rinehart, and Win-ston.

Rudman, H. C. (1987). Testing and teaching: two sides of the same coin? Studies in Educational Evalua-tion, 13, 73-90.

Rudman, H. C., Kelly, J. L., Wanous, D. S., Mehrens, W. A., Clark, C. M., & Porter, A. C. (1980). In-tegrating assessment with instruction: A review (1922- 1980). East Lansing, MI: Michigan State Univer-sity College of Education, Institute for Research on Teaching.

Shavelson, R. J., & Stem, P. (1981). Research on teachers' pedagogical thoughts, judgments, decisions,and behavior. Review of Educational Research, 51(4), 455-498.

Shepard, L. A. (1976). Setting standards and living with them. Paper presented at the annual meeting of theNational Council on Measurement in Education, San Francisco.

Slavin, R. E. (1978). Separating incentives, feedback, and evaluation: Toward a more effective classroomsystem. Educational Psychologist, 13, 97-100.

Solo, L. (1977). What we do because testing doesn't work. National Elementary Principal, 56, 63-66.

Stewart, W. J. (1975). A multi-dimensional evaluating-reporting system in the elementary school. ReadingInstruction, 12, 174-176.

Stiggins, R. J., Conklin, N. F., & Bridgeford, N. J. ( 1986). Classroom assessment: A key to effective ed-ucation. Educational Measurement: Issues and Practice, 5(2) 5-17.

Terwilliger, J. G. (1977). Assigning grades -- philosophical issues and practical recommendations. Journalof Research and Development in Education, 10, 21-39.

Thomdike, R. L. (1969). Marks and marking systems. In R. L. Ebel (Ed.) Encyclopedia of educational re-search (pp. 759-766). New York: Macmillan.

Walling, D. R. (1975). Designing a "report card" that communicates. Educational Leadership , 32, 258-260.

Wilson, S. (1976). You can talk to teachers: Student-teacher relations in an alternative high school. Teach-ers College Record, 78, 77-100.

Wise, R. I., & Newman, B. (1975). The responsibilities of grading. Educational Leadership , 32, 253-256.

Withers, G. (1986). Profile reports-A typology and some caveats. Studies in Educational Evaluation, 12,325-334.

1 21 8