-A12 721 SUfFT1U I ORUElliU QUg ERUEM: IN-ANCIN4GJIHE mqERlk

UNCLASS IFIED ~ D~MM4C85- NT 2 9 GO~ MLL~C

11111I.OL, 110

I1.25 1

Ml( Wk CP Rf',o.iUI.N 'i ' -HAW

AD-A182 721

Human Resources Research

3LPERVISOR VERSUS OBSERVER: ENCHANCING

THE EXTERNAL VALIDITY OF LABORATORY

PERFORM&MNCE APP1 -,SAL STUDIES

Roseanne J. Fotiand

Brett A. Cohen

September, 1986TM-ONR-5

Te\as A&,\ Lniversitv-and

Virginia Poltchnic Institute

DTIC'-- JLECTE

UJUL 21 W97

87 7 20 02(i - - - -

SUPERVISOR VERSUS OBSERVER: ENCHANCING

THE EXTERNAL VALIDITY OF LABORATORY

PERFORMANCE APPRAISAL STUDIES

Roseanne J. Fotiand

Brett A. Cohen

September, 1986TR-ONR-5

Department of PsychologyVirginia Polytechnic Institute

and

State University

Prepared for:

Office of Naval Research800 N. Quincy Street

Arlington, Virginia 22217

This report was prepared for the Manpoier R&D Program of theOffice of Naval Research under contract N00014-85-k-0289.

Reproduction in whole or in part is permitted for any pur-

pose of the United States Government.

iij

Unclassified

SE-CURITY CLASS IIATION OF THIS PAGE (~Mn Daea Entered)_________________

EAD INSTRUCTIONSREPORT DOCUMENTATION PAGE BEFORE COMPLETING FORM1. REPORT NUMUERi 12 ovT ACCESSION NO. S. RECIPICHT'S CATALOG NUMBER

TR-ONR-5 I, A4. TITLE (and Suheill*) S. TYPE or REPORT a PEthoD COVERED

Supervisor versus Observer: Enhancing the Technical ReportExternal Validity of Laboratory Performance C. PERFORMING One.REPORT NUMBERAppraisal Studies

7. AUTHOR(a) I. CONTRACT OR GRANT NUMUUER~a)

Roseanne J. Foti & Brett A. Cohen N00014-85-k-02899. PERFORMING ORGANIZATION NAME AND ADDRESS 10. PROGRAM 91.9111110T. PROJECT, TASW

Department of Psychology AE OKUI UURVirginia Polytechnic Institute & State Univ.Blacksburg, VA 24061 =~475-036

11. CONTROLLING OFFICE NAME AND ADDRESS 12. REPORT OATE

Office of Naval Research October, 1986Department of the Navy IS. NUMBER OF PAGES

Arlington, VA 22217WS MONITORING AGENCY NAME S AODRESS(If different Inei Controllig Offc) 1S. SECURITY CLASS. (of this ropert)

Unclassified194. DECk ASLIFICATION/0011IGRADING

SCHEDUL

1S. DISTRIOUTION STATEMENT (of this Repot)

Approved for public release; distribution unlimited

17, DISTRIBUTION STATEMENT (of the "strgeet oItdft Ba lock ". 1149100MV111 hum Report)

Ill. SUPPLEMENTARY NOTES

Supported by the Office of Naval ResearchManpower R&D Program

IS. KEY WORDS (ceaMijve ean Fewee* old*1 umeew A40 4 IF 411111~ 401#14FA Week aON1o)

Performance appraisal, performance evaluation, rater error

20. ABSTRACT fCe.,thw on revere old* i .. eosdedn d soffir OW SheA ine)

This research evaluated differences in the psychometric quality of supervisor vsobserver performance ratings. Specifically, type of rater (supervisor vs. ob-server) and type of instructions (rating vs. neutral instructions) were manipu-lated in a 2 X 2 factorial design to compare the traditional laboratory perf or-

nce appraisal approach vith a more realistic experimental design. Results indicated supervisors demonstrated more halo and leniency error in rating subord-inates' behavior than did observers Type of instructions given to the ratershad no effect.

.IM7 slatI OF I wev as6a eefsu55 Hmlai

DOIS'N #02- LP.0Old-S00I

UnclassifiedmCUTATVv CI*S ptCAy" or Tell oS mb= &We Swam*

Of** S@Lx.IN. "of

Abstract

This research evaluated differences in the psychometric quality of supervisor

vs. observer performance ratings. Specifically, type of rater (supervisor vs.

observer) and type of instructions (rating vs. neutral instructions) were

manipulated in a 2 X 2 factorial design to compare the traditional laboratory

performance appraisal approach with a more realistic experimental design.

Results indicated supervisors demonstrated more halo and leniency error in

rating subordinates' behavior than did observers. Type of instructions given to

the raters had no effect.

S0

--AvBai1btlitY CodeS

,Avail and/or

SdSt special

rI

Enhancing the External Validity of Laboratory

Performance Appraisal Studies

For a laboratory experiment on performance appraisal to be externally

valid, it should be as similar to the actual work setting as possible. Feldman

(1981) suggests that the laboratory evaluator should be required to perform

other tasks In addition to observing the ratee, since a supervisor in an actual

work setting would not be able to concentrate solely on the employee's

performance. Similarly, Banks and Murphy (1985) have emphasized the importance

of incorporating contextual variables present in organizational settings into

laboratory experiments on performance appraisals. The traditional laboratory

experiment in which the subject is informed that s/he will be evaluating a

stimulus person, and subsequently views a videotape of that person performing

various tasks (Bigoness, 1976; Murphy, Martin & Garcia, 1982), is missing some

crucial characteristics of the actual work setting. As a result, the entire

cognitive process induced by this experimental procedure may be different than

the one which would generally be employed by the rater In an actual work

setting.

By instructing the subjects that their sole task in the experiment is to

rate the performance of a target person, the experimenter is forcing the raters

to attend to a specific type of stimulus input. The experimenter has induced a

controlled process of attention in the subject. A rater in an actual work

setting is concerned with a number of tasks, each of which requires some degree

of attention. It is therefore likely that any information available to the

rater in this setting will have been encoded by means of an automatic process of

attention (Schneider & Shiffrin, 1911). The manner in which information is

3

encoded will strongly influence the availability of that information on

subsequent judgments of the stimulus person (Srull & Wyer, 1979). A controlled

process of attention should result in the specific behaviors observed by the

rater being available during the subsequent performance appraisal. An automatic

process of attention should result in the information being encoded Into general

schemas. The performance ratings in these instances will be influenced by the

prototypes of the categories in which the information was stored, and should

therefore exhibit rating errors (halo and leniency) based on a global impression

of the ratee (Nathan & Lord, 1983).

The present experiment employed a 2 X 2 factorial design (rating

instructions vs. neutral instructions X supervisor vs. observer) to compare the

traditional laboratory performance appraisal approach with a more realistic, and

therefore externally valid, experimental design. The dependent variables were

the amount of halo and leniency errors committed on the rating forms. It was

hypothesized that: 1) the supervisors would exhibit more halo and leniency

errors in their ratings than the observers, and 2) the subjects receiving the

neutral instructions would exhibit more halo and leniency errors than the

subjects in the rating instructions condition.

Method

Subjects

One hundred and fifty introductory psychology students volunteered for the

study, and received experimental credit for participating.

Task

All work groups performed a manufacturing game developed by Foti (1981).

The purpose of the game is for the group to manufacture as many models in the

4

fastest time possible, in order to maximize profits. The subjects start out

with $3,UO (play money), and are given a price list for raw materials and the

finished products. The game requires some strategy in terms of deciding what to

manufacture (there are instructions for 2 different models), the amount of raw

materials to buy, anI how to divide up the labor.

Procedure

Five subjects were run in each experimental session. One individual was

randomly assigned to view the group on a monitor in a separate room; and one of

the remaining four subjects was randomly assigned the role of supervisor for the

group. Once the subjects were seated, they were either told that they would be

asked to rate the performance of the group members on the task (rating

instructions) and then instructed to follow the task directions as closely as

possible, or they were merely instructed to follow the task directions as

closely as possible (neutral instructions). The subjects were then given 10

minutes to read the instructions, and discuss how they would perform the task.

It was the supervisor's responsibility to assign tasks, keep a running tab of

monies spent on raw materials and monies earned, as well as evaluate the quality

of each model produced. The person viewing the monitor received the same

instructions. The work yroup was then given two 15-minute sessions to construct

the models while the observer in the other room watched on the monitor. At the

end of the second session the experimenter administered the questionnaire to all

all five of the subjects.

Uependent Variables

Rating Scales. The measure of subordinate behavior consisted of 10

performance dimensions which were rated on 5-point Likert scales with anchors of

poor and excellent. The dimensions were obtained from a pilot study using 15

subjects (three b-person groups) who performed the manufacturing game and

subsequently listed what they felt were relevant performance dimensions. The

dimensions used are as follows: behavior flexibility, quality of

decision-making, organizing/planning, delegatory skills, communication skills,

construction skills, idea contribution, cooperation, quality of product and an

overall evaluation of each subordinate.

Halo and leniency. Halo was operationalized as a subject's (either

supervisor or observer) standard deviation across all nine performance

dimensions for each subordinate (Saal, Downey & Lahey, 1980). Less dispersion

among the dimension ratings, as evidenced by smaller standard deviations,

indicates a greater halo effect. Leniency was operationalized as simply the

average dimension rating for each subordinate.

Data Analysis Procedures

For the halo measure, a 2 X 2 X 3 (instructions X type of rater X

subordinates) fixed-factor ANOVA with repeated measures on the latter two

factors was performed. For the leniency measure, a 2 X 2 (instructions X type

of rater) ANOVA with repeated measures on the rater factor was performed. For

both analyses, groups were the unit of analysis. This was necessary because

task groups were not constant across each supervisor/observer pair.

Results

It was hypothesized that supervisors would exhibit a stronger halo effect

than observers. Results of the 2 X 2 X 3 ANOVA indicated a significant main

effect for type of rater, F (1,28) = 4.06, p< .05, eta = .04, with means of

.56 and .69 for supervisors and observers respectively. Additionally, it was

hypothesized that supervisors would also exhibit more leniency. Results of the

2 X 2 ANOVA showed a significant main effect for type of rater, F (1, 28) =

b

13.77,p < .001, eta * .12, with means of 3.89 and 3.51 respectively for

supervisors and observers. Thus, both components of hypothesis I were

supported.

Our second hypothesis, that subjects in the neutral instructions condition

would exhibit more halo and leniency error than subjects in the rating

Instructions condition was not supported. Results of the Z X 2 X 3 ANOVA to

assess the effects of type of instructions in halo error was nonsignificant, F

(1, 28) .73. p< .39, as well as the results of the 2 X 2 ANOVA on leniency

error, F (1, 28) - .54, p < .38.

Discussion

The results of the present study suggest that there is a significant

difference in the psychometric quality of performance ratings given by active

supervisors versus passive observers. We found that supervisory ratings

exhibited more halo and more leniency error. Thus, this study lends Support to

the contention that laboratory evaluators cannot be made to focus exclusively on

the stimulus person's behavior (Banks & Murphy, 1985; Feldman, 1981), if the

results are to be directly (rather than theoretically) relevant to employment

settings.

Our second hypothesis, that type of instructions given to raters would

impact on the psychometric quality of the ratings was not supported. It may be

that given a laboratory situation, any type of instructions will induce

controlled information processing, and only by making the observations part of

some other tasks that the person is doing will automatic processing occur.

In conclusion, although the present research has demonstrated the

difference in halo and leniency errors between supervisor and observer ratings,

the issue of accuracy was not addressed. Given that there may be a weak

positive relationship between certain rating errors (e.g., halo) and accuracy

(Cooper, 1981), future research should focus on this issue.

,mma

References

iants, . ,. and Murphy, K.k. (i95). Narrowing the research-practice yap in

Oerformance apprdIsal: The appraisal task dnd the role of observation.

oersonnel ksjcholOLy. J.d, 33-34-.

.04oness, N. J. 191b). Effect of applicant's sex, race, and performance on

empi jver's performdnce ratings. Some additional findings. Journal of

App Ied tsjLnolo 0Y . , Piu-d4.

elj ln, '. M. Ild?1. Beyond attribution theory 'oynitive processes in

pertirinnte appraisdl. Journal of ApI.I1e 0ycrnoloyY, bt, ill-14t'.

Mjrpny, K. k., Martin, L. 4 (arcia, M. 1 9el . JO bendwloral observation scdies

measure observation' Journal of Applled Psycnolo. b 5b6e-3b.

Nathan, 6. q., & Lord, R. (j. j196j). '.oynltlve CdtegorIZdtIOn and jimensional

schemata: A process approach to the study of nalO in performance ratings.

Journal of Applied Psychology. b 1iZ-l14.

Saal, F. E., Downey, R. G., & Lahey, M. A. (196U). Hating the ratings:

Assessing the psychometric quality of rating data. Psycnological Bulletin,

_., 41j-428.

Schneider, w., & Shiffrin, R. M. (I9/). Controlled and automatic human

information processing: I. Detection, search and attention.

Psychological Review, 84 1-66.

Srull, T. K., & Wyer, R. S. (1979). The role of category accessibility in the

interpretation of information about persons: Some determinants and

implications. Journal of Personality and Social Psychology. 37 16b0-16l1.

knnumr. rsonel.ad Eraining R&D Program

Director of Research Prograns Technical DirectorOffice of Naval Research (Code 11) NPRDC (Code 01)Arlington, VA 22217-5000 Sun Dieco, CA 92152-6900

Chairman, MPT R&D Planning Consittee Director, Manpower and PersonnelOffice of the Chief of Naval Research LaboratoryCode 222 NPRDC (Code 06)Arlington, VA 22217-5000 San Diego, CA 92152-680O

Lif e Sciemn Technology Program Departnt of Administrative SciencesManager (Code 125) Naval Postgraduate School (Code 541a)

Office of the Chief of Naval Researob Moterey, CA 93943-5100Arlington, VA 22217-5000

Defense Technical Information Center Program DirectorMTCIDf*A-z Manpower Research A Advisory Services

Cameron Station, Building 5 biltheooian InstituitionAlexandria, VA 22314 601 North Pitt Street

Alexandria, VA 22314

Saiesce and Technology Division Staff Specialist for TraliigLibrary of Cogrs and Personnel Systems TechnologyWashington, 0C 20540 Office of the Under Secretary of

Defense for Research and Engineering3D129, The Pentagon

Office of the Amistnt lecoetary of Masbinoton, DC 20301-3080the Navy (Manpower A Reserve Affairs)

SUWO,' The PestagoaWashington, DC 20350-1000

Team Nead, Manpower, Persosel * and Technical DirectorTraining Section U.S. Army Research Institute f or the

Office of the CIG (Op- 9 14D) Behavioral and Social Sciences4A378, The Postatam 5001 simshower AvenueWashington, DC 20350-1000 Alexandria, VA 22333

Assistant for Research, Development Dr. BenJamin Schneiderand studies Department of Psychology

Office of the DUCO(IP T) (Op-OIB7) University of Ma"landDeparteent of the Navy College Park, MD 20742Washington, DC 20370

Headquarters U.S. Marine Corps Dr. Albert S. GlickmanCode WI-20 Department of PsychologyWashington, DC 20360 Old Dominion University

Norfolk, VA 23509Head, Leadership A Command

Effectivens Branch ("M21 Prof. Bernard W. BassNaval Mililar Personnel Command School of MamnaetDepartment of %be Navy University Center at BinghamtonWashington, DC 20370-5620 State U. of New York

Binghamton, IT 13901Head, IHomas Factors LaboratoryNaval Training Systems Center (Code 71) LibraryOrlando, n 32613-7100 Naval War College

Newport, RI 02940

Mmmine. Puaael an Training I rai

Lt. Col. Losa Petty Col. R est=r

NU4ulart.r, unc Nudquartes, UNCN. nLab@3 1 DC 20380 Umblagba, DC 20380

rATE

LM D