164 614

UTHORITLE

NSTITUTION

UB DATEOTE

DRS PRICEESCRIPTORS

DENTIfIERS

DOCUMENT RESUME

TM 008 187

Cockrell, John T.Television as Stimulus Input Synthetic PerformanceTesting. Technical Paper 303.Army 'Research Inst. for the Behavioral and SocialSciences, Arlington, Va.Aug 7849p..

MF -$Q.83 HC-$2.06 Plus Postage.Enlisted Personnel, *Feasibility Studies; Job Skills;*Performance Tests; Response Style (Tests) ;*Stir: lation; *Television; *Testing; Test Validity;*Visual Measures; *r:lork,Sample TestsPaper and Pencil Tests

BSTRACTAs part of an effort to provide more economical

ethods for job-skill evaluation in the Army Enlisted Personnelanagement System, a study investigated the use of television as aeans of presenting test items for a sample of tests from the jobjell of tank crewman.. The television scene provided the job settingor each item and then posed a question that required a real-timeesponse from the examinee. All items were job-connected and_herefore represented simulated skill items.,The examinee respondedirectly to the face of the television screen by means of anlectronic stylus or gun reticle. The test was compared'with aaper-and-pencil test which covered the same items and a hands-onerformance test which covered many of the same items. The resultsndicated 4- at television testing is very acceptable to soldiers andeasible. ne validity of the prototype test could not be determinedrecisely _ecquse the criterion hands-on test was not usable; mostxaminees made a perfect score on the hands-on test. (Author/GDC)

***********,i*****************************************- A***************Reproductions supplied by EDRS are the best that can be made

from the original document.*********.************************************************************

Technical Paper 303

U S OE PAR THE NT OF HEALTHEDUCATION L WELFARENATIONAL INSTITUTE OF

EDUCATION

Tut', DOCUMENT HAS BEEN1")1F:C) EXACTL Y AS RECEIVE° r ROMTHE PERSON OR ORGANt/AliON OrtnOiN.AT 'NC, IT POINTS Or ON OPINIONSSTATED DO NOT NECESSARIL REPT:L-SEN T Of FICIAL NATIONAL iNT,T1 TUT F OfEDUCATION P051 TION OR POI ICY

AD

TELEVISION AS STIMULUS INPUT IN

SYNTHETIC PERFORMANCE TESTING

John T. Cockrell

ARI FIELD UNIT AT FORT KNOX, KENTUCKY

SCOPE OF INTEREST NOTICE

The ERIC Facility has cosignedthis durtunent for pr oreto:

In our iudgement. this documentis also of Interest to the clear ing-hr:trtes noted TO the right. I ode

reflect their specie!tots of .tiew.

U. S. Army

Research Institute for the Be,avioral and Social Sciences

Auaust 19781

Approver-1 for pultl:c Ielttase; h'-,tributinn

2

U. S. ARMY RESEARCH INSTITUTE

FOR THE BEHAVIORAL AND SOC;AL 'ENCES

A Field Operating Agency under the Jurisdiction L.',1 the

Deputy Chief of Staff for Personnel

1#11)NElk1)ireci-or (I ) sil_H:ite)

V,'!1.1.1AN1 11:111S1 IZ

(:()Incl, US Aim(.(>111mandur

NOTICES

%IS pL;TtON Primary dist!. ibut,011 this er.', I hcs een m, ARI Please address corresbonder.oncerning distribution of reports to U. f). Army Hetear.t.h I n ite for the Behavioral and Social Sciences,ATTN r'E i P, 5ool Alt.arlitda ,:7333

FINA.1)1!3F'r,)51TION This report rntiy be da5trovec wt- ,t is no longer ,,eecied. Please do not return it totne U 3 Army Research Ir,%itute for the Behavioral end Sciences.

NOTE Tne findings in This report are not be construed es en official Depertmen of toe Army pos,t1pn,unless so designated by other euthorized documents

SECURITY A "NON OF THIS PAGE (When Deo, rnferrrl)

REPORT DOCUMENTATION. PAGEI REPORT NUMBER 12. GOVT ACCESSION NO.

,-H; I 'l p- 1

4 fITIFInnel.S.h.fINe)

Tia i r I 1,1

7. AUTHOR(s)

II, II. 'I 'L " I

',INFORMING ORGAN' LA TION NAME AND ADDRESS

Al-mv Institute foriehavioral ang 'ill rluiences

nue , Alexandria

11 CON TEOLL INC. OFFICE NAME ARC' ADDRESS

aad 101tEIMLTrimanh

READ INSTRUCTIONSBEFORE COMPLETING FORM

3 RI CIPIEN VS CATALOG NUMBER

yPE OF NEPOR I A PEFIMO COVE Iwo

PERF ORMING ORG. REPORT NUMBER

B. CONTRACT OR GRANT NUMBER(30

ill PROGRAM LLIMENT, P1-10.) EC. r, TASKAREA A WORK UNIT NUMBERS

-.l()7711A77o

14 McNITORIItO AGENCY NAME A AODRESSOI tflflrarer0 from -.,0rrollina ()thee)

12, REPORT DATE

107813. NUMBER OF PAGES

313 pages15. SECURITY CLASS. (of rt./ft report)

Enelae flied

, DECL FICATION/DOWNGRADINGSCHEID! C

EMEN T (of thler Reporr)

:eproveil E r public release, distribution unlimirecl.

eh.,,rer-/ en,fre,1 Itt Merk 20, If iffffert-r frnr

IN SUPPL EME NT ARY NOTES

,"..E .V WORDS (Continue. on reyer.qo sirfo if necemsary ern! ich-nrify by hi/lc-It number-1.

:iimulated performance resting.:fltHnI is ro 1. Hi;

Television teslii'ing

testlinu

ABSTRACT (Centime., on reveres sit, +1 nr.,,Pssary and iderltfly 5w '1,-I, number)

part of an effort to provide more economical methods far job-skill

-valuation in the irilisLed Pernonnel Mangement System (EPMS), a- study

investgated the lin.- of television as means of providing test item

rosenraricn. The' television scene provided the job setting for each item

and then posed a euehtion that required a real-ti,me response from the

examinee. Ali items were job-connected and thertfore represented simulated

skill Hems. The examinee responded directly ) the face of the television

1 meanh of an elec.- ')nic stylus or gan reticle. The test was

EDITION OF I NOV I:5 IS OBSOLETEDO , ,,,,T.°75 1473 Uncla.tl.nified3ECUFBTY CLASSIFICATION OF -1-1415 PAGE (Whern Date Entered)

7,200

SECURITY CLASSIFICATION OF TNI5 PAGE,(3 rhari Det ErNotect) -.--kt 1.,1 J...? 1 f i1 , I I ,J JI. ." III 1 f. , ' ! 1 J j 1 Jj J j.J:11 j1 J I J 'ij r'1 ' 1 1,1 t 1,, " ', tint ' 1 t t 'tn. , ,11:-' t

i H tit' 1. -, 'II 1.1 t )1 ir,111,-, I , -:.1 '.....!ill 'III t't ),,r1`1 , ! 111,1!1`," l 't t Iv . till... i t .ni: : Th.,.;i11:-. 'inLii.....,.i 1,1 t 11.1 ,,1. ,:i 1:L.)11 f ,.....t .111,1 ,Itt.l 'I 't '' 'dd. ' t ' "'"I'll it' t

l',, I t '',t ;ON lt,_ Th.' tt..t.I Iti i t ,.. ttt t 111' 1'1'1'11 nt ,,,,,,J1,1 nut I' 1.,t .'1111114,` I

( t li: .11,1h -1- It ,I1 11,1!1,H-,,i, 1 ,.. 1 II;,I , l i' Lt..',I11!;.' t . iI l

LinciassiciedSECURITY CLASSIFICATION OF THIS PAGE(When Data Er,red)

Tet Paper 303

TELEVISION AS STIMULUS INPUT IN

SYNTHETIC PERFORMANCE TESTING

John T. Cockrell

and

Donald M. Kristiansen, Work Unit Leader

ARI FIELD UNIT AT FORT KNOX, KENTUCKY

Submitted as complete andtechnically accuro.1c.,,, :

Donald F. HaggardField Unit.Chief

Approve(

E. Ralph Dusek, DirectorINDIVIDUAL TRAINING AND PERFORMANCERESEARCH LABORATORY

Joseph ZeidnerTECHNICAL DIRECTOR (DESIGNATE)

U.S ARMY RESEARCH INSTITUTE FOR THE BEHAVIORAL AND SOCIAL SCIENCES5001 Eisenhower Avenue, Alexandria, Virginia 22333

Office, Deputy Chief of Staff for PersonnelDepartment of the Army

August 1978

Army Project Number2Q763731A77^

EMBEWEISIMIIIMIMMINICT: TnITM1.-112WMINMEMRIMMal

i formance-OrientedIndividual Trainirg

Approved for public release; distribution unlimited.

Affil FiCI)OrtS ''arid -Technical Pawns are intended :or sponsors ofR&D tasks and other research and military agencies. Any findings ready forimplementation at the curie of publication are presented in the latter par t ofthe Brief. Upon completion of a major phase of the task, formal recommendit11011S for official action normally are conveyed to appropriate militaryarrocies by briefing or Disposition Form.

hm iw!tttti..71p.ti t)1

HI1 .1 t i (111,11 '.;() it] i (.r:: 1.() nit, .1 N.1 Ht ,),11

"%t Hi H f -11, iwn,iv 1 Am.! :,1)(.i,L I :4 (10' f 1 .

4'1 11(1 i 1 1 Ot,ve,1()pmt t irmt I l',\:1111It IA 01.,'t I H C(-111,C! ni,(1 It MpriHV t (711_slICY arl(l 1`((111t1IW:

,)1 11(ti_l; Ht '1 t 1..) n r1(1 111,1 i on. 111 ti rna trT

i In} iv, n(, 1111 r,t )1 , i II, Army tItt.tt1t t.1,1 t 1.1,

!.,n I.l h., ....,:mmitt 1 n.itt,i(it (1.1'7.U;) . Tli !;y:;t (tilt

-1' i !it. iridivi lii.tI t t m mtt t t i litH 1 itt ! iti.itt

tylt ()r (hid] .;0.1.di.(-!r:; tor car-er

FP:!; ritr,rion-rterence performance tc:;tin(1 of ,(dualrtiLhor thjm generaliz.ed knowledge. fit an (.fort to Twhieve mon.'

oc:nnomy teting required for L1i EPN;:;, re!-4,,irch

ti.:-ogram L;eeking to develop :-7:imulateil performance to:it:-; Iwonnno promiing line of c.ndeavor ir= the 11:-;e of variouH audio-

vi:Jul media to pivldc inpuL and for LI-AL

pront roport:; the re.!-;ult!; of a stony

v,:;t. L,ja Lnq the input:: in eon pinntii on withhan electroni,. rcHpondinc vehiel- whiTh recair, ral-time cleciiow; and

re 0,11-,.h wc-1 don at 1 -Ile API !.'ic ld Unit at Fort Knox, Kv. , in

reH.ponc, to icduircmc!nt:"; ot Airily PITojc.c't 2fl:t,37U_A770 Ind to -;):4._A--1.1-

t or tin Trait and Pont-. r tie ccrirriand (TPAI)0(') , Port :',1onrn,

OSFPH ZE.1611E

,nical Director (Designate)

i't I

T j T,Ii t tlit t Il 1

It.

ticIti t1:1.111>11:1111-11 i ; ,! I It if t I t () (10,1 e.1-riljil, i

I ."!! .t, it t r ,1 1'1.! till '1' I 1.'1 . i H I .,11 d

tt t....t. I' ".t tt".1".,' t..)' (-)t )1.1.1(1.;;-it ..t.t. t..I. 1..! Y.; . 11'. t.. \t,i int .1.11 it it III

Iii

t ; L e r : t c iri1, iii H I e t t t tit 11 te 1 i.e.' I I ctql .1tt t I it. t ilIli I ii: thip111

w i. Lit'l.:.`1. TO. I ld .'t; I Wit; OflI, i di" r H 1 1.)t-./t Itn1:1:11t1L('

: t. t 1 j t.. t. I1( iIt na I tasks the examinees we rereguireel pertnrm 1)11 the job. I'ii lent as administered to T.) soldier

nee who had com1 eteri a1VC1,11C(...'d is ra in in g in the sub cc t IRe titer .

made by these same trainees on a hands-on performance Lestwhich hod similar items were alsn_obtained. The hancls -on test was

administered routinely by the Army to all trainees at the end of theadvanced training. A parti!lel paper-an(1-pencil Lost wHs administeredI I) it'', t raj [lees , and han.Litolt 'ores wet e a] sn obta ned for

I

Th- results favred the felsihility of television toting. The

I' we. roduced and administered without difficulty, and the examineeshad a very favorable attitude. The examineL-s had no trouble under-standing and respnnling to the items. The exalt; nees judged the test

as "fair" (impartia.fl in terms of testind them on important tasks they!ti-IH have mastorcd.

The validity of the result-5 was inconclusive. The criterion scorestor th- hands-on test were unsatisfactory in Lhat most examinees milea perfect score. The correlation between the television and hands-- antests was led but nonsignificant. Comparison between thetelevision and parallel paper-and-peucil tests also showed no overall.Ai:ierence, altherh there were sidnificant differences between manyi.Is

it tsti ,)1 tt,T;;

f S Zvi i,:, I ;III I i 'ti I.1'11( 't t )1 IC I i I (Ii it- yi)1 t; V I.; c+11 i 111)10 1-; .lit 1 1)1 11.1111:, 11

10,111,', ; . .1 t.t mt to, Ill( )1'4 ' p ' I ilt t t !; 1( )11.t 1 )1l11 `, ill i " t4 7, )1J1.14 'I) 111'0 rt. !,,t f; try

t!, ,! 1 1;t.1- c)tt t "y;t ,1 flint t Itt)t-tmttli ri it inn )I t-li( )i=a, I

t '11 ;,' ,'1,trii'(to'111:; I it...it..nti1t..11 ttitr;t t- :;t

T I - I 1. ' ;')'1'.1:111,11)., I ';11'1' 1^,)'i f

i 1 ? it ':t

1

)` )t, !!

'11 1'4 1 t ) 1, 1

i

11)11 )1, 1

! 1!. ' 1! I1 .1

1 !, 1 I i t ; .! I r-1 I

'1' 1 %." 1 ; ; ;

!()!) (!) Tt icy ):)1()1,, l',119 -)incl-Pfl ,))),)i1)111(1;-' )11 T ' I ';

,):;!-; 11(_)N,), AND 1.):L3C 7'3):)17,

...

i11-1'l.:7:1)1y A i LIE I, I rii;?, F01-3 THE DEVI ..I.XX S ON

:A:IPL)I Di' PAPER-Arm-PENCII)

Fi2.););\ 1)T;AT TIIF TELESTPAT01.3 Eciti

3I

33

3'7

kli;T or TARIY;;

It' I . 1 r t i,)ri t t i t t '111H nt, Lh t

ean porc4.int

I `,./ ! j f, 4 t

t;c1iulk.i And dittrihnt ion it ,xaminoei;

urroJ made onand-Hinciit

viHion and paper-

4. An,ilviti!, uf variancie for tel(iiviitin aid paper-And-i.niHl

Hat tmoit ttiniii WL.111(1--1,(.'1,Ci. I )n the t .--f

ti,levjun And Liper-AnJ-penH1trIU; rc 5ii I r rot irratnft,(1 lc/ n.i-tpont-te . .

cotri.lationiti !ii.tweoti the televi:;inn and 1-ti,t..And-

1 ticorTL; And the nAnd5-;-on i_wrforman(ty n_-C) re; .

Cl'SIT FTMIRES

ridure t:on(it,IATion of otlinrilus fidelity and feailiiii.tyttradeoff

tiirtiulus and respons;ci fideliti AHlensions 5

Roponse components for tank: Lo,Hing round intomaitA t-tiun on an M60A1 tank

TELEVISION AS STIMULUS INPUT TN SYNTHETIC PERFORMANCE TESTING

INTRODUCTION

Oyer the past 10 years or so, Army has tried to convert more ofits testing to the "hands-on" performance mode, especially at trainingcenters and at the beginning skill levels. Even more emphasis has beenplaced on performance testing in the last 2 or 3 years with the begin-,ning of the Skill Qualification Testing (SQT) program. Performancetesting is highly desirable because of it:T high face validity and high

acceptability; however, this type of testing is very costly, hardto standardize, and often not feasible.

The alternative to hands-on performance testing has generally beenthe standard, group-administered, knowledge-type, paper-and-penciltest. Although relatively easy to produce and administer, this typeof test is generally considered to have low validity and low user

Osborn (1970) has Suggested that a compromise validity-feasibilitytradeoff point might be reached by using synthetic performance test:According to Osborn, the term "synthetic performance test" -!fers to

any performance test that is less than a full hands-on test, but morethan the group-administered, knowledge-type, paper-and-pencil test.Synthetic performance tests include all-tests that use any type ofsimulated, inputs or responses. Part-task tests, ih which only one ora few response components of a task are measured, are also included

under synthetic performance tests. The s-mthetic performance testis conceived as less costly than a hands-on test, but as a test that

still ha!-- reasonable validity and user acceptability.

To support the Army's adoption of performance testing, the U.S.

Army Research Institute has initiated a broad-based research program

to investigate the possibilities of synthetic performance testing as a

cost-effective alternative to the usual hands-on procedures. The goal

of this research is to develop a psychometric h for boti hands-on

and synthetic methods.

The research focus beel-, on the use audiovistal media to pro-

vide the simulated stimulus inIDut. The reasoning behind this focus isthat audiovisual media stand midway in the stimulus fidelity range, and

at the same time, are at the medium to high end of the feasibility scale .-L

Thus, audiovisual media may represent a good fidelity-feasibility tradeoff

1 Stimulus fidelity as used here refers to how closely the test stimulus

resembles the real world, and stimulus feasibility refers to how much

it cos' to present the test stimulus in a testing situation (high

feasib_ ty equals low cost);

LOWSTPIULL:S

FIDELITYASI IMULUS. FIDELITY AND FEASIBILITY

PRINTED STILL SLIDESWORDS PICTURES PLUS

AUDIO

MEDIUMSTIMULUSFIDELITY

ATELEVISION

ORMOTIONPICTURES

SIMPLE3-DMOCK-UPS

ORSIMULATORS

NIGHSTIMULUSFIDELITY

COMPL REALTRAINING EQUIPMENTSIMULATORS

HIGHSTIMULUSFEASIBILITY

MEDIUMSTIMULUSFEASIBILITY

LOWSTIMULUSFEASIBILITY.

Figure 1. Conception of stimulus fidelity and feasibility tradeoff.

2 14

point insofar aS stimulus input is concerned. Fic:ure 1 shows a concep-

tion of this fidelity-feasibility tradeoff.

The overall research ,program has ti - follow.ng objectives:

1. To explore the parafieters of the various audiovisual media todetermine the media's applicability to synthetic performance testing.

To explore various responding modes and response devices thatcan be used with audiovisual stimulus inputs.

3. To determine whether those response compents of a cask thatcan be measured using audiovisual media are sufficient to yield anacceptable measure of the entire task.

4. To develop a task classification system that will enable asynthetic performance test developer to determine by analyzing the task(a) when audiovisual media should be used as the stimulus input, (b)

which medium is advisable, and (c) which response components shouldbe measured.

Several experiments in this research program are now in processusing a number of different audiovisual media. This paper, which isconcerned with television as the stimulus input, presents the results ofthe first of these experiments.

This first experiment was limited in nature and focused on thefeasibility of-using television as the stimulus input. As such it wasconcerned mostly with the first research objective--applicability ofmedia to testing--with some exploration into the second and thirdobjectives, responses to stimuli and test-task comparisons.

Background and Rationale

The impetus for this research stemL from the Army's decision to.substitute the Skill cualification Testing (SQT) program for the currentMOS testing program as a means of assessing the job skills of enlistedpersonnel. The SOT program is intended to be based or job-sample testswherever practical, as contrasted -to the current MOS paper-and-pencilknowledge test.

This change was brought about partially as a result of the researchof a number of investiga -s (Engel,'July 1970; Engel, October 1970;7ngel &'Rehder, 1970;.Sf.J__Acey, 1965; Urry, Shirkey, & Nicewander, 1965)whc questioned the validity c' the MOS test for job skill assessment.In 1966 the Army convened a special board of inquiry (Brown Board) tosurvey the entire question of written MOS tests for assessing job skills

and job knowledge. This board recommended that Performance tests besubstituted for written tests wherever practical (U.S. Army, 1966).Following-the publication of the findings of the Brown Board, the Armyhas made subst.ntial progress in implementing the recommendation (e.g.,

3

the Tank Crewman Advanced Individual Training perform-tests adMin-istered in,the form of a "county fair," with examinees movingt,from

`test to test around the examination area, during and at the end ofeach training cycle). However, due to high costs and difficulty i.maintaining standardization, the rprformance test obviously is limitedin terms of making up a substantial part of each SQT test.. This isparticularly true at the higher skill le.';21s and for many hard-to-measure tasks. Occhialini (1972) , for eNaple, presents evidence thatperformance tests are extr_rnely difficult to prepare and administer,and are of questionable alidity. Engel and Rehder (1970) review thearguments againsT the use of performance tests for part or all of theSQT battery. Their general conclusion is that the exclusive use of 4

performance tests in an SQT batterN would be too costly and impractical,

Reacting to the pros and cons of paper-and-pencil vs. performancetests, several researchers have prOposed compromises. Engel and Rehder(1970) advocate a mixture-of-measurement technique in each SQT test,combining work samples, simulated tests, peer ratings, and paper-and-pencil tests. They present evidence indicating that cognitive itemscan be measured adequately by paper-and-pencil tests; that motor-manipulative items require work sample or simulated tests; and thatpeer ratings can be used to judge social, leadership, and overallability.

Osborn's (1970) approach is concerned with developing synthetictests that it is hoped wfil eliminate some of the i4racticality ofadministering performance tests, while reducing the vLrbal componentand improving the validity of paper-and-pencil tests. Osborn visual-izes a continuum bounded on one extreme by paper-and-pencil knowledgetests and on the other by job-sample skill tests. Within thiscontinuum, a number of synthetic tests more or less removed from eachextreme can be constructed.. The continuum is conceived of as beingscaled in psychological units and Varies along the dimensions ofstimulus fidelity and response fidelity (or a mixture of 'both).

In any combat situ_Lion, the stimulus dimension would be a largecomplex composed of visual, auditory, tactile,-kdresthetic,-olfactory,pain, and stress inputsl: The response dimension would be an equallylarge complex of cognitive, motor-manipulative, and perceptual outputs.'For the purposeS. of illustration, the stimulus and response fidelitydimensions forarmor crewmen might be conceptual4.zed as shown in gure2. Osborn maintains, in an analysis similar to the one shown in Figure

f,

2, chat one must pull away from each excreme of the continuum to develop 71synthetic tests that are both feasible and more valid than paper-and-pencil tests.

An important aspect of Osborn's conception is his reasoning withregard to part-task testing (Osborn & Ford, 1976). In this conception,each task is composed of a number of response components divided intoCognitive, perceptual, and motor behaviors. -Figure 3 shows a task brokenL

4

Li'

Written Itels

STIMUS FIDELITY ITIENSIM

2 35

'\\....//

W'ritten items

it `till

i'iritton Items

Plus fotiuu

Auditory Itm

P1v IOtior

Job Samplr

Items or Simulvd

,10 Sample

-,ls on

'Ilctir.:al 'IiITIOUVOT:',

with Lnril.p:nt

Pidtt,vs Pik:tun

(Televisiou

Pioturrs

(Television

Euuigrot jiiJatIon.il

F

Siimil:itior,

ViSd3]

C

SC

Respons(H

FIPILjTi EDP..`;M`I

;INJ

Witory !uulitery

Ti:ned

kspons,'s t

fl

trt .(11(1;) Pt'S,rin

66

Al.r.ditnrY

ThVy

dPj d

resivuses

Figure 2, Stimulus and response fidelity dimens.ions.

dart

8

ombat

a

18

HEAR

FIRE COMMANDPERCEPTUAL

RECALLPROCEDURECOGNITIVE

CHECKSAFETY SWITCH

MOTOR-

OPENBREECHMOTOR

RECALL

SAFETY RULESCOGNITIVE

IflENTIFTI

_AT() 1--

PERCEPTUAL

PICK-UP PLACE ROUND PUSH ROUNC ENGAGEAMMO IN CHAMBER INTO CHAMBER FIRE SUITCH>

MOTOR MOTOR MOTOR MOTOR

VRECALL

SAFETY RULESCOGNITIVE

AV

RECALLSAFETY RULESCOGNITIVE

CLEAR PATHOF RECOILFERCEPTUNE

NOTIF(CREWMOTOR

Figure 3. Response components for task: Loadinground into main gun on an, M60A1 tank.

6

down into response components. (This task is performed by the loader

on an M60A1 tank.)

Pilo reasoning behind part-task testing is that it may not be neces-

sary to test every response component in a particular task in order

to determine how well the whole task can he performed. It may be

possible to get a good indication of whole-task performance by measuringonly a few'response'components or perhaps measuring only one critical

response component.

Part-task testincj becomes crucill when audiovisua] stimulus inputsare used because the nature of the medium precludes obtaining anymeasurements on most motor-response components.-In order to obtainmeasurements on motor-response components one needs to test on realequipment or a hands-on simulator. Since the measurable 'responsecomponents in audiovisual simulation are limited to perceptual andcognitive ones, it follows that the usefulness of audiovisual stimulusinputs is dependent upon the validity of the part-task testing concept._One objective of the research program is to check the part-task testing

concept by correlating scores made on bart-taks using audiovisualstimulus inputs with scores made on'the corresponding whole task testedin the hands-on mode.

li of Television in Testing. Television has been used in testingprimarily as a recording medium (Cockrell, 1974; Hays & Pulliam, 1974).A study by Shriver (Shriver, Hayes, & Hufband, 1974) explored the

possibilities of using television as the stimulus input in a perfor-mance,test. After developing the test, Shriver concluded that tel

did not offer much promise in terms of replacing hands-on testing.He listed eight disadvantages of'the television medium and decided to

abandon the method and not attempt a systematic comparison between thetelevision test and hands-on performance tests. Some of the disad-

vantages mentioned follow:

1. Television tests place the subject in a. passive role, watching

someone else perform and evaluating the Correctness of the performance.

There is no reason to believe that success in this evaluation role,will

insure success in the active role of performing the task.

2. Television violates a major ground rulq7of criterion-referenced.testing in that it emphasizes process measurement rather than product

measurement.

3. Television costs are'very high compared to those of slides orgraphics because of the ldrge amount of equipment needed and the largepersonnel time'requirements-

Shriver's criticisms are informative, but they do not necessily' settle the case. The nature of the medium does-include some practical

difficulties both in producing-the stimulus tapes and in administering

7

20

the H!;t. :Iowever, these difficulties are minor compared to the complexLi. of administering hands-on performance tests. If televisIn can show

eorrolat ion with job-sample tests and also show advantages overwritteh iii other audiovisual tohts, it well Pe tilt=' extra cost.

r'

rimary ohj,ctivos of the present ext--rlment were to appraise.1, or I practical. difficulties in using television as the stimulus

and to ma)-:e a rough comparison among television, papr-and-Pencil,:inn nds-on performance tents The secondary objective was to conduct

of a responding device (Telestrater) designed to permitto respond directly to imagos on a screen (se Appendix c).

:itieally, the oljeetivos were as Follows:

L. De rmi ne the Peas ibi 1 i ty of ...u:=;) n(j on i

it ii. under consideration here were

lnderstandahility of test items

Case of responding,

Time allotment for responding

d. Diffieultie., and costs involved with administering tele-vision test-

Determine the acceptability of television testing by examinees.

1. Compare the results made on the television at wit) -se made

on the papr-and-pencil arrd hands-on performance t ;ts.

d. Conduct a checkout of the Telestxator response device.

METHOD

The overall methOd consisted of (1) producing a television test fora sample of -tasks from the job field of tank crewman (11E MOS), (2) pro-

ducing a parallel paper- and pencil test covering the same items, and (3)

comparing the results made or,these two tests with the results made onan existing-hands-on performance test that covered many of the same items.

The job field of tank crewman was selected because much priorresearch had been done in this field. A complete, task analysis wasavailable, and a hands-on performance test has been In use for the TankCrewman Advanced Individual Training course for 2 years. This existinghands-on performance test was felt to,be a good bade , against which tocompare the television and parer- and-- pencil tests.

8 21

The first step in producing the television tape was to select the

critical tasks in consultation with military experts. The selection

criteria were set by the military and included such considerations as

importance to fulfilling the mission; safety to the crewman, and safetyto the equipment. The critical tasks selected were quite similar tothe task, coyered in the Tank Crewman Advanced Individual Train

course. After the critical tasks were selected, they were ordered

according to skill I

Eos the final test, tasks were selcoted from skill levels 1, 2, and

3.- Vur the purp .es Of this exirimnt, the tasks can he considered

to ringe taisty eany to yes :I.: tic-ult. Tasks WOre also selectd

such that each the four pc. .ions (driver, loader, gunner, and tank

CL=Ar) e s cOVt_il:ud, ,! few tasks pertained LO the crew at large.

Inconsuitation with e military, each task was broken down into

cognitive, percepth. , and motor components; and each response componentexamined for its :1-iticalitv to the tank. Practical considerations

Jon an overall test running time, tins to televise each item, number of

:sponse components needed to cover a particular task, and achieving

a balanced test (see Appendix A) eliminated many critical response

components. !,or each of the remaining critical rponse components a

television test item was conceived and a television shooting script waswritten. Each item was televised in a crude fashion with a handheld

eter.a and a p(ntable videotape recorder.

The raw footage was edited roughly into a prototype television test

by the addition of narration and titles... The prototype tape was intended

only as a model for a professional tape to 1 produced later and as a

vehicle to check technical accuracy and television feasibility.3

7ilitary experts checked the prototype tape for technical accuracy

and understandability. A revised televsion script incorporated

suggestions; a final television tape was produced using professional

television personnel, cameras, and editing facilities. The shooting and

editing of his final tape required approximately 30 calendar days (about

15 actual working days) .

The final tape consisted of /17 test items plus 4 p-acl;ice items and

had a running time of 53 minutes. The items ranged running time from

1There are five skill levels for each PROS ranging from skill level 1

(beginning) to skill level 5 (most advanced)...--"Th

2Work on the preliminary television tape and the task selection required

to produce it were done by Human Resources Research Organization under

contract to the U.S. Army Research Institute.

.K

Lrt r

.1 b

va

fir,' r.3!

I gun

tlal night

.., , .',...1:1L-t,r". .1,1r1 taro -t

..,...(ht-tiovkil tari ...

1,..,

Ye.;

IE,,;

Y(!:

Drawinc;

Drawrol .

ture . MI i h ,rirt.-..t.rt.ion.lry ''..arirt. Mn:: Orawila

7. !.W.E.,_ mr:....-rnovin.; targ.'t Y,:. No` 'oraw1Lo

ern orrutt.orl

,. t

. 1:(

'L

)%or

choke.

Mot Or rr,aN ipulation

cholce

Error ri tection

':1;n1.0,r ,., r cr,,,,;.r. u... Motor minipulation

.-..h" : P(..rcrptual Motor mauipulatioh

runr.. I'err,pt oa I Motorrimanii.ulation

r;urr.. : c),.rec.r.1)...u,, i Motor manipulation

..1' lifforent poty.t L1 ti.tininl,

ern ornitt,..1

(27 not ihulud!!(3 ou',(uy thel(1:.-n

cm riot rov,rrr during

23

w.;t . f

p,a - Te!

1. . tt. t !".1. ;(.1ht.

2!;1":1

1

2, 211r!

Ta2210 i (con t 12:11e..)

Item or Type ut Crow

w;ell t on paper .. oicit ion

u-on ;encl.; Ind

Yo!!,

!; A

r,tr.-4.1w;

rdwmc,

r .

TV t eot

typi.

Pe re,. ptIld 1

Pit r,:er 1. 0.131

tor nanipulat ion

motur manipulation n

Mut or manipulation

Motor manipulation

d itl.oet Ion

;!,1-r ",r '!eteot 1021

.tr1. 11 ie

mxt

ir;dre

:;12

vi7e

uLnI.F 310

3F Orion;

.' ii11 .t!,ct

10';

.;:. OcrJ hit a

4 i. Fire cum-pa20 t,ank

41, Eirn 201121and truck

p, Fire 20mniEl c,uix

41,, :'Irt. ,..mamir.,.: truup2;

17. Cn6uni priot i y target -

el)

22

:.0'1

0°

If:)d

',;(fl

4e.4

1

No`

Nod

tioLl

d!!c2

: raw In":

Plot IT'

P1Cttilr

.'" 11

uri12.;

',Curd!. ,

h'ord2;

Word;

Tank 1.::t] r

Tank Cmdr i

Ta;:k Cmdr I

TA4.

7.22.k .',,tnjr 3

Tank Cmdr 7

Tank Cmdr 3

Tank ad r

Tar,k Cr

Tank Cdr 3

ereptual

Pc r'eptua 1

Cod:: t 1 ve

Coo: t ve

C.tdn t ve

Cogn 2 ti vo

Cognitive

Coin it ive

Cognitive

Ter7optual

t pl .22:Dice

r or ]e',.oction

Multiple cbirioc

Multiple choice

Mu; ! iJc cholc:2,

Er ror detect 2on

Error detection

Error (le tec tion

Error ti,.ft0CtIOn

Multiple choice

2M UMIttoll iotrouue te.:tod it i22 n ulnt in t rain:n(1.

rm .Jmittet 1A.J0 aiur te ii 1.FiiFO plYt Unto

rm not Inc lulled unc any i.and!- or, !

Ifl nut cyJo re n'; t

a

24

reSpOnSU.

Table 1 provides a description of the final television tape. The

Categorizin:: of response components into perceptual, cognitive, or

.motor, types was somewhat intuitive. The intent was to show the pre-

dominant element of each response component and not to imply that

other elements were not present.

Ot the 47 items shown Table 1, only 37 were administered

to the examinees in the expo invent and only 30 were scored. Most Skill,

level 3 items were eliminated before the experiment upon the rec..mmen-

dation of the Hllitary staff at the Armor Center. These items were

considered too aanced for the examinees, Aft.er'the start of the

experir it, several Firy advisers recommended the elimination of

six morn items, and one item was eliminated d., to a poer'teleisionnicture. These sev n i.,J-Hus were administered but not scored. The

,otnotes in Table 1 give the reason for the elimination of any item

and also explain wk(i certain it :m were not included on the hands-on

test.

A more eciffic description of each respone type shown in Table 1

follows;

(1) 'Iultiple choice. The examinee was required to selict one

answer from a list of three, four, or five alternatives. These al-

ternatives were somet:mes the s as those in the usual paper-and-

pencil, testnamely, words on the screen--and sometimes cohsisted of

in. jes on the screen.

(2) Error detection., The examinee was required to watch a procedure

being performed on the screen and to indicate the time an location of

an error, if one occurred, at the time it occurred. The examinee was

shown the procedure twice and responded on the second showing.

(3) Motor manipulation. The examinee placed a plastic gun tetiiple

(thbse teticles used with the main gun in theM60A1 tank) on various

stationary and moving targ2ts as'if preparinq)to fire the main gun. Th-e-

reticles were also used to simulate the asijk6tment of fire that would

be made if the first round missed the target. The motor manipulations'

)rnponse was supposed to be a,crude simulation of the actual response

in aimind the main gun. However, the movements required were so far

down on the scale of response fidelity that the motor component appeared

not to he Measured at all. Perhaps the reticle response was primarily

per,:eptual and cognitive .

.1

Paber-and-Pencil Test

The 1.15por-and-pencil test phsalleled the televiKion test on hnitem-by-item basis. The stimulus inpu' en this test was primarilyPrinted words, but some pictures and drawing were UsCtd on Percentua]itemK. Table i shows the ...timuLus input for each item.

Ah wit the television test, only 37 of er-and-pen c i t.

items wero adminiKt_ered and enly 3D were ;(:10ref-i- 'The-.'

were the the . scar ' for (=, television trust

The paper-and-pencil items and the tel'-,vision items differed dre4.1:1 the amount of time allotted to repond to each item. The total ti,.

llmit wee the same for both tests; however, examinees could allocatethe response time any way they chose on the paper-and-pencilwere restricr,i to 10 second per item on the televi.:ion test.

Dn the papernd-l_:encil test., cix.aminees could c; sqe their-- on rn,

items and anhwor later, and review their answers; on the tele-ioilDn test., none of thin flexibility was bermitteC,

These diflorences between the twa tests were retained because eachmedium lends itself most readily to the type of procedure used. Any

Her procedures or a common procedure for both tests would hac requiredmuch more control and thereby reduced administration feasibility.

Hands-On Performance Test

The hands-on-.test was cne-routineLy administered to tank crewmentrainees as a final examination for the Advanced IndividnalTrainin-':r7,r course. This test was given in the foLm of a county fair with

ations and 30 performance measures. Examinees were graded on a"go/no-go" basis for each performance, measure. For each no-go, examneeswere required to seek out remedial training a:, report back later for aretest. If the 'retest was a no-go the exami had to report back thenext day, after further remedial training, for a second and final test.:Par the purposes of the present experiment, the score recorded for eachexaminee was the number of first-round no-go's. This won not a partic-ularly good criterion because the number of no-go's was very small%

Response Equipment

,A secondary objective of the study was to check out the televisionresponse equipment (Telestrator). This equipment consists of a clQarplastic electronic tablet and associated recording and programingcomponents. The electronic tablet covers the television screen ,(theLiLlet is approximately inch away froM the screen at the center ofthe screen and approximately,1 inch away at the edges of the screen).

13

The examnee looks through the Htblet to view the te. items. Responsesare made by touching the face of the tablet with an electronic stylus oran electronic gun cuticle Et a particular time and location. Before thetest, he correct answers sLime and location) are programed. on the tele-vision tape. During the test, examinees are credited with a correct an-swer if they touch the screen at the correct preprogramed time and loca-tion. Any other response by an examinee is counted as incorrect. Onlyone answer is" permitted for each time, and the first answer--correct t !-

incorrect--made during the 10-second response period iL; counced.

The response equipment was in prototype foLdr and bee Ise of opera-tional difficulties could not be used for the 07(periment. However, if

proved possible to test the operating concept of the equipment by placinga human grader behind each examinee and having this observer record en asheet of paper whether. the examinee touched the correct location atcorrect time. This gre Lng task was quite simple, and during a pilot runwith eight examinees there were no difficulties in grading.

The television monitors were black and white and measured 15 inchesdiagonally. The examinees sat approximately 2 feet from_the sets atself-regulated distances so that they could manipulate the respb/.re im-plements comfortably. Prior to the start of the experiment, it waS,de-cided to remove the electronic tablets from in f-nnt of the screens be-cause of parallax problems After the tablets were removed, the accuracyof the responding and scoring improved to a very precise level.

The response implements consisted of a stylus used for all multiple-choice._and error-detection items, and two plastic gun reticles used forrotor- manipulation items. The stylus was simulated by using the eraser

an ordinary lead. pencil. The two plastic gun reticles, the samethe M32 and M1U5D main gun reticles in the M6OA1 tank, were

manipulated by small wooden knobs glued to the plast:_c reticles.

Examinees

The examinees were tank crewmen who had just completed the AdvancedIndividual Training/Armor course. Altogether, 134 examinees assignedfrom three different companies were tested. Examinees were drawn fromthe companies by a selection process best described as haphazard ratherthan random; however, there is no reason to believe that selective biaswas present_ As each group of examinees arrived for the experiment foreach session, the group was randomly assigned to the television or paper-and-pencil test. Originally, 144 examinees were scheduled for the ex-periment, but 2 were lost due to scheduling problems and 8 were lost dueto scoring problems.

Procedure

Testing was conducted over a-5-day period in three morning and fiveafternoon sessions. The actual schedule and distribution of examineesare given in Table 2.

14

2?

Table

Schedule and DistribW ion of Examinee!;

andday

DaysTotals1 2 3 4

Television 16 1.6 16 11 11 70

Morning 8 8 0 24

Afternoon 8 8 0 11 11 46

Paper-and-pencil 16 11 1R 11 8 64

Mornin, ! 8 4 10 22

Afternoon 8 7 a 11 9 4")

Each group of subjects reported at 0800 o/ 1300 and was given an

Orientati, session explaining the purpose of the experiment.. All ofthe paper-and-pencil group was administered the paper-and-pencil testright after an orien_ation. The television test was administered tofour examinees at a time; the rest ef the televi sion group was assigned

to a waiting room. Both the televiton and the paper-and-pencil testsrequired approximately 1 hour to complete.

Ar>sroxima 10 minutes of training were required to teach theexaminees the methods for responding to the television items. Most oF

this training was concentrated on the use of the plastic reticle. The

examinees were trained by having them respond to the four practice test

items. rf any examinee had difreuity with the reticles, such as choos-

ing the incorrect reticle or holding reticles incorrectly, the tape wesstopped and the four Practice items presented again. In no case was it

necessary 0 present the practice items more than twice.

RESULTS

reasi ility of Using Television in Testing

The ee_Iminees did not appear to have any difficulty in understanding

the items. All of the r. ntent had been covered in the Advanced Individ-ual Training course, and the examinees had been tested on similar items

several times. Pll of the items were also performance based and posed

questions ` -het fl-7.ur normally in everyday operati'ons.

15

The rasponding proceeded smoothly for most items. The examineresponded very Adickly on the easy items iapproximately 1-2 secondswith the stylus, 1-4 seconds with the reticles) . on difficult items,the am.-tult of respon:e time allotted (10 seconds ) still appeared ample,although there nsualii would be a lot of hesitating over the answers.On only a few items the examinees failed to respond. When queriedafter the completion of the test about the amount of response IEmost ey,lainees indicated that for the most part the response time wasadeque A few examinees sai that more response time should havebeen Lotted to some items.

The administration of the television test was more time consumingthan that of die paper-and-pencil test hecause of the need to provid:,preliminary training 1, the correct way to respond and the limit i.

four examinees per session. Administration could be made more asibleby incr,',-k!' r the n-mber of television monitors, but Auld still 1.)

- Lu have test administrator for cch four -J.Iinees becausef the examinees' unfamiliarity wii:h the response methoC mared tothe administration time for hands-on testing, however, L(' sjon testing

much less costly.

Acceptance of Tel Testing

The reaction of the examinees to t*e television test appeared to bequite favorable. Postexamination interviews indicated that most ex-aminees actually preferreb the television test to the hands-on testand all examinees thought the television test was fair. Even whenqueried about the test's being used as a loasi, For promotion or extrapay, the examinees still thought it was fair Mme examinees pre-ferred the hands-on mode of testing, but no ,refereed the paper-and-pencil mode.

Some reasons merit- fled for preferring the television mode follow:

1. Scoring is fairer and not dependent upon the whims of the testadministrator.

2 Testing is faster and not so drawn out...

3. In television testing. no one is shouting at you and orderingyou around.

Some of the reasons for ereferring the hands-on mode follow:

There is more time to think and to respond.

Testing is more spread out and doesn't come so fast.

3 Television nurts the eyes.

16

There is ''1.111Ce to walk around between items.

The .xaminees also indicated that television testinq would beletter than paper-and-pencil tes because the quetions would bemore uplerstandable 1-,,guire much less reaCting..

Comparison of Television, Paper-and-Pencil, and Hands-On Tests

The comparison between the mean percent error made on the televisiontest and that made or.. the paper-and-pencil test is shown in Table 3.The means for the television and paper -arid- pencil tests do not cliiferto any great degree, indicatinq that the difficulty levels of the twotests are 7airly equal.

Table

Mean Percent Error Ende on the Television and Paper- and Pencil Tests

Test andLimn of day 1

Days3 4 5 Mean

TelevisionMorningAfternoon

Parer-and-penci_iMorningAfternoon

10.63 15.00 14.7529.3d 28 .00

28.3F; 20.7527.13 24.71

22.35

16.4627.34

Unweighted mean 22.00

29.90 27.3824.2326.05

Unweighted mean 25.14

One interesting facet of the data is that afternoon televisionexaminees made many more errors than the morning groups- These resultsare L:,,nvineing because they are consistent across the first 3 days ofthe exueriment and because the afternoon means for Days 4 and 5 areconsistent with the other afternoon means. There c' as not appear to beany mornin-afternoon effect -mr the paper-and-pencil test.

Th lalysis of variance using the unweighted means analysis forunequa. Jell frequencies (Winer, 1962)-is shown in Table 4. This anal-ysis shows no difference between the television test ad the paper-and-

17

test in term:; t); lt,,m Th : ;Iqui f

111,0rni1,1--,If I ,,rn; fr..1 , but t 11(' i'w)? if ill re'!-,u t :; t lc

significant mean s,tuare (M!';) interaction. Analyhis of this 7,1-; inter-action reveals that the morning-afternoon effect is concentrate,i nilthe tele7cHion test and not on rhe paper and -peneil test.

W-4 to c!lo_'Ck On Wh(' t t he al terrinf.) X art:LI-WO?, May 1111v, been1 ;lit t lien the morn i_nci , i r!;t= round nn-qnrroM hang:-;-on test were analved. These results are shown in Tal)le

lnspection of tne means indicate!,1 little differene- between thetelevision and paper- and pencil groups, or Between fhe morning andal-tornoon groups. If anything, the afternnpn group performed slightlybetter than the morning group. AM analysis of variance of theseresults showed ne, significant difference for any of the variablen.

Yarianc,.

rCe

TV vs. t (method)

Table 4

for Television and Papei-and-Peneil -ents

dt

ort ng vs. afternoon (session)

x session

Within cell 130

MS

9,50 1.04 ns

124.57 13.1)0

.40

9.16

Although overall scores on the television and paper-and-penciltests did not differ, there might he differences among the variousitems. Accordingly, the items were grouped by response type !multiplechoice, error detection, and motor manipulation) and log linearChi-square tests (Shaffer, 1973) were comnnted for each item. Table 6sfic:s that there was a wide variation of difficulty among the itemsnnn nn from 10 to q1^, error. Per the i toms there

was L.tt-le difference between the television and paper-and-pencilversions. ,illy 1 of 13 items showed a significant difference. Forthe error-detection items there was a sub-7tantial difference, withsix out of nine items showing a significant difference.

18 31

`Fah;

Moan Percont Error Made by rho Television and,In-Pencil (r, ,s on the (Hands-on ToKt

Test andtime of day

TelevisionMorning c).00 6.00Afternoon ().00 9.no

Papar-and-pencilMorning 1.00 6.00Afternoon 7.52 9.72

Days

3 5

9.526.52

8.005.52

0.56

Unweicjhted mean

12.00 6.52

Unweighted mean

Mean

8.166.52

x).08

8.48

8.78

It is interesting to note that errors of commission are more diffi-cult to detect on television; whereas errors of omission an,1 no-erroritems are more difficult to detect on paper-and-pencil. Three of theeight motor-manipulatioe items show some significant difference, andall three of these items show more difficulty for the television test.The net result of this item difficulty analysis shows five items moredifficult on television tests and five items more difficult on paper-ancl-penciJ t=ests. This canceling effect is reflected in the overallnonsignificant difference between the television test and the paper-and-pencil test.

The last analysis, in Table 7, shot7s the correlations rf the hands-0;, est with -.le paper-and-pencil test and the television test. Thosecorrelations are also broken down for the morning and afternoon groups.There is a lo,/ positive correlation between the television and hands-ontests and also between the paper-and-pencil and hands-en tests. The

paper-and-pencl correlation is significantly differe _ from 7...er,);

however, there is no significant difference between the televisionversus the handy-on and the paper-and-pencil versus hands-oncorrelations. The breakdown for morning and afternoon groups shows asomewhat higher positive correlation for the afternoon group and verylittle correlation for the morning group. Once again, there is nosignificant difference between the television and paper -and- pencilcorrelations with the hands-on test.

19

Comparison Retwen TolevisjonItems tlercent I.rro

so typ(:: and

it(m number

andArriinii t hy

Papc-r-and-PenctiHsponsc Typos

crrolX-

MuLtip1 e ultoL.

1 i r) ns:) 0 :, ns4 43 il 1.435

:21 1.4 .

6 3H 72 13.83*7 47 5. ns0 20 1 P C

') 10 117 ns10 17 16 ns'

1.1 35 ns

12 0 0 ns17 1 3 ns32 81 77 ns

Error detection(commission)

3 51 RR 7.47*13 11 11 ns14 40 12 11.09*37 46 50 1.42

Error lotection(omi:T..don)

16

1

10

4

17-,,

. 4714.3e*18.90*

23 13 45 11.26*Error detection(no error)

15 16 19 ns20 17 66 30.36*

Motor manipulation(reticles)

19 46 16 12.24*21 51 17 16.17*24 4 5 ns25 27 11 -,.75k

26 11 2 ns27 14 .. 11 ns28 23 34 1.4530 23 28 ns

*p <.01.**p <.10.

20

Table 7

or:-IaLionsHetweem the Televinon and Paper-and-Pencil:;(.:ory:; and the Hands-nn Performance orcs

t o)f. day

ane corre .3("(A

overoll

'ii vs. nand:;-on .".)=1 5H

Pape-I-and-pencil vs. nand -on

Mornjr-ig

Television vs. hands-onPaper-and-pencil vs. hands-on

Afternoon

Television vs. hands-on .47

nimF, F.iCafi

tference is nom-iigniFicantns

.01 diffeLence is nonsignificantPaper-and-pencil vs. hands-on .40 .01

CONCLUSIONS AND DISCUSSION

ie result from this research indicate that it is Hossible to producea HTntl-lc,tjct nsi.no teicision as the stimulus inbut. The examinees

can under:: .nd the problems, make proper responses, and accept the testas "fair" for career evaluatioh.

The experience gleaned from the production and administration of thisprototye test indicates that television testing is more costly thanpaper-and-pencil testing but far less costly than hands-on testing. The

production of she tape, from conception to final editing, required severalmonths and used the services of a substantial number of professionalpeople. Televison tests are also somewhat inflexible, not only in thedifficulty in e-Lfectihg changes in the test, but also in the timingdecisions -the amounts of time to allot for posin each question and foreach rese--tht h71, to be made before the producinn of the test.

Television testing will have a much mOrie promising future if apresentation and response device can be designed which will permit theexaminee to advance to the next item as soon as the present one is

answered, to see the same item twice, to chahge answers to an item, and

to review the entire test. Such a capability Would permit the ,flexibilityof presenting multipart items, such as in troubleshooting and would per-mit the presentation of multimedia items, such as using both televisionand technical manuals in the same item.

21

The pree ex1n provides evidenee that fele,,Ision tesngin highlyacceptable to the examinees- Their pre,f)minant attitude wast.h,it: the test was little di. erk-mt from t hands-on to:-7,t,n in the AdvaneYd Individual Training course, except that teloviion was quicketand less sub-ject to senrinq error. Al t of the !cc,nt:-; were famil-iar to the examinem and the items were one.; that they had been study-ing tor H-13 weeks.

Televiion used in the multiple-ehc)ido format aPpears to off' 1.

advantage over slide or paper-and-pencil format:;. liefore the experi-ment, it: was felt that television would an advantage tor thoseitems in which motion was an integral part of the stimulus. For. ey.am-

spangenburq (1971) has shown that ;itching a ¶-i'lev,sionA prncedurosequence ofbe true for

involving motion lead!-; to more learning than watching a

still shots. However, this advahLa(je of motion proved toone motion item in the oresent re r;earch (item 6, Tattle 6)

but not 1-.rue for two items (5 and 7) . Perhaps if more motion-typeitem.-, had been included in the multiple-choice cot:egery an advantagemight_ have been shown.

In the error-detedtion eaVegory there did appear to be a cear-cutdifference between television and paper-and-pencil item. Were thefidelity of the stimulus did seem to play a role, and the enrichedstimulus of the television picture may have presented cues to the ex-aminee". The two :._r.or-deLection that proved to be more ,difti-calt for 'elevision examinees liter ant'. 14, Tables 1 and 6) weretwo of the first error-detection itm_ to be presented. Since errordetettion was an unfamiliar response for the examinees, this unfamil-iarity may have caused some difficulty. This same phenomenon can beseen in the motor-manipulation items which involve an even more unfa-miliar response_ Here tna television examinees had more diffici31twwith the first few items town did the paper-and-oencil examinees.

The correlaLlions between the synthetic and the hands-on L'aSL5- aretoo low to warrant recommenJing the suostitution of synthetic for hands-.on tests. However, the correlations for the afternoon groups are highenough to encourage further research. The hands-on criterion test usedin the present experiment was somewhat unsatisfactory because of thelarge number of perfect scores.

The drop in the scores on the television test for the afterhon-group as comt e- to the morning group was interesting but unexpect.One possible explanation for the drop may be that the examinees werrequired to stare continuously at a fairly large television screen Hrema very close distance for approximately 1 hour. A human being mayable to tolerate this strain to the morning but by the afternoon accumu-lated fatigue plus a heavy Army lunch may have combined with the stra:nto produce a letdown.

22

rz:

; 414 444i Ti_1.; _1t)(1

: ' , , 4 4 1 ' ! 4 ( ) 1 4

, 4 4 ,

1 I . \ 4 0 [ 4 4 1, P.-1 '; r:4. 1 0.44414114-4"41.1.141

I 11:44j-.:4 4 414 ; 4.41.4.4 1'44, 1.4 .j 111;4111 I .'.1)1) 1.; 41.1 .,,t4,1p1A:.1

41'111

11,40m_3,Li !J I p lit) ,L1 , '11 3 1 1 1 1( '11 1i' 3 1 [

1 1 , 1.)11 M >1 1: I

3 p i. i It H I t u(>1:;

I 1 I H I tA U ;R.!. I II', {LULL.) ' 1...1 I (II, I I 3 ").,. 1 1 1".;(,,h1 ,'AL 1.:1'M

:1,111'13 1)111' 11(11 :; ;1,, ,)11 tA 3 11() I 1 3 :).,1.10,..) 1 I 13 11

r,

11111.:.:., 11( )-;;311.11'11 31'13 1 A ').3,L1'1!; 1 3 Z-",;1111 ,"11 3 _IL LI

It()1.1. 3 1 L. ,11,), IL 3 ).)11 3 I;I'r,1 ";.3 111:;,) ..c.1 4 A ;

, tt31,7X :113 3 13 3 1 33 1 (11' 131,111 ;r\l'11 1)111'

.411 I I .4 111)41i' 114,1,1111ln _t 4411411,411 4)1(11 1.4,....)...;1,41.1 1 II , u tulpu 1 ! "

. . , 0 1

, 1 1 ' 1 1 ( P : 1 ' .

1 I 1 I )1 )( 011 111.%) 1; 3 bit 3

3(1 t I ,0 3 ,1LH A11 3.; !; 13 1 1', I

I 12, tt t t 1 ' , ! ; 1 d,

t f .1 1,,II-144; , I ;!- I

I .;

!!1 !.,! I I !it I,' . '1,

.11 .

; '!! t I,. 4, 4 r

iiF e

, LLt

1.11i 1.'1..11 I' , t

! t, : i'. It ' . :,,,i, 1 il ." !1.! 1; 1 ' 1 1:1

!,-4..! , 4 . , - 1 .,. , .

, ..

... I! ,, (1,. ; t!,. I '. t! 1, , I !

2:7, '.. TILl I i '1') ! ": '",I,"!

1 I '1.11 . 1 11 1 ILL I L1 1 T' II12- I' ::111:.! ,", .11 ''1 -: ii" i oil

iH.s:!! .22

,1 1.11 :nt 2.71.),1..11;;.-111 1tH.,1-- 4.

1 !

'. ci.:1!' : I i it- .14 In1. ()Li

! r Dt.! N I cimi FL I [iiIi (:) 1 .1 2'; , . I :

:" 'it t iT .11E. I 0; fl.1[11,..:1 fox' 12th Ar:I"

, ,

Prot .. .! 2 1:11M-111 P NI C-11

.

. NIL, . , I'..

i7;-.

P. in 1.11iLl;_-"JLoc;ica.i. Du lje , 1 97:_ 1 7-

P. (2. IreLiminary t op lli,INIL1.L.:::-1-1-_!]) Study Na. ?,) ) Fort n

I I Jr F. : '21; 2 Army En 1. i-tL-(1 ()I)

2_5 t

r \.rI I,. 11111'1,1/1 1,A 111,1r1 ),I, 1t ! ,) ,

I r r (AHILI, V;) ) .

I r "

111 1 t I I .It-11,1 011! . li 1.. 111 1 11 '

I , 1 ,

! \ t ,r1 .r.rrr,,

j \ 1

111 I ,',1

1 11 1 1 1 11 1 .

, r'r1 t II, ' .1 it tr 'II 'Fr I r'.r! +11:,11, 1 r i .1

r1,1,, .' ) . II-1 '1,, rrr 1.1th i !Litf I

r l 1r' , I

11 t1 11 11. 1r rip I rr i i rtr, n i,

r r,1 I

111. 1 . It

I

I

L LH: )i ": ! 'I' '\.' : t't

111.1 () t 1,(tlt)!i . i i III . '11/' Oil i Ld It ,)r mi I it

t .;t :;11t vt I t; (YL.)1 t t (II . , 1 't /4) an if not ( 1)1! t ,,r(' t

1.1(1 tti. I ',tt ".-rtt ',1+t,11 1 V .'1() :111, / t t ' ),!';1'ki 011 i 1 t

h `V. t. 11,;11. nf r1. t t ow III t

If I it I 1 i:" 1i 1+,1. I 1)14.44 , rth,fly nt, .1.;. t ht.

,r tt i !I- 11T117, t

ill:; T LI 1 I : 1 I 11 1 If 1 1 ! I 1 1 , I t ,111 ' I t I

.ArtH t tig t 1 !1f t. it: I i 1 tl-, 1 -11 t 1 ,11 ft,r 'Pt tli 5 111ff Mt .1,.1),ALt,L t tro. I 1 I t I V 1 I I-

11,t11 AL; t 'l0tiO I II III I I 111[1111 It t-

,Ilti g11,- , 01111 1.1111,1.11 ()Ilet11''JItll.Ilt I 1111111 )V r 111: i:; II.- 111w i

t L (. i .t ;i1L I tnt t hi.. ,L ttvi H1031 i,..tt. , thl' .1055r t.'

h()l1:-.1t1 1 vi t..)!1 L.L;t i mt.! LI ti-) 1 i tn i tI 4 t'(-) (1 !IeF

mh,.h (11 the wa,,- ot it my L,e useful tI del;crihe the develop-rwuta an ,1 ot 1: t t-trt,i! t

o the the test it_ was decided to aim for aminut- I jill it.,i Lime, to cover 1011 MiS tank crewmen at skill

1(tVt I , 0111] 1 111(1 Lho iL11 t-tosit of driver , loader , gunner, and,min a tt LIT. Thi-itest w11-: to be A group test with indivi.:ual TV

fn tvin

I,1(t of th .cren with a styll. or reticle.tounh7-17 tj'Y

The fir step Was to ask various military trainingHunnc'ry, automotive, and such) to submit a list of critical

taki; whic .;hou1,1 1)e tested. These departments su5 tted a total of 75Rosa -Ie only a limited number of tasks con be used on the final

tape, the list had to he pare down considerably. Many t'.asks were elim-tnat,cd in ordot i.e ha lance the test among skill levels and crew Do,sitions.I:or example, 4D of the tasks received from the departments were for skilllevel. 3 and only 5 of these ask.; were on the final -emainingexes:; tasks were eliminaed by deciding to limit tle Lest to tasksassociated with the actual operation of the tank. Critical areas such asdrug abue, first aid, leadership, and tactical decisions, and complextasks, such as s;-,:,tching In area map and tasks that required excessivelylong televtion inInning times, were eliminated.

The first step in developing specific test items for each task was2 list all response components making up a task and decide whether ei.1L

y. :-)nse component was primarily cognitive, perceptual, or motor. Each,Tjnitive or perceptual resnn:le component was then examined for criticality

27

35

D4.7..11'01 1_11D74. I ...")U1rT-.7 ;'4,11_1-41 O 1 1...-P.o!.1.11JC1! 11_111.I1L,

IWCX 0 0[414 ii.(1 p121/41 ' 1 (111 ;

,;": TAT i), 4 rul..4. () .A-114 .; 1 701

3112+ 4i..z !Olt 'Ili' 4;110 I 1T2 [A-op 0_11)[

.1 .J-67) -3,Ab1 L'.; Ii 1 ' . i),,,t

Iturxt', -Pliu f':_1; . 4

[n.) L 1 41_ 4

1:1L' 1 1;'". 14. 1

.

Lai_ 1!. L;) 4.1ILI ;V.( I 1, I: !

!!1!)1., .11:1 1.4,)11/!,1 1;11 I_II1_I 1 !

1...:1-1.41:=.; .014; :111t2i...1=t111.1:4')'4471D 1)1 no:i (0,1 / i)", .1 41- ...Au ; .;

; 2; Litit) 1.1-1-1-2111 1111 4_,H.111,

1 441 .11 - ciLLIL'X r "T( 4 //A i 1 4 114..;; : 1;

1 11?... ..,;444'1 1.1114.4 14./..+4/1.; 1.011' . ,_!1/..401.! 1'111' .1)N1.11

11,!)

4 I 1); "....1(4,42_7..; 411;114414y, 1_1.1t1;.,.1. +14 1 4141.

.1 j 111.1 1'1 1. '11 11

11111 1,;(1A(1 1'11,"1:; '1 41'11) 1111'..; L

1 1.11.) Li _I p 0_11' .14 4 1,1;;

111'; 41'1; H._'. 1.'111_1.1 .1( I/. 1 1;11(",i'4:',_; -L,H;

' , L . 1 I' . 1 1 1 LI I I

".

11.1.1, ) ; 1.1.1 dill

.1 0.11 ;. ; 1... 11111' 10:11,"._1 1.1 1'; ;'. f I 1.)1.1. I.H(1.1': 1, 1,

11( ; AI' ; 1 1 ;4;11 1 ;11

Ii 1 !A !

!,. 1 11 11. I

1 4 1 ' ; i ' . ; )11 4

, . 4 4 1 1 1

4 I ' 1: ; . .. :. , 1 1

1;;1;; ;. 1114...[ 4:;; ;11 11'11 1 4 11..; 44 411 ;1;1

1 1 1 ) ' 111 1' /..,11' 1 '1 i . i '

1 11 ; ' t' '' ' ' { I 111 ; .

1 1111, '1.

1

,

1 1 , 1 1 1 1

l .1'4 r 1 HI 1 ? 1 I

- 111 1 .

I ' 1 I . 0 1

, ' 1 1 ; 1 1 ..

i 1

I

1 1 1; % : ( {

. I I. , . i

I. I .1.1 1

1 ( '

't r

i ! 4 4 1. , i 1 )

. 1: 1 1 ,. . 1 . i i L

I

,

!

i 1111 i 4 i i 1

I. 4 .

,

I 1 1 1 ',

. 1

1 I 1

, ' . 1 1 ,

1 I

1:11

+1 1 .114 ; ;

1 4;

..1 ' ;

li II 1 1 IL :1

' 1

i , 1111..., 1 1 1 I 1 ,

1

- ',

. ; 1 1 t ' l l i 1 1

.

1 ;', . 4 ;

, . 1

1

1

i ; 4 , 1 i I 1 ' .

5 I ' .' ; ' I 4 " 4 OP , 11.1. ,f.; ; 1 I 4 1 '

1 i ' 4 ; 1 1 1

,

1 ( 1 ; 4 1 1 1

1 1 1 1 1 : 0 0 . 1 ; . ;

I I . H , 1 4 z ;, .4 ,141 1 1; 1

4 1 1 ' 1

; 4 I ' 1 ; ; . 0 1 I 1 1 1 1 1

. 4 1 - I. I l i 4

, , .',1; , t , 1

_ , 1 ,i

i ' . I, , , 4 . 1

1 1 t ! I

,

1 1 ' - i l ' I i i ' 4

1 1

. . 1 1

, ; .

1 1 1 1 . ;

. ,

4

1 1 , ' 1

' i 1 ' L 1 1

,

1 1 1 1

l ' ' 4 ; ; ; ; 1 1 1 ; ,

1 ; 4

1 1

. , 1

; ; I 1 4

.

1 1 ' 1 .

1 1 1 1 , ; .. 1 11

The error- ,otection r on!--;e type was include:. to test: the examinee?'

knowledge of incorrect as ,:t.!.11 correct actions. Many incorrect actionsoccur very inCrequentiy but can he very serious when they do occur.Performing the correct action in a hand::,-on test does not necesarilyindicate awareness of dany.12r points, and examinee:: need to be testeddirectly as to awareness -H.7 incorrect actions. :,--)wever, Shriverpoints out, wat'ching somo,, else perform is er-: different from doi: 4l.t your se f i 1 error may indi loci: of edg,--, or-

it may indicate inohility to noti,. error .

The overall r:(-)n luion is that , rror cl,- -.ion,I _ibt-f-ul resnone

t,,-pe and more thought and reerch are needed 72riflr Lc_s anc,.,:,tanc.: as

a useful 1)rocedure.

3. Motor manipulation. This respons- type was rather ecific to thisnarticular study and the response equipment being evaluat,A. 12 fact,

..:trong selling point of the response equipment was its provsion fortesting the motcmanipulation ite. All test items under his particula[respo:.-;0 type pertained to wn, re the examinee should place tne reticle onthe television screen when simulating 'firing the main g lc underconditions. gowever, analysis reveals that this res!,-H,_ type isreally a test of motor ability, but rather a test of a combination ofperceptual cognitive abilities. The cognitive element. was knowingthe correct lead and elevation for each target and the perceptual elementwas being able to discriminate the correct lead and elevation. There isno evide! t :Le that the ability to manipulate a plastic reticleon a screen has any correlate-in with the motor element involvedin aiming an actual gun. On the plus side, this response type is more ofa recall item than a multiple-choice question and therefore should providea more exact me, -, of recall. On the rn!itiL3 side is the requirementto learn a new re. onse gu,ckly (manipulating plastic reticles). Incorrectresponses may he caul -ed by lack of knowledge or perceptual ability, ormerely by failure to master the new response of manipulating plastic

_

rticles.

This response type, like the error-detection tcsi, '110 type, needsmuch more thought and research prior to a(..ptane as a procedure.

Some comments on a few miscellaneous topics may also he useful:

1. Ise of a time_period to indicate error. One item (3i Tables 1and 6) n the television test used the pass,ige of time as the cue forthe examinee to note an error. That is, the actor waited only 5 seconsbefore continuing a procedure, where the prescribed procedure in toetechnical manual calls for a 120-180-second wait. Some cri.ism hasindicated that this time:-passage technique might confuse examineesbecause Americans have been conditioned through exposure to motipictures and televisi,.)n to accept any length time period shown on ascri,en as the 4propriate time. Item 3 did seem to confuse many elcfaminees.

29 41

2. Long 'toms. Several items on the television tape had relativelylona runnig Lmes (approximately 3 minutes). Some critics claim thatincluding !Irinh 1,;ng items may be unwise because coverage of the totalsubject matter is res. :L.T(1 at best, given a 50-60 minute time limitfor the test. rii_hough a long item may not necessarily confuse theexaminees, it noteworthy that 3 out of 4 long items retained in thetest did prove very difficult for the television examinees and all 4of the long items omitted from the study appeared to be confusinLJ

pilot ru.,1:_;. Another reason for omitting long items is thei-fficulLY in getting an actor to perform a long Sequence in letter-)erf fashion. One very long item on the tape (evacuate injuredcrewnein) was never completely satisfactory. The final take wasacceLi her:ause the director b; came concerned with the safety of theactor playing the role of injured crewman.

3. 'resolution. Unlike the human eye, television cannot capturehoth wide-angle view and good resolution at the same time. Forseen , that require Cfn,Cd resolution it is a good idea to zoom in on asce and remain there. To attempt to show more than one cll_oseup inany one sequence tends to confuse the viewer.

Restricted view. Even with a wide -angle lens, television givesa, very restricted view and care must be taken to provic setting shots.Precise judgments as t(, the :placement of controls are difficlt tomake from a television pictiure.

5; Poor depth perception. Much depth perceptioi. is lost in atelevision picture. Items that depend upon jud(:ment of depth shouldbe omitted.

G. Closeup and motion. Any kind of motion hi a closeup shot isconfusing. Necessary movements must be i2ry slow and precise. However,it should be noted that slowness is per rived an error by many people.

More research needs to be accumulatd before a more precise set ofguidelines can be prod,. Fur television testing. Particularly neededis development of a muJe adequate stimulus presentation and response-recording device. Also needed are researchers well grounded in thecapabilities and limitations of television and the use of televisioncameras, lighting, and editing equipment. Television testing offersmuch potential but before this potential can be reached, much preparatorywork r .ains to be done.

30

APPENDIX B

EXAMPLES OF PAPER-AND-pENCtL ITEMS

A. Multiple choice.

12. You are the driver of an m60A1 tank. What response do you make tothe following ground guide signal given at night with a flashlight?

a. Move backward.

b. Start engine.

c. Stop engine.

0.. Tarn lett.

H. Error Detection

6. You are the loader in an m60A1 tank. The tank commander gives thefollowing fire command:

"GUNNER, BEEHIVE TIME, TROOPS, ONE SIX HUNDRED"

The firinr' switch has een checked and the breech is olsix steps in order:

(1) Select. a BEEHIVE round.(2) Insert the round 2/3 of the way into the chamber.(3) Push the roun into the chamber with the heel of(4) Clear the path of the recoil.(5) Turn the firing switch to FIRE.(6) Announce "UP."

Di.d you do anything wrong?

a. Step (2) is wrong.

b.' Step (3) is wrong.

L;.1111kA step is missing,

d. All of the steps are correct.

31 43

You do the

.ght hand.

Motor Manipulation (Reticle Manipulation)

FOR THE NEXT PROBLEMS ENGAGE ALL STATIONARY TARGETS A. CENTER OF MASSAND ASSUME ALL MOVING TARGETS ARE TRAVELING AT 15 MPH.

275. You are gunner on an OGLA] tank. The tank commander gives the firecommand:

"GUNNEP,, SABOT, TANK"

Which (--E tbe following sight pictures would you take up using the periscopereticle?

a.

C.

32 4 4

b.

d.

1

APPENDIX C

EVALUATION OF THE TELESTRATOR EQUIPMENT4

One of the reasons the television test was designed and producedwas to evaluate the Teletrator equipment (also known as the Telestarequipm,mt). The novel component of this equipment is an electronictablet which can be fitted over thefece of a television screen. Thetablet will record the horizontal and vertical (XY) location when itis touched with an electronic contact point embedded in a stylus orsimilar indicator (such as a gansight reticli). By the proper use ofauxiliary recording equipment it is possible to record the place andtime the screen is touchr!d. The recording equipment includes a counterwhich keeps a running toAl of the number of items, number of answersattempted, and oumber of correct answers.

The complete system includes the electronic tablet, a programingunit, and several student units. The programming unit is used by the testdeveloper to place electronically on the television tape the XYcoordinates or the correct answer for each test item and the timeperiod during which the equipment will accept this answer. The studentunit compares electronically the programed answer and the examinee'sanswer an.1 records the results.

The student unit provides three types of feedback to the examineefor each test item. Immediate feedback is provided by a high-pitchedtone and a small red light that comes on for a correct answer versusa low-pitched tone and no light for an incorrect answer. Slightlydelayed feedbaCk comes from a counter which shows'new totals of itemsand correct answers at the end' of the programed time period foranswering each problem.

As to whether the lelestrator equipment has any merit or not,' itis necessary to examine both:

1. The equipment itself, as designed and produced, andThe overall testing strategy which includes .(a) Individualresponding, (b) Television stimulus, (c) Immediate feedback,and (d) Time limit on each response.

A. The Euai;-2rnent

As.with most newly designed equipment, the Telestrator containedmany bugs and never worked properly. HoweVer,.it was possible to testsome aspects of the equipment by using human graders to record rightor wrong answers by the examinees according to the time and place thescreen was touched. Several pi 7)t tests were run with the followingresults.

`This suunary Telestrator operation was submitted previously to theTraining Support Division, TRADOC.

3; 45

(1) Accuracy. There is a fundamental flaw in the Telestratordesign insofar as precise responding is concerned. The equipment c. Lsclaimed to he accurate t0 1/4 inch. However, due to parallax theactual accuracy was mure on the order of 1 or 2 inches. This gross-ness effectively eliminated the use of the reticle test items becausewith any reasonable size reticle no discrimination was possible `or_leads or rar. The grossness also eliminated many test items inwhich the examinee was required to discriminate among several taneontr, is. The spa ing between these controls as shown on the screenwas not grat enough to permit exact_ programin of the answers, andone answer box would overlap another. The parallax results frommounting the electronic tablet at some distance from the actualtelevision screen (due to curvature of the television screen the parallaxincreases as one approaches the edge of the screen) .

in e. der to continue the experiment and test the idea of respondingto television, the electronic tablets were removed from the televisionmonitors and the examinees were reguirdd to touch the face of theactual television screens. This completely eliminated all parallax andpermitted the use of reticle items and other precision responding.

(2) Video pre-sentation. The electronic tablet is constructed insuch a manner that it blocks a 1-inch-wide area around the outer edge ofthe television screen. This is a serious limitation because it isnecessary to use a small. -size monitor for such closeup work and thisouter 1 inch covers a substaftt..ial part of the available-screen area.

B. Testing Strategy

Because of the device's failure to work properly and the poor designof the electronic tablet, it was not possible to evaluate the testingstrategy completely. However, by eliminating the parallax (removingthe electronic tablets) and using human graders to record responses,it was possible to make a limited test of' the strategy.

(1) Responding to television. The examinees seemed to have a littletrouble understanding the test items, and responded very precisely.Three types of test items were used; Multiple choice; Error detection,and Reticle manipulation.

No training other than instructions was required for learning torespond to the multiple-choice and error-detection'items. Approximately10 minutes were required for training on the reticle-manipulation items.

(2) Time to respond. Ten seconds were allowed for responding toeach item. The time limit was generous and most examinees responded tomost items in less than 5 seconds.

(3) Perception of test. The examinees perceived the test as being"fair" and most actually preferred the television test over the hands-ontest.

34

4o

(4) Comparison with other ,sts. The telovisi n Lest was compared

to parallel paper-and-pencil and hands-on tests.

(a) Paper-and-pencil test. Overall there was little difference:Jetwn the mean scores on the television and the paper- and -p. rcilHowev.r, on an item-hy-item b.:ihis there was coniderable diffference for

T;I:; . -..c!ction, the television score were much hotter:ole manipulatiotH the television scores were worse.

P ) There was a low positive correlation betweent± tetc:.-ision test JH sire hnods-on tent. This correlation was less than

desi,ehble.

The exrui Ht was too limited to permit any conclusions at this timeto the vol .Aity of the above results.

C.7)) Feedback. Because the eaminces wore tested four at a time andbecause the Telostrater equipment was not work it was not ,.ossible

to provide immediate feedback after each iter'

(6) Eye fatigue. The television test and the responding moder'7uired the examinees to stare continually at the television monitor.Thcire were many coriaints of eyestrain and there is some evidence thatth,.-J afternoon television examinees performed more poorly than the morningtelevision examinees.

Conclusions and Recommendations

(1) The Telestrd'Jor equipment as presently designed should be rejected

because of the pa_allax Taroblem.

(2) The television method appears to offer enough promise to warrantthe testing of other response devices which do not have the parallaxproblem.

(1.) There are many unknowns in television testing and the overalltesting strategy, and the research etfort needs to be greatly expandedsoon as:

(a) A more definitiv comparison with hands-on tests.

(b) research on the "immediate feedback" idea.

(c) Using alternative response device.

(d) Comparison with slide-tape devices.

(e) Further research on eye fatigue.

DISTRIBUTION

4 OASD2 HODA IDAMI-CSZ!

HODA (RAPE -PBPHODA IDAMAAR)

1 HODA (DAPEHREPO)1 HODA {SCR D10)

HODA iDAMtT)0T-C11 HODA -DAPC4'MZ-Ai

HOD,,.. )ACH-PPZ Al1 HODA (DAPEHREI1 HODA !DAPEMPO.C)1 HODA (DAPEDW)1 HODA (DAPEHRL.'1 HODA IDAPECPSh1 HODA (DAFD-MFA)1 HODA IDARD-ARS-P)1 ' -IUD. (DAPCPASA)1 HEIDA (DUSA1 HODA IDAMO,R0R11 HODA (DASG),1 HODA IDATOP1)1 Chief. Consult Div (DAOTSGi, Adelutn, MD

1 Mil Asst. Hum Res, ODDR&E, OAD (E&LS)USARAL, APO Seattle, ATTN: ARAGP-R

1 HO First Army, ATTN: AFKA01-TI2 HO Fifth Army, Ft Sam Houston1 Dir, Army Stf Studies Ofe, ATTN: OA, ..A IDSP)1 Ofc Chief of Stf, Studies Ofc1 DCSPER, ATTN: CPS/OCP

1 The Army Lib, Pentagon, ATTN: RSB Chief1 The Army Lib, Pentagon, ATTN: ANRAL1 Ole, Asst Sect of the Army I R&D11 Tech Supt.ort Ofc, OJCS1 USASA, Arlington, ATTN: IARD-T1 USA Tisch Ofc, Durham, ATTN: Life Sciences Dir2 USARIEM, Natick, ATTN: SGRDUECA1 USATTC, Ft Clayton, ATTN: STETC-MO-A

1 USAIMA, Ft Bragg, ATTN: ATSUCTD-OM1 USAIMA, Ft Bragg, ATTN: Marquat Lib1 US WAC Ctr & Sch, Ft McClellan, ATTN: Lib1 US WAC Ctr & Sch, Ft McClellan, ATTN- Tng Dir1 USA Quartermaster Sch, Ft Lee, ATTN: ATSM-TE1 Intelligence Material Dev Oft:, EWL, Ft Holabird1 USA SE Signal Sch, Ft Gordon, ATTN: ATSO-EA1 USA Chaplain Ctr & Sch, Ft Hamilton, ATTN: ATSC-TE-RD1 'JSATSCH, Ft Eustis, ATTN: Educ Advisor

1 USA War College, Carlisle Barracks, ATTN: Lib'2 WRAI 9, Neuropsychiatry Div1 DLI, SDA, Monterey1 USA Concept Anal Agcy, Bethesda, ATTN: MOCA-WGC1 USA Concept Anal Agcy, Bethesda, ATTN: MOCA-MR1 USA Concept Anal Agcy, Bethesda, ATTN: MOCAJF1 USA Artic Test Ctr, APO Seattle, ATTN: STEAC-MOASL1 USA Artie Test Ctr, APO Seattle, ATTN: AMSTE-PL-TS1 USA Armament Cmd, Redstone Arsenal, ATTN: ATSKTEM1 USA Armament Cmd, Rock Island, ATTN: AMSARTDC

1 FAA-NAFEC, Atlantic City, ATTN: Library1 FAANAFEC, Atlantic City, ATTN: Hum Engr Br1 FAA Aeronautical Ctr, Oklahoma City, ATTN: AAC44D2 USA Fld Arty Sch, Ft Sill, ATTN: Library1 USA Armor Sch, Ft Knox, ATTN: Library1 USA Armor Sch,.Ft Knox, ATTN: ATSB-DI-E1 USA Armor Sch, Ft Knox, ATTN: ATSB-DT-TPi USA Armor Sch, Ft Knox, ATTN: ATSB-CD-AD

ARI L,-,stributIon List

2 HOUSACDEC, Ft Ord, ATTN: Library1 HOUSACDEC, Ft Ord, ATTN: ATEC-EX-E-Hum Factor;2 USAEEC, Ft Benjamin Harrison, ATTN: Library1 USAPACDC, Ft Benjamin Harrison, ATTN: ATCP-HR1 USA Comm-Elect Sch, Ft Monmouth, ATTN: -EA1 USA,EC, Ft Monmouth, ATTN: AMSEL-CT-HDP1 USAEC, Ft Monmouth, ATTN: AMSE L1 USAEC, Ft Monmouth, ATTN: AMSEL-Sl1 USAF!', Ft Monmouth], ATTN: C, Facl Dev Br1 USA Materials Sys Anal Agcy, Aberdeen, ATTN: AMXSY -P1 Edgewood Arsenal, Aberdeen, ATTN: SA R EA- BL-H1 USA Ord Ctr & Sch, Aberdeen, ATTN: ATSL-TEM-C2 USA Hum Engr Lab, Aberdeen, ATTN: Library /Dir1 USA Combat Arms Tng Bd, Ft Benning, ATTN. Ad Supervisor1 USA Infantry Hum Rsch Unit, Ft Berming, ATTli: Chief1 USA Infantry Ed, Ft Benning, ATTN: STEBC-TE-T

USASMA, Ft Bliss, ATTN: ATSS-LRCI USA Air Def Sch, Ft Bliss:ATTN: ATSA-CTD-ME1 USA Air Def Sch, Ft Bliss, ATTN: Tech Lib1 USA Air Def Ed, Ft Bliss, ATTN: FILES1 USA Air Del Bd, Ft Bliss, ATTN: STEED-PO1 USA Cmd & General Stf College. Ft Leavenworth, ATTN: Lir)1 USA Cmd & General Stf College, Ft Leavenworth, ATTN:: TSW-SE -L

1 USA Cmd & General Stf College, Fr Leavenworth, ATTN: Ed Advisor1 USA Combined Arms Cmbt 'Dev Act, Ft Leavenworth, ATTN: DepCdr1 USA Combined Arms Cmbt Dev Act, Ft Leavenworth, ATTN: CCS1 USA Combined Arms C:-nbt Dev Act, Ft Leavenworth, ATTN: ATCASA

USA Combined Arms Cmbt Dev Act, Ft Leavenworth, ATTN: ATCACO --E1 USA Combined Arms Cmbt Dev Act, Ft Leavenworth, ATTN: ATCACC-Cl1 USAECOM, Night Vision Lab, Ft Belvok, ATTN: AMSEL-NV-SD3 US/ -omptrter Sys Cmd, Ft Belvoir, ATTN: Tech Library1 USA, RDC, Ft Belvoir, ATTN: STSFB-DO1 USA E '",ch, Ft Belvoir, ATTN: Library1 USA Topographic Lab, Ft Belvoir, ATTN: ETL -TD-S1 USA Topographic Lab, Ft Belvoir, ATTN: STINFO Center1 USA Topographic Lab, Ft Belvoir, ATTNtETL-GSL1 USA Intelligence Ctr & Sch, Ft Huachuca, ATTN: CTD-MS1 USA. Intelligence Ctr & Sch, Ft Huachuca, ATTN: ATS-STD--MS1 USA Intelligence Ctr & Sch, Ft Huachuca, ATTN: ATSI4TE

1 USA Intelligence Ctr & Sch, Ft Huachuca, ATTN: ATSlf- TEX -GS1 USA Intelligence Ctr & Sch, Fr Huachuca, ATTN: ATS1-CTS-OR1 USA intelligence Ctr & Sch, Ft Huachuca, ATTN: ATSI-CTD-DTI USA Intelligence Ctr & Sch, Ft Huachuca, ATTN: ATSI-CTD-CS1 USA Intelligence Ctr & Sch, Ft Huachuca, ATTN: DAS/SRD1 USA Intelligence Ctr & Sch, Ft Huachuca, ATTN: ATSI-TEM'1 USA Intelligence Ctr & Sch, Ft Huachuca, ATTN: LibraryI CDR, HO Fr HuactOca, ATTN: Tech Ref Div2 CDR, USA Electronic Pivg Grd, ATTN: STEEP-MT-S1 CDR, Project MASS-TER, ATTN: Tectl Info Center1 Hg MASSTER, USATRADOC, LNO1 Research Institute, HQ MASSTER, Ft Hood1 USA Recruiting Cmd, Ft Sherdian. ATTN: USARCPM-P1 Senior Army Adv., USAFAGOD/TAC, Elgin AF Aux Fld No. 91 HQ USARPAC, DCSPER, APO SF 96558, ATTN: GPPE -SE1 Stimson Lib, Academy of Health Sciences, Ft Sam Houston1 Marini: Corps Inst., ATTN: Dean-MCI1 HOUSMC, Commandant, ATTN: Code.MTMT 511 HOUSMC, Commandant, ATTN: Code MPI-202 USCG Academy, New London, ATTN: Admission2 USCG Academy, New London, ATTN: Library1 USCG Training Ctr, NY, ATTN: CO1 USCG Training Ctr, NY, ATTN: Educ Svc OfcI USCG, Psycho! Res Br, DC, ATTN: GP 1/621 HQ Mid-Range Br, MC Det, Quantico, ATTN: P8.6 Div

I rifrki/UL, Y t Monroe, I N 313F{ _AD

1 USATAADOC, Ft Monroe. ATTN: ATTS- -EA1 USA Fon:es Cmd, Ft McPherson, ATTN: Library2 USA Aviation Test Bd, Ft Bucker, ATTN: STEBG -PO1 LISA f(); Aviation Safety, Ft Rucker, ATTN: L:b ary1 USA Ay.; for Aviation Safety, Ft Rucker, ATTN: Educ Advisor1 USA Avi,rtror, Sch, Ft Rucker, ATTN PO Drawer 01 HDUSA Aviation Sys Cmd, St Louis, ATTN: AMS.AVZDR2 USA Aviation Sv3 Telt Act , Edv.ards AFB, ATTN: SAVTET1 USA Air Oef Sr.h, Ft Bliss, ATTN: ATSA TEM1 USA Air Mobility Fisch & De" Lab. Moffett Fld, ATTN: SAVDLAS1 USA Aviation Sch, Res Tog Mgr, Ft Rocker, ATTN: ATSTTATM

! - -SA Aviation Sch, CO, Ft Rocker, ATTN: ATSTDAHO, DARC01,1, Alexandria, ATTN: AMXCD -TLtic], DARCOM. A:exam:no, ATTN. CDRUS Military Acadilmv West Point. ATTN: Serials Unit

1 US M Diary Acatrny, West POint, ATTN: Ofc of Milt LdrshoUS Military Acad.:my, W-..st Point, ATTN: MAOR

1 USA )i)tandarcitzatinn Go, UK, FPO NY, ATTN: MASE -GCrib: of Naval Rsch, AtIntr;ton, ATTN. Cods' 452Of... of Naval Fisch, Arlington. ATTN Code 459

1 Qtr. Rwh, el leatt,ci, ATTN. Code 4501 On: ref Nava! R,ch. Ai lington, .1)(1/3-N7 Coat! 441I Nail Aen3,,pc., Med lies bah Pensacola, ATTN: Acous Sch Div/ Naval Acrostic Med Re, Lab, Pens.;(cola, ATTN: Coda L511 Naval Acro,:u.: Med Re,: Lab, Pensacola, ATTN: Code L51 C.i1 i t f NavPer: ATTN: Perr.-0131 NAVAIRSTA, Norfolk ATTN: Safety Ctr1 Nov Ot-itai(olca(0):c. DC, ATTN: Code 6251, Ch;)rtt, & Tech1 center of fl,ivai Anal, ATTN. One Cir

(,),-../AitSy(a/;iarn ATTN. AIR -5313Cr4.1.4 71.3

NavHeiicupterStibSona 2, FPO Sr' 966011 AFHRI... hf:11 AFB1 ATHHL (f/ T1 Lown, AFB1 APHRL (AS) WPAFB, OH2 AFHPL IDOJZ) Brooks AFB1 i'Ar-HRL (DOJN) Lackland .AFB1 HOUSAF (INYSD)1 HOUSAF (DPXXA)1 AFVTG RD) .'andolph APB3 AMRL (HE) WPAFB, OH

1-1,P Inst n./ Tcchs, ',WAIT, OH, ATTN ENE/SL1 AFC IXPTD I Randolph: AFB

USAF Li), Brooks AFB ISUL-4), ATTN: DOC SEC1 AFUSR 101_1. Arlington1 Al Lng C.rnth, McClellan AFB, ATTN: ALC/DPCSI Air F( Acaslenly, CO, ATTN: Dept ni Bel Scn5 N,vIA'rs & Dev Ctr, Sari Diego2 NJ-ay Med Neurousychiatric Rsch 1.1int, San Diego1 Nil Ee,00nic Lab, San Diego, AT Res Lab

Nip: -frnor.en, San Diego, ATTN: Code 9000Lib1 Nas:PustGraSch, Monterey, ATTN: Code 55Aa

NoePostGr.)Sc:1, Monterey, ATTN: Code 21241 NavTrnyEquioCtr, 0( lando, ATTN: 'Tech Lib1 0.5 nt;p,' of Labor, QC, ATTN: Manpower AdminI US Dew of Justice, DC, ATTN. Drug Enhorce Admin1 N3; BUS Stanrfartfs, DC, ATTN: Computer Info Section

1 Nat Clearing House' for MH Info, Rockville1 Deaver Federal CAL Lake,.vood, ATTN: BLM

12 Defense Documentation Center4 Dir Psych, Army Hg, Russell Ofcs, CanberraI S ientlfic Advsr, MA Bd, Army Itio, iussell Ofcs, Canberra1 MA and Air Attache, Austrian Ernb((sSy1 Ct,11rf. ,13 Recherche Des Facteurs, Humaine de la Defense

Nationale, BrusselsCenadtan Joint Staff Washington

1 :JAI'. Staff, Royal Canadian AF, AT TN: Pers Std Anal Br3 Chief, Canadian 00 Rsch Staff, ATTN: CiCRDS(VV)4 British Def Stafr, itish Embassy, Washington

38

mulitterusykratoqrsic t !Anise, Copehagsn1 Military Attache, French Embassy, ATTN: Doc Sec,1 Medec'n Chef, C,E,R.P.A.Arsenal, ToutoniNavai Flvrice1 Prin Scientific Off, Appl Hum Engr Rsch Div, Minist y

of Defepse, Na y., Delhi1 Jars Rsch Ofc Library, AKA, Israel Defense Forces

1 Min,ster;s van Defensie, DOOP/KL Atd SociaalPsychologi$che Zakert, The Hague, Netherlands

4,9