ED 227 162

AUTHORTITLE

INSTITUTIONSPONS AGENCY

REPORT NOPUB DATE

/CONTRACT// NOTE

PUB TYPE

EDRS PRICEDESCRIPTORS

IDENTIFIERS

DOCUMENT usum

TM 830 183

Mckinley, Robert L.; Reckase, Mark D.The Use of the General Rasch. Model withMultidimensional Item ResponseData.American Coll. Testing Program, Iowa City, Iowa:Office of Naval'Research, Arlington, Va. Personneland Training Research Programs Office.

,ONR-82-1Aug 82N00014-81-K081738p.

Reports -,Researth/Technical (143)

MF01/PCO2 Plus Postage.*Goodness of Fit; Item Analysis; *Latent TraitTheory; *Mathematical Models; *Psychometrics;Research Methodology; Statistical Analysis; TestInterpretation; Test Items*Multidimensional Approach; Multiple MdasuresApproach; *Rasch Model

ABSTRACT. Several special cases of the general Rasch model,

varying in complexity., were investigated to determine whether theycould successful* model realistic multidimensional item responsedata. Whether the parameters of the model could be readilyinterpreted ieas also investigated. The models investigated included:(1) the vector model; (2) the product term model; (3) the vector and.product term model; (4) the reduced vector and product term model;and-(5) the item cluster model. Of the models investigated, all but

- the reduced vector and product term model and thd item cluster model .

were rejected.as being incapable of mOdelling realisticmultidimensional data. The item cluster model appears to be a usefulmodel, but .,its applications may be limited in scope. Of the modelsstudied, the reduced vector and product term model was found to bethe most capable of mOdellitg realistic multidimensional data.Although the estimation of the parameters of the reduced vectors andproduct term model may. be more difficult than it would be for othermodels, this model alipaars to be the model that is most worthpursuing. (Author/PN)

*t********************************************************************** Reproductions supplied by EDRS are the best that can be made ,*

* from the original document. *

**********************************************************************

r..

4U S DEPARTMENT OF EDUCATION

NATIONAL INSTITUTE OF EDUCATIONEDUCATIONAL RESOURCES INFORMAtION

CENTER 1E+310T h6 do,,,rnvo% bOVI, 11,1)10,loi thi

ei eAr rd lone p.. Peoo or ortorotdIOnurnJtirrijtMavir L'odoje, have 1.11,e0 r11015 , to 411010Vf

ovoorJoc ton o0.11,tV

PO nts WO/ 0, opirpons stated in ttns dote'nye? dn 00( flet (.1,3,4 represent oft, al NIE

-postOon Or polil

The Use of the General Rasch Model.with Multidimensional, Item Response Data

\t,

Robert L. McKinleyand

Mark D. Reckase

Research Report ONR 82-1August 1982

'FERMItSION TO REPRODU4 THISMATERIAL HAS BEEN GRANTED BY

-HA& OLic..

ga-t4-,A

TO THE EDUCATIONAL RESOURCESINFORMATION CENTER (ERIC)"

The American College Testing ProgramResident Programs Department

Iowa City, Iowa 52243

Prepared under Contract No. N00014-81-10817with the Personnel and Training ,Research Programs

Psychological Sciences DivisionOffiee of Naval Research

Approved for public release; distribution unlimited.Reproduction in whole or in part is permitted forany purpose of the United States Government.

')4

f

SECURITY CLASSIFICATION OF THIS PAGE (*hen Data Entored)

REPORT DOCUMENTATION PAGE READ INSTRUCTIONSBEFORE COMPLETING FORM

1 REPORT NUMBER

ONR 82-1

2. GOVT ACCESSION NO. 3 RECIPIENT'S CAfALOG NUMBER

4 TITLE (and Subtitle) ..,, c

The Use of the Generar Rasch Model withMultidimensional Item Response Data .-

5 TYPE OF REPORT 6 PERIOD COVERED

Technical Report

6 PERFORMING ORG. REPORT NUMBER

I AUTHOR(*) .

Robert L. McKinley andMark D. Reckase

8 CONTRACT OR GRANT NUMBER(*)

'N00014-81-K0817

9 PERF,ORMING ORGANIZATION NAME AND ADDRESS

The American College Testing ProgramP.O. Box 168Iowa City, IA .52243

.

10 PROGRAM ELEMENT PROJECT. TASKAREA 6 WORK UNIT NUMBERS

P.E.: 61153N Proj.: RR042-04T.A.: 042-04-01W.V.: NR150-474

ti CONTROLLING OFFICE NAMEAND ADDRESS .Personnel and Training Research PrOgrams:Office of Naval Research . iArlington, VA 22217 - 'N

12 REPORT DATE

August 1982 .

.

13 NUMBER OF PAGES

2314 MON.ITORING AGENCY N AME 6 ADDRESS(11 different from Controlling Office)

. .9

15 SECURITY,CL ASS. (of tiliIeport)

.

Unclassified,154. pj15.64R111CATION'DOWNGRADING

i:,.

16 DISTRIBUTION STATEMENT (of this Report) . . .

Approval for public release;'-distribution unlimited. Reproduction inwhole or in part is .permitted *for any purpose of the United StatesGovernment.

. t.

t..

17 DIerRIBUTION STATEMENT (of fh abtract entered in Block 20, if different from Report) .

t

. '18. SUPPLEMENTARY NOTES ..

. .

.* .

1

1 .

19. KEllWORDS (Continuo on revers aide if noceetiaty and identity by block number).

Latent Trait-Models .

.

General Rasch Model ..

Multidimensional Models ,

.

. ,

20

I

ABSTRACT (Continu on revere. side it ncioaitary end identify by block number)Several special.cases of the general Rasch mw.lel, varying in complexitvo '

were investigated to determine whether.they could successfully modelrealistic multidithensional item response data. Also investigated waswhether the parameters of the model could be readily interpreted. Of

the formulations of the model investigated, all but two were found tobe incapable of modeling realistic multidimensioual item response

.°clata. One of'the remaining formulations of-the model was found tohave limited applications. The version of the modeT found ,to be most

DD"" 1473.AN 73 EDITION OF 1 NOv 65 IS OBSOLETE

3 N 0)02:i.F. 014- 6.601 SECURITY CLASSIFICATION OF THIS PAGE Mien Data Entottod)

vg.

SECURITY CLASSkICATION OF THIS PAGE (When Data Entorod)

useful is an extension of the two-parameter logistic model.

S/N 0102- LE- 01.4-6601

SECURITY CLASSIFICATION OF THIS PAGE(lrhen Data Entered)

CONTENTS

Page

Introduction1

Multidimensional Latent Trait Models1

Summary 4

Method5

Design5

Analyses /

5

Results, 6

.

Vector Model6

Yroduct Term Model9

vector and Product Term Model 410

Reduced Vector and Product Term Model ,;i

12Item Cluster Model

15

Discussion19

Vector Model 19Product Term Model. . ..... . 20Vector and Product Term Model 20Reduced Vector and Product Term Model 20Item Cluster flodel 20

Summary and Conclusions21

References22

The Use of the General Rasch Modelwith Multidimensional Item Response Data

Citent trait theory has become an increasingly popular area for researchand application in recent years. Areas of application of latent traitfheory have included iai1ored testing (McKinley and Reckase, 1980), equating(Marco, 1977; Rentz and Rashaw, 1977), test scoring (Woodcock, 1974), andcriterion-referenced measurement (Hambleton, Swaminathan, Cook, Eignor, andGifford, 1978). While many of these applications have been successful,they are limited to areas in which the tests used Measure predominantly onetrait. This limitation is a result of the fact that most latent traitmodels that have been proposed assume unidimensionality. Because of thisrequirement, in some situations latent trait models have not been successfullyapplied. For examPle, in achievement testing the goal is not to measure asingle trait, but to sample the content covered by instruction. Therefore,most latent trait models are inappropriate since tests designed for thiss'purpose generally cannot be considered-to be unidimensional. Even when thegoal is to measure a sihgle ttait, if dichotomously scored items are usedno gepprally accepted method exists for forming unidimensional item sets,for'ittermining_the dimensionality of existing item sets, or for-testingthe fit of the unidimensional model to the data.

An alternative to trying either to construct unidimensional item setsor to fit a unidimensi-onal model to already existing item sets is to developa multidimensional latent trait model. Several such models have beenproposed (Bock and Aitkin, 1981; Mulaik, 1972; Rasch, 1961; Samejima, 1974;Sympson, 1978; Whitely, 1980), but little research has been done usingthese models. Some work has been completed on the estimation of the parametersof the13ock_and Aitkinmodel (Bock and Aitkin, 1981), the multidimensionalRasch mo el (Reckase, 1972), and the Whitely,lodel (Whitely, 1§80), but noextensiv resehrch-has been completed44'the qharacteristics and propertiesof any of ese model's. The purpose of thrsliaper is to present the resultsof research on the characteristics and properties of the multidimensionalRasch model. Before presenting these results, however, the multidimensionalmodels that have been proposed will be briefly discussed, as will the

, research that has investigated theNchar.acteristics of these models.

Multidimensional.Latent Trait Models

Three of the multidimensional latent trait models thst have beenproposed have been extenions to the multidimensional case of the unidimen-sional Rasch model (Rasch, 1960). The unidimensional Rasch model is givenby

.

. exp(x (0 + cyi)) .

P(x I ) ='ii i

. (1)11 °.i i 1 + exp(O + )

ii

4

where0.is the ability parameter for Pergon j,-0, is the item easiness

parameter for Item i, and lAx.1 0

j i,0.) is the probability of response xij / j

(0 or 1) to Item i by Person j. The multidimensional model proposed bylysch (1961) is given by

11,(x ) ,\ ttxpWx)'3 + 0'.)((x)0. 4- :(x).1 (2)'jj'-j Yk(2.15f1i) - -j - -1 - j-, -1

-2-

where 6 G. e and P(x 6 ,cr.) are as defined abov; I, x, and p are-1 -1

scoring functions which are functions,of x,only; and y(0 ,a.) is..a normal-

izing factor necessary to make the,probabiIities of the response alternatives .

sum to 1.0. The scoring functions and X act as weights for the

parameters, while the p teTm is used.to adjust the scale for different

scoring procedures. Both'th'd scOring-functions and fhe p term.depend ox

the score obtained by a person on the item. In brdet to apply this model

to multidimensional data, G., I,.and are defined as column vectors,

I(x)-6 and (x)-cr. are inner products of vectors, and i(x) is defined as a-1

matrix. The\terms I and now represent vectors of weights for the different

elements in the cr-,and 6-vectors. The x matrix is a matrix of weielts.

Rasch never attempted to apply this model.. :

.

Reckase (1972) tried to apply the generalized Rasch mOdel,to real and

simula'ted item response data with'limited success, In this study, the

multidimensional model fit multidimensional data no better than did the

unidimensional Rasch model. hoWever, Reckase did"not include the 6 -x(x)(3.-1

term in the model, which may,have resulted,in the poor fit of the model to

multidimensional data. In additibn, several methodological problems may;

,have contributed to the poor results of the study. First, the sample size

used to estimate the parameters of the model was relatively small for the

number of parameters estimated. Second, in addition to estimating the'parameters of the model, Reckase also estimated the values of the I and

scoring functions. ' Finally, in order to estimate the parameter,vectors,the dichotomously s'cored items used in the study were combined into clusters

,to form nominal response patterns. The most appropriate way to form the

, cluAers, was not known, which may have caused problems in the estimation of

.the/parameters. :Pespite these difficulties, a least squares estimationprocedure was developed which did yield somewhat reasonable parameter

estimates.

Mulaik (1972) also proposed a multidimensional model that is agenerallzation of the Rasch model. The model proposed by Mulaik is"given

byJ

2:exp[(0. )x ]

3k ik ijk=1

P(x le ;C1 )

1 + Eexp(e.k

+ aik

)j

k=14

wheret)j

thk ikis e ability parameter for Person j on Dimension k, and

,the difficulty .rameter for Item i on Dimension k. Although Mulaikjiever

applied this mod 1, he did suggest procedures for estimating the parametet.s

of the model far hree separate cases: when item responses are normally

distiibuted and ha e a common variance for all items apd,subjects; whenitem reponses follo a Poisson distribution; and when item responses are

dichotomous.

-3-

Samejima (1974) proposed a multidimensional latent trait moderthet is. a generalization of a different unidimensional model. The model.proposedby Samejima is1based on the two-parameter normal ogive model: This model

' is given by

= V[a.-(9. b.)) (4)

where4)00-isthe,dormaldistributionfunction,a.is a column vector of

item discrimination.parameters, b, is a column 'Vector of item difficulty

-parameters, and 0, is a column vector of abiaity parameters for Person j.

Unfortunately, the basic derivation of this model used the continuouresponse version of the normal ogive, model. Therefore, its.use withdichotomous data requires that item scores be translated to the continuousscale. Since no procedure for translating item scores to the continuousscale is available, the moAel cannot at present be applied to dichotomousdata. Like Rasch and Mulaik, Samejima never applid this model, but onlysuggested how the parameters might be estimated.

Sympson (1978) proposed a multidimensional model based on the three-parameter logistic model. The Sympson model postulates that the probabilityof a correct response is determined by the product of the conditionalprobabilities of a correct response on each of the imensions being measured.The three-parameter logistic model is gively

P(x. =1 18. a.,b ,c.) = C. + (1 c )ij j' i

I + expfDa.(e.11):))

j

expplai(ei - b.)j

(-5 )

,

where' is the ability parameter for Person,j, aiis the item discrimination

.

parameter for Item i, bi

is the item difficulty parameter for Item i, c. is

a pseudo-guessing parameter.for Item i, and D.= 1.7. The three-parameterlogistic model is used-to model the conditional probability for each dimension,although the c

iparameter does not, have a separate value for each dimension,

but rather is a scalar parameter related ,to the item as a whole. Themultidimensional model is given by

m expCa (9 b I

F('-.1 a. ,b.,c.) = - c . ) IT ik jk ik 1jx . 0,1 (6)

k=1. 1 + exp[a.k(9jk

- b.ik

)]i

where the parameters are as defined above and m is the number of dimenpions.Although Sympson has 'done'some work on estimating the parameters of thismodel, no application of the model to multidimen&tonal data has yeC beenattempt9d.

The model proposed by Whitely (1980) is somewhat similar to Sympson'smodel. This model, called the multicomponent latent trait model, definesthe probability of a correct response to an item.as the product of theprobabilities of-performing §uccessfully on each cognitive component of theitem. The Whitely model is given by

-4- .

m expi(9 b )x. 1

.p(x. 1r; jk ik 1j

k=11 + exp19. - b

Lk

where all th parameters are as previously defined, Itican be seen't atthis model :s essentially another extension of the Rascfi model. The modelfocuses on t.he different cognitive skills required to perform on an itemrather thad the global dimensions hypothesized by Sympson. Estimation '

procedures/have been developed for the model and some applications havebeen made 6 rear data. However, because of the emilhasis placed on identi-'fying the !different cognitive skills required by an item, the applicationof this model is limited to data collected under very restricted experimentalconditions.

, x = 0,1 (7)

Bock and Aitkin (1981) have proposed a multidimensional two-paramenor9011 ogive model for,use with dichotomously scored response data. Thimodel is given by

P(x..=11 0.,y.,a.) = + a.'0.) / ac] (8)

whereY $.is the difficultarameter for Item i, a. is a column vector ofi

y p-1

discrimination parameters for Item i, 0. is a column vector of ability-J

parameters4or Person j, 4)(x) is the normal distribution function, and a. sis given by

= (11 ik

) (9)

Bock and Aitkin described a method for estimating the parameters ofthis'model, and presented the results of the application of the model tothe data for the Law School Admissions Test (LSAT) presented in Bock andLieberman (1970). The results of the application of the model to the LSAT.indicated.that a two-dimensional solution fit the data better than a one-dimensional solution. Fit was assessed using a likelihood ratio chi-squaretest.

Summary

Six different latent trait models have been Proposed for use with multi-dimgnsional item resporise data. .0f these six Models, two are of littleidterest here. The Samejima model is not designed for use with dichotomouslyscored item response data, and the Whitely model is appropriate only forspecial experimental conditions. Of the remaining four models, only theBock and Aitkin model and a specialtcase of the Rasch moderhave beenapplied, and no attempt has been made to extensively investigate the ,

characteristics and properties of any of the models. The purpose of this'research is to extensively investigate the characteristics of one of thosemodels, the generalized Rasth model:

-5-

Method

Design

The design of this research was to start with the most simple formulationof the multidimensional Rasch model, investigate its ability to describemultidimensional item response data, and if necessary to investigate increasinglymore complex versions of the model until good model/data fit was obtained.At each level of complexity We properties ot the model were investigated,and the reasonableness 4nd usefulness of the model were explored. This wasdone by generating test data to fit the particular form of the model beinginvestigated, and analyzing that data in an attempt to assess how well thecharacteristics of the data matched the characteristics of real data with

14

multidimensional charecteristics.

The most general formulation of the model investigated in this researchis the model described by Rasch (1961), given by Equation 2. The simplerformulations of the model used in this Yesearch were obtained by eliminating .

different terms from the model statement by setting the appropriate scoringfunctions equal to zero for all item scores.

For each model statement that was obtained, simulated tet data weregenerated to fit the model. Using the known parameters and mcidel stetement,predictions were made as to.the dimensionality of the generated data and

1'the characteristics of the hypothetical items. Analyses were,then performedon the simulation data in order to teSt the predictions. If:1j were foundthat a model statement could not be used to generate realist. data, interms of either dimensionalitY or item characteristics, .then e model was,rejected, and a different model statement was investigated.. s involvedalteriag the terms of the general model (Equation 2) that wer ncluded andthose that,were zeroed out.' In some cases, all of the terms a particularrejected model statement were retained, and one or mire addit al termsfrom the general model were added.

Analyses

The first analysis performed-on the simulation data generathe models was a factor analysis. Factor,analysis, in this caabeing used as a means of validating thq mopels, but as a proceddetermining whether the data generated from the models have chasimilar to those of real,test data. All of the factor analysesin this research were performed using the principal componentsphi coefficients. When the obtained and expected factor structudata did not match, follow-up analyses were performed in an attedetermine why the obtained factor s,tructure was different from wexpected.

Follow-up analyses included plotting the true item partAmetethe factor loadings and against traditional item statistics suchcorrect difficulty values and point biserial diCrimination valuanalyses were performed using both the unrotated factor loading mthe factor loading matrix rotated to the variMax criterion. Theof these analyses were three-fdld. ' One purpose was to determine

usingis not

e forcteristicserformedhod ons of thet to

was

against

proportion-Theseix and

posesther

-6-

the obtained factor structure was a result of the model statement, thevalues used for the model parpmeters, or both. The second purpose was tofacilitate interpretation of the model parameters, and the third purposewas to determine whether the model yielded items with reasonable characteris-tics. ln many cases it was necessary to generate additional data, usingdifferent values fox the parameters of the model, in order to answer specificquestions about a paqicular model statement.

Using the results of all of the analyses performed for a particularmodel, a decision was made as to whether the model adequately generateddata similar to real test data. If a model statement were rejected, anattempt was made to determine from the results of the analyses what changesin the model would yield a more acceptable model.

Results

Vector Model

4The first model that was investigated was a simple vector parameter

model. the 0. 'x(x)P. and p(x) terms were' eliminated, yielding the modelgiven by

-1.

P(x..1=3 ,s7 ) =11'-j exp0(x)"". + ;!(x)''.)_ _1 _

,ce )

where all the terms areNts defined for Equation 1, and 0., G., 1 and are-3 -1

vectors. For this model the scoring functions all took on values of Onefor a correct response and zero for_an incorrect response. This model wasselected first because it appeared to be a straightforwaxd extension of theunidimensional Rasch model (Rasch, 1960) to the multidimensional case. .The

dxpectation was that data generated according to this model vouldhave adimensionality that wOuld vary with the number of elements in the parametervectors. For instance, when data were generated using two elements in boththe item and person parameter vectors,- it was expected that the data would,yield a two-factor solution when factor anagrzed. This was not the case,however. Regardless of are number of elements in the parameter vectors,this model yielded one predominant factor. This was true regardless of theactual values of the parameters or the values sthat were used in the scoringfunctions.

Table 1 shows the first two .eigenvalues from a typical principalcomponent solution for the vector model. As can be seen, there is a Oominantfirst factor, with one minor factor. Table I also shows the unrotatedfactor loading matrix obtained for these particulan data, as well as theproportion-,correct difficulty and-the inner product of the item parametervec.tor and scoring function for each item (sum of the item parameters). As

can'be seen, thee.is little variation in the loadings on the first factor,while the minor factor is related to item diffitulty. Factor Irgenexallyhas pos,itive loadings for easy items and.negative loadings for hard items.Once ct was ascertained that the vector mbdel would not yield multi-factordata, i was not difficult to determine why. Equation 8 can be written as

1PU... ,c7 ) exp(a. 4-- .,.)

1.] j _i y(0 ,u.) j t

--i 1.

:

-7-

TABLE 1

Princip'al Component Factor Loadings Based onPhi Coefficients with ttle Sums of the Item Parameters

and Observed Prdportion Correct forTwo-Dimensional Vector asch Model

Item Sum of ItemParameters

ObservedDifficulty

I

Factor I Factor II

1 .89 .65 .57 .28

2 -.89 .33 .52

3 .43

4 2.02 .80 .47 .29

5 .59 .60c--)

.59 .27

6 -.91 .3 .52 -.14

7 -1.44 .24. .51 -.27

8 .47 .58 .56.

.13.

9 -1.05 .30 .51 -.24

10 -1.76 .20. .48 -.19

11 .98 .64. .58 .07

12 2.58 .88 .43 .35

13 -1.31 .25 .50 -.34

14 1.22 .71 .53 .29

. 39te

.

16 .05 .48 .56 .06

17 -2,33..

s ,-

.15 .46 -.36

.39

4 ., .38 -.53 .

20 2.26 .82 .43 .39

1Eigenvalues 5.42 1.18

r

WherelY1 =100-e..al (x)-G. . Equation 11 is the unidimensional. -1 -1

Rasch model, with inner producs af vectors as parameters. Therefore,regardless of the values.of thpeftedel parameters, as long as the inner,prodiict remains the same, the probability of a,response is the same.Therefore, the dimensionality of the vectors is unimportant, only tileproduct is critical. The'model is still a unfdimensional model. The

factor analysis* results typified by the solutidn,shown in Table i serve as. 'an empirical:demonstration that the vector model is a.unldimensional model.

It can also be empirically demonstrated that the inner p.roducts of the. scoring function and parameter vectors serve as parameters for the model.

Pi.gure 1 shows a plot of proportion-correct difficulty by the inner productof the scoring function and item parameten vectors, which for this case isjust the sum of the item parameters.. As can be seen, .there is an almostperfect relationship between the inner products and the proportidn-coerectscores. When data were generated using the unidimensional Rasch model,with the inner products from the two-dimensional model as parameters,exactly the same plot was obtained.

,i'igure 1

Relationship Between the Proportion Correct and the Sumof the Item Parameters for the Two-Dimensional

Vector Model

co

j 0 33s

- J16

o1

a.

ci

17

II 1

14

20

12

I 1 1

9-3.00 -12.00 -A.00 o'.00 1.00 2.00 3.00SUM OF ITEM FARRMETERS

Note: The numbers in the plot are the item numbers,

1 3

-9-

,f

Product Term Model

It was clear from the resulbs just reported that using parametervectors in an otherwise unidimensional model did not make it a multidimensionalmodel. Therefore, the vector model was rejected as a multidimensionalmodel. The nexf model that was investigated contained only the 0 'x(x)G.

term. This was the next model investigated because It involved more thansimple inner products of scoring and parameter vectors, but was simplerthan using both inner products and the 0,-x(x)G. term.

-J -1

When 0, and G. are vectors, x(x) riust be a matrix. The product 6.0.-

-J -1 -j-irepresents a matrix of products of all possible pairs of the elements inthe - and G.-vectors. For two-dimensional -.and G.-vectors,0j ei

i

.0102,

L2 1 2 21

The x(x) matrix is a scoring mptrix having an element for each element, ,

'of the 0.0. matrix. If the x(x) rfiatrix for the matrix in Equation 12 were-j-i

x(x) =0 1

(13)

A

foPa particulaeresponse x, then the numerator of the model statement forthat response would be exp(01G1 + 0202), The nonzero elements of the x(x)matrix indicate which elements of the 6.G.- matrix are included in the

exponent. It is clear from this that by selectively using zeros in theX(x) matrix, various products of 0 and G. elements can be selected.

-1Varying the values of the nonzero elements in x(x) assigns different weightsto different combinations. Thus, the product term model, given by

*-'

- 1p(x le ,o ) .exp[0" '(x)(-' 1 '..ii -.i -i- j- -iY (0 ,(3 )

-i -i-

..

is a very rich model in terms of the number of alternative formulations ofthe exponent of the model that are available. Unfortunately, when datawere simulated using some of these14

alternativ s, it was discovered,thatts model had an inconvenient property. Rega dles's of which formulationof.thi44,model was used, and regardless of what values' were taken on by theiteM parameters, the item proportion-correct difficulties were an approx-imately equal to each other. A closer examination of the product termmodel indicates why this occurred. Using the item parameters shown inTable 2, data were generated using

.

(14)

1

1 0

(15)

-10-

for a correct.response, and

01v(x. = 0) = (16)

I 0 0 I

for an_incorrect response. This yields a model given by

exp(9.1

a. 4. '3j 2%.-1)

p(x.. = 11

-1 1. +

5j1"-Ii2)13

(1'7)

where! the, EIJI( and . terms are elements in the 6- and a-vectors. Fromajk

Equation 17 it,can be seen.that the item parameters are similr to thediscrimination parameter in the'uni.dimensional two-parameter logistic (2P0model presented bY Birntaum.(1968) 'since they are multipliers of the person

parameters. In tact, if written as

P(x =-3

+ 0) + ji2

(0j2'+

0))ij -1--4:- Y(e ,c )

, 11 j1

.4,.. --j -i,

the model is essentially b two-dimensional two-parameter logistic mode,1wig.' both of the difficulty parameters equal to zero for all items. Becausethe data used for Table 2 were generated using a bivariate N(0,1) with

p = 0 distribution of ability, difficulty parameters of'zero.yielded apredicted Proportion-correct difficulty of .5.

(18)

A principal components analysis of phi coefficients yielded evidencethat the use of two item Parameters resulted in a two-dimensional model.The first three eigenvalues obtained for the data generated using the itemParameter values in Table 2 were 4.0, 2.4, and .9. The role of the itemparameters as-discrimination parameters in'this model is indicated bycomparing the item parameters shown in Table 2 with the rotated factorloading matrix, also shown-in Table 2. The correlations between the itemparameters and the factor loadings indicaced that there was a strong linearrelationship between the item parameters and factor loadings r = .98 for

al with Factor II, r = .99 for a2 with Factor I), supporting the conclusionthat al and a2 are acting as discrimination parameters.

Vector and Broduct Term Model?

The vector model that was investigated first was essentially a uni-

dimensional model that contAined a'difficulty parameter (the inner productas the only item parameter. The product term model is a multidimensional

mode), that contins discrimination pardmetes as the only item patameters.In order to obtain a multidimensional. modelAich contained a difficultyparameter, the vector and product term models were combined. A combination

of these two models is given by

it

..

TABLE 2. .

Item Parameters, Ptoportion Correct Item Difficulty and Factor L9adingsfrom a Varimax Rotated,Principal Components Solution on

Phi Coefficients for the Producct Term Model

.

Item al p Factor I,

'Factor IT

1 0.150 1.150 .48 .52 .11 ,

2 1.280 0.200 .50 .08 .56

3 0.260 1.350 .51 .57 .09 ,

4 1.000 , .0.300 .52 .14 . .52

0.250 . 1.050 .49 .47 ' .09,).

6 1.040 0.100 .51 .03 .57

7 0.110 1.150 .47 .52 -.01

8 0.200 1.200 .49 .57 .09

N9 1.400 0.300 .50 .12 .58

.q

10 . 0.300 1.200 .48 .54 .07.

11 1.350 0.150 .51 .08. .59

12 0.400- 1.200 .50 .53 .14

13 1.150 0.250 .50 .01 .52

14 - 0.150 1.300 :49 .61 . .04

15 1.000 0.250 .51 .11 .49

16 0.100 1.400 .46 .61 .04

17 1.3.50 0.150 .50 -.01 .59

18 1.250 .0.100 .52 7.03 .59

19 "0.200 1.500 .48 .62 .03

20 1.150 0.500 .51 s, .25 .46

'N

\

ot

-12-

, 4

1P(x.. e.,a.) exp (x)'a. + 0.'X(x)o.j. (19)

13 -3 -1 -1 -3 --3 -1

As can be seen in Equatiop 19, the (x)-6. term was eliminated when the two-Jmodels were combined.

Table 3 ghows the item parameters used to generate data to fit thevector and product term model. These data were-generated using two-dimensionalpargmeters. The scoring functions were also two-dimensiohal and werevectors of ones'for,a correct response and vectors of zeros.for an incorrectresponse for all elements. Table 3 also shows the rotated factor loadingsobtained for the first two factors from the principal coMponents analysis-of phi coeffiCients obtained for that data. The first three eigenvaluesfrom the solution are 5.26, 2.28, and 1.07. Initial analyses indicatedthat this model coal& be used to model multidimensional data, and that itemdifficulties were not constant (see Table 3). However, these analyses alsoindicated that it was hot realistic to use the same item parameters in boththe parameter vectors and the product term. The problem is indicated bythe magnitude of the correlation of the item proportion-correct difficultieswith the item point biserials. Because of the double role played by theitem parameters, the proportion-correct scores and point biserials had acorrelation of r = .94. That is not a very realistic situation. Therefore,this model was also 'rejected as.a reasonable method for describing multi-dimensional item response data.

Reduced Vector and Product Term'Model-

Since the analyses of the vector and product term model indicated thatthe same item parameters should not appear in both the parameter vectorsand the product term, the item parameter vector ahd the scoring funCtionswere altered so that parameters appeared in one or ,the other, but not both.In order to facilitate this, two additional elements were inserted into theitem parameter vector. For a correct response the first two elements werezeroed out of the product,term, while the last two were elements werezeroed out of the vector term.. This procedure results in the first twoitem parametert acting as difficulty parameters and the last two parametersacting as discrimination parameters. 'Although four item parameters wereused, only two dimensions were modelled in this case. For an incorrectresponse all of the-parameters were zeroed out. All nonzero elements inthe scoring functiiig were set equal to one. The resulting model is givenby

P(xii = 118j,21.) = exp[ail + a.1.2 + 0130j1 + a.i.40j4] (20)

-3 -1

where ,the 6 and crterms-are elements of the corresponding vectors.

The first three eigenvalues obtained from the principal componentsanalysis for this model are 5.39, 1.30, and .99. Table 4 shows the itemparameters that were used to generate the data, as well ai the factor

-13-

TABLE 3

Item Parameters, Proportion Corr'ect Item Difficulty,and Rotated Factor Loadings for

the Vector and Product Term Model

Item

1

2

3

4

5

7

8

11

e

12

13

14

15

16.

1f7

18

19

20

1 a2 Factcal I Factor II

.230 .190 .56 .63 .12

.880 2.180 .77 .73 -.10

.900 -1.920 :35 :08 .71

-.900 -.110 .32 .39 -.22

-.640 .830 .52 .58 -.35

-.540 -1.040 .16, _

.104.31

1.730i -1:350 .54 .34 .65

.946 .630 .69 .69 .21

.036 -.110 .46 .56 .11

-1.610 -.570 .13 .01 -.37

-1.170 1.260 .74 .75 .05

-.550 -1.070 .17, .04 .22

-.420 -.480 .31 .32 -.01

.220 -1070 .53 .60 .15

-.020 -1.676-

.21 -.04 .55

2.420 .370 .78 .61 .40

1.230 .400 .69 .68 .28

.250 .410 .58 .65 .08

.140 .760 .64 .67 -.10

-1.770 .550 .30 .27 -.60

6

TABLE 4

Item Parameters and Rotated Factor Loadingsfor the Reduced Vector and Product Model

Item a1

a2

a3

. Factor I Factor II

14 .206 -.503 .373 .997 .51 .01

2 -.164 .888 1.205 1.832 .60 .32

3 .448 .261 .766 .876 .34 .36

4 .814 -.008 1.321 1,714 .55 .34

t 5 .111 -.908 1.344 1.216 .41 .42

6 . -.947 .044 1.758 1.694 .46 .51

1

7 -.490 .111 .687 .738 .40 .24

1

8 P.553 -.502 .347 1.454 .61 , .07

9 -.344 .639 1.307 .127 -.06 .64

4

10 -.257 .303 .11 .824 .26 .39

11 -.069 -.542 .472 .404 .22 .25

_ 12 .779 .432 .392 .656 .30 .13

13 -.611 .571 .578 .1.252 .59 .13.0

14 -.140 -1.032 .334 1.066 .60 7.04

15 -.705 .Q81 .821 -.480 '.07 .44

.....

16 =.386 -.164 1.912 .24/ 4. .03 .71

17 -.154 .044 1.193 .537 .16 .56

-18 .474 .249 1.385 1.287 .49 .44

19 .438 -.210 '1.320 1.110 .42 .45

20 :294 .190 ' 1.634 1.492 .45 .49

1 4t

-15-

loadings obtained from a varimax rotation of the first two principal components.The item parameters that were used as multipliers (03 and G4) were allpositive in order to avoid having items with negative discriminations.

The results of the factor analysis of these data indicate that adominant first factor is present. However, there was a second componentpresent in the data which wa's strorigly related to the item parameters(r = .87 fcr 03 and Factor II, r = .87 for 04 and Factor I). The itemparameters in the product term were related to the factor loadings, whilethe sum of the item parameters in the vector term were found to be relatedto the proportion correct difficulty. The correlation between the sum ofthe parameters in the vector term and the proportion correct difficulty wasr = .98, indicating that the sum of the vector parameters act as difficultyparameters. There was not a significant correlation between the itemdifficulty and point biserial values (r =' .12). The sum of 03 and 04 had acorrelation of r = .96 with the item point biserials.

The analyses of the model set out in Equation 20 indicate that it hasAny desired characteristics. The rotated factor loadings are highlyrelated to the item parameters in the product term, the item difficulty ishighly correlated with the sum of the item parameter vector elements, andthere is no correlation between item difficulty and item discrimination.

One prob,lem that does exist with the data that were generated Is thatthe factor analysis resqlts indicated that the data had only one predominantfactor. One possible reason for this is that so many of the items hadlarge values for both of the'item pxrameters'in the product term. In orderto test this, data were generated for the set of item parameters shown inTable 5. The eigenvalues from,the principal components analysis for thesedata are 2.49, 2.28, 1.05, and 1.03. As can be seen, when using the itemparameters from Table 5 to pnerate data, there are twq factors of approxi-mately equal magnitude present in the data.

Ttem Cluster Model

Although the reduced vector and product terfn model appears to adequately'model multidimensional data, the presence of the produCt term compflcatesparameter estimation, since the separation of the itemCand person parametersis not possible through techniques of condit,ional estimation. Because ofthis, one more model that does not have a product term was investigated.This model is the item cluster model.

One'of the reasons the item vector model, given by EquSion 10, doeknot adequately model multidimensional data is that no information about thedifferent dimensions is preserved in the item score when the-item is dichoto-mously scored. The elements for the different Aimppsions are summed, andthe sums are treated as parameterS. If qt were paeSighje to score thedimensions separately, then the vector model thight.be able to model multi-dimensional data. This requires, however, polychatomous item scoring.

-16-

TABLE 5

Item Parameters for.the ReducedMector and Product Model

Item a1

a2

a3

a4 .

1 .977 .258 1.000 .000

2 .359 .728 .000 . 1.000

3 -.322 - .377 1.000 .000

4' -1.289.

1.128, .000 1.000

e5 -.613 .219 1.000 .000

6 1.299 .797 .000 1.000

7 .029 -.213 1.000 ' .000

8 -.360 -.862 .000 1.000

9 .769 1.000 .000 '

10 -1.447 2.092 .000'

1.000

11 -1.252 -.24 1.000.

.000

*1

.12 - -.778 -1..426 . .000 ,t, 1.000

,

:1'13 .668 -160 Lobo .000. .

14 2.102, -.025 .000 1.000

15 -:724 .968' 1.000 .000

16 1.230 -.535 .000 1.000

4

20 -.206 -.525 .000 1.000

r

17 .260 -1.216 1.000 .000

18 -1.092 -,432 ' ..0,00 1:000

19 ' -.994 1.479 1.006 .000

-

-17-

Scoring an item on each dimension would require 2n 'response ca.tegories,where n is the number of dimensions. UnfortunatelY, most test data are notscored polychotomously.

r

An alternative to haviag polychotomous item scoring is to considermore than one item at a time. If twp dichotomously scored items are clusteredtogether, area the cluster is tteated as a single unit; then the,cluster has'22 or 4 response categories - (0,0), (0,1), (1,0), and (1,1). The modelgiven by Equation 10 can then be applied, with the exception that thea-vector now represents a cluster rather than a single item, the scoring -functions now take'on values for 4 response categories instead of 2 and theresponse x is a vector with two elemehts. Further, the number of elementsin the a vector need not be the same as the number of items in the clusqr,but rather should reflect the dimensionality of the cluster. .

The procedure by which this model was investigated is as follows. Fon thetwo-dimensional case, item parameters were selected for 20 items. The ,*

items were paired so that Items 1 and 2 formed Cluster 1, Items 3 and 4Pormed Cluster 2, and so on.until 10 clusters were formed. For each clusterthere were four response categories, which Pere scored as follows:

a) t(x) = (x) = (81 for x equal to both items incorrect;

b) t(x) = t(x) = [(11 for x equal to the first item incorrect, thesecond item correct;

c) t(x) = (x) = N51 for x equal to the first item correct, the seconditem incorrect;

and d) t(x) = (x) = [1] for x equal to both items correct.

For,prey one cluster two responses were generated, one for each dimension,using th6 parameters shown ift Table 6. Table 6 also cohtains.the unrotated.factor loadings for the first two fagors from a principal componentsanalysis of phi coefficients obtained for these data. The first foureigenvalues were 3.61, 3.06, 1.33, and 1.21.

As can be seen, for the factor analysis the simulation data weretreated as 20 items, rather than as 10 clusters. The eigenvalues listedabove indicate that there were two roughly equal components,in the data.Table 6 shows that the first component was defined bythe items that wereplaced first in the clusterrand the second component was delined by theitems that were in the second position in the 'ciluster. Consistent with thescoring functions, there were two equal independent factors.

In order to demonstrate that the factors need not be independent, thesame item parameters were used to generate data using the following scoringfunctions: g,

-18--

TABLE 6

Unrotated Factor Loadings,on First Two Principal Componentsfor the Two-Dimensional Item Cluster Model

/O.

/Item

ItemParameters

t

Indapendent Model Dependen't Model

Factor I Factor IIA

Factor.I Factor II

V

1 .893 .56 .00 .40 .21 ,

Is2 -.850 .02 .65 .33 -.36

3 1.892 .66 -.02 .38 '.24

4 .690 .01 .66 ) .33 -.33

.430 .64 .00 .41 .20

6 3.200 .02 .19 .25 -.24 ..---

7 2.016,.

.36 -.04 .36 .16

-3.310 ..07 .22 .10 -.35

9 .594 .61 -.05 .40 .33

10 . ' .470 .00 .69 ..26 -..39

11 -.913 .66 .01 .47, .30

12 , 1.220 .06 .55 .32 -.38

13 -1.437 .58 -.01 .37 .20,-

14 -1.260 , .00 .58 .32 -.43

15 ' .467 .65 -.07 .45 .28

16 .880 .04 .62 .28 -.36

17 -1.048 .66 -.04 .44 .28

t.....

181.,

-.970 -.02 .64_

.28 -:37

19 -1.760 .56 .07 .43 .10

20 -2.140 .01 .42 .20 -.31

/,

-19-

a) 1(x) = (x) = [8] for x equal to both items incorrect;

.1b) 1(x) = (x) = [

.9) for x equal to the first item incorrect,

the second item correct;

.9-'c) 1(x) = ()(') = [

.1I for x equal to the first item correct,_ _

the secona item incorrect.f\

and d) 1(x) = (x) = [1] for x equal to both items correct.

The principal components analysis of phi coefficients for the datagenerated according to this model yielded.six factors with eigenvaluesgreater than one [2_46, 1.83, 1409, 1.08, 1.01, 1,00). Table 6 shows theunrotated factor loadings. As can'be seen, there are still two factorspresent in the data. However, the factors are no longer defined only bythe items in the corresponding position in the cluster. The first .componentis a general factor, while the second component indicates the position ofthe item in the cluster. Clearly these two sets of items are not independentin this case.

Discussion

"The use of simulation data to study the characteristics of a modelbefore selecting it for application is perhaps atypical of research onlatent trait models. Usually a model is aoropted, estimation procedures arederived, and the model is applied without ever going through the processthis study has employed. 'In this study this approach has been taken fortwo main eeasons. First, it was felt that when dealing with multidimensionallatent trait models much of the acquired wisdom concerning latent traitmodels might no longer apply. It was felt that considerable iresearch wasnecessary in order to gain an understanding of how these models work andwhat the model parameters represent before they could be applied. Thisbelief has been borne out several times in this study by findings indicatingthat the models were not behaving in fhe anticipated manner.

1

A second reason for taking this approach wat that it seemed impracticalto attempt to develop estimation procedures for some of these models.Specifically, the general model set out by Rasch has'a very large number ofparameters. It seemed impractical to try to estimate all of them, and itwas hoped that research on the model could help simplify the estimationprocess by eliminating some terms from the model and by discovering restrictions'on the range of values for the parameters. With these considerations inmind, the results of this study will now be discussed.

Vector Model 4

fhe simplest formulation of the general model that was investigatedwas the vector model. This model is simply the unidimensional Rasch model,but with vectors for-parameters instead of scalars. This mookel was foundto be totally inadequate for.modelling multidimensional data. When datawere generated according to this model, the resulting data were unidimen-sional, with item characteri'stics determined by the inner'product of theitem parameter vectors and scoring functions. From this it follows that

. this model would fit multidimensional data no better than a unidimensionaimOdel having pararlieters' equal to the.inner -products from the vector model.

2,1

--20-

Product Term Model

Because of its slight similarity to Birnbaum's two-parameter logisticmodel, it was felt that the product teim model would be better able tomddel:multidimensional data. It was anticipated that the item parameters,in the product term would behave as discrimination indices, and that is howthey did behave._ Unfortunately, without the vector terms in the model

, there were no terms playing the eole of difficulty parameters. The data .

generated for this model had items of constant.difficulty. From this itwas concluded that this'model would be ,useful only for modelling items ofconstant difficulty, and when items have varying difficulties this model isinappropriate.

Vector and Product Term Model

B sed on the findings for the vector model and the product term model,it was hypothesized that a combination of the two models woulebe necessaryto mad t items that were both multidimensional and of varying di;fficulty.Analys s of the vector and product term model indicated that it would modelmultiqmensional data, and that it would model items of varying difficulty. .

licowever, it was also found that, as long as the item parameter vectorelements appeared both in the vector terms and in the pioduct term,°theitem difficulties.and disciminations would be highly correlated. Sincethis i rarely the case in real test data, it was concluded that this modelwould be useful only in a very limited number of circumstances.

Reduced Vector and Product Term Model

In order to overcome the deficiencies of the vector and product termmodel, it was clear that a given item parameter vector element shouldappear only in the vector term or the product term, but not both. It wasanticipated that similar problems might exist if the person parametervector elements occurred in both the vector term and the product term, sothe same restriction was placed on" the person parameters as was placed onthe item parameters.

The resulting model appears to be quite successful at modelling realisticmultidimensional data. It is capable of modelling correlated as well asindependent dimensions, and the item parameters are-readily interpretable.The only real problem there seems to be with this model is with the estimationof the parameters. Although there are fewer parameters to estimate than isthe case with the general model, there are sfu1l a lair number-to estimate.Moreover, there are no observable sufficient statistics for the parametersin the product lerm. These problems do not make estimation of the modelparameters impossible, and probably not even impractical. However, they domake estimation mckre diffichlt.

Item Cluster Model,

. The item cluster model was proposed as an alternative to the vectormodel. This model 'does not involve a product term, but it still cansuccessfully model multidimensional data. However, it does involve clustering

-21.7

items, which gives rise to a number of new problems. For instance, as yetit is unclear what the effect is of forming different combinations of.jtems, or whether all items should be clustered with the same item.Preliminary investigations seem to indicate that the optimal clusteringprocedure is to cluster all items on a subtest wip one item taken from a,different subtest. Another alternative, which has not been explored, is toapply the model only in situations where items are already clustered, suchas is the case with passage units. Clearly more research is needed on thistype of application of the item cluster model.

Summary and Conclusions

The purpose of this study was to investigate the application of thegeneral.Rasch model to multidimensional data, Several formulations of themodel, varying in complexity, were investigated to determine whether theycould successfully model realistic multidimensional data. Also investigatedwas'whether the parameters of the models 'could be readily interpreted. The,models investigated included: a) the vector model; b) the product term

del; c) the vector and product term model; d) the reduced vector androduct term model; and, e) the item cluster model.

f.

Of the models investigated, all but the reduced vector and productm model and the'item cluster model were rejected as being incapable of

'mo lling realistic multidimensional data. The item cluster model appearsto be a useful model, but its applications may be limited in scope. Of themodels studied, the reduced vector and product term model was found to bethe most capable of modelling realistic multidimensional data. Arthoughthe estimation of the parameters of the reduced vector and product termmodel,tmay'lloe more difficult than it would be for other models, this modelappeaN to be the model that is most worth pursuing.

,

-22-

REFERENCES

Birnbaum, A. Some latent trait models and their use in inferring anexaminee's ability. In F.M. Lord and M.R. Novick, Statisticaltheories of mental test scores. Reading, MA: Addison-Wesley, 1968.

Bock, R.D., and Aitkin, M. Marginal maximum likelihood estimation of itemparamaters: An application of an EM,algorithm. Psychometrika,1981, 46, 443-459.

Boh, R.D., an&Lieberman, M. Fitting a response model for n dichotomoslyscored items. Psychometrika, 1970, 35, 179-197.

Hambleton, R.K., Swaminathan, H., Cook, L.L., Eignor, D.R., andGifford, J.A. Developments in latent trait theory, models, technicalissues, and applications. Review of Educational Research, 1978,48, 467,-510.

Marco, .L. Item characteristic curve solutions to three intractabletesting problems. Journal of Educational Measurement, 1977, A.4,139-160.

McKinley, R.V., and Reckase, M.D. A successful.application of latenttrait theory to tailored achievement- testing (Research Report80-1). Columbia: U 'versity of Missouri, Department of EducationalPsychology, February

Mulaik, S.A. A mathematical investigation of' some multidimensional Raschmodels for psychological tests. Paper presented at the annual-meetingof the Psychometric Society, Princeton, NJ, March, 1972.

Rasch, G. On general laws and the meaning of measurement in psychology.Proceedings of the Fourth Berkeley Symposium on MathematicalStatistics and Probability, Berkeley: University of California Press,1961, 4, 321-334.

Rasch, G. Probabilistic models for some_intelligence And attainment tests.Copenhagen: Danish Institute for Educational Research, 1960.

Reckase, M.D. Development and application of a multivariate logisticlatent trait model (Doctoral dissertation, Syracuse University,1972). Dissertation Abstracts International, 197, 33. (University

Microfilms No. 73-7762)

Rentz, R.R., and Bash* W.L. TheiNational Reference Scale for Reading:An application of the Rasch model. ,Journal of Educational Measurement,1977, 14(2), 161-180.

4

Samejima, F. Normal ogive model for the Continuous response level inthe multidimensional latent 'space. Psychometrika, 1974, 39.,111-121.

0

-23-

Sympson, J.B. A model for testing with multidimensional items. InD.J. Weiss (Ed.), Proceedings of the 1977 Computerized. Adaptive

e'Testing Conference. Minneapolis: University of Minnesota, 1978.

Whitely, S.E. 'Measuring aptieude processes with multicomponent latenttrait models (Technical Report No. NIE-80-5). LawrenCe: Universityf Kansas, Department of Psychology, July 1980.

Woodcock, R.W. Woodcock Reading Mastery Tests. Circle Pines, Minnesota:'American Guidance Service, 1974. a'

4

Navy

Dr. Ed AikenNavy Personnel R&D Center'

San Diego, CA 92152

Meryl S. BakerNPRDC

Code P=09.44ri.'biego, CA 92152

Dr. Jack R. Borsting

Provost & Academic Dean-U.S. Naval Postgraduate SchoolMonterey, CA 93940

1 Dr. Robert BreauxCode N-711.

NAVTRAEQUIPCEN

Orlando; FL ,32813

. 1 Chief of Naval Education and TrainingLiason Office

4ir Force Human Resource LaboratoryFlying Training DivisionWILLIAMS AFB, AZ 85224

1 CDR Mike CurranOffitce of Naval Research800.N. Quincy St.

Code 270Arlington, VA 22217

Navy

1 Dr. Jim HollanCode 304Navy Personnel R & D Center

,San Diego, CA 921S?

1 Dr.,,,..erman J. Kerr

G.f.:lef of pval Technical TrainingNaval Air Statlbh Memphis (75)Millington, TN 3805N

.1 Dr. William L. Maloy

Prilbcipal Civilian AdOsor forEducation and TraiRing

Naval Training Command, Code 00A

Pensacola, FL 32503

CAPT Richard L. Martin, USNProspective.Commanding QfficerUSS Carl Vinson (CVN-70)Newport News Shipbuilding and Drydock CoNewport News, VA 23607

1 Dr. James McBride ,

Navy Personnel PAD CenterSan Diego, C4 9215

1 Dr William MontagueNavY Personnel R&D CenterSan Diego, CA 92152

1 Mr. William Nordbrock1 6R. PAT FEDERICO Instructional Program Development

NAVY PERSONNEL R&D CENTER Bldg. 90SAN DIEGO, -CA 92152 NET-PDCD

Great Lakes Naval Training Center, .1 Mr. Paul Foley 0 IL 60088

Navy Personnel R&D CenterSan Diego, CA 92152 1 Ted M. I. Yellen

Technical Information Office, Code 2011 Dr. JOhn Ford NAVY PERSONNEL R&D CENTER

.Navy Personnel R&D Center SAN DIEGO, CA 92152San Diego, CA 92152

1 Library, Code P201L1 Dr.'Patrick R. Harrison Navy Personnel R&D Center

Psychology Course Director San Diego, CA 92152LEADERSHIP & LAW DEPT. (7b)DIV. OF PROFESSIONAL DEVELOPMMENT 1 Technical DirectorU.S. NAVAL ACADEMY Navy Personnel R&D CenterANNAPOLIS, MD 21402 San Diego, CA 92152

Navy

6 Commanding MinerPeserTch Laboratory

Code 2627

WasninIton, DC 209".".)

1

1

PsychologistONR Branch OfficeBldg 11u, Section D666 Summer StreetBoston, MA 02210

Office of Naval ResearchCode 437

800 N. Quincy,SStreet

Arlington, VA22217

Office of Naval ResearchCode 441

800 N. QuincY StreetArlington, VA 22217

1

Vf

Navy

Dr. Worth Scanland, Director

Research, Development, Test & FvaluationN-5

Naval E6pcation and Training Co-ImandNAS, Pensacola, FL 32908

1 Dr. Robert G. SmithOffice of Chief of Naval OperationsOP-987H

Washington, DC 2050a

1 Dr. Alfred F. Smode.TrainingALnalysis & Fvaluation Group

(Tilt)

Dei)t. of the Navy

Orlando; FL F2813

1 Dr. 'Richard SorensenNavy Personnel R&D CenterSan Diego, CA 92152

5 Personnel & Training Research Programs 1 W. Gary Thomson(Code 458)

Nav'al Cceah Systems,CenterOffice of Naval Research / Code 7132Arlington, VA 22217 / San Diego, CA 92152

1 FtychologistONR Branch Office

1030 East Green Street'Pasadena, CA 91101

1 Roger WeissingerBaylonDepartment of Administrative SciencesNaval Postgraduate schoolMonterey, CA 93940

1 Office of the Chief of Naval Operations 1

Research Development & Studies Branch(0P-115)

Washington, DC 20350

1 LT Frank C.'Petho, MSC, USN (Ph.D7Selection and Training Research Division 1Human Performance Sci'ences Dept.Naval Aerospace Medical Research taboratPensacola, FL 325O 8

1 Dr. Bernard Rimland (03B)

'Navy Personnel R&D CenterSan Diego, CA 92152

1

Dr. Ronard WeitzmanCode 54 WZ '1

Department of Administrative SciencesU. S. Naval Postgraduate SchoolMonterey, CA 93940

Dr. Robert WisherCode 309

Navy Personnel R&D CenterSan Diego, CA 92152

Mr 'John H. Wolfe

Code P310

U. S. Navy Personn'el Research andDevelopment Center

San Diego, CA 92152.

Army

1 , Technical Director

U. S. Army Research Institutie for theBehavioral and Social 8cieves

. 5001 Eisenhower AvenueAlexandria, VA 2233

1 Mr. James RakerSystems Manning Technical AreaArmy Research Institute5001 Eisenhower Av.Alexandria, VA 22333

.1 Dr. Beatrice J. Farr

U. S. Army Research Institute

5001 Eisenhower AvenueAlexandria, VA 22333

1 Dr. Myron FischlU.S. Army Research,Institute for the

Social and Behavioral Sciences5001 Eisenhower AvenueAlexandria, VA 22333

1 Dr. MicbaelvKaplanU.S. ARMY RESEARCH INSTITUTE5001'EISENHOWER AVENUE

ALEXANDRIA, VA 22333

1 Dr. Milton S. KatzTraining Technical AreaU.S. Army Research Institute5001 Eisenhower AvenueAlexandria, VA 22333

1 Dr. Harold F. 0!Neil, Jr.Attn: PERIOKArmy Research Institute5001 Eisenhower AvenueAllexandria, ,vtii 22333

1 DR. JAMES L. RANEYU.S. ARMY RESEARCH INSTITUTE5001 EISENHOWER AVENUEALEXANDRIA, VA 22333

1 Hr. Robert Ross

U.S. Army Research Institute for theSocial and Behavioral Sciences

5001 Eisenhower AvenueAlexandria,, VA 22371:

Army

1 Dr. Robert SasmorU. S. Army Research Institute for the

Eehavioral and Social Sciences5001 Eisenhower AvenueAlexandria, VA 2232

1 Commandant

US ArMy Institute of Administration' Attn: Dr. Sherrill

FT Benjamin Harrison, IN 4626

1 Dr. Joseph WardU.S. Army Research'Institute5001 Eisenhower AvenueAlexandria, VA 22333

Air Force

Air Force Human Resources Lab

'AFHRL/MPD

Brooks AFB, TX 78135

U.S. Air Force Office of Sc.ientificResearch

Life Sciences Directorate, NLBalling Air Force BaseWashington, DC 20332

1 Dr. Earl A. AlluisiHQ, AFHRL (AFSC)Brooks AFB, TX 78235

1 Dr. Alfred R. Fregly

AFOSR/NL, Bldg. 4101Bolling AFB\Washington, DC 20332

1 Dr. Genevieve HaddadProgram Manager ,

Life Sciences DirectorateAFOSRBolling AFB, DC M332

1 David R. HunterAFHRL/HOAMBrooks AFB, TX 78235 .

3

Air ForLe.

1 Research and Measurment DivisionResearch Branch, A5MPC/MPCYPRRandolph AFS: TX 78148

1 Dr. Malcolm ReeAFHRL/MPBrooks AFB, TX 78235

2 3700 TCHTW/TTGH Stop 32Sheppard AFB, TX 76311

1 Dr. Joe Ward, Jr.AFHRL/MPMDBrooks AFB, TX 78235

Marines

1 H. William Greenup

Education Advisor (E031)Education Center, MCDECguantico, VA 22134

1 ,Director, Office Of Manpower UtilizationHO, Marine Corps (MPU)ISCB, Bldg. 2009

Quantico, VA 22134. ,

1 Headquarters, U. S. Marine CorPsCode MPI-20Washington, DC 20380

1 Special Assistant for MarineCorps Matters

Code 100M

'Office of Naval Research800 N. Quincy St.Arlington, .VA 22217

1 ,Major 'Michael L. Patrow, USMC.-lesdquarters, Marine Corps(Code MPI-20)Washington, DC 20380

1 DR. A.L. SLAFKOSKY

SCIENTIFIC ADVISOR (CODE RD-1)HQ', U.S. MARINE CORPS

WASHINGTON, DC 20380

CoastGUard

Chief, Psychological Reserctr BranchU. S. Coast Guard (G-P-1/2/TP42)Wshington, DC 2C593

Mr. Thomas A. WarmU. S. Coast Guard InstituteP. C. Substation 18Oklahoma City, OK 73169

Other DoD

12 Defense Technical Information CenterCameron Station, Bldg 5Alexandria, VA 22"14Attn: TC

1 Dr. William GrahaMTesting DirectorateMEPCOM/MEPCT-PFt. Sheridan, IL 60037

1 Director, Research and Tata'OASD(MRA&L)3B919, The Pentagon

Washiqgtqp, DC 20301

1 Military Assistant for Training and'Personnel Technology

Office of the Under Secretary of Defensefor Research & Engineering

Room 3D129, The PentagonWashington, DC 20301

1 Dr. Wayne Sellman

Office of the Assistant Secretaryof Defense (MRA & L)28269 The PentagonWashington, DC 20301

1 DARPA

1400 Wilson Blvd.Arlington, VA 22209

Civil Govt

1 Dr. Paul G.' Chapin

Lingui,Atics ProgramNational Science FoundationWashington, DC: 20550

1 Dr. Susan ChipmanLearning and DevelopmentNational Institute of Education1200 19th Street gWWashington, DC 20208

1 Dr. John MaysNational Institute of Education1200 19th Street NWWashington, DC 20208

1 Dr. Arthur Melmed

National Intitute of Education1200 19th Street NWWashington, DC 20203

1 Dr. Andrew R. 'Molnar.

Scienc9 Education Dev.'and Research

National Science FoundationWashington, DC 20550

1 Dr.. H. Wallace StnaikoProgram DirectorManpower Research and Advisory ServicesSmithsonian Institution801 North Pitt StreetAlexandria, VA 22314

1 Dr. Vern W. UrryPersonnel R&D CenterOffice of Personnel Management1900 E Street NWWashington, DC 20415

1 Dr. Frank WithrowU. S. Office of Education400 Maryland Ave. SWWashington, DC 20202

1 Dr. Joseph L. Young, DirectorMemory & Cognitive ProcessesNational Science Foundation

Washington, DC 20550

Non Govt

1 Dr. James AlginaUniversity of FloridaGainesville, FL :2611

1 Dr. Erling B. AndersenDepartment of StatisticsStudiestraede 61455 CopenhagenDENMARK

1 Anderson, Thomas H., Ph.D.Center for thi Study of Reading174 Children's Research Center.51 Gerty DriveChampiagn, IL 61820

1 Dr. John AnnettDepartment of PsychologyUniversity of Warwick .

Coventry CV4 7ALENGLAND

1 1 psychological research unitDept.. of Defense (Army'Office)

Campbell Park OfficesCanberra ACT'2600, Australia

1 Dr. Alan BaddeleyMedical Research Council

Applied Psychology Unit15 Chaucer Road

Cambridge CB2 2EFENGLAND

1 Ms. Carole A. BagleyMinnesota Educational Computing

Consortium2354 Hidden Valley LaneStillwater, MN 55082

Dr. Isaac BejarEducational Testing ServicePrinceton, NJ 08450

1 Capt. J. Jean BelangerTraining Development DivisionCanadian Forces Training SystemCFTSHQ, CFB TrentonAstra, Ontario KOK 1BC

3,3

Non Gov t

1 Dr. Venucha BirenbaumSchool of EducationTel Aviv UniversityTel Pviv, Ramat Aviv 69978Israel

Dr. Werner Pirke

DezWPs imStreitkraefteamtPostfach 20 50 03D-5300 Bonn 2WEST GERMANY

1 Dr. R.. Darrel Bock

Department of EducationUniversity of ChicagoChiCago, IL 60637

1 Liaison ScientistsOffice of Naval Research,Brapch Office , Lond,on

BOX 39 FPO New York 09510

Dr. Robert Brennan

American College Testing ProgramsP. O. Box 168Iowa City, IA 52240

1 bR. JOHN F. BROCKHoneywell. Systems & Research Center(MN 17-2318)

2600 Ridgeway ParkwayMinneapolis, MN 55413

1 DR. d. VICTOR BUNDERSONWICAT INC.

UNIVERSITY PLAZA, SUITE 101160 SO. STATE ST.OREM, UT 84057

1 Dr. John B. Carroll

Psychometric LabUniv. of No. Carolina

Davie Hall 013A 'It

Chapel Hill, NC 275144

Non Govt

1 Dr. Norman CliffDept. of PsychologyUniv. of So. CaliforniaUniversity Park'Los Angeles, CA 9007

1 Dr. William E. Coffman..

Director, Iowa Testing Programs334 Lindquist Center

Universitylof IowaIowa CityfIA 52242

1 Dr: Mereclith P. Crawford

hmerican Psychologic'al Association1200 17th Street, N.W.Washington, DC 20036

1 Dr. 'Hans CrombagEducation Research CenterUniversity of LeydenBoerhaavelaan.22334 EN LeydenThe NETHERLANDS

1 Dr.,Fritz DrasgowYale School of Organization ana Yanageme'Yale UniversityBox 1A

New Haven, CT 06520

1 Mike Durmeyer

InstruCtional Program DevpmentBuilding 90NET-PDCDGreat Lakes NTC, IL 60088

1 ERIC Facility-Acquisitions4833 Rugby AvenueBethesda, MD 20014

1 Dr. Benjamin A. Fairbank, Jr.McFann-Gray & Associates, Tnc.5825 CallaghanSuite 225.

San Antonio, Texas 78228

Non Govt

1 Dr. 4eonard Feldt 1

Lindquist Center for MeasurmentUniversity of Iowa.Iowa City, IA 52242

Non Govt

1 Or. Delwyn HarnischUniversity of Illinois ,242b Education-Urbana, IL 61801'

1 Dr. Richard L. Ferguson 1 Dr. Chester HarrisThe American College Testing Program Schoolpf EducationP.O.'Box.168 University of CaliforniaIowa City, IA 522u0 Santa Barbara, OA 93106

1 Dr. Victor FieldsDept. of PsychologyMontgomery CollegeRockville, MD 20850

1 Dr. Dustin H. HeustonWicat, Inc.Box 986Orem, UT 84057

1 Univ. Prof. Dr. Gerhard Fischer' 1 Dr !t Lloyd Humphreys

- Liebiggasse 5/3 Department o PsychologyA 1010 Vienna University IllinoisAUSTRIA Champaign, -61820

1 Professor Donald Fitzgerald 1 Dr. S teven Hunka

University,of New England Department of EducationArmidale, New South Wales 2351 University of AlbertaAUSTRALIA Edmonton, Alberta

CANADA1 DR. ROBERT GLASER

LRDC

UNIVERSITY OF PITTSBURGH3939 O'HARA STREETPITTSBURGH, PA -15213-**

1 Dr. Earl HuntDept. of PsychologyUniversity of WashingtonSeattle, WA .98105

1 Dr. Daniel Gopher 1 Dr. Jack HunterIndustrial & Management Engineering . 2122 Coolidge St.TechnionIsrael Institute of Technology Lansing, MI 48906HaifaISRAEL ' 1 Dr. Huynh Huynh

College ofEducation1 Dr. Bert Green University of South Carolina

Johms'Hopkins University Columbia, SC 29203Department of PsychologyCharles V 34th Street 1 Professor John A. KeatsBaltimore, MD 21218 University of Newcastle '

AUSTRALIA 23081 Dr. Ron-, aMbieton

School dPEducation 1 Mr. Jeff KeletyUniversity of Massechusett's Department of Instructional TechnologyAmherst,-MA 01002 University of Southern California

Los Angeles, CA 92007

Non Govt

1 Dr., Stephen Kosslyn

Harvard UniversityDepartment of Psychology33 Kirkland StreetCambridge, MA 0217F

Dr. Marcy LanmanDepartment of Psychology, NI 25University of WashingtonSeattle, WA 98195

1 Dr. Alan LesgoldLearning R&D Center

University of PittsburghPittsburgh, PA 15260

1 Dr, Michael Levine

Department of Educational PsychologY'210 Educatipn Bldg.Universitypof IllinoisChampaign, IL 618,01

1 zDr. Charles LewisFaculteit Sociale WetenschappenRijksuniversiteit GrOningenDude Boteringestraat 239712GC GroningenNetherlands

1 Dr. Robert LinnCollege of EducationUniversity of IllinbisUrbana, IL 61801

Dr. Frederick M. LordEducational Testing ServicePrimceton, NJ 08540

1 Dr. James LumsdenDepartment of Psychology'University of Western AustraliaNedlands W.A. 6009AUSTRALIA

1 Dr, Gary Marco

Educational TeSting ServicePrinceton, NJ-08450

Non Govt

1 Dr. Scott MaxwellDepartment of PsychologyUniversity of HoustonHouston, TX 77004,

1 Dr. Samuel T. MayoLoyola University of Chicago820 North Michigan AvenueChicago, IL 60611

1 Professor-Jason MillmanDepartment of EducationStone Hall

Cornell UniversityIthaca, NY 14853

1 Dr. Allen nunro

Behavioral Technology Laboratories1845 Elena Ave., Fourth FloorRedondo Beach, CA 90277

1 Dr. Melvtin R. Novick

356 Lindquist Center for MeasurmentUniversity of IowaIowa City, IA 52242

1 Dr. Jesse OrlanskyInstitute for Defense Analyses400 Army Navy Drive

Arlington, VA 22202

1 Wayne M. PatienceAmerican Council on EducationGED Testing Service, Suite 20One Dupont Cirle, NWWashington, DC 20036

1 Dr. James A. PaulsonPortland State UniversityP.O. Box 751Portland, OR 97207

1 MR. LUIGI PETRULLO2431 N. EDGEWOOD STREETARLINGTON, VA 22207

tr.

Non Govt.

1 Dr. Steven E. Poltrock_Department of PsychologyUniversity of DenverDenver,C0 30208

DR. DIANE M. RAMSEY-KLEER-K RESEARCH & SYSTEM DESIGN3947 RIDGEMONT DRIVEMALIBU, CA 90265

1 MINRAT,M. L. RAUCHP II 4

BUNDESMINISTERIUM D5R VERTEIDIGUNGPOSTFACH 1328D-53 BONN 1, GERMANY

1 Dr. Mark D. Reckase

Educational Psychology Dept.University of Missouri-ColUmbia4 Hill HallCdlumbia, MO 65211

1 Dr. Lauren Resnick'LRDC

University of Pittsburgh

3939 O'Hara Str'eet

Pittsburgh, PA 15213

1 Mary RileyLRDC

University of Pittsburgh3939 O'Hara Street ,

Pittsburgh, PA 1521?

1 Dr. Leonard L. Rosenbaum, ChairmanDepartment of PsychologyMontgomery CollegeRockville, MD 20850

1 Dr. Ernst Z. RothkopfBell Laboratories600 Mountain AvenueMurray Hill, NJ 07974

1 Dr. Lawrence Rudner403 Elm AvenueTakoma Park, MD 20012

Non Govt

1 Dr. J. Jiyan

Department of Education%

University of South CirolinaColumbia, SC 2920q1

1' PROF. FUMIKO SAMEJIMADEPT. OF PSYCHOLOGYUNIVERSTTY OF TENNESSEEKNOXVfLLE, TN 37916

1 Frank L. SchmidtDepartment of PsychologyBldg. GG '

George Washington UniversityWashington, DC 20052

1. Dr. Kazuo ShigemasuUniversity of TohbkuDepartment of Educational PsychologyKawauchi, Sendai 980JAPAN

1 Dr. Edwin Shirkey

Department of,PsychologyUniversity of -Central FloridaOrlando, FL 32816

1 Dr. Richard SnowSchool of EducationStanford UniversityStanford, CA 9430'

1 Dr. Kathryn T. 'SpoehrPscyhology Department8roWn UniversityProvidence, RI 02912

a

1 DR. PATRICK SUPPESTNSTITUTE FOR MATHEMATICAL STUDIES IN

. THE SOCIAL SCIENCESSTANFORD UNIVERSITY'STANFORD, CA 94305

1 Dr. Hariharan SwaminathanLabratory of Psychometrjc and

Evaluation ResearchSchool of EducationUniversity of VassachusettsAmherst, MA 01003

Non Govt

1 Dr. Brad Sympson

Psychometric Research GroupEducationql Testing ServicePrinceton, NJ 0541

1 Dr. Kikumi Tatsuoka

Computer Based Education ResearchLaboratory

252 Engineering Research LaboratoryUniversity of IllinoisUrbana, IL 61801

1 Dr, David,ThissenDepartment of Psychology _

University of KansasLawrence, KS 66044

1 Dr. Douglas TowneUniv. of 'So, California

Behavioral Technology Labs1845 S. Elma Ave.Redondo Beach, CA 0277.

1 Dr. Robert TsutakaWaDeparthent of StatisticsUniversity o40,MissouriColumbia, MO 65201

4

Dr. David ValeAssessment Systems Corporation2395 University'Avenue ,

Suite .306

St. Paul, )IN

1 Dr. Howard WainerDivision of Psychological StudiesEducational Testing'ServicePrinceton,.NJ o854p

DR. SUSAN E. WHITELYPSYCHOLOGY DEPARTMENTUNIVERSITY OF KANSASLAWRENCE, KANSAS 66044

1 Wolfgang WildgrubeStreitkraefteamtbox 20 50 03D-5300 Bonn 2

owe