Model, Objective Measurement, Equating, and Robustness · 437 The Rasch Model, Objective Measurement, Equating, and Robustness Jeffrey A. Slinde Science Research Associates, Inc

437

The Rasch Model, Objective Measurement,Equating, and RobustnessJeffrey A. SlindeScience Research Associates, Inc.

Robert L. Linn

University of Illinois, Urbana-Champaign

This study investigated the adequacy of the Raschmodel in providing objective measurement whenequating existing standardized reading achievementtests with groups of examinees not widely separatedin ability. To provide the context for the assessmentof objectivity with the Rasch model, informationrelevant to several assumptions of the model wasprovided. An anchor test procedure was used toequate the various pairs of existing achievementtests. Despite the considerable lack of fit of thedata to the model found for all tests used, theRasch difficulty estimates were reasonably invariantfor replications with random samples as well assamples that differed in ability by one grade level.Furthermore, with the exception of the data for onetest pair and one grade level, the Rasch modelusing the anchor test procedure provided a reason-ably satisfactory means of equating three test pairson the log ability scale for the examinees at twograde levels.

Many authors (e.g., see the special Journal ofEducational Measurement summer, 1977, issueon applications of latent trait models) have indi-cated that latent trait models offer a potentiallygood means of solving many difficult psycho-metric problems, such as vertical equating,tailored or adaptive testing, and criterion-refer-enced testing. The reason for this optimism is

that objective measurement, which consists ofperson-free test calibration and item-free personmeasurement, is a consequence of such models.Person-free test calibration refers to the estima-tion of item parameters that are invariant for allgroups of examinees. Item-free person measure-ment implies that except for errors of measure-ment, the same ability estimates would be ob-tained for an individual no matter which subsetof calibrated items was used in the measure-ment. It is easy to see that if these two propertiescan reasonably be achieved, then the difficultiesassociated with many testing problems woulddisappear.One of the simplest and most tractable latent

trait models is the one-parameter logistic latenttrait, or Rasch, model (Rasch, 1960). Only oneitem parameter-difficulty or location with re-spect to the item characteristic curve (ICC)-isspecified by the Rasch model. The Rasch modeldoes not include parameters for the lower

asymptote or slope (discrimination) of the ICC.Like most of the latent trait models, the Raschmodel also assumes that the items can be ac-counted for by only one latent trait or psycho-logical function. In addition, the Rasch model,as well as other latent trait models, requires thetests to be unspeeded; otherwise, the probabilityof correctly answering the last items depends notonly on the probability of success due to at-

APPLIED PSYCHOLOGICAL MEASUREMENTVoL 3, No. 4 Fall 1979 pp. 43 7- 452@ Copyright 1979 West Publishing Co.

Downloaded from the Digital Conservancy at the University of Minnesota, http://purl.umn.edu/93227. May be reproduced with no cost by students and faculty for academic use. Non-academic reproduction

requires payment of royalties through the Copyright Clearance Center, http://www.copyright.com/

438

tempting the item but also on the probability ofattempting the item.Several studies (Anderson, Kearney, &

Everett, 1968; Rentz & Bashaw, 1975; Tinsley &

Dawis, 1975) have investigated the person-freetest calibration property of the Rasch model,reaching the conclusion that the difficulty esti-mates were relatively independent of the abilitylevel of the calibration sample. The results ofother studies (Tinsley & Dawis, 1977; Whitely &

Dawis, 1974; Wright, 1968), which investigatedthe item-free person measurement property ofthe Rasch model, indicated that the Rasch abil-

ity estimates were reasonably independent of thedifficulty of the items.The extent of fit of data to the Rasch model

with existing tests has received only limited at-tention in connection with the results concerningobjective measurement, however. Also, cross-

validations of Rasch results were not reported inthe studies dealing with item-free person mea-surement. That is, in all three studies the sampleused to estimate the parameters was the same asthat used to test the item-free person measure-ment claim of the Rasch model. In addition, theresults of the Slinde and Linn studies (1978,1979) suggested that item-free person measure-ment may not be realized with sets of multiple-choice items that differ markedly in difficulty.In Slinde and Linn’s studies, objective mea-

surement was investigated within the context ofthe vertical equating problem by using cross-validations of Rasch results for groups of quitedifferent ability and for tests of quite differentdifficulty. Since extreme groups and tests wereused, the test of the adequacy of the Raschmodel for the vertical equating problem wasacknowledged to be a severe one. Slinde andLinn also indicated that better results might beexpected under such circumstances if an anchortest were used. Thus, it seems important andnecessary to investigate further the claim for ob-jective measurement with the Rasch model.The primary purpose of this study, therefore,

was to investigate the adequacy of the Raschmodel in providing objective measurement when

equating existing standardized reading achieve-ment tests with groups of examinees not widelydifferent in ability. More specifically, the assess-ment of objectivity with the Rasch model in-volved two considerations: (1) whether approxi-mately equal item difficulty parameter estimatesare obtained by the Rasch model when different(random and nonrandom) groups of examineesare used for the item calibration and (2)whether, on the average, a group of examinees(different from those used for the estimation) ob-tained the same ability estimates from two teststhat differ in difficulty level. The first considera-tion refers to investigating the person-free itemcalibration property of the Rasch model,whereas the second consideration involves in-

vestigating the item-free person measurementproperty. To provide the context for the assess-ment of objectivity with the Rasch model, infor-mation relevant to several assumptions of themodel was provided. Also, an anchor test proce-dure was used to equate the various pairs ofexisting achievement tests.

’

Procedure

To investigate whether objective measurementis achieved with the Rasch model, item responsedata were taken from the Anchor Test Study(ATS; Bianchini & Loret, 1974). For the ATS astratified random sample of students from thepopulation of all fourth-, fifth-, and sixth-gradestudents in the United States was administeredthe reading comprehension and vocabulary sub-tests of two standardized achievement test bat-teries. A separate group of examinees was ad-ministered each possible combination of pairs ofthe seven publishers’ tests originally included inthe ATS. For purposes of this study only thefourth- and fifth-grade item response data forfour reading comprehension subtests and threevocabulary subtests were selected.The four publisher’s tests included the Cali-

fornia Achievement Tests (CAT; CTB/McGraw-Hill, 1970), Comprehensive Tests of Basic Skills(CTBS; CTB/McGraw-Hill, 1968), SRA



439

Achievement Series (SRA; Science Research As-sociates, Inc., 1971), and Sequential Tests ofEducational Progress (STEP; Educational Test-ing Seivice, 1969). The SRA vocabulary item re-sponse data were not available except for the 12vocabulary-in-context items embedded withinthe reading comprehension subtest. These 12items were included with the regular 48 readingcomprehension items, creating a 60-item SRAreading comprehension subtest, which was usedin the subsequent analyses. The number of itemsand the appropriate grade levels by test, form,and level are given in Table 1.Table 2 depicts the ATS data collection proce-

dure for the three publishers’ vocabulary sub-tests used in this study. In Table 2 each of the sixcells represents an independent sample of ex-aminees who were administered the pair of vo-cabulary subtests corresponding to the row andcolumn headings. Cells above the diagonal referto fifth-grade samples and cells below the diag-onal refer to fourth-grade samples. For example,Cell a represents an independent sample of fifth-grade students who were administered the CATand CTBS vocabulary subtests. Since each vo-cabulary subtest was paired with the other twovocabulary subtests at both grade levels, four in-

dependent data sets (replications) were asso-

ciated with each vocabulary subtest, e.g., two

fifth-grade samples (Cells a and b) and twofourth-grade samples (Cells d and e) were ad-ministered the CAT vocabulary subtest. A simi-lar design was associated with the reading com-prehension subtests except that a fourth pub-lisher’s test, SRA, was also considered. Conse-

quently, there were six independent sets of ex-aminee data, three at each grade, for each of thefour publishers’ reading comprehension sub-tests. Ignoring the order of administration, thesample sizes, ranging from 1,372 to 1,921, foreach pair of publishers’ tests are given in Table3.

Model-Data Fit

Preliminary data were obtained concerningthe differing discrimination values of the items,and the speededness and the unidimensionalityof each of the standardized achievement tests.The preliminary data provided informationabout the limits of applicability of the Raschmodel, even though violations of the underlyingRasch assumptions could not be systematicallyvaried. That is, information about the degree of

Table 1Number of Items and Appropriate Grade Levels

by Test, Level and Form

aThe SRA Vocabulary items were not all available for this study.bIncludes the 48 regular reading comprehension items plus the12 vocabulary-in-context items embedded within the readingcomprehension subtest.



440

Table 2

Representation of the ATS Data Collection Samplesfor the Three Vocabulary Tests Used in this Study

Note. Fifth grade samples are above thediagonal and fourth grade samples arebelow the diagonal.

misfit, depending upon the results, gives someidea about the limits to which the Rasch as-

sumptions can be violated in achieving objectivemeasurement.

Unidimensionality. By inspecting the eigen-values from the factor analysis of interim tetra-choric correlations computed on the combinedfourth- and fifth-grade data, a check on the rea-sonableness of the unidimensionality of eachsubtest was made. The tetrachoric correlations

require normality assumptions, which are notnecessary to the Rasch model. Also, with mul-tiple-choice items the normality assumptioncannot be expected to hold because of the effectsof guessing. Nonetheless, the factoring of thetetrachoric correlation matrices provides some

indication of the plausibility of the unidimen-sionality assumption of the Rasch model. Cer-tainly, if several large factors were indicated inthe factor analysis, the assumption of unidimen-sionality would be highly suspect.The particular factor extraction technique em-

ployed was the principal axis method with esti-mated communalities in the diagonal. The com-munality of each item was estimated by thesquared multiple correlation with every otheritem. In addition to the factor analysis of eachsubtest, correlations between equated tests arealso provided.Nonspeededness. A rough notion as to the

speededness of each subtest is given by the pro-portion of combined fourth- and fifth-grade ex-

Table 3Sample Sizes for the Various Test Pairs

Note. Grade 5 sample sizes are above the diagonaland Grade 4 sample sizes are below thediagonal.



441

aminees failing to respond to each item. Of

course, some speededness is expected for moststandardized achievement tests. Typically, a

nonspeeded test is considered to be one in whichthe proportion omitting the last few items isabout .10 or less. In the long run, however, thedifferential amount of speededness between

tests to be equated is likely to be a more impor-tant factor than the absolute degree of speeded-ness, because an examinee’s test behavior is ap-preciably different for two differentially speededtests than for two equally speeded tests.Equal discriminations. As previously men-

tioned, four independent sets of examinees’ itemresponse data (two at each grade) were availablefor each of the three publishers’ vocabulary sub-tests. For each set of examinee data the propor-tion of very good-fitting items, those with dis-crimination values in the range .80 to 1.20

(Panchapakesan, 1969; Rentz & Bashaw, 1975),was calculated. The average of the four propor-tions for each of the publishers’ tests providedan overall picture of the extent to which the as-sumption of equal discrimination across allitems was met. Proportions of items with dis-crimination values less than .50, .50 to .79, .80 to1.20, 1.21 to 1.50, and greater than 1.50 werealso calculated for the equating samples. Asimilar procedure was followed for each of thefour publishers’ reading comprehension subtestsexcept that six independent sets of examinees’item response data were available.

Person-Free Item Calibration

Evaluating the person-free item calibration

property of the Rasch model involved determin-ing how stable the item parameter estimateswere when estimated across different groups ofexaminees. All Rasch item calibrations were

performed using the BICAL computer program(Wright & Mead, 1977). Six independent cali-brations of the Rasch item parameters (three ateach grade) were obtained for the reading com-prehension subtests of the four publishers’ testbatteries. Only four independent calibrations

were possible for the vocabulary subtests. Inorder to save on computer costs, the items foreach test pair were commonly calibrated (recallthat each examinee was administered two test

batteries), i.e., the two tests were treated as onelong test. This was done separately for the read-ing and vocabulary subtests of each battery. Forboth subtests of each publisher’s test the inter-correlations, standard deviations, and means ofthe independent item parameter estimates werecomputed.

Item-Free Person Measurement

To assess the item-free person measurement

property of the Rasch model, a paradigm depict-ing an equating situation that uses an anchortest must first be described. Generally, the para-digm involved treating the ATS data as thoughthe easiest test, along with an anchor test, wasadministered to only fourth-grade students andas though the hardest test, along with the sameanchor test, was administered only to fifth-gradestudents. Specifically, for the three vocabularysubtests the CTBS was selected as the anchor

test; the CAT, using fifth-grade data, was

equated to the STEP; while STEP used onlyfourth-grade data. For the four reading compre-hension subtests two equating paradigms weresimulated, both involving the CAT as the anchortest. Both the SRA and STEP using Grade 5data were equated to the CTBS, while the CTBSused only Grade 4 data. A schematic representa-tion of equating the easier STEP vocabularysubtest to the more difficult CAT vocabularysubtest is shown in Figure 1.In general, the difference between the diffi-

culty levels (p-values) for the easiest and hardesttests was about .06 at each grade level. Also, interms of p-values, the average difference be-tween fourth and fifth graders was about .09.Thus, the difference between the groups and thetests were not extreme, as in the Slinde and Linnstudies, yet they were not all that small.After calibrating the item parameters for pairs

of tests to be equated, the Rasch log ability esti-



442

Flgure 1

Schematic Representation of Equatingthe Two Vocabulary Subtests

a. Equated thru their common item calibrationusing grade 4 examinees.

b. Equated thru their common item calibrationusing grade 5 examinees.

c. Equated using the 2 sets of anchor test(CTBS) log difficulty estimates.

mates for the two tests were then placed on thesame scale by using the two sets of difficultyparameter estimates for the anchor test. Since, as N

approaches infinity, the scales for the two sets

of difficulty estimates have the same varianceand the correlation between the estimates is 1.0,there is only a difference in the origin of the twoscales. Thus, the difference between average dif-ficulties estimates the additive constant, placingnot only the difficulty estimates but also theability estimates of the two tests to be equatedon the same scale. For example, to equate thetwo vocabulary subtests, the mean of the CTBSdifficulty estimates based on the CAT samplewas subtracted from the mean of the CTBS diffi-

culty estimates based on the STEP sample. Thedifference was then added to all CAT difficultyestimates to place CAT item difficulties on thesame scale as STEP. A similar procedure wasfollowed for the two pairs of reading comprehen-sion subtests.

Evaluating the item-free person measurementproperty of the Rasch model involved notingthat the equated easy and difficult tests actuallyhad both been administered to a group of

fourth-grade students and a group of fifth-gradestudents. Therefore, for these two groups of stu-dents, summary statistics for the Rasch log abil-ity estimates were computed for each of the

equated easy and difficult tests. In particular,the means, standard deviations, and correlationsof the log ability estimates for the two equatedtests provided data pertinent to item-free personmeasurement.

Results and Discussion

Model-Data Fit

Unidimensionality. The first 10 eigenvaluesof the matrices of tetrachoric correlations

among the items of each subtest are reported inTable 4. With one exception the percentage ofcommon variance accounted for by the first fac-tor for each of the subtests ranged from 65% to75%. The exception involved the SRA readingcomprehension subtest, which had only 55% ofthe variance accounted for by the first factor.Since the second and successive latent roots wereof similar magnitude and substantially smallerthan the first, one very large dominant factorwas associated with each subtest. Conventionalrules for determining the number of factors indi-cated that there was at least one minor factor as-sociated with all but one of the subtests, how-ever.

Nonspeededness. The proportion of ex-

aminees failing to respond to each item was usedto provide an indication of the speededness ofthe subtests. The proportion of omits for each ofthe subtests was, with rare exception, nonde-creasing with respect to item number, i.e., if theproportion of omits was .06 for Item 7, then theproportion of omits was at least .06 for Item 8.Reported in Table 5 are the distributions of theproportion of omits for each subtest. An ex-amination of the data reported in Table 5 in-dicates that the four reading subtests and



443

0s

4-o

Ncmc=:fO·r- +:S- vma

..c.e =youEE£8kV a

LL4-0 !-

C<U 4-CD<C t/4-) CC CQ) or-U +S- C’tU r-c..a.

« s-0 S.

a~ a~ cr- 4-) <-J~i f0ft$ ·r (·r- U or-

0 !-U%c00 ~:

CX::<..T

-Os.C +-<0 <L

....NNoar+<0 ’t-asc a<U 4-m£Or- )&horbar;

w

CNH

-CI)i

1



444

Table 5Distributions of the Proportion of Omits in

Terms of the Proportion of Items in Each of the Subtests

the three vocabulary subtests were differentiallyspeeded with respect to their content areas. Infact, the CAT vocabulary subtest was extremelyspeeded. The proportion of omits for the last sixitems of this test ranged from .21 to .30. Thus,not only was the assumption of nonspeedednessnot met to a reasonable approximation withmost of the subtests but the degree of speeded-ness was considerably different between the sub-tests of a given content area.Equal discriminations. As described earlier,

four independent sets of examinee item datawere available for each of the vocabulary sub-tests, and six sets of data were available for eachof the reading comprehension subtests. For eachsubtest half of the independent sets of data in-volved Grade 5 examinees, while the other halfinvolved Grade 4 examinees. Across the replica-tions the average proportion of items having dis-crimination values in the range .80 to 1.20 were

.575, .506, and .633 for the CAT, CTBS, andSTEP vocabulary subtests, respectively. For theCAT, CTBS, STEP, and SRA reading compre-hension subtests, the average proportions acrossthe six replications were .409, .444, .550, and.367, respectively. On the average, the vocabu-lary subtests had a larger proportion of verygood fitting items than the reading comprehen-sion subtests, but for none of the subtests wasthe model-data fit exceptionally good in terms ofthe discrimination index. Distributions of the

item discrimination values, in terms of the pro-portion of items in each subtest, are reported inTable 6. These distributions were constructed

using only those samples that were involved inthe equating paradigms. As noted from thetable, several subtests contained a few items forwhich the discrimination value deviated by .51or more from the standard value of 1.00.

Implications. Results pertaining to model-data fit indicated that the assumption of unidi-mensionality for each of the reading subtestswas not met, strictly speaking, although one verylarge major factor was associated with each ofthe subtests. In addition, the assumptions ofnonspeededness and uniform item discrimina-tions were violated to a substantial degree. Withonly four options per multiple-choice question itis probable that guessing was also a factor. Thisis likely to be true for many national stan-

dardized achievement tests, since such tests are

developed along the same general guidelines andthe examinees being tested span a wide range ofability. Consequently, the Rasch model is apt tobe considered suspect when the sole criterion in-volves satisfying the assumptions of the model.Although in terms of satisfying the antecedent

conditions of the model, fit is an important con-sideration, the issue of primary importance is

whether the model achieves what it purports to,even though the underlying assumptions of themodel are not met. In fact, the empirical portion



445

Table 6Distributions of the Item Discrimination Values

in Terms of the Proportion of Items in Each of the Subtests

aEnclosed in parentheses are the paired test and grade level of thecalibration sample.

of the present study was largely concerned withjust this issue: Is the Rasch model robust withrespect to existing standardized reading achieve-ment tests? The consequences or desirable out-comes of the Rasch model were defined in termsof person-free item calibration and item-free

person measurement, and the degree of misfitwas that which is naturally associated with thestandardized reading achievement tests used inthis study.

Person-Free Item Calibration

The intercorrelations, means, and standarddeviations of Rasch difficulty estimates werecomputed for each subtest across the several

replications. Since two grade levels were in-

volved, both random (within grade) and nonran-dom (between grade) replications were con-

sidered. For the vocabulary subtests the sum-mary measures are reported in Table 7 and forthe reading comprehension subtests the indicesare reported in Table 8.

Correlation of item difficulties. As can beseen in both tables, the intercorrelations were

extremely high, with only a few coefficients lessthan .98 and none less than .94, with the STEPreading comprehension subtest providing thelowest intercorrelations (five of the six correla-tions were less than .97). Thus, the ordering ofRasch difficulty estimates for each subtest wasreasonably constant across the several replica-tions.Standard deviation of difficulties. For each

subtest the standard deviations of the difficultyestimates were very similar. Typically, the rangeof these standard deviations across the replica-tions was about .08. With one notable exception,there was no tendency for the standard devia-tions to be higher or lower at one ability level(grade) than another. The exception involved theCAT reading comprehension subtest. On the

average, the CAT standard deviations based onGrade 4 data were about .13 log unit smallerthan those based on Grade 5 data. Despite theone exception, the unit of the scale of measure-



446

Table 7Intercorrelations, Means and Standard Deviations of the

Vocabulary Subtest Item Difficulty Estimates

ment for item difficulty was essentially the sameacross the random and nonrandom replications.Mean item difficulties. Because each subtest

was commonly calibrated with another subtest,it is inappropriate to compare the means of thedifficulty estimates for equality across all repli-cations of a given test. The reason for this is thatin order to overcome the indeterminacy in thescale of difficulty, the mean of all commonlycalibrated items (e.g., CAT and CTBS items)was arbitrarily set equal to zero. Consequently,the mean for any subset of commonly calibrateditems (e.g., just the CAT items) is dependent onwhat other items are included in the calibra-tion. Since the same set of items were commonlycalibrated for certain pairs of nonrandom repli-cations (different grade levels), comparisons ofthese means was appropriate.For example, the means of the CAT difficulty

estimates, which were based on the CTBS-4 andCTBS-5 samples, can be compared; and simi-larly, for the STEP-4 and STEP-5 samples.Therefore, two comparisons of means were

made for each of the three vocabulary subtests

and three comparisons were made for each ofthe four reading comprehension subtests. Mostof the differences between the 18 pairs of meanswere about .04 log unit or less, but the STEPreading comprehension subtest provided the

largest mean differences of .09 and .08. Con-sidering the size of the standard deviations, theorigin of the scale of measurement of the diffi-culty estimates was very similar across the non-random replications.Implications. To a reasonable approxima-

tion, the results concerning person-free itemcalibration indicated that across the randomand nonrandom replications, the ordering ofRasch difficulty estimates was constant and themeans and the standard deviations of the diffi-

culty estimates were very similar.The results indicating that the Rasch difficulty

estimates were reasonably stable would seem es-pecially convincing, since no attempt was madeto delete poor-fitting items, whereas the dataconcerning the assumptions of the model indi-cated considerably less than satisfactory model-data fit. But it must also be emphasized that



447

Table 8Intercorrelations, Means, and Standard Deviations of

the Reading Comprehension Subtest Item Difficvlty Estimates

very large samples (1,372 to 1,921) were used inthis study; and as suggested by the Rentz andBashaw (1975) and Tinsely and Dawis (1975)studies, the effect of using very large samplesmay have been the reduction of any distortion inthe person-free item calibration results, whichwould otherwise have occurred if small samplessizes (100 to 500) had been used. All in all, theresults of this study and of previous studies

(Anderson, Kearney, & Everett, 1968; Rentz &

Bashaw, 1975; Tinsley & Dawis, 1975) stronglysuggest that Rasch difficulty estimates are rela-tively independent of the ability level of the cali-bration sample for groups not widely different inability. The range of ability of the calibrationsample in relation to the difficulty level of theitems, the degree of model-data fit, and the sizeof the calibration sample undoubtedly interactto determine the extent to which invariance is

achieved, however.



448

Item-Free Person Measurement

Within the two content areas (reading compre-hension and vocabulary), three equating para-digms using an anchor test were described.Since each pair of equated tests was actually ad-ministered to a sample of Grade 4 and a sampleof Grade 5 students, the means, standard devia-tions, and correlations of the Rasch log abilityestimates for each of the three equated test pairsprovided data relevant to item-free person mea-surement. The summary measures for the vo-

cabulary subtests are reported in Table 9, andfor the reading comprehension subtests the mea-sures are reported in Tables 10 and 11.Correlation of ability estimates. For all three

equated test pairs, and for both grade levels, thecorrelations between Rasch log ability estimateswere relatively high (ranging from .78 to .85), in-dicating that the examinees were ordered simi-larly on the equated test pairs. Although con-sistently lower, the correlations between Raschlog ability scores differed by less than .02 fromthe corresponding raw score correlations. Thus,little, if any, information was lost due to use ofthe Rasch model to equate the various test pairs.Mean and standard deviation of ability esti-

mates. The remaining results pertaining to

item-free person measurement were generallyfavorable, but not ideal. Perhaps the onlyequated test pair which can be treated as statis-tically equivalent throughout the range of mea-sured ability (Grades 4 and 5) was the CTBS andSRA reading comprehension subtests. Even forthis test pair the difference between the Grade 5log ability means was -.076, which was morethan four standard errors from zero. This is

likely to be more a purely theoretical ratherthan a practical difference, however, since themean difference was rather small in relation tothe size of the standard deviation of log abilityscores for each test.For the two equated vocabulary subtests the

difference between the two means and the twostandard deviations was negligible for the Grade4 data; but for the Grade 5 data the differencebetween means was .13 and the difference be-tween standard deviations was . 10. Although theGrade 5 differences are not exceptionally large,they seem to indicate that there was less than anideal equating of the two vocabulary subtests.The equating results for the CTBS and STEPreading comprehension subtests clearly in-

dicated that these two tests did not provide sta-tistically equivalent log ability estimates

throughout the range of measured ability. The

Table 9Comparison of the Log Ability Scores for the

Equated CAT and STEP Vocabulary Subtests

aEnclosed in parentheses is the correlation between raw scores.



449


Equated CTBS and SRA Reading Subtests


Grade 5 mean difference was -.08, which wasnot especially large; but the Grade 4 mean dif-ference was -.21, and the difference between thestandard deviations was almost .10. The differ-

ence of -.21 was approximately one-fifth of thestandard deviation of the log ability scores foreach test, and therefore item-free person mea-surement was not achieved with respect to

Grade 4 data.The person-free item calibration results asso-

ciated with STEP could account for the pooritem-free person-measurement results for theSTEP and CTBS reading comprehension sub-tests. STEP provided the lowest correlations andthe largest mean differences between the diffi-culty estimates computed for each subtest. Inaddition, the correlational data and the infor-mation from content specialists obtained in theoriginal ATS (Bianchini & Loret, 1974) in-

dicated that the STEP reading subtest may bemeasuring something apart from what the re-maining reading comprehension subtests are

measuring.Other criteria. The item-free person mea-

surement results can also be evaluated in rela-tion to two other criteria-the standard error ofmeasurement and the log ability difference be-tween adjacent raw scores. For example, the dif-

ference between adjacent raw scores in themiddle of the distribution was about .14 log unitfor the CAT vocabulary subtest and about .20log unit for the STEP vocabulary subtest. As ex-pected, the log ability differences were consider-ably larger at the extremes of the score distribu-tion. Therefore, the Grade 5 mean difference of.13 log units for the two equated vocabulary sub-tests represented a difference that was typicallyless than one raw score point in the middle of thedistribution and considerably less than one rawscore point at the extremes.Furthermore, the standard error of measure-

ment for scores in the middle of the distributionwas about .38 for the CAT and about .45 for

STEP. Thus, the mean difference of .13 was no

larger than a third of the error of measurementfor either subtest. Although this is not trivial, itis probable that for many testing situationsmean differences as large as the one for the twoequated vocabulary subtests could reasonably betolerated in relation to the types of decisions

likely to be made with equated tests.With the exception of the Grade 4 mean dif-

ferences for the CTBS and STEP reading com-prehension subtests, the remaining mean differ-ences were rather small (if not trivial) in terms oftheir absolute value, the standard deviation of



450


Equated CTBS and STEP Reading Subtests


the log ability scores, the log ability differencebetween adjacent raw scores, and the standarderror of measurement. The Grade 4 mean dif-ference of -.21 for the CTBS and STEP read-

ing subtests, however, was approximately one-fifth of a standard deviation of the log abilityscores for either test, representing a difference inthe middle of the distribution that was as muchas two score points between adjacent raw scoresand as large as two-thirds of the error of mea-surement. Thus, the equating of the CTBS andSTEP reading comprehension subtests was lessthan satisfactory for Grade 4 examinees.Other than this exception, however, the Rasch

model using the anchor test procedure providedperson measurement with existing standardizedreading achievement tests, which (to a rough ap-proximation) was item free throughout the

range of measured ability. These results are en-couraging, since there was considerably lessthan satisfactory model-data fit in terms of theassumptions of uniform item discriminations;nonspeededness; unidimensionality; and, verypossibly, zero asymptotes (no guessing).

Conclusions

The information pertaining to model-data fitindicated that the assumption of unidimen-

sionality for each of the reading subtests was notmet, strictly speaking, although one very largemajor factor was associated with each subtest.In addition, the assumptions of nonspeedednessand uniform item discriminations were violatedto a substantial degree. It is likely that with onlyfour options per multiple-choice question,guessing was also a factor.Despite the considerable lack of model-data

fit, the Rasch difficulty estimates for the vocabu-lary and reading comprehension subtests werereasonably invariant for replications with ran-dom samples and samples differing in ability byone grade level. Furthermore, with the exceptionof the data for one test pair and one grade level,the Rasch model using the anchor test proce-dure provided a reasonably satisfactory meansof equating three test pairs on the log abilityscale for examinees at two grade levels. Conse-quently, the Rasch model provided approxi-mately item-free person measurement in all butone case with existing standardized readingachievement tests, even though the assumptionsof the model were not adequately met.The item-free person results of this study were

not as good as those for the Wright (1968),Whitely and Dawis (1974), and Tinsley andDawis (1977) studies, which were judged to pro-



451

vide strong support for the claim for item-freeperson measurement with the Rasch model, butthey do lend reasonable support for the claim. Itmay be that the almost ideal item-free person re-sults reported in those studies were the result ofnot using cross-validations with samples thatdiffered in ability, whereas this study and theSlinde and Linn (1978, 1979) studies did involvecross-validations with differing ability level

samples.The results of this study suggest that the

Rasch model has greater utility for the purposeof vertical equating than was apparent in theprevious Slinde and Linn studies. The results re-ported here, however, are not inconsistent withthe results reported in the earlier studies bySlinde and Linn, since their tests of the Raschmodel were quite severe. The comparisons be-tween the nonparallel tests in the Slinde andLinn studies were more extreme in terms of thewide separation of the high- and low-abilitygroups than are apt to be encountered when

equating tests over different grades. They alsoindicated that better results might have been ex-pected with an anchor test procedure. In thisstudy an anchor test procedure was used and thedifference between the high- and low-abilitylevel calibration samples was one grade level. Interms of the standard deviation unit on the logability scale, the separation between high- andlow-ability groups for the hard and easy testswas about five to six times larger for the Slindeand Linn studies than that used here.

Apart from considerations of model-data fit, itmay be that item-free person measurement canbe achieved with the Rasch model to the extentthat samples of examinees differ by the equiva-lent of one or two grade levels. This does not pre-clude the possibility of equating several levels ofa standardized achievement test (assuming uni-dimensionality), since the equating can still be

accomplished grade by grade with the anchortest procedure, although the extent of &dquo;linkingerrors&dquo; is not known and hence should be inves-

tigated.

The results of this study indicate that there isreason to be optimistic about using the Raschmodel to scale existing test data in order that ob-jective measurement can be achieved. The two-and three- parameter logistic latent trait modelsmay be expected to provide even better results,however, since these models attempt to take intoaccount more information about each item byincorporating additional item parameters fordiscrimination and for the lower asymptote. Inaddition, the results of the Slinde and Linnstudies (1978, 1979) and those for monte carlostudies (Dinero & Haertel, 1977; Hambleton &

Traub, 1971) suggest that the Rasch model maynot be adequate for low-ability examinees;hence, models containing additional item para-meters may be required in such circumstances.The higher order models, however, require quitelarge sample sizes with presently used estima-tion techniques, and consequently they mayhave limited application (e.g., regional or na-tional testing). In addition, the conditionalmaximum likelihood estimation procedure can-not be strictly applied to test data when the dis-crimination values are not equal, since the suffi-cient statistic will depend on unknown and un-observable parameters (Hambleton & Cook,1977). Thus, it is not known whether there is anygain in using the higher order models over theRasch model.A modified version of the Rasch model

(Waller, 1976) may prove to be very useful inthose situations in which the Rasch model is not,since the procedure attempts to remove the ef-fects of random guessing from the estimates ofdifficulty and ability without altering the basicproperties of the Rasch model (e.g., mathe-matical and computational simplicity and

uniqueness). The modified procedure is an ap-plication of the abilities removing random

guessing (ARRG) model developed for the two-parameter latent trait model (Waller, 1974). Theassumption underlying the ARRG procedure isthat examinees who guess in an essentially ran-dom manner do so on those items that are too



452

difficult for them. Thus, the ARRG procedureremoves from the estimation procedure thoseitem-person interactions in which a person re-sponds to an item estimated to be very difficultfor that person. Consequently, the modifiedRasch model can be considered a rough ap-proximation to tailored testing, and research us-ing this model should hence prove useful.

References

Anderson, J., Kearney, G. E., & Everett, A. V. Anevaluation of Rasch’s structural model for test

items. British Journal of Mathematical and Statis-tical Psychology, 1968, 21, 231-238.

Bianchini, J. C., & Loret, P. G. Anchor test study:Final report (Project Report and Vols. 1-30) andAnchor test study: Supplement (Vols. 31-33),1974. (ERIC Document Reproduction ServiceNos. ED 092 601-ED 092 634)

CTB/McGraw-Hill. Comprehensive Tests of BasicSkills (1968 ed.). Monterey, CA: Author, 1968.

CTB/McGraw-Hill. California Achievement Tests

(1970 ed.). Monterey, CA: Author, 1970.Dinero, T. E., & Haertel, E. Applicability of the

Rasch model with varying item discriminations.Applied Psychological Measurement, 1977, 1,581-592.

Educational Testing Service. Sequential Tests ofEducational Progress (1969 ed.). Princeton, NJ:Author, 1969.

Hambleton, R. K., & Cook, L. Latent trait modelsand their use in the analysis of educational testdata. Journal of Educational Measurement. 1977,14. 75-96.

Hambleton, R. K., & Traub, R. E. Informationcurves and efficiency of three logistic test models.British Journal of Mathematical and StatisticalPsychology, 1971, 24, 273-281.

Panchapakesan, N. The simple logistic model andmental measurement. Doctoral dissertation, Uni-versity of Chicago, 1969.

Rasch, G. Probabilistic models for some intelligenceand attainment tests. Copenhagen: Danmarks In-stitut Paedagogiske, 1960.

Rentz, R. R., & Bashaw, W. L. Equating readingtests with the Rasch model (Vols. 1 & 2). Athens:University of Georgia, Educational Research

Laboratory, August 1975. (ERIC Document Re-production Service Nos. ED 127 330-ED 127 331)

Slinde, J. A., & Linn, R. L. An exploration of the ade-quacy of the Rasch model for the problem of ver-tical equating. Journal of Educational Measure-ment, 1978, 15, 23-35.

Slinde, J. A., & Linn, R. L. A note on vertical equat-ing via the Rasch model for groups of quite dif-ferent ability and tests of quite different difficulty.Journal of Educational Measurement, 1979, 16,159-165.

Science Research Associates. SRA AchievementSeries (1971 ed.). Chicago: Author, 1971.

Tinsley, H. E., & Dawis, R. V. An investigation of theRasch simple logistic model: Sample-free itemand test calibration. Educational and Psychologi-cal Measurement, 1975, 35, 325-339.

Tinsley, H. E., & Dawis, R. V. Test-free person mea-surement with the Rasch simple logistic model.Applied Psychological Measurement, 1977, 1,483-487.

Waller, M. I. Removing the effects of random guess-ing from latent trait ability estimates (ETS RB74-32). Princeton, NJ: Educational Testing Serv-ice, 1974.

Waller, M. E. Estimating parameters in the Raschmodel: Removing the effects of random guessing(ETS RB 76-8). Princeton, NJ: Educational Test-ing Service, 1976.

Whitely, S. E., & Dawis, R. V. The nature of objectiv-ity with the Rasch model. Journal of EducationalMeasurement, 1974,11, 163-178.

Wright, B. D. Sample-free test calibration and per-son measurement. Proceedings of the 1967 Invita-tional Conference on Testing Problems. Prince-

ton, NJ: Educational Testing Service, 1968.Wright, B. D., & Mead, R. J. BICAL: Calibrating

items and scales with the Rasch model (ResearchMemorandum No. 23). Chicago, IL: University ofChicago, Department of Education, Statistical

Laboratory, 1977.Wright, B.D., & Panchapakesan, N. A procedure for

sample-free item analysis. Educational and Psy-chological Measurement, 1969, 29, 23-48.

Author’s Address

Send requests for reprints or further information toJeffrey A. Slinde, Science Research Associates, Inc.,155 North Wacker Drive, Chicago, IL 60606.



Documents

Model, Objective Measurement, Equating, and Robustness · 437 The Rasch Model, Objective Measurement, Equating, and Robustness Jeffrey A. Slinde Science Research Associates, Inc