Which Comprenhension... 1980.pdf

8/13/2019 Which Comprenhension... 1980.pdf

1/5

Which Comprehension? Artifacts in the Measurement of Reading ComprehensionAuthor(s): Janet Ross Kendall, Jana M. Mason and William HunterSource: The Journal of Educational Research, Vol. 73, No. 4 (Mar. - Apr., 1980), pp. 233-236Published by: Taylor & Francis, Ltd.Stable URL: http://www.jstor.org/stable/27539756.

Accessed: 25/01/2014 23:35

Your use of the JSTOR archive indicates your acceptance of the Terms & Conditions of Use, available at.http://www.jstor.org/page/info/about/policies/terms.jsp

.JSTOR is a not-for-profit service that helps scholars, researchers, and students discover, use, and build upon a wide range of

content in a trusted digital archive. We use information technology and tools to increase productivity and facilitate new forms

of scholarship. For more information about JSTOR, please contact [email protected].

.

Taylor & Francis, Ltd.is collaborating with JSTOR to digitize, preserve and extend access to The Journal of

Educational Research.

http://www.jstor.org
http://www.jstor.org/action/showPublisher?publisherCode=taylorfrancishttp://www.jstor.org/stable/27539756?origin=JSTOR-pdfhttp://www.jstor.org/page/info/about/policies/terms.jsphttp://www.jstor.org/page/info/about/policies/terms.jsphttp://www.jstor.org/stable/27539756?origin=JSTOR-pdfhttp://www.jstor.org/action/showPublisher?publisherCode=taylorfrancis


2/5

Which Comprehension? Artifacts inthe Measurement of Reading ComprehensionJANETROSS KENDALL

Simon Fraser University

JANA M. MASONUniversity of Illinois, Urbana-Champaign

WILLIAMHUNTERMount St. Vincent University

ABSTRACT Differences in fifth graders' reading comprehension scores were obtained using four different taskstypically employed to measure comprehension (multiplechoice, recall, cloze, and maze) and four different reading passages that were equated according to readabilityformulas. Data analyses revealed significant effects for passage, task, and an interaction between task and passage. Itwas concluded that the choice of a particular comprehension passage and testing procedure, whether in research orpractice, does not allow generalization to other operational definitions of reading comprehension. These resultssuggest serious limitations of most contemporary readingcomprehension research and testing.

Severaltasks have been used tomeasure readers' com

prehension. For many years the multiple-choice format has been incorporated in standardized achievementtests and in basal reading materials. Alternatives to the multiple choice test include recall, inwhich, after people reada passage, they are urged to describe everything they canremember about it [e.g., Bransford & Johnson (3)]. Another, cloze, is usually constructed by deleting every fifthor seventh word (2) and scored by counting as correctonly the exact words that were deleted. Somewhat similar to the cloze ismaze inwhich every fifth or seventhword can be replaced by three choices, one of which isthe correct word (7).

Often practitioners and researchers presume that written materials (passages) within a given study are similar orthat scores from different measures of comprehensionwould rank individuals in the same order. That is, theyassume an equivalence of passage type or an equivalenceof comprehension task type.These assumptions of equivalence are not universallyaccepted, however. For example, Clark (4) has arguedthat researchers must be cautious in generalizing from re

suits based on one set of materials to others?committingthe language-as-fixed-effect fallacy. Moreover, Gibsonand Levin (6) state, ... almost any measure raises questions of validity, reliability, and generalizability to anyother material or task (p. 409). Some comprehensiontasks rely on memory (recall), others rely on search(maze) or on organization of information (cloze). Thuscomprehension tasks actually tap quite different strategies and processing mechanisms. Passages, too, can be constructed in quite different ways, for example, descriptivestories and myths. These differences are described morefully in Pennock (12). Further, because passages contain

more or less familiar content to particular individuals,comprehension can be affected by the topic (14). Therefore, it can be true for any investigation of reading comprehension that different conclusions might have beendrawn if some other sets of tasks or passages had beensupplied.The primary purpose of this study was to examinesome variations in reading comprehension scores that result from different task and passage measurement combinations.

MethodThe subjects were 164 fifth graders from four elemen

tary schools located in a large eastern Canadian city. Inorder to assure comparability, participating schools werechosen only if the average of the raw scores for fifthgraders on theMetropolitan Achievement Test (MAT)Comprehension subtest corresponded to grade equivalents between 5.2 and 5.7.

Multiple choice, recall, cloze, and maze tasks wereconstructed for four reading passages from the McCallCrabbs Standard Test Lessons inReading (11). This setof reading materials was selected because it has been thecriterion for readability formulas more often than anyother single criterion (9:53), because the passages were

This content downloaded from 148.206.53.9 on Sat, 25 Jan 2014 23:35:31 PMAll use subject to JSTOR Terms and Conditions
http://www.jstor.org/page/info/about/policies/terms.jsphttp://www.jstor.org/page/info/about/policies/terms.jsphttp://www.jstor.org/page/info/about/policies/terms.jsp


3/5

234 THE JOURNALOF EDUCATIONAL ESEARCHshort and complete, and because each passage was accompanied by eight to twelve normed, multiple-choicequestions. Passages were chosen that show a grade-levelcomprehension range of 5.2 to 5.6 when 70 percent ofthe questions are answered correctly. The readability(grade five, according to the Fry formula), passage length(127 to 142 words), and location of main idea (first sentence) of the passages were equated, but the structure ofthe passages was deliberately varied. One passage, Woodpecker, iswritten in a fairy tale form. Ants is descriptive, though it also has a story form. The other two areexpository: Paper contains a list-like explanation of thesteps inmaking paper; Whales uses a causal structure todescribe why people tag whales.The passages and tasks were organized into a 4 X 4Latin Square design with students randomly assignedto one of four groups. Each student read the four passages, each one under a different comprehension condition. So that instructions and practice for each comprehension task could be given to the students as a group,the conditions were given in this order: (a) cloze, withevery fifth word replaced by a blank; (b) McCall-Crabb

multiple-choice comprehension questions; (c) free recall,with students asked to write down everything they couldremember about the passage; and (d) maze, with everyfifth word replaced by three alternatives.The students worked at their own pace with no timelimits; the only constraint was that for the multiplechoice and recall tasks they were not allowed to lookback at the text. The number of correct responses wasthe dependent measure for three of the tasks. On the recall task, the number of idea units was measured using atechnique based on work by Johnson (8).

ResultsInspection of proportions correct (Table 1) shows

that the maze and multiple-choice tasks were consistently easier than recall and cloze, and that the fairy talepassage, Woodpecker, was consistently easier than theothers for all task types. Ants, which is nearly a storyform, was always second or third in difficulty, and

Table 1.?Average Proportions Correct for Each Task/PassageConditionAverageTasks across tasks

Cloze Choice Recall MazeWhales .465 .856 .276 .862 .615Woodpecker .554 .951 .378 .938 .705Ants .510 .835 .334 .935 .654Paper .426 .816 .374 .906 .631

Average acrosspassages .489 .864 .341 .910

Whales and Paper, the two expository passages,were the more difficult. These observations are confirmed by the analysis of variance (Table 2). The interaction between tasks and passages, which was also significant, shows that the difficulty of the Whales andPaper passages seemed to depend on the type ofmeasurement task employed. Paper was the more difficult to comprehend with the cloze task, whereas Whales'1was the more difficult with the recall and maze tasks.

A second set of analyses was carried out to measurethe effects of reading ability on comprehension tasks andpassages. Students were divided into three groups on thebasis of their Canadian Tests of Basic Skills Comprehension subtest scores. The analyses showed expected maineffects of reading ability, passages, and tasks; the interactions between ability and passages and between abilityand tasks were not significant. (See Tables 3 and 4.)Discussion

The finding that the maze and multiple-choice taskswere consistently easier than recall and cloze is not surprising (1, 2,13). Children generally find comprehensiontasks easier when selecting from given answers (maze andmultiple-choice) than when remembering an answer (re

Table 2.-Analysis of Variance, Passage and Task EffectsSource SS dfS F

Between-subjectsGroups .07 3 .02 .435Subjects within groups 7.35 160 .046

Within-subjectsPassages .79 3 .263 17.08*Tasks 38.65 3 12.88 835.68*Passages X tasks .38 6063 4.11*Error 7.4 480 .0154

*p < .001

Table 3.-Analysis of Variance, Task and Reading Ability EffectsSource SS dfS F

Between-subjectsAbility 3.97 2 1.99 48.23*Error 6.71 163 .04

Within-subjectsTasks 38.10 3 12.70 695.92*Tasks X ability .22 604 2.02Error 8.92 489 .02

*p < ,001



4/5

KENDALL-MASON-HUNTER 235

Table 4.-Analysis of Variance, Passage and Reading Ability EffectsSource SS df MS

Between-subjectsAbility 3.97 2 1.99 48.23*Error 6.71 163 .04

Within-subjectsPassages .75 325 2.66**Passages X ability .25 604 .44Error 46.30 489 .09

*p < .001**p < .05

call) or constructing an answer (cloze). Such differencesin difficulty should be considered when planning a comprehension test.Also to be considered is the type of passage. Performance varied significantly among the four passages, eventhough they were carefully selected and were controlledfor readability. This suggests that seemingly comparablepassages are very difficult to equate. As a result, until webetter understand what factors in text make passages moreor less difficult, practitioners and researchers must becautious when interpreting comprehension test resultsbased on different reading passages.The combinations of the various tasks and passagesalso produced systematically different results. Thus theproblems associated with comparing studies using different operational definitions may be even greater than previously suspected. While the fairy tale Woodpecker wasalways the easiest passage, Paper was the hardest on twoof the tasks, and Whales was hardest on the other two.It is conceivable that the two expository passages haddifferential effects depending on their cohesiveness.Paper had well-sequenced information whereasWhales presented the conclusions first and the motivation information last. These kinds of deviations have been

shown to effect recall (15) and reconstruction of stories(10). Thus the task and passage interaction providesfurther evidence of heretofore unsuspected differences.The main effect of reading ability in the analyses ofability with tasks and passages was not at all surprising. Itshowed that the better readers obtained higher scores onthe comprehension tasks and passages than did the poorerreaders, a finding teachers would certainly expect.

ConclusionThe interaction between passage and task raises the

question of which, if any, of the operational definitionsused here is actually reading comprehension. The answer, of course, is that each of them is, and, at the sametime, none of them is. That is, as Davis (5) has argued,comprehension is not a unitary construct. Several impli

cations of this conclusion are warranted. First, researchersshould be cautious about presenting research that purportsto label different passages as equal; critical analysis ofmaterials is necessary aswell as awillingness of researchersto make their materials readily available to others. Second,further research is needed to determine what other factors, such as text cohesiveness, text structure, or topicfamiliarity, affect passage difficulty and interact withmeasurement of reading comprehension. Third, teachersought to consider a variety of tasks and passage typeswhen instructing students in reading comprehensionsince any one type of measure may be too restrictive.Fourth, these results suggest a need for more varied comprehension measures for testing and evaluating children.It is apparent that the difficulty found here in obtainingcomparable tasks and passages makes comprehensiontest interpretation risky.

NOTEMaterials are available from the first author on request atthe Faculty of Education, Simon Fraser University, Burnaby,

B.C., Canada.

REFERENCES1. Bormuth, J. R. Comparable Cloze and Multiple Choice

Comprehension Test Scores. Journal of Reading 10 (1967):291-99

2. Bormuth, J. R. The Cloze Readability Procedure. ElementaryEnglish 45 (1968): 429-36.3. Bransford, J. B., and Johnson, M. K. Contextual Prerequisites for Understanding: Some Investigations of Comprehension and Recall. Journal of Verbal Learning and VerbalBehavior 11 (1972): 717-26.4. Clark, H. H. The Language-as-Fixed-Effect Fallacy: ACritique of Language Statistics in Psychological Research.Journal of Verbal Learning and Verbal Behavior 12 (1973):335-59.

5. Davis, F. B. Research in Comprehension in Reading. Reading Research Quarterly 4 (1968): 499-545.6. Gibson, E. J., and Levin, H. The Psychology of Reading.

Cambridge, Mass.: M.I.f. Press, 1975.7. Guthrie, J. T. Reading Comprehension and Syntactic Responses in Good and Poor Readers. Journal of EducationalPsychology 65 (1973): 294-99.8. Johnson, R. Recall of Prose as a Function of the Structuralor the Linguistic Units. Journal of Verbal Learning and Verbal Behavior 9 (1970): 12-20.

9. Klare, G. R. The Measurement of Readability. Ames, Iowa:Iowa State University Press, 1963.10. McClure, E.; Mason, J.; and Barnitz, J. Story Structure andAge Effects on Children's Ability to Sequence Stories, Tech.Rep. No. 122. Urbana: University of Illinois, Center for theStudy of Reading, May 1979.11. McCall, W. A., and Crabbs, L. M. Standard Test Lessons in

Reading. New York: Teachers College Press, Columbia University, 1961.12. Pennock, C, ed. Reading Comprehension at Four LinguisticLevels. Newark, Delaware: International Reading Association, 1979.13. Rankin, E. F., and Culhane, J. W. Comparable Cloze and

Multiple Choice Comprehension Test Scores. Journal ofReading 13 (1969): 193-98.14. Steffensen, M. S.; Joag-Dev, C; and Anderson, R. C. A



5/5

236 THE JOURNALOF EDUCATIONAL ESEARCHCross-Cultural Perspective on Reading Comprehension.Reading Research Quarterly 15 (1979): 10-29.

15. Stein, N., and Nezworski, T. The Effects of Organization andInstructional Set on Story Memory, Tech. Rep. No. 68.Urbana: University of Illinois, Center for the Study of Reading, January 1978.

^WESTERN EUROPEAN \EDUCATIONA Quarterly Journal of Translations

Editor: Raymond E. Wanner, U. S. Office of Education

... an important contribution to the scholarly studies of comparative education. ? Library Journal

Contributing Editors

Suzanne Audebert, Institut P?dagogique National, Paris; William W.Brickman, University of Pennsylvania; Max Eckstein, Queens College, City University of New York; Sebastiaan J. C. van Eyndhoven,

University of Amsterdam; Christoph F?hr, Deutsches Institut f?rInternationale P?dagogische Forschung, Frankfurt; Marcello Luketti,

University of Rome; Jose M. Diaz de Rabago, University of Santiago,Spain.

Forthcoming Issues

EDUCATIONAL REFORM AND THE WORLD OF WORKEducational Planning and Secondary School Reform in ItalyEconomic Change, Educational Needs, and Secondary School Reformin SwedenEducational Needs, Economic Change and Secondary School Reformin France

Educational Policy and Secondary School Reform in EnglandQuarterly. First issue: Spring 1969.Institutions: $105.00Individuals: $30.00

?sM. E. Sharpe, Inc., Publisher901 N. Broadway, White Plains, N.Y. 10603


Documents

Which Comprenhension... 1980.pdf