Let’s Teach Research Evaluation

Joe Dan AustinEmory University

Atlanta, Georgia 30322

INTRODUCTION

This paper describes a course in reading, evaluating, and using educa-tional research taught to inservice mathematics teachers in the EmoryUniversity Master of Arts in Teaching program. It is argued that such acourse can begin to solve one of the most perplexing problems in improv-ing education�how to transmit knowledge gained through educationalresearch to teachers. If results from educational research are to have anyimpact on education, teachers will have to implement and use these re-sults in their own classrooms. However, there is strong evidence thatmerely publishing research studies is not sufficient to disseminate thisinformation. For example, Reys and Yeager (1974) surveyed 40 ele-mentary teachers and found that while 97.5% "Always or Frequently"read educational journals, 87.5% indicated they "Seldom or Never"read research flavored articles. The i\\’o main reasons postulated forthese percentages were (1) the lack of relevance of most published re-search for classroom teachers and (2) the lack of exposure to research inmost preservice teacher preparatory programs. This paper argues for acourse in research evaluation and gives some suggestions for the contentof such a course.

BASIC COURSE PHILOSOPHY

When advocating a course in research evaluation for teachers, it isimportant to point out that teachers have neither the time nor the interestto systematically evaluate and synthesize educational research in general.These tasks are best done by the research educator. Like Cogan (1975, p.218) it is NOT advocated that school teachers become researchers. How-ever, teachers do need to use research results. Therefore a course inevaluating research should seek to show how the classroom teacher canprofit from research. [Note that it is NOT advocated that research willsolve all classroom problems for the teacher. See Rappaport (1972) for adiscussion of this.] A study of evaluating research will lead to questionson research design, data collection, and statistical analysis. Again it isnot advocated that teachers become experts in these areas. However, astudy of evaluating research should help teachers better understand whattype of questions can be answered by research and when research result^are likely to be valid in the teacher’s own school and class environment.A basic competency in understanding and evaluating research

could�as Cogan (1975, p. 218) argues�

425

426 School Science and Mathematics

(a) keep teachers (and their administrators) from hopping on every bandwagon thatcomes down the street; (b) help them to maintain a critical approach to innovation,rather than making apparently eternal and unexamined commitments to new pro-cedures every year or so; and (c) inform them about new research findings that couldcause them to question, discard, or revise their modes of teaching in the light of newknowledge and careful speculation.

In addition, the teacher can better understand what types of questionscan be answered by research. This understanding would permit a betterassessment of whether the teacher (or school program) was having thedesired effects.

EXAMPLES FROM A SPECIFIC RESEARCHEVALUATION COURSE

The following specific suggestions are based on a graduate level re-search evaluation course developed by the author at Emory University.This twelve week course meets once a week for three hours. Almost allstudents are full time mathematics teachers. Clearly some modificationswould be necessary in the recommendations if the course were offered tostudents different from those in the Emory program.A large number of different text books are available for a course in re-

search evaluation. Following the review by Bright (1972), Tuckman(1972) was selected. This text uses many of the ideas in Campbell andStanley (1963). In addition to the text, students were required to read andevaluate in writing specific articles similar in design to those being con-sidered in the text. [To facilitate this evaluation, the evaluation forms ofFarquhar and Krumboltz (1959) and Suydam (1968) were provided.]During part of the class period, the assigned research paper was dis-cussed. Then the class considered other researchable hypotheses relatingto the same general area as in the assigned paper. This was followed bythe author reviewing the research in this area. The area selected was al-ways relevant to mathematics education and/or classroom teaching ofmathematics. These two activities were felt valuable as they gave the stu-dents a better understanding of what type of questions could be re-searched and which actually had been investigated. The state of the artreview helped disseminate research information that related to classroomteaching. Some specific topics and survey papers for the review are asfollows: discovery learning [Bittinger (1972)], the relation of sex tomathematical achievement [Fennema (1974)], the prediction of geo-metric achievement [Sowder (1974)], the relationship of attitude toachievement in mathematics [Neale (1969)], the relation of teacher atti-tudes toward mathematics to student attitudes [Chase and Wilson(1958)], programmed instruction [Carroll (1968)], what makes a goodteacher [Hamachek (1969)], and homework in mathematics [Austin(1978)]. Of course, there are many other equally acceptable topics. [See,for example, Suydam (1974).]

Research Evaluation 427

Since most of the students were full-time teachers, they were alsorequired to do an empirical investigation. The author carefully outlined aseries of possible studies. Most studies took six weeks to complete, so theresults were discussed when they became available. For students whowere not teaching, alternate empirical investigations were suggested. Ifany student had a particular study he wished to investigate, it was con-sidered. If this study seemed feasible, given the student’s backgroundand teaching constraints, it w^as accepted. Some studies suggested by theauthor replicated existing studies while others were new studies. Theprojects were used to show the students that research results could applyto their own school environment. Thus the plausibility and usefulness ofthe research literature (although carefully selected to be sure), couldhopefully be seen. To avoid statistical problems, the author analyzed alldata with computer programs available at Emory University.

SAMPLE CLASS PROJECTS

The following selected projects are briefly summarized to show thetypes of questions investigated and how teachers can obtain empiricalinformation on these questions.One student (Lois Hogan) investigated the relationship of selected

academic measures to students’ grades in tenth grade (Euclidean) geom-etry. All academic measures were taken from the students’ permanentrecords in the school. The following eight variables were selected: (1) lan-guage percentile scores on the Georgia standardized test given to eighthgrade students, (2) mathematics percentile scores on the test described in(1), (3) first year algebra grade, (4) eighth grade mathematics grade, (5)ninth grade geography grade, (6) ninth grade science grade, (7) cumula-tive grade point average for grades eight through ten, and (8) school-wideclass rank. The students’ I.Q. scores were not available. A total of 59regular (non-accelerated) students was selected at random for thisinvestigation. Using a stepwise regression program, the predictionequation selected two variables, first year algebra grade and cumulativegrade point average. Using a = 0.05 none of the other variables were

added to the regression equation. The regression equation had an R =

0.672. Note this one study is particular to one school and has only 59 stu-dents. In spite of these possible difficulties, this study replicates wellsome previous work. Douglass and Kinney (1938) surveyed studies donein the 1930’s and found a range of R2 values of 0.63 to 0.78. Caldwell(1970) found that the best one predictor variable for geometry was stu-

dents’ first semester algebra grades.Seven students investigated whether writing comments on students’

homework papers significantly improved students’ test scores. This was

an attempt to generalize Page’s (1958) paper on the effects of written

428 School Science and Mathematics

comments on test papers on student achievement. The combined dataresulting from these projects is discussed in Austin (1976). The resultstended to indicate that (i) comments significantly improved performancein geometry, (ii) comments could significantly improve performance ingeneral mathematics classes but do not necessarily do so, and (iii) com-ments were not shown to have any measurable effect on achievement inalgebra classes. Pooling all 222 students in the experiment indicated thatcomments significantly improved achievement for the group as a whole.These results showed that a teaching technique could have measurable ef-fects with certain courses (or type students) and not with others.Two students (Dana Parkans and Margaret B. Schick) investigated the

effects of two methods of assigning homework. The two methods wereVertical and Spiral. [See Peterson (1971) and Laing and Peterson (1973).]As each student taught two classes of the same subject, each studentchose one class at random for the Vertical homework method. The otherclass of each student had the Spiral homework method. The initial com-parability of the classes was tested using a t-test on the classes’ previoussix weeks grades. For one pair of classes the initial comparability was re-jected. This served to illustrate one of the major problems in educationalresearch�the possibility of non-homogeneous intact classes (or groups).One student (Penny Reese) who was not teaching sought to investigate

the effects of a course in research evaluation on teachers’ attitudestoward educational research. A Likkert type scale with 21 items \\as con-structed. This questionnaire was completed by the 12 students in the classand then each student was asked to have one or more fellow teacherscomplete the questionnaire�preferably teachers without a course onevaluating research. A total of 29 teachers completed the question-naire�17 having had a research evaluation course and 12 not having hadsuch a course. (The coefficient a for this questionnaire was 0.66 forteachers with a course on evaluating research and 0.92 for teachers nothaving such a course.) Using this (decidedly non-random and small)sample, the teachers with a research evaluation course had a more favor-able opinion of educational research as measured by 18 of the 21 items.Using a t-test on the sum of the 21 questionnaire items, the researchevaluation group showed a more favorable attitude that was notsignificant at a = 0.05 but was significant at a = 0.10. The three itemsthat showed the largest differences between groups were as follows (dif-ference favorable to research group):

7. Do you ever read research articles in education?21. Do you feel research results are impractical for teachers to use? (This item is

worded in the negative, and the scale was reversed on the scoring.)19. Do you ever go to professional conferences?

The item showing the largest reverse differentiation was

Research Evaluation 429

11. Do you feel that the statistics in research articles is a hindrance to your understand-

ing of research.

Clearly this questionnaire is NOT sufficient to base any claims for acourse in research evaluation. Both a more carefully developed question-naire and a more nearly random sample of teachers are needed. In fact,the first item (No. 7) may only reflect the fact that students are usuallyrequired to read research reports in a research evaluation course. How-ever, the second item offers some positive encouragement that the useful-ness of research to the classroom teacher is more often seen by the grouphaving had a research evaluation course. The third item (No. 19) may re-flect a confounding of the sample�the research evaluation studentsseem to go to more professional meetings. Finally, the negative item mayreflect that students with a research evaluation course have read researchreports and found that the statistics are indeed a hindrance to theirunderstanding of research reports. In summary, this project raises morequestions than it answers. A more complete investigation is needed.However, it does provide a good example of attitude assessment andevaluation.

CONCLUSIONS

This paper argues for a research evaluation course somewhere in theteacher preparation program. The logical place seems after some class-room teaching experience. Such a course should help the practicingteacher read, evaluate, and use�when possible�the results of ed-ucational studies. With inservice teachers a small scale empirical studycan be used to show the applicability of known research results and togive a better understanding of some of the questions that can be re-searched. While teachers do not have time to become practicing re-searchers, a course in research evaluation should provide some familiar-ity with how to read and interpret research in education. It is unlikelythat this one course will solve all the communication problems betweenresearchers and classroom teachers. However, this course may be thelogical first step. The second step may be as Reys and Yeager (1974)point out, the researcher needs to write in a manner that is easier for theclassroom teacher to understand and to research problems more directlyrelated to classroom teaching.

REFERENCESAUSTIN, JOE DAN Homework research in mathematics. School Science and Mathematics^

1978, In press.AUSTIN, JOE DAN Do comments on mathematics homework affect student achievement?

School Science and Mathematics, 1976,76, 159-164.BITTINGER, MARVIN L. A review of discovery. The Mathematics Teacher, 1968, 61, 140-

146.BRIGHT, GEORGE W. Research methodology textbooks: 1970-1972. Educational Studies,

1972.3.191-195.

430 School Science and Mathematics

CALDWELL, J. R. Structure-of-intellect factor abilities relating to performance in tenth-grade modern geometry. (Doctoral dissertation, University of Southern California) AnnArbor, Mich.: University Microfilms, 1970, No. 70-11361.

CAMPBELL, DONALD T. AND STANLEY, JULIAN C. Experimental and quasi-experimental de-signs for research. In N. L. Gage (Ed.), Handbook of Research on Teaching. Chicago:Rand McNally, 1963.

CARROLL, JOHN B. Basic and applied research in education. Harvard Educational Review,1968,38, 273.

CHASE, W. LINWOOD AND WILSON, GILBERT M. Preference studies in elementary schoolsocial studies. Journal of Education, 1958 (April), 140, 2-8.

COGAN, MORRIS L. Current issues in the education of teachers. In Kevin Ryan (Ed.)Teacher Education, The Seventy-fourth Yearbook of the National Society for the Studyof Education. Part II. Chicago: The University of Chicago Press, 1975.

DOUGLASS, R. R. AND KINNEY, L. B. Prediction of achievement in mathematics. Review ofEducational Research, 193 8, 8, 51-57.

FARQUHAR, WILLIAM W. AND KRUMBOLTZ, JOHN D. A check list for evaluating experi-mental research in psychology and education. Journal of Educational Research, 1959,52, 353-354.

FENNEMA, ELIZABETH Mathematics learning and the sexes: A review. Journal for Researchin Mathematics Education, 1974,5, 126-139.

HAMACHEK, DON E. Characteristics of good teachers and implications for teacher educa-tion. Phi Delta Kappan, 1969,50, 341-345.

LAING, ROBERT A. AND PETERSON, JOHN C. Assignments: Yesterday, today, and tomor-row�today. The Mathematics Teacher, 1973, 66, 508-518.

NEALE, DANIEL C. The role of attitudes in learning mathematics. The Arithmetic Teacher,1969,76,631-640.

PAGE, ELLIS BATTEN Teacher comments and student performance: a seventy-four class-room experiment in school motivation. The Journal of Educational Psychology, 1958,49(4), 173-181.

PETERSON, JOHN C. Four organizational patterns for assigning mathematics homework.School Science and Mathematics, 1971, 77, 592-596.

RAPPAPORT, DAVID Research�The new panacea. School Science and Mathematics, 1972,72, 169-172.

REYS, ROBERT E. AND YEAGER, TERESA Elementary teachers and research in mathematicseducation. School Science and Mathematics, 1974, 74, 431-436.

SOWDER, LARRY High versus low geometry achievers in the NLSMA Y-population. Journalfor Research in Mathematics Education, 1974,5, 20-27.

SUYDAM, MARILYN N. An instrument for evaluating experimental educational researchreports. The Journal of Educational Research, 1968, 61(5), 200-203.

SUYDAM, MARILYN N. A Categorized Listing of Research on Mathematics Education (k-12)1964-1973. ERIC; The Ohio State University; Columbus, Ohio; 1974 (August).

TUCKMAN, BRUCE W. Conducting Educational Research. New York: Harcourt Brace Jo-vanovich, Inc., 1972.

SCARLET PRAWNS CLUSTER IN OIL-POLLUTEDSEABOTTOM AREAS

Oil is generally thought to harm marine life, but in some instances oceanorganisms seem to prefer and thrive in oil-laden areas. In studies off the coast ofAruba in the Dutch West Indies, scarlet prawns, Crustacea dwelling on the oceanbottom (benthos), were found to be three times more abundant in oily placesthan in adjacent nonoily areas. The prawns are most likely attracted by petro-leum-derived compounds formed by bacteria acting upon the oil. Prawns fromthe polluted areas accumulate petroleum-derived hydrocarbons selectively, ac-cumulating only those that are normally present in prawns from unpollutedareas. However, in prawns from oil-polluted areas, these characteristic hydro-carbons are present in greater concentrations and different proportions�a find-ing that suggests metabolic usage.