EXPLORING SOME MYTHS IN RESEARCH DESIGN*

GLEN ROBINSON, Ph.D.

National Education Association, Washington, D.C.

This paper is in the nature of a review of some things that all good researchers know and do to the best of their abilities. What I would like to do is to call to mind a few general points about some of the problems and pitfalls that beset researchers in the educational, social, and health sciences.

We live and work in a scientifically-oriented society, where people often tend to make a kind of religion out of science. They look to scientific investigation for answers to their questions, solu- tions to problems, directions that will tell them what to do. Potentially, this is a wholesome and productive situation. Public confidence in sci- entific research opens the door to progress on a sound scientific basis and gives research the op- portunity to make an impact on society and official policy. It provides the researcher with the opportunity to promote changes for the gen- eral and the individual good.

But this is also a potentially dangerous situa- tion. The unquestioning faith that so many people have in science today can lead to myths about research, to false ideas about what re- search can do and the way it operates. Thus along with the researcher’s power to make an impact on society goes the responsibility to use that power properly, to see that studies are well designed and reported, that the results are not distorted or abused, the responsibility to be aware of the misconceptions that may arise and to combat the myths that grow up around research.

Probably the most basic myth about research is the one I have just mentioned: this tendency of popular thought to make a religion out of science and to expect research to provide absolute truth and absolute imperatives of action. Quite obviously, the first charge incumbent upon re- searchers is the necessity to fight this tendency to attribute to research powers which it does not have, and to resist the temptation to become prophets of a modern-day cult, handing down authoritative pronouncements.

The idea that the purpose of research is to “prove” or “confirm” a theory is so prevalent in our attitudes and experiences as scientists that it sometimes shocks us to think that in a very fun- damental sense this is not what we are doing in our research design. In research we never prove or confirm a theory or hypothesis, but rather, as Campbell and Stanley’ have pointed out, the successful theory is merely one that is tested and escapes being disconjrmed. The results of re- search probe but do not prove a theory. “An adequate hypothesis is one that has repeatedly survived such probing-but it may always be displayed by a new probe.”

No researcher should start out with the expec- tation that he will find definite answers to all the questions about his topic, nor should he attempt to represent his findings as if they are complete and conclusive answers. The results of research are inevitably limited and incomplete, and we need to remember that the findings of any study are susceptible to rectification and supplementa- tion by further study.

Along with the myth that gives research exces- sive powers, there is a myth that tends to arrange research methods into a hierarchy and assign the place of honor to the disciplined precision and efficiency of experimental research. However, many of the most important problems in the educational, social, and health sciences defy study by true experimental designs. When working in these problem areas, the researcher seldom has complete mastery to schedule treatments and measurements for the optimal statistical efficiency sought in the true experimental study. For ex- ample, in studying the effects of malnutrition on the learning processes of children, ethical con- siderations make it impossible to subject an ex- perimental group of children to a controlled process of gradual starvation in order to deter- mine the effect on their mental activity. In the face of problems like this-and there are many such in the field of social science-the experi-

*Presented to the Research Council Scientific Forum of the Forty-Third Annual Meeting ?f the American School Health Association, Philadelphia, Pennsylvania, November 8, 1969.

The Journhl of School Health-Seplember, 1970

‘Campbell, Donald T., and Stanley, Julian C., “Ex- perimental and Quasi-Experimental Designs for Re- search on Teaching,” Chapter 5, Handbook o Research

tion, Rand McNally and Company, Chicago, 1963, p. 205.

335

on Teaching, American Educational Researc i Associa-

mental method invalidates itself because of its practical inapplicability.

In order to study significant problems in these areas, we must be content to use methods that approximate as closely as possible the efficiency of the experimental design. We must devise quasi-experimental, analytical, and descriptive designs to study conditions as they exist instead of manipulating conditions in a controlled experi- ment and studying the results. A great deal can be achieved by such methods. This is dra- matically illustrated by the studies of smoking and health compiled and evaluated in the Surgeon General's report, surely a research document of major social and economic significance to our time. A large part of the data presented there is derived from population studies, either retro- spective studies of the smoking history of persons afflicted with various diseases, or prospective studies projecting mortality rates by cause for smokers and non-smokers. Both these methods, the retrospective study and the prospective study, represent ways of approximating an experimental design in studying a problem for which a true experimental design cannot be used.

There is thus no hierarchy of research meth- odology, and the user of an experimental design has no cause to imagine himself superior to his colleagues using quasi-experimental, analytical, or descriptive designs. Nor should a researcher allow himself to become an exclusive devotee of experimental design; for if he does, the limited utility of experimental design may restrict him to studying only simple and insignificant prob- lems, and the results of his research will be equally insignificant. The importance of the problem under study should always be para- mount; the research design is only an instrument for studying the problem. And the research de- sign that best fits the problem under study is the best design in terms of appropriateness, efficiency, and respectability.

In population studies of the type we were just discussing, a crucial factor is the selection of the sample to be studied. Here we encounter another widely held erroneous belief: that the bigger the sample, the better the study. It is, of course, the representativeness of the sample which is the decisive factor, not the sample size. Large sam- ple size does not guarantee the accuracy of find- ings; nor, conversely, does a small sample neces- sarily mean that results of the study are not valid. Probably the most dramatic illustration of the inadequacy of sample size as a criterion of accuracy was the failure of the huge Literary

Digest Poll to predict correctly the outcome of the 1936 Presidential election, a t the same time that more representative small sample surveys proved more successful in obtaining accurate results.

All samples have some limitation and bias; and in drawing conclusions from sample data we always need to keep in mind the external validity of the sample. In the first place, if the sample has, by selection, been restricted to a specific population, it cannot, except with extreme cau- tion, be used as a basis for drawing conclusions about other populations. If the sample repre- sents the population of a particular state, the members of a particular profession, or the stu- dents in a particular school, we must be careful about inferences applying to other groups or populations.

This consideration has a bearing on retrospec- tive health studies, such as those used in studying the effects of smoking. The retrospective tech- nique, in which the characteristics of a group with a certain disease are compared with the characteristics of a group without the disease, utilizes a research design which does not provide a sample of the'total population. This limits the conclusions which can be drawn from the data. It is not possible, on the basis of such data alone, to calculate the probability that a person with a specific characteristic will contract the disease; it is only possible to determine whether certain Characteristics are more prevalent among those with than without the disease. To proceed further and estimate risk, it is necessary to know the prevalence of the disease in the total popula- tion, and also to assume that the sample group with the disease represents the total population with the disease and that the control group is representative of the total population without the disease.

These assumptions introduce another problem in the use of sample data: the problem of sample bias. No sample perfectly represents the popu- lation it is intended to represent. The retro- spective health study affords one type of ex- ample. In such a study, the sample group of persons with a particular disease consists of iden- tified cases of that disease. Cases of the disease which for some reason or other have not been identified remain an unknown factor. It is a sample selected by diagnosis, rather than a ran- dom selection from among all persons with the disease. In circumstances where reporting and diagnosis are well developed, this factor may not present a serious problem of bias; where these

336 The Journal of School Health

conditions do not exist, however, the representa- tiveness of the sample is open to question.

There are also other factors that can bias a sample, such as, for example, the use of volunteer participants or of volunteer recruiters to obtain participants for a survey. Some major national groups have based widely publicized conclusions on information collected by volunteer members who go out and contact a few friends and neigh- bors to take part in the survey. This method offers no assurance that the group of persons con- tacted constitutes a representative sample of the adult population of the United States. Some- times, voluntary participation may even seriously limit the validity of the results of a study. This is probably what occurred in the case of the Kinsey Report. The only persons whose char- acteristics are recorded in the Kinsey Report are those who were willing to answer Dr. Kinsey’s questions, and it may well be doubted whether they are typical of the population a t large.

Limitations of this type do not invalidate sampling as a method of research; rather they are some of the dangers inherent in the method which we as researchers must guard against if we are to produce results in which confidence can be placed.

A number of the research myths we have been considering are of a fairly subtle nature. Some others present much simpler problems: for ex- ample, the computer mystique. How many studies have you read in which the researcher makes a major point of having fed his data into a computer and come out with certain results, seemingly implying that because the percentages and correlation coefficients were computed elec- tronically they are of a higher order of statistical measurement than if they had been computed on a simple hand calculator? We should not deceive ourselves about this. The computer, it is true, has made possible complex interaction analysis and elaborate analytical measures, but the under- lying assumptions on which these statistical tech- niques rest must be both thoroughly understood and observed in their interpretation. The fact that coefficients have been calculated by a com- puter at lightning speed is of little value if they are not carefully interpreted.

When we turn to the problems of analyzing data, we find two principal dangers that we need to guard against: oversimplification and over- sophistication. The dangers of oversimplifica- tion are obvious. Take a fact like this: Cancer deaths increased from about 12,000 in 1900 to

82,000 in 1’360, an increase of 70,000 over tt 60- year period. This is a perfectly true statement; but misleading; for analysis will show us that 20,000 of the 1960 total were due to population growth, and another 15,000 to aging of the popu- lation, leaving about 35,000 residual, part of which must be attributed to better reporting, improved diagnosis and other similar factors. In other words, the true increase in cancer deaths from 1900 to 1960 is somewhat less than half the simple increase in numbers.

On the other hand, it is possible to go too far in analyzing data and end up with meaningless hairsplitting. For instance, in evaluating an experimental educational program, it might be possible to find a statistically significant increase of one-tenth of a grade level in the reading ability of the experimental students. This may be a significant difference, in terms of statistical mea- surement, but it is not likely to be a meaningful difference, in terms of the educational worth or cost of the program. This type of ineffective, and even misleading, over-analysis of data occurs more often than it should in attempts to evaluate experiments in education, public health, and other social problem areas, We need to remember that a statistically significant difference is not always a meaningful difference in terms of the problem being studied or the purpose for which it is being studied. And we should not allow the scientific fascination of measuring minute dif- ferences precisely to seduce us from the major aims of our work.

Finally, when we come, to the last stage of a study-the drawing of conclusions. We en- counter what is perhaps the most potent of all the myths about research: the belief that sta- tistical association establishes a causal relation- ship. It is, of course, the power of scientific investigation to probe the mysteries of causality that gives research, not only its prestige in the public eye, but its own basic raison d’etre. Yet we cannot afford to forget that, causality is not something that can be ‘Lprovedll by scientific methods. Causal relationships are a matter of judgment that goes beyond statistical association.

What confronts us here is the familiar logical trap of the irreversible proposition : a causal relationship of the type producing differences measured in research studies does imply correla- tion; but a correlation derived from differences measured in research studies does not necessarily imply a causal relationship. In the first place, a statistical association is not necessarily a relevant association. It is not difficult to think of irrele-

The Journal of School Health-September, 1070 337

Vant associations. One might, for instance, be able to establish that 99 percent of heroin addicts drank milk in early childhood. This would yield a very high correlation between childhood milk- drinking and later-life heroin addiction, but it would certainly not be correct to infer that raising a child on milk would cause him to become a heroin addict.

Yet even in cases where a statistical association does possess relevance, such an association does not suffice to establish the existence of a causal relationship. The determination of causality re- mains a matter for the researcher’s judgment. It is a judgment that should be made in the light of all the available evidence and with reference to several criteria. To make a judgment about causality on the basis of a statistical association, we need to consider the consistency, strength, specificity, and logic of the association. The pre- ponderance of the evidence, when judged on these criteria, determines where it is reasonable to assume that a causal relationship cannot be rejected.

So far I have been reviewing some of the myths that may interfere with the effectiveness of re- search. In conclusion, I would like to suggest a few things which help contribute to a good re- search study. The first essential of a good study is to have a good problem. A good problem is an important problem. If we are not studying im- portant problems, our research will have little importance regardless of the sophistication of the design and treatment of data. Our business in all branches of the social sciences is to study the problems that are of importance to our society, and we should not retreat before the challenge of major social questions because we cannot see a neat experimental design for studying them.

A second point of importance is to select the type of research design best suited to the problem under study. We need a design that will provide the maximum statistical precision and efficiency for the problem we are studying. But although we should be ambitious in the type of problems we undertake, we should not pin unrealistically high hopes on the results of our studies. All studies are in some way going to be less than we would hope for, and we should not be disappointed with data that yield only partial results.

When we find ourselves with limited data, we should try not to overgeneralize from them and remember to qualify our findings wherever neces- sary. We need to admit and call attention to the difficulties in the study and avoid the use of

elaborate statistical analyses that draw attention from basic faults.

Last, we must be prepared to take on the next follow-up project. Whatever the public may think, science is not religion, and research has no absolute answers. There is never a last word in research, only a latest word. We should not try to answer all the questions a t once. We need to proceed by specific steps, defining and studying the problem that can be handled now, building upon the results and moving on to new questions and new problems, constantly improving our knowledge in a systematic way that avoids the myths in research.

* * * * *

Nurses who have been dreaming of world travels now have a chance to make their dreams come true. Experienced nurses are needed overseas to work with American children. For the first time in history the Department of Defense will employ nurses for schools operated in foreign countries for children of the Depart- ment’s military and civilian personnel.

The Department is recruiting nurses for the educational institutions being operated in 192 different locations around the world for more than 160,000 American boys and girls of school age. Nurses who are interested in world travel will find the job openings offer an opportunity to see Western Europe, Turkey, Morocco, Ethiopia, Japan, Tiawan, Oltinawa, Korea, Mid- way, the Azores, Bahamas, Bermuda, or some other foreign country.

Nurses will be offered a starting salary of $6,630 per school year plus fringe benefits, will be furnished transportation, will be provided free housing or a housing allowance, and may receive additional compensation for service in some areas.

Individuals must hold a baccalaureate degree in nursing from an accredited college or uni- versity, possess current registration to practice professional nursing, and hold school nurse certification or its equivalent. In addition, nurses are required to have at least two years of successful full-time nursing experience, includ- ing one year in a controlled, supervised nursing environment.

Handling applications are all State Employ- ment Service offices or Directorate for Depend- ents’ Education, Department of Defense, R.oom 1A-658, The Pentagon, Washington, D. C. 20301.

* * * * *

338 The Journal of Scliool Health