Vol. 71, No. 2, pp. 137-148.©2005 Council for Exceptional Children.

Exceptional Children

Research in Special Education:Scientific Methods andEvidence-Based Practices

SAMUEL L. ODOMIndiana University

ELLEN BRANTLINGERIndiana University

RUSSELL GERSTENInstructional Research Group

ROBERT H. HORNERUniversity of Oregon

BRUCE THOMPSONTexas A &M University

KAREN R. HARRISVanderhilt University

ABSTRACT:r: This article sets the context for the developrhent of research quality indicators and

guidelines for evidence of effective practices provided by different methodologies. The current con-

ceptualization of scientific research in education and the complexity of conducting research in spe-

cial education settings underlie the development of quality indicators. Programs of research in

special education may be viewed as occurring in stages: moving fiom initial descriptive research, to

experimental causal research, to finally research that examines the processes that might affect wide-

scale adoption and use of a practice. At each stage, different research questions are relevant, and

different research methodologies to address the research questions are needed.

Should science guide practice in to identify and disseminate practices that havespecial education? Most indi- scientific evidence of effectiveness. In education,viduals would say "Yes." How- national policies such as the No Child Left Be-ever, the "devil is in the hind Act (NCLB) require that teachers use scien-details." Major initiatives in tifically proven practices in their classrooms. Yet,

other disciplines such as medicine, the allied there is concern about the quality of scientific re-health professions, and psychology are attempting search in the field of education and disagreement

Exceptional Children 137

about the type of scientific information that is ac-ceptable as evidence (White & Smith, 2002). Anoft-cited report from the National ResearchCouncil (NRC) states that science in educationconsists of different types of questions and thatdifferent methodologies are needed to addressthese questions (Shavelson & Towne, 2002). Incontrast, other agencies and research synthesis or-ganizations (e.g., the What Works Clearinghouse[WWC]) have focused primarily on the questionof whether a practice is effective and proposedthat the "gold standard" for addressing this ques-tion is a single type of research methodology—randomized experimental group designs (alsocalled randomized clinical trials or RCTs; WWC,2003b).

In January 2003, the Council for Excep-tional Children's (CEC) Division for Research es-tablished a task force to address these devilishdetails as they apply to special education. The op-erating assumptions of this task force were thatdifferent types of research questions are importantfor building and documenting the effectiveness ofpractices, and that different types of methodolo-gies are essential in order to address these ques-tions. The task force identified four types ofresearch methodologies in special education: (a)experimental group, (b) correlational, (c) singlesubject, and (d) qualitative designs. The task forcewas to establish quality indicators for eachmethodology and to propose how evidence fromeach methodology could be used to identify andunderstand effective practices in special educa-tion. The subsequent four articles in this issue ofExceptional Children will describe the quality indi-cators and provide guidelines for how eachmethodology contributes evidence for the effec-tiveness of practices in special education.

This article provides a context and rationalefor this endeavor. We begin with a discussion ofthe importance of multiple scientific methodolo-gies in special education research. Next, we exam-

There is concern about the quality of scien-tific research in the field of education anddisagreement about the type of scientificinformation that is acceptable as evidence.

ine efforts to identify high-quality researchmethodology and then examine initiatives in thefields of medicine and education to identify evi-dence-based practice. In conclusion, we proposethat research and development on effective prac-tices in special education exists on a continuum,with each methodology matched to questionsarising from different points of the continuum.Also, it is important to acknowledge that al-though basic research serves as the foundation forthe development of effective practices and is criti-cally important for our work in special education,the issues addressed in this article will be most rel-evant for applied research.

R A T I O N A L E FOR M U L T I P L E

S C I E N T I F I C R E S E A R C H

M E T H O D O L O G I E S IN S P E C I A L

E D U C A T I O N

The rationale for having different researchmethodologies in special education is based onthe current conceptualization of research in edu-cation and the complexity of special education asa field. The history and tradition of special educa-tion research, when employing multiple method-ologies, has resulted in the identification ofeffective practices.

CURRENT CONCEPTUALIZATION OF

RESEARCH IN EDUCATION

A primary emphasis in education policy today isto improve the quality of education for all ofAmerica's children. This policy, exemplified byNCLB, compels educators to use "teaching prac-tices that have been proven to work" (U.S. De-partment of Education, 2003). However, ageneral concern has been voiced about the qualityof research in education (Levin & O'Donnell,1999; Mosteller & Boruch, 2002). To address thisconcern, the National Academy of Sciences(NAS) created a committee to examine the statusof scientific research in education. An operatingassumption of this committee was that researchquestions must guide researchers' selections of sci-entific methods. The NAS committee proposedthat most research questions in education couldbe grouped into three types (Shavelson &C Towne,2002, p. 99): (a) description (what is happen-ing?); (b) cause (is there a systematic effect?); and

1 3 8 Winter 2005

(c) process or mechanism (why or how is it hap-pening?). The committee conveyed two impor-tant points about these types of research and theirassociated questions. First, each type of questionis scientific. Second, the different types of ques-tions require different types of methodologies. Itfollows that each type of methodology that em-pirically, rigorously, and appropriately addressesthese questions is also legitimately scientific. Sci-entists and social philosophers as diverse as B. F.Skinner (1972), John Dewey (1938), and J.Habermas (1971) have emphasized that the ap-propriate match between question and methodol-ogy is an essential feature of scientific research.

COMPLEXITY OF SPECIAL EDUCATION AS A

FIELD

In his commentary on the NAS report on scien-tific research in education and the policy empha-sizing use of RCTs implied by NCLB, Berliner(2002) noted that such a conceptualization of sci-ence is based on hard sciences, such as physics,chemistry, and biology. He proposed that sciencein education is not a hard science but it is the"hardest-to-do science." Berliner stated.

We [educational researchers] do our scienceunder conditions that physical scientists find in-tolerable. We face particular problems and mustdeal with local conditions that limit generaliza-tions and theory building—problems that aredifferent from those faced by the easier-to-dosciences [chemistry, biology, medicine], (p. 18)

Special education research, because of itscomplexity, may be the hardest of the hardest-to-do science. One feature of special education re-search that makes it more complex is thevariability of the participants. The Individualswith Disabilities Education Act (IDEA) identifies12 eligibility (or disability) categories in specialeducation (Office of Special Education and Reha-bilitation Services [OSERS], 1997), and withinthese categories are several different identifiableconditions. For example, in addition to "typical"learning disabilities, attention deficit/hyperactivedisorder is often subsumed undet the SpecificLearning Disabilities category. Autism is nowwidely conceptualized as a spectrum consisting offour disorders. Mental retardation varies on therange of severity. Emotional and behavioral disor-

Special education research, because of itscomplexity, may be the hardest of thehardest-to-do science. One feature of spe-cial education research that makes itmore complex is the variability of theparticipants.

ders consist of externalizing and internalizing dis-orders. Visual and hearing impairments range inseverity from mildly impaired to totally blind orprofoundly deaf. Physical impairment can be ex-hibited as hypotonia or hypertonia. Other healthimpaired may incorporate health conditions asdistinct and diverse as asthma, epilepsy, and dia-betes. Adding to this variability is the greater eth-nic and linguistic diversity that, unfortunately,occurs in special education because of overrepre-sentation of some minority groups (Donovan &Cross, 2002).

A second dimension of complexity is theeducational context. Special education extendsbeyond the traditional conceptualization of"schooling" for typical students. Certainly manystudents with disabilities attend general educationclasses. However, the continuum of special educa-tion contexts is broader than general education.At one end of the chronological continuum, in-fants, toddlers, and many preschoolers receive ser-vices in their home or in an inclusive child caresetting outside of the public school settings (e.g..Head Start Centers). For school-age students withdisabilities, placement sometimes occurs in specialeducation classes or a combination of special edu-cation and general education classes. For adoles-cents and young adults with disabilities, specialeducation may take place in community living orvocational settings in preparation for the transi-tion out of high school and into the workplace.

Complexity in special education has severalimplications for research. Researchers cannot justaddress a simple question about whether a prac-tice in special education is effective; they mustspecify clearly for whom the practice is effectiveand in what context (Guralnick, 1999). The het-erogeneity of participant characteristics poses asignificant challenge to research designs based onestablishing equivalent groups, even when ran-

Exceptional Children 1 3 9

special education extends beyond the tra-ditional conceptualization of "schooling"for typical students.

domization and stratification is possible. Certaindisabilities have a low prevalence, so methodolo-gies that require a relatively large number of par-ticipants to build the power of the analysis maybe very difficult or not feasible. In addition, be-cause IDEA ensures the right to a free appropriatepublic education, some research and policy ques-tions (e.g.. Are IEPs effective in promoting stu-dent progress?) may not be addressable throughresearch methodologies that require random as-signment to a "nontreatment" group or condi-tion. Last, in special education, students withdisabilities are often "clustered" in classrooms,and in experimental group design, the classroomrather than the student becomes the unit onwhich researchers base random assignment, dataanalysis, and power estimates (see Gersten et al.,2004).

HISTORY OF SPECIAL EDUCATION

RESEARCH

Special education research has a long history inwhich different methodologies have been em-ployed. In the early 19th century beginning withItard's (1962) foundational work. The Wild Boy ofAveyron, there was a tradition of discovery, devel-opment, experimentation, and verification. Ini-tially, the research methods employed in the fieldthat was to become special education researchwere derived from medicine. Many of the earlypioneers in services for individuals with disabili-ties (Itard, Seguin, Montessori, Fernald, Gold-stein) were physicians. Similarly, early services forindividuals with disabilities occurred in residentialfacilities and training schools, which were basedon the medical tradition of care (Scheerenberger,1983).

As psychology, sociology, and anthropologybecame academic disciplines, they providedmethodological tools for research in special edu-cation. For example, Skeels's (Skeels & Dye,1939) and Kirk's (1958) works, respectively, onearly experiences and preschool education for in-fants and young children with mental retardation

employed experimental and quasi-experimentalgroup designs prominent in psychology. Edger-ton's (1967) research on individuals with mentalretardation who left institutions and moved to thecommunity, drew from methods in sociology andanthropology. In academic instructional studies,Lovitt and Haring based their methodology onthe then newly created single-subject designmethodology of the time (see Lovitt, 1976). Far-ber's (1960) important early work on femilies ofchildren with disabilities and continuing throughwork by Blacher (2001) and Dunst (2000) had itsroots in family sociology. Many of the currentspecial education research tools now frequentlyemployed, such as sophisticated multivariate de-signs, qualitative research designs, and programevaluation designs, have their roots in general ed-ucation and educational psychology. Today arange of methodological approaches are availableto researchers in special education (Martella, Nel-son, & Marchand-Martella, 1999) as a result ofthis rich history.

MORE THAN ONE RESEARCH

METHODOLOGY IS IMPORTANT IN

SPECIAL EDUCATION RESEARCH

A current initiative of the U.S. Department of Ed-ucation is to improve the quality of research in thefield of education (Whitehurst, 2003), with therationale that improved research will lead to im-proved practice. A major effort to improve qualityhas come through the establishment in 2003 ofthe Institute of Education Sciences (IES), whosemission is to expand fundamental knowledgeabout education (Institute of Education Sciences,2004). A central theme advocated by IES is tofocus research on the questions of effectivenessand to employ high-quality research methods toaddress these questions (Whitehurst). The goldstandard for research methodology that addressesthese issues is the use of RCT methodology(Mosteller & Boruch, 2002; WWC, 2003b). TheIES acknowledges that different methodologies areimportant for addressing different questions.

The increased use of RCT methodology,when conducted well, will undoubtedly enhancethe quality of research in education and specialeducation. Rigorously conducted RCT studieshave greater capacity to control for threats to in-ternal validity than do quasi-experimental designs

Winter 2005

that are often used in special education. Becauseof this greater experimental control, Gersten et al.(2004) propose that random assignment to exper-imental groups is one indicator of high-qualitygroup design research. The IES and Departmentof Education policy of encouraging RCT studiesmay well move the field closer to the goal of iden-tifying evidence-based special education practices.But again, there are devilish details that challengethe near exclusive use of this methodology for in-vestigating effective practices in special education.

In special education, other methodologies,such as single-subject designs, are experimentaland may be a better fit for some research contextsand participant characteristics (see Horner et al.,2004). Powerful correlational methodologies maysuggest causal relationships by statistically con-trolling for competing hypotheses and may be es-sential for addressing causal-like questions whenresearchers are not able to conduct experimentalgroup or single-subject design studies (seeThompson, Diamond, McWilliam, Snyder, &Snyder, 2004). The discovery and development ofnew effective practices may require researchers towork in naturalistic contexts where they may notbe able to exert experimental control and/or indesign experiments (Cobb, Confrey, diSessa,Lehrer, & Schauble, 2003), or where they havethe flexibility of changing certain elements of anintervention based on students' responses. Suchdescriptive and process-oriented research may re-quire the use of qualitative methods (Brantlinger,Jimenez, Klingner, Pugach, & Richardson, 2004).Educational researchers have acknowledged thevalue of mixing methodologies to provide a com-plementary set of information that would moreeffectively (than a single method) inform practice(Greene, Caracelli & Graham, 1989; Li, Mar-quart, & Zercher, 2000).

QUALITY INDICATORS OP

RESEARCH METHODOLOGY

Quality indicators are the feature of research thatrepresents rigorous application of methodology toquestions of interest. They may serve as guidelinesfor (a) researchers who design and conduct re-search, (b) reviewers who evaluate the "believabil-ity" of research fmdings, and (c) consumers whoneed to determine the "usability" of research find-

Researchers cannot just address a simple• question about whether a practice in spe-cial education is effective; they must spec-

! ify clearly for whom the practice iseffective and in what context.

ings. High-quality research is designed to rule outalternative explanations for both the results of thestudy and the conclusions that researchers draw.The higher the quality of research methodology,the more confidence the researcher and readerswill have in the findings of the study.

Textbooks on educational research describethe methodology that investigators should follow,but they usually do not provide a succinct or un-derstandable set of indicators that are useful forindividuals who lack graduate training on re-search methodology. Several professional organi-zations have developed standards for describingand, in some cases, evaluating research. Division16 of the American Psychological Association(APA) and the Society for the Study of SchoolPsychology have established criteria for evaluatinggroup design, single-subject design, and qualita-tive methodology used in research on practices inschool psychology (Kratochwill & Stoiber, inpress). Similarly, APA Division 12 Task Force onPsychological Interventions has established crite-ria primarily for experimental group designs stud-ies used to provide support for therapies inclinical psychology (Chambless & Hollon, 1998)and clinical child psychology (Lonigan, Elbert, &Johnson, 1998). The CEC Division for EarlyChildhood (DEC) created procedures for describ-ing research methodology for studies using group,single-subject, and qualitative research methodol-ogy (Smith et al., 2002), which they used to de-termine recommended practices for earlyintervention/early childhood special education.These standards have been used to determine thequality of research methods employed (Odom &Strain, 2002; Snyder, Thompson, McLean, &Smith, 2002); however, they have not been pub-lished as quality indicators that other researcherscould use.

Exceptional Children

Educational researchers have acknowl-edged the value of mixing methodologiesto provide a complementary set of infor-mation that would more effectively (thana single method) inform practice.

In their work on summarizing evidence foreffective practices, which will be described in thenext section, some research synthesis organiza-tions have established evaluation criteria andmethods for determining the quality of research.For example, the WWC created an evaluation in-strument, called the Design and ImplementationAssessment Device (DIAD), with which a ratercould conduct an extremely detailed evaluation ofa research article. At this writing, a DIAD hasonly been created for experimental and quasi-ex-perimental group design, but the WWC reportsthat DIADs are also being constructed for single-subject and qualitative group designs (WWC,2003b). Other research synthesis organizations,such as the Campbell Collaboration (2003) andthe Evidence for Policy and Practice InformationCentre (EPPIC; 2003), have somewhat similar re-search evaluation procedures.

Efforts described here illustrate the progressthat professional and governmental organizationshave made toward establishing standards for qual-ity in research. To date, however, such quality in-dicators have not been identified specifically forresearch in special education. As noted, the pur-pose of the work conducted by this task force wasto establish a set of quality indicators that wereclearly stated, understandable, and readily avail-able for use as guides for identifying high-qualityresearch in special education. These quality indi-cators are presented in the articles appearing inthis special issue.

EVIDENCE-BASED PRACTICE

The type and magnitude of evidence needed toverify a practice as evidence-based is a prominentissue in the discussion of scientific research andeffective educational practices. These devilish de-tails are critical for policymakers, practitioners,educational researchers, and consumers. Currentendeavors to establish standards for evidence-

based practice as well as to identify the evidence-based practices themselves, are occurring throughtwo different, but related, initiatives. In this sec-tion, we describe briefly the history of identifyingeffective practices first in medicine and then othersocial science fields, efforts by professional organi-zations to identify effective practices, and similarefforts being conducted by research synthesis or-ganizations.

IDENTIFYING EVIDENCE-BASED PRACTICE

Like the evolution of special education researchmethods noted previously, the search for evi-dence-based practice originated in the field ofmedicine. Although the practice of evidence-based medicine extends back to the mid-19thcentury, the modern era of evidence-based prac-tice emerged in the early 1970s and 1980s (Ben-nett et al., 1987) and came into fruition in GreatBritain in the early 1990s (Sackett, Rosenberg,Gray, Haynes, & Richardson, 1996). Cutspec(2003) tracked the evolution of evidence-basedmedicine from a movement that began with theintent to address the gap between research andpractitioners' provision of medical care, moved tothe use of the literature to inform practice deci-sions, and then became an approach to practicingmedicine. Evidence-based practice is now a cen-tral part of medical education (Grad, Macaulay,& Warner, 2001), education in allied health pro-fessions such as nursing (Newell, 2002), andcounselor education (Sexton, 2000).

General and special education have fol-lowed suit in adopting scientific evidence as theappropriate basis for selecting teaching practices(Carnine, 1999; Davies, 1999; Oakley, 2002).The impetus for the current evidence-basedmovement in education is similar to that inmedicine: A concern that effective educationalpractices, as proven by research, are not beingused in schools. This current concern reflects along-standing discussion in the field of special ed-ucation regarding the distance between researchand practice (Gersten & Smith-Jones, 2001;Greenwood & Abbott, 2001). In response, a largenumber of initiatives have been established toidentify practices that will generate positive out-comes for children (Dunst, Trivette, & Cutspec,2002). Two types of groups are sponsoring theseinitiatives: research synthesis organizations and

1 4 2 Winter 2005

professional associations that propose standardsfor practice.

Research Synthesis Organizations. Researchsynthesis organizations systematically evaluate anda^regate findings from the research literature inorder to inform practitioners. Perhaps the largestand longest standing synthesis organization is theCochrane Collaboration (www.cochrane.org), lo-cated in Great Britain and founded in 1963. Thisorganization, which focuses on medical andhealth research, consists of over 50 collaborativereview groups and has completed over 1,300 re-views. Following this model, the Campbell Col-laboration (http://www.campbellcollaboration.org/Fra/Vbout.html) was established in the UnitedStates in 1999 to assist individuals in educationand the social sciences to make informed deci-sions about what works based on high-quality re-search and reviews. In Great Britain, EPPIC atthe University of London Institute of Education(http://www.researchtopractice.info/) was createdin 1993 to conduct systematic reviews of researchon social interventions. This organization was re-cently funded to conduct reviews specifically oneducational practices, which it plans to makeavailable through their Research Evidence in Edu-cation Library (http://eppi.i6e.ac.uk/EPPIWeb/home.aspx?page=/reel/intro.htm).

In the United States, the IES has estab-lished the WWC (http://w-w-c.org/about.html),which is jointly managed by the Campbell Col-laboration and the American Institutes for Re-search. WWC conducts reviews of educationalpractices supported by high-quality research andmakes this information available to practitionersthrough Web-based databases. The U.S. Depart-ment of Education funds the Research and Train-ing Center on Early Childhood Development(CED; www.puckett.org), which is conducting aset of practice-sensitive research syntheses on theeffectiveness (and ineffectiveness) of practices forinfants and young children with disabilities andtheir families. Whereas other organizational ef-forts primarily provide evidence that a practice iseffective, the CED has created a more functionaloperational definition by stating that evidence-based practices are "informed by research, inwhich the characteristics and consequences of en-vironmental variables are empirically establishedand the relationship directly informs what a prac-

titioner can do to produce the desired outcome(Dunstetal., 2002, p. 3).

To examine the effectiveness of programsfor children with autism, a committee formed byNAS established guidelines for the strength of evi-dence provided by individual studies (Committeeon Educational Interventions for Children withAutism, 2001). The dimensions of the studiesevaluated were internal validity, external validity,and generalization, with strength of evidence (i.e.,from I to IV) evaluated for each.

Textbooks on educational research de-scribe the methodology that investigatorsshould follow, but they usually do notprovide a succinct or understandable setof indicators that are usefiilfor individu-als who lack graduate training on re-search methodology.

A key feature in these research synthesis ini-tiatives is the methodological criteria establishedto select or exclude research studies for the syn-thesis. Most organizations confined evidence ofeffectiveness to research studies that have em-ployed RCT methodology or rigorously con-structed quasi-experimental designs. The CEDresearchers took a broader view of the empiricallinkage between a practice and an outcome andlooked for descriptions of the process of the inter-vention practices that led to the outcome. Theleadership of the WWC has noted that qualitativeresearch may provide information about the waysin which interventions work and can be used tosubstantiate "promising practices" in education,although they proposed that clearly efficaciouspractices would require verification through RCTs(WWC, 2003b). For the EPPIC, Oakley (2002)reported that they incorporated qualitative re-search in their reviews, but they had encounteredmultiple problems in their evaluation of qualita-tive studies.

Professional Associations. Professional associ-ations and groups have also examined the litera-ture to determine effective practices. These groupshave often established the level of evidence

Exceptional Children 1 4 3

General and special education have fol-lowed suit in adopting scientific evidenceas the appropriate basis for selectingteaching practices.

needed to identify a practice as efFective. For ex-ample, the Child-Clinical Section of Division 12of the APA established the Task Force on Empiri-cally Supported Psychosocial Interventions forChildren (Lonigan et aJ., 1998). They proposedthe types and amount of evidence needed to iden-tify a practice as (a) well-established (i.e., twowell-conducted group design studies by differentresearchers or nine well-conducted single-subjectdesigns); or (b) "probably efficacious" (i.e., twogroup design studies by same investigator or atleast three single-subject design studies).

The Division for Early Childhood of C E Cestablished a process for identifying recom-miended practices that incorporated evidencefrom the research literature (Smith et al., 2002).Mentioned previously, DEC conducted an exten-sive literature review to identify support for rec-ommended practices and also to incorporateinformation from focus groups of experts, practi-tioners, and family members in the final identifi-cation of practices. The level or type of evidenceneeded to support a recommended practice wasnot identified.

The American Speech-Language-HearingAssociation (ASHA; 2004) proposed that differ-ent types of evidence may be important for differ-ent clinical activities. For questions of treatmentefficacy, they propose that different frameworksare available for evaluating the level of evidencethat documents efficacy. They provide as an ex-ample one such framework developed by the Ox-ford Centre for Evidence-based Medicine (2001).This system could be used to classify practices ac-cording to four levels of evidence:• Level I evidence derives from meta-analyses in-

cluding at least one randomized experimentaldesign or well-designed randomized controlstudies.

• Level II evidence includes controlled studieswithout randomization and quasi-experimentaldesigns.

• Level III consists of well designed nonexperi-mental studies (i.e., correlational and case stud-ies).

• Level IV includes expert committee report,consensus conference, and clinical experience ofrespected authorities.

The ASHA policy emphasizes that other frame-works are currently available or evolving andcould be useful for specific questions related totreatment efficacy.

To date, the special education communityhas yet to develop systematic guidelines for speci-fying the types and levels of evidence needed toidentify a practice as evidence-based and effective.The Division for Learning Disabilities (DLD)and the Division for Research (DR) jointly pub-lished a document entitled Alerts, in which an ex-pert from the field reviews the literature relevantto a specific practice and describes the evidence orlack of evidence that underlies the practice. Thesealerts, however, have been based on individual au-thors' reviews of the literature and, although quiteuseful, different authors may well be using differ-ent criteria for the evidence they include. The sec-ond goal of the current DR Task Force, therefore,was to describe the types of results generated byeach research methodology and to recommendguidelines for using the results as evidence of ef-fectiveness, or lack of effectiveness, of practices inspecial education.

WHERE DO WE GO FROM HERE7

The Department of Education is under pressure toprove to Congress that there are educational prac-tices that have evidence of effectiveness and thatsupporting educational research is a good invest-ment of public funds. In specifying RCT method-ology as the gold standard for research, theDepartment of Education is investing the bulk ofresearch funding in addressing the question of ef-fectiveness, which is clearly important. However,Berliner (2002) urges us to avoid confusing sci-

Professional associations and groups havealso examined the literature to determineeffective practices.

Winter 2005

To date, the special education communityhas yet to develop systematic guidelines forspecifying the types and levels of evidenceneeded to identify a practice as evidence-based and effective.

ence with a specific method or technique. It ismore impottant to look at the broader goal ofusing science to improve education for all chil-dren.

To accomplish such a goal, educational sci-ence might be more appropriately seen as a con-tinuum rather than a fixed point. Levin,O'Donnell, and Kratochwill (2003) suggest that aprogram of educational research might bethought of as occurring in four stages, (see Figure1). The first stage would involve observational.

F I G U R E 1

Stages of Programs of Research

focused exploration and flexible methodology,which qualitative and correlational methodsallow. Stage 2 would involve controlled laboratoryor classroom experiments, observational studies ofclassrooms, and teacher-researcher collaborativeexperiments. Design experimentation involvingqualitative methodology, single-subject designs,quasi-experimental and/or RCT design could beuseful at that stage. Stage 3 research would thenincorporate knowledge generated from these pre-vious stages to design well-documented interven-tions and "prove" their effectiveness through well-controlled RCT studies implemented in class-room or naturalistic settings by the natural partic-ipants (e.g;, teachers) in the settings. We proposethat single-subject design studies could also ac-complish this purpose.

If research ended here, however, the move-ment of effective, evidence-based research intopractices that teachers use on Monday morning

- •

Stage V.

PreUmiiiaiy ideas, liypatheses, obsenotiDns, and pilol work

y/

Controned laboratoiy• experiments

—

Stage 2 I

\Ciassraom-based

demonstrations and designexperiments

Stage 3:

Randomized classroom trials studies

Stage 4:

btformed dassroom practice -—

From "Educational/psychological Intervention Research," by J. R. Levin, A. M. O'Donnell, & T. R. Kra-tochwill, 2003, in Handbook of Psychology, Vol 7: Educational Psychology (pp. 557-581), edited by W.Reynolds and G. Miller. Copyright 2003 by Wiley Publishers. Reprinted with permission.

Exceptional Children 1 4 5

would likely fail. A further step in the develop-ment process (Stage 4) would be to determine thefactors that lead to adoption of effective practicesin typical school systems under naturally existingconditions. That last step would require researchinto organizational factors that facilitate or im-pede adoption of innovation in local contexts(Fullan, 2001). The research methodologies thatwould generate this information are more likelyqualitative, correlational, and mixed methods, aswell as RCT and large-scale, single-case designs.Researchers may well draw from such disciplinesas sociology, political science, economics, as wellas education, in this research. Research at thisstage may best occur though a partnership amongresearchers from education, researchers fromother disciplines, local education agencies, andteachers. Indeed, an initiative is emerging fromanother panel convened by the NRC (Donovan,Wigdor, & Snow, 2003), which is proposing abroad federal initiative that would create just sucha partnership.

CONCLUSION

If different methodologies are appropriate for ad-dressing important questions in special education,then we, as a field, need to be clear about (a) thematch between research questions and methodol-ogy, (b), the features of each methodology thatrepresent high quality, (c) and the use of researchfindings for each methodology as scientific evi-dence for effective practices in special education.To date, we have numerous texts and papers thatdescribe each methodology, but they are not a co-ordinated, clear index of how each contributes tothe present research-to-practice challenge. Thenext four articles in this special issue identifyquality indicators of research in special educationand propose the use of research fmdings as evi-dence for practice. In the first article, Gersten etal. (2004) examine quality indicators for experi-mental and group experimental design and pro-pose guidelines for using the results of groupstudies for evidence of effective practices. Horneret al. (2004) propose quality indicators and stan-dards for evidence-based practice for single-sub-ject design in the second article. Qualityindicators for correlational design and ways inwhich results from correlational studies may con-

tribute evidence of effective practices appear inthe third article by Thompson and colleagues(2004). In the fourth article, Brantlinger and col-leagues propose quality indicators for qualitativedesign and uses of qualitative research as evidencefor effective practices in special education.

REFERENCES

American Speech-Language-Hearing Association.(2004). Evidence-based practice in communication disor-

ders: In communication disorders: An introduction (Tech-nical Report). Rockville, MD: Author.

Bennett, K. J., Sackett, D. L., Haynes, R. B., Neufeld,V. R., Tugwell, P., & Roberts, R. (1987). A controlledtrial of teaching critical appraisal of the clinical litera-ture to medical students. Journal of the American Medi-cal Association, 257, 2451-2454.

Berliner, D. C. (2002). Educational research: The hard-est science of all. Educational Researchers, 3J(S), 18-20.

Blacher, J. (2001). Transition to adulthood: Mental re-tardation, families, and culture. American Journal onMental Retardation, 106, 173-188.

Brantlinger, E., Jimenez, R., Klingner, J., Pugach, M.,& Richardson, V. (2004). Qualitative studies in specialeducation. Exceptional Children, 71, 195-207.

Campbell Collaboration. (2003). Campbell systematicreviews. Retrieved November 6, 2003, fromhttp://www.campbellcollaboration.org/Fraguidelines.html.

Carnine, D. (1999). Perspective: Campaigns for mov-ing research into practice. Remedial and Special Educa-tion, 20(1), 2-6.

Chambless, D. L, & HoUon, S. D. (1998). Definingempirically supported therapies. Journal of Consultingand Clinical Psychology, 66, 7-18.

Cobb, P., Confrey, J., diSessa, A., Lehrer, R., &Schauble, L. (2003). Design experiments in educa-tional research. Educational Researchers, 32(\), 9-13.

Committee on Educational Interventions for Childrenwith Autism. (2001). Educating children with autism.Washington, DC: National Academy Press.

Cutspec, P. A. (2003). Evidence-based medicine: The firstevidence-based approach to best practice. Manuscript inprogress. Information retrieved September 20, 2004,from http://www.researchtopractice.info/products.php#centerscope.Davies, P. (1999). What is evidence-based education?British Journal of Educational Studies, 47, 108-121.

Dewey, J. (1938). Logic: The theory of inquiry. NewYork: H. Holt.

Winter 2005

Donovan, M. S., & Cross, C. T. (Eds.). (2002). Minor-ity students in special education and gifted education.Washington DC: National Academy Press.

Donovan, M. S., Wigdor, A. K., & Snow, C. E. (Eds.).(2003). Strategic educational research partnership. Wash-ington, DC: National Academy Press.

Dunst, C. J. (2000). Revisiting "rethinking early inter-vention." Topics in Early Childhood Special Education,

20, 95-104.

Dunst, C. J., Trivette, C. M., & Cutspec, P A. (2002).Toward an operational definition of evidence-basedpractice. Centerscope: Evidence-based approaches in earlychildhood development, / ( I ) . Retrieved August 19,2003, from http://www.evidencebasedpractices.org/centerscope/centerscopevol 1 no 1 .pdf

Edgerton, R. B. (1967). The cloak of competence: Stigmain the lives of the mentally retarded. Berkeley, CA: Uni-versity of California Press.

Evidence for Policy and Practice Information Centre.(2003). EPPI-Centre introduction. London: Author. Re-trieved November 6, 2003, from ht tp: / /eppi.ioe.ac.uk/EPPIWeb/home.aspx.

Farber, B. (1960). Family organization and crisis: Main-tenance and integration in families with a severely men-tally retarded child. Lafayette, IN: Child DevelopmentPublications of the Society of Research in Child Devel-opment.

Fullan, M. (2001). The new meaning of educationalchange (3rd ed.). New York: Teachers College Press.

Gersten, R., Fuchs, L. S., Compton, D., Coyne, M.,Greenwood, C , & Innocenti, M. (2004). Quality indi-cators for group experimental and quasi-experimentalresearch in special education. Exceptional Children, 71,149-164.

Gersten, R., & Smith-Jones, J. (2001). Reflections onthe research to practice gap. Teacher Education and Spe-cial Education, 24, 356-361.

Crad, R., Macaulay, A. C , & Warner, M. (2001).Teaching evidence-based medical care: Description andevaluation. Eamily Medicine, 33, 602-606.

Greene, J. C , Caracelli, V. J., & Graham, W. F (1989).Toward a conceptual framework for mixed-methodevaluation designs. Educational Evaluation and PolicyAnalysis, 11(3), 255-274.

Greenwood, C. R., & Abbott, M. (2001). The researchto practice gap in special education. Teacher Educationand Special Education, 24, 276-289.

Guralnick, M. J. (1999). Second-generation research inthe field of early intervention. In M. Guralnick (Ed.),The effectiveness of early intervention (pp. 3-22). Balti-more, MD: Paul Brookes.

Habermas, J. (1971). Knowledge and human interests.Boston: Beacon Press.

Horner, R., Carr, E., Halle, J., McGee, G., Odom, S.,& Wolery, M. (2004). The use of single-subject designresearch to identify evidence-based practice in specialeducation. Exceptional Children, 71, 165-179.

Institute of Education Sciences. (2004). Director's in-troduction. Washington, DC: Author. RetrievedSeptember 20, 2004, from http://www.ed.gov/about/offices/list/ies/director.html.

Itard, J. M. (1962). The wild boy of Aveyron. NewYork: Appleton-Century-Crofts.

Kirk, S. A. (1958). Early education of the mentally re-tarded: An experimental study. Urbana, IL: University ofIllinois Press.

Kratochwill, T. R., & Stoiber, K. C. (in press). Procedu-ral and coding manual for review of evidence-based inter-ventions. Published by the Task Force onEvidence-Based Interventions in School Psychology.Retrieved November 1, 2004 from http://www.sp.-ebi.org/workingfiles/EBI7.pdf

Levin, J. R., & O'Donnell, A. M. (1999). What to doabout education's credibility gaps? Issues in Education:Contributions from Educational Psychology, 5, 177-229.

Levin, J. R., O'Donnell, A. M., & Kratochwill, T. R.(2003). Educational/psychological intervention re-search. In W. Reynolds & C. Miller (Eds.), Handbookof Psychology, Vol. 7: Educational Psychology (pp. 557-581). Hoboken, NJ: John Wiley & Sons.

Li, S., Marquart, J. M., & Zercher, C. (2000). Concep-tual issues and analytic strategies in mixed-methodstudies of preschool inclusion. Journal of Early Interven-tion, 23, 116-133.

Lonigan, C. J., Elbert, J. C , & Johnson, S. B. (1998).Empirically supported psychosocial interventions forchildren: An overview. Journal of Clinical and ChildPsychology, 27, 138-145.

Lovitt, T. C. (1976). Applied behavior analysis tech-niques and curriculum research: Implications for in-struction. In N. G. Haring & R. L. Schiefelbusch(Eds.), Teaching special children (pp. 112-156). NewYork: McGraw-Hill.

Martella, R. C , Nelson, R., & Marchand-Martella, N.E. (1999). Research methods: Learning to become a criti-cal consumer Boston: Allyn & Bacon.

Mosteller, E, & Boruch, R. (Eds.). (2002). Evidencematters: Randomized trials in education research. Wash-ington, DC: Brooldngs Institute.

Newell, R. (2002). Research and its relationship tonurse education: Focus and capacity. Nurse EducationToday, 22, 278-284.

Exceptional Children 1 4 7

Oakley, A. (2002). Social science and evidence-basedeverything: The case of education. Educational Review,54, 277-286.

Odom, S. L., & Strain, P. S. (2002). Evidence-basedpractice in early intervention/early childhood specialeducation: Single-subject design research. Journal ofEarly Intervention, 25, 151-160.

Office of Special Education and Rehabilitation Ser-vices. (1997). Individuals With Disabilities EducationAct, 1997 Reauthorization. Washington DC: Author.Retrieved October 22, 2003, from http://wwvi'.ed.gov/ofFices/OSERS/Policy/IDEA/the_law.html.

Oxford Centre for Evidenced-based Medicine. (2001).Levels of evidence and grades for recommendations. Ox-ford, UK: Author. Retrieved September 21, 2004, fromhttp://vv'ViTv.cebm.net/levels of evidence.asp.

Sackett, D. L., Rosenberg, W. M., Gray, J. A., Haynes,R. B., & Richardson, W. S. (1996). Evidence-basedmedicine: What it is and what it isn't. British MedicalJournal, 312, 71-72.

Scheerenberger, R. C. (1983). A history of mental retar-dation. Baltimore: Paul Brookes.

Sexton, T. L. (2000). Reconstructing clinical training inpursuit of evidence-based clinical training. CounselorEducation and Supervision, 39, 218-227.

Shavelson, R. J., & Towne, L. (Eds.). (2002). Scientijicresearch in education. Washington, DC: NationalAcademy Press.

Skeels, H. M., & Dye, H. B. (1939). A study of the ef-fects of differential stimulation on mentally retardedchildren. Proceeding and Address of the American Associ-ation on Mental Deficiency, 44, 114-136.

Skinner, B. E (1972). Cumulative record: A selection ofpapers, 3rd ed. New York: Appleton-Century-Crofts.

Smith, B. J., Strain, P. S., Snyder, P, Sandall, S. R.,McLean, M. E., Boudy-Ramsey, A., et al. (2002). DECrecommended practices: A review of 9 years ofEI/ECSE research literature. Journal of Early Interven-tion, 25, 108-119.

Snyder, P., Thompson, B., McLean, M. E., & Smith, B.J. (2002). Examination of quantitative methods used inearly intervention research: Linkages with recom-mended practices. Journal of Early Intervention, 25,137-150.

Thompson, B., Diamond, K., McWilliam, R., Snyder,P., & Snyder, S. (2004). Evaluating the quality of evi-dence from correlational research for evidence-basedpractice. Exceptional Children., 71, 181-194.

U.S. Department of Education. (2003). Proven meth-ods: Questions and answers on No Child Left Behind.Washington, DC: Author. Retrieved October 29, 2003,

from http://www.ed.gov/nclb/methods/whatworks/doing.html.

What Works Clearinghouse. (2003b). Standards.Washington, DC: Author. Retrieved November 6,2003, from http://www.w-w-c.org/standards.html.

White, E., & Smith, C. S. (Eds.). (2002). Theme issueon Scientific Research in Education. Educational Re-searcher, 31(9,), 3-29.

Whitehurst, G. J. (2003). The Institute of EducationSciences: New wine, new bottles. Paper presented at theannual conference of the American Educational Re-search Association, Los Angeles. Retrieved September20, 2004, from http//www.ed.gov/rschstat/research/pubs/ies.html.

ABOUT THE AUTHORS

SAMUEU u. ODOM (CEC #407), Otting Profes-sor of Special Education, School of Edtication;and ELLEN BRANTLINGER (CEC #407), PrO-fessor Emeritus, School of Education, IndianaUniversity, Bloomington. RUSSELL. GERSTEN(CEC #108), Director, Instructional ResearchGroup, Signal Hill, California. ROBERT H.

HORNER (CEC OR Federation), Professor, De-partment of Special Education, University of Ore-gon, Eugene, B R U C E THOMPSON (CEC T XFederation), Professor and Distinguished ResearchFellow, Department of Educational Psychology,Texas A & M University, College Station.KAREN R. HARRIS (CEC #263), Professor andCorey-Ingram Chair, Department of Special Edu-cation, Vanderbilt University, Nashville, Ten-nessee.

Correspondence may be addressed to Samuel L.Odom, School of Education, Indiana University,Bloomington, IN 47405-1006; e-mail:slo-dom@indiana.edu.

The authors wish to thank the CEC Division forResearch for sponsoring the Quality IndicatorTask Force and its work.

Manuscript received December 2003; acceptedApril 2004.

Winter 2005