Evidence-based Education: It Isn’t as Simple as You Might Think Ronnie Detrich Randy Keyworth Jack States Wing Institute

Evidence-based Education: It Isn’t as Simple as You Might Think

Ronnie Detrich

Randy Keyworth

Jack States

Wing Institute

Everybody’s Talking

• In the last few years there has been a great deal of discussion about evidence-based education.Term has become nearly ubiquitous.

• Sources of InfluenceNo Child Left Behind (NCLB). Individuals with Disabilities Education Improvement

Act (IDEIA).Several initiatives within APA

Divisions 12, 16, 53.

What Did They Say?

• NCLB requires that interventions used to improve educational performance are based on scientific research. In NCLB there are over 100 references to scientific

research.

• IDEIA (2004) requires that interventions are scientifically based instructional practices.

Special Education and Evidence-based Education

• Pre-service and professional development for all who work with students with disabilities to ensure such personnel have the skills and knowledge necessary to improve the academic achievement and functional performance of children with disabilities, including the use of scientifically based instructional practices, to the maximum extent possible.


• Scientifically based early reading programs, positive behavioral interventions and supports, and early intervention services to reduce the need to label children as disabled in order to address the learning and behavioral needs of such children.


• The Individualized Education Program (IEP) shall include a statement of the special education and related services and supplementary aids and services, based on peer-reviewed research to the extent practicable, to be provided to the child, or on behalf of the child, and a statement of the program modifications or supports for school personnel that will be provided for the child.

Ethical Conduct and Evidence-based Interventions

• Most national psychological and educational organizations have ethical standards requiring science-based practices to address problems.American Psychological Association Ethical Standard

2.04: Psychologists’ work is based on the established scientific and

professional knowledge of the discipline.

Ethical Conduct and Evidence-based Education

• National Association of School Psychologists Standard III F 4.

School psychology faculty members and clinical or field supervisors uphold recognized standards of the profession by providing training related to high quality, responsible, and research-based school psychology services.


• National Association of School Psychologists Standard III F 4.

School psychology faculty members and clinical or field supervisors uphold recognized standards of the profession by providing training related to high quality, responsible, and research-based school psychology services.

Standard IV 4.School psychologists use assessment techniques, counseling

and therapy procedures, consultation techniques, and other direct and indirect service methods that the profession considers to be responsible, research-based practice.


• Behavior Analysis Certification Board Standard 2.09a

The behavior analyst always has the responsibility to recommend scientifically supported, most effective treatment procedures. Effective treatment procedures have been validated as having both long-term and short-term benefits to clients and society.

Standard 2.09bClients have a right to effective treatment (i.e., based on the

research literature and adapted to the individual client).

Scope of the Problem

• Kazdin (2000) identified 550 named interventions for children and adolescents.A very small number of these interventions have been

empirically evaluated.Of those that have been evaluated, the large majority of them are

behavioral or cognitive-behavioral.Evidence-based interventions are less likely to be used

than interventions for which there is no evidence or there is evidence about lack of impact (Kazdin, 2000).

In many instances practitioners are not aware of evidence-based interventions (Kratochwill, Albers & Shernoff, 2004).

Scope of the Problem

• Situation may be worse for autism interventions.Long (2006) reported a Google® search on autism

cure:Feb., 2005 528,000 hitsFeb., 2006 5,290,000 hits.

Science does not move fast enough to evaluate all of these new interventions.

Some of these interventions may be helpful, some may have no impact, some may be harmful.We do not know.We cannot afford to implement interventions with unknown

effects.

What is Evidence-based Practice?

• Sackett et al (2000) defined evidence-based practice in medicine as: “the integration of best research evidence with clinical expertise, and patient values.”

• EBP is a decision-making approach that places emphasis on evidence to: guide decisions about which interventions to use. evaluate the effects of an intervention.

What is Evidence-based Practice?

• Ultimately, EBP is a consumer protection issue.Assumes that evidence-based interventions are more

likely to be effective than interventions that are not evidence-based.

By validating interventions as evidence-based there is the implication that there are standards for reviewing interventions.Standards should be transparent.

• EBP is more than identifying evidence-based interventions.

Evidence-based Practice

Identifying Evidence-based Interventions

• Controversies to identifying evidence-based interventions.There is no consensus about what constitutes evidence.

NCLB permits both quantitative and qualitative evidence without specifying the types of questions that each approach best answers.

In this context, we are most often concerned with evidence that establishes a causal relation between an intervention and a class of social or academic behaviors.

Identifying Evidence-based Interventions?

• Even if we accept causal demonstrations to be evidence, we have no consensus.Randomized Clinical Trials have become the “gold

standard.”There is controversy about the status of single

participant designs.Most frequently criticized on the basis of external validity.


• Identification is more than finding a study to support an intervention. Identification involves an evaluation of the body of

knowledge about an intervention.Standards of evidence specify:

the quantitythe quality of evidence necessary to validate an intervention as

evidence-based.There are no agreed upon standards.

No single resource for decision makers.May result in other criteria than evidence to influence

decisions.


• Two approaches to establishing standardsThreshold approach:

Evidence must be of a specific quantity and quality before intervention is considered evidence-based.

Hierarchy of evidence approach:Best available evidence approach. Strength of evidence falls

along a continuum with each level having differential standards.


• Several organizations have established standards but there is limited agreement among them.What Works ClearinghouseNational Autism Center (proposed)Society for Prevention ResearchTask Force on Evidence-based Interventions in School

PsychologyCouncil for Exceptional Children

It is possible for an intervention to meet one standard but not a second.

Identification of Evidence-based Interventions

• Types of Evidence:Efficacy trials: Intervention demonstrated to be

effective when implemented under optimum conditions.Primarily designed to demonstrate impact of an independent

variable.

Effectiveness trials: Interventions implemented under more typical practice conditions.Often interventions effects compromised when taken to scale.

Implementing Evidence-based InterventionsWhere Good Interventions Go to Die

• Identification of evidence-based interventions is necessary but not sufficient to assure that interventions will be effective.

• Non-science based issues influence selection of intervention:Personal experienceExpert opinionCost impacts adoption

Less effective but cheaper interventions may be adopted.Effort

Contextual fit.Complexity of intervention.Training.

Implementing Evidence-based Interventions Contextual Fit

• Logically it would seem to make sense to always implement intervention that produces greatest impact.There may be exceptions:

If high impact intervention requires great resources, specialized training, and is very different from current practices it may not be implemented with integrity.

May be better to implement effective but lower impact intervention that is better contextual fit and will be implemented with greater integrity.

Implementing Evidence-based InterventionsContextual fit

• Adoption or AccommodationAdoption: Implementing intervention as evaluated to be

effective.Assures evidence-based intervention is being implemented.

Accommodation: adjusting intervention to meet local circumstances.May result in intervention no longer being evidence-based.May increase implementation with integrity.

Implementing Evidence-based Interventions Complexity of Intervention

• Level of precision may increase complexityBe as precise as necessary but no more.

Catch’em being goodGood behavior gameIndividualized intervention plan

Implementing Evidence-based InterventionsTraining

• Implementing a new intervention requires trainingPresents significant financial and logistical challenges.

How is everyone to be trained?Who will provide the training?How will acquisition be assured?How will maintenance be assured?How will new implementers be trained after initial training?

Implementing Evidence-based InterventionsDissemination

• Issue for dissemination is getting information into hands of practice-based decision makers. Journals are poor means for dissemination.

Not typically read by decision makersDecision makers not necessarily qualified to interpret resultsOften lack sufficient detail for replication in a practice setting.

Clearinghouses are in their infancy.Often do not have the information a decision maker is seeking.

Implementing Evidence-based Interventions

• Often the evidence for a particular set of problems is inadequate for identifying an evidence-based intervention.What becomes the basis for decision making?

An alternative to evidence-based:Evidence-based: meets well defined standards.Empirically supported: based on scientific principles of

behavior.When no evidence-based intervention then build intervention

with exclusively with principles from science of human behavior.

» Not as strong as evidence-based.

Implementing Evidence-based InterventionsSustainability

• If intervention is effective then it is desirable to assure sustainability of the program. Defining features of sustainable programs:

Maintains over time. Maintains across generations of practitioners. Supported with existing resources of system.

There is an emerging science of sustainability.

• The larger the scale of implementation the greater the degree of complexity. The science of large scale implementation is not well

established.

Evaluating Evidence-based InterventionsProgress Monitoring

• Implementation of evidence-based intervention does not assure success.Necessary to evaluate impact in local context.

No intervention will be effective for all students.Cannot predict who will benefit.

Progress monitoring is practice-based evidence about evidence-based practices.

Consistent with legal requirements and ethical standards.

Ethical Standards and Progress Monitoring

• National Association of School PsychologistsStandard IV C 1b.

Decision-making related to assessment and subsequent interventions is primarily data-based.

Standard IV 6.School psychologists develop interventions that are

appropriate to the presenting problems and are consistent with the data collected. They modify or terminate the treatment plan when the data indicate the plan is not achieving the desired goals.

Ethical Standards and Progress Monitoring

• Behavior Analysis Certification Board Standard 4.04

The behavior analyst collects data or asks the client, client-surrogate, or designated other to collect data needed to assess progress within the program.

Standard 4.05The behavior analyst modifies the program on the basis of

data.

Legal Requirements for Progress Monitoring

• Fundamental to IEP process.Must report on same schedule that grades are reported

in general education.

• Response to Intervention is accepted as alternative means for determining eligibility for Learning Disability classification.Progress monitoring is the heart of RTI.

All students routinely and systematically monitored to assure adequate progress is occurring.

Evaluating Evidence-based Interventions

• Progress monitoring is a systems level intervention.Contingencies must be in place to assure:

Data are collectedData are reviewedDecisions are based on the data.

If contingencies are not in place, response effort associated with data collection will compromise data-based decision making.


• Curriculum based measurement is a powerful means for evaluating impact of academic interventions.Scores on CBM correlated with scores on high stakes

test.Can be used to predict how students will perform on state-

wide tests.


• Monitoring effects of intervention of students is necessary but it is also necessary to systematically assess accuracy of implementation (treatment integrity). Implementation without integrity results in an

unspecified intervention being implemented for which there is no evidence-base to justify the intervention.

If student is not benefiting from intervention and there is poor integrity then no conclusions can be drawn about the effect of the intervention.If integrity is low, increase integrity before evaluating impact.

Where Are We?

• The legal requirements to implement evidence-based interventions requires system level changes.Systems requirements:

Process for selecting interventions.Process for training and maintenance.Process for progress monitoring for both treatment integrity

and student performance.These changes to the system will not occur rapidly. Behavioral systems perspective can facilitate change process.

Gilbert (1996) Human Competence: Engineering Worthy Performance

The Effects of Being Evidence-based

• Personal preference for interventions is de-emphasized.Evidence guides decisions at all levels.

• Training programs are not necessarily teaching evidence-based interventions (Kratochwill, Albers & Shernoff, 2004).To realize the long term benefits of an evidence-based

practice approach it will be necessary to impact the content of university level training programs.

Documents

Evidence-based Education: It Isn’t as Simple as You Might Think Ronnie Detrich Randy Keyworth Jack States Wing Institute