The Ethical and Legal Basis for Evidence- based Education: Implications for the Profession Ronnie Detrich Wing Institute TED Conference November, 2008

The Ethical and Legal Basis for Evidence-based Education: Implications for the

Profession

Ronnie Detrich

Wing InstituteTED Conference

November, 2008

Dallas, Texas

Overview

• Discuss the recent legal and regulatory requirements to base educational interventions on scientific research.

• Describe ethical requirements to rely on scientific knowledge.

• Describe what it means to be evidence-basedDefinitions IssuesControversies

• Appraisal of current status and future directions.

The Legal Basis

• No Child Left Behind (NCLB): Interventions used to improve educational performance are based on scientific research.Over 100 references to scientific research in

NCLB.

• Individuals with Disabilities Education Improvement Act [IDEIA] (2004): Interventions are scientifically based instructional practices.

The Legal Basis

• Specific requirements of IDEIA include: Pre-service and professional development to

improve the academic achievement and functional performance of children with disabilities, including the use of scientifically based instructional practices, to the maximum extent possible.

The Legal Basis

• Scientifically based early reading programs, positive behavioral interventions and supports, and early intervention services to reduce the need to label children as disabled in order to address the learning and behavioral needs of such children.

The Legal Basis

• The Individualized Education Program (IEP) services based on peer-reviewed research to the extent practicable.

The Legal Basis

• In determining if a child has a specific learning disability, a local education agency may use a process that determines if a child responds to a scientific, research-based intervention as part of the evaluation procedures.

The Ethical Basis

• Most national psychological and educational organizations have ethical standards requiring science-based practices to address problems.American Psychological Association Ethical

Standard 2.04: Psychologists’ work is based on the established scientific

and professional knowledge of the discipline.

The Ethical Basis

• National Association of School Psychologists Standard III F 4.

School psychology faculty members and clinical or field supervisors uphold recognized standards of the profession by providing training related to high quality, responsible, and research-based school psychology services.

The Ethical Basis

• National Association of School Psychologists Standard IV 4.

School psychologists use assessment techniques, counseling and therapy procedures, consultation techniques, and other direct and indirect service methods that the profession considers to be responsible, research-based practice.

The Ethical Basis

• Behavior Analyst Certification Board Standard 2.09a

The behavior analyst always has the responsibility to recommend scientifically supported, most effective treatment procedures. Effective treatment procedures have been validated as having both long-term and short-term benefits to clients and society.

Standard 2.09bClients have a right to effective treatment (i.e., based on

the research literature and adapted to the individual client).

Another Ethical Responsibility

• Education services are largely funded through public dollars (taxpayers dollars).

• There is an implicit assumption that the money will be spent for the public good.

• A fiduciary responsibility exists when one person or organization is charged with managing another person’s money.

Another Ethical Responsibility

Act solely for the benefit of the other party

Fiduciary

Carries the weight of ethical

conduct

Assure that the taxpayers are receiving the

greatest possible return on their

investment.

How Do We Meet Our Fiduciary Responsibility?

• Interventions that have an evidence base are more likely to produce positive effects for students.Does not assure positive outcomes but increases

the probability.

• The impact of a non-evidence based intervention is unknown.

How Do We Meet our Fiduciary Responsibility?

• Implications:Using a non-evidence-based intervention when

there are evidence-based interventions available constitutes unethical practice.

Use of a non-evidence based intervention should be considered research. All of the safe-guards afforded research participants and

their families should be in place.

Conducting research with tax-dollars provided for education services may constitute a violation of our fiduciary responsibility.

Becoming Evidence-based

• Clearly, the intent of Congress, the U.S. Department of Education, and Office of Special Education Programs is to rely on interventions that have a scientific basis.

• Professional organizations place great value on scientific knowledge.

• What does it mean to be evidence-based?

What is Evidence-based Practice?

• At its core the EBP movement is a consumer protection movement. It is not about science per se. It is a policy to use science for the benefit of

consumers. “The ultimate goal of the ‘evidence-based

movement’ is to make better use of research findings in typical service settings, to benefit consumers and society….” (Fixsen, 2008)

What is Evidence-based Practice?

• Evidence-based practice has its roots in medicine.Movement has spread across major disciplines in

human services:PsychologySchool PsychologySocial WorkSpeech PathologyOccupational Therapy

What Is Evidence-based Practice?

• EBP is a decision-making approach that places emphasis on evidence to:guide decisions about which interventions to use;evaluate the effects of an intervention.

Professional Judgment

Best available evidence

Client Values

Sackett et al (2000)Professional Professional

JudgmentJudgmentBest Available EvidenceBest Available EvidenceClient Client

ValuesValues

IdentifIdentifyy

ImplementImplementEvaluateEvaluate

Evidence-Evidence-basedbasedInterventioInterventionn

IdentifyIdentify

ImplementImplementEvaluatEvaluatee

Phases of Evidence-based Intervention

What is Evidence-based Education?

• The term “evidence-based” has become ubiquitous in last decade.There is no consensus about what it means.

At issue is what counts as evidence.Federal definition emphasizes experimental

methods.Preference for randomized trials.

Definition has been criticized as being positivistic.

IdentifyIdentify

What Counts as Evidence?

• Ultimately, this depends on the question being asked.Qualitative methods are best for some questions.

• In EBP the goal is to identify causal relations between interventions and outcomes.Experimental methods do this best.

IdentifyIdentify

What Counts as Evidence?

• Even if we accept causal demonstrations to be evidence, we have no consensus.Randomized Clinical Trials (RCT) have become

the “gold standard.”There is controversy about the status of single

subject designs.Most frequently criticized on the basis of external validity.

IdentifyIdentify

How Are Evidence-based Interventions Identified?

• Identification is more than finding a study to support an intervention.

• Identification involves distilling a body of knowledge to determine the strength of evidence.

IdentifyIdentify


• Distillation requires standards of evidence for reviewing the literature.Standards specify:

the quantity of evidencethe quality of evidence

IdentifyIdentify

Continua of EvidenceQuality of the

Evidence

Personal Observation

Expert Opinion

Current “Gold Standard”High Quality

Randomized Controlled Trial

Uncontrolled Studies

General Consensus

Single Case Designs

Semi-Randomized Trials

Well-conducted Clinical Studies

Quantity of the Evidence

Janet Twyman, 2007

Meta-analysis (systematic

review)

Single Case Replication (Direct

and Parametric)

Single Study

Various Investigations

Repeated Systematic Measures

Convergent Evidence

Threshold of Evidence

IdentifyIdentify


• Two approaches to validating interventionsThreshold approach:

Evidence must be of a specific quantity and quality before an intervention is considered evidence-based.

Hierarchy of evidence approach:Strength of evidence falls along a continuum with each

level having differential standards.

IdentifyIdentify


• There are no agreed upon standards. It is possible for an intervention to be evidence-

based using one set of standards and to fail to meet evidence standards using an alternative set.Difficult for consumers and decision makers to sort out

the competing claims about what is evidence-based.

IdentifyIdentify

IdentifyIdentify

Evidence-based InterventionEvidence-based InterventionEvidence-based InterventionEvidence-based Intervention

IdentifyIdentify

Most likely with threshold

approach

Most likely with hierarchy

approach

Effective IneffectiveE

ffec

tive

Inef

fect

ive

TruePositive

TrueNegative

FalsePositive

FalseNegativeA

sses

sed

Ass

esse

d E

ffect

iven

ess

Effe

ctiv

enes

sActual EffectivenessActual Effectiveness

Effective IneffectiveIn

effe

ctiv

eE

ffect

ive

IdentifyIdentify

Choosing Between False Positives and False Negatives

• At this stage, it is better to have more false positives than false negatives.

False Negatives:

Effective interventions will not be selected for implementation.

As a consequence, less likely to determine that they are actually effective.

False Positives: Progress monitoring will identify interventions that are not effective.

IdentifyIdentify

Why Do We Need Evidence-based Education?

550 named interventions for children and adolescents

BehavioralCognitive-behavioral

Empirically evaluated

Evidence-based interventions are less likely to be used than interventions for which there is no evidence or there is evidence about lack of impact.

Kazdin (2000)

Are We Training Educators to be Evidence-based?

Survey of School Psychology Directors of Training

29%

Evidence-based interventions

Knowledge

(Shernoff, Kratochwill, & Stoiber, 2003)

training41% programs

0

5

10

15

20

25

30

35

40

0 1 2 3 4 5

What Are Teachers Being Taught About Science of Reading?

Percent

# of Reading Elements TaughtNational Council on Teacher Quality, 2006

Implementing Evidence-based Interventions

Where Good Interventions Go to Die

• Intent of both legal and ethical guidelines is to have positive impact.Evidence-based interventions are assumed to give

to give us that chance.

• Identification is necessary but not sufficient to assure that intervention will be effective.

• Must address complex issues associated with implementation.

ImplementImplement


• Implementation is where the research to practice gap is most evident.

• Many innovations in education have very short life spans.18 mos. (Latham)Most often a result of poor implementation.

ImplementImplement

0

20

40

60

80

100

2000 2002 2004

80% of initiatives ended within 2 years

90% of initiatives ended within 4 years

Data from Center for Comprehensive School Reform

Implementation is Fundamental

Well Tested Programs Often Fail Large Scale Implementation

• Elliott & Mihalic (2004) review Blueprint Model Programs (violence prevention and drug prevention programs) replication in community settings.Programs reviewed across 5 dimensions

Site selectionTrainingTechnical assistanceFidelitySustainability

ImplementImplement

Keys to Implementation

• Critical elements in site readinessWell connected local championStrong administrative supportFormal organizational commitmentsFormal organizational staffing stabilityUp front commitment of necessary resourcesProgram credibility within the communityProgram sustained by the existing operational

budget

ImplementImplement


• Critical elements of trainingAdhere to requirements for training, skills, and

education.Hire all staff before scheduling training.Encourage administrators to attend training.Plan and budget for staff turnover. Implement program immediately after training.

ImplementImplement


• Critical elements of Technical AssistanceProactive plan for technical assistance.

• Critical elements of FidelityMonitor fidelity.

• Critical elements of SustainabilityFunction of how well other dimensions are

implemented.

ImplementImplement


Dimensions of Implementation

• Contextual fit• Complexity of intervention

ImplementImplement


Contextual Fit

• Contextual Fit: the degree to which an intervention matches the culture, training, and resources of a particular setting.These characteristics of a setting can be

measured.Degree of contextual fit may moderate the impact

of an intervention.

ImplementImplement


Contextual Fit

• Adoption or AccommodationAdoption: Implementing intervention as evaluated

to be effective.Assures intervention is evidence-based.Does not assure implementation.

Accommodation: adjusting intervention to meet local circumstances.May result in intervention no longer being evidence-

based.May increase implementation with integrity.

ImplementImplement


Contextual Fit

• Logically it would seem to make sense to always implement intervention that produces greatest impact.There may be exceptions:

If high impact intervention requires great resources, specialized training, and is very different from current practices it may not be implemented with integrity.

May be better to implement effective but lower impact intervention that is better contextual fit and will be implemented with greater integrity.

ImplementImplement


Complexity

• Level of precision may increase complexity.Be as precise as necessary but no more.

Com

plex

ity

Precision

Catch’em being good

Good Behavior

Game

Individualized intervention

plan

ImplementImplement

Evaluating Evidence-based InterventionsProgress Monitoring

• Implementation of evidence-based intervention does not assure success.Necessary to evaluate impact in local context.

No intervention will be effective for all students.Cannot predict who will benefit.

Progress monitoring is practice-based evidence about evidence-based practices.

Consistent with legal requirements and ethical standards.

EvaluatEvaluatee

Ethical Standards and Progress Monitoring

• National Association of School PsychologistsStandard IV C 1b.

Decision-making related to assessment and subsequent interventions is primarily data-based.

Standard IV 6.School psychologists develop interventions that are

appropriate to the presenting problems and are consistent with the data collected. They modify or terminate the treatment plan when the data indicate the plan is not achieving the desired goals.

EvaluatEvaluatee

Ethical Standards and Progress Monitoring

• Behavior Analyst Certification Board Standard 4.04

The behavior analyst collects data or asks the client, client-surrogate, or designated other to collect data needed to assess progress within the program.

Standard 4.05The behavior analyst modifies the program on the basis

of data.

EvaluatEvaluatee

• Fundamental to IEP process.• Response to Intervention is accepted as

alternative means for determining eligibility for Learning Disability classification.Progress monitoring is the heart of RTI.

All students routinely and systematically monitored to assure adequate progress is occurring.

Legal Requirements for Progress Monitoring

EvaluatEvaluatee

Evaluating Evidence-based Interventions

• Progress monitoring is a systems level intervention.Systems must be in place to assure:

Data are collectedData are reviewedDecisions are based on the data

If systems are not in place, response effort associated with data collection will compromise data-based decision making.

EvaluatEvaluatee

Progress Monitoring as an Intervention

• Progress monitoring 2-5/week in math and reading:4 times as effective as 10% increase in per pupil

spending;6 times as effective as voucher programs;64 times as effective as charter schools;6 times as effective as increased accountability.

(Yeh, 2007)

EvaluatEvaluatee

Evidence-based Education and Treatment Integrity

• Progress monitoring allows data based decision making about effects of an intervention. It is impossible to make informed decisions without

knowing how well the intervention was implemented.

Positive NegativeH

igh

Low

Continue Intervention

Change Intervention

Unknown reason Unknown reason• Intervention problem?

• Implementation problem?

• Other life changes?

• Unknown intervention?

• Intervention is effective?

OutcomeIn

tegr

ity

Positive NegativeH

igh

Low

Where Are We?

From a university in the U.S.

Where are We?

• Being evidence-based is more than a good idea, it is the law and it is ethical conduct; however, it is not as easy as it might seem.

• Lack of consensus about evidence may do harm to consumers.

• The research to practice gap limits the impact of evidence-based education.The science of implementation and sustainability

is in its infancy.

Where are We?

• Pre-service training should change to reflect current policy.Changes in both method of training and content.

OUTCOMES(% of Participants who demonstrate knowledge, demonstrate

new skills in a training setting, and use new skills in the classroom)

TRAININGCOMPONENTS

KnowledgeSkill

DemonstrationUse in the Classroom

Theory and Discussion

10%

5% 0%

..+Demonstration in Training

30% 20% 0%

…+ Practice & Feedback in Training

60% 60% 5%

…+ Coaching in Classroom

95% 95% 95%

Joyce and Showers, 2002

Effects of Training

Treatment

EvidenceValues

Basis for Choosing TreatmentSzatmari (2004)

Do Nothing

None

Unethical

Clinical Paralysis

Do Nothing

Clinical Paralysis

Unethical

None

Toss a Coin

None

Unethical in light of evidence

Do Nothing

Clinical Paralysis

Unethical

Toss a Coin


None None

Training

NoneOutdated

Perhaps some

Current

Do Nothing

Clinical Paralysis

Unethical

None

Toss a Coin


None

Training

Outdated None

Etiology

LimitedDifficult

Do Nothing

Clinical Paralysis

Unethical

None

Toss a Coin


None

Training

Outdated

Etiology

LimitedDifficultNone

ABA

Robust

Not very humane

Effective

ABA

Not very humane

Do Nothing

Clinical Paralysis

Unethical

None

Toss a Coin


None

Training

Outdated

Etiology

LimitedDifficult

ABA

Not very humane

Robust

None

Developmental sociocognitive

NoneYet

Highly preferred

Developmental sociocognitive

Highly preferred

Noneyet

Inform Parents of Options

Do Nothing

Clinical Paralysis

Unethical

None

Toss a Coin


None

Training

Outdated

Etiology

limitedDifficult

To be Ethical:

ABA

Not very humane

Robust

None

Thank you

Copy of this presentation may be downloaded at

www.winginstitute.org

Documents

The Ethical and Legal Basis for Evidence- based Education: Implications for the Profession Ronnie Detrich Wing Institute TED Conference November, 2008