Upload
others
View
8
Download
0
Embed Size (px)
Citation preview
A+ Evidence: How to Tell if Your Evidence-Based Intervention Meets
ESSA Requirements
@EducateIN@GinaGRomano
@Education_AIR@eisenbartht
Gina Romano | Senior Data Coach Specialist | IN Department of EducationTrent Eisenbarth | TA Consultant | American Institutes for Research Patti Furlano | Senior TA Consultant | American Institutes for Research
Title Con 2020 | April 16th, 2020
@EducateIN @Education_AIR @GinaGRomano @eisenbartht
• Introductions• Background/Importance• Process• Applying the Process
@EducateIN @Education_AIR @GinaGRomano @eisenbartht
@EducateIN @Education_AIR @GinaGRomano @eisenbartht
Actions that have been proven effective through rigorous outcome evaluations (i.e., quantitative-focused studies). If implemented with fidelity, the intervention is expected
to positively change outcomes.
@EducateIN @Education_AIR @GinaGRomano @eisenbartht
@EducateIN @Education_AIR @GinaGRomano @eisenbartht
@EducateIN @Education_AIR @GinaGRomano @eisenbartht
@EducateIN @Education_AIR @GinaGRomano @eisenbartht
● The evidentiary threshold can’t be met with a couple of citations○ Provide copies of the studies in your application
● Quantitative studies are required○ Randomized control trials, quasi-experimental designs, etc.○ This does not mean that qualitative studies (e.g., case studies,
conceptual papers) aren’t important, they just are not included in determinations of EBI thresholds for ESSA
@EducateIN @Education_AIR @GinaGRomano @eisenbartht
You can also access the rubric here: https://tinyurl.com/EBI-Rubric
We are going to dissect this to help with your understanding
about EBI’s
https://www.doe.in.gov/sites/default/files/news/june-14-ebi-clarification-sig-imp-grant-review-rubric.pdf?utm_content=&utm_medium=email&utm_name=&utm_source=govdelivery&utm_term=https://www.doe.in.gov/sites/default/files/news/june-14-ebi-clarification-sig-imp-grant-review-rubric.pdf?utm_content=&utm_medium=email&utm_name=&utm_source=govdelivery&utm_term=
@EducateIN @Education_AIR @GinaGRomano @eisenbartht
@EducateIN @Education_AIR @GinaGRomano @eisenbartht
Using the WWC to Identify
ESSA Evidence Based
Interventions
https://nam01.safelinks.protection.outlook.com/?url=https://www.youtube.com/watch?v%3Dhu4XnpyiKxw&data=02|01|[email protected]|9aae27ebc5ba4a8b348008d79ebafa31|9ea45dbc7b724abfa77cc770a0a8b962|0|0|637152398292157445&sdata=qUJkTXzI0%2B6RFiAf61C80y0%2Br11pDgMv2OHPtyw/S/M%3D&reserved=0
@EducateIN @Education_AIR @GinaGRomano @eisenbartht
1. Select Meets WWC standards with or w/out reservations
2. Select study design3. Select topic area 4. Select ESSA Tier 1 & Tier 2
5. Results are
displayed here
@EducateIN @Education_AIR @GinaGRomano @eisenbartht
Tiers of Evidence
@EducateIN @Education_AIR @GinaGRomano @eisenbartht
@EducateIN @Education_AIR @GinaGRomano @eisenbartht
● Research-based vs. evidence-based○ Research-based: theories behind it (abstract)○ Evidence-based: empirical proof (concrete)
■ The intervention was compared to something■ Outcomes measured with valid and reliable instruments■ Description of how program was implemented: replicability■ Effect sizes reported
● No meta-analyses or reviews of research● Quality over quantity
@EducateIN @Education_AIR @GinaGRomano @eisenbartht
● Understandably, the most difficult part! ● Some good places to start are located in the EBI Resources
document, which can be found in the following folder: https://tinyurl.com/EdDataResources
● Abstract/General vs. Concrete/Specific● Include the original study
https://tinyurl.com/EdDataResources
@EducateIN @Education_AIR @GinaGRomano @eisenbartht
@EducateIN @Education_AIR @GinaGRomano @eisenbartht
You can typically find well-designed and well-implemented studies from these places:● An entity from the National Center for Educational Evaluation and Regional Assistance
(NCEE), including the What Works Clearinghouse (WWC) the Regional Educational Laboratories (REL), or Educational Resources Information Center (ERIC)
● A peer-reviewed academic journal (e.g. American Educational Research Journal )● A report published by a reputable organization focused on education research and
evaluation (e.g., AIR, WestEd, Abt Associates, Evidence for ESSA, Best Evidence Encyclopedia, Results First Clearinghouse, RAND Report on School Leadership Interventions)
● These are all linked in the EBI Resources document, which can be found in the following folder: https://tinyurl.com/EdDataResources
https://tinyurl.com/EdDataResources
@EducateIN @Education_AIR @GinaGRomano @eisenbartht
● Randomized control trials where participants were randomly assigned
into a treatment and control groups
● Some sort of intervention used to change outcomes
Strong
Evidence
Moderate
Evidence
Promising
Evidence
Demonstrates
a Rationale
● Typically quasi-experimental designs (QED), where participants were not
randomly assigned into treatment and comparison groups
● Some natural change created a group that received the treatment/intervention,
and one that did not
● Includes correlational studies, with statistical controls for selection bias.
● Will not include treatment and control/comparison groups
● Researchers examine relationships among specific variables and the
outcomes
● Not an acceptable level of evidence, though they might have a strong logic
behind the intervention, or some promising preliminary data
@EducateIN @Education_AIR @GinaGRomano @eisenbartht
● Statistical significance○ Something is unlikely to happen by chance○ In other words, the intervention did something to change the outcomes
that can’t be attributed to business as usual
● Positive effect○ It’s a positive change in the outcome, not necessarily a “+” sign○ In other words, was the change large enough?
@EducateIN @Education_AIR @GinaGRomano @eisenbartht
● Outcomes measured in the study should be relevant to the setting and student population that you serve ○ State the extent to which the participants in the study are similar to
those who would participate in the intervention○ It helps if this study was conducted at multiple classrooms, schools or
districts● Is the study setting similar to your LEA (grade level, urban/rural/suburban)?● Is the student population (FRL EL, SpED, race/ethnicity) in the study similar
to your LEA?● The outcome(s) measured should also be reliable (can be replicated) and
have face validity (measures what it claims to be measuring)
@EducateIN @Education_AIR @GinaGRomano @eisenbartht
● Once you are done, make sure you cite the source and include the document● Please use APA format
@EducateIN @Education_AIR @GinaGRomano @eisenbartht
@EducateIN @Education_AIR @GinaGRomano @eisenbartht
Stage 1: Intentional Data Review (IDR)
1. Comprehensive Needs Assessment (CNA)
1. Root Cause Analysis Process (RCA)
@EducateIN @Education_AIR @GinaGRomano @eisenbartht
Stage 2: Research & Identify EBI’s
Search WWC to identify and choose an evidence-based intervention (EBI)
@EducateIN @Education_AIR @GinaGRomano @eisenbartht
1. Review WWC and IDOE Resources for possible evidence-based interventions that meet Tiers 1 and 2.
https://ies.ed.gov/ncee/wwc/ReviewedStudieshttps://www.doe.in.gov/school-improvement/resources-tsi-schools
@EducateIN @Education_AIR @GinaGRomano @eisenbartht
2. Create a chart of possible EBIs and sources.
@EducateIN @Education_AIR @GinaGRomano @eisenbartht
3. Assign homework to research identified EBIs.
@EducateIN @Education_AIR @GinaGRomano @eisenbartht
4. Share-out EBI research summaries with team for selection process.
5. SIP Development: SMART Goals, Drivers, Actions, and Measures
@EducateIN @Education_AIR @GinaGRomano @eisenbartht
▪ What Works Clearinghouse
▪ WWC Standards Handbook and Procedures Handbook
▪ What Works Clearinghouse Practice Guides
▪ Using the WWC to Find ESSA Tiers of Evidence
▪ WWC Reviews of Individual Studies
▪ Using the WWC to Identify ESSA Evidence Based Interventions Video
▪ ERIC: online library of education research and information, sponsored by the IES
▪ REL Midwest: Putting Research into Action
https://ies.ed.gov/ncee/wwc/FWWhttps://ies.ed.gov/ncee/wwc/multimedia/2054https://ies.ed.gov/ncee/wwc/PracticeGuidesabout:blankhttps://ies.ed.gov/ncee/wwc/ReviewedStudies#/OnlyStudiesWithPositiveEffects:false,SetNumber:1https://nam01.safelinks.protection.outlook.com/?url=https://www.youtube.com/watch?v%3Dhu4XnpyiKxw&data=02|01|[email protected]|9aae27ebc5ba4a8b348008d79ebafa31|9ea45dbc7b724abfa77cc770a0a8b962|0|0|637152398292157445&sdata=qUJkTXzI0%2B6RFiAf61C80y0%2Br11pDgMv2OHPtyw/S/M%3D&reserved=0https://eric.ed.gov/https://ies.ed.gov/ncee/edlabs/regions/midwest/default.aspx
Gina Romano | Senior Data Coach Specialist | [email protected] | Twitter: @GinaGRomano
Trent Eisenbarth | TA Consultant | American Institutes for [email protected] | Twitter: @eisenbartht
Patti Furlano | Senior TA Consultant |American Institutes for Research | [email protected]
mailto:[email protected]:[email protected]
Structure BookmarksFigureFigureFigureFigureFigureFigureFigureFigureFigureFigureFigureFigureFigureFigureFigureFigureFigureFigureFigureFigureFigureFigureFigureFigureFigureFigureFigureFigureFigureFigureFigureFigureFigureFigureFigureFigureFigureFigureFigureFigureFigureFigureFigureFigureFigureFigureFigureFigureFigureFigureFigureFigureFigureFigureFigureFigureFigureFigureFigureFigureFigureFigureFigureFigureFigureFigureFigureFigureFigureFigureFigureFigureFigureFigureFigureFigureFigureFigureFigureFigureFigureFigureFigureFigureFigureFigureFigureFigureFigureFigureFigureFigureFigureFigureFigureFigureFigureFigureFigureFigureFigureFigureFigureFigureFigureFigureFigure