Evalua&ng  Evidence  about  Educa&onal  Programs  for  Children  with  Special  Educa&on  Needs  (EEEP)  

 AERO  Spring  Conference—June  1st,  2012  

Presenters:  Jayme  Herman  &  ChrisBe  Fraser  

AGENDA  

By  the  end  of  this  presentaBon,  you  will  have  an  understanding  of:  

 

•  the  EEEP  project  •  a  tool  for  evaluaBng  evidence  •  collaboraBon:  

• during  the  EEEP  project  •  through  the  use  of  the  EEEP  tool  

Collabora&on  and  the  EEEP  Project  

•  during  the  process:  • between  AERO  and  OISE  members  • with  aLendees  and  colleagues  

•  using  the  product:  • managing  what  we  know  •  generaBng  a  common  language  and  framework  •  tool  for  communicaBng  

Fic&onal  Scenario  

You  work  in  an  Ontario  school  board  called  Whispering  Pines  DSB.  The  board  area  contains  1  medium  to  large  city,  several  small  ci&es  and  rural  areas.  WPDS  is  a  public  board  and  has  60  elementary  and  15  secondary  schools  under  its  jurisdic&on.  Two  of  the  elementary  schools  are  located  on  reserves  and  have  only  aboriginal  students.      

Fic&onal  Scenario  con’t  

About  10  and  15%  of  students  in  the  ci&es  need  ESL  support  and  a  smaller  propor&on  in  the  rural  areas.  SES  is  typical  for  this  rural/urban  mix.  Your  superintendent  comes  to  you  very  enthusias&c  and  really  ready  to  immediately  adopt  an  interven&on  for  special  educa&on  students,  perhaps  PALS  for  junior-­‐level  mathema&cs.  The  SO  asks  what  you  think—What  does  the  evidence  say?        

Background  &  Context  

•  history  of  the  project  

•  an  OISE-­‐AERO  collaboraBon  funded  by  KNAER    •  EvaluaBng  Evidence  About  EducaBon  Programs  For  Students  With  Special  EducaBon  Needs  (EEEP)  

Goal  of  the  Project  

 The  purpose  of  this  project  is  to  increase  the  capacity  of  Ontario  school  boards  to  use  evidence  to  inform  their  decision-­‐making,  especially  about  educaBonal  programs  for  students  with  special  educaBon  needs.    

 

The  Purpose  of  Educa&on  Programs  

•  according  to  Ontario’s  Educa&on  Act  ,  “the  purpose  of  educaBon  is  to  provide  students  with  the  opportunity  to  realize  their  potenBal  and  develop  into  highly  skilled,  knowledgeable,  caring  ciBzens  who  contribute  to  their  society”  (secBon  0.1(2))    

•  one  of  the  principal  ways    a  school  board  achieves  this  purpose  is  by  “deliver[ing]  effecBve  and  appropriate  educaBon  programs  to  its  pupils”  (secBon  169.1  (1))  

Four  Principles  for  Choosing  Programs  

•  EffecBveness  •  Equity  •  Efficiency    •  Appropriateness  

Effec&veness  

What  does  it  mean  for  an  educaBonal  program  to  be  effecBve?  

•  “successful  in  producing  a  desired  or  intended  result”  (Oxford  Dic&onary  of  English,  2012)  

•  framework  provided  by  the  Educa&on  Act    •  not  only  does  the  program  meet  its  stated  objecBves  but  do  those  objecBves  also  meet  one  or  more  of  the  Ontario  goals  without  detracBng  from  others?      

Equity  

•  equity  of  programming  provides  students  with  equal  opportuniBes  to  reach  their  potenBal  development,  to  achieve,  and  to  experience  well-­‐being  

•  the  requirement  of  equity  implies  fairness,  not  necessarily  equality,  in  allocaBng  educaBonal  resources  and  in  defining  and  judging  student  outcomes      

Efficiency  

•  the  resource  requirements  of  a  program  relaBve  to  other  programs  that  might  have  similar  levels  of  effecBveness  and  equity    

•  the  Educa&on  Act  reminds  us  as  well  that  school  boards  are  responsible  for  “stewardship  of  the  board’s  resources”  

•  efficiency  can  have  implicaBons  for  equity    

The  3  E’s  and  Appropriateness  

EffecBveness  ×  Efficiency  ×  Equity  =    

Appropriateness  

The  EEEP  Tool  

PHASE  1  

FIND  the  evidence  about  the  effec&veness,  equity,  &  efficiency  of  the  program:  

 ① Define  the  scope  of  evaluaBon  –  Specify:  topic/program,  

special  populaBon,  grade  level,  context  (i.e.,  community  type,  public/private,  SES),  type  of  anBcipated  effect  

② Gather  the  evidence  –  E.g.,  published  research,  developer  informaBon,  expert  &  professional  opinion,  media  coverage,  anecdote  

③  Filter  the  evidence  –  Revisit  the  scope.  Keep  only  evidence  relevant  to  the  scope.  

 

PHASE  2  

SORT  &  ASSESS  the  evidence  about  the  effec&veness,  equity,  &  efficiency  of  the  program:  

Sample—Research  Reviews  &  Studies    Applicable  to  all  Quan&ta&ve  Designs:  1.  Measures  including  tests,  items  and  quesBons,  if  

applicable,  were  appropriate  and  sufficient  measures  of  the  construct(s).  

2.  The  analysis  is  appropriate  to  the  level  of  measurement.  

3.  The  analysis  is  described  in  such  a  way  that  the  comparisons  made  and  relaBonships  explored  are  clear.  

4.  The  summary  of  findings  accurately  reflects  the  results  of  the  staBsBcal  tests.  

5.  Effect  sizes  are  presented.    

Sample—Professional  Judgement  

Quality  Indicators  for  Professional  Opinion:  1.  There  is  evidence  of  experBse  from  the  

individual’s  degrees  or  other  formal  qualificaBons.  

2.  The  expert’s  record  shows  skill,  experience  and/or  publicaBon  in  the  field.  

3.  The  expert’s  knowledge  is  up-­‐to-­‐date.    

Sample—Media,  Journalism,  &  Anecdote  

Quality  Indicators  for  Media,  Journalism  &  Anecdote:  

1.  The  author  is  credible,  either  by  direct  experience  with  the  program  or  issue  or  by  background.  

2.  The  content  is  current,  explicit,  and  well-­‐wriLen.  3.  The  opinions  expressed  are  objecBve  and  

moderate.  4.  The  source  includes  references  to  supporBng  

evidence.    

PHASE  3  SUMMARIZE  &  SYNTHESIZE  the  evidence  about  the  effec&veness,  equity,  &  efficiency  of  the  program:  

 

•  Is  there  sufficient  and  consistent  evidence  to  inform  a  decision  about  the  program  under  examinaBon?  

•  Have  issues  of  effec&veness,  equity,  and  efficiency  been  considered?  

•  Based  on  the  evidence,  is  the  program  appropriate?  

 

EEEP  Handbook  &  Website    

•  handbook  containing  background  materials  which  support  the  EEEP  tool  

•  website:  hLp://www.aero-­‐aoce.org/eeep.html        

EEEP  as  a  Collabora&ve  Tool  

How  can  the  EEEP  tool  be  used  to  collaborate  in  educaBonal  research?  

 •  managing  what  we  know  •  generaBng  a  common  language  and  framework  

•  tool  for  communicaBng      

Collaborators  &  Contributors    •  AERO  Team—Greg  Rousell,  Susan  Palijan,  Kim  BenneL,  

Stephanie  Pagan  •  OISE  Team—Ruth  Childs,  Susan  Elgie,  Jayme  Herman,  

ChrisBe  Fraser  •  AERO  Members  present  at  the  Fall  SIG  2011  •  Don  Buchanan  •  Colleagues  in  the  OISE  course  Advanced  Issues  in  

Educa&onal  Policy  Analysis  and  Program  Evalua&on  •  ALendees  of  the  CEPP  Seminar  Series  •  AERO  Spring  Conference  2012  •  Funding  by  KNAER—Knowledge  Network  for  Applied  

EducaBonal  Research      

THANK  YOU!  

 We  look  forward  to  the  opportunity  to  discuss  collaboraBon  and  our  project  with  you  during  the  roundtable  session!