IEFE 2013 Workshop

Designing a Curriculum Evaluation Dr. Dorothy Harnish

Workshop Outline �  Overview of Evaluation Design Components �  Hierarchy of evidence model �  Template for developing an evaluation plan �  Complete sample plan ◦  Identify the evaluation questions ◦  Identify indicators or evidence desired ◦  Describe sources of information ◦  Describe types of data from each source ◦  Describe methods of collecting information ◦  Describe how data will be analyzed

�  Participants design an evaluation plan to use in their school �  Share several examples with group �  Questions, concerns, issues related to the evaluation process

Evalua&on  Ques&ons  

Evidence/  Indicators  

Sources   Type  of  data    

Data  collec&on  

Analysis  

Evaluation Questions

Ø  Is the program being implemented as it was designed/intended? Have each of the key elements of the program been put in place? If not, why not?

Ø What changes have occurred in teaching methods, student behaviors, learning outcomes, or school processes? What evidence exists for this?

Ø Are those implementing the program (e.g., teachers, administrators) sufficiently trained to use the new methods, materials, or equipment?

Evaluation Questions Ø Are resources (e.g., materials, technology, time, staffing)

sufficient to implement all parts of the program?

Ø How satisfied are parents, students, teachers, and administrators, with the program and results? What problems have been identified by stakeholder groups?

Ø What changes are needed to improve the program?

Ø Do the results justify continued expenditure of funds on this program or expansion to other sites?

Information Sources

� People � Products � Processes � Documents/Materials �  Importance of “triangulation” through

multiple sources of information

Data Collection

Ø Methods determined by evaluation purpose and questions

Ø Qualitative and quantitative methods § on-site field observations § individual interviews § focus groups § questionnaires § document or program materials review § student test or performance data

Data Analysis & Reporting � Experimental, quasi-experimental,

longitudinal, and qualitative analyses � Pre-post analyses to assess growth and

impact � Comparison groups � Randomization to control extraneous

variables � Generalization of findings � Difficulty of cause-effect conclusions

Evalua&on  Ques&ons   Evidence/Indicators   Sources   Type  of  data     Data  collec&on   Analysis  

1.  How  well  is  the  Reading  Excellence  program  being  implemented  in  elementary  classes?  (process  evalua>on,  implementa>on)  

 -­‐  Are  all  components  

of  the  curriculum  being  taught?  

-­‐  Are  new  teaching  approaches  being  used  effec>vely?  

-­‐  Are  new  materials  being  used  as  intended?  

Materials  are  available  in  all  classes    Teachers  are  trained  to  use  methods    Teachers  understand  the  new  program  goals,  materials,  methods    Teachers  use  program  strategies  in  their  daily  teaching  

Teachers    Students    Administrators    Curriculum  experts/trainers      Curriculum,  training    materials  

Field  observa>ons  of  reading  teachers      Ra>ng  scale  of  program  implementa>on  criteria  used  in  observa>on  visits  to  classrooms    Interviews  with  teachers      Survey  of  reading  teachers,  students,  school  administrators,  curriculum  experts    Training  workshop  documenta>on  

Classroom  visits  by  evaluator(s)  4-­‐6  >mes/year    Classroom  visits  by  supervisor  or  principal  2-­‐3  >mes/month    Interviews  with  teachers  conducted  by  evaluators  mid-­‐term,  end-­‐of-­‐course    Teacher,    student  ,  administrator,  curriculum  expert  surveys  administered  at  end  of  course  

Implementa>on  criteria  achieved  by  teachers,  as  documented  by  observa>on  interview,  and  survey  data  (components,  methods,  materials)    Comparison  of  survey  feedback  from  4  sources  

Evalua&on  Ques&ons   Evidence/Indicators   Sources   Type  of  data     Data  collec&on   Analysis  

2.  To  what  extent  does  student  reading  skill  improve  with  use  of  Reading  Excellence  methods?  (outcomes  evalua>on,  impact)  

 -­‐  Do  students  read  

with  greater  fluency  and  understanding?  

-­‐  Do  students  have  a  greater  vocabulary?  

-­‐  Do  students  read  more  frequently  in  and  out  of  class?  

Results  of  student  reading  assessments  show  posi>ve,  significant  changes  in    vocabulary,  fluency,  comprehension    In-­‐class  reading  performance  has  improved    Out-­‐of-­‐class  reading  ac>vi>es  are  more  frequent  

Student  achievement  measures    Test  results/in-­‐class  student  assessments    Teachers    Parents    

Tests  of  achievement  in  vocabulary,  fluency,  comprehension  (criterion  referenced)    Teacher  assessments/  grading    of  in-­‐class  reading  performance        Teacher  interviews    Parent  survey  (observa>on  of  out-­‐of-­‐class  reading  frequency,  ability,  enjoyment)  

Pre-­‐test;  mid-­‐term;  post-­‐test  administra>on    Interviews  with  teachers  conducted  by  evaluators  mid-­‐term,  end-­‐of-­‐course    Student  grades  (mid-­‐term,  end-­‐of-­‐course)    Parent  surveys  (beginning,  end  of  school  year)  

Comparison  of  pre-­‐post  standardized  test  scores,  amount/  significance  of  change      %  of  students  achieving  end-­‐of-­‐course  passing  grade  vs.  previous  year;      %  of  students    receiving  A  –  F  grades;  %  improving  grades  from  mid  to  end  of  year    Comparison  of  pre-­‐post  student  reading  frequency  reported  by  parents