How to Use Ed360 and Other Data Tools for Year End Evaluation · •Decide what you want to know....

How to Use Ed360 and Other Data Tools for Year End Evaluation

Presented by: Gina FultonModule #11

May 27, 2020

Learning Objectives Module #11

2

To review and understand evaluation models and frameworks

To conceptualize and gather the data needed for the evaluation

To discuss theory and questions related to program evaluations

To refresh everyone’s memory about the data available in Ed360 for supporting program evaluation

Program Evaluation Definition

“A common definition used to separate evaluation from research is that program evaluation is conducted for decision-making purposes, whereas research is intended to build our general understanding and knowledge of a particular topic and to inform practice.”

-Spaulding, Dean T.

https://www2.ed.gov/about/offices/list/oese/sst/evaluationmatters.pdf

https://www2.ed.gov/about/offices/list/oese/sst/evaluationmatters.pdf

Why Evaluate Your Program?

Let’s use the Title I Program for an example as you’re required to evaluate Title I annually.

Brainstorm and list reasons why you would evaluate your Title I program.

Besides it being the law…

Brainstorm Ideas--Responses

To research if Title I program is effective To determine if we’re meeting the needs of all

students To determine if our resources align to our needs To decide whether there are areas we can improve

upon To determine if some areas of the program are not

working

What other questions might you want to include in your program evaluation? Why did you choose these?

What questions would you ask?

GENERAL IDEAS FOR PROGRAM EVALUATION QUESTIONS:(1) Has the Title I program been effective?(2) What has worked well in the Title I program? (3) What has not worked well in the Title I program(4) How should the Title I program be refined?

The data are analyzed, and the results of these analyses are used as the source of evidence to determine the answers to the four questions.

What are the steps in an evaluation?

• Decide what you want to know. (PURPOSE) • Form your evaluation questions. (What do you want to

know?)• Use a good model, design or evaluation matrix. (Logic

model and program overview slides)• Determine what data you want collected. (Types and

methods)• Collect data throughout the year. (Action plan)• Analyze data. (frequency, % change, aligned to ?)

What are the steps in an evaluation?

• Make connections to what the data is saying through triangulation. (3 or more valid data points/sets)

• Write a summary of program effectiveness. (Narrative style, using a standard template—see Rockwood 5* example.)

• Share findings and recommendations.• Ask for feedback on evaluation. What can others

do to help support the necessary changes?

Sample Evaluation Matrix

Evaluation Objective

Stakeholder Group

Tools Used to Collect Data

When Purpose

To document depth and breadth of activities that support theTitle I program during the 2019-2020 school year.

Certified and non-certified staff members, administrators, any providing services to Title I students.

InterviewSurvey

October during Parent/Teacher Conference weekOr throughoutthe year at different times.

Summative

Questions?15

Pause & Ponder16

Please write down your reflections.What ideas stuck with you?What aspects do you disagree with?What aspects do you want to ponder further?What resonates?How does this perspective differ from previous knowledge?

17

Demonstration of Ed360-Dashboard

18

Academic Overview, Grade 3, Low Income with All Races

19

Academic Overview, Grade 3, Students with Disabilities and All Races

20

Academic Overview, Grade 3, English Learners, All Races

21

KIDS Survey

22

KIDS Results by Readiness Developmental Areas Fall 2018 - 2019 with 2019-2020 School Overview

23

KIDS Readiness Measures by School

24

KIDS, Percentage of Students Demonstrating K Readiness-Male

25

KIDS, Percentage of Students Demonstrating K Readiness-Female

26

KIDS, Percentage of Students Demonstrating K Readiness, All Students, Hispanic or Latino

What data could your collect? Why?

Brainstorm activity: Share and discuss the types of data and the importance of the timeline in data collection.What data have you collected in your district for program measures?Once you collected data, what occurred? Why are the types important? Why is the timeline important?

Types of Data

Quantitative Qualitative Mixed Measures (both) Summative—collected at end of project to measure

outcomes and how those outcomes relate to the overall judgment of program (both quant. & qual.)

Formative—collected during project to measure outcomes and report back to stakeholders to be able to make timely adjustments to program

Tools for Collecting Data

Surveys Interviews Questionnaires Observations Document review Achievement data Student demographic data Student behavioral data

NAEP DATA

https://www.nationsreportcard.gov/profiles/stateprofile?chort=2&sub=MAT&sj=AL&sfj=NP&st=MN&year=2019R3

Data Analysis

Table 1-A Illinois Snapshot for NAEP Mathematics Grade 4 for 2000-2019

https://www.isbe.net/Documents/naep-2019-math-state-4-8.pdf

Triangulate Data

Be sure to include various data sets or points of collection

Check for outliers Make sure there isn’t bias or other factors that

skew your data Make sure you use evidence, grounded in facts—

not person feelings.

Quick reminders from the start of the DT Roadmap with online sessions

District Data Team• Meets monthly• Identifies District:

– Goals– Student Outcome Indicators – District Targets – Adult Action Indicators

• Develops District Strategic Improvement Plan

• Evaluates Programs

District Level Data

Data Analysis Protocol

School Level Data

Data Analysis Protocol

Classroom/Student Level Data

Data Analysis Protocol

Remember, it’s all about practicing the 5-Step process that works for your DT

Reeves’ 5 Steps: Collect & chart data Analyze strengths and

obstacles Establish goals: set,

review, revise Select instructional

strategies Determine results

indicators

Discuss the Process in DT: Consider a problem you face

as a leaders Record data you can

reconstruct from memory or have with you

Set a SMART goal Brainstorm strategies for

moving toward goal List indicators of fidelity and

success

Just remember…

If it is not written down, it doesn’t exist!

And

Document your process, analysis and results to demonstrate program effectiveness and areas of needed improvement annually.

Questions?44

Pause & Ponder45

Please write down your reflections.What ideas stuck with you?What aspects do you disagree with?What aspects do you want to ponder further?What resonates?How does this perspective differ from previous knowledge?

Wrap Up46

All Modules for Building Your Data Team (DT) Roadmap Training Modules can be found on Ed360’s Resource page https://www.isbe.net/ed360

Thanks for your participation.

IMPORTANT READING SUGGESTIONS

https://www2.ed.gov/about/offices/list/oese/sst/evaluationmatters.pdf

Content questions about this presentationGina Fulton: gfulton@isbe.net

47

Ed360 Outreach Teamed360@isbe.net

Ed360 webpageisbe.net/ed360

Ed360 Logined360.isbe.net

CONTACT INFORMATION