Upload
hubert-pope
View
213
Download
0
Embed Size (px)
Citation preview
Early Childhood Outcomes CenterEarly Childhood Outcomes Center 1
Using Data for Program
Improvement
Christina Kasprzak, NECTAC/ECO
Ann Bailey, NCRRC
July 2010
Measuring Child and Family Outcomes
Early Childhood Outcomes CenterEarly Childhood Outcomes Center 2
Systems Thinking:
Systems are made up of interrelated, interconnected components
Systems change involves changing the capacity, interrelationships, and interdependencies among parts, levels and stakeholders
Desired changes in one part or level of the system must be accompanied by changes in other parts or levels
Measuring Child and Family Outcomes
Early Childhood Outcomes CenterEarly Childhood Outcomes Center 3
SPP as long term plan for systems change:
EI/ECSE are systems of complex, interrelated components with goal of achieving outcomes for children and families
Changes to EI/ECSE systems require a combination of improvement activities that are interconnected and support changes to infrastructure that work together to achieve the desired results
Measuring Child and Family Outcomes
Early Childhood Outcomes CenterEarly Childhood Outcomes Center 4
Developing improvement activities:
Effective Policies, Procedures, and MOU Data Systems and Monitoring TA and Professional Development Fiscal Management Evaluation
Measuring Child and Family Outcomes
Early Childhood Outcomes CenterEarly Childhood Outcomes Center 5
Evaluating SPP/APR Improvement Activities
Measuring Child and Family Outcomes
6
The Need for Evaluation
Provides an organized way of assessing work in progress and results obtained
Assesses the impact of an activity on the area targeted for improvement
Identifies strengths and weaknesses in the implementation of the improvement activity
Measuring Child and Family Outcomes
7
Types of Evaluation
Process evaluation
Evaluates the improvement process itself
Impact evaluation
Evaluates the results produced by the process
Measuring Child and Family Outcomes
8
Process Evaluation Questions
Process evaluation questions might answer To what extent is the improvement activity
being implemented as intended? To what extent is the improvement activity
reaching the target audience (i.e., children, staff, parents)?
Is everyone doing what they said they would do?
Are resources still available to adequately support this improvement activity?
Measuring Child and Family Outcomes
9
Impact Evaluation Questions Impact evaluation questions might include
Did the improvement activity accomplish what it was supposed to?
Which parts of the improvement activity worked well?
Which parts of the improvement activity did not work well?
Should the agency continue the improvement activity?
What has changed as a result of implementing the improvement activity?
Measuring Child and Family Outcomes
10
Reviewing Improvement Activities Improvement activities are aligned to the
indicators, including if reflected across related indicators.
Improvement activities reflect state priorities.
Improvement activities are actionable.
Improvement activities are realistic.
Measuring Child and Family Outcomes
11
Reviewing Improvement Activities Improvement activities include measures of
performance.
Improvement activities include timelines.
Improvement activities identify responsibility for implementation.
Improvement activities include technical assistance needs.
Measuring Child and Family Outcomes
12
Categorizing Improvement Activities
Training and Professional Development Improve Data Collection Improve Systems Administration and Monitoring Improve Collaboration and Coordination Program Development Clarify/Examine/Develop Policies & Procedures Provide Technical Assistance Increase/Adjust FTE Evaluation
Measuring Child and Family Outcomes
13
Developing a Plan for Evaluation
Identify the goal of the evaluation.
Frame the evaluation questions to be answered.
Identify evaluation methods and measurement options.
Identify data sources.
Determine data analysis techniques.
Establish timelines.
Measuring Child and Family Outcomes
What’s Next?
Moving beyond data quality
Understanding and manipulating the data you have
Using data to make program improvements
Measuring Child and Family Outcomes
Early Childhood Outcomes CenterEarly Childhood Outcomes Center 15
Looking at Data:
To better understand issues and areas of concern to focus improvement activities... What does the data analysis say? What are the
‘root causes’ of issues or challenges?
Measuring Child and Family Outcomes
Early Childhood Outcomes CenterEarly Childhood Outcomes Center 16
Using data for improvement
Evidence
Inference
Action
Measuring Child and Family Outcomes
Early Childhood Outcomes CenterEarly Childhood Outcomes Center 17
Evidence
Evidence refers to the numbers, such as
“89% of families reported ...”
The numbers are not debatable
Measuring Child and Family Outcomes
Early Childhood Outcomes CenterEarly Childhood Outcomes Center 18
Inference
How do you interpret the #s? What can you conclude from the #s? Does evidence mean good news?
Bad news? News we can’t interpret?
To reach an inference, sometimes we analyze data in other ways (ask for more evidence)
Measuring Child and Family Outcomes
Early Childhood Outcomes CenterEarly Childhood Outcomes Center 19
Action
Given the inference from the numbers, what should be done?
Recommendations or action steps
Action can be debatable – and often is
Another role for stakeholders
Measuring Child and Family Outcomes
Early Childhood Outcomes CenterEarly Childhood Outcomes Center 20
Program improvement:
At the state level – TA, policy
At the regional or local level – supervision, guidance
At the service/classroom level– implement high quality individualized family centered services
Measuring Child and Family Outcomes
Early Childhood Outcomes CenterEarly Childhood Outcomes Center 21
Key points
Evidence refers to the numbers and the numbers by themselves are meaningless
Inference is attached by those who read (interpret) the numbers
You have the opportunity and obligation to attach meaning
Measuring Child and Family Outcomes
Early Childhood Outcomes CenterEarly Childhood Outcomes Center 22
Continuous Program Improvement
Plan (vision) Program characteristics
Child and family outcomes
Implement
Check(Collect and analyze data)
ReflectAre we where we
want to be?
Measuring Child and Family Outcomes
Early Childhood Outcomes CenterEarly Childhood Outcomes Center 23
Tweaking the System
Plan (vision) Program characteristics
Child and family outcomes
Implement
Check(Collect and analyze data)
ReflectAre we where we
want to be?
Is there a problem?
Why is it happening?
What should be done?
Is it being done?
Is it working?
Measuring Child and Family Outcomes
Early Childhood Outcomes CenterEarly Childhood Outcomes Center 24
Outcome questions for program improvement, e.g.
Do outcomes vary by
Region of the state? Level of functioning at entry? Services received? Age at entry to service? Type of services received?
Measuring Child and Family Outcomes
Early Childhood Outcomes CenterEarly Childhood Outcomes Center 25
Looking at Family Outcomes by Subgroups
System Characteristics
Family Characteristics
Child Characteristics
Service Characteristics
Measuring Child and Family Outcomes
Early Childhood Outcomes CenterEarly Childhood Outcomes Center 26
Are there differences in outcomes across family characteristics?
Race/ethnicity
Family Income
Primary language
Family structure
Etc
Measuring Child and Family Outcomes
Early Childhood Outcomes CenterEarly Childhood Outcomes Center 27
Are there differences in outcomes across child characteristics?
Race/ethnicity
Type of disability
Length of time in services
Etc
Measuring Child and Family Outcomes
Early Childhood Outcomes CenterEarly Childhood Outcomes Center 28
Examples of process questions
Are ALL services high quality?
Are ALL children and families receiving ALL the services they should in a timely manner?
Are ALL families being supported in being involved in their child’s program?
What are the barriers to high quality services?
Measuring Child and Family Outcomes
Early Childhood Outcomes CenterEarly Childhood Outcomes Center 29
Working Assumptions There are some high quality services
and programs being provided across the state.
There are some families who are not getting the highest quality services.
If we can find ways to improve those services/programs, these families will experience better outcomes.
Measuring Child and Family Outcomes
Early Childhood Outcomes CenterEarly Childhood Outcomes Center 30
Action Given the inference from the numbers,
what should be done? Develop improvement activities that
are: Targeted based on data analysis Based on evidence based practices Interconnected, work together to
accomplish the desired result
Measuring Child and Family Outcomes
Early Childhood Outcomes CenterEarly Childhood Outcomes Center 31
Small Group Scenarios
Measuring Child and Family Outcomes
Early Childhood Outcomes CenterEarly Childhood Outcomes Center 32
Small Group Scenarios
Each table is assigned a scenario Choose a table / scenario As a group, walk through the scenario.
Discuss and answer questions. Jot down your ideas and be prepared to
share back some highlights.