Are your C4 data reflective of the families you serve? Joy Markowitz, Director Jean Dauphinee, TA Specialist Measuring Child and Family Outcomes Conference,

Are your C4 data reflective of the families

you serve?

Joy Markowitz, Director

Jean Dauphinee, TA Specialist

Measuring Child and Family Outcomes Conference, August 28, 2008


2

Purpose of Collecting C4 data

• To figure out how to serve our families better specifically related to Indicator C4:

Percent of families participating in Part C who report that early intervention services have helped the family:

– A Know their rights;

– B. Effectively communicate their children's

needs; and

– C. Help their children develop and learn.


3

Why do we care who responds to our surveys?

•We don’t want to invest time and money in improvement activities if data do not reflect the opinions and/or experiences of the families/children we serve.

•This is an issue if you collect data from all families (census) or a sample of families because not all families who receive a survey respond.


4

Population

Population may be defined by the state and could be (for example):• all Part C families• all Part C families exiting in a given year, or• all Part C families in program more than 1 year.


5

Other terms used

Population• Target group• Target population

Respondents• Response Pool• Respondent Pool• Respondent Group


6

Nonresponse Bias

•Representativeness is a word we have been

using. It’s not a statistical term.

•Correct term –nonresponse bias.

•Meaning: Are the families who did not

respond different from the families that did?


7

What is a Sample?

•A sample is a subset of the population thatyou define. However, this sample mustbe derived using an approved samplingplan.

•OSEP collaborates with DAC to review andapprove submitted sampling plans.


8

Questions about Response Rates

•What is high enough?

•How do you determine if your respondentsare reflective of the population youdefined?

Whether using census or sampling, yourreturn or response rate is unlikely to be ashigh as you want it to be; therefore, you must address nonresponse bias.


9

Example 1

Families served

Black 30

White 50

AI/AN 20

Total 100

Survey respondents

Black 24

White 44

AI/AN 2

Total 70

Do these data reflect the population?

Families served in 2006-2007 (n=100):


10

Example 2

Families served

English 75

Other language 25

Total 100

Survey respondents

English 65

Other language 5

Total 70


Families exiting in 2006-2007 (n=100):


11

Example 3

Families served by agencies

Agency 1 100

Agency 2 100

Agency 3 100

Agency 4 100

Total 400

Survey respondents

Agency 1 49

Agency 2 63

Agency 3 66

Agency 4 21

Total 199


Families served in programs for at least one year (n=100):


12

State Data Collection Methods/Strategies

Methods• Census (n=41 states)• Sample (n= 15 states)

–oversampling populations that are known to be hard to reach.

Analysis strategies• Frequencies and

percentages• Weighting• Sampling among

respondents


13

Summary of C4 Findings(Based on analysis of 54 state 2006-2007 APRs)

67% (n= 36 states) reported nonresponse bias, 26% (n=14 states) did not.

Among the 67% who reported nonresponse bias, most reported by one variable (e.g. race/ethnicity, region, gender, or child’s age)


14

Summary of C4 Findings(Based on analysis of 54 state 2006-2007 APRs) cont.

Most states measured nonresponse bias by:

Race/ethnicity or

Race/ethnicity and other factors such as:

a. length of time in program,

b. region of the state,

c. child’s age, or

d. gender.


15

From SPP/APR Instruction Sheet

States are allowed to use sampling when so indicated on the Part C Indicator Measurement Table. When sampling is used, a description of the sampling methodology outlining how the design will yield valid and reliable estimates must be submitted to OSEP. The description must describe the: (a) sampling procedures followed (e.g., random/stratified, forms validation); and (b) the similarity or differences of the sample to the population of children with disabilities in the early intervention program (e.g., how all aspects of the population such as disability category, race, age, gender, etc. will be represented).


16

From SPP/APR Instruction Sheet (cont.)

The description must also include how the Lead Agency addresses any problems with:

(1) response rates;

(2) missing data; and

(3) selection bias.

Samples from EIS programs must be representative of each of the EIS programs sampled, considering such variables as eligibility definition (diagnosed condition or developmental delay), age, race, and gender.


17

From SPP/APR Instruction Sheet (cont.)

In reporting on the performance of small EIS programs, the Lead Agency shall not report to the public or the Secretary any information on performance that would result in the disclosure of personally identifiable information about individual children or where the available data is insufficient to yield statistically reliable information, i.e., numbers are too small.

Source: Part C State Performance Plan (SPP) and Annual Performance Report (APR) Instruction Sheet dated 10/19/2007.


18

Examples from States

How have states tackled these issues?

State representatives:

•Wendy Whipple, NV

•Sue Campbell and Rosanne Griff-Cabelli, DE

•Alice Ridgeway, CT


19

Questions?

•What questions are you getting from OSEP that you need help to answer?

•What guidance do you need from OSEP?

•What are realistic expectations of states for reporting nonresponse bias?

Documents

Are your C4 data reflective of the families you serve? Joy Markowitz, Director Jean Dauphinee, TA Specialist Measuring Child and Family Outcomes Conference,