Child Outcomes Quality Assurance

January 8, 2010

Presented by Sherlock Center on Disabilities at RI College and

The RI Department of Human Services

For more information contact Maureen Whelan: [email protected]

Today’s Learning Outcomes Understand how the Child Outcomes

Summary Forms (COSF) gets translated and reported to OSEP (the Office of Special Education Programs)

Learn ways to review your own program data Discover how to improve the validity and

reliability of your data and Begin to use the data for quality improvement

The Sherlock Center on Disabilities at RI College and RI Department of Human Services


1. Children will demonstrate positive social emotional development (including positive social relationships)

2. Children will acquire and use knowledge and skills, including early literacy skills.

3. Children will use appropriate behavior to meet their needs.


• Moving from Compliance to Performance• Summary Statements• Target Setting• RI’s Current Process: COSF



Outcome Statements1. Children will demonstrate positive

social emotional development (including positive social relationships)

2. Children will acquire and use knowledge and skills, including early literacy skills.

3. Children will use appropriate behavior to meet their needs.

Outcome 1: Children will demonstrate positive social emotional development (including positive social relationships)

Involves:Relating with AdultsRelating with ChildrenFollowing and understanding rulesRegulating emotions and behaviors

Includes areas like:Attachment/separation/autonomyExpressing emotions and feelingsLearning rules and expectationsSocial interactions and play


Regulate emotions A 3 – 6 month old child can be comforted or

soothed by caregiver when over stimulated A 12 – 18 month old may use a security object

to soothe, may have small tantrums but can be easily directed

A 24 – 36 month old begins to verbalize feelings, can wait for short periods of time, can transition between activities and follows directions with little resistance


Outcome 1: Children will demonstrate positive social emotional development (including positive social relationships)

Outcome 2:Children will acquire and use knowledge and skills, including early literacy skills.

Involves: Thinking Reasoning Remembering Problem Solving Using symbols and language Understanding physical and social worlds

Includes areas like Early concepts – symbols, pictures, numbers Imitation Object permanence Expressive language and communication Early literacy


MemoryA 3-6 month old will begin to repeat an

enjoyable activity (i.e. shaking a rattle)A 12-18 month old will show object permanence

and remember social games or actionsA 24-36 month old can retell stories, remember

simple rules, and remembers simple life events


Outcome 2:Children will acquire and use knowledge and skills, including early literacy skills.

Involves: Taking care of basic needs Getting from place to place Using tools (e.g. fork, toothbrush, crayon) Health and Safety

Includes areas like: Integrating motor skills to complete tasks Self-help skills Using appropriate communication to ask for help

when needed Acting on the world to get what one wants


Outcome 3:Children will use appropriate behavior to meet their needs.

Communicating and satisfying hungerA 3-6 month old will have a different cry to mean

hungry, and will begin to hold bottleA 12-18 month old may have a few words or signs

for familiar foods, can use hands to feed self, and explores with utensils

A 24-36 month old can ask for food using familiar phrases and simple sentences, feeds self independently with utensils


Outcome 3:Children will use appropriate behavior to meet their needs.

Functional Skills are INTEGRATED

Have Positive Social

Relationships

Acquire & Use Knowledge &

Skills

Take Appropriate Action to Meet Their Needs

NOT domain-specificThe Sherlock Center on Disabilities at RI College and RI Department of Human Services

The Ratings: Things to RememberWhen assessing children for the outcomes:

• Involve multiple sources• Involve multiple measures


Ask yourself:• What does the child typically do?• What is the child’s actual performance across settings and situations?• How does the child use skills to

accomplish tasks?

Outcomes are NOT:• The child’s capacity to function under unusual

or ideal circumstances• Necessarily the child’s performance in a

structured testing situation


The Ratings: Things to Remember

MOST IMPORTANTLY!!!What are the typical functional skills and behaviors for this child across a variety of settings?

What do we need to know when filling out the COSF?

Understand the contents of the three outcomes

Understand age-expected child developmentUnderstand age-expectations for a child

functioning within the child’s cultureKnow about the child’s functioning across

settings and situationsKnow how to use the rating scale


COSF Ratings7 Completely

The child shows behaviors and skills expected in all or almost all everyday situations that are part of the child’s life

The child’s functioning is considered appropriate for age

No one has significant concerns about the child’s functioning in this outcome area


Outcome 3:Children will use appropriate behavior to meet their needs.Example: 28 month old child7 Completely

Child is independently mobile in all settings. Child dresses, brushes teeth, puts away own clothes, and eats independently. Child is able to communicate with a variety of people wants, needs, and asks for help using verbal language. Child remains safe in all settings, and demonstrates an understanding of rules at home and at childcare.


6 Between Completely and SomewhatThe child’s functioning is generally considered

appropriate for age, but there are significant concerns about the child’s functioning in this outcome area


COSF Ratings

6 Between Completely and SomewhatChild is independently mobile in all settings. Child dresses, brushes teeth, puts away own clothes, and eats independently. Child is able to communicate with a variety of people wants, needs, and asks for help using verbal language most of the time. Child requires constant supervision at home due to safety concerns (jumps off furniture and throws objects). Child is able to follow rules, but this is more consistent when in a highly structured environment.


Outcome 3:Children will use appropriate behavior to meet their needs.Example: 28 month old child

5 SomewhatThe child shows functioning expected for age

some of the time and/or in some situations

The child’s functioning is a mix of age-appropriate and not-appropriate function

The child’s functioning might be described as like that of a slightly younger child


COSF Ratings

5 SomewhatChild is independently mobile in all settings. Child is able to take clothes off independently, but needs help to finish dressing. Child can brush teeth with help, and eats independently with utensils. At times, the child needs a reminder to use verbal language to communicate with a variety of people his wants, needs and to ask for help. Child remains safe in all settings, and is more likely to demonstrate an understanding of rules at child care than at home.



4 Between emerging and somewhatChild shows some age appropriate functioning

some of the time or in some situations or settings, but most of the child’s functioning would be described as not yet age appropriate

The child’s functioning might be described as that of a younger child


COSF Ratings

4 Between Somewhat and EmergingChild is independently mobile in all settings, although child tends to trip a lot when an object is in his way. Child dresses and undresses with adult help, brushes teeth with help, and eats most of a meal using utensils before using his fingers to self-feed. At times, the child needs a reminder to use verbal language to communicate with a variety of people his wants, needs and to ask for help. Child requires much supervision at home and at child care due to safety concerns. Child is beginning to follow some daily routine activities, but requires a lot of assistance and guidance with other rules and non-routine directions within his daily routine both at home and at child care.



3 EmergingThe child does not yet show functioning

expected of a child his/her age in any situationThe child’s behaviors and skills include

immediate foundational skills on which to build age appropriate functioning

The child’s functioning might be described as like that of a younger child


COSF Ratings

3 EmergingChild has recently become stable with walking long distances, but still has times where he does fall especially if an object is in his path. Child needs assistance with dressing and undressing although he likes to try it on his own first. Child finger feeds to eat, is beginning to use utensils, but is very messy at meal times. Child has a limited vocabulary, but is using some single words to request favorite foods. Child will tug on his mother’s clothing to indicate he wants her help, but will usually resort to crying when he needs help at child care. Child can follow basic routine directions, and is beginning to follow other simple directions and commands with some guidance from an adult.



2 Between Emerging and Not YetThe child does not yet show functioning

expected of a child his/her age in any situationThe child’s behaviors and skills have some of

the immediate foundational skills on which to build age appropriate functioning, but these are not displayed often.

The child’s functioning might be described as that of a younger or much younger child


COSF Ratings

2 Between Emerging and Not YetChild is stable most of the time while walking, but will revert to crawling when he is tired. Child needs assistance with dressing and undressing although he likes to try it on his own first. Child finger feeds to eat, is beginning to use utensils, but is very messy at meal times. Child has a limited vocabulary, but is using some single words to request favorite foods. Child will usually cry or fuss when he needs help, but sometimes will tug on an adult’s clothing to gain their attention for help. Child will follow basic routine directions and simple commands if the adult pairs the command with a physical gesture.



1 Not YetThe child does not yet show functioning

expected of a child his/her age in any situationThe child’s skills and behaviors also do not yet

include any immediate foundational skills on which to build age-appropriate functioning

The child’s functioning might be described as like that of a much younger child

Child with 1 ratings still have skills, just not yet an immediate foundational skill


COSF Ratings

1 Not YetChild currently crawls to get from place to place and has pulled to stand on his sofa a few times. Child requires full assistance with dressing. Child finger feeds to eat at times, but is usually fed by an adult. Child drinks from a bottle, and is not able to drink from an open cup. Child will sometimes point to what he wants, but usually will cry, fuss, or tantrum when he is not understood by others.



Decision TreeChild: Julie Age: 30 monthsOutcome: 3

Summary: Julie is able to walk from place to place, but will fall if there is an object in her way. She will often bump into things and is a bit unbalanced on uneven surfaces (sand, grassy areas). Julie requires assistance to climb up and down stairs, and up an down from chairs that are not toddler height. Due to this, Julie is sometimes either unsafe in areas that are cluttered or at times unwilling to even approach an area that is cluttered or challenging. Julie can drink from an open cup, use utensils, change her clothes, and brush her teeth with adult assistance. Julie seems to be more independent with these skills at home than at her child care center as reported by her mother and child care teacher. Julie uses some intelligible single words to request foods (drink, cookie, more, banana) and to request favorite activities (book, dora, swing-to mean playground, nana-to mean grammas house). When she is not understood by others, or if she becomes frustrated while trying to do something for herself, she will immediately fall to the floor and tantrum.


Progress Y/NThe ECO Center defines progress as “the acquisition

of at least one new skill or behavior related to the outcome”

Compares child to her/himself over time (not to same-age peers, as with the 1-7 ratings)

HINT: The answer should almost always be YES, unless the child has a degenerative or extremely disabling condition

“Impossible” – 1-7 ratings indicate growth but ‘no progress’ indicated


Where Do We Go From Here?COSF data needs translation to

interpret

Entry (1-7) + Exit (1-7) + Progress =

Categories a-e


Category “a”

Child did not improve functioning from entry to exit

Child didn’t gain any new skills while in the program OR

Child regressed during the programThis category should include only those children with

degenerative conditions or very significant disabilities


Child “a”: SallySally was enrolled in EI at age 2 due to recent

loss of some single words but was pointing to request preferred foods and toys

Sally was diagnosed with Rett Syndrome at age 25 months

At exit, Sally had no spoken language and had lost her ability to point to request things


Developmental Trajectory for “a”

Level of Development

7

6

5

4

321

Age in MonthsEntry Exit


Category “b”Child who improved functioning but not

sufficient to move nearer to functioning of same-age peersChild who acquired new skills but grew at the same

rate throughout their time in the programChild made gains but did not change their rate of

growth


Child “b”: TimmyTimmy entered EI at age 2, and he was using

a few single words to request, but his parents often could not understand his speech

By the time Timmy was 3, he was using more words at home but unfamiliar people still found his speech difficult to understand


Developmental Trajectory for “b”



7

6

5

4

321


Category “c”Children who improved functioning to a level

nearer but not equal to same-age peersChild who accelerated their rate of growth during

their time in the programChild made progress toward “catching up” to

peers but still functioning below age expectations“Narrowed the gap”


Child “c”: KareemKareem was enrolled in EI at 9 months of age

because he was not sitting independently or crawling

By age 3, Kareem was walking skillfully but still required support to manage uneven surfaces when playing with peers in community settings (parks, library)


Developmental Trajectory for “c”



7

6

5

4

321


Category “d” Children who improve functioning to reach a

level comparable to same-age peers Child who was functioning below age

expectations when they entered program but are age-appropriate when they leave program

Child “caught up” while in the program


Child “d”: Maria Maria lives in a bilingual home and entered EI

at age 2 with a few Spanish single words, but was mostly pointing or getting things for herself

By age 3, Maria was a bit shy with new people until she became comfortable, but was consistently using 2- and 3-word phrases in both Spanish and English


Developmental Trajectory for “d”



7

6

5

4

321


Category “e”Children who maintained functioning at a

level comparable to same-age peersChildren who were functioning at age

expectations when they entered the program and kept up with age expectations throughout their time in the program


Child “e”: Alex Alex was enrolled in EI at 3 months of age

due to a diagnosis of neurofibromatosis and possible risk for vision problems and developmental delays

Alex was discharged from EI at age 15 months of age because he had not developed vision problems and was age-appropriate in all areas of development


Developmental Trajectory for “e”



7

6

5

4

321


Converting COSF data to OSEP progress categories

Child's Name Entry Rating Exit Rating Progress? OSEP Category

Tammy7 7 yes

Patty6 7 yes

Jonas3 5 yes

Phoenix4 4 yes

Angela1 1 no

Juan2 4 yes

Terry 1 1 yes

Leroy3 5 yes

Maria4 7 yes

Tony5 3 yes

Tutor:

Conversion of Child Outcomes Summary Form (COSF) data

to OSEP Reporting Categories

Entry Exit OSEP

Outcome Outcome Progress Category

--

Instructions

Entry: Enter rating number for outcome (1 through 7)

Exit: Enter rating number for outcome (1 through 7)

Indicate whether or not progress was made since Entry

(Yes or no: format "y" or "n")

Reporting category will appear in "OSEP Category" column

a - Children who did not improve functioning

b - Children who improved functioning but not sufficient to move nearer to

functioning comparable to same age peers

c - Children who improved functioning to a level nearer to same-aged peers

but did not reach it

d - Children who improved functioning to reach a level comparable to

same-aged peers

e - Children who maintained functioning at a level comparable to same-aged peers

Progress? - No determination of progress has been provided yet (please fix)

Impossible - The combination entered could not possibly occur. For example,

a child cannot go from a 5 to a 7 and show no progress (please fix)



What do you see?6’s and 7’s Outcome 1 = 46%Outcome 2 = 36%Outcome 3 = 26%

Does what you see make sense?

6’s and 7’s are lowest in Outcome 3


• “Few children receiving services would be expected to be considered functioning as typically (few scores in the typical range) at entry”

• “Functioning in one outcome area will be related to functioning in the other outcome areas”


What do you see?


• Do you have high 6’s and 7’s?Add up the % of 6’s and 7’sThen compare your % with the State as

a whole

• Does Outcome 3 have a lower % of 6’s and 7’s than Outcomes 1 and 2?

• Compare % of 6’s and 7’s across all Outcomes


Look for examples of all 6’s and 7’s at entry

Look for 2 Outcomes entering at 6 or 7 and 1 Outcome entering at lower rating

Look for wide discrepancies in ratings across Outcomes


What do you see?

• 6’s and 7’s increase with all Outcomes• Increase in 6’s and 7’s for Outcomes 2

and 3• Decreases in 5’s



What do you see?• Do 6’s and 7’s increase with all

Outcomes?• Compare 6’s and 7’s at exit to entry

ratings in all Outcomes • Compare exit and entry of other

ratings?Does what you see make sense?


Look for examples of all 6’s and 7’s at exit in all/or 2 Outcomes areas for children who exited to Part B


• % of e’s is higher than all other categories in Outcome 1 and 2

• Low % of a’s • % of d is higher in Outcome 3 compared to Outcomes 1 and 2

Missing Information

Impossibles

Progress Questionable

What do you see?


Program Data ScoresWhat do you see?

• Does your program have a high % of e’s• Is there a difference in % of e’s between

Outcomes?• Is there a difference in the % of d’s between

Outcomes?• Is there another score category that is high?

%b?• Does your program have missing data?

Why do you think this is?


Look at e’s…Do you have records with 6’s and 7’s at entry and exit in 2 or more Outcomes

Look at a’s… Is the progress question correct?

Look at b’s…Does the written summary support a rating that has stayed the same?

Look at records with Progress… Is the progress question blank or is it a data entry issue?

Look at the Impossibles….Was the question understood?

Look at Missing records…Is it a data entry problem or a systems issue (Note: Attaching the COSF to the discharge sheet is an effective way to lessen “Missing Data”


Summary StatementsSummary Statement 1:

Of the children who entered below age expectation, the % of children who substantially increased their rate of growth by exit

c+d / a+b+c+d

Summary Statement 2: The % of children who were functioning

within age expectations by the time they exited

d+e/ a+b+c+d+e


RI Process and Guidance•Upon Entry and Exit for ALL children •COSF’s for transfers

•In-State – use previous programs entry status

•Out-of-State – begin new entry status rating

•More than 1 EI experience – begin new rating upon each entry

•COSF Review Protocol


RI Process and GuidanceRating Resources• Backside of COSF• RI Child Outcomes Guidance• ECO Decision Making Trees


What is it?

• Excel spreadsheet, found in the RI Early Intervention Care Coordination System in Welligent - updated quarterly• Contain information for all providers – no identifying data• Can compare to statewide data or another provider• Identify missing data• Analyze program data for trends – identify training needs


Where is it?Go to the communications center then click on downloads

Outcomes Summary Spreadsheet


Click saveHow is it downloaded?

Click file to download

Then select where to save the file then click save

Note: If your agency’s server does not allow you to save the file on your PC, you may be able to save the file to a thumb drive instead of your PC



Main components of the spreadsheet

Definition tab – describes what the spreadsheet contains and the meaning of

terms

Missing Data tab – review data to find what children have missing or invalid

data

Main Entry tab – contains tables and graphs to review each outcome entry

data *

Main Exit tab – contains tables and graphs to review each outcome exit data

*

Main Scores tab – contains tables to review each outcome scoring data and

outcome statements *

Raw Tab – contains all the raw data

Data Patch – contains data to make sure the outcome statements work

correctly

* Providers may change parameters on this sheet to analyze different aspects of the data



Parameters are drop-down lists that the user may change to view filtered information. The default is “all” meaning all data is used for the calculations. The user can change the following drop-downs to filter the out-come data.

• Discharge SFY – Select one state fiscal year (July – June) based on discharge date

• Discharge year – Select one calendar year (January-December) based on discharge date

• Disc –Select one Discharge Status• SFY- Select one state fiscal year (July- June) based on referral date• Year - Select one calendar year (January- December) based on referral date• Eli – Select one Eligibility Category• Provider- Select one EI Provider• Group Months in EI – Select a length of stay in EI based on referral date• Group Months from entry – Select a length of stay in EI based on the COSF

entry date• Group (or Entry) More Than 6 – Older versions of excel only allow the user to

select one option from each drop down list. If you are using a older versions this drop down allow you to select children who have been in EI (based on referral or COSF entry) “less” than six month or “More” than six months. Users with a higher version of excel do not need this option as then can select one or more age groups from the option above.



To work with any table- right mouse click on the table and click refresh data

Click on View then Zoom to display the tab in a larger or smaller print!Change any the parameters fields by using the drop down listsClick on just the graph to print the graph only

Click on chart and then press {shift}{ctrl}F to format the chart. This works if you are using Microsoft Excel 2000. For high or lower versions of Excel you may need to format manually or create a macro.


Tips


The missing data tab will assist in identifying the IDs of children with incorrect or missing data. This spreadsheet only contains children enrolled in EI who have been discharged. If the child has been enrolled, providers, in most cases, should be able to fill out an exit outcomes form based on their knowledge of the child.There will be exceptions to the rule when a provider has not served the child. The parameters in the missing data tab can help providers sort through problems with missing or incorrect data.Incorrect data must be corrected

• Even for children who have been in EI less then six months the form should be filled out – However the parameter “Group months in EI” can be used to identity children over or under six months in EI.

• Certain discharge categories may indicate the reasons for missing data such as unable to contact but this would also depend on how long the provider has served the family.


Missing Data Guidelines


EI TUNESOutcome Essentials

Documents

Child Outcomes Quality Assurance