Measuring Effectiveness Doing what works

DR Fiona KennedyConsultant Clinical Psychologist

Working with

Dr. Dave PearsonConsultant Clinical Psychologist

Vishal TalrejaCofounder & CEODream A Dream

Why do we need to measure outcomes?

Surely stories of success are enough?

Surely counting numbers of children reached is enough?

Surely questionnaires are enough?

Any other ways we can measure outcomes?

Quantitative measuresLies, darn lies and statistics!

A good measure will be standardised (see later)

There are existing standardised measures of, e.g. educational achievement, personality, mental health….

But these are all standardised on Western populations

Because our young people are different

Adversity puts a strain on the body and the brain

Why do we need a new scale?

Adversity

can include:

Abuse

• Abandonment

• War experiences

• Extreme poverty

• Poor care

• Poor nutrition

Children in life skills programmes will have experienced many of these

Malnutritionnot having enough to eat

and/or not being loved

Causes bodies and brains to develop differently

Indian children..the factsHunger and Malnutrition (HUNGaMA) Survey Report (2011)

(Naandi Foundation, 2012)reported up to 59% of children ’stunted’

Children in India (2012) Report(Government of India Ministry of Statistics & Programme Implementation)in 2006 – approx. 48% children ‘stunted’approx. 20% ‘wasted’

stunted=height for age wasted=weight for height

Why does this matter?Children in adversity have challenges

Thinking skills (understanding, concentrating, remembering)

Relationship and attachment skills Maturity may ‘collapse’ under pressureAnxiety control skills

Life skillsThese challenges appear in everyday life as life-

skills challenges

For example, – interacting with others– overcoming obstacles and solving problems– taking initiative– managing conflict– understanding and following instructions

Meeting the challengeResearch shows:

With the right help, young people can meet and overcome these challenges

But how do we know what is the right help?

We need a new scaleBecause the young people we work

with are a special group

Because they have challenges resulting from adversity

Because they are in a developing world context

The Life Skills Assessment Scale: Measuring life skills of disadvantaged children in the developing world

Kennedy, F., Pearson, D., Brett Taylor, L. & Talreja, V. (2014). Social Behavior

and Personality: An international journal, 42 (2),

197-210.

Scale development

Drag picture to placeholder or click icon to addDrag picture to placeholder or click icon to add

How to measure cloth

A good ruler is reliable

• Should give the same length every time it’s used (test-retest)

• Should give the same length when I use it as when you use it (inter-rater)

A good ruler discriminates

Should tell the

difference between

a long and a short

piece of cloth

(discriminant

validity)

Every inch mattersEach inch should contribute to the ruler: if I take an inch away

from a ruler, that should make a difference (internal reliability)

An inch should be an inch

The world over, we know what is an inch: somewhere in the world is a ‘gold standard’ inch! (normative group)

First StepsLiterature review World Health Organization (WHO,

1997) definition of life skills as a globally influential framework.

Developing scale items then produced some items describing behaviour that could

be answered

by observers rating children and young people involved in life skills programmes

Life Skills Assessment Scale (LSAS)

interacting with others

overcoming problems and solving problems

taking initiative

managing conflict

understanding and following instructions

5-point Likert-type scale:

1 (does not yet do),

2 (does with a lot of help)

3 (does with some help)

4 (does with a little help)

5 (does independently)

An overall score is calculated as the mean of all five skills.

Rater instructions• Use judgment as to what a child might

reasonably be expected to manage at a given (chronological) age

• Note both chronological and apparent ages of the child/young person.

• Note any difficulties resulting from the use of other than first language

• Give examples of the observations on which ratings are based (optional)

Developing the scaleBetween 2009 and 2010 DaD employee raters

completed the Life Skills Assessment Scale (LSAS)

observing 1232 children and young people aged 8-16 years

from at-risk communities in Bangalore

as they participated in group activities

A Base for comparison (norms)

This data was analysed to produce normative scores

average score = 2.5

standard deviation = .75

for three age groups, 8-10 yrs, 11-13 yrs and 14-16

Does the scale hang together?

The ruler: are all the inches (items) part of the same ruler?

Yes! ☻

Does each inch do its bit?If we take each inch (item) off the ruler one at a time, does the measurement go down?

Yes! ☻

Inter-rater reliabilityUsing a ruler: if I use the

ruler (scale) then you use the ruler to measure the same thing, do we get the same result?

Yes! ☻

Test-re-test reliabilityUsing a ruler: if I use the

ruler (scale) then use the ruler again to measure the same thing, do I get the same result?

Yes! ☻ ☻

Discriminant validityCan the ruler (scale) tell the difference

between a long piece of cloth and a short piece of cloth?

comparing LSAS scores for adversity children with scores for children from an Army Public School

Yes! ☻

DiscussionThe statistical analysis indicates that

the scale can be used with confidence.

It is a reliable and valid measure of life skills

Age appropriatenessNo significant differences in the scores for each of the three

age groups

Shows rater success in making age-appropriate judgments

E.g. 8-yr-old “please bring me the football”; 14-yr-old “please collect all the footballs and put them away”.

Practical ConsiderationsThe LSAS is scored by observer rating and has only five items

It is a very simple and practical assessment tool for use with at-risk children

An explanatory session can be held or role-play scenarios acted out with new raters so that any questions about its administration can be raised.

Wider Use of the LSASParticipants for this study from Bengaluru and surrounding

areas in India

The children came from a mixture of urban and rural backgrounds

As this simple scale is child-centered, it could be used worldwide with children from disadvantaged backgrounds

Measuring program impact

Why is this scale so simple?

Surely it should be complicated, long and difficult to be good??!!

Seven years work to make the complex simple

One item represents many skills: e.g. managing conflict

E.g. fastening a button

Who can use it?NGOs - Give feedback to stakeholders, inform programme strategy and

maximise effectiveness

Donors - Make better informed decisions about funding allocation and support

Researchers - Advance the study of Life Skills

Clinicians - Use Life Skills as an outcome measure alongside mental health measures and know it is appropriate for disadvantaged Indian children

Programme developers - Check out programme performance at an early stage

Carers - Keep track of the progress of children in your care

What can they use it for?

Measuring programme impact - Find out how effective interventions are at increasing Life Skills among disadvantaged children in India

Getting a skills profile for an individual child - Look at five different Life Skills to see relative strengths in an individual child

Comparing an individual’s scores with norms - See how one child compares with the average scores for their age group

Allocating a child to the programme best for them - See how a child thrives in different programmes and help them choose the best one for them

Comparing one programme with another - Discover which Life Skills are best developed by a given programme

Feedback on progress - Let stakeholders (donors, parents, teachers, carers, children) see progress visually

Getting a skills profile for an individual child

For example: Raju

inte

ract

ing

ovec

omin

g

initi

ative

confl

ict

inst

ruct

ions

over

all

0

1

2

3

4

scores

scores

Comparing an individual’s scores with norms

For example:

interacting ovecoming initiative conflict instructions over all0

0.5

1

1.5

2

2.5

3

3.5

4

4.5

scoresnorms

Allocating a child to the programme best for them

For example, you may know

That your computer program is strong on developing the understanding and following instructions lifeskill

Young Raju may need to develop his skills in this area

So you may suggest he joins your computer program

Measuring individual progress

For example: Raju

Over all performance

For example:DaD All Program Impact 2013

1905 young people showed positive Life Skills development in one or more of the five Life Skills Assessed.  

3.145 2.915 3.008 2.848 3.263.749 3.421 3.621 3.439 3.804

Baseline Endline

Program performance

For example:

Comparing one programme with another

For example: before vs after

Seeing which skills are impacted by which

program

For example: Mentoring:

Seeing how long a child needs to attend

Longevity of Engagement

Seeing how often the child needs to attend

Life Skills and Attendance

Feedback on progress As well as using this scale to measure your own

life skills impact,

you can also prepare any or all of the above reports to feed back to stakeholders of every sort

In pairsDiscuss how you might use this scale in your

own setting

Feedback

Questions?

Goodbye!