Care to Share?

Capturing learning from the

Short Breaks Fund

Graeme Reekie, Evaluation Support Scotland

0131 243 2770 graeme@evaluationsupportscotland.org.uk

Background

• Learning:

Funded Groups

Shared Care

Government

Carers and cared-for

“What works, for whom, in what circumstances?”

The Plan!

• Quick reminder on outcomes

• Measuring soft outcomes – (using indicators)

• Evaluation tools, including participatory

• Evidence

• Reporting

• Planning for action

Reminder

Outcomes:

• The change or difference

• That ‘comes out’ of your activities

Tips:

• Use change words (e.g. increase, reduce)

• Ask ‘So What?’

IndicatorsWant to:

• Involve users?

• Spread the burden?

• Strengthen your evidence?

• Make better use of evaluation tools?

Here’s how…!

Soft outcomes

Need to break down into proxy indicators

Take each outcome.

What would it look like if we achieved it?

Pick 2-3 most important,

most likely, easiest to

collect

Draw out the methods

Indicators must be …• Specific

• Simple

• Recognisable to your service users

• Able to be measured more than once and show change

• Ideally you need a baseline in order to measure progress or change.

Writing from the perspective of the user

If it helps, write indicators from the perspective of the user:

I feel …

I do…

I can ….We can…They can

5

I don’t feel sad as much as I used to4

I can keep my temper when I need to3

I find it easier to deal with the tough stuff going round my head

2

I am less scared about things than I used to be1Young people are better able to manage

their anger and frustration and anxiety

B

If I have to I find it easier to stand up to people who are mean to me.

5

I have made new friends.4

I can take part in activities.3

I make suggestions2

I feel surer of myself and less nervous about doing stuff1Young people feel more confident

A

Outcome indicatorOutcome

Common sources of information on indicators

Assessment

Observed behaviour changes

Hard evidence

Records

Third party tells you

They tell you

Outcome

Common methodsThey tell youThird party tells you

Behaviour changes

Records

Hard evidence

Assessment

Record casual feedback, Questionnaires, Interviews, Focus groups, Video diaries, Exercises, mapping and visual progress tools

Observation notes

Attendance sheets, Record of activities, Use of services

Tracking what happens, Comparing statistics (eg illness, employment, debt, truancy)

Badges, checklists, standards

Traditional methods

People tell you:

• Questionnaire

• Interview

• Focus group

Either stand alone

OR

Part of activities

Questionnaires • Your questions must relate to your outcomes and

indicators.• Short and simple - one piece of paper. • No leading questions! (‘How good is the service?’).• Think about closed or open questions. • Think about scales: numbers, smiley & sad faces etc• Stamped addressed envelopes and give a deadline.

Offer rewards for replying.• Do you need to get information from everyone or can

you select a sample. • Try it out first to check it makes sense (piloting).

InterviewsTips for group interviews:• Your topic guide must relate to your outcomes and

indicators.• If possible use a colleague to facilitate if you need very

personal feedback.• Give info in advance about the times, venue, expenses.• Provide refreshments, introductions, ice breakers.• Keep the discussion moving on when issues have been

addressed.• Do not let one person dominate the group, invite less

vocal people to offer their views.• Include group interview as part of an existing group

activity?

Other methods

Observation Group discussion (set and measure indicators) Diaries & video diaries Third party feedback (referrers, family

members etc) Visual representations of progress – eg trees,

wheels, dart boards, ladders Body maps Time lines

More info on our website

Relationship map (social network)

At the start of the project At the end of the project

XX

Liz

Mum

Carol

Tom

Mum

Liz

Carol

Tom

Dad

Callum

Di

Mary

Susan

Andrew

Body maps: Example 1

Before participation

After participation

Understand what units of alcohol are

Understand Understand the effect of the effect of alcohol alcohol on on the the liver brain Know how to The trainer prevent drinks worked well from being with the spiked group The information was clear

Balanced Wheels

1. Ask people to shade portions of a circle

2. Could represent the importance of different things to them

3. Or time spent

money

friends

family

relationships

education

work

hobbies

health

Exercise

The Short Breaks Fund generates increased understanding of the role of short breaks and respite in supporting caring relationships.

What would that look like?

How do you like your evidence?

• Poached?

• Scrambled?

• Hard boiled?

• Fried?

• All in one basket?

Jigsaw or mosaic?

• Evidence comes from a variety of sources

• Your can build pictures using those different sources

So that you can identify:• what outcomes• for whom• in what circumstances• and why

Where do you get evidence for your

decisions?

The last person you spoke to?

The Esoteric Journal of Perpetual Enquiry, vol 5349?

Take your evidence and TREBL it!

• Transparency: methods clear, limitations acknowledged

• Relevant: up to date, appropriate• Enough: strength of evidence v proportionality• Believable: accurate, representative, reliable• Legitimate: coming from the right sources

What goes in a report?

• Original aims• Planned outcomes and activities• Headline achievements• Main facts about activities• Main facts about outcomes• What went wrong and why? • Case studies• Learning and changes for the future• Financial information• Appendices

Scotland Funders’ Forum

The best reports come from organisations that can

• evidence their work

• tell their story

• use reporting to reflect on what they have achieved and learned

(Scotland Funders’ Forum: Harmonising Reporting Working Group 2010)

Summary

• Outcomes: What ‘comes out’? So What?• Indicators: What would that look like?• Tools: Traditional, maps, wheels.• Evidence: TREBL • Reports: Telling the story

• More help on our website:• www.evaluationsupportscotland.org.uk