08/07/2016

1

Evaluating Health Interventions

Jo Mackenzie Public Health Analyst Trainee Health Psychologist Hertfordshire County Council E-mail: jo.mackenzie@hertfordshire.gov.uk

Learning objectives

• To be able to:

– Demonstrate why evaluation is important

– Define aims and objectives

– Write SMART objectives

– Recognise the different types of evaluation

– Describe the steps of evaluating within the project development cycle

08/07/2016

2

What we will cover today

• What evaluation is and why we do it

• Evaluation in relation to the project development cycle

• Writing SMART objectives

• Using the Logic Model Framework

• Evaluation measures

• Data confidentiality and participant consent

How would you describe evaluation?

and

How do you currently feel about

evaluation your projects?

08/07/2016

3

What is evaluation?

• Basic part of good project management

• Evidence of success of an intervention

• “Did I achieve what I set out to do?”

• “Did the intervention make the changes it set out to make?”

• Did it meet the objectives?

• It is about making judgements about the intervention

Evaluation myths

• Requires expensive consultants

• Not just counting everything

• A report on the costs of the project

• Something that can be done at the end of a project

08/07/2016

4

Why evaluate?

• Essential for improving public health programmes:

– Should we run this intervention again?

– To find ways to improve a campaign or intervention.

– Justify the cost.

– Is the programme effective?

– Did anything not work?

Purpose of evaluation

• Should be able to complete the follow sentence

“The evaluation is for the funders. To answer the question ”Does the intervention improve health and wellbeing?” In order that the user make better decisions about “repeating this intervention on a larger scale”

08/07/2016

5

When to evaluate

• Evaluation needs to be considered before or during the intervention design

• Evaluation takes place at the planning stage

• This can:

– enable the outcome and process measures to be matched to the intervention’s theory of change

– inform the design of the intervention.

Project development cycle

1. Planning (why, who, what , when)

2. Clarify objectives &

outcomes

3. Select indicators & sources of

evidence

4. Design methods & collect data

5. Analyse data

6. Reflect on practice & share

learning

08/07/2016

6

1. Planning

• Short document or project brief outlining: – The project title

– The aims and objectives • Outputs to be measured

• When will they be measured

– Funding source

– The health issue to be addressed

– The prevalence of the health issue

– Target population

– Description of the intervention

Planning cont….

– Evidence for the approach to be taken

– How will you recruit participants?

– Resources available • Physical resources

• Staff

– Stakeholders involved

– Intervention timescales

– Intervention delivery dates

– Location and setting

– Who will be responsible for the evaluation

– Ethics and Consent

08/07/2016

7

Intervention design

• Different ways to design interventions

• Dependant on available resources

• We will focus on a simple, before and after design

2. Setting the objectives

We cannot evaluate without well written aims and objectives! These need to be agreed with everyone involved before the project starts.

08/07/2016

8

To run a workshop on healthy eating

Objectives versus activities

• Objectives and activities are often confused

–Objectives state what we want to achieve

–Activities state what we will do to meet out objective(s)

08/07/2016

9

By the end of the workshop, participants will be able to identify key components of a healthy meal

Aims versus objectives

• Aims – Describes what the

project/intervention hopes to achieve

– The planned affects

– Very general

– Need to be testable

– Normally one aim

• Objectives – Very specific

– Must be measureable

– State the outcome to be achieved

– Give the criteria for deciding the outcome has been achieved

– The target population

– 3 – 5 objectives are recommended

08/07/2016

10

Writing objectives

• Objectives help us answer our evaluation questions

• What is a SMART objective? – S - specific, significant, stretching

– M - measurable, meaningful, motivational

– A - agreed upon, attainable, achievable, acceptable, action-oriented

– R - realistic, relevant, reasonable, rewarding, results-oriented

– T - time-based, time-bound, timely, tangible, trackable

• When writing SMART objectives it is easier to use the following order:

– M - measureable

– A/R – achievable/realistic

– S – specific

– T - timely

08/07/2016

11

Measureable

• How will you measure this objective?

• Some objectives are easier than others

• What tools could you use?

– What have other interventions in this area used?

• Where can you find suitable measures?

• Some measures are licenced and have an attached cost.

• How will you collect the data?

Achievable/Realistic

• Achievable and realistic work together

• Is your target achievable? – It is highly unlikely you will achieve a 100% success

rate

– How much change do you expect from your target group?

• What is a realistic expectation? Consider: – Budget

– Time

– Resources?

08/07/2016

12

Specific

• Be clear and do not use jargon

• Does the objective mean the same to everyone involved? – There should be no confusion here

• Should state – What

– Why

– Where

– When

– Who

Time

• This is about setting deadlines

• Use dates or specify a set time period.

• Need a clear start and end date

– Extending the time period makes the objective non-specific!

08/07/2016

13

SMART objectives exercise

Evaluation questions

• The evaluation question will determine the approach and design of all the data collection activities.

• Question examples include: – Has the health of the target group improved?

– Has the expected percentage of the target group adopted the health behaviour changes?

– Has the uptake of a health service increased by the desired amount?

08/07/2016

14

Inputs, outputs and outcomes

• Inputs – these are your resources

• Activities – What did you do for the intervention?

• Outputs – monitoring data e.g. how many people participated?

• Outcomes – Short, medium and long term effects of your intervention.

Logic Models

• Tells the story of the intervention in a condensed format and links:

– Resources or inputs

– Activities

– Outputs

– Outcomes

• Informs monitoring and evaluation

08/07/2016

15

Logic-Model Framework

Inputs Activities Outputs Outcomes

Human, financial or

material resources

used in the intervention

What activities

were carried out for the

intervention? E.g training

What did you see happen?

What was the effect of

this intervention?

Short term Medium

term Long term

Context

What is the problem?

What does

the evidence show?

Start here

3. Selecting indicators

• Key measures are indicators – They indicate if the project/intervention has made

the change/impact it was supposed to make.

– Linked to the SMART objectives

• Referring back to the logic model we can have: – Process indicators

– Outcome indicators • Short term

• Medium term

• Long term

08/07/2016

16

Types of Indicators

• Process indicators

– Monitor the progress of the project/intervention

• Outcome indicators

– Short term – Immediately at the end of the project. E.g. attitudes and knowledge

– Medium term – follow ups at 6 months/1 year. E.g. behaviour changes

– Long term - measurable outcome e.g. PHOF indicator

4. Design and data collection

• The design of your evaluation will depend on

– Your evaluation questions

– Your selected indicators

• How will you collect your data?

• How do you want to use the collected data?

– Who will be the audience of that data?

08/07/2016

17

Types of evaluation

• Outcome

– Assesses if the project met its intended outcomes

– What people think of when discussing evaluation

• Process

– Describes what happens when a project takes place

– Overlaps with monitoring

• Formative

– Carried out in the planning stage

– Used to develop an intervention

Types of data

• Quantitative – Required to measure the outcomes

– Uses statistical techniques to determine outcome

• Case studies – Participant stories which highlight the impact of

the intervention

• Qualitative – Can be used as part of the monitoring process

– Captures the experiences of the participants

08/07/2016

18

Data to collect - outcome evaluation

• This will relate to your question, aims and objectives

• Questionnaire data

– Improvements in wellbeing

– Weight loss

– Increased physical activity

– Reduction in alcohol consumption

– Increased use of an existing health service

Data to collect – process evaluation

• Number of enquiries about an intervention

• Participants signed up to an intervention

• Attendance rates

• How many from deprived backgrounds

• Number of people completing an evaluation form.

• Participant feedback on the intervention

08/07/2016

19

Monitoring data

• Demographics – helps your to analyse and compare groups within your data – Sex

– Age group

– Ethnic group

– Socioeconomic status

– Area of residence

• Attendance rates

• Number of evaluation forms returned

Exercise

In groups discuss what measures you are already using for your interventions

What are your reasons for your choices?

08/07/2016

20

Outcome measures

• Dependant on the type of intervention – Weight Management

• Weight (KG)

• BMI

• Change in consumption of food types

• Attitudes towards food consumption

– Physical Activity • Single item measure (7 day recall)

• Incidents of exercise per week

• Type of exercise

• Benefits of physical activity

Evaluation measures cont..

– Mental health/wellbeing • (S)WEMWBS - Wellbeing • GAD7 - Anxiety • PHQ9 – Depression • SF36 – Quality of Life • SF12 – Quality of Life (shortest version)

– Dependant on the target group • Children • Working age adults • The Elderly • Adolescents

• Beware that some measures have costs!

08/07/2016

21

When to collect data

Baseline

•Pre-intervention

•Demographics

Post intervention

• Immediate

Post intervention

•Follow up (e.g. 3 months after end of intervention)

This is a basic example, you can have more data collection points and it is recommended if you are carrying out a large intervention

If we do not collect this data before starting our intervention we have lost the chance to evaluate well!

Data collection methods

• Methods – Interviews

• Face to face • Telephone

– Online surveys – Postal survey – In person beginning and end of course surveys – Focus groups

• Consider how you are going to get the best response rate – Make it easy for them!

08/07/2016

22

5. Analysis

• Who will carry out the analysis? – They should be involved before any data collection

– What tools do they need for this?

• Level of analysis – How in depth will this be?

• What are you going to compare?

• Do you require statistical tests?

• How will you present the data?

• What are the limitations of the data?

Data confidentiality

• Need to carry out a Privacy Impact Assessment

– Participants personal data is sensitive data.

– Make this anonymous as soon as practically possible.

– Always store in a secure location

• Check your organisation’s data protection rules.

08/07/2016

23

Participant consent

• When collecting data from participants you must get their consent. – You need to let them know that

• The information they give will be keep anonymous

• The data will be stored securely

• What you plan to use the data for

• That they may withdraw consent at any point before, during or after the intervention – Let them know how to do this.

– Participants have the right to say no!

6. Reflection

• What were the findings

• Conclusions

• Final report

• Sharing the information

• What have you learnt

08/07/2016

24

Questions??

Resources and support

• Evaluation guides by PHE – http://www.noo.org.uk/core/Frameworks

• Mental health and wellbeing measures – http://www2.warwick.ac.uk/fac/med/research/platform/wemwbs – http://www.euroqol.org/ – http://www.phqscreeners.com/

• Guide for measuring diet and physical activity interventions – http://www.noo.org.uk/core/frameworks/SEF_PA

• Ethics – http://www.hra.nhs.uk/documents/2013/09/defining-

research.pdf

• Hertfordshire Health Evidence – www.hertshealthevidence.org