Evaluation

Bruce Etherington15th May 2013

This presentation is developed from a number of presentations originally created by the National

Coordinating Centre for Public Engagement and the Beacons for Public Engagement through the HE STEM

Programme. http://www.publicengagement.ac.uk/evaluating-stem-

outreach

As such, this presentation is released under the same Creative Commons licence of Attribution-Share Alike 3.0

Unportedhttp://creativecommons.org/licenses/by-sa/3.0/

Aims of day

• To help develop a shared set of approaches to evaluating engagement across Wellcome Trust Centres

• To develop skills of participants to develop high quality evaluation strategies

• To help participants to make strong cases for engagement with research and for the evaluation of this activity

Timetable

9am Arrive (tea and coffee)9.15 Introductions9.20 Why Evaluate?9.40 How do I know what I am evaluating?10.30 Break10.45 How do I know what I am evaluating? (cont)11.45 Who is the evaluation for?12.30 Lunch13.30 Making the case for engagement (and evaluation of engagement)15.00 End

Introductions

• Who you are

• What experience you have in evaluation

• What you are hoping from the day

WHY EVALUATE?

1. Why evaluate?1. Why evaluate?

I had no idea that wormswere so important.

Beginner’s Guide to Evaluation

Why and Who of evaluation

Post up as many reasons as you can think of why we evaluate our activities on the WHY flipchart.

HOW DO I KNOW WHAT I AM EVALUATING?

Source: Ingenious evaluations: A guide for grant holders, The Royal Academy of Engineers

2a: What are you aiming to do?2a: What are you aiming to do?

Beginner’s Guide to Evaluation

1. Aim (what do you want to achieve? Big picture!)2. Objectives (what you need to do to achieve your aim?)3. Evaluation questions (what do you want to know?)4. Methodology (what strategy will you use?)5. Data collection (what techniques will you use to collect

your evidence?)6. Data analysis (how will you analyse your data?)7. Reporting (who will be reading your report?)

What goes in an evaluation plan?What goes in an evaluation plan?

Beginner’s Guide to Evaluation

• Pick an activity that you know well

• Pair up with someone you do not know and explain you activity to each other– Why you do the activity– What you hope to achieve by doing the activity

Activity

Inputs Outputs Outcomes

Basic Logic Model

Term Definition Example

Inputs Public sector resources required to achieve the policy objective

Resources used to deliver the policy

Activities What is delivered on behalf of the public sector to the recipient

Provision of seminars, training events, consultations etc.

Outputs What the recipient does with the resources, advice/training received, or intervention relevant to them

The number of training courses completed

Intermediate Outcomes

The intermediate outcomes of the policy produced by the recipient

Jobs created, turnover, reduced costs or training opportunities provided

Impacts Wider societal and economic outcomes

The change in personal incomes and ultimately wellbeing

HM Treasury Definitions (p22)

• Pick an project you are familiar and start to work out the steps of a logic model for it

• Consider:– Inputs – Resources Used– Activities – What the project did/does– Outputs – What the participants did/do– Intermediate Outcomes – What changed in the

participants– Impact – Wider societal effects

Your activity

• Understanding the theory of the change you are aiming for improves evaluation– You can see what you need to evaluate– You can see what you do not need to evaluate– You can see the assumptions you may be making

Logic Models & Evaluation

Pool of long term unemployed who

lack skills

Obtain Placements and undertake

training

Improve Qualifications and workplace skills of

attendees

Job Training scheme example

Obtain Interviews and

Job Offers

Increase in jobs and incomes

Lower overall unemployment

HM Treasury, Magenta Book, p23

• What evaluation questions might you want to ask about this project?– Are we promoting it sufficiently to the target

audience?– Are the training courses at the right level? – Are they improving the skills and qualifications of

attendees?– Are they getting more interviews? If not, why not?– Etc.

Job Training scheme example

Other Templates

Assumptions

Assumptions

Nick Temple/School for Social Entrepreneurs

Assumptions

Activities Immediate Effects

Medium term

outcomes

Long term impact

1. Analysis of the project’s Context 2. Stakeholder Analysis3. Problem Analysis/Situation Analysis4. Objectives Analysis5. Plan of Activities6. Resource Planning7. Indicators/Measurements of Objectives8. Risk Analysis and Risk Management9. Analysis of the Assumptions

Other Templates – Logical Framework

• Participatory approach to evaluation• Looks to understand the contribution of a

project to changes in practice of stakeholders• Needs skilled facilitation and a budget

Other Templates – Outcomes Mapping

• “So That” chains• UNDP template:– Identify the desired change– Identify the agents of change– Identify the assumptions– Pathways to Change– Indicators of Change

• Theory U (www.presencing.com)

Other Templates

• HM Treasury, The Magenta Book: Guidance for evaluation (2011)

• Annie E Casey Foundation, Theory of Change: A Practical Tool for Action, Results and Learning (2004) http://www.aecf.org/upload/publicationfiles/cc2977k440.pdf

• www.outcomemapping.ca• www.theoryofchange.org

References

WHO IS THE EVALUATION FOR?

Why and Who of evaluation

Post up as many audiences that you can think of for our evaluation work on the WHO flipchart.

What do these audiences want?

In groups consider the following questions:• Why is this audience interested in your

evaluation?• What are the top three things they would

want to know?• What are the things you, as the organiser,

would like them to know?

2. Making sure your evaluation is fit for purpose2. Making sure your evaluation is fit for purpose

Activity

Look at the example evaluation strategies.

Is the approach suitable for all the potential audiences of evaluation?What else could the organisers do to help improve their evaluation plan?

Read the feedback – do you agree/ disagree with the suggestions?

Corrosion Summer BallA family activity during the Manchester Science Festival with 4 table-top interactive experiences related to corrosion science. The aims of the activity were to:• inspire the general public with an introduction to corrosion.• communicate that corrosion is interesting and relevant to people's daily lives.• provide an exciting and memorable learning experience.• make universities more accessible to the general public.

What could be the possible outputs, outcomes and impact of this activity?

http://www.publicengagement.ac.uk/how/case-studies/corrosion-summer-ball

Outputs, outcomes and impact

Challenges of measuring impact

What are the key challenges to measuring impact?

Results

Behaviour

L e a r n I n g

R e a c t I o n

MAKING THE CASE FOR ENGAGEMENT AND EVALUATION

3. Making an impact with your evaluation

How can you make use of evaluation?

• Self reflection• Reports– Case studies and other formats eg presentations/ video/

audio etc

• Making a case for future funding/ support

What worked well? Why? What did not work well? Why not? What will I do the same next time? What will I do differently next time?

What are the key things you need to include in a report?

Reports

Other ways of reporting

Case studies/ Video etc

What are the pros and cons of using case studies as a way of reporting on your evaluation?

Making a case

• Things you can evidence–History of evaluative practice informing

development of activities–Learning (self/team reflection)–Approach is informed by target audiences –Effective practice–Commitment to future evaluation to

inform activity

Beyond the report

What are the opportunities for sharing your evaluation with others?

• On your website• With funders• With partners• With others e.g. NCCPE; Collective Memory

Top tips• Think about your audience• Develop your evaluation plan at the beginning• Don’t collect data you can’t use• Beware of misrepresenting your data• Back up qualitative data with quantitative data• Don’t hide mistakes – learn from them• Reflect on what you would do differently next time• Recognise the challenges of measuring impact• Be realistic about what you can measure• Remember the value of using evaluation during the

project• Share what you have learnt

Evaluation can always have an impact.....

........if you let it

NCCPE http://www.publicengagement.ac.uk/how/guides/evaluation/resources

Manchester Beacon Evaluation Guide http://www.manchesterbeacon.org/about/

UCL Evaluation Toolkithttp://www.ucl.ac.uk/public-engagement/research/toolkits/Event_Evaluation

RCUK Evaluation Guidehttp://www.rcuk.ac.uk/documents/publications/evaluationguide.pdf

HE STEM http://www.hestem.ac.uk/evaluation

Inspiring Learning for All http://www.inspiringlearningforall.gov.uk/toolstemplates/

Useful Resources