Measuring Impact

An Engage Workshop on Monitoring & Evaluation

Workshop Aims

• To deepen ROEYP grant holders’ understanding of monitoring and evaluation.

• To assist grant holders in starting to identify appropriate methods and means of measuring the impact of their work.

• To help ROEYP grant holders network and make connections between their projects

Intended outcomes for workshop

 • Increased confidence in using the language/

terminology of monitoring & evaluation• Greater understanding of evaluation methods and

means to measure impact• More connections between projects funded under the

ROEYP Programme

Workshop overview• Why measure impact (evaluate)?• Definitions & clarification of terms• Learning from other participants: approaches & tools Evaluation Template, Youth Concern Rickter Scale, Include Youth, Wemwebs, YANI Assessment Tool based on Farrington’s Factors,

NIACRO• Involving young people in monitoring & evaluation –

some ideas• Discussion: questions, queries and challenges• Resources, evaluation & close

Why measure impact (evaluate)?

Some reasons ...• Accountability (is the money being spent as it

should be?)• To find out what is the impact of the project • To explore how the project could be improved• To understand user perspectives (what is their

experience of the project?)• To help with decision-making about the project (cut,

or continue – all or part?)• To inform the replication or expansion of a pilot

project• To inform future marketing or publicity about the

project

What is evaluation?

• Derived from the word ‘value’ (how much? how good?)

• Relates to a specific project, programme or service, at a particular point in time

• A series of planned and systematic activities• Has a clearly defined purpose• Gathering evidence in order to make

judgements

• Multi-method (typically)• Multiple perspective (if possible)

Evaluation: a working definition

• Evaluation consists of a set of systematic, planned activities designed to assess the worth or value of a project or programme.

Over-arching evaluation questions

• What / how much did we do? (monitoring)

• How well did we do it? (process)

• Did we make a difference? (outcomes/impact)

Monitoring

• Underpins any evaluation • Answers the ‘what/how much/how many?’ question• A systematic record of relevant information about the

project• Recording ‘inputs’ and ‘outputs’• Keeping track of project progress• Relevant information about service users• Monitoring data will usually be in numerical form

(quantitative) • Monitoring is not simply collecting this information, it is

about using it (reviewing, reflecting, comparing, analysing)

Outcome / impact (evaluation)

• Answers the ‘did we make a difference?’ question• Assessing the ‘difference’ that the project has made – eg

to users (and others?)• Outcomes and impacts need to be expressed in the

language of change (improve, decrease, enhance etc)• To measure change conclusively, a before-after model is

required (although there may still be issues of attribution)• Sometimes ‘softer’ assessments of change (eg qualitative

views, or retrospective accounts) may be more practical for a DIY evaluation

• It may be appropriate to consider longer-term or broader impacts (including unintended outcomes)

Methods of collecting evidence

• Adapt or extend existing systems• ‘Spoken word’ methods (face-to-face or phone,

individually or in groups)• Written methods eg questionnaires, diaries• Standard ‘tests’ or assessments• Observation• ‘Creative’ methods eg pictures, photos

Evaluation Template

• Ricky Linton• Youth Concern, Whitehead

The Rickter Scale

• Nicci Smith• Include Youth

Wemwbs

• Eliz McArdle• YANI

Assessment tool (bases on Farrington’s Factors)

• Tony Martin • NIACRO

• Evaluation does not have its own exclusive set of methods

• No one method is ‘better’ than any other, each have their advantages and disadvantages

• Important to select the most appropriate method(s) for your specific needs and circumstances

• Can mix and match methods for different aspects of your evaluation

• Different methods can provide complementary data

Methods of collecting evidence

When selecting methods, things to bear in mind

• Who are your informants? (age, skills, expectations)

• What kind of information are you collecting – Qualitative and/or quantitative?– Sensitive?– Complex?

• Likely numbers involved• Anonymity / confidentiality issues• Your own analysis skills / confidence• Resource implications• Time implications

Lunch

Involving children & young people in monitoring and evaluation

• Some ideas….

When to collect evidence?

• Ongoing (for monitoring data)• After the ‘intervention’ is complete (for

retrospective view)• Before and after the intervention (if you want to

measure change, or compare expectations with reality)

• Additional follow-up (if you want to look at longer term impact or outcomes)

• At other appropriate points eg midway point if you want to assess progress and make changes, or ‘snapshot’ to gather wide range of views at particular point in time

Analysis

The last minor (!) steps: • Analysis - making sense of all the information you

have collected• Keep it simple!

– Think back to your evaluation questions– Collate your data into a workable format– Describe the facts in a balanced way (include positives and

negatives)– Interpreting the facts – what do they signify and why?

Reporting

• Reporting – presenting it in a way that makes sense to target audience– Planned structure (logical sequence)– Be concise and keep it simple– Include evidence to support your findings (e.g frequencies or

percentages for quantitative data, quotations for qualitative, cameos for observations)

– Tables and graphs a useful way of presenting quantitative data

Measuring impact…

• Challenges• Concerns• Queries

Measuring impact

• Resources• Evaluation

Want more info?

Teresa Geraghty

tgeraghty@ncb.org.uk

028 90891733