v

MEASURING IMPACT: USING QUANTITATIVE RESEARCHEibhlín Ní Ógáin

London, November 2012

AGENDA

1 What is impact measurement

2 NPC’s approach

3 Quantitative approaches

4 Tips and things to remember

TRANSFORMING THE CHARITY SECTOR

Increasing the impact of charities

eg, outcomes frameworks

Charity

Funder Sector

We are a think tank and

consultancy working at the nexus between charities and

funders

Increasing the impact of funders

eg, effective commissioning

Strengthening the sector

eg, collaboration towards shared

goals

WHAT IS IMPACT MEASUREMENT?

5

Impact: The difference an organisation or activity makes to the people or problem it aims to help

Impact measurement: Planning, managing, tracking and reviewing how much difference you are making

NPC’S APPROACH TO MEASURING SOCIAL IMPACT

7

NPC’S APPROACH TO MEASURING SOCIAL IMPACT

Strategic vision / goals

A well-developed Theory of Change

Existing evidence Appropriate measurement tools

QUANTITATIVE RESEARCH

8

9

WHY USE QUANTITATIVE RESEARCH

• Qualitative: understand something in detail. Rich information and insight.

• Quantitative: how many people? How large an effect? How much impact and how robust?

• Use quantitative research when dealing with large numbers of people and when you want to show a consistent effect.

• E.g does new type of psychotherapy lead to improved mental well-being? Need consistent measure and need large sample.

• Depends on type of ‘intervention’ and questions you are trying to answer.

• Know a lot, measure a little, know a little, measure a lot.

10

DIFFERENT METHODS IN QUANTITATIVE RESEARCH

• Ranges from easy to measure to hard to measure

• Easier: Hard and objective outcomes: move into employment, gain a qualification, reduction in reoffending.

• Harder: Softer, difficult to observe outcomes: changes in feelings, personal growth, well-being.

• NPC’s preference is for previously tested scales and questionnaires.

• If there is a gap, develop own questions.

11

TYPES OF QUANTITATIVE TOOLS AND THEIR USES

11

Type of toolIdentify needs of group

Measures progress for individuals

Aggregate to show change

Robust measure of change

Explain why change has happened

Admin. data ? ? ? Case work tool (eg, Recovery Star) ? Clinical tools Scales (developed by researchers)

Single questions

Hard outcomes (eg, statutory data)

12

WHEN TO MEASURE

• Pre-post design: how outcomes change over time by collecting data before (pre) and after (post) the programme.

• When you think change happens influences your decision about:

– When to collect the before measure

– When to collect the after measure

12

Before measure

After measure

Programme / service

13

THINK ABOUT ATTRIBUTION

• Is the programme the only factor influencing any changes? What would have happened anyway?

• Control groups

• Compare before and after data for participants and non-participants, ideally matching on their key characteristics

• Holy grail: Randomised Control Trial, where potential participants randomly assigned to control group

13

14

THINK ABOUT SAMPLING

• Representativeness

– Does your sample represent your population?

• Response rates

– Is your response rate high enough to avoid bias?

• Statistical validity

– Are your sample sizes large enough for analysis?

• Oversampling (segmentation)

14

Biased/small sample => Avoid percentages

15

YOU DON’T NEED TO MEASURE EVERYTHING

16

WHAT: BEFORE YOU THINK ABOUT COLLECTING DATA ASK YOURSELF:

Has anyone already proved the causal link you want to make?

Yes

No

Is it really important you need data on this outcome?

Do you already collect this data?

Don’t collect unless you decide it is really important

Use research to limit your data collection

Yes

No

Yes

Can you access this data?

No

Develop your own data source

Use this data (if it is fit for purpose)

Does anyone else collect this data?

NoNo

Yes

Yes

17

PRIORITISE OUTCOMES THAT:

• you directly influence (rather than indirectly support)

• are important / material to your mission

• are not too costly to measure

• will produce credible data?

18

HOW ROBUST DOES EVIDENCE OF YOUR SOCIAL IMPACT NEED TO BE?

• What do your target stakeholders (investors?) think?

• What is possible, given your resources etc?

19

LEVELS OF EVIDENCE

Randomised control trial

Anecdotes / quotes

Before and after survey

Self-reported change

Case studies

Control groups

Credibility

Basic Advanced

Nesta level 1 Nesta level 2 Nesta levels 3, 4, 5

20

6 TIPS FOR MEASURING YOUR IMPACT

2. Keep it simple. Concentrate on what you really want to know and don’t try to be too ambitious in the scope of your evaluation.

1. Decide what’s important. Define your research question and be clear what you want to achieve.

3. Know your limits. You need to choose an approach that matches your resources.

4. Consider investing in training. If you are not confident, get training or seek external help.

5. Think about the future. Don’t build a system that will quickly go out of date and look at the year-on-year costs.

6. Use the information that you collect: If you don’t use it, don’t measure.

21

RESOURCES

• Review of evidence of ICT and learning: http://educationendowmentfoundation.org.uk/toolkit/ict

• A good website to look for measures and indicators: http://wilderdom.com/tools/ToolsSummaries.html

• How to communicate your results: Hedley, S et al (2010) Talking about results. New Philanthropy Capital.

• Bradburn et al. (2004) Asking questions: the definitive guide to questionnaire design. Jossey-Bass: San-Francisco.

• http://www.theoryofchange.org/• The innovation network’s logic model workbook:

http://www.innonet.org/client_docs/File/logic_model_workbook.pdf• The Centre for What works (http://www.whatworks.org/) has an outcome portal where

you can browse different outcomes and their corresponding indicators. • TRASI (http://trasi.foundationcenter.org/) has a database where you can search

different approaches to impact assessment. • NPC has a page on shared measurement where it outlines some indicators for different

sectors. http://www.philanthropycapital.org/publications/improving_the_sector/measuring_together.aspx

22

RESOURCES

• Survey question bank: A website where you can research widely used surveys and single questions by theme or using key word searches. http://surveynet.ac.uk/sqb/

• CES’s guide to theory of change: http://www.ces-vol.org.uk/downloads/makingconnectionsusingatheoryofchangetodevelopplan-800-808.pdf

• Forum for Youth Investment’s guide to measuring soft skills in youth programmes: http://www.forumfyi.org/files/Soft_Skills_Hard_Data.pdf

• A list of the most common psychological scales and questionnaires: http://www.ull.ac.uk/subjects/psychology/psycscales.shtml

• Harvard Family Research Project’s tools for out-of-school time programmes: http://www.hfrp.org/out-of-school-time/publications-resources/measurement-tools-for-evaluating-out-of-school-time-programs-an-evaluation-resource2

• Bradburn et al. (2004) Asking questions: the definitive guide to questionnaire design. Jossey-Bass: San-Francisco.

• Ritchie et al. (2003) Qualitative Research Practice – A Guide for Social Science Students and Researchers. Sage Publications Ltd

• Measurement and assessment tools for family and parenting interventions: http://www.actionforchildren.org.uk/media/3986553/outcome_tools_1.pdf

vEibhlín Ní Ógáin-eibhlin.niogain@thinkNPC.org