Creating Value, Evaluating Impact

Kerry Traynor

November 2012

What is Evaluation and why do it?

About the Project

Measuring Impact

Theories and Models

Next Steps

‘Self-evaluation is a vital part of the discipline imposed in undertaking creative work’ (Moriarty, 2002)

What is evaluation?

Evaluation involves gathering evidence before, during

and after a project and using it to make judgements

about what happened.

The evidence should prove what happened and why, and

what effect it had.

The evidence can also help you to improve what you

are doing during the project and what you do next

time.

(Woolf, 2004 cited in Arts Council, 2011)

Why evaluate? ‘Self-evaluation is hard work and time-consuming. The reward is

that it can give us the ability to do things beyond the best of our present available knowledge’ (Moriarty, 2002)

evaluation helps with planning, as it makes you think about what you’re aiming to do, how you will do it and how you will know if you’ve succeeded

on-going feedback keeps you on track and helps to avoid disasters

evaluation helps you to adapt/change as you go along

evaluation is a good way of dealing with ‘quality assurance’ – you’re keeping an eye on things to make sure quality is maintained

evaluation helps prove the value of what you are doing

evaluation records your contribution to the field you are working in

your evaluation can help others working in the same field

information you collect can also be used for reporting back to those with an interest in the project (eg participants, funders) and telling others about what you’ve done

the evidence you collect can support future funding applications

(Arts Council, 2011)

Evaluation helps the arts…

Rigorous evaluation of our work enables us to :

accumulate a collective body of evidence

contributes to the ‘collective practice wisdom of the sector’

builds a record of our ‘history and achievement’

(Arts Victoria, 2002)

Cannot evaluate all projects in the same way and in the same depth

Can be more or less formal, and more or less detailed

Process as well as product - quality and impact of both

Artistic judgements about process, materials, form and content

Judgements about the results of what you did and what you have produced

(Arts Council, 2011)

Planning your evaluation:

What kinds of information or evidence you are going to include in your evaluation

e.g. what people say, what they have done (process and finished work), what you have done, how participants/audiences responded?

What questions you are going to ask?

How you plan to answer those questions –what sort of information you need to answer the questions and how you will collect it.

When you should collect the information.

How you will collect the information?

e.g. keeping a register, asking people in a questionnaire, asking them to video their thoughts about a project, keeping a diary, taking photographs, etc.

How you are going to make sense of the information you have collected?

How you are going to present the results of the evaluation?

Who you are going to share it with and how?

(Arts Council, 2011)

About the Project

Awards for All funding / Nov 12 – Aug 13

Aims Understand how impact is currently measured and evaluated in small enterprises in

the creative industries

Understand the strengths/weaknesses/opportunities/risks of evaluating impact

Review models of impact evaluation and identify appropriate model(s) for use in creative communities network

Identify needs and deliver support to evaluate impact more effectively

Outcomes ◦ Impact Evaluation Strategy for Safe Productions and the Creative

Communities Network – finding a shared approach that can measure the impact of the whole network

◦ One-to-one support for members in designing and implementing your own impact evaluation strategies

◦ Interactive blog / social media site / web archive of useful info

Background

Director / Chair of Safe Productions for 10 yrs

Charity Manager in Toxteth for 6 yrs

Grants Assessor for BBC CIN for 10 yrs

Researcher & lecturer at LJMU for 8 yrs

Manager of Community Media Enterprise

MSc in Governance (Creative Industries)

PhD in Impact of Local Media

What do we mean by impact?

Inputs + Outputs = Outcomes

Outcomes – What would have happened anyway = Impact

Difference achieved by a project/service

Social, environmental, financial

Exercise

Introduce yourself and your organisation

What do you believe are the main impact(s) of your organisation?

What is your experience of impact evaluation?

What has worked well for you?

What hasn’t worked so well?

What would you like to do better?

What do you hope this could achieve?

What are your fears / concerns?

Models of Impact Evaluation

Social Return on Investment

European Foundation for Quality Management (EFQM) Excellence Model

Prove It?

Quality First

Social Enterprise Balanced Scorecard

SIMPLE

3rd Sector Performance Dashboard

AA1000 Assurance Standard

The Big Picture

Volunteering Impact Assessment Toolkit

ISO9001:2008

Global Reporting Index (GRI)

Investors in People (IIP)

Eco-Mapping

EU Eco-Management and Audit Scheme (EMAS)

PQASSO

Exercise

Look at one of the models

Identify 3 or 4 key points

Think about how this could be

applied in your organisation

Discuss the pros and cons

Social Return on Investment

Inputs : £20,000

Outputs : 20 unemployed people complete a 6 month training programme and get a qualification

Outcomes : 5 people get a job

But : 2 people would have got a job anyway

Impact = 3 people move into employment

Attach monetary values (returns) to that impact e.g.

◦ Reduced costs to state of paying unemployment benefits

◦ Increased income to the state from employment taxes

Calculate returns over 5 years

Divide total returns by investment e.g.

◦ £100,000 / £20,000 = 5:1

Or, for every £1 invested, £5 is created in benefit for society

But does that really capture impact… ?

The Challenges…

Creativity / the Creative Industries

Individual > Social > Community

Community development

Pride, identity, shared vision

Perceptions of a community

Challenging negative, promoting positive

“Feelgood factor”

Social and Cultural Capital

Health & Wellbeing

References

Moriarty, G (2002). Sharing practice: a guide to self-evaluation in the context of social exclusion. London: Arts Council of England. This guide focuses on processes which can be used by arts organisations, individual practitioners and participants to reflect on and develop their practice. It also discusses why self-evaluation is important for all artists and arts organisations. Available at www.newaudiences.org.uk , News, July 2003.

Arts Victoria (2002). Evaluating Community Arts and Community Well Being. Available at www.arts.vic.gov.au , ‘publications’. This guide presents a complete approach to evaluating community arts work, using a framework based on process, impact and outcomes. It has practical guidance on planning, conducting and presenting an evaluation. There are lots of sample tools, such as sample evaluation indicators, a focus group outline, a sample participant questionnaire and project journal guidelines.