Performance Measurement in the Conservation Community Status, progress, barriers, and next steps...

Performance Measurementin the Conservation Community

Status, progress, barriers, and next steps

Elizabeth O‘Neill, Conservation Auditor, WWF International

& Matt Muir

on behalf of the

Measuring Conservation Effectiveness Summit Research Committee

INTRODUCTION

Purpose: Survey of 29 conservation implementers and funders to ask about their Systematic Performance Measurement (SPM) practice.

Implementors:• “OECD-DAC principles for evaluation”

• “annual plans and performance reviews”

• “peer-review audits”

• “implementation of the Open Standards”

Funders:• “communication tool for our board”

• “rigorous post-grant monitoring”

• “constant and direct assessments of progress”

• “qualitative approach to evaluate grantee results”

Plan

Adapt Do

Check

INTRODUCTION

Respondents:

15 conservation implementers: • ~2.1 billion US$

• ~7,000 projects

• 16,000 staff

14 conservation funders: • ~1.1 billion US$

• ~1,000 grants

• 578 staff

1. SPM is widely acknowledged as important...

2. ...but is not done to a great extent across our community…

3. …however, we can improve by leveraging the progress we have made…

4. and overcoming several critical obstacles.

What we learned:

INTRODUCTION

SPM IS IMPORTANT IN CONSERVATION

1. SPM is widely acknowledged as important in conservation.

SPM is generally viewed favorably and as a priority by most managers and boards.

SPM IS IMPORTANT IN CONSERVATION

Strongly disagree

Strongly agree

Moderately disagree

Moderately agree

? ?

SPM IS IMPORTANT IN CONSERVATION

When SPM is implemented, we say the primary driver is to improve effectiveness.

Do we understand why projects fail?

Are actions cost-effective?

Are we having intended impacts?

Are we adapting and improving?

What can be learned to improve?

Can we demonstrate credible results?

85 to 90% of all respondents say answering these

questions is important

FUNDERS

Do we understand why projects fail?

Are actions cost-effective?

Are we having intended impacts?

Are we adapting and improving?

What can be learned to improve?

Can we demonstrate credible results?

% of respondents

SPM IS IMPORTANT IN CONSERVATION

Not at allVery wellMinimally

wellSomewhat

well

0 100

FUNDERS ORGS

Do we understand why projects fail?

Are actions cost-effective?

Are we having intended impacts?

Are we adapting and improving?

What can be learned to improve?

Can we demonstrate credible results?

SPM IS IMPORTANT IN CONSERVATION

Not at allVery wellMinimally

wellSomewhat

well

% of respondents0 100

FUNDERS ORGS $$$

Do we understand why projects fail?

Are actions cost-effective?

Are we having intended impacts?

Are we adapting and improving?

What can be learned to improve?

Can we demonstrate credible results?

SPM IS IMPORTANT IN CONSERVATION

Not at allVery wellMinimally

wellSomewhat

well

% of spending0 100

SPM IS NOT DONE WIDELY

2. That we often cannot answer key performance questions with confidence is likely because...

...SPM is not done to a great extent across our community.

For every conservation dollar spent, we say that only 10-30 cents worth is guided by SPM…

SPM IS NOT DONE WIDELY

Survey Question: Of the total budget for your organization’s conservation efforts, what % is guided by SPM?

…and only about 5% of projects go through a full SPM cycle (i.e., plan-do-check-adapt).

only ~350-500

only ~2500

Of ~7000 projects

have good conservation plans in place, but…

currently undertaken by implementing organizations,

have completed the SPM cycle

SPM IS NOT DONE WIDELY

By $$$

Practice tends to break down at certain places in the cycle

DESIGN

IMPLEMENT

MONITOR

EVALUATE& ADAPT

SHARE

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

KEY INGREDIENTS TO PROGRESS TO DATE

3. To improve our ability to measure effectiveness, we can leverage several ingredients key to our progress to date.

SPM:

The importance of having a mandate

Key ingredients to the progress we’ve made to date:

KEY INGREDIENTS TO PROGRESS TO DATE

(1) Institutional mandate

(2) Presence of an SPM champion

(3) A vision for what could be accomplished with SPM

(4) Evidence that SPM helps

(5) Dedicated SPM staff

(6) Dedicated SPM funding

CRITICAL OBSTACLES TO PROGRESS

4. To improve our ability to measure effectiveness, we also must overcome several critical obstacles.

CRITICAL OBSTACLES TO PROGRESS

OBSTACLE 1: LACK OF TIME TO DO SPM

• 90% of respondents indicate this is an extreme or major obstacle

CRITICAL OBSTACLES TO PROGRESS

OBSTACLE 2: LACK OF MONEY

Most implementers indicate they invest no more than 1-5% of total conservation spending in SPM.

By Org

% of annual conservation spending

OBSTACLE 3: LACK OF STAFF DEDICATED TO SPM

CRITICAL OBSTACLES TO PROGRESS

578foundation

staff

By Funders

19 dedicatedSPM staff

(30:1)

By Org

16,400organizational

staff 71 dedicatedSPM staff

(230:1)

CRITICAL OBSTACLES TO PROGRESS

OBSTACLE 4: FEELING THAT SPM IS TOO COMPLEX

Strongly disagree

Strongly agree

Moderately disagree

Moderately agree

OBSTACLE 5: LACK OF PRESSURE/DEMAND FROM ‘ABOVE‘

By Org By $$$

CRITICAL OBSTACLES TO PROGRESS

Smaller organizations seem to do SPM to a greater extent than larger ones.

Is SPM practiced well at your organization?

Key obstacles identified:

• Lack of donor pressure

• Lack of board pressure

• Lack of demand from upper management

Insufficient time + Weak funding + Few staff + Limited understanding + Lack of demand from ‘above’ =

An apparent low priority placed on SPM

Unanswered research questions:

Why is SPM a low priority? What are root causes?

What will it take to make major advances?

CRITICAL OBSTACLES TO PROGRESS

1. SPM is important in conservation

We say it is important and a top priority.

We want to be able to answer key questions regarding effectiveness.

But we generally cannot answer these questions.

2. SPM is not done widely

Only 10-30% of current conservation spending is guided by SPM.

Very few projects do more than basic design plus implementation.

Things break down at more rigorous design and monitoring, evaluation, adaptation, and sharing learning.

CONCLUSIONS

Performance Measurement in the Conservation Community

3. We can improve by leveraging key factors to progress to date...

Performance Measurement in the Conservation Community

4. ...and by overcoming several critical obstacles.

CONCLUSIONS

Time

Money

Dedicated SPM staff

Knowledge/understanding

Demand from ‘above’

Institutional mandate

Champions for SPM

A vision for what SPM can do

Evidence that SPM helps

THANK YOU!!

Frog illustrations © Ted Kahn 2010Original artwork used with permissionNeotropical Conservation Foundationwww.neotropicalconservancy.org

Thank you to all survey respondents!

Summit Research CommitteeSheila O'Connor (chair)Bernd CordesWilliam CrosseBrett JenksRichard MargoluisMatthew MuirElizabeth O'NeillNick SalafskyKristin Sherwood