Changing the way the New Zealand Aid Programme monitors and evaluates its Aid

Ingrid van Aalst

Principal Evaluation Manager

Development Strategy & Effectiveness Division

Key national environmental drivers

The New Zealand Government’s priorities

enhancing the operational effectiveness of ODA focus on effective aid that supports sustainable economic development priority to the Pacific Region and targeted approach to Asia, very small

programmes in Africa and LA focus on NZ’s comparative advantage and to address our partner’s needs requirements for greater accountability for the use of public money and

demonstrate value for money

Key international environmental drivers

the effectiveness of aid expenditure is now at the forefront of development thinking

transparency, accountability, and confidence that significant aid investments are both effective and efficient in fulfilling their objectives

commitments to Paris Declaration on Aid Effectiveness, the Accra Agenda for Action and the Millennium Development Goals, with better monitoring and evaluation for both donors and development partners

An effective and focussed monitoring and evaluation function

contribute to our understanding of whether our aid interventions are effective, efficient and provide value for money

help us improve activities, programmes, and how we deliver aid what is working effectively (or not) & why what progress are we making towards outcomes

be aligned to the New Zealand Aid Programme’s core strategic and operational goals, and priorities - be at the forefront and integrated into our work

support accountable requirements for spending by providing credible, independent and objective assessments of New Zealand Aid Programme’s key strategies, policies, sectors, programmes and activities - we need to better demonstrate results or “tell our story”

be integrated with the New Zealand Aid Programme’s internal and external planning and reporting

New Zealand Aid Programme’s Performance Framework

Planning · Statement of Intent · Information Supporting

Estimates · MFAT Strategic Directions

Planning · Programme Plans · Sector Plans(??)· Joint Commitment for

Development / Country Strategic Frameworks (incld Results Frameworks)

Planning · Activity Designs (incld

Results Frameworks)

Reporting NZ Accountability · MFAT Annual Report · Section 32 A Report International Reporting · Aid Effectiveness Reporting

(e.g. Cairns Compact)Internal Management · Quality Management

Review Report· Dashboard · Aid Programme Focus and

Concentration Report

Reporting· MFAT Annual Report · Programme Reports· Sector Reports(??)· JCfD / CSF Annual Reviews· Aid Programme Focus and

Concentration Report

Reporting · MFAT Annual Report · AMAs and ACAs· Implementing Partner

Reporting (based on Results Framework)

Level 1- New Zealand Aid Programme

Level 2 - Programme (including Sector Strategies)

Level 3 - Activities

Qu

alit

y A

ssu

ran

ce

Qu

ality

Man

agem

ent R

evi

ew

In

tern

al A

udi

t

New Zealand Aid Programme Performance Framework

International Development

PolicyStatement

NZ Aid Programme

Strategy

Individual Performance Plans Divisional Operational Plans

Str

ateg

ic E

valu

ati

on

Wo

rk P

rog

ram

me

Pro

gra

mm

e, P

olic

y, S

ect

or

and

Re

spo

nsiv

e (

Ma

nage

d b

y D

SE

)

Monitoring &

Evaluation

Ac

tiv

ity

Ev

alu

atio

ns

Pri

ma

rily

man

aged

by

Pro

gram

mes

Monitoring &

Evaluation

Monitoring &

Evaluation

New Zealand Aid Programme’s monitoring and evaluation

Development, Strategy and Effectiveness Division

New Zealand Aid Programme’s strategy and policy, work programme, planning, performance and reporting

monitoring, evaluation and research

cross cutting co-ordination

procurement

process improvement, policy coherence, development outreach

best practice programme and activity management

The Evaluation team

developing and managing a strategic evaluation work programme

building evaluation capacity in and across the New Zealand Aid Programme, including at activity level

reviewing and developing evaluation policies, strategy, plans, processes and systems

The New Zealand Aid Programme’s strategic evaluation work programme

Four strands in the work programme

Sector Programme – The New Zealand Aid Programme has 24 Programmes Policy Responsive

a rolling programme of 5-7 evaluations each year across a number of years report to the Evaluation and Research Governance Group anticipate participating in a number of joint evaluations focus first is on developing the strategic evaluation work programme

Activity level monitoring and evaluation

simple and manageable monitoring and evaluation approach, which builds on previous monitoring and evaluation approaches

supports design of the activity at start and during implementation

greater focus on results (outputs, outcomes and goal) - away from inputs and activities

imbedded into the Activity lifecycle management processes and systems - ‘part of business as usual’

activity monitoring and evaluation is the responsibility of the Programme Teams

Activity Results Framework

Results Diagram identifies logic supporting the Activity and intended results

Results Measurement Table how results will be monitored and measured, including indicators and

targets

Monitoring and Evaluation Work Plan tasks and resources required to support monitoring and evaluating

results

Increased use of clean water

Improved health of community

Key Outputs

Long-term Outcomes

Activity’s Goal: For a healthier population contributing to a more prosperous country

Medium-term

Outcomes

Short-term Outcomes

Water management

training provided Wells Built

Community engagement /

training provided

Improved water quality

Improved water practices by community

Improved water management

processes

Water management

operations plan developed

Improved guidance for operations

Results Measurement Table

Results Indicators Baseline and Targets Methodology & Data Sources

Long-term Outcomes: Improved health of the Community

1. Cause specific morbidity rates (per 100,000)*2. Selected infectious diseases rates (per 100,000)*

Baseline: 450 (2005) and 470 (2010)Target: 460 (2012) and 400 (2015)Baseline: 460 (2005) and 480 (2010)Target: 480 (2012) and 420 (2015)

Information is available each October from the MoH. There is a one year lag in available information.

Medium-term Outcome: Increased use of clean water

1. % of households making use of clean water

Baseline: no information currently available Target: all households using clean water (2013)

Information will be collected through a specifically designed survey and captured by the MoH. Available each November.

Short-term Outcome: Improved water quality

1. % of water testing samples that comply with agreed quality standards

2. Incidence water born diseases in the community*

Baseline: 20% of samples comply with quality standards Target: 90% of samples comply (2012)

Baseline: 60 incidence per year Target: 30 incidence (2012) and less than 10 incidence (2015)

Information will be collected through a specifically designed audit process and captured by the MoH.

Output: Wells Built

1. Wells built to specification to time and to budget^

Baseline: no information available Target: Compliance achieved (2012)

Information will be collected through a specifically designed audit process and captured by the MoH. Information will be available in November.

M&E Work Plan

Tasks Approach (methods, processes)

Timeline Roles and Responsibilities

Deliverables and Reporting

Indicative Costs

Evaluation (Note: must be based on soon to be released Evaluation Policy and Guidelines)

Plan and undertake year three evaluation

4.1 Develop terms of reference

Identify in consultation with key stakeholders the purpose, scope, objectives / Document and confirm ToR with the Steering Committee

June-July 2012 Lead: MFAT Documented and agreed terms of reference

TA funding: 2 days ($4k)

4.2 Consult with stakeholders and develop eval. Plan

Identify and contract evaluator / Develop and agree eval. Plan

June –July 2012 Lead: MFAT Agreed and high quality eval. Plan

Funding for contract evaluator: $16k

4.3 Undertake year three evaluation

Complete the evaluation June- Aug 2013 Lead: Lead Evaluator Eval report signed-off by Eval. Steering Group

Funding for contract evaluator (s): $60k

4.4 Year three evaluation: Report results and develop action plan to strengthen on-going delivery

a) Present results and b) Workshop to determine response to results and how services will be strengthened

Sep 13 Lead: Evaluator Presentation on the results / Workshop: response to the evaluation, work plan to strengthen services

Grant funding: $20k

Overall Monitoring and Evaluation Budget Grant Funding $106k

TA Funding $14k

Full Funding $120k

New Zealand Aid Programme’s Performance Framework

Planning · Statement of Intent · Information Supporting

Estimates · MFAT Strategic Directions

Planning · Programme Plans · Sector Plans(??)· Joint Commitment for

Development / Country Strategic Frameworks (incld Results Frameworks)

Planning · Activity Designs (incld

Results Frameworks)

Reporting NZ Accountability · MFAT Annual Report · Section 32 A Report International Reporting · Aid Effectiveness Reporting

(e.g. Cairns Compact)Internal Management · Quality Management

Review Report· Dashboard · Aid Programme Focus and

Concentration Report

Reporting· MFAT Annual Report · Programme Reports· Sector Reports(??)· JCfD / CSF Annual Reviews· Aid Programme Focus and

Concentration Report

Reporting · MFAT Annual Report · AMAs and ACAs· Implementing Partner

Reporting (based on Results Framework)

Level 1- New Zealand Aid Programme

Level 2 - Programme (including Sector Strategies)

Level 3 - Activities

Qu

alit

y A

ssu

ran

ce

Qu

ality

Man

agem

ent R

evi

ew

In

tern

al A

udi

t

New Zealand Aid Programme Performance Framework

International Development

PolicyStatement

NZ Aid Programme

Strategy

Individual Performance Plans Divisional Operational Plans

Str

ateg

ic E

valu

ati

on

Wo

rk P

rog

ram

me

Pro

gra

mm

e, P

olic

y, S

ect

or

and

Re

spo

nsiv

e (

Ma

nage

d b

y D

SE

)

Monitoring &

Evaluation

Ac

tiv

ity

Ev

alu

atio

ns

Pri

ma

rily

man

aged

by

Pro

gram

mes

Monitoring &

Evaluation

Monitoring &

Evaluation

Building the monitoring and evaluation capacity and capability across the New Zealand Aid

Programme developing simple user-friendly, practice-oriented guidelines and templates covering all aspects of monitoring, and the process of commissioning and managing Activity evaluations

developing and delivering result monitoring and evaluation training

providing coaching and support for the more complex Activity evaluation challenges

re-establishing a Community of Practice for results monitoring and evaluation

supporting results monitoring and evaluation champions in each Division

Monitoring and evaluation policies, processes and systems

In the next 12 months

developing activity and programme results monitoring and evaluation policy, processes, systems and guidelines, and evaluation policy and strategy

establishing the Evaluation and Research Governance Group, and a panel/pool of results monitoring and evaluation providers

developing processes for quality review of evaluation deliverables; protecting the independence of evaluators and their reports; disseminating learnings; management response and action from evaluations; link to planning, reporting and management decision-making

Any questions?