Evaluation Talk

From Uganda, Ukraine and beyond – Evaluation lessons from parliamentary strengthening?

A practical example of the design and execution of a

multi-country end-evaluation

By: Karin Weber Kampala, April 9th 2014

1. Introduction 2. Scope of evaluation 3. TWC evaluation in numbers 4. Purpose of evaluation 5. Evaluation criteria 6. Evaluation methodology 7. Evaluation framework 8. Limitations & Challenges 9. Top tips

2

Presentation Outline

Introducing…

Karin Weber - Senior Consultant - M&E specialist - Worked as independent consultant, NGO director, United

Nations M&E specialist, MoFA policy advisor in Europe, Africa and Asia.

Delta Partnership - International development consultancy firm - Offices in UK, Kenya and Uganda - Recently became part of WYG - Multi-sector and multi-disciplinary

TWC end evaluation in numbers…

TWC end evaluation

7 Consortium members

1 lead

7 Countries in

5 different regions

50 Implementing

partners

9 Evaluation Criteria

130+ respondents

5 languages

4 thematic areas

5 outputs

Evaluation Scope (1/3)

- The programme: ‘’Strengthening Human Resource Development in Southern Parliaments’’

- Implemented by The Westminster Consortium (TWC): - led by the Westminster Foundation for Democracy (WFD) - Commonwealth Parliamentary Association UK Branch, - The House of Commons Overseas Office, - The International Bar Association’s Human Rights Institute - The National Audit Office (NAO) - The Thomson Reuters Foundation (TRF) - The University of Essex Institute for Democracy and

Conflict Resolution

- Funded by DFID’s Governance and Transparency Fund (GTF)

Evaluation Scope (2/3)

• TWC programme aimed to establish self-sustaining Parliamentary Studies Centres (PSC) and building the capacity of local partners to deliver effective parliamentary development

• Twinning model between UK partners and key partners in each country.

• Four thematic focus areas: 1. Parliamentary Process and Management; 2. Human Rights and the Rule of Law; 3. Financial Oversight; and 4. Access to Information.

Evaluation Scope (3/3)

• Period under evaluation: 2008 – 2013

• Geographical scope: UK, Georgia, Ukraine, Uganda, Mozambique, Lebanon and Morocco

Purpose of Evaluation

In line with GTF guidelines the purpose of the Final Evaluation was to: Identify the impact of the programme and ways that this

may be sustained Record and share lessons Account to local stakeholders for the programme’s

achievements Improve future programme design and management Verify funds were used effectively and efficiently to deliver

results Enable DFID to evaluate the performance of the GTF as a

whole, making sure the overall portfolio has increased accountability and responsiveness.

Evaluation Criteria

• Relevance • Impact • Economy • Efficiency • Effectiveness • Equity • Value for money • Sustainability • Replicability

Methodology

• The international evaluation team: Karin Weber and Aileen Lyon • The end evaluation was conducted between September 2012 and

August 2013 • Step 1: Planning meetings with the client • Step 2: Stakeholder mapping (to identify potential respondents) • Step 3: Development of an evaluation matrix • Step 4: Development of an evaluation plan • Step 5: Development of interview topic list/ questionnaires • Step 6: Conducting the desk study • Step 7: Carrying out 6 country visits (up to 5 days in each country)

– Mid-term update report to client including 3 visit reports and emerging findings

– Country reports validated with in-country teams

• Step 8: Analysis and draft reporting • Step 9: Validation meeting with TWC team • Step 10: Finalisation of evaluation report

Example of Evaluation Framework

Criteria Key evaluation questions Methodology Data sources/key stakeholder

Effectiveness • To what extend have the planned results been achieved to date?

• What has been the most significance change of the programme?

• Which factors contributed to the success of the programme?

• What were the key challenges affecting programme performance?

• Were there any unintended results in the programme? Were they positive or negative and in which ways did they affect the different stakeholders?

Desk study Semi-structured interviews Focus Group Discussions

Project documents National level studies and surveys Project staff, Ministerial policy advisors

Cont. Criteria Key evaluation questions Methodology Data

sources/key stakeholder

Efficiency • How economically were resources/inputs (funds, expertise, time, etc.) converted to programme results?

• To what extend did the budget and actual expenditure at the country level reflect identified programme priorities?

Value for Money analysis Cost-benefit analysis Desk study

Project documents Finance officers Audit reports

Relevance • Was the programme formulated and implemented in accordance with national and local strategies?

• To what extend did the programme answer the needs of the intended beneficiaries?

Interviews Desk study Satisfaction survey

Project document National policies Beneficiaries

From ToR to Framework, Tools and Analysis

Analysis and Reporting

Labelling, grouping, summarising Report outline/ chapters

Tools for data collection

Interview topic lists & Questionnaires Desk review checklists & Achievement rating scale

Evaluation framework

Evaluation Criteria & Evaluation Questions Methodology & Data sources & Key stakeholders

Evaluation objectives

In Terms of reference Link to Evaluation criteria

Desk Study

TWC Programme documents

• Log frame

• Budget

• Quarterly reports

• Annual reports

• Financial reports

• Risk management report

• Case studies

Other documents

• Final Evaluation Guidelines for Governance and Transparency Fund Grant Holders

• National level data (incl. bills, laws, policies, statistics etc.)

• Academic literature related to topic

• Articles by trained journalists

• Handbooks and guides developed by the programme

• Partner documents

• Websites

Evaluation

documents

• Mid Term Review Report

• Management response to MTR

• Internal evaluations

Methodology (cont.) • Country visits:

– Observation, Semi-structured interviews and Focus Group Discussions with: • TWC Programme Managers and Administration and Finance Officers • local partners • MPs and Parliamentary staff • Local trainers and Trainees • Journalists and representatives from Civil Society Organisations (CSO). • DFID and other donors working with the parliament in-country

– Use of a translator in Georgia, Ukraine, Morocco, Lebanon and Mozambique

– Use of voice recorder where permission was granted by respondent

– Additional in-country document review where relevant

• Performance analysis - to assess performance against the logframe, for outputs, outcome and impact

Methodology (cont.)

• Data analysis and synthesis – Notes from document review

– Transcriptions from interviews and FGDs

– Labelling and synthesis in line with evaluation matrix

– Further analysis of financial data to assess Value for Money

• Reporting – Draft report in line with good practice, the ToR and GTF

guidance

– Discussion and validation with the TWC team

– Incorporation of feedback

– Finalisation of the evaluation report.

Challenges & Limitations (1/2) • Limited evidence

– Especially at the higher purpose (outcome) and goal (impact) levels.

– Focus on monitoring activities and expenditure during implementation instead of intermediate and long-term results

– Limited use of the logframe for monitoring (seen in-country as a HQ document)

– Poor follow-up after trainings, study tours and other capacity building activities

– No systematic monitoring of bills passed and implementation of laws

– Lack of clarity on the responsibility for data collection during monitoring

• Data collection during country missions took place at different times in the last programme year

(between October 2012 and June 2013)

Challenges & Limitations (2/2)

• In some cases WFD staff members were present during interviews; risk of bias

• Security (e.g. Lebanon mission postponed due to car bombing next to Parliament)

• Political sensitivity (e.g. parliamentary proceedings in Lebanon are not public)

• The end evaluation steering committee consisted of TWC members only; no independent members

• Gender unbalanced evaluation team

• Different quality of translators

Top Tips (1/2)

• Check specific donor guidelines for end evaluations • Frequently refer back to ToR and evaluation matrix • Ensure a good preparation and initial analysis before the

country missions: – Go straight to the monitoring data and reports early on to get a

grip on the data they have available – Ensure that you have a ToC up front – not just the logframe.

Jointly develop one if ToC doesn’t exist. You can then test the assumptions and how it all worked out as you go into in-country conversations

• Check country reports, data and facts with respondents and/or managers before the final analysis of the data

• Make time for a joint analysis with all the team who worked on it – much more efficient

Top Tips (2/2) • Ensure highest adherence to ethical standards,

including: – Independence (emphasise this with client where

necessary, don’t be shy!)

– Honesty, integrity and transparency (report on limitations)

– Respect for dignity and diversity

– Confidentiality- protecting sources of information

• Apply triangulation in methodology and data sources

• Use participatory M&E methods and integrate gender and human rights

• Keep the client informed, also during the process

Thank you! Questions?

For more information:

karin@deltapartnership.com www.deltapartnership.com

For the full evaluation report:

http://www.wfd.org/upload/docs/GTF%20394%20Annex%20A6%20-%20Final%20Evaluation.pdf

21