Monitoring & Evaluation System“Learning to Improve”
Making evidence work for development
Contents• Rationale of the M&E• Users and needs • Principles of the M&E• Units of analysis and dimensions of study• Elements of the M&E• Products of the M&E• M&E share of responsibilities• M&E constraints • M&E linkages: KM, decision making and learning
Rationale: Why should the MDGF have a M&E system?
• It is a requirement included in the legal agreement between the Donor (Spain) and UNDP
• It is an obligation, included in all signed joint programs
• It is necessary: if we want, scaling up programs into policies and spread solutions to achieve MDGs at Global level
• It is useful: As it is part of the program managing cycle and is the best way to measure progress, detect problems, correct them, improve performance and learn at local and global level
Who are the users of the M&E system
what are their needs in terms of Information?
For 5 minutes discussion:
Are we missing anyone?
Are most of the views of stakeholders included?
This is an attempt to let your tacit knowledge flow
What kind of principles should the MDGF’s M&E system incorporate?
PRINCIPLES
What kind of principles should the MDGF’s M&E system incorporate?
• Accordance with UNEG and DAC/OECD standards
• Oriented to well balanced learning and accountability purposes
• Evidence-based: consistent data, information or knowledge to support judgments and conclusions of monitoring and evaluation
What kind of principles should the MDGF’s M&E system incorporate?
• Built on an aggregation scheme: Elements of M&E (indicators + evaluations, etc) in lower levels add up in higher level of inquire
• Measure (DELTA), describe, analyze understand the object of study (JP+C+W+MDGs) and use results to improve program and policy performance
What dimensions should the MDGF M&E system cover?
What questions should it answer?
Joint Programs Countries
1st M&E LevelJoint programs
2nd M&E LevelCountries
Environment and Climate Change
Gender Equality Women’s empowerment
Culture & development
Economic democratic Governance
Youth, Employment and Migration
Conflict prevention & Peace Building
Children Food Security and Nutrition
Development and the Private Sector
3th M&E LevelWindows
Monitoring AspectsInput-Products-Results-ProcessesEvaluation Dimensions-Quality of the program formulation-Program objectives attained-Contribution to MDGs &other development indicators, gender-Replication: Scale up-Innovation -UN system Coordination-Delivering as One -Ownership-Alignment-Harmonization-MFDR-Mutual Accountability-Delta change, effects incitizens’ life
Monitoring Aspects-Results (UNDAF+PRS)-Processes (Coordination)Evaluation DimensionsInduced effects on:
-MDG at country level-Other Development indicators-UN system Coordination -Delivering as one-Ownership-Alignment-Harmonization-MFDR-Mutual Accountability-UN country pilots
4th M&E LevelMDG Achievement Fund
Monitoring Aspects-ResultsEvaluation DimensionsQuality of partnership Spain/UNDP:Added Value as a mechanism to Progress MDGs achievementThe Secretariat’s role and added valueInduced effects on:-Linkage of windows and MDGs-Other Development indicators-UN system Coordination -Delivering as one-Ownership-Alignment-Harmonization-MFDR-Mutual Accountability-Effects on citizens' life
Joint Programs
Joint Programs
Countries
Countries
Monitoring Aspects-Results as an aggregate of the JP-HR/Gender/EnvironmentEvaluation DimensionsNew themes:: Culture & migrationInduced effects on:-MDG at country level-Other Development indicators(Peace and Culture)-UN system Coordination-Delivering as one -Ownership-Alignment-Harmonization-MFDR-Mutual Accountability
Please in 5 minutes
What are the 3 most burning questions an M&E system should it answer?
What elements comprise the MDGF M&E system• Monitoring indicators: to measure progress and trends in the short
and medium-term at the (inputs, activities, outputs, outcomes)• Field visits: to monitor JP in depth and prepare and manage
evaluations as well as disseminate results and provide feedback from recommendations
• Evaluations: to review programs and value the worth of the dimension of study JP, at country level, in thematic windows and the MDGF as a whole
• Meta-evaluations: to review the quality of the evaluations (JP + Country) conducted and produce robust evidence at window level to link these evidence to MDG achievement
• Desk reviews and data collection & analysis: from a variety of sources to contribute with information and knowledge to the M&E+KM system
What products will the MDGF M&E system create and offer?
• Field monitoring reports• Country monitoring reports• Mid-term evaluations reports• Impact evaluation reports• Country evaluations (case studies)reports• Meta-evaluations reports• In depth review reports • MDGF global reports (midterm + final)• Special activities under the M&E+ Information & Advocacy
focus country initiative
MDG F Achievement Fund
M&E USER Responsibilities to be taken Deliver the products
MDG-F
Secretariat
Support the UN country teams in the collection of the information and the reporting procedure for monitoring indicators and activities
Design 24 thematic and process indicators to feed MDGF M&E system.
Design generic terms of reference for mid-term evaluations.
Work with UN country teams customizing mid-term evaluation’s terms of reference
Design 8 meta evaluations studies 1 for each window of activity
Support and facilitate the ongoing evaluations, assuring quality and participation through all their phases: desk review, field work report drafting, publication and dissemination.
Assure that recommendations from monitoring system and evaluations mainstream the manager’s decisions and correct programs’ deviations, guarantee that knowledge created serves as a input for the knowledge management system.
Implement a calendar of field missions to carry out M&E activities
Elaborate a protocol guide to monitoring visits
Implement the calendar of field visits to carry out the above mentioned activities
The strategy of M&E for MDGF
Country monitoring reports+ special countries with high learning and knowledge transfer potential
The MDGF mid-term evidence-based report
Mid-term evaluations
Country evaluations
At least 3 Impact evaluations ( experimental or/and quasi-experimental)
meta evaluation studies
The MDGF evidence-based final report
MDG F Achievement Fund
M&E USER Responsibilities to be taken Deliver the following products
UN Country Team
Collect and report information on the following indicators:
On joint program implementation at 3 levels :(3 input, 3 output, 3 outcome) (best, medium and worst indicators, to observe trends) (quarterly reported)
3 indicators on joint program coordination (supplied by MDGF Secretariat) (quarterly reported)
3 thematic indicators per active window in the country (every six months, supplied by MDGF Secretariat)
Report on all Joint Program indicators designed by themselves ( annually)
Facilitate the M&E processes and activities according to the guidance and principles supplied by the MDGF Secretariat
Integrate recommendations from M&E to improve programs
Adapt the templates for TOR for final evaluations joint program and country evaluations as well as financing t final evaluations
Special activities under the initiative focus countries (Ethiopia, Mauritania, Morocco,Timor Leste, Philippines, Bosnia Herzegovina, Brazil, Ecuador, Honduras)
Monitoring Quarterly reports
Final evaluations
M&E framework activities
Report in coordination indicators
And thematic indicators
MDG F Achievement Fund M&E Timeframe
Activities 2009 Jan Feb Mar Apr May Jun jul Aug Sep Oct Nov DecM&E Document finishedSocialization process of the M&E strategy: consensus reachedDesign of common indicators: 24 thematic + 3 on coordinationSelect 9 pilot countries for M&E and communication purposesSpecific m&E activities designed and implemented9 Pilot country monitoring reportsselect and design 3 impact evaluationsConduct baseline study for impact evaluationDraft M&E Guidelines document elaborated M&E Guidelines document testingM&E Guideline document in operationTraining sessions for Secretariat Collaborative work with MDTF for monitoring and reportingCollection of information on indicatorsReview of the JP quaterly reportsField visits for monitoring and evaluationElaborate general terms of reference template for mid-term evaluationsElaborate customized terms of reference template 5 mid-term evaluationsProcurement process for 2009 midterm evaluations consultants2 regional Workshops intorducing M&E and trainingActivities 2010 Jan Feb Mar Apr May Jun jul Aug Sep Oct Nov DecElaborate customized terms of reference for midterm evaluationsHire consultants and procurement processmanagement and coordination of Mid-term evaluations 5management and coordination of Mid-term evaluations 4management and coordination of Mid-term evaluations 10management and coordination of Mid-term evaluations 6management and coordination of Mid-term evaluations 3management and coordination of Mid-term evaluations 3management and coordination of Mid-term evaluations 7management and coordination of Mid-term evaluations 7management and coordination of Mid-term evaluations 6management and coordination of Mid-term evaluations 8management and coordination of Mid-term evaluations 66 evaluations started and 45 reports are completed 7Colborative work with MDTFCollection of information on common indicatorsConduct a second baseline study for impact evaluationReview of the JP quaterly reportsField visits for monitoring and evaluationKnowledge management activities, compilation, digestion and disseminationDesign and conduct the meta anlaysis evaluation round 1Meta analysis reportThematic reports on the windows round 1Pilot Country monitoring reportsEvidence based mid-term report of the MDGF FundActivities 2011 Jan Feb Mar Apr May Jun jul Aug Sep Oct Nov DecElaborate customized terms of reference for midterm evaluationsHire consultants and procurement processmanagement and coordination of Mid-term evaluations 2management and coordination of Mid-term evaluations 7management and coordination of Mid-term evaluations 10management and coordination of Mid-term evaluations 10management and coordination of Mid-term evaluations 10management and coordination of Mid-term evaluations 12management and coordination of Mid-term evaluations 5management and coordination of Mid-term evaluations 3management and coordination of Mid-term evaluationsmanagement and coordination of Mid-term evaluationsmanagement and coordination of Mid-term evaluationsmanagement and coordination of Mid-term evaluationsCollaborative work with MDTFCollection of information on common indicators
59 evaluations started and 80 reports are completed
real data as december 2008
What are the constraints we would face in such a enterprise? budget, time, data, political constraints
• What level of involvement and what work load can the staff from the JP take?
• Organizational issues, Who is responsible for what part of the system?
• Joint program, for real? Do joint programs have the necessary coordination mechanisms to allow for joint monitoring and evaluation?
• Will the users of the M&E system willing to collaborate to implement it ?
• Is data available at a reasonable cost and on time?• Will we have the political support to implement it?
M&E linkages: KM, decision making and learning• M&E is an enormous source of knowledge and will generate an
extraordinary amount of valuable explicit and tacit knowledge as well as organizational and general knowledge that will have to be delivered at the right moment, to the right persons in the adequate format for the purposes needed.
• KM system could also become a source of data, information and
knowledge for M&E system, is a reciprocal 2 way relation. • The ultimate goal of evaluation is recommend actions to improve
decision making and improve performance of programs and policies so all evaluations should focused on utilization of their findings and recommendations