Upload
florence-ryan
View
214
Download
0
Tags:
Embed Size (px)
Citation preview
DIRECCIÓN DE PLANEACIÓN Y NORMATIVIDAD DE
LA POLÍTICA DE EVALUACIÓN
The Importance of the Institutionalization
of M&E: The Mexican Case
Gonzalo Hernández Licona
2013
Institutional milestones Mexico 1997-2009
1997 Progresa Evaluation 2000 Congress’ Decree: annual external evaluations to
all federal programs 2001 Evaluation Units within ministries 2001 National Audit Office. Congress 2002 First official poverty estimates. Ministry of Social
Development 2004 Law of Transparency and Public Access to Information 2005-6 Social Development Law
CONEVAL. National Council for the Evaluation of Social Policy
2006 Budget Law: Performance Evaluation System 2007 Evaluation Guidelines
CONEVAL, Ministry of Finance, National Audit Office 2009 CONEVAL. Multidimensional Poverty
Methodology 2009 Government Accountability Law
Changes in social policy due to the M&E System
The government launched in January 2013 an aggressive strategy to reduce hunger and extreme (multidimensional) poverty.
In September 2013 the President sent a bill to Congress to reshape the way social security has been working for almost 50 years.
The fiscal reform of 2013 changed the way resources will be allocated between States and Municipalities.
Since 2009, the Ministry of Finance uses information from the evaluation process to elaborate the annual budget.
There is a constant public and in-house debate and analysis (Presidency, ministries, programs, Congress, media) about the public evaluations of social programs and policies.
CONEVAL published that poverty and food deprivation increased between 2008 and 2012, and the Government acknowledged it.
Multidimensional poverty estimations showed that the larger deprivation is the access to social security (61%)
The reform has the goal to reduce extreme poverty according to the new multidimensional poverty methodology.
There are now annual information for almost 270 social programs. For some of them there are impact, process, design evaluations.
All evaluations are discussed with policy makers and all of them are public.
Changes in social policy due to the M&E System
The budget for the Cement Floor program was increased nearly 400% between 2007 and 2012.
The program Employment for the Youth was cancelled in 2009.
The food supplement of the Progresa-Oportunidades program was modified in order to have a better iron formulae in 2003.
The rural program Procampo is now less regressive, due to a design change in 2009. program were changes due to the evaluation.
Due to the design evaluation of the National Crusade Against Hunger, the strategy improved definitions, diagnosis and coordination tools
The impact evaluation demonstrated reductions of gastrointestinal diseases for children do to the program
The design evaluation of the program showed various design and operational flaws at a considerable cost.
The impact evaluaton showed the iron had not been absorbed by children.
The distributional analysis of various programs illustrated that the program was very regressive.
In July 2013 CONEVAL launched and evaluation to the CNCH, which was discussed with the Ministry of Social Development and also was public. The goal: On going improvement of the strategy.
/1 Information from the Diagnostics of the Indicators Matrixes for Results 2008 and 2010. /2 Information from the Follow-up of Aspects Susceptibles of Improvement System. The estimation considers the aspects susceptibles of improvement attenden by a 100% to march 2012./3 The estimations includes two programas that finished their obligations but are not in operation since 2011.
Various social programs and their budget have changed due to the “Evaluation-Improvement Mechanism”:
Type of Improvement2010/1 2011-2012/2
Programs Relative Participation Programs/3 Relative
Participation
Improvement in programs´ activities or processes 19 16.8% 35 47%
Improvement in the services offered by programs 22 19.5% 10 14%
Reorientation of programs 71 62.8% 26 35%
Programs were merged - - 3 4%
The program was cancelled 1 0.9% 0 0%
TOTAL 113 100.0% 74 100%
Challenges for constructing an Evaluation system:
Institutional: It’s almost impossible to have a public Evaluation system without a proper institutional arrangement: evaluation mandate, evaluation unit, feedback procedures for policy improvements, norms about transparency.
Technical: Suitable methodologies, trained evaluators, administrative records, information.
Constructing an M&E system is a political and institutional task with technical elements….not the other way round….
Identify the main social problems to prioritize strategies and resources.
Public Policy Actions Which programs can
be related to the social priorities?
Improvement of Programs and Policies
• Measuring Poverty (various dimensions)
• Evaluation of Programs and Policies
M&E: Budget decisions based on Results.
Evaluation guidelines for all institutions, together with the Ministry of Finance: The Demand for evidence was clearer now
Ministries’ Strategic objectives
Logical Framework:All Programs
Results
Recommendations’ follow-up
Consistency & ResultsEvaluations
Process Evaluations
Policy Evaluations
Impact Evaluations
Annual Evaluation Plan
Planning Evaluation
NationalDevelopment Plan
Annual Performance Report
Products and facts There are poverty figures at a national, state and municipality level Almost 600 programs have Log Frameworks. All social programs plus
others… We hired ECLAC to help us with the capacity building for the Log
models. We have offered an evaluation course every year for policy makers
and every 2 for researchers. 25% of all indicators are oriented to measure Results 150 programs are evaluated every year (now every two). There are
around 179 social programs. This is 90% of the total budget. 270 one-page summaries every year. Between 3-4 impact evaluations are done every year. CONEVAL budget is around $23 millions on even years (household
survey); $16 millions on odd years. But there is more budget for evaluation within ministries.
We can find on the internet: Poverty estimates All the evaluations The program’s point of view about its evaluation Each program’s Work Plan
Why all these changes in a country where citizens usually mistrust government information and where political parties fight for resources?
Since 1997 there is a balance of power between Congress and the President.
Congress demanded the creation of an independent institution, CONEVAL, for the measurement of poverty and for evaluation.
Information on poverty and evaluation is public and transparent. The Finance Ministry has been an important ally in the process. CONEVAL and the Ministry of Finance produced Guidelines for the
Evaluation of Programs. The center of the Guidelines: Results The Guidelines also created an “Evaluation-Improvement Mechanism”
Most ministries have cooperated in the design of the M&E system. Fine balance between: Accountability and Policy Improvement. The (autonomous) Statistical Office has invested a lot on data
collection. Good academic skills and suitable evaluation methodologies have
been increasingly disposable for the past 15 years.
Institutional milestones Mexico 1997-2007
1997 Progresa Evaluation 1997 Balance of power between Congress and the Executive 2000 Congress’ Decree: annual external evaluations to
all federal programs 2001 Evaluation Units within ministries 2001 National Audit Office 2002 First official poverty estimates2004 Law of Transparency and Public Access to Information 2004-5 Social Development Law
CONEVAL. National Council for the Evaluation of Social Policy
2006 Budget Law: Performance Evaluation System 2007 Evaluation Guidelines
CONEVAL, Ministry of Finance, National Audit Office
CONEVAL Mandate:
Measurement of Poverty at the National, State and Municipality level
Evaluation of social programs and policies
Governance CONEVAL is part of the Executive, but The Board has 8 seats. The majority of Board
members (6) are academic researchers elected by all the States, representatives from Municipalities, Congress and the Executive (44 votes)
Technical and managerial independence
Evidence must be part of a planning system
Identify and measure social challenges
Analysis What works?
Program design Budget Implementation
Monitoring & Evaluation
Why do we need evidence? Improve social
policy
Make better decisions (management, design, budget…)
Accountability
Measuring poverty by Law
Social Developme
nt Law
Dimensions for poverty
measurement
•Current income per capita
• Educational gap
• Access to health services
• Access to social security
• Quality of living spaces
• Housing access to basic services
• Access to food
• Degree of social cohesion
National States
Municipalities
• Multidimensional Poverty Measurement (National and State Estimations)o Economic wellbeingo Education lago Access to health serviceso Access to social securityo Quality and spaces of the
dwellingo Access to basic services in the
dwellingo Access to foodo Social cohesion
• Social Gap Index• Poverty Labor Trend Index
(ITLP)• Poverty Maps• Income Poverty Measurement
• Annual Evaluation Plan• Log Framework: Programs• Consistency & Results
Evaluations• Impact Evaluations• Complementary Evaluations• Performance Evaluations• Programs’ Performance
Summary • Social Development Programs’
Inventory• Thematic Evaluations• Policy Evaluations• Recommendations’ Follow-up• Document for Budgetary
Considerations
Social Programs Evaluation
Poverty Measurement and
Analysis
Main Products
-10.0
-8.0
-6.0
-4.0
-2.0
0.0
2.0
4.0
6.0
-10.0
-8.0
-6.0
-4.0
-2.0
0.0
2.0
4.0
6.0
-10.0
-8.0
-6.0
-4.0
-2.0
0.0
2.0
4.0
6.0
-10.0
-8.0
-6.0
-4.0
-2.0
0.0
2.0
4.0
6.0
Sorce: estimations of CONEVAL based on information of MCS-ENIGH 2008 y 2010
Population whose incomeis below the wellbeing
tresholdAccess to
food
Access to basic servicesin the dwelling
Quality and spaces of the dwelling
Access to social security
Access to health services
Educational gap
Millions of persons
ExtremePovertyPovert
y
200844.5 %
48.8 millions
201046.2 %
52.0 millions
200810.6 %
11.7 millions
201010.4%
11.7 millions
6
4
2
0
-2
-4
-6
-8
-10-9.0
-2.9-2.5
-2.3
-0.8
4.13.5 3.2
0.0
Social Deprivations
4.8
Population whose income is below the minimum wellbeing
treshold
Evolution of multidimesional poverty, Mexico 2008-2010
Percentage of population living in poverty by municipality. Mexico, 2010
Sorce: estimations of CONEVAL based on information of MCS-ENIGH 2010 and the Census 2010
Rangos
[ 0 - 30 ] [ 30 - 50 ] [ 50 - 70 ] [ 70 - 100 ]
Total de municipios
97 347 790 1222
Ranks
Total municipalities
Programa de Empleo Temporal (PET)
NA Moderate Adecuate Adecuate Adecuate 379.59% 92.5%The program is
VERY PROGRESSIVE
99.5%
Programa IMSS-Oportunidades
NA Adecuate Adecuate Adecuate Moderate SD 90.0%The program is
VERY PROGRESSIVE
100.0%
Seguro Popular (SP) Adecuate NA Adecuate Adecuate Outstanding 88.54% 100.0%The program is
VERY PROGRESSIVE
100.0%
Programa Comunidades Saludables
NA Adecuate Moderate ModerateOpportunity for Improvement
SD 80.0%Without
Information100.0%
Programa Caravanas de la Salud (PCS)
NA Moderate Adecuate Adecuate Moderate SD 100.0%Without
Information100.0%
Reducción de Enfermedades Prevenibles por Vacunación
NAOpportunity for Improvement
Opportunity for Improvement
OutstandingOpportunity for Improvement
SD NAWithout
Information100.0%
PROCAMPO para Vivir Mejor NA Adecuate Adecuate Adecuate Adecuate 99.43% 75.0%The program is
VERY REGRESSIVE
100.0%
Fondo de Apoyo para la Micro, Pequeña y Mediana Empresa (Fondo PYME)
Adecuate NA Outstanding Outstanding Outstanding 150.30% 69.0%Without
Information100.0%
Progressivity Level
Spent Budget/ Modified Budget
Program
Valuation
Improvements on the Delivery of Goods and
Services
Improvements on Indicators
and Goals Analysis
Coverage Coverage Efficiency
% of Achievement on
Following the Recommendations from External
Evaluations
Impact of the Program
Improvements on the
Achievement of its Objectives
RESULTS FROM THE SPECIFIC PERFORMANCE EVALUATION (EED) 2010-2011(External evaluation coordinated by CONEVAL and elaborated with information from the Performance Evaluation System of the
Ministry of Finance and Public Credit) Distributional Impact
2010 Budget Compliance
Results Related to the Program´s Objectives
Evaluation:Programs’ Performance Summary
Better coordination between evaluation institutions and budget allocation decisions based on results: Planning vs Budget; CONEVAL, Ministry of Finance and Ministry of Audit
More capacities on evaluation and measurement tecniques
Improve indicators of diverse programs Indicators of quality of educational and health services
List of beneficiaries Evaluation of legislation and norms’ changes Evaluation of Sectors/Ministries. Strategic objectives. Rigorous evaluations and transparency in states
and municipalities. Take more into account the information from
evaluations in budgetary, operative and strategic decisions.
Challenges for Mexico
Lessons Countries should find their own institutional arrangement: Chile,
Colombia, South Africa, USA, Canada, Mexico, China, etc. The balance of power between Congress and the Executive is
important Public pressure helps
Credibility is at the center of the institutional arrangement If the country engaged in specific evaluations previously, use
the experience to set up a system in the future Champions are always a key factor Internacional help is important…but try to build your own
path Impact evaluations are not always the first step, more basic
evidence is sometimes more important South-South exchanges of knowledge and innovation Keep the fine balance between Transparency and Policy
Improvement. On of the main users: Ministry of Finance