Elements of poverty reduction program

monitoring and evaluation

Sophisticated development context: the Kazakh experience

Ayse KudatSocial Assessment

2004

WHAT IS OUR CHALLENGE IN KAZAKHSTAN?

START A PRPM PROCESS DO SO BY TAKING MODEST SUSTAINABLE STEPS AS A FIRST STEP, INTEGRATE NATIONAL AND OBLAST LEVELS PRP ELEMENTS INTO A

UNIFIED SYSTEM AND SUPPORT THIS INTEGRATION WITH REAL-TIME AND GEPOGRAPHICALLY SPECIFIC COMPUTER APPLICATION

DESIGN A PROCESS OF CAPACITY BUILDING FOR PRP AND PRPM BY LINKING INPUTS, ACTIVITIES, OUTPUTS, OUTCOMES AND IMPACTS AT THE OBLAST LEVEL

INITIATE A SIMPLE SYSTEM OF PRPM COORDINATION AND CREATE PRP MONITORING WORKING GROUPS IN MEBP, MLSP AND IN THE 14 OBLASTS

ENSURE MONITORING CONTRIBUTIONS OF NGOS CURRENTLY COOPERATING WITH OBLASTS IN PRP PREPARATION AND IMPLEMENTATION

SUPPORT THIS PROCESS IN 2 PILOT OBLASTS WITH FOCAL POINTS FINANCED BY THE TA PROJECT

SUPPORT THE PARTICIPATORY PRPM GOALS OF OBLASTS BY HELPING 2 PILOT REGIONS STRENGTHEN M&E CAPACITY IN THE RAYONS---HELP THESE OBLASTS ENHANCE LOCAL M&E CAPACITY WITH MODEST RESOURCES AVAILABLE IN TA PROJECT

CREATE INSTITUTIONAL ARRANGEMENTS FOR EVALUATION CONDUCT PILOT STUDIES FOR IMPACT ASSESSMENT AND INITIATIVE SURVEYS TO

TRACK PUBLIC EXPENDITURES ASSESS EQUITY AND EFFICIENCY IMPLICATIONS OF PUBLIC SPENDING ON PRP EVALUATE IMPLICATIONS OF TA PROJECT ACTIVITIES FOR NATIONWIDE

IMPROVEMENTS IN THE M&E OF PRPS AND PRP

Q: WHAT DO WE LEARN FROM INTERNATIONAL EXPERIENCE

OF POVERTY REDUCTION STRATEGIES/PROGRAMS?

A: PUBLIC SECTOR EXPENDITURES AND ACTIVITIES DEVOTED TO POVERTY REDUCTION PROGRAMS (PRP) CANNOT BE JUSTIFIED OR CAUSALLY LINKED TO POVERTY TARGETS UNLESS A RESULTS

BASED MONITORING IS ESTABLISHED

In Kazakhstan we know more about outcomes than about the means of achieving them

Output

Activity

Input

Outcome

Impact

Results Monitoring

Implementation Monitoring (Means and Strategies)

Imp

lem

enta

tio

nR

esu

lts

Sources of Information for PRPM

Output

Activity

Input

Outcome

Impact

Households surveys, qualitative surveysPublic expenditure Tracking Surveys,

Sector specific surveys

Imp

lem

enta

tio

nR

esu

lts

Oblast budgets, oblast payroll,Rayon budgets, rayon payroll

Oblast PRP workshops, task forces,Policy reviews,

community infrastructure inventories, Rayon M&O achievements, WSSSystems repaired/constructed

TA financed qualitative surveys,Public expenditure Tracking Surveys in

2 pilot oblasts,

1. Conducting readiness assessment 2. Agreeing on Outcomes to monitor 3. Selecting indicators to monitor outcomes 4. Establishing baseline 5. Selecting realistic targets 6. managing for results 7. Role of evaluations 8. Reporting findings 9. Using findings 10. Sustaining M&E systems

Progress made in PRPM&E in Kazakhstan

Much is already known

What data are collected (source) When data are collected

(frequency) How data are collected

(methodology) Who collects the data For whom the data are collected Who reports the data

Where revenuesAre collected,

Expenditures madeAnd activities held,

Outcome data Are not

systematically gatheredRAYONS

MEBPInter-Ministry Steering Committee SecretariatMonitoring Working Group

Two Pilot Oblasts

Focal Points

Oblast PRP working groups, with Oblast M&E Coordinators

NationalStatistics Agency

Local Univ., andcivil society

organizations

Ministry of Laborand

Social ProtectionJoint evaluationGroup

RayonAdministrations

Other Line Ministries

Steering Committee

Recommendationsto the Parliament and the Cabinet for

Poverty Reduction Strategy

Remaining OblastsCITIZEN

FEEDBACK

Households

Stakeholders in KAZAK PRS monitoring

Central Ministries (Finance, Economy) MEBP consolidate information,

Line Ministries (MLSP) Provide oblast based data, helps evaluate results Reporting arrangement to be structured between MEBP and other line Ministries under

the TA Universities, research institutes

Local universities contribute to training and impact data gathering for small scale impact studies

Rayon governments Key provider of input and output data to Oblasts

National statistical agencies Conducts reliable and large scale, nationwide household surveys representative at the

oblast level Develops rayon level capacity for collection of output data

Civil society organizations Trade unions provide feedback information, Association of SME can contribute impact

data Parliament/Cabinet

MEBP through the Steering Committee to carry out Evaluation and provide strategy feedback to Parliament and Cabinet

Participation and Feedback

Civil society involvement in policy formulation exists at Oblasts and through the TA financed focal points their contributions for M&E can be structured.

Through PRPM the program planning for future PRPs (2004+) can be improved. Targets, inputs, outputs can be better defined.

Sharing PRP results with public through the Newsletter of the national NGO consortium, as well as the website of the MEBP (with links to other line Ministries) can help generate citizen feedback and ensure sharing of PRP/PRS information

Client Feedback

Limited resources available for all studies, including report cards, are extremely limited under the TA.

Specific citizen feedback will be generated through the NGO Newsletter

Citizen/user feedback will be assessed for 2 projects (e.g., a- SME development for the unemployed, and b- micro credit for rural areas) in 2 pilot oblasts through surveys carried out by local universities. These studies will also include budget tracking surveys

Qualitative impact studies of pioneering programs in 2 pilot oblasts (a-process of social passport issuance and targeting; and b- emigration from ecological disaster areas)

PRPM works shops in 2 pilot oblasts with rayon and select settlement representatives; participatory evaluation of PRP as an input to next steps of PRP preparation.

Types Of Evaluation Proposed within TA project

Impact Evaluation ofHealth sector expenditures

Process Implementation-

commissions

SME training for The unemployed

Meta-Evaluation

Case Study of Immigrants to 2 oblasts

Pre-ImplementationAssessment

Monitoring data flow

In terms of institutional mechanisms, the data on poverty will flow through several different channels

From RAYONS to OBLASTS to the central level MEBP for monitoring PRP

From RAYONS to Statistical Agency for monitoring of the outcomes Sectoral information, MIS data and/or data from sample surveys conducted by different central agencies, including the MLSP to MEBP

Data from key NGOs (e.g., trade unions, association of business and farmers’ association) to Oblast governments and transmitted to MEBP

Case studies, budget tracking studies and client satisfaction studies conducted at the Oblast level and shared with MEBP

Input, Activity and Output Indicators: Unified and re-prioritized in accordance with the National Poverty Reduction Strategy

Data obtained from the oblast levelData obtained from the implementing line ministries

Outcome Indicators

Data obtained from the oblastsData obtained from the National Statistics Agency

Sources of Monitoring Indicators

Real-time Monitoring and Evaluation Software

Analysis of specific actions underlined at the National Poverty Reduction Strategy in terms of actual outputs provided, and poverty reduction outcomes achievedMaking these results available at the oblast levelEnabling direct linkages between resources put through specific actions and policy outcomes of these actions, assisting policy makers in designating new priorities

Expected Results from the Monitoring Software

Sample Indicators – How the Software Will Work

Inputs

Name of Activity: Primary school repairsAmount of financing (to be indicated)If financing not available, indicate estimated number of man-hours dedicated to this activity

Activities and Outputs

Number of schools undergoing repairsEstimated number of beneficiaries (ie the number of pupils in these schools)

Outcomes

Number of primary school age childrenstudying in schools – urban and ruralPercent of pupils in schools that study onthe third shift

OblastsN

atio

nal

Lev

el

Charts and Graphsfor summary information

and evaluation

Results will be made available on a geographical system(sample indicators on education)

30 schools repaired15,000 pupils benefited

20 schools repaired13,000 pupils benefited

Almaty Oblast Pavlodar Oblast KzylOrda Oblast

0

5

10

15

20

25

30

35

Number of schools

10 schools repaired9,000 pupils benefited