Upload
umi-hanik
View
173
Download
0
Embed Size (px)
DESCRIPTION
Paper presentation for 6th International Evaluation Conference 2014 Malaysian Evaluation Society (MES) Park Royal Hotel, Kuala Lumpur, 24-28 March 2014
Citation preview
TOWARDS RESULTS BASED MONITORING AND EVALUATION SYSTEM: LEARNING FROM FAILURES OF THE MONITORING AND EVALUATION SYSTEM OF SCHOOL OPERATIONAL ASSISTANCE (BOS) PROGRAM IN INDONESIA
UMI HANIK Co-founder and Chair for Department of Networking and Advocacy Indonesian Development Evaluation Community (InDEC); Currently working as Monitoring and Evaluation (M&E) Specialist for Poverty Reduction Support Facility (PRSF) an Australian Aid Initiative project in Indonesia managed by GRM International on behalf of the Australian Government
!
*) Disclaimer: The paper is shared mainly for knowledge sharing and academic purposes. None of the concepts, approaches, or statements in this paper should be taken as any official viewpoint of neither the Government of Indonesia nor any other agency mentioned in the paper.
Paper presentation for 6th International Evaluation Conference 2014 Malaysian Evaluation Society (MES) Park Royal Hotel, Kuala Lumpur, 24-28 March 2014
BACKGROUND q According to law, Indonesian citizens have the right to
education and the Government has an obligation to finance basic education without charging fees
q In 2005, the government introduce School Operational Assistance or Bantuan Operasional Sekolah (BOS) to provide operational funds directly to schools based on a per-student formula
q BOS objective was to improve access to quality education for all children of ages 7 to 15 in Indonesia by strengthening school-based management (SBM) and community participation
q The Government of Indonesia with the World Bank support work to increase the effectiveness of the BOS M&E system
OBJECTIVE
The paper will outline the BOS M&E System, to explore the failures and learnt on how to perform RBM system in real world evaluation for the social assistance program in education sector
CONTRIBUTION TO FIELD
The paper is expected to provide the value adds of practical knowledge of learning from failures on the Results-based M&E System in real world evaluation for the social assistance program or education issues in countries.
METHOD q The paper is
summarized and rewrite from BOS KITA M&E Capacity Development project’s proceedings 2009-2012
q Assessment focused on three aspects of the M&E activities.
The system, approach, and procedure for M&E of BOS program
The institutional (regulatory framework and institutional mandates)
the organizational context (organizational structures and functions) under which the system take place
LEARNING FROM FAILURES
M&E RESOURCES q Only one person was employed for each
level and seems didn’t meet the requirement to secure quality RBM implementation
q The staff was also working for a unit within Education Office, which potentially brings up conflict of interest
q Staff did not have appropriate knowledge and skills to carry out the tasks
COMMUNITY PARTICIPATION q The community as an external component of BOS
Program on M&E is limited
q The community does not familiar with any system of M&E within BOS Program
q School Committee (in particular those of government schools) has become a non target impact of BOS Program.
q Committee holds limited access to carry out budget monitoring, the major activity of Committee
M&E INSTRUMENT q the nature of monitoring is a management audit q most respondents did not have common
understanding on the final goal q information gaps and inconsistency between BOS and M&E
guideline
q Not sure which BOS Management is accountable to the achievement of participation of Local Government
q The instrument to collect the relevant data was not there, q the report available only to prepare the administrative
requirement.
QUALITY OF M&E IMPLEMENTATION q Coordination among BOS Management has not been addressed
proportionally q All BOS Managements targeted schools as the object of M&E activities
which potentially brings up duplication q BOS Management tends to focus its M&E activities on outputs monitoring
at school level q BOS Management staff was very skeptical toward the utility of monitoring
for broader education purposes q They were not informed nor have knowledge on how useful is monitoring
results to decision makers q There was not any follow up actions made as recommended in the report q The program continue and funds are disbursed, disregard the quality of
monitoring q There is not any reward for conducting good monitoring or preparing good
monitoring report. q Monitoring report did not clearly show quality of the Program q The report does not show any further analyses of each score for each
indicator to find out the causes – why the score is low or high, what factors influence achievement of a particular indicator.
IMPROVED RBM SYSTEM
CAPACITY BUILDING STRATEGY q Focus on building capacity of BOS Management to
continue conduct RBM activities;
q Improve and strengthen monitoring instruments that ensure monitoring results provides benefits to the Central and Local Governments to develop their education program, and refine tune the system as necessary;
q Select at least two Provincial BOS Managements as the active partners of Capacity Building team, in addition to Central BOS Management;
IMPROVED SET OF OUTCOME AND INDICATORS q Refine the tune of BOS M&E system is significant to ensure clarity in
the goal and intermediate goals of BOS Program.
q Adopting cause and effect logic that highlights interrelation between Problem and Causes with Impact and Effects (or Outcomes) may
helps BOS defining its ultimate goal and intermediate goals
q clear goals and indicators of achievement brings advantages for BOS Management in conducting quality monitoring and preparing quality report.
q Monitoring will not focus only on outputs, but include outcomes to measure impact or changes that BOS Program would like to achieve.
q Report will no longer present only figures, but will highlight changes that the Program has achieved, how they are achieved and why the expected changes do not take place.
q In addition, report will present situation to quality education which is very useful for decision makers developing further interventions.
CLEAR ROLES AND RESPONSIBILITY q Each BOS Management authority and responsibility should clearly be
defined according to the level. q The authority and responsibility should not overlap one to another. q Each BOS Management should support one to another,
Level! Authority! Responsibility!Central! • Socialization at Provincial level
• Monitor quality implementation at Provincial level
• Monitor achievement of all goals!
• Ensure quality implementation at Central and Province levels
• Report on consolidated achievement of all goals at national level
!Provincial !
• Socialization at District/City level • Monitor quality implementation at
District/City !
• Ensure quality implementation at Province and District/City levels
• Report on consolidated achievement of all goals at Provincial level
!District City !
• Monitor quality implementation at school level
• Monitor fulfillment of children’s rights !
• Ensure quality implementation at District/City and school levels
• Report on consolidated achievement of all goals at District/City
• Report on the agreed dimension of change !
School !
• Implement quality activities at school level
• Monitor fulfillment of children’s rights!
• Ensuring quality implementation at school level • Report on the progress of activities • Report of the agreed dimension of change !
IMPROVED SYSTEM AND INSTRUMENT ü M&E activities should not give another burden to the
government official or BOS management
ü Improved RBM system should consider the capacity of the government officials or BOS management to manage the improved RBM system in a daily basis. ü covering simple monitoring form ü BOS Management could use instantly ü handy instrument that support data collection, monitoring
and supervision, reporting, evaluation, and the utilization ü include an automate and user-friendly excel-based
application ü generating BOS progress and update on results
from the schools up to national level easily and instantly
ü feed BOS decision makers with instant and sound information on BOS results.
Ø With one simple excel sheet application the BOS management will be able to : Ø use the monitoring form, Ø data input-process, and Ø preview the results and visualization,
instantly!! Ø With such instrument and application will
lessen the burden to carry out the M&E in a daily basis
YOU WILL GET THIS, INSTANTLY!!! Instant analysis at district/city/school level
SEE HOW IT WORKS Let’s open the excel sheet application now ;)
CONCLUSION q RBM in real world evaluation
ü Applicability and simplicity is the key to successful RBM ü Applicable RBM system will guarantee the buy in from the
government or the expected actors ü User-friendly application is very much needed to support the
applicable RBM implementation ü Capacity of the M&E resources to perform or to carry out the
system in a daily basis is significant ü Ensure continue resource allocation ü Intensive nurturing and coaching for two years full on initial
implementation to change old behavior and to ensure the applicability
ü Participative approach for the engagement and continuous learning to ensure compliance