Adm

1

Annex 9

Proposed Monitoring and Evaluation System

1. INTRODUCTION This document describes the framework developed for M&E system of the BEAM-ARMM ADM project. The Monitoring and Evaluation (M&E) system describes how the performance and quality will be continuously monitored and how the outcomes and impact of the assistance will be periodically evaluated. This framework is linked to the over-all BEAM-ARMM Results Based Monitoring and Evaluation System and the Quality Assurance and Accountability Framework of the Department of Education. It also builds on the strengths of BRAC Education Program M&E developed over the last 26 years. A careful analysis of the project design was undertaken to determine: the purpose and scope of the M&E system; performance measures and the requirements of users of the M&E information; the sources of information and gathering methods; the responsibilities for M&E; critical reflection processes and events; how M&E information is to be reported and used and capacity building. More specifically, the M&E framework provides the following basis for monitoring: the attainment of defined target results through outcome monitoring; the information needs at different levels of the management structure; methods to assess project progress and performance against work plans; resource schedules and budgets; the quality of key project activities and outputs; responsibilities for undertaking monitoring activities at all levels; formats for reporting progress and achievement and issues/problems and remedial actions. Similarly, the M&E framework should provide the following basis for evaluation: systematic collection, analysis and assessment of potential project impacts and associated indicators through outcome evaluation; identification of key stakeholders impacted by the project; critical questions to explore - based on hypotheses, assumptions and major risks inherent in the design; summary descriptions of key tools and methods to be used for evaluation; responsibilities for undertaking evaluation activities and associated reporting and implementation plan for evaluation activities. The draft M&E implementation plan is summarized in M&E Matrix in page 31. This can be further refined by the Managing NGO in consultation with BRAC, DepEd and Implementing NGOs. 2. OBJECTIVES OF THE M&E As articulated in the mother BEAM-ARMM design, the over-all purpose of the M&E is the measurement and assessment of performance in order to effectively manage the outcomes and outputs known as development results. Performance is defined as progress towards achievement of results. Traditional M&E focuses on assessing inputs and implementation processes. In this framework, the focus is on assessing the contributions of various factors to target development outcomes. Likewise, this framework will provide management information needed for checking on the progress of support activities and to involve key stakeholders in learning to improve project implementation. The M&E system will provide quantitative and qualitative performance data by which the achievements of the desired results can be measured and judged to inform the strategic planning process at critical points. Secondary data from the Department of

2

Education (DepEd) Basic Education Information System (BEIS) will be used and compared in the reports for the various components of the project. Specifically, the objectives of the results-oriented M&E are to:

• Enhance learning and development- lessons from experience will provide information on what work and not and thus serve as basis to adjust the approaches or redesign if necessary;

• Ensure informed decision-making – quantitative and qualitative data will have to be established as baseline and updated periodically for policy development;

• Support substantive accountability – responsibility for ensuring value for money, cost efficiency and transparency in the implementation

• Model for DepEd-ARMM and Civil Society Partners to enhance and build capacities in the delivery and monitoring of ADM project

3. GUIDING PRINCIPLES AND APPROACHES This BEAM-ARMM ADM M&E framework is developed consistently following the guiding principles and approaches of BRAC Education Model on M&E, the Quality Assurance and Accountability Framework of DepEd and the results-based monitoring and evaluation system. 3.1 Managing for Results As indicated in the project log frame, outcome monitoring will be the major emphasis of the M&E activities. There will be a continual and systematic collection and analysis of data to measure the performance of interventions towards the achievement of the defined outcomes of the BEAM-ARMM ADM. It is not time-bounded activity; outcome monitoring must be periodic so that change can be ascertained. Project components will accumulate information on progress against the target outcome, which will then be periodically compared against baseline. This will be carried out by tracking the outputs of each of the 4 Project Components and measure their contributions towards the achievement of target outcomes by assessing the change from baseline conditions. Relevance of the interventions- whether the BEAM-ARMM ADM intervention contributes to the achievement of key outcomes; supports the DepED ARMM priorities and targets appropriate groups, will have to be assessed periodically. In order to conduct effective outcome monitoring, the Project will need to establish baseline data, select outcome indicators of performance, and design mechanisms that include planned actions such as filed visits, stakeholders meetings and systematic analysis or reports. Outcome monitoring is not the same as implementation monitoring. However, it is expected that the project will still monitor the implementation of activities such as collection of inputs, activities, and immediate outputs and systematically report this to provide information on administrative, implementation and management issues that will have an impact to the achievement of desired results. This monitoring should be planned and integrated with outcome monitoring to avoid duplication and to lower transaction cost latter 3.2 Evidence based Outcome evaluation will assess and validate what results were achieved, how and why they were or were not achieved. Periodic and in-depth assessment will be undertaken to obtain evidence of the actual impact of the project strategies focusing on improving access and quality to basic education of students in target communities. The assessment will be done cooperatively among the various implementing partners such as BRAC, the Facility NGO, provincial based Civil Society Organizations and DepEd-ARMM. The assessment will also look at the partnership strategies purposely to establish impact working with such partners in

3

a conflict environment like ARMM. Evidences will be gathered through the periodic reporting formats, research and evaluation activities and most significant change stories. All data will be gender disagreggated to allow analysis of the progress on addressing performance disparity between boys and girls.

3.3 Analysis, learning and accountability In order to maximize the impact of the project, it is critical that effective learning process and analysis are nurtured so that innovations can be shared and applied. The results of both the outcome monitoring and outcome evaluations will provide understanding of why the change is or is not taking place. The analysis will help improve the implementation processes and overall project strategy. This framework will also fulfill its responsibility for being accountable to stakeholders for project expenditures, activities, outcome and impacts. Reflecting the critical role of analysis, learning and accountability in monitoring and evaluation, the framework allows stakeholders to: understand what changes are needed to improve results and performance; identify lessons learned for future projects and programs, and establish the success of the implementation – ie assess whether project activities should be expanded. The M&E system will need to provide performance and impact information to stakeholders to inform project management decision making and enable stakeholders to: assess progress against schedules and targets; allocate the necessary resources and funds for effective implementation; compare use of inputs and expenditure against resource schedules and budgets; assess the quality of implementation. BRAC, DepED-ARMM along with all implementing NGO partners will be involved in planning and review of the M&E system, as well as analysis on the effectiveness of the innovations and approaches. 3.4 Gender Equity This principle meant to promote and ensure that gender and development (GAD) goals are integrated in various aspects of the project’s operations and M&E. The BEAM-ARMM ADM ensures that education opportunities are equally accessible to marginalized boys and girls. Activities and strategies in this new education alternative delivery modality are congruent with the gender equality goals and the results that the project is expected to attain- enable young children-boys especially- to attend school at the same time help their families earn a living; make better school facilities accessible to rural boys and girls; greater access of rural boys and girls to good quality education; longer stay in school of boys, where male drop-out is particularly high in ARMM; improved school performance of boy and girl students. Likewise, project monitors and evaluators will be guided by GAD checklist for education in their assessment for the gender responsiveness of the ADM project. This checklist will be drafted by the Managing NGO using the harmonized Gender and Development Guidelines developed and produced by National Economic Development Authority (NEDA), in collaboration with the National Commission on the Role of Filipino Women (NCRFW) and the ODA-Gender and Development Network. Collection of sex-disaggregated data, gender specific information and conduct of research studies to further analyze gender issues/gaps of education in ARMM shall be undertaken. The research findings will inform the detailed implementation of activities to ensure gender-fair teaching and learning processes. 3.5 Simplicity

4

This principle underpins the BRAC Education Program M&E which is also adopted in the development of this framework. BEAM-ARMM ADM M&E System is simply constructed yet capable of producing detailed information. It is designed to track results and record activities implemented using select limited set of Key Performance Indicators (KPIs). With the view to obtain consistency in the data management, it is essential that indicators and data collection methods be simple and clear, and the number of indicators to be collected be kept to an absolute minimum. It is much better to have complete and accurate data on a small number of useful measures than to have a comprehensive set of measures for which the integrity and completeness of the data is uncertain. Reflecting the simplicity principle, the framework consists of three key elements: tracking inputs and outputs of project activities; organizing regular discussions with key stakeholders on progress and problems with implementation; and annual review on critical performance questions as the basis for the annual plan As required, the monitoring and evaluation framework may expand and grow in complexity to reflect changing needs and as insights for improvement are gained from experience. 3.6 Ethics Monitoring and evaluation of BEAM-ARMM ADM activities are designed to comply with the Australian Evaluation Society (AES) Code of Ethics. In brief, the code requires

• systematic, data-based inquiry • competence of evaluators • integrity and honesty • respect for the security, dignity and self-worth of target groups, clients and other

stakeholders • careful consideration of the potential risks or harm to the client, target groups or staff

of the project • articulation and consideration of the diversity of interests and values that may be

related to the public welfare Moreover, the implementing NGO partners and staff will be trained on the Do No Harm (DNH) analysis as part of their annual planning and M&E work. This will include a more detailed examination of how specific activities will engage (or are engaging) the communities and the possible (or actual) positive and negative impacts these may have.

3.7 Modeling and Capacity building Consistent with the overarching objective of the M&E is to model BRAC best practices on M&E and build the capacity of DepEd-ARMM and CSO partners in ADM Project delivery and monitoring. DepEd-ARMM and NGO partners will significantly learn from the intensive BRAC M&E practices and experience developed over the years. For the BEAM-ARMM ADM, outcome indicators will come from the Unified Basic Education Information System (UBEIS) of DepEd-ARMM. Specific baseline outcome indicators will have to be shared and gathered to serve as the basis in measuring the direct impact of the intervention. Improvement of UBEIS as well as DepED ARMM’s M&E capacity will have to be enhanced through trainings and systems improvement. Key elements of the capacity building approach will be established to support communities of practice in project evaluation and to promote a learning culture so that evaluation findings will inform future actions. The M&E/Quality Assurance Unit of DepEd-ARMM will also be capacitated.

5

3.8 Coordination with other BEAM-ARMM Program Components The need for coordination with other BEAM-ARMM Program Components will also take on greater importance to ensure complementation of activities and avoid duplication. Likewise, the project is likely to place considerable demands on the time of DepEd leaders and staff in ARMM, therefore, the scope of monitoring extends to other initiatives to improve basic education. Similarly it is also important that BEAM-CLC capitalizes on lessons learned from other projects in order to avoid repeating mistakes and optimize the likelihood of success. 4. COMPONENTS OF THE M&E FRAMEWORK The M&E framework provides specific guidance for the implementation of the M&E System through a delineation of activities, methods, indicators, timing and responsibilities. It is linked to the BEAM-ARMM ADM Logframe and supported by the specific procedures within Project’s quality assurance system. Both framework and the procedures are intended as working document that will be updated annually in the project annual plan to reflect changes in the project’s scope and activity based on the lessons learned during the implementation. The framework is divided into 5 components: a.) Review Information Needs of Major Stakeholders and Responsibilities; b.) Plan for Data Gathering, Analysis and Evidence-based Reporting; c.) Plan for Milestone Review and Critical Reflection Events and Processes; d.) Plan for Quality Communication and Reporting; e.) Plan for the Resources and Capacities Required 4.1 Review Information Needs of Major Stakeholders and Responsibilities This section aims to assess the information needs, interests and decision making responsibilities of key project stakeholders-the intended users of information. This component will help provide key stakeholders the information they need to make good decision about the project based on the established M&E System. See table 1 for details.

Table 1 Major Stakeholders, Responsibilities and Information Needs

Stakeholders M&E Responsibilities Information Needs

1. DepED National, and other national agencies

• Support the institutionalization of national education innovation in ARMM through technical assistance and capacity building for DepED ARMM

• Support the scale-up of successful education innovations in ARMM to other regions of the Philippines

• Development plans or concepts for muslim education innovations

• Lessons learned and best practices

• Results of RAMSE test

2. BRAC, Managing NGO, Implementing, NGOs, DepEd-ARMM and other Implementing

• Develop the M&E tools • Train all stakeholders on

M&E • Reporting and

communicating the progress and outcomes

• Outputs and progress of implementation

• Results and impact • Client satisfaction • Performance of

Implementing Partners

6

Partners to all stakeholders • Conduct of in-depth

research

(BRAC, Managing NGO, Implementing NGOs)

3. ARMM Regional Planning Office

• Harmonize the M&E tools and reports across various Donors in ARMM

• Disseminate program results

• Data on education investments from GOP and Foreign assistance

• Population data • Legislative agenda in

support of education 4. DepED ARMM • Initiate Policy Dialogue

with national and field level offices

• Maintain an updated BEIS database

• Support the Divisions, Districts and Schools on planning and monitoring

• Operationalize the M&E framework

• Use data and analysis for policy development, budgeting process

• Indicators for access and completion

• Results of RAMSE and NAT

• Baseline on education data and periodic updates

5. BEAM - ARMM ADM • Prepare BEIS reports on school and student data

• Share school performance data with community

• Gather activity specific data

• Data on pupils, teachers, school physical facilities in accordance with BRAC reporting requirements

• Actual interventions, support received from the project

• Feedback on activities from pupils, parents

6. Local Government Units, School Management, Committee, Parents

• Share population data and community profiles

• Provide the Local School Boards with information on education performance

• Share information on various assistance to education

• Progress and impact of implementation

• Issues and problems in the LGU affecting implementation such as security, right of way

• Counterpart requirements

4.2 Plan for Data Gathering, Analysis and Evidence-based Reporting M&E is only useful if it results in improved decision-making. This is the underlying value of the results-based approach to M&E and requires that data to be gathered are transformed into information and knowledge. Collection of sex-disaggregated data and gender related information is crucial to inform project implementers and decision makers if there is a need to adjust on strategies that will address constraints to girls’ and boys’ improved participation to education. This component will determine how to gather, organize and analyze and report on data. This is, in a sense, the heart of the M&E System. The logframe indicators will be linked with the data gathering forms/tools and then making sure those forms/tools link with the evidence-

7

based reporting format. Tools and forms will help ensure that each Key Performance Indicator will be gathered in a systematic way. The Managing NGO will coordinate with implementing NGOs and DepED ARMM on establishing the baseline data for the outcome indicators. Once the baseline has been gathered, it will serve as a guide to all implementing NGO partners for periodic reports. In selecting information gathering methods, due consideration will be given to the cost, reliability of data, skill needed, ability to quantify results and the richness of information generated. This will also require some capacity building for those responsible for collecting, analyzing and reporting evidence based data. See page __ for details of the table below that is linked to the Monitoring and Evaluation Plan (to be drafted by the Managing NGO). Table 2 Sample of data gathering and evidence based reporting that is linked to M&E Plan Performance Indicators Method of data gathering

(Tools/forms) Findings to be Presented in (Monthly report (MR); Quarterly Report (QR); Annual Report (AR))

Component 1 E.g Number, by gender, children ages 4-5 attending BEAM-CLC pre-school classes

Site visit and interview with beneficiaries BEAM-ARMM ADM Monitoring Form Report on Newly Opened School (see annex X Forms/Tools)

MR

4.3 Plan for Milestone Review and Critical Reflection Events and Processes In this component systematic planning time for reflection and analysis will help ensure that data are transformed into valuable knowledge. By doing this, project management decisions will likely build on the lessons learned as the project progresses. Likewise, M&E working group can plan routine reflection events in order to validate project information coming from observations, monitoring data and project visits; analysze the findings; and then use these findings to inform project decision-making. Critical reflection events should be held frequently during the project implementation. Regular project review meetings maybe held monthly, quarterly, semi-annual or as deemed necessary by the project. Periodic events maybe scheduled, such as a learning review following a major training event or a lessons-learned workshop following a project evaluation. Formal critical reflection and milestone review can occur at meetings of the Project Management Unit, Senior Management Group and Steering Committee where a decision can be made to proceed or not or scale up the project. 4.4 Plan for Quality Communication and Reporting In this aspect of M&E it is important to think about when information is needed to make critical management decisions with partners and key project stakeholders. Communicating M&E findings is very important. Primarily, there are two fundamental reasons for quality communication and reporting: 1.) for accountability purposes, and 2.) to maximize impact by sharing information on project successes and shortcomings if there are any.

8

For primary stakeholders, they have an interest and right to know how the project is progressing and provide them feedback to ensure the accuracy of the results. For AusAID, BRAC, DepEd-ARMM, and MINDA it is important to account for the investment in project inputs, progress toward goals and contribution to impacting on the quality of and access to basic education in ARMM as well as to understand the challenges faced to make good decisions. Communication and reporting is also a key element of the ‘managing for impact’ or results-based approach of BEAM-ARMM ADM. In particular M&E findings generate knowledge of what works, what doesn’t and why under certain conditions which needs to be recorded and shared through formal and informal means to maximize its impact. Sine M&E information is very important to inform project management decision, creating an M&E Calendar will help ensure that information is available at the time required by those who need it to inform management decision. The timely provision of information will enhance the likelihood that it will be used. Further, an M&E Calendar establishes critical deadlines for reporting. Please see below reporting M&E calendar. Table 3. E.g. of the M&E Calendar Type of Report Format Key Messages Prepared by

BRAC and Managing NGO

Submission to AusAID and DepEd-ARMM

E.g. Progress Monthly Report

Electronic narrative Microsoft word A4

Achievements: Facilitators trainings conducted, parents forum undertaken

10th of every month 15th of every month

Bi-monthly Monitoring Report


Pedagogic issues which includes the subject-based teaching-learning issues

15th and 30th of every month

5th of the following month

Yearly Progress Report


Achievements: Summary of outputs, progress 4 major components and initial project results (if there’s any; evidence-based)

Every 18th day of the 12th month

25th of the 12th month

4.5 Plan for the Resources and Capacities Required

9

The key staff and partners with M&E responsibilities will be ensured to have the knowledge, skills, tools and support to carry out their respective tasks. A well functioning M&E System requires human resources, training as well as materials and financial resources. The Managing NGO will lead the orientation on the M&E Plan/System to ensure that Implementing NGOs, DepEd-ARMM and other key partners are familiar and have the necessary skills on how to implement the M&E plan so they can effectively use the system to record, store and analyze data as well as produce the necessary reports to inform decision-making. The project will also endeavor to investing project staff time in building capacities within the project to integrate GAD or promote gender equality into the M&E. Table 4: Example of Capacity Building Required for Stakeholders with M&E Responsibilities Stakeholders with M&E Responsibilities

M&E Responsibilities Resources & Capacities Required

BRAC, Managing NGO through its M&E Unit

• Develop the M&E tools • Train all stakeholders on M&E • Reporting and communicating the

progress and outcomes to all stakeholders

• Conduct of in-depth research

• Review of existing BRAC M&E tools and enhance/develop tools suitable for BEAM-ARMM ADM Project

Implementing NGO Partners and staff

•

DepEd-ARMM through their M&E and Quality Assurance Unit

•

ARMM Regional Planning Office

•

5. SCOPE OF M&E The scope of M&E activities is informed by the project strategy as documented in the Project Design Document (PDD) and further defined in the Work Plan (to work on by Managing NGO & Implementing NGO). The changing context and new understanding gained from experience demands that the project strategy be adapted from time to time to maximize the impact on improving the quality of, and access to, basic education in ARMM. 5.1 Project strategy In determining the scope of the M&E framework, the first step was to review the logical framework (logframe) incorporating outputs and activities defined in the work plan and cost schedule of the Project Design Document. In addition, the LogFrame (Annex X) provides a single project description against which to monitor progress and assess impacts and services delivered. It is also reflected in the detailed activity schedule (Gantt Chart-to be further refined by Managing NGO), resource schedule and cost schedule to better manage project implementation and monitor project performance. 5.2 Annual plan

10

Translating a project strategy as outlined in the LogFrame into an annual plan that is clear to the staff of Managing and Implementing NGOs and partner organizations transforms ideas into action. Accordingly, activities listed in the Work Plan (to work on by Managing NGO and Implementing NGO partners) are included in the M&E system. The Project Management Unit will periodically review the project LogFrame and Work Plan in close consultation with primary stakeholders to monitor changes in the planned activities. 6. PROJECT PERFORMANCE MEASUREMENT & INFORMATION NEEDS Measuring the effectiveness and efficiency of BEAM-ARMM ADM to improve the quality of, and access to, the basic education system in the ARMM is a complex task. Socioeconomic factors, geographic location, age, gender, Indigenous status, language background, educational attainment of parents and the performance of other government agencies (particularly health, law and justice services) also influence educational outcomes. Nonetheless the M&E system has to provide a reliable means to measure the performance of the project in delivering outputs, producing outcomes and having an impact on basic education and peace and development in Mindanao. The relative contribution of the project on various outcomes, the project will gather, where possible, information on the outcomes and impacts primarily through Most Significant Change reports. Information from these reports may be useful to the scale-up of the project. 6.1. Activity, output and outcome indicators The activities under each component will have to be outlined in the work plan. This will allow the project some flexibility to modify, adapt or revise activities based on the changing context and lessons learned to be responsive to changing needs as well as optimize the allocation of resources to maximize project results and impact. In this respect, indicators of project activities and outputs are detailed on an annual basis within the overarching project design. The cause-effect/ means-end relationship of project activities, outputs and outcomes enables all the data collected for activities under a specific output to be collated to monitor progress toward achieving that particular output. Similarly the results under each output can also be collated to monitor progress toward achieving a particular project outcome, expressed a component objective. This hierarchical linkage also extends to collating data from project components to assess the contribution of the project to its stated goal and purpose. The indicators of project activities, outputs and outcomes are detailed in the Work plan (see annex X-to work on by Managing NGO) and M&E framework (refer to Annex X-to be further refined by Managing NGO). 6.2 Key performance indicators (KPIs) Key performance indicators have been identified to enable management to focus on “big picture issues” and monitor achievement of the project objectives. The KPIs provide the basic accountability information for the project detailing what the project has achieved at any point in time. It should be noted that the KPIs are largely compound indicators or indices consisting of a number of measures to assess the performance of the project. Particular consideration has been given to making the KPIs attainable and realistic by ensuring that they reflect a more pragmatic and achievable target. Below are the key milestone outputs that can be the basis in the formulation of the project’s KPIs. The Outcome Indicators

11

Project Goal: Improved access to and quality of basic education for Out of School Boys and Girls thereby contributing to the over-all improvement of basic education performance in ARMM Project Purpose: To provide opportunity for Out of School Boys and Girls in selected communities in ARMM acquire pre-school and elementary education through the BRAC-like alternative delivery model (ADM).

Strategic Objectives:

1. Increased access, participation and completion rates of boys and girls in basic education

2. Improved learning achievements of boys and girls in elementary education 3. Engaged communities in basic education delivery through an alternative delivery

model 4. Capacitated DepEd-ARMM and NGO Partners in basic education delivery through

an alternative delivery model Table 5: Milestones Outputs Total Component 1.Curriculum, Materials and Assessment 1.1. Preschool education curriculum and instructional materials 1 set 1.2. Primary education curriculum and Instructional materials 1 set 1.3. Student assessment tools 1 set Component 2. Community Learning Centers 2.1. Preschools Learning Centers established 1,128 2.2. Primary education Learning Centers established 299 2.3. Preschool education completers 33,840 2.4. Primary education completers 8,970 Component 3. Capacity Building 3.1. Trained Managers (ARMM) 20 3.2. Trained Managers (Other regions) 12 3.3. Trained Supervisors and Trainers (ARMM) 170 3.4 Trained Supervisors and Trainers (Other regions) 25 3.5. Trained Project Coordinators and Organizers 200 3.6. Trained CLC Learning Facilitators 1,128 3.7. Trained parents and community leaders 40,000 Component 4. Management, Monitoring and Evaluation 4.1. Steering Committee organized 1 4.2. Provincial Steering/Advisory Committee organized 5 4.3.Project Management Unit established 1 4.4. Local NGOs engaged and capacitated 15 4.5. Provincial Field Offices established 5 4.6. Project Design for BRAC- ADM Philippines 1

12

7. INFORMATION GATHERING 7.1 Data collection methods Based on the output, outcome and impact indicators to be measured and evaluated, consideration has been given to how data is to be collected, specifically:

• moving from data collection to explaining successes and failures, deriving understanding from various issues with stakeholders and making decisions

• the reliability of data collected. In many cases triangulation of data generated from different methods is proposed and validation of the results.

• existing data gathering systems/tools from BRAC shall be utilized. Particular consideration has been given to how these systems can be used and strengthened where needed to meet the requirements of stakeholders

• providing precise, scientifically verifiable data to measure the status or change of a specific indicator and rich qualitative information on peoples’ experiences and opinions

Having considered the input of stakeholders and assessed existing capacity, a range of data collection methods have been identified in the M&E framework including:

• Sampling- random sampling will be employed to assess pupil learning outcomes, quota sampling used for evaluating many key performance indicators.

• Stakeholder analysis- identification of stakeholders, their relations, and external factors which may affect the performance of BEAM-ARMM ADM initiatives

• Documentation review- to understand the performance of the activities. Where information gaps exist or there is a contradiction of evidence, other methods of data collection will also be employed such as interviews and questionnaire/ surveys.

• Case studies- will be used to document the sequence of events or story of an individual activity to obtain insights into the impact of the project. This form of data collection lends itself to qualitative information and the capacity building elements for BEAM-ARMM ADM.

• Interviews- to gain answers from an individual or small group using a broad series of questions to guide conversations which allow a degree of flexibility in reframing questions and clarifying issues in order to provide an in-depth understanding of qualitative issues

• Focus groups- will be used to generate group discussion on project activities, outputs, outcomes and impacts, particularly in assessing opinions of change, the quality of project services and service providers and identification of areas for improvement

• Questionnaires/ surveys- to gain structured data to specific questions from a large number of people to provide data on specific performance questions and indicators

• Photos- are a convenient method of recording changes over time and giving an impression of the projects activities which produce tangible physically observable results

• Direct observation and field visit- where feasible, direct observation of project performance based on clearly defined criteria. It extends to site visit and and system inspections and;

• Most significant change- will be used to identify the most significant changes, whether positive or negative, relating to key objectives which are difficult to quantify such as capacity building

7.2 Data collation, analysis & storage The M&E framework is structured to provide reports at activity, output and component levels as well as produce data to evaluate the outcome and impact of the project. As such it

13

enables quantitative data to be aggregated to permit analysis at various levels such as school/ divisional/ regional reports to answer performance questions specific to a particular geographic location. Similarly the capability of project M&E systems also enables data to be collated at different levels of the project hierarchy to answer performance questions specific to each component of the project. To this end, the requirements of the project M&E systems, particularly the training database has to be articulated to provide standardized fields and formats. The collation of qualitative data requires special attention to the categorization of raw data, analysis and formulating conclusions to non-standard questions and opinions. Data collection includes gender equality targets and indicators which require collection of sex-disaggregated data on education performance. Good record keeping underpins the M&E framework. Stored information serves as a record or organizational memory for new staff members or in verifying progress on past performance. Storing data serves to meet accountability requirements and also inform the project strategy whilst also consider the cost-benefit of having excess information which is poorly used. With respect to collecting and storing data for learning, the focus is on documenting key lessons learned and ensuring the utilization of this knowledge through formal reporting and the project website and informal means such as Communities of Practice. 8. CRITICAL REFLECTION PROCESSES AND EVENTS Reflecting the on principles and purposes of the M&E framework which are managing for results and learning to improve project implementation need critical reflection processes and events. This includes individual and team reflections, team meetings, reflecting with stakeholder groups, using steering committees and documenting lessons learned and actions for improvement. 8.1 Schedules for Critical Reflection Processes and Events Critical reflection begins at individual level but extends to collaborative inquiry to address differing perspectives by articulating assumptions about practice and understand where assumptions have been made and identify what alternatives are possible. This demands that critical reflection events are systemized into a sequence of learning events. This structure recognizes the important contribution of stakeholders in providing different perspectives in uncovering new information, limiting biases and constructing a shared understanding of the situation to ensure actions are meaningful, shared and owned by all. At the same time, critical reflection activities also acknowledge the differing levels of willingness and ability in engaging in change-oriented learning processes. As such, BEAM has structured learning processes and events which follow reporting and the hierarchy of decision-making. Table 5: Example of Critical reflection schedule

Process or event Purpose & Description Whom to Involve Timing

E.g Monthly Progress review

Review project progress against targets defined in the Annual Plan, plan of activities for subsequent six months

Country Director, Project Manager, Deputy Project Manager and Senior Managers

Monthly

14

8.2 Capturing lessons learned In order to manage for results, it is necessary to consciously reflect on action and capture the lessons learned to avoid the mistakes of the past and increase successes in the future. The key elements to be used in recording lessons learned are:

• LogFrame reference- a reference to the project output which the lesson learned was derived

• Theme of the lesson learned- a brief description of the core innovation or problem encountered

• Original understanding/ assumption- a short description of the challenge/ problem and the governing assumption/s

• Revised understanding- a brief description of the insight derived • Supporting evidence- at least 3 forms of evidence will be needed to substantiate

the finding from a range of different sources to ensure its reliability and validity • Prompt for the insight- a simple description of what triggered the contributors to

derive its new understanding (eg critical incident, field observation, after action review etc)

• Contributors- the people/ stakeholders involved in deriving the lesson learned • Author- the name of the person who recorded the lesson learned

9. COMMUNICATION AND REPORTING 9.1 Communication and reporting measures The BEAM-CLC shall employ a range of formal and informal communication and reporting measures including:

• BEAM-CLC website- communicates project activities and achievements to primary stakeholders who have access to the internet.

• E-mail, letters, facsimile- regular formal and informal communication to stakeholders on planned and implemented project activities

• Team meetings- formal and informal communication within and across project teams on project implementation issues

• Senior Management Group meetings - meeting of DepEd-ARMM Secretary, Project Manager and Deputy Project Manager of Managing NGO, Heads of Implementing NGOs and Senior Staff to ensure coordination of project activities and alignment with DepEd-ARMM activities.

• PMU Staff meetings- to be held every month, alternating with SMG meetings, where all component and line managers in the PMU reflect on progress and achievement, address issues and coordinate the planning of BEAM-CLC activities

• Quarterly reports – detailing progress, financial status, staffing and procurement, with a narrative summary of progress and implementation issues which serves as a progress report to BRAC, AusAID, DepEd-ARMM and Steering Committee.

• Implementing NGO Partners’ reports- monthly exception reporting detailing the progress of activities on ground and reason why a target has not been achieved and what remedial action has been taken

• Financial status reports- statements of expenditure for staffing, procurement and training included in regular reports to stakeholder groups

• Issues-based reports- produced by exception whenever a Project issue emerges in the view of either the Managing NGO or the Implementing NGO Partners is of sufficient importance that AusAID requires notification. Issues-based reporting will

15

describe the issue in full, and identify the proposed remedial action, its likely impact, and the responsibility and timing for implementing it.

9.2 Frequency of reporting The following reports and or studies shall be prepared during the project implementation. Table 6: Example of Reports to be produced for various M&E Users Type of Report Contents Responsibility Recipients Monthly Report Quarterly Progress Report

Completion Report Research and Special Studies

Advisers/Consultants Reports

10. MANAGEMENT INFORMATION SYSTEM The development of Management Information System (MIS) is very vital to effectively and efficiently handle complex data, store and transform them into useful information and knowledge. The MIS itself comprises of a number of different related systems development for the various outputs and components of the project. Wherever possible basic packages such as Microsoft Office products (including MS Excel and MS Access) are used to minimize the recurrent cost of software licenses and avoid the large costs of developing and maintaining customized software. The key elements of the MIS for the BEAM-CLC include the following: 10.1 Training database A user-friendly and flexible training database shall be developed and utilized during the BEAM-ARMM ADM Project. A consultation shall be undertaken among NGO Implementing partners to identify the information needed and ensuring that the data entered is of high quality by using inputs, limiting lists and making key fields compulsory so that the reports produced are accurate and reliable. The database allows results to be disaggregated and analysed by gender, special needs groups (e.g. Lumads, Learning Facilitators, Parents), and calculate absolute training numbers and the percentage of the target population that has received training. 10.2 Student Tracking System- Establish student tracking system to monitor and assess progress and attainment of learning competency of each pupil during project implementation. It shall also provide vital information about status of pupils after completion in BRAC-like school especially those who will enter in the public elementary and secondary schools. Collection and analysis of sex-disaggregated data are an important requirement of this system. 10.3 NAT and RAMSE A simple MS Access database to allow easy input of NAT and RAMSE test results across different sites, compilation of results and basic analysis which also interfaces with more advanced analysis packages. This system serves as a resource for all PMU staff and

16

primary stakeholders in evaluating the impact of initiatives to improve the quality of basic education. Collection of sex-disaggregated data is an important requirement in the analysis of NAT and RAMSE results. 10.4 Quality and Financial management system- Establish chart of accounts to be used in the BEAM-ARMM ADM Project reflect outputs during the project implementation. The financial management system can generate reports of reimbursable expenditure grouped as training, procurement and other costs at component level. Similarly financial reports on expenditure covering fixed management fees and milestone payments can also be produced. 11. RESEARCH AND EVALUATION A small research wing can be placed under the M&E Unit to provide technical support to the Managing NGO to undertake research studies. Some of these can be outsourced and some can be done jointly by the wing and local experts. Following are some proposed studies. First five can be conducted during the first two years. Similar studies are proposed for pre-school and elementary education.

1. Baseline household survey This may include socioeconomic and educational information of both participant and non-participant households. Information on other educational facilities in the communities and reasons behind participation and non-participation can also be explored through this survey. The baseline survey can be done by the POs after inception of each learning centre. This database would be used in tracer studies conducted in future to explore external impact of the project (further education and livelihoods of the graduates).

2. A closer look into monthly refreshers training This qualitative study can document the way of conducting the monthly refreshers training. Whether the learning facilitators place their problems in the training and how their problems are addressed can also be explored. Training observation can be the main tool for exploration; however, in-depth interviews of POs and learning facilitators can also be included.

3. A study on community participation in project implementation The parents, other community members and local elite can participate in operation of learning centres in various ways. Who participates, how do they participate and what is their observation about the provision can be included in this study. Level of participation of the mothers in monthly parent-teacher meeting may have special focus of this study. Issues discussed in such meetings can also be included. Observation and in-depth interviews can be the ways of investigation.

4. An exploration of classroom teaching-learning This study would investigate classroom teaching-learning provision through observing a number of classrooms. Lesson plan preparation, way of teaching, students participation, teaching materials used, students assessment, remedial measures for weak students, co-curricular activities etc. can be the issues for exploration. Classroom observation may be the main technique of data collection; however, interviews of the facilitators and students can also be added.

17

5. Learning achievement of the pre-school learners A simple school readiness test can be developed to assess a sample of pre-school learners to know their level of achievement at the end of one year course. Learning performance can also be correlated with their socioeconomic background.

6. Mid-term project review A mid-term project review is suggested after 2-3 years of the start of the project. All major components of the project can be brought under this review exercise. A review team comprised of external and internal experts can be suggested for conducting the review.

7. Impact evaluation of the project An impact evaluation study against the project objectives is suggested six months before the end of first phase of the project. Similar to the above, all major components of the project can be brought under this study. Results of this study would help understanding the impact of the present project and would provide necessary inputs for possible extension of the project. A team comprised of external and internal experts can be suggested for conducting the impact evaluation study.

8. Gender analysis and identification of gender issues in ARMM A research study to inform the over-all BEAMM-ARMM Program will be carried out to further analyse the underlying causes of why girls outperform boys in ARMM. The research findings will inform the detailed implementation of activities of 4 components to ensure gender-fair teaching and learning processes.

GAD Summary checklist for the assessment of ADM Project in ARMM

Scores carried over Element or requirement

No Partly yes* Yes

Result

From Box 5 (p. 10) or sector-specific checklist

1. Involvement of women and men in project conceptualization and design (max score: 2)

2.0 Men and women were engaged and consulted in the conceptualization and design phase of the ADM Project.

2. Collection of sex-disaggregated data and gender-related information at the planning stage (max score: 2)

1.5 Analysis of ARMM BEIS data or education indicators reflected in the project design highlights sex-disaggregated data except for , teachers.

3. Conduct of gender analysis and identification of gender

The project design clearly incorporated activities addressing the

18


No Partly yes* Yes

Result

issues at the project identification stage (max. score: 2)

1.5 improvement of boys and girls participation and performance in target remote communities. The ADM M&E System ensures that quality education opportunities are equally accessible to marginalized boys and girls. In-depth analysis of gender issues will be undertaken prior to implementation

From Box 6 (p. 15) or sector-specific checklist

4. Presence of gender equality goals, outcomes and outputs (max score: 2 points)

2.0 Gender equality is explicit in the project’s objectives and targets. This is clearly illustrated in the recommended interventions across the project design and delivery.

5. Presence of activities and interventions that match gender issues identified to produce gender equality outputs and outcomes (max score: 2)

2.0 Gender equality outputs and outcomes are outlined in the four project’s components. Each of the project component explicitly incorporated activities and interventions that address gender issues in its actual delivery in remote communities.

6. Conduct of gender analysis of the likely impacts of the designed project

1.5 Gender analysis is included in the project’s design document. To reduce gender disparity is one of the primary foci of the project and this is

19


No Partly yes* Yes

Result

specified in the targets.

Gender strategies are embedded in the ADM Project components and this education model maximizes the involvement of women especially in the project implementation at the classroom level in far flung communities.

7. Presence of monitoring targets and indicators (max score: 2)

2.0 Gender equity is part and highlighted as one of the guiding principles and approaches of the ADM Project Monitoring and Evaluation System. This M&E system clearly incorporates targets and objectives to reduce gender disparity.

8. Provision of collection of sex-disaggregated data in the M& E plan (max. score: 2)

2.0 The project’s M&E plan puts emphasis on the collection of sex disaggregated data through out the project implementation.

Research to determine and address other gender issues will also be conducted.

9. Commitment of resources to address gender issues (max score: 2)

1.80 Addressing gender issues are incorporated in all key activities of each of the project component with specific budget items. An expert on GAD will be

20


No Partly yes* Yes

Result

engaged throughout the project implementation as well as in the M&E for all components.

10. Inclusion of plans to coordinate/relate with the agency’s GAD efforts (max score: 2)

2.0

ADM Project design considers all lessons learned from the recently completed Australian funded education initiatives in the Philippines.

TOTAL GAD SCORE FOR PROJECT IDENTIFICATION AND DESIGN

18.30

*As evident from the scores in Boxes 5 and 6 and the sector-specific checklists, the summary score for a “partly yes” for an element or requirement can range from “0.1” to “1.99.”

Documents

Adm