Module 5

Module 5. Monitoring and Evaluating a Child Rights Program Page 1

cleanreputations.com

The ultimate test of (project) management is achievement and performance.

Peter Drucker

Module 5

Monitoring and Evaluating

a Child Rights Program

Introduction Did you know that every plane that takes off is being tracked until it reaches its final destination? Tracking is done both by the pilot and the on-ground air traffic controller, each one using different tracing devices. According to an aircraft engineer, the pilot uses a “glidescope” which measures the plane’s vertical distance and a “localizer” which corrects the plane’s glide. The air traffic controller uses flight radar and GPS to trace where the aircraft is located, its speed, including its fuel capacity bearing. If there are troubles along the way, the air traffic controller gives advice to the air crew. Further internal to the aircraft is a computerized “transponder” that automatically signals the air traffic controller of the status of the aircraft while in mid-air. Up in the airplane’s cockpit are at least three crew – the captain, the assisting captain and the aircraft engineer, each one performing specific roles unique from each other. Any mishap that takes place is immediately known to the ground crew. Airplane accidents can be caused by many internal, external and extra-external factors, i.e. mechanical defect in the airplane, pilot error, maybe ground crew’s error as well, and worst is terrorism. This aircraft tracking is similar to what we do when CRPs are implemented. For program implementation, this is called monitoring. The airplane metaphorically can stand for the CRP and its aims or targets. The air traffic controller can be the CRP supervisor who does the monitoring while the plane’s crew are the main CRP implementer and other duty bearers. The gadgets represent the tools that the CRP implementers and supervisors use in monitoring project implementation. Thus, I know that you associate the accidents with problems in project implementation. But wait… Let’s go back to our story about the aircraft. Have you observed whenever you take plane rides those men and women who start clumping under the airplane upon touchdown and before another batch of passengers board? They are the aircraft engineers and technicians who check on the condition of the aircraft after every travel and ensure that everything in and about the aircraft is A-ok before it goes to another destination. This checking is what we call evaluation in CRPs.


"Projects go wrong for the same reasons all the time. There are no new sins. We can look at a

project in its first two months and know if it will be a success or not. Many organizations are failing to heed painful lessons learned from past projects"

This module will refresh you about what monitoring and evaluation are all about and how useful participatory methods are as consensus building and decision making tools.

Objectives At the end of this module, the students are expected to:

1. Discuss the principles of participatory monitoring and evaluation 2. Describe basic monitoring and evaluation approaches and techniques, and 3. Design practical tracking and evaluation methods and tools for their selected

CRPs.

Monitoring and Evaluating CRP milestones Monitoring is different from evaluation, although in some cases, people mix them

together, i.e. M/E. Monitoring is the systematic and continuous assessment of the progress of a piece of work over time (UNICEF, EAPRO Programme Strategy Toolkit). It is more concerned with the regular examination and tracking down of results of project activities being undertaken in a CRP.

Evaluation, on the other hand, is the systematic and objective assessment, at one point in time, of the impact of a piece of work against planned results. It is not done as frequent as monitoring. Both monitoring and evaluation are conducted to ensure that projects ultimately reach what have been set right at the planning stage. Now let’s go back to the PLOCS functions of management. What function do you think M/E fall on? Yes, M/E are control processes. Let’s recall what controlling means. According to Henri Fayol, controlling involve seeing that everything is being carried out in accordance:

with the plan that has been adopted

the orders that have been given

the principles that have been laid down

Its object is to point out mistakes in order that they may be rectified and prevented from recurring. It is about checking current performance against pre-determined standards contained in the plans, with a view to ensure adequate progress and satisfactory performance (EFL Breach ___). Don’t you think Fayol’s second description of controlling is what makes some implementers scared about M/E? M/E is not created to merely look for mistakes but also


seek for lessons, success stories and good practices which can serve as bases for scaling-up and expanding similar projects.

What about failures? Controlling implies that there are mistakes or failures to correct or mitigate. For most of us, failure is such a troublesome word. However, failures in projects shouldn’t discourage implementers. Failures may mean that:

The plan, right at the very first instance, was not practical and therefore has to be modified and improved.

The baseline or pre-project situation from which the project was based on has changed and therefore new strategies and activities may be required. This is true with individual or community beneficiaries, which are suddenly, stricken by disaster, war or plague.

CRP implementers need to update their capabilities so as to suit the very situation on the ground.

The timing is not right.

Resources are not sufficient to cope up with the demands of the present ground condition.

And many other reasons.

Failures are challenges. They are supposed to be precedents and therefore make us more resilient when faced with similar situations.

Why Participatory M/E in CRPs?

Participatory monitoring and evaluation is that process by which beneficiaries and duty bearers are involved in determining key results areas (KRA), documenting, collecting information, measuring achievements, analyzing results and giving feedback. The results of this process are useful in determining the next steps. Remember that we are encouraging ownership building here. Ownership includes the beneficiaries’ stake over all information related to the project. Evaluation results help define whether the project strategies used can be expanded, scaled-up or otherwise be shelved. Experts, implementers and beneficiaries can determine such analysis concertedly.

FAO also calls this process as Participatory Assessment, Monitoring and

Evaluation (PAME). PAME is characterized by FAO as:

A systematic recording and periodic analysis of information that has been chosen and recorded by insiders with the help of outsiders.

One that focuses on beneficiary- field staff and beneficiary-community relationships.


Is built on two-way communication, clear messages, and a joint commitment to what "works" for a certain community or beneficiary.

Is a combination of three interlinked parts: the idea, the methods and the tools.

However, FAO admits that while it may not always be possible to adopt the whole PAME approach in every project, it is possible to experiment with some activities to see if PAME works.

What has to be monitored and evaluated? The conduct of M/E basically looks back at what have been set at the preliminary stage of the CRP --- the plan. Recall our discussions in module 3. According to UNICEF, M/E basically looks at the project goal, outcomes, outputs and activities. See the table below.

Set at planning

stage

What to monitor/evaluate

Program/Project goal

Impact Evaluation Baseline with targets compared with the end of the program or project

Expected Outcomes

Outcome evaluation Baseline vs target (end line)

Project outputs Output monitoring and outcome evaluation Baseline, during and end line.

Project activities Implementation Monitoring: During activities Are things proceeding as planned? What are the quality, satisfaction and level of participation? Are there requirements for re-adjusting?


Principles of Participatory Monitoring and Evaluation

According to the EAPRO Programme Strategy Toolkit, a Participatory M/E process should:

1. Reduce the focus to a manageable range of activities 2. Build efforts around the standards and norms of the monitoring and evaluation

profession 3. Engage stakeholders 4. Make the terms of reference the key step in which quality is systematically

inserted 5. Take steps to ensure good implementation of the terms of reference; convert the

implementation effort into an evidence-based report. Besides UNICEF’s suggested M/E principles above, it is also important to note of the following guidelines when undertaking:

The process is consultative and participatory. The beneficiaries are the main characters involved in the actual conduct of M/E. Participation is voluntary and not-forced.

Duty bearers/CRP implementers can serve as facilitators in the process. Initial M/E steps should aim not just at gathering data but also at orienting and training the beneficiaries of the process so that the beneficiaries can conduct the succeeding M/Es with minimal intervention from the CRP implementers.

Involvement in M/E develops the sense of ownership of data or information gathered.

Values consensus building is encouraged.

Does not end with a written report but with movement based on the findings and recommendations.

Stakeholders (both beneficiaries and duty bearers) make use of M/E results for learning and improving.

CRP success is determined by not just looking at what were achieved based on set plans but also on how they were reached.

Analysis of results lies in the values of honesty, objectivity and neutrality. Gerison Lansdown also emphasized that “child participation” is best done with actual children’s participation. Involving the beneficiary children in the process is more effective. In order that their involvement is both ethical and effective, the following principled guidelines need to be taken into consideration:

Monitoring and evaluation with and by children is addressed during the planning stages, as an integral part of any project or program initiative.

Girls and boys (especially the most marginalized) are supported to participate in feedback, monitoring, evaluation, and follow up processes.


Confidential reporting mechanisms are made accessible to girls and boys in their local communities to ensure that they can easily share concerns or reports about child abuse which are then followed up sensitively and promptly by the appropriate agencies

Children are equipped with the skills and confidence to use participatory monitoring tools to support their active role in monitoring and evaluation.

Child sensitive indicators are developed with children enabling agencies to understand the priority concerns of children themselves and the goals they aspire to.

All information and findings are disaggregated according to gender, age, ethnicity, caste, religion, disability, HIV status, socio-economic status and other relevant factors.

Children are given rapid and clear feedback on the impact of their involvement, the outcome of any decisions, next steps and the value of their involvement.

The results of monitoring and evaluation are communicated back to the children involved in an accessible and child-friendly way and their feedback is taken into account in future work.

Feedback reaches all children involved.

Mistakes identified through evaluation are acknowledged and commitments given about how lessons learned will be used to improve participatory practice in the future.

Adults evaluate how they have translated and implemented children’s priorities and recommendations into their policies, strategies and programs.

Sustainability of support is discussed with children. Adults provide clear feedback to children regarding the extent/limit of their commitment to support children’s ongoing initiatives and organizations. If ongoing support is not possible, adults provide children with resources and support to make contact with other agencies who can support them.

UNICEF simply puts all these principles into three basic assumptions:

1. Stakeholders’ voices and knowledge should be the basis of defining success 2. Stakeholders should benefit and learn from the process of evaluating and

tracking results 3. M/E should enable stakeholders to be, and hold other authorities accountable for

results

Activity 5.1 Consider the monitoring and evaluation system of a CRP you are familiar with. Which of the principles discussed above are evident and not so apparent in the CRP? What could be the possible reasons for this?


Just as a navigator continually takes reading to ensure whether he is relative

to a planned action, so should a manager continually take reading to

assure himself that his project is on right course.

Donnel

When successive project milestones are missed this is a sure sign of a

project that is failing.

Monitoring CRPs In most cases, projects are tracked more rapidly during its preliminary or pilot stages. These are the stages when CRP implementers usually go through different challenges in building sense of trust of the beneficiaries towards the implementer and the project. Rapid monitoring is likewise conducted at the project’s tail-end since it is at this stage when the implementers need to determine if there are unmet objectives that the project needs to catch up with. In the EAPRO Programme Strategy Toolkit, UNICEF cited two concentrations of monitoring:

1. Implementation monitoring

Done on a continuous basis

Provides CRP implementers, management and other stakeholders with early indications of progress or lack thereof in the achievement of outputs

Answers the folloiwng: Is the program reaching the main beneficiaries? How are the stakeholders participating in implementation? Are all stakeholders satisfied with the program? Are all inputs, activities materials and outputs of good quality?

2. Performance monitoring

Measures progress in achieving specific results in relation to an implementation plan, whether for programs, strategies or activities.

A core accountability component for effective work planning and review.

Will be useful as an evaluation framework when looking at specific items set during the planning stage

Answers the following: How far are we from set targets? What are the reasons for the fast or slow achievements? What are the difficulties surrounding our achievements?


AMORE-Coca Cola Foundation WASH Project Turnover amore.org.ph

To further describe participatory monitoring, let us pretend that we are all inside a jeepney. A jeepney has open windows and this does not deter the passengers to look outside and determine whether he/she is near his/her destination. The driver and passengers may altogether see the rugged portions of the road. Sometimes, it is the passengers, which alert the driver of a landslide, a child crossing the street, another passenger waving for a ride, etc. Once it rains, jeepney drivers unroll the window plastic curtains, which then make it difficult for the riders to determine their exact location. Participatory monitoring is like riding a jeepney on a clear day and with uncovered windows. It is collectively tracking project results.

Uses of Participatory Monitoring Participatory monitoring yields the following benefits to CRP implementers and beneficiaries:

Allows CRP implementers and beneficiaries to make necessary and immediate actions on project-related problems and concerns

Serve as information bucket needed in the review and refinement of short term plans as well as in project evaluation

Basis for updating plans, budgets and staffing requirements

Serves as staff-presence and -performance tracking Only with adequate and timely information is it possible to know the scale of the problem, to engage in evidence-based advocacy, to assess progress, or to learn from successes and mistakes. Without adequate monitoring information, it is untimely impossible to reach the goal of projects (Wash in Schools: Monitoring Package, UNICEF 2009).

Use of Team Monitoring Monitoring becomes less burdensome when we create teams. Team formation allows the sharing of load as well as exchange of ideas when analyzing data gathered. It also avoids bias when dealing with sensitive and critical issues or findings. The team however should:

be made aware of what to monitor (activities, outputs, timeline, actors)

be trained how to gather data using the tools or methods designed for the project and agreed-upon with the team members


be equipped with the necessary materials and equipment when conducting monitoring

Sample Monitoring Tools and Techniques

There are several CRP monitoring tools that can be used. But you are encouraged to make your own set, try other modes that will encourage active participation and honest delivery of data from beneficiaries. Some examples that you might want to browse over are: The Wash in Schools: Monitoring Package, UNICEF 2009. This a booklet of different monitoring questionnaires, observation checklists, and Focus Group Discussion tools (pages 33-85) used for gathering information about Wash Schools and updates regarding water, sanitation, hygiene and waste disposal and drainage. See: http://www.unicef.org/wash/schools/files/wash_in_schools_monitoringpackage_.pdf The Child Protection System: Comprehensive Mapping and Assessment, UNICEF and Maestral International, January 1900. This is a toolkit on how baseline information for CRPs can be gathered. The same or portions of it can be used in the conduct of monitoring. See: http://www.unicef.org/aids/files/UNICEF_webinar_111010.pdf Participatory Assessment, Monitoring and Evaluation, FAO. This describes how M/E is done in a participatory manner. It includes some techniques and tools tested by FAO community development practitioners. See: http://www.fao.org/DOCREP/006/T7838E/T7838E02.htm Can you cite some more CRP M/E tools?

Activity 5. 2

This is a reading by Jade Mingus about an actual fund raising drive for children. Read through. What monitoring tactics or methods and set of KRAs should we set if this project is ours and if we want to scale-up the effort?

Artists, chefs and generous Central Texans came together Sunday to fight childhood hunger. The 15th annual Austin Empty Bowl Project drew hundreds to the Marchesa Hall and Theatre. The handmade ceramic bowls are tools in the fight against childhood hunger. They came in all shapes, sizes and colors at the Austin Empty Bowl Project. "This is my fifth or sixth year coming, and we always get here really early," Joy Kohl said. Kohl is among the hundreds who lined up outside the Marchesa hall and Theater in North Austin. For a donation of $20, they could bring home the bowl of

http://www.unicef.org/wash/schools/files/wash_in_schools_monitoringpackage_.pdf

http://www.unicef.org/aids/files/UNICEF_webinar_111010.pdf

http://www.fao.org/DOCREP/006/T7838E/T7838E02.htm


We learn little from victory and learn more from defeat.

Japanese Proverb

their choice, and have it filled with soup. "I'm looking for the ugliest bowl I can find," Kohl said. All the bowls were created and donated by local students, artists and potters. "We never know what you are looking for -- we just come and see what strikes us," Kohl said. A few people had trouble choosing just one. "I like the Texas Longhorns because I'm a Texas Longhorn at heart," one shopper said. "This is my sister's because she likes turtles. These I'm getting for my sisters-in-laws." Ninety special bowls were part of a silent auction. Many had celebrity signatures form Paula Deen to Joan Rivers. All the money raised benefits the Capital Area Food Bank's after school program, Kid's Cafe. Every $20 buys 45 meals. After volunteers washed their bowls, generous Central Texans got to eat. Hoover Alexander from Hoover's Cooking, was among the local chefs donating gourmet soups."We do soup every year, and really look forward to it," Alexander said. "It sets the tone for the holiday season. It's a season for giving and sharing -- it's a perfect venue for that." It's also a perfect way to fight childhood hunger, one bowl at a time. Source: Empty Bowl Project benefits hungry children at http://www.kvue.com/news/Austin-empty-bowl-project-

benefits-hungry-children-134203618.html

Evaluating CRP Outcomes and Impacts According to UNICEF, like monitoring, it is also advisable to engage the beneficiaries

and duty bearers in the evaluation process. Such process is called Participatory

evaluation. Evaluation helps in adjusting and redefining objectives, reorganizing institutional arrangements or re-allocating resources as necessary. Monitoring and evaluation allow a continuous surveillance of the CRP. In evaluation, we look at 2 main things:

Performance Evaluation - measures quality and efficiency of a CRP; if it is organized and it met the activity and output targets.

Impact Evaluation - measures the results of a program or policy on children, their families and their communities. It is concerned with M/E of both performance and situation.

http://www.kvue.com/news/Austin-empty-bowl-project-benefits-hungry-children-134203618.html

http://www.kvue.com/news/Austin-empty-bowl-project-benefits-hungry-children-134203618.html


Need External Evaluators? There are times when external program evaluators are contracted to conduct evaluations. This is primarily because CRP organizers or coordinators are keen at:

objectivity

fairness

truthfulness

speedy completion (which local, organic staff complain about because of many other tasks, pending activities and reports, and the like)

Participatory Methods in Monitoring and in Evaluation There are many ways of conducting participatory M/E which makes the process:

Less intimidating to informants

More informal

Enjoyable and fun

Educational (as information sets are processed and discussed with the beneficiaries)

Leading to collective analysis and consensus decision making

Types of evaluation CRPs can be evaluated based on the type and schedule of this activity. While other projects propose only items a-c, item d is also recommended as an important part of evaluation:

a Pre-project evaluation Conducted even before a project begins. It serves as a situational or environmental scanning method. It intends to establish the baseline of the project

b Mid-project evaluation Conducted anytime during the project implementation. It can be done on a regular schedule (e.g., annual, biennial or at the completion of every main activity) to get a sense of how the project is accepted by the target beneficiaries; checks if the planned strategies and activities match the real situation/need. It checks, at the minimum scale, the quantity and quality results and outputs of project activities, and at the maximum the concrete outcomes if there are any.


Example: Elementary schools provided with computer and science laboratories; improvement of grades and performance in national achievement tests

c Project terminal evaluation

Conducted at the completion of the CRP. The intent is to check the outcomes of the activities or strategies, and at the maximum are the impacts of the project based on set objectives. Example: Objective: Develop among the out-of-school youth (OSY) the entrepreneurial skills i.e. silk screen and souvenirs production Evaluation focus:

Expressions of behavioral and skills changes among the youth

Expressed benefits derived by the OSY from the project

Success and failed stories/realizations

Income/profit raised from the enterprise

Reactions and testimonials from the OSY parents, barangay officials, non-participating OSYs, and the general community

Awards & citations received so far

Etc.

d Post-project evaluation

Conducted after the CRP has been terminated. This may be done months or a year after project completion. It tries to look at how the project objectives have been sustained. Ex. CRP on Cyber-Sex Watch in University Belts The focus may be on:

Sustained zero cyber sex operation

Active daily and spot rounds by Barangay ‘tanods’ of internet cafes and dormitories

Policy developed and enacted on anti-cyber sex

Sustained campaign on anti-cyber sex


blog.kevineikenberry.com

Activity Which of the previously described evaluation approaches are being used in the CRP you are familiar with? How has this type of evaluation enhanced your operation? Did you encounter any difficulties in implementing this evaluation? Why?

The Use of Documentation

Documentation of processes and outcomes is an important aspect of a CRP. M/E is not complete without complete recordings of events and outputs, otherwise everything will all be hearsays and considered baseless therefore. One of the more popular ways of conducting documentation of projects is Process Documentation or PD. PD intends to capture how a project is introduced, planned and transpired in a community; how it is accepted or rejected by the participating beneficiaries; and why they reacted so to the project interventions. PD is useful especially when:

Piloting new strategies in a new area or with a new set of beneficiaries

Piloting new strategies in the same area or with the same beneficiary

Expanding old strategies in a new area or with a new set of beneficiaries

You can read more about Process Documentation. There is a mass of materials about it available now in the net.

Feedback Giving

Feedback is a process by which results of the M/E are relayed to the stakeholders, i.e. beneficiaries, funders and partners. Even if a certain sector was involved in the actual M/E process, it is still best to feedback the results to them. Feedback is also given as an immediate natural reaction to activities conducted, statements expressed, body language, written reports, announcements, and many more. The success of the feedback lies on how the feedback giver delivers the feedback and on how the feedback-recipient takes the message. It is therefore is two-way communication, and is called the “feedback loop”. Feedback allows both parties (feedback giver or recipient) to be heard and to speak as well. Hence, it is important for both parties to take the feedback loop or the giving and taking process objectively.


Taking feedback karatebyjesse.com

The two-way process leads to the validation and analysis of data, and ultimately in the determination and agreement on the next actions to take. Its purpose is not to dwell on the present condition but on what needs to be done in the future. Recipients’ reactions to a feedback may graduate from rejection, defense, explanation, understanding, and to changing. Others may react to feedback positively without going through the rejection or denial. Every feedback giver aims for a reaction that leads to change. Taking from the adage “love begets love” and using the illustration in the figure above, it is important to note that these 5 reactions may also bring about another set of responses from the feedback giver, causing therefore a cycle of feedback responses. For instance, if there is rejection and defense on the part of the one being recipient, then a feedback giver may continue to insist on his message (e.g., findings and analysis), entailing further explanation of the idea. The feedback giver may have to modify his/her message and style. This is especially done if he/she senses that there is total rejection. When a feedback recipient tries to explain his/her side, idea or position, feedback giver may have to do further reasoning. This third reaction from a feedback recipient may turn into the fourth reaction (i.e. understanding). However, if there is poorer reasoning on the part of the feedback giver, then the feedback recipient may turn into defense and rejection of the idea or proposal. Hence, feedback givers must work to help the feedback recipient understand the idea, so in the end, the latter may be expected to change (i.e., improve performance). Once the recipient gains understanding and acceptance of improvement suggested by the feedback giver, the latter will only need to mentor and ensure that the change process is undertaken smoothly.

Activity 5.3 Some feedback recipients do not easily take feedbacks. They also go through the process of reflection and analysis before they admit understanding and thereafter improve or change based on the M/E findings.

Feedback recipient’s reaction Rejection Defense Explanation Understanding Change Insistence/ Advocacy/modify message Reasoning Mentoring

Feedback-giver’s corresponding reaction


You must have encountered this many times in your workplace, in CRPs or other projects. How did you approach such feedback recipients?

Institutionalizing best practices and Ensuring Program/Project Sustainability

Sustaining a CRP is an indication that initial efforts of the project are effective and efficient. Lessons and good practices derived from CRPs will become more useful if they are institutionalized. Institutionalization is the process and stage in a CRP when experiences are formulated into policies, new programs and new strategies or the CRP itself is scaled-up or expanded in other areas. Some of the ways by which lessons and good practices from CRPs are institutionalized and disseminated are through:

Case study development and publication

Video documentation of testimonials, stories and good practices

Discussion fora and conferences

Submission of manifesto or position paper from the beneficiaries (and implementers) about a concern well-documented by the CRP

Audience to policy makers to advocate item f

Policy recommendations

Media coverage

Representation of the sector in local and international gatherings about children rights

Turning-Over CRPs

To make full ownership of the CRP, it is apt to consider turning-over its management to the ultimate beneficiaries, in due-time and in the most appropriate manner. The sense of timing can be gauged either according to target project period or according to the capacity or readiness of the beneficiaries. International donors practice this turn-over of projects to local partners (e.g., local government structures, NGOs, community people’s organizations, families). Local implementers can also do the same to their local partners and beneficiaries.


References: Gerison Lansdown. A Framework for Monitoring and Evaluating Children’s Participation: A participatory draft for piloting (May 2011) Wash in Schools: Monitoring Package, UNICEF (2009) Participatory Assessment, Monitoring and Evaluation (PAME), FAO. 2007. The Art of Giving and Receiving Feedback. Shirley Poertner & Karen Massetti Miller. 1996

Documents

Module 5