EVALUATION OF SIGHTSAVERS INNOVATION FUND ROUND 1: FINAL REPORT

MAY 2013

TIGER CONSULTINGMary Ann Mhina: maryanmhina@gmail.com

Graham Bennett: grahambennett777@hotmail.co.uk

Jill Edbrooke: jilledbrooke@hotmail.com

Lorraine Wapling: consultant@wapling.me.uk

1

CONTENTS Executive Summary (page 4) Chapter 1 –Introduction and Background to the Evaluation of Sightsavers

Innovation Fund Round 1(page 7) Chapter 2 – Methodology (page 11) Chapter 3 - Bangladesh: Disability-Inclusive Disaster Preparedness and

Management focusing on Alternative Livelihoods (page 13) Chapter 4 - Tanzania: Zanzibar Comprehensive Eye Health Project (page 19) Chapter 5 - Pakistan: Community-based Diabetes Prevention and Control (page 24) Chapter 6 - Pakistan: Ensuring social inclusion through accessible water and

sanitation (page 29) Chapter 7 - Findings and Overall Recommendations (page 34) Chapter 8 - References and Appendices (page 37)

List of Appendicesa. Evaluation Terms of Referenceb. Innovation Fund Guidelinesc. Topic Guide (initial UK interviews)d. Topic Guide (Country Visits)e. People consultedf. Inception Report

Acronyms and AbbreviationsAIEH Al Ibrahim Eye HospitalBCO Bangladesh Country OfficeCEC Comprehensive Eye CareCMP Commitments Management ProcessCO Country OfficeCP Community PharmacistsDFID Department for International Development, UK GovernmentDPO Disabled People’s OrganisationDRB Diabetes-Related BlindnessDRRA Disabled Rehabilitation and Research AssociationFGD Focus Group DiscussionsHRDS Human Resources Development ServicesICON Incidence and Causes of Ophthalmia NeonatorumIEC Information/education/communicationLHW Lady Health WorkerMoHZ Zanzibar Ministry of HealthMCH Mother and Child HealthMCBI Muhimbili Child Blindness InitiativeMoH Ministry of HealthON Opthalmia NeonatrumPC Project Coordinator (Zanzibar)PCO Pakistan Country OfficePHL Public Health Laboratory, Pemba

2

PPA Programme Partnership ArrangementPWD Disabled person (Person with Disabilities)RCH Reproductive and Child Health workers (Pemba)SHG Self Help GroupTCO Tanzania Country OfficeVHC Village Health CommitteeWatSan Water and Sanitation

AcknowledgementsThe evaluators would like to thank all the partners and staff of Sightsavers International who assisted us with our work, especially those in the four countries we visited; the CO staff, in particular, accepted us and opened up their work to us. We would also like to thank the very many disabled people and children and families that we met, who inspired us with how they are changing their own lives.

Mary Ann & Graham

3

Executive SummaryThe Innovation Fund is part of the Sightsavers Programme Partnership Arrangement (PPA) with the UK’s Department for International Development (DFID). It was established to encourage innovative projects to be tried and tested, with the aim of learning lessons that could lead to breakthroughs in how Sightsavers does its development work and, through sharing and communication, influence development actors including DFID and other agencies working to improve eye health. The fund primarily used DFID’s definition of innovation, which states that innovation does not “necessarily mean ‘brand new’ “(e.g., innovative practices, innovative partnerships, innovative approaches). It noted that there is an “element of experimentation” involved, with the “risk of failure”1. Allocating part of its PPA to encouraging innovation is a positive and creative move by Sightsavers. This evaluation also shows clearly that some excellent development outcomes are being delivered through the Round 1 projects.

Five projects in four countries were selected in Round 1 of the Innovation Fund in September 2011 with the intention that they would start as soon as possible though in practice some projects started later due to a variety of issues. The selection process involved initial recommendation by UK-based Sightsavers staff, who referred promising projects from the Commitments Management Process (CMP) to Innovation panels of internal and external experts for further assessment. The Nigeria project was excluded from the evaluation due to its nature and duration.

This evaluation was commissioned in late 2012 to investigate progress, to examine each project using three basic criteria (structure, process and outcomes/results), and to make recommendations for adjustments and changes to ensure each project could deliver on its aims. This included asking questions about the way the projects were organised to deliver their results, what monitoring systems were in place to track progress, what additional inputs might be needed to ensure that desired results would be delivered on time and to budget, and what plans were in place to collect, analyse and communicate the lessons learned from the projects. The evaluators were also required to review what plans were in place to scale up and replicate projects and methods that had proved to be successful and innovative.

The evaluation used desk research and initial semi-structured interviews to establish the background and rationale for the Innovation Fund and how its projects were selected. Each project was then visited on the ground, where interviews were conducted with implementing Country Office (CO) staff and partners of Sightsavers, as well as beneficiaries and other relevant stakeholders such as advocacy targets and private sector collaborators. This phase also involved observation and examination of locally-produced data and information, as well as participation in various group discussions and project-related seminars and forums.

The evaluators found that the projects varied enormously in aims, content, duration and budget, and in general were broadly on track to deliver their objectives as set out in their logframes. Concerns had been expressed (e.g. in the TOR for this evaluation) about the progress of these Round 1 projects, but we found these concerns were generally unfounded. Work on the ground was seen to be having real impact; in Pakistan, for

1DFID GPAF Innovation Fund Guidelines; http://www.dfid.gov.uk/Documents/funding/gpaf/glob-pov-act-fnd-guide-appls.pdf

4

example, inclusive and accessible water and sanitation was making fundamental changes for how disabled people were living, for their confidence, and for how they were accepted as part of their communities. The work in Bangladesh on including disabled people in disaster planning, and in Tanzania on strengthening provision of eye care, were also making a difference for target communities. We have written a chapter for each project, highlighting achievements and, where relevant, areas that could be improved for that specific project.

The evaluators addressed the issue of whether the projects are innovative, since their ultimate success, according to the guidelines for the Innovation Fund, is tied to their delivery of lessons and to learning and the possibility of replication and scale-up. Applying the full implications of the DFID definition including the aspect of experimentation and risk, most projects contained activities that were innovative, though some of the techniques and approaches, while effective and appropriate, were not innovative. We recommend that Sightsavers is very clear, before the stage is reached of documenting and communication learning, exactly what is the innovation being highlighted, to aid future replication and scaling-up. We did not find risks had explicitly been taken, and we recognise risking possible failure is counter-intuitive, especially when spending donor money. We would encourage management to support more risk-taking, in the spirit of the Innovation Fund, because the potential gains are enormous.

We found the main area that needs addressing is the planning for lesson-learning, scaling up and replication. There were positive examples (e.g., the links being established in Pakistan with Comsats University to document the lessons from the water and sanitation work); but even though attempts such as this to collect learning from some projects were in the pipeline overall the projects lacked a clear and consistent approach to this essential aspect. This lack of evidence may be due to the relatively early stage in the Round 1 projects’ cycles at which the evaluation was carried out, and there is certainly time to adjust this aspect through better communication about what is expected and the provision of external resources if required. The lack of focus on the learning commitment appeared to be a result of the Round 1 process where project proposals were not originally written for the Innovation Fund, with its additional conditions, with the result that some CO staff did not realise the full implications in terms of learning and communication.We understand this has been addressed in Round 2 of the Innovation Fund

One suggestion from the evaluators to increase risk-taking and innovation, and therefore to ensure there are breakthroughs in new approaches, partnerships and techniques to document and share, is to use the Innovation Fund for shorter, explicitly experimental projects. These could be more focused in scope and more clearly pilot projects that could, if successful, lend themselves more readily to generating learning and to scaling up. The Innovation Fund is a good idea; it requires conviction, courage, leadership and good communication within Sightsavers to make it work, not least in supporting CO staff and overseas partners to be bold. The Innovation Fund has the potential to deliver, to the benefit of Sightsavers and to others who learn from its lessons, if innovation is actively encouraged, as we understand is happening with Round 2.

Evaluation Matrix: Overall Rating for Innovation Fund Round 1 projects (i.e., are they “on track”?)

GREEN = evidence that the project is on track for that particular criterion

5

AMBER = evidence that, with some adjustments, the criterion will be satisfactory

RED = lack of evidence, or evidence that major adjustments are needed

Project Rating

Bangladesh: Disability Inclusive Disaster Preparedness and Management focusing Alternative Livelihood

GREEN

Tanzania: Zanzibar Comprehensive Eye Health Project

GREEN

Pakistan: Community-based Diabetes Prevention and Control

AMBER

Pakistan: Ensuring social inclusion through accessible water and sanitation

GREEN

6

Chapter 1Introduction and Background to the Evaluation of the Sightsavers Innovation Fund Round 1 The Innovation Fund is an integral part of Sightsavers’ PPA) with DFID. The PPA provides Sightsavers with an estimated £11.2m between 2011 and 2014. Sightsavers originally allocated approximately 5% of the PPA (£750,000) to the Innovation Fund but in practice, following the selection process that took place in the second half of 2011, £1.1m was made available for Round 1 of the Innovation Fund. The Innovation Fund Terms of Reference (TOR) outline the need for innovation in order to improve quality under three main thematic areas; eye health, inclusive education and social inclusion. The emphasis is placed on ‘“out of the box” approaches and projects that hold the promise of effecting large scale change and contribute to global learning’. The guidelines for Round 1 of the Fund state that it “requires projects that clearly demonstrate and provide evidence of why the work is innovative, how success is to be disseminated, and its impact. Projects will have a clear plan for replicating and scaling up the successes”. They also say that the Fund “will allow for a degree of risk-taking necessary when developing new approaches”.

What is Innovation?

Innovation can be achieved, according to the Fund guidelines, through innovative practices, innovative partnerships, and innovative approaches. Like DFID’s evaluation guidelines, Sightsavers Innovation Fund explicitly states that innovation does not necessarily mean brand new but may instead be testing a new approach or practice in an existing context.

How innovation is defined is critical to the working of the Innovation Fund, to which projects were selected and how, to the assessment of success, and to the learning achieved as a result of the projects. It is understood that Sightsavers is currently working on a more refined definition of innovation and an overall Innovation Strategy; however in this evaluation innovation has been assessed using the following criteria drawn from the TOR for the Innovation Fund2:

Do projects have a clear plan for replicating and scaling up successes? Do projects have a rigorous design approach? Do projects demonstrate innovative practices, partnerships or approaches? Do projects demonstrate experimentation and an element of “risk of failure”

(DFID, as quoted in the Innovation Fund guidelines) Are projects likely to lead to large scale change or contribute to global learning? Can those implementing the project explain the innovation?

In assessing the degree of Innovation of each specific project, account is taken of the context and existing work by other agencies as well as the project work observed during field visits to assess whether these key criteria are met. Innovation is reviewed both in the local context of each project and at organisational, national and international levels. The relative innovation of these projects and the extent to which they are likely to affect large scale change and global learning is considered.

Projects were selected for the Innovation Fund initially because staff in the UK felt that they had potential for innovation; applications were redirected from the Commitment Management Process (CMP), which considers all Sightsavers projects. Proposals

2Sightsavers Programme Partnership Arrangement: Innovation Fund Guidelines (no date)7

submitted to the Innovation Fund went first to an internal and then to an external selection panel before they were accepted. All of the projects supported during Round 1 are managed by Sightsavers country offices with various partners. Funds were allocated in Round 1 to five projects in four countries:

1. Bangladesh: Disability Inclusive Disaster Preparedness and Management focusing Alternative Livelihood (3-year duration, cost = £67,824, 7.3m BDT))

2. Tanzania: Zanzibar Comprehensive Eye Health Project(5 years, cost = £395,599, 989m TZS)

3. Pakistan: Community-based Diabetes Prevention and Control (3 years, cost = £95,200)

4. Pakistan: Ensuring social inclusion through accessible water and sanitation (3 years, cost = £600,350)

5. Nigeria: Integrating primary eye care and primary ear care into Patent Medicine Vendors’ work (1 year, cost = £70,886, 12.5m Naira)

The Nigeria project, largely research-based and of one-year duration, was considered different in nature to the other projects, and has been excluded from this evaluation.

Purpose of the evaluationThe Terms of Reference (see Appendices) set out the purpose, framework and timescale for this evaluation. Primarily, the TOR asked the evaluators to look at three areas for each project in the Innovation Fund Round 1, with questions under each:

1. Structure (e.g., how partner and Sightsavers are set up to manage and coordinate the project, systems of documentation and monitoring, etc)

2. Process (e.g., barriers to progress, additional technical inputs, project revisions needed, etc)

3. Outcomes/Results (on track, scope for replication, etc)

In addition, the evaluators committed to giving some basic assessment of the progress of the projects according to current standard DFID criteria (e.g., value for money, sustainability, efficiency, effectiveness, etc).

The TOR also state explicitly that “significant progress has been made in one of the [above] projects while in the others a number of concerns have been expressed as progress has been slower than anticipated”. Part of the evaluation’s rationale was therefore to “validate these reports and concerns and to draw out recommendations on how to ensure that funded projects deliver on expected results”.

Critical points of interestAs the evaluation developed, both at the inception stage and in the field visits, a number of key points emerged which had a bearing on the purpose and scope of the evaluation.

(1) “Concerns”: though explicitly stated in the TOR, the concerns relating to the projects seemed to have lessened as the inception stage progressed. Internal financial reports showed slow expenditure rates and the Independent Progress Review (IPR) carried out as required by DFID on the Sightsavers PPA showed expenditure of £16k on the Innovation Fund against an allocated budget of £1.1m,

8

suggesting substantial delays and/or under spending, with the inference of slow or non-existent project implementation. Subsequently, underspends were shown to be a consequence of a delayed start dates for the projects, due to miscommunication and misunderstanding about dates for release of funds. Nevertheless the project visits set out to validate the extent to which the concerns did or did not remain valid.

(2) Selection process of Innovation Fund Round 1 projects: An understanding of the Round 1 funding process was felt to be significant to this review because of the extent to which the selection and set-up of projects was likely to impact on demonstrated innovation and achievement of learning and scale-up. The importance of the selection process became clear when the findings of our field work suggested that most of the projects submitted by country offices were not initially designed for the Innovation Fund, but were submitted to the Sightsavers CMP. This raised issues for us about whether CO understood the particular conditions they would be expected to satisfy if awarded funding, especially those relating to lesson-learning, documentation and communication. Such conditions might not unreasonably be seen as part of any project’s conditions; but since the Innovation Fund lays particular emphasis on these processes, an assessment of progress such as this needs to review the processes and systems that have been established to ensure that these things happen. Since the selection of projects for Round 2 had already taken place, it was clear that this review was not intended to evaluate and make recommendations specifically about the selection methodology. Therefore this review does not comment on the selection processes themselves but does look at how those processes have affected how these projects are perceived and implemented by both CO and the staff in other parts of the organisation (Programmes, Policy and Programme Support, Regional Offices etc.) and addresses the extent to which these have, or have not been effective in producing the kind of work which the Innovation Fund was established to encourage.

(3) Comments from the Innovation Selection Panel on Round 1 projects: the documentation provided to us as evaluators indicated that there were many comments from the external members of the Innovation Panel on the Round 1 projects. During project visits, the extent to which these comments had been acted on before projects went ahead was reviewed. No documentation was provided to us that showed revised applications with comments incorporated; it was not possible to know, therefore, whether comments had been accepted.

(4) Timing of the evaluation: the Round 1 Innovation Fund projects were expected to start soon after the selection date of September 2011. In practice some started in April 2012, and in exceptional cases even later (e.g., Bangladesh, where the NGO Affairs Bureau approval process resulted in a seven months’ delay). This meant that in some cases projects had only been operating by the time of field visits for 9-10 months of a three-year cycle (or 13 months of a five-year cycle in the case of Zanzibar). We had therefore to make an assessment of what results and progress could reasonably be expected in that time frame.

9

10

Chapter 2

Methodology Each of the five Innovation Fund Round 1 projects has been evaluated using the basic framework outlined in the TOR: structure, process and outcomes/results. In addition the projects are rated in relation to DFID’s evaluation criteria, namely Efficiency, Value for Money (VFM), Results, Effectiveness, Innovation, Sustainability, Lesson-learning and the potential for Scale-up/replication. The specific question, is the project innovative, is also addressed both specifically in relation to each project and in general of the projects in Round 1.

Inception: In the Inception and Initial Review stage a thorough document desk review was carried out and this was supplemented with five semi-structured interviews with Sightsavers’ UK-based staff from the Policy and ProgrammeStrategic Support Department. These interviews utilised the Topic Guide (see Appendix c). As the projects were very different in nature it was felt that all five needed, in the first instance, to be visited and assessed separately. In order to develop some consistency, tailored Topic Guides (see Appendix d) were developed for use in all four projects. Field visits were planned to all five projects, with five/six days allocated for each country visit. What evaluation activities could be undertaken during this time varied depending on the project site, security, travel and logistics?Data Collection; Between 13 January and 9 February 2013 visits were made to all four countries for the purpose of data collection. During the visits the methodology included semi-structured interviews, direct observation of project activities and focus group discussions (FGDs). Discussions and interviews were carried out with both implementers and where possible end-user beneficiaries. Because these projects are still in their early stages more emphasis was placed on implementers (at various levels), as process and structure were key focuses for this evaluation. In some cases where DPOs are implementing activities, for example, the implementers and beneficiaries overlap.

The evaluation involved many semi-structured and informal interviews and interactions with staff, beneficiaries, partners and other key stakeholders, and used various group meetings (including FGD and seminars) to gather data for later analysis and interpretation. Meetings and individuals spoken to are detailed in the appendices.Challenges encountered during field visits

Bangladesh: the project sites in Bangladesh were remote, involving use of local transport (e.g., cycle rickshaws, boats) to visit particular visits. There was also a general strike on one day, related to the judicial process for a group of anti-liberation war, pro-Pakistan activists, which meant all motorised transport was being physically attacked, and was therefore not an option. This experience of remoteness and distance from Dhaka was important context for understanding the challenges and achievements of the project (e.g., logistical and communication issues, difficulty of transport, physical isolation of some communities).

Tanzania: During the field visit the time allocated by the CO to the evaluation was limited. The findings of the evaluation in Tanzania are based largely on the field visits to project sites in Zanzibar and therefore the comments made in this report in relation to structures and processes are observed primarily from the perspective of the implementers in the field. Follow up work was done from the UK to clarify issues by email with the CO during the analysis phase of the evaluation, which provided some additional information.

11

Pakistan: the visit to Pakistan was delayed for two weeks because the security situation in Islamabad was a concern. In Karachi, where some of the project sites are based, there was further recurrence during the visit of the killing of female anti-polio campaigners working for the Red Cross, as well as bombings in Quetta. Security arrangements at all public buildings were extensive. All this is important context for understanding the challenges of doing development work in Pakistan, both for local and international organisations. Internal travel was also disrupted (e.g., flights cancelled) due to adverse weather conditions, another factor that, with the large distances involved, necessarily limited direct observation of some of the projects (e.g., water and sanitation work in Chitral, in the north-west, was inaccessible due to heavy snow).

Data Analysis: Following the field visits, two discussions with Sightsavers UK staff and two internal Tiger Consulting workshops have taken place to dodata analysis and compare outcomes so far with reasonable expectations of where the projects should be at this stage, Team workshops included Jill Edbrooke (JE) and Lorraine Wapling (LW) who helped to ensure that the analysis was rigorous and inclusive.

A coding system was used to identify the most common themes, which emerged from the overall analysis, including individual stories and data collected from group discussions, as well as our own observations. The quantitative data we were able to collect was mostly about project outputs; where possible we did collect relevant data about beneficiary circumstances when relevantto the purpose of this review. In FGD we applied Most Significant Change techniques, asking groups to identify the ‘most significant thing’ that had happened” and the ‘most significant challenge’ which they faced.

During this process we used of a traffic light indication of progress (see below); red areas will be basis for recommendations.There was further consultation with country offices where necessary to clarify any issues or inconsistencies that emerged during analysis.

Where challenges with project implementation, structure or process were identified during field visits,Action Research techniques, (i.e., asking informants, as part of the evaluation process, what strategies they felt would best solve challenges they faced) were utilised to identify solutions

As well as a concise narrative exploring these criteria for each project, we use the matrix below as a summary of our assessment across each project on a range of criteria that includes those currently in use by DFID in its key funding schemes such as the PPA and GPAF. We have used the traffic light system as a quick indication of project/criterion status; the coding is explained below, and is used for each project, as well as an overall assessment in the Executive Summary

GREEN = evidence that the project is on track for that particular criterion

AMBER = evidence that, with some adjustments, the criterion will be satisfactory

RED = lack of evidence, or evidence that major adjustments are needed

12

Chapter 3Disability-Inclusive Disaster Preparedness and Management focusing on Alternative Livelihoods (Bangladesh: 3-year duration, cost = £67,824)IntroductionBangladesh is a country prominently at risk from natural disasters such as cyclones, flooding, water-logging and earthquakes, primarily due to its geographical position and its low-lying coastal lands. Within this context, Bangladesh has a high population density, not least in its southern coastal areas that are most impacted by flooding and tidal surges connected to major cyclones. Many of those living in vulnerable areas have low incomes, and rely on agricultural production and related services. Such livelihoods tend to face massive disruption in natural disasters, either temporarily or permanently (e.g., areas that used to have fresh groundwater are now experiencing saline contamination, as observed in this evaluation).

The government and citizens of Bangladesh have worked to put in place better plans and infrastructure to deal with natural disasters, exemplified by the cyclone shelters that stand out on the flatlands, commonly schools or other civic buildings that have an upper floor that acts as a refuge from floods. However, disabled people have largely been excluded from both the physical forms of protection (e.g., many shelters are inaccessible to physically disabled people), and also from the planning and disaster preparedness processes. This project, funded with £67.8k from the Sightsavers Innovation Fund, aims over three years to address in particular the absent voice of disabled people from disaster planning, as well as examining the issue of livelihoods for disabled people living in disaster-prone areas.

The project is managed primarily by Disabled Rehabilitation and Research Association (DRRA), an established Bangladeshi NGO, with support for the Sightsavers Bangladesh Country Office (BCO), and its logframe outlines the following summarised aims:

Overall Objective: National policies and processes on disaster risk reduction are responsive to the needs of disabled people (people with disabilities, PWDs)

Objective 1: Enhance capacities of PWDs to demonstrate effective disaster risk reduction

Objective 2: Explore evidence-based alternative livelihoods for disability-inclusive development

Objective 3: Build capacities of various stakeholders (e.g., local councillors, Union councils) to include PWDs in disaster planning

Objective 4: Policy advocacy to strengthen national systems for disability-inclusive disaster planning.

The theory of change that underpins this logframe is not made explicit, but there is logic in increasing the interaction between PWDs and those primarily responsible for current disaster planning in order to better reflect and plan for the additional challenges faced by PWDs, as well as scaling this up through local initiatives and plans to national level through advocacy and examples. Objective 2, though important in developing thinking about (alternative) livelihoods for PWDs to ensure their inclusion, independence and economic survival, appears slightly appended to the project, both on paper and in the way this aspect is discussed by stakeholders that include PWDs themselves. It may, in effect, constitute a separate project, and was not explicitly related to disaster risk reduction by

13

BCO staff or beneficiaries. None of the alternative livelihoods observed in the field or proposed by the GMark Consulting Ltd study commissioned by the project to identify appropriate alternative livelihoods would necessarily be more or less likely to be reconstituted post-disaster; the lasting impact might be enhanced confidence of PWDs to be independent and recover from disasters, we understand relevant groups have initiated discussions around livelihoods options.

This project was intended to start in January 2012, but was delayed through the necessity of obtaining permission to go ahead from the NGO Affairs Bureau; all NGO projects require this permission. One indicator of the innovative aspects of this project, and of its advocacy and awareness-raising effect, is that the Bureau’s seven-month delay was largely due to finding independent experts to advise it on disability and disaster-preparedness.

Assessment of project progress

The project operates in the two southern districts of Khulna and Satkhira, in 12 locations Unions; it has a wide, ambitious geographical coverage, and so activities, processes and structures are of necessity sampled for this evaluation. However, seven specific village locations were visited, there was exposure to three self-helps groups (SHGs) and interviews with individual PWDs who were members of such groups; two joint Union/SHG disaster planning meetings were attended and minutes/participant list of other meetings obtained, plus various documents/reports of training, baseline surveys and other activities. One multi-stakeholder event was attended, at Shyamnagar on 30 January 2013, bringing together local and district politicians and officials responsible for disaster planning with PWDs and staff from DRRA and BCO; the free exchange of views was constructive and impressive. One evening awareness-raising event in a local town was attended, entirely presented to the local community by PWDs, as well as a presentation by GMark of its work on alternative livelihoods for disabled people. In addition, considerable time was given by both DRRA and BCO staff to explaining approaches and discussing challenges and results so far, together with future plans.

Structure

SHGs of PWDs form the bedrock of this project. This is not a new way of working, but DRRA has managed to establish 24 such SHGs within a short space of time. The SHGs grew out of initial survey and baseline work by DRRA staff, putting PWDs in touch with each other. The SHGs attended in the evaluation were dynamic, were largely gender-balanced (though appointed officers tended to be men, something DRRA will address), and were instrumental in reducing isolation of individual PWDs, in building awareness of disaster and other issues, of increasing confidence to lobby for influence at various levels in the disaster planning structures, and to deliver tangible benefits to members.

One SHG Chair, interviewed at his bicycle repair business, was very articulate in explaining what his SHG was doing, what it had achieved, and how its members benefitted in various ways beyond the immediate issue of disaster preparedness. One elderly female SHG member, who was primarily involved on behalf of her grandson with MS for whom she is the primary carer, had obtained free physiotherapy to enable him to walk. In all cases, these SHGs appeared to be active and important as a building block for all the project activities, but also as a means to develop confidence and profile for PWDs.

SHG members also take an active role in advocacy and representation at the Union planning level, where decisions and resources are assigned for disaster preparedness. In one Union meeting it was clear that the PWDs who were present were active speakers and well-briefed. There was clear evidence of change arising from this influence; for

14

example, one cyclone shelter visited in Satkhira district now has ramp access, and one disaster Union plan has clear procedures for informing and evacuating named PWDs in advance of future cyclones and other predicted natural disasters.

DRRA staff are very active in servicing the SHGs and assisting them to link with other structures to further their influencing aims. Discussion with some of them, focusing on the reported issue of high staff turnover, indicates that working with 24 SHGs may not be sustainable, and higher quality of intervention may be more achievable if less SHGs are actively supported and workloads for staff are reduced. Although not actively engaged on this project, DRRA staff does include disabled people. Though still in an early stage, there is no indication that learning is being documented (e.g., why are some SHGs more effective than others, how can gender stereotypes be reduced to enable women PWDs to take on more officer roles?) in any systematic way, and key monitoring processes (e.g., budgets/expenditure) are rudimentary. DRRA does, though, have a track record of good documentation and packaging of case studies, primarily for fundraising purposes.

Process

Bangladesh local and national government has responded positively and effectively to the challenges of the country’s natural disaster proneness, and this openness seems to have extended to being receptive to the coordinated voice of PWDs on this issue. Advocacy has not, therefore, been as difficult as might have been expected, and the project has enabled PWDs to get influence, modify disaster plans, and make tangible improvements for PWDs in disasters and disaster-preparedness. The NGO Affairs Bureau process slowed the project down at the start, but it has recovered.

DRRA has a dynamic, inspirational Founder-Director who is the primary decision-maker. In observed discussions, and in strategic decision-making with her management team, the Director’s views were pre-eminent, though the quality of her understanding and leadership was high. A draft transition and succession plan is being developed to ensure continuity in leadership, though its evolved, but its time frame of five years is long, and enforced loss of leadership (e.g., through illness) would be very challenging for the project, and might entail greater active involvement of the BCO. The DRRA, as an implementing partner, has influence, expertise and experience in working across a whole range of disabilities, though less so on blindness, but its capacity to deliver on the scale required in this project, together with its other funded work, may be a limiting factor. DRRA has moved, largely, beyond the medical model of disability (i.e., disability as a condition to be treated), though, to the rights-based approach (i.e., disabled people have a right to a dignified, respected life as citizens of different ability).

The use of external consultants – for example, the previously mentioned GMark, which has been evaluating which income-generating activities are best suited to PWDs in terms of profit potential and practicality (e.g., crab fattening, an activity which some PWDs were already observed to be doing successfully) – is helpful, and the BCO has been key in encouraging and facilitating this use of external capacity. Another example is the training on human rights, gender and advocacy delivered by the Bangladeshi NGO Ain o Salish Kendra (ASK) for SHG members and DRRA staff in September 2012, building their capacity to undertake key aspects of this project and building on knowledge of disability rights.

How the project records expenditure could be improved, moving from objective-based budgeting (e.g., the cost of a bicycle), to activity-based budgeting, which might then provide a more complete picture (e.g., in DRRA staff time) for activities such as capacity-building and support of SHGs. This would also provide assurance to Sightsavers that

15

activities are happening and at the scale planned. The overall budget for the project is sufficient, and represents very good value for money.

Outcomes/ResultsThe project is delivering on interim results (e.g., 24 SHGs supported, baseline data on numbers and types of PWDs present in the project area, active influence and involvement of PWDs in practical issue such as income generation evidenced by many case studies, observable participation of PWDs in lobbying activities at Union and Upazilla levels, etc). DRRA has assisted PWDs in some cases to receive state disability allowances and other formal indicators of recognition, as confirmed in interviews with two PWDs (e.g., state ID that enables PWDs to receive certain services).

A random sample of the results listed in the Year 1 report were tested in interviews and through examination of documents such as meeting minutes and attendance lists, and there is strong evidence that PWDs are as active as reported, and are gaining access to decision-makers. It is reasonable, therefore, to assess that the project is on track with respect to Objectives 1-3; it is harder, at this early stage, to assess impact at the national level (Objective 4).

There is an absence of any discernible plans at this stage for further replication and scaling up, though the model of inclusive disaster relief planning involving PWDs would have potential in other parts of Bangladesh, and elsewhere in the world. This is an area that needs to be stressed, not least because there is real potential in this work for DFID to be interested in replicating it.

InnovationThe linking of disability and disaster planning and relief, especially in the context of climate change and the growing likelihood of disaster occurrence, is demonstrably innovative and important, and has huge potential for documentation, case studies and replication, both within Sightsavers but also more broadly. DFID, in particular, may be interested in this linking of these key concepts, which may have scope for replication and scaling up within other disaster-prone contexts and regions.

Findings

This project is broadly on track at this point in its implementation, and both DRRA and the BCO are actively involved in monitoring progress and ensuring delivery. Both in the SHGs and in individual cases, there is active participation and real change and benefit to people’s lives. The fundamental aim, that PWDs will be better prepared and involved in planning and disaster relief, is being fulfilled. The policy change aspects of the project are being delivered at the local level; this advocacy and influence needs to be extended at the regional and national levels, with full participation of PWDs, but there is no reason to suggest DRRA in partnership with the BCO will not attend to this aspect of the project. There is documentation of case studies and statistics as part of the monitoring of the project; greater attention needs to be focused on the higher-level lessons learned about how the innovative linking of disasters and disability has worked in practice. DRRA is a skilled and capable implementing partner, and there is a positively critical relationship with the BCO where ideas and practices can be challenged. DRRA itself faces challenges in its capacity, and it needs to strengthen senior management to reduce an over-reliance on its Founder-Director.

16

Project-specific Recommendations (see also general recommendations in Chapter 7)

1. Documentation of innovation, continuing with documentation of individual case studies of PWDs and the benefits they gain through the project, but also focusing on the learning re systems for including PWDs collectively in disaster planning, the set-up and functioning of SHGs, the approach to collective advocacy in the existing disaster planning structures, and other key innovative aspects

2. Improve the systems for financial monitoring and reporting, ensuring that the real cost of activities (e.g., advocacy, support to SHGs) is recorded, so that the financial picture will give a clear indication of any areas being neglected

3. Reduce the number of SHGs supported, concentrating on actively supporting a smaller set of groups that are most successful in their functioning and advocacy, and enabling DRRA staff to focus on quality of results rather than quantity. We understand this is already under consideration.

4. Strengthen DRRA’s leadership, reducing reliance on the Director, and increasing through training and support the senior management tier. This action would be supported by the current Director, and DRRA leadership is currently being supported through the BCO risk matrix to assist performance

Evaluation Criteria Matrix

Criterion Rating Evidence

Structure GREEN DRRA is a strong partner with experience and established outreach. The BCO has strong, regular contact with DRRA re monitoring and support

Process AMBER The linking of disasters and disability is a new one to the Bangladesh administration, as evidenced by the NGO Affairs Bureau’s slowness in sourcing expertise, which will make policy change slower. The scope of the project is large, leading to some unevenness in quality (e.g., functioning of SHGs)

Results GREEN SHGs are functioning, with high participation from PWDs, and there are clear gains for PWDs (e.g., ramps in cyclone shelters, piloting of alternative livelihoods, access to financial assistance)

Innovation GREEN Linking of disaster and disability is new to Bangladesh, and has potential for replication and scaling up

Lesson-learning AMBER Weak systems are in place to document

17

lessons learned; there is awareness amongst BCO staff that this is a weak area

Replicability GREEN High potential for replication in other parts of Bangladesh and in other disaster-prone regions

VfM/efficiency/effectiveness GREEN Relatively small investment giving big returns (e.g., 24 SHGs supported) and tangible change in confidence of PWDs (e.g., active presence in local disaster relief forums)

On track? GREEN YES

18

Chapter 4Zanzibar Comprehensive Eye Care Project (Tanzania; 5 years, budget = £399,772)Introduction The project covers the whole range of activities that Sightsavers currently undertakes in both Unguja and Pemba (the two main Islands which make up Zanzibar). It includes screening, eye care and corrective surgeries, capacity building of the primary health care system (including in Pemba the maternal and child health system), capacity building of secondary and tertiary health facilities and lobbying of government to seek to improve eye health and care. The work is effectively a continuation and scale-up of an existing partnership between Sightsavers and Ministry of Health Zanzibar (MoHZ), to deliver Comprehensive Eye Services in Zanzibar which began in 2006. The project will be supported by the Innovation Fund for another 4 years until 2016, with the possibility of other funding being leveraged to support aspects of it.

The objectives of the project are:

1. To streamline coordination and management of eye care services in Zanzibar by 2016

2. To demonstrate how to build partnership with communities and district health services in delivering eye services in 5 districts (3 in Unguja and 2 in Pemba) of Zanzibar by 2016.

3. To demonstrate a system for early identification and intervention for blinding conditions from child birth through childhood in Pemba by 2016

4. To increase the provision of quality cost effective eye care services to a hard to reach population of approximately 500,000 (Pemba) Island of Zanzibar by 2016

5. To strengthen Monitoring and Evaluation system for capturing performance, learning and information sharing by 2016

Many of the activities began before the start of the Innovation Fund, which has underwritten the whole project as well as funding key elements, and it was expressed by staff3 that the innovative element of this project was a Pemba-specific component working under objective 3 to demonstrate early identification and intervention in childhood. This component focuses upon capacitating primary level maternity workers to provide basic screening and services to new-borns. It also includes research, the plan for which has recently been revised. This research, now called Incidence and Causes of Ophthalmia Neonatorum (ICON), was about to commence when the evaluators visit took place and will look at a specific eye problem, which occurs in neonates called Ophthalmic Neonatrum (ON). This was one of two proposals for Innovation Fund work in Tanzania that were reviewed at inception phase. The Innovation Review Panel’s comments suggested that these two projects be combined in some way. The evaluators found some confusion remained about the extent to which research was a key part of the agreed project activities.

The visit to Tanzania was conducted between 28January and2 February 2013. Visits were made to Unguja and Pemba, as well as a brief visit to the Tanzania country office (TCO) in Dar es Salaam. Three villages in Unguja and two in Pemba were visited. Four focus

3 This was confirmed in Interviews with both CO and Policy and Strategic Programme Support Team staff and with the PC and partners in Zanzibar.

19

group discussions were carried out (two in Unguja and two in Pemba) and seven individual interviews were also carried out. Both interviews and focus group discussions were carried out in KiSwahili. Medical facilities at Muhimbili National Hospital, MnaziMmoja Hospital Unguja, Vision Centre at ChakeChake Hospital, Wete Eye Theatre as well as a number of primary and secondary level facilities were all visited. TCO staff were unable to do much accompaniment on the visits to the Islands, so this review is based largely on what was observed during visits Unguja and to Pemba.

Assessment of project progressStructure

This project is a partnership between the Zanzibar Ministry of Health (MoHZ), who are the implementing partner, Sightsavers and Muhimbili University of Health and Allied Sciences4 (Muhimbili Child Blindness Initiatives).

John Hopkins School of Public Health (JHBSPH) and the Public Health Laboratory in Pemba (PHL-IDC) are also partners in the research being carried out in Pemba. The Sightsavers Project Coordinator is based at the MoHZ working alongside all the focal people from MoHZ in Unguja and Pemba

The project works directly with government, supporting their strategies and plans to improve eye care. Planning is carried out by the Comprehensive Eye Services (CES) Team which includes both government staff and the PC and some activities are funded by MoHZ and others by Sightsavers. Staff from Muhimbili provide additional capacity, for example carrying out surgery (observed in progress at MnaziMmoja hospital) and providing training for researchers and practitioners at various stages of the project.

Focus groups with health workers and an interview with the Director General of Health (MoHZ) confirmed that all of the planned activities in partnership with government were on-going. The Director confirmed that he supported the project and that there had been few challenges or delays from his perspective. Focus groups also confirmed that refresher training had been provided to primary health care workers and that eye screening was being supported. Community involvement was evident during a focus group held in Kitope Village (North B, Unguja) where a Village Health Day had recently been supported. As a result of these screening days (which cover a range of medical conditions) a number of patients with more serious eye problems had been referred to MnaziMmoja in Zanzibar Town for surgery or vision aids. In addition the project was supporting some inclusion activities being undertaken by the Ministry of Education and Vocational Training, Zanzibar (aFGD was held with teachers responsible for inclusive education).

The stated innovation of the work looks at the eye health of Neonates in Pemba (originally called NEST) and commenced in 2012 with training being provided to 100 Maternal Child Health Aids (now known as Reproductive and Child Health workers - RCH) and Traditional Birth Attendants (TBAs). There was evidence that specific treatment for newborns was being procured by the primary health coordinator at the Health Centre visited by the evaluators in Pemba following the training deliver by the project in 2012. This was confirmed during Focus Group Discussions held in Pemba.

Research activities

As well as the Pemba activities which commenced in year one there is also the ICON research which did not go ahead because approval for the research protocol was not obtained. According to CES staff thiswas because the government felt that the proposal to

4The National Teaching and Referral Hospital for Tanzania in Dar es Salaam.20

use Public Health Laboratory field workers (who are linked to John Hopkins University) who were not trained in eye examination was not acceptable. A new research proposal was therefore developed and has now been approved. The research is now (February 2013) about to start, though the new activities do not appear in the logframe or implementation plan for the Innovation Fund project being used by the team in Zanzibar. Internal project documents reviewed in the evaluation appear to show activities were addedat a second stage of the application process.

ICON, as revised, will seek to establish the prevalence and causes of ON by surveying 10,000 newborns over a period of six months. This research element is overseen by a Steering Committee, which shares responsibility for making decisions about the research. Now that permission for the research has been obtained there is no reason that the project cannot carry out this research within the time frame proposed. It is effectively action research because some RCH staff have already been trained to deliver treatment and therefore they are already offering improved diagnosis and treatment for ON and will continue to do so whilst the research into proliferation and causes is on-going.

There is some lack of clarity about the extent to which this research is part of the Innovation Fund project, resulting in some confusionabout decisionson the research and the rationale behind them (e.g., changes to the research protocol, amount of Innovation Fund budget allocated to research).

This year’s plan, budgets and financial forecasts for Tanzania had yet to be finally approved due to Sightsavers internal process, meaning that project implementers in Zanzibar were unclear what funds would be available for 2013 (e.g., how much of the Innovation Fund budget will be spent in Zanzibar and how much will be spent directly by the TCO in Dar es Salaam). There was some lack of clarity on various financial issues (e.g., tracking of budgets, expenditure reports, transfers, what funds are available for project activities in Zanzibar in 2013).Procurement of equipment and the time scales for ensuring it reached Zanzibar and was available for use by the project was an issue.

ProcessData about the numbers of people who are offered screening and health care via the comprehensive eye care project is collected by the MoHZ focal persons by extracting it from the data that health care workers are collecting as part of Zanzibar Health Information Management system. This provides statistics to the project as well. They also report on KPIs. There is a comprehensive logframe for this project (which does not include ICON activities); almost all activities in the logframe are being implemented as planned. Key activities not yet done are the baseline and the M&E framework for this Innovation project as a whole. The PC is awaiting information about overall Sightsavers strategy so that the framework designed can be integrated with it. The CO also requires more support with M&E from the Policy and Strategic Programme Strategic Support Department with this aspect of the project, and we understand a formal request has been made by the TCO.

The linkages between the ICON research and the rest of the work are unclear. This may be an opportunity for Sightsavers to test the integration of health care for newborns in primary care initiatives as part of a wider programme, but at present the structures and

21

processes being employed encourage a piecemeal approach which detaches the research from the rest of the work.

Outcomes/Results

For the most part the activities in the project are, at this early stage (less than a year in to a five year project), proceeding as planned, with some just starting (e.g. National Eye Care Strategy). The following outputs were verified:

Training of Primary Health Care workers in Unguja (8 were interviewed) Participation of Comprehensive Eye Service (CES) staff in Village Health Days (1

village was visited) Provision of spectacles (2 vision centres visited and a number of clients observed) A surgical camp was observed (2 operations and 10 consultations were observed.

47 surgical operations were carried out during the week we visited Unguja (patients were provided with transport to travel from Pemba)

MoHZ focal person and Director Generalconfirmed that the Ministry have allocated 16m shillings. Their funds are currently being utilised as funds have not been received from Dar since September 2012

Health workers confirmed that eye care data is being collected at PHUs and that it is being integrated in to the Health Information Management System

The new operating table at Wete hospital was observed 1 school in Pemba where screening was carried out was visited (a school in

Zanzibar could not be visited because an early morning flight was not obtained) FGD with Health workers in Pemba confirmed that Reproductive and Child Health

staff (RCH) had received training on Basic Child Eye Management Essential eye examination tools were observed in five locations Work in villages, at all level of the health care system and in schools seemed to be

progressing positively though it is too early to evaluate the results of this fully

Innovation

The research element sits within a more general programme of work which is all underwritten by the Innovation Fund, which challenges the implementers in terms of being able to demonstrate innovative impact. More thinking about the work with newborns and treating ON could be used to demonstrate integration to other programmes of work with which Sightsavers and other agencies are involved.

Project-specific Recommendations (see also general recommendations in Chapter 7)

1. Greater clarity on financial issues should be establishedwithin the CO as part of improving overall communication (e.g., how much of the budget is allocated to work in Zanzibar and Dar es Salaam respectively) and funds transferred accordingly

2. The baseline should be conducted as a matter of urgency and an M&E framework developed as a shared process between staff in the mainland and those in Zanzibar. Part of this process should be to develop a plan for learning and Sightsavers UK should offer support with this, particularly in terms of outlining their specific expectations of the Innovation Fund.

22

3. Detailed support should be provided to TCO to clarify and assist with the commitments around learning developed from this project. This support should include identifying the key innovations, a plan for collecting and disseminating learning, and a clear understanding of the coherence of the research and project work

4. Project Co-ordinator should be given greater clarity about the research elements of the project and possibly participate in the Steering Committee to give greater continuity to the project.

5. Consider how this work with newborns, treating ON, could be used to demonstrate integration in to other programmes which Sightsavers and other agencies are involved in)

Rating Evidence

Structure RED Information flows (e.g., financial) between different

elements of the project need to improve

Process AMBER Baseline and M&E framework not yet completed

Results GREEN The bulk of project activities are progressing as

planned

Innovation AMBER Innovation is around including basic eye care in

community-level maternal care. This element forms a

small part of the overall project and conceptual

linkages could be improved.

Lesson-learning RED Lack of M&E framework and baseline limits learning

potential

Replication/Scale-Up

AMBER No clear plan to learn although project report points

out that (for example) the approach under outcome 4

to working with RCH workers could be extended to

ZNZ too.

VFM /Efficiency/ Effectiveness

AMBER Whilst the funds spent to date seem to deliver good

value for money there is a need to streamline

budgeting and financial transfers to ensure that the

23

project stays in track.

On track? GREEN

Chapter 5

Community-based Diabetes Prevention and Control (Pakistan: 3 years, cost = £95,200)Introduction

Pakistan is at number seven in the list of countries with the highest incidence of diabetes; the diabetic population in Pakistan is over 7 million, and is projected to reach 11.5 million by 2025. The National Blindness Survey5 estimated the prevalence of blindness at 1% across Pakistan, with 1.5 million blind people. Diabetic retinopathy (DR) is a major cause of blindness in around 4% of patients suffering from type-I diabetes, and1.6% of those suffering from type-II diabetes. Studies carried out by the Diabetic Association of Pakistan show 10% of the general population aged 30 years or above has diabetes type-II. Undiagnosed diabetes and related retinopathy due to virtually non-existent primary eye care is the issue this project is addressing, seeking to change treatment-seeking behaviour in a low-income population in areas close to Karachi.

The project is managed by local partners, with Al Ibrahim Eye Hospital (AIEH) in a lead role supported by the government-sponsored Comprehensive Eye Care (CEC) Cell Sindh, with further support from the Sightsavers Pakistan Country Office (PCO), and its logframe outlines the following aims:

Overall Objective: Reduce burden of preventable vision impairment by early screening and treatment of diabetes related blindness (DRB)

Objective 1: Early screening to improve care of people with diabetic eye diseases

Objective 2: Establish state of the art diabetic retinopathy unit at AIEH Objective 3: Create awareness of the issue, especially women and health

professionals Objective 4: Train local health professional, especially lady health workers

(LHWs), in referral and awareness raising Objective 5: Functioning data management system to record and monitor

interventions and follow-ups

This project represents a specific strategy, combining medical and community interventions, to address a very specific issue of DRB. The theory of change, establishing a hierarchy of referral and treatment designed to reduce DRB, is logical and achievable, though there are important local factors, notably the presence of the AIEH and a sister institution in Nowshero Feroze, and the driving vision of Dr Muhammad Saleh Memon as

5Prevalence of Blindness and Visual Impairment in Pakistan: The Pakistan National Blindness and Visual Impairment Survey, Mohammad Z. Jadoon et al, Invest. Ophthalmology. Vis. Sci.,November 2006, vol. 47 no. 11 4749-4755

24

Director of AIEH and key figure in the project, that are fundamental to the potential success of the project, and may make replication elsewhere difficult.

Assessment of project progress:Structure

Fundamental to the functioning of this project, which is trying to change treatment-seeking behaviour, is the presence and competence of the first point of referral, which is the LHWs. The LHWs are an existing network set up by a previous government administration to provide family planning advice and other basic health information; their allowances are paid by the state. Utilising this existing network is practical and effective. The claims on the total number of LHWs trained so far in the project on DRB (c 1700) can be verified through records, and two meetings with LHWs did take place, involving 130 LHWs in all, and the women present showed themselves to be articulate and motivated on this issue. There have been successful efforts made through the project to increase their pay and status, and one impact has been an enhanced image of women and the role they play in the community.

Also key is the structuring of levels of awareness-raising, referral and treatment; this needs to be accessible, logical, affordable and initially based in the community. The lowest level of awareness-raising is the village health committees (VHCs), of which 12 are claimed to have been set up through this project. There is lack of documentary evidence available for functioning VHCs (e.g., minutes), but the evaluators participated in one VHC which had an agenda and did focus on the diabetes issue and how to tackle it. The VHCs form a meeting point between the LHWs and other community representatives, and provide discussion points for how to improve referral for treatment in the community, though it is unclear how vital a role they place overall in the project. Referral from the LHWs go to community-based clinics, where CEC has been instrumental in obtaining government funds to refurbish these centres; two centres were visited, of five within the Karachi project area, which had basic diabetes screening equipment and were adequately equipped with some new medical kit as part of the project. One was also staffed by a qualified doctor who gave one day per week to the centre, and who had attended one of the project briefings cited in the Year 1 report. There were basic records, though it was hard to get a clear picture of how many patients had been screened after referral form the LHWs (3,198 are claimed across the centres). Record keeping of patients screened at the AIEH was more advanced, recording data such as age and gender, and using computerised systems.

The AIEH and its sister institution in the neighbouring Nowshero Feroze area are key to the intervention, and to the overall aim of reducing DRB; they provide the nub of treatment, and are symbolic of a functioning system that gives confidence to users. AIEH was visited, and the facility was well-equipped and staffed, with new equipment available. Individual staff were interviewed, including female staff, and understood how the screening was working and the contribution to diabetes identification, though the issue of DRB was not central to discussions, and other blindness causes (e.g., cataracts) were understood to be central to good eye health in the community.

The interaction between different actors in the project at the advocacy level was demonstrated by a project meeting and research presentation at the AIEH, with representatives from local government and health departments, the CEC, the Diabetic Association of Pakistan and the PCO. The Director of the AIEH is a key figure in the project, and he expressed strong views about both sustainability and replicability of the project, as well as the level of sustainable cooperation amongst the different actors (state, private sector, non-governmental); others were more positive about progress being made.

25

The Director of the AIEH stressed the use of key performance indicators (KPIs) and “surprise visits” in monitoring performance of the screening and treatment at the different levels; at present the KPIs are still in the process of being set through baseline surveys that are being carried out, as evidenced by the community statistics on diabetes prevalence and treatment figures presented at the stakeholders meeting at AIEH.

ProcessThe project confronts a common problem in changing treatment seeking behaviour of low-income populations; people would prefer not to know if they are prone (as in this case) to diabetes, and to DRB. The project, by utilising the existing network of LHWs to raise awareness and provide first point of referral, is well-thought out and executed in its approach; but the economic consequences of diabetes, and the probable costs of transport, drugs and other components mean that some low-income families would prefer not to know the diagnosis (e.g., ongoing cost of insulin). A special counter, observed as part of the evaluation, does act as a no-cost point for referred at-risk diabetes community members to be seen and given the detailed screening to confirm or otherwise diabetes and possible retinopathy.

The project, from logframe to implementation, displays some uncertainty about whether it is about screening for diabetes in general, or about DRB and screening for retinopathy in particular. This lack of clarity impedes the work of the project, including setting and monitoring targets, and the treatment cycle and network. In practice, the approach from LHWs upwards, seems to be more about detecting diabetes and encouraging treatment-seeking behaviour for that issue, with a small subset of those identified as diabetic being at risk of DRB. Assistance in making this clarification would be useful.

Technically, the project is well resourced to service those who come forward to testing and treatment. The counselling aspects of support, and the costs associated with diabetes, seem less considered. People with diabetes were largely or wholly absent from discussions about the project and its aims and outcomes, and there is a sense of experts designing an intervention strategy for recipients. The developmental aspects of this project, as opposed to its medical components, are not clear; there is no clear linking to economic development, to stigma, or to other social components (e.g., the impact on income and livelihoods of DRB, the inclusion of blind people within the community).

Outcomes/ResultsThe infrastructure for screening and testing has been established within the project areas. Two of the satellite centres in Gadap Town were viewed, of the five claimed by the project, and these had new equipment and seemed accessible for physically disabled and elderly people (e.g., ramps). Detailed figures, compiled using bespoke software at AIEH, gave numbers of community members referred and screened at different stages (e.g., referrals by LHVs, screening at satellite clinics, referred patients received at AIEH), though this data was hard to verify (e.g., 3198 patients screened for diabetes at the satellite centres, with gender-disaggregated data collated; 1674 LHWs trained, of which each is calculated to service 1000 community members; 12 VHCs formed; 563 at risk diabetes cases referred to AIEH, of which 29 presented themselves at AIEH). The challenge of the project is to change behaviour, and there is an evident drop off of referrals to those who actually seek treatment at AIEH, indicating other factors (reluctance to know reality re diabetes, cost of transport, fear of medical system, loss of earnings in time spent, etc). There is scope for research and modification of process, and the project team meeting at AIEH in which the evaluator participated did include robust discussion of results and what they meant.

26

The documentation provided to the evaluators on the projects omits any targets for screening and treatment, merely that this would be increased; and baseline data on real incidence of DRB in the project areas, and on how far this would be reduced, was lacking. It would be helpful to know more about what percentage of the project population is actually receiving screening and treatment against targets; though the actual numbers are large and seemingly significant. An important issue also is what levels of referral would have happened without the project intervention; there is no significant control data for comparison

InnovationDiabetes is undoubtedly a major health issue in Pakistan and, by association, DRB; though, as stated in the project documents, cataracts is a more major contributor to blindness, and the AIEH is playing a significant role in this area with surgery and treatment. As a methodology the levels of screening and referral, built up from the existing network of LHWs, is logical and appears to be working at a basic level. The absence of those with DRB, as advocates or referrers, is one example of how innovation could be improved. The project also appears to be unclear at some points in whether it is focusing on DRB or on diabetes in general. In terms of the learning and potential for scale-up, this lack of clear focus could be important, though Sightsavers itself that this is a community based diabetic retinopathy project aimed at raising community level awareness about diabetes in order to avoid progression to retinopathy. This particular issue, of DRB, may have obtained the level of importance to merit Innovation Funding due to Sightsavers’s mission, though there are other key developmental issues in this area. The drive of the Director of AIEH to do this work is clear and impressive; but it is less clear what the overall benefits are to the community are, which raises issues about value for money.

Findings This project has established a functioning network to encourage and respond to a change in behaviour for those seeking treatment for diabetes. It has also enabled the purchase and installation of new equipment to deal with the issues arising from the detection and treatment of diabetes and DRB. The project remains unclear about whether its main focus is diabetes in general, or more specifically DRB. If the latter, there is a question mark about how important is the issue in comparative terms, against other challenges faced by the population, and how they themselves prioritise it.

Project-specific Recommendations (see also general recommendations in Chapter 7)

Clarify the logframe and aims of the project: is it about diabetes or DRB? Project activities will need to be revised depending on the answer. Logframe re-examination will also make clearer what are the developmental aspects of this project, as opposed to the medical

Involvement of people with diabetes needs to be improved and expanded to make it more relevant, responsive and grounded in their needs. This would make the project feel less top-down

Evaluation Criteria Matrix

Criterion Rating Evidence27

Structure GREEN The inter-linking of LHWs, CEC satellite clinics and the central hospitals provides a logical implementation structure

Process AMBER The project is adequately resourced and has the necessary technical capacities. It needs greater clarity on its aims

Results AMBER Medical interventions have been recorded, though inadequate baselines limit assessment of rates of interventions accorded to the project. Awareness raising has been limited

Innovation AMBER The use of the LHWs, adding to their responsibilities and giving them extra status, has been positive; and the partnership between public, private and NGO partners has worked well so far

Lesson-learning RED There are no specific plans for lesson-learning at this stage, and no real lessons from the work have yet emerged, though the use of IEC materials for awareness raising has been effective

Replicability AMBER LHW networks exist elsewhere, but the presence of the hospital facilities at the centre of this project, and the vision, energy and influence of the project Director are unique aspects. Awareness at community level, referral from community to secondary and to tertiary care, are also replicable and sustainable

VfM/efficiency/effectiveness

AMBER Relatively low cost intervention, but limited overall community benefit, at least if DRB is indeed the main focus

On track? AMBER

28

Chapter 6Ensuring Inclusion through Accessible Water and Sanitation (Pakistan: 3 years, cost = £600,350)Introduction

Improving access for the poorest people to accessible water and sanitation (WatSan) is a central target in the Millennium Development Goals (MDGs), and is seen as key to unlocking many other economic and social development objectives. Disabled people (People with Disabilities, PWDs) are often further excluded from community because their basic needs, such as using the toilet or washing, are not understood and addressed even where poor urban and rural communities do get improved WatSan.

The project is managed by local partner Human Resources Development Society (HRDS), which has worked before with the PCO on a pilot phase of this project and has extensive experience in developing WatSan infrastructure in Pakistan, though without the focus on disability. Its logframe outlines the following aims:

Overall Objective: Reduce social exclusion of PWDs with community-managed accessible and affordable WatSan services in three districts

Objective 1: Engage with relevant service providers and stakeholders at local levels

Objective 2: Develop a strong evidence base of effective WatSan solutions at community/local/national levels

Objective 3: Demonstrate accessible WatSan solutions at communal locations

The theory of change implicit in the project’s logframe uses the very practical issue of basic WatSan as an entry point for raising awareness within poor communities about disability issues, and enabling PWDs to become more visible, participative and confident. It has also helped to raise issues relating to other marginalised groups such as the elderly and women for whom inaccessible WatSan creates significant problems relating to dignity and quality of life

Visits to one of the project sites were restricted by security and extreme weather, so assessment of the project in Chitral and Mansehra is limited to discussion and documentation. In Shahdadkot a number of sites were observed, and meetings at various levels were attended, including community mobilisation meetings in three village locations. The clearest expression of success was the presence of PWDs themselves, in the committees and meetings, expressing their views. For example, a blind elderly man with visual impairments, without apparent awkwardness, demonstrated how he can now go to the toilet facility unaided. PWDs talked about how initial designs for toilets and washing facilities had been modified based on their particular needs (for example, the provision of both squat toilets and “English”-style seated facilities in the same structure; and the provision of lever-type handles for taps). In interviews, it was clear that communities had contributed materials and labour as part of their commitment to obtaining facilities, as explicit part of the HRDS strategy to encourage ownership and awareness of the wider issues of disability. Some visiting members of neighbouring communities not serviced directly by HRDS had developed and built facilities themselves without support.

Assessment of project progress

29

Structure

Both HRDS and PCO staff make regular monitoring visits to WatSan project sites, as evidenced by travel logs and diaries, and also by the strong individual rapport between some community members and project representatives. HRDS and the PCO are monitoring explicitly a range of indicators across the three output areas; for example, a 46-slide PowerPoint presentation of the current status of the project, including field data from Chitral, Mansehra and Shadadkot, was made available to the evaluator, backed up with triangulated evidence (e.g., photos, research outlines, meeting minutes). As an example, there is detailed evidence of how and why latrines were modified in particular cultural circumstances based on surveyed views of PWDs. The powerpoint, incorporating excel spreadsheets, also records progress against a range of detailed output targets, including less tangible activities such as group formation and functioning. Detailed monitoring appears to have been instilled as a fundamental process integral to showing success and highlighting problem areas so that solutions can be found.

The impact of doing this work on HRDS has been to raise the awareness amongst its staff about both the policy and practical issues faced by PWDs in using WatSan facilities, and more generally (e.g., access to HRDS’s new office building was made more accessible). HRDS staff, in discussions and interviews, genuinely seem to have internalised disability issues, increasing the likelihood that HRDS will continue to build inclusive WatSan infrastructure beyond the life of the project.

The PCO staff have also absorbed the lessons from this project, and discuss disability and related issues with knowledge and sensitivity (e.g., staff stated that this project, in particular, had made an impression on them). The PCO also has one disabled member of staff, a visually impaired woman, who is fully included in all activities, and for whom adjustments have been made (e.g., exploration of voice-operated software), providing evidence of a real commitment to disability. The PCO has linked up with other key organisations to form the alliances which will be the basis for advocacy alongside the practical demonstration of WatSan facilities within the communities. One such organisation, which was visited as part of the evaluation, is STEP, a disabled people’s organisation (DPO) entirely staffed by PWDs with a variety of impairments (physical, learning, visual, auditory). STEP has successfully lobbied for a human rights desk at the Ministry of Justice which will take responsibility for disability issues; the PCO, and through them HRDS, is linking with effective strategic partners such as STEP to increase the potential effectiveness of policy change at government level. The project is also supporting the formation of DPOs in Mansehra and Shadadkot, as evidenced with detailed minutes, agendas and full lists of participants. This is all part of ensuring the project remains shaped by disabled people, though there is an issue of how much the process of forming DPOs can be driven by non-disabled people, as it is at the moment, as well as the fact that this process does not feature in the logframe.

The project is also using the fact of Pakistan’s signatory status to the UN Convention on the Rights of People with Disabilities (UNCRPD) as the cornerstone for its advocacy, taking advantage of an established foothold to demand access and time of key government officials and agencies, demonstrating tactical awareness.

ProcessThe security and geographical challenges for this project in working in certain rural areas of Pakistan can provide real barriers, particularly for monitoring work and ensuring deliverables at key dates. At the time of the evaluation visit, the Chitral region was entirely inaccessible due to adverse weather, and the political situation in the country, which ebbs and flows, was heightened by bomb attacks, anti-corruption demonstrations and the fatal

30

attacks on Pakistani female anti-polio health workers, restricting travel and leading to general feelings of insecurity, not least for NGO workers. The PCO staff and their counterparts at HRDS take a pragmatic approach to their work in these circumstances; for example, by constantly adjusting the travel and meetings schedule to provide as much access as possible to project sites and beneficiaries as possible for the evaluator.

HRDS appears to have the resources, track record and technical competence to build the necessary inclusive WatSan facilities that are the focus of this project, and to work successfully on obtaining community ownership. The PCO has played and continues to play a key role in ensuring disability issues are central to the physical construction; both in a seminar at the HRDS central office, and in the field, professional discussions and exchanges were observed which were positive and knowledgeable. HRDS is also outsourcing areas where it has less capacity or understanding, such as action research and baseline establishment. The evaluator participated in a discussion, including with senior researchers by Skype, where Comsats University presented initial findings and outlines for further work. The latter will establish how many disabled people live in the project areas, what range of disabilities ispresent, what is the position of disabled people in the community, what are the key economic activities of the community, and what are the attitudes to WatSan. Answers to these questions will enable HRDS to better design and target its interventions.

Outcomes/Results

HRDS, in partnership with the PCO, maintains a clear, professional vision of what this project is trying to achieve; staff at different levels, along with Comsats researchers and other external and outsourced resources, hold a clear picture of the project logframe structure, indicating the inherent clarity of the theory of change and also the efforts made to ensure a coherence and consistency amongst those implementing the work. The targets under Objectives 1 and 3 are already being met, and monitoring is regular and vigorous, with the PCO playing a key role in pushing for results. The basic groundwork, involving the construction of inclusive WatSan and the formation of local institutions and structures that provide stewardship of these resources, are being met (for example, 15 of the projected latrines and accessible water points have been built; 10 DPOs have been established, though it is not clear exactly what process has been used, whether this is led by PWDs themselves, and what relationship the DPOs have to the project). With this early track record of achievement, it is likely that all the project outcomes will be met on time and budget; though making change happen for PWDs at the national policy level will be challenging, and will require greater emphasis on advocacy and documentation of lessons learned. Objective 2, especially at the national level, will be more challenging, and much will depend on systematic documentation, research and advocacy; the project is building a good base in these areas.

Replication of the basic model will depend on resources, and on documentation. Building inclusive WatSan facilities and developing the community ownership and institutions as an entry point to a more inclusive attitudes and practices in general towards PWDs and other marginalised groups, with the underlying theme of positive social change, appears to be working well, and this project is already an example of scaling up from an original pilot.

InnovationIncluding PWDs in mainstream Pakistani society by focusing on a key practical issue of WatSan is innovative within the Pakistan context, though inclusive WatSan is not in itself a new concept. The linking up with existing DPOs, and the aim of forming new groups, is also innovative in ensuring the project remains owned and influenced by PWDs though, as has been noted earlier, the process of forming the DPOs does not appear to have been

31

led by PWDs. As has already been stated, the project follows on from a successful pilot, so its process is not necessarily innovative; but the scale of the project is new and challenging. In principle, the methodology of the project could be applied in other contexts outside Pakistan, as well as being extended within Pakistan.

Findings

This project is, in effect, the scaling up of an earlier pilot collaboration between HRDS and the PCO; for that reason, its status as an Innovation Fund project must be questioned (i.e., it is working with an existing partner). However, its very scale and impact can be argued to single it out as innovative, as well as its implementation in some very remote and physically and culturally challenging situations. It is making a real difference in the lives of the PWDs interviewed as part of this project, and the facilities built by HRDS provide a symbolic and physical focus for discussions about the inclusion of PWDs and others whose lives are made difficult by lack of access to basic washing and toilet facilities. PWDs are very much central to the project, and appear to influence design and siting of WatSan, as well as beneficiaries. HRDS is an efficient and effective partner, and its collaboration with other actors such as STEP and Comsats University on lobbying and action research provides a solid basis for advocacy success in influencing regional and national policy on disability issues.

Project-specific Recommendations (see also General Recommendations in Chapter 7)

1. Forming DPOs: this should be clarified in terms of who is leading it, and to what purpose within the project

2. Target schools for WatSan: in discussion amongst project partners, the idea of changing focus from building inclusive WatSan as general community resources to targeting schools as a way of improving inclusive education for disabled children and girls in general has been put forward. This should be seriously considered

Evaluation Criteria Matrix

Criterion Rating Evidence

Structure GREEN HRDS is an accomplished partner with the technical and human resources to lead the project, as well as the openness to learning (e.g., WatSan constructed, designs modified based on needs of PWDs)

Process GREEN Security and remoteness of some locations is an issue in implementation, meaning project sites are inaccessible at certain times and project processes are disrupted on the ground, but partner HRDS has experience in such zones

Results GREEN Accessible WatSan has been built to schedule and budget, various training has been completed, baseline and action research has been

32

commissioned and initial findings presented, lobbying contacts being developed

Innovation GREEN Accessible WatSan is not new, but its application in Pakistan, and its use to improve quality of life for PWDs but also as a means to instigate greater integration with the community is innovative

Lesson-learning AMBER HRDS has a good track record on documenting its work, and the PCO has already demonstrated its interest in drawing out lessons from this project (e.g., in its powerpoint presentation) in terms of future project implementation as well as overall process and substance of this project

Replicability GREEN The model used for accessible WatSan could be used in other contexts with little necessary cultural adaptation

VfM/efficiency/effectiveness

AMBER Project budget exceeds £600k, and local community contributes in materials, labour and upkeep

On track? GREEN

33

Chapter 7Findings and Overall Recommendations Introduction

We find that Sightsavers should be congratulated for conceiving and setting up its Innovation Fund to try out new ideas, encourage experimentation and the risk of failure, find new ways of achieving change for blind and disabled people, and sharing lessons with others so that learning, replication and scaling up of successful interventions is encouraged. The five projects approved in Round 1 of the Innovation Fund, in Tanzania, Bangladesh, Nigeria (excluded from this evaluation) and Pakistan (2), vary in aims, budget, timescale and subject matter and quality; some are delivering excellent work that is impacting positively on blind and disabled people.

We find the initial execution of the Fund, in terms of guidelines, UK-based systems for management of learning, process for choosing projects, and aims of projects initially chosen, could have been better, and adjustments must be made both to individual projects and to centralised systems to extract the maximum benefits for Sightsavers. Although outside our TOR, we have found the initial lack of clarity and communication around the Fund and its objectives has placed in doubt the success of these projects in terms of delivering learning in particular; which is why we have commented on it. We find that the absence of coherence across the small group of initial projects chosen is symptomatic of a lack of clarity about what the Fund overall is trying to achieve on innovation. This has made calculation and comparison of value for money difficult, for example. Fortunately we feel there is time to make the adjustments needed within the timescales imposed by the Round 1 projects and the wider PPA, and we understand that Round 2 of the Innovation Fund has already set right many of the issues we encountered.

This evaluation is timed, in theory, just over one year into the lifecycle of the Innovation Fund Round 1 projects. In practice, projects started at January 2012 at the earliest, making assessment of overall progress at such an early stage difficult.

Each project has its own strength and challenges, and needs to be supported in different ways to achieve its objectives. Recommendations for each project are repeated below. In addition, we offer some overall recommendations about the Innovation which may assist it to deliver on its potential

Project-specific recommendations1. Bangladesh: Disability Inclusive Disaster Preparedness and Management

focusing Alternative Livelihooda. Documentation of innovation, continuing with documentation of individual

case studies of PWDs and the benefits they gain through the project, but also focusing on the learning re systems for including PWDs collectively in disaster planning, the set-up and functioning of SHGs, the approach to collective advocacy in the existing disaster planning structures, and other key innovative aspects

b. Improve the systems for financial monitoring and reporting, ensuring that the real cost of activities (e.g., advocacy, support to SHGs) is recorded, so that the financial picture will give a clear indication of any areas being neglected

c. Reduce the number of SHGs supported, concentrating on actively supporting a smaller set of groups that are most successful in their functioning and

34

advocacy, and enabling DRRA staff to focus on quality of results rather than quantity. We understand this is already under consideration.

d. Strengthen DRRA’s leadership, reducing reliance on the Director, and increasing through training and support the senior management tier. This action would be supported by the current Director, and DRRA leadership is currently being supported through the BCO risk matrix to assist performance

2. Tanzania: Zanzibar Comprehensive Eye Health Projecta. Greater clarity on financial issues should be established within the CO, as

part of improving overall communication (e.g., how much of the budget is allocated to work in Zanzibar and Dar es Salaam respectively) and funds transferred accordingly

b. The baseline should be conducted as a matter of urgency and an M&E framework developed as a shared process between staff in the mainland and those in Zanzibar. Part of this process should be to develop a plan for learning and Sightsavers UK should offer support with this, particularly in terms of outlining their specific expectations of the Innovation Fund.

c. Detailed support should be provided to TCO to clarify and assist with the commitments around learning developed from this project. This support should include identifying the key innovations, a plan for collecting and disseminating learning, and a clear understanding of the coherence of the research and project work

d. Project Co-ordinator should be given greater clarity about the research elements of the project and possibly participate in the Steering Committee to give greater continuity to the project.

e.f. Consider how this work with newborns, treating ON, could be used to

demonstrate integration in to other programmes which Sightsavers and other agencies are involved in)

3. Pakistan: Community-based Diabetes Prevention and Controla. Clarify the logframe and aims of the project: is it about diabetes or DRB?

Project activities will need to be revised depending on the answer. Logframe re-examination will also make clearer what are the developmental aspects of this project, as opposed to the medical

b. Involvement of people with diabetes needs to be improved and expanded to make it more relevant, responsive and grounded in their needs. This would make the project feel less top-down

4. Pakistan: Ensuring social inclusion through accessible water and sanitationa. Forming DPOs: this should be clarified in terms of who is leading it, and to what

purpose within the projectb. Target schools for WatSan: in discussion amongst project partners, the idea of

changing focus from building inclusive WatSan as general community resources to targeting schools as a way of improving inclusive education for disabled children and girls in general has been put forward. This should be seriously considered

35

Overall recommendationsDegree of innovation

1. Make more explicit the definition of “innovation” used for the Innovation Fund, encouraging experimentation and the “risk of failure” as per DFID guidelines, and encouraging CO staff and partners to really think about ways of accelerating the changes they and their beneficiaries want

Learning and communication of lessons generated from the Innovation Fund

2. Develop and communicate clear guidelines on what is expected from CO and partners in this area, including consulting on the draft guidelines

3. Offer additional support and funding to CO for this area4. Put in place clear processes and procedures in the UK for maximum capturing

and exploitation of any learning generated5. Designate one single, senior SMT person to take an overview of the Innovation

Fund and support those staff working on all aspects of Innovation Fund projects in the UK and in COs

6.

Process of selection of the Innovation Fund projects, including acting on comments from the external Innovation Fund panel

7. Summarise simply and in writing the conditions and commitments attached to gaining funds from the Innovation Fund and communicate these to COs, so that they can discuss with implementing partners how these commitments will be met, including making any requests to the UK for additional funds or technical input (e.g., on learning)

8. Clarify what is the final, approved version of an Innovation Fund proposal, which should include amendments as a result of comments from the Innovation Fund panel. All proposals should have clear and final sign-off.

Maximising the benefits of the Innovation Fund

9. Actively promote risk as an integral part of the Innovation Fund, which would involve senior management support and leadership to give people confidence to take risks

10.Use the Innovation Fund primarily to fund focused, short-term pilot projects that explicitly set out to test new and innovative ways of working, with the aim of learning lessons and applying them in scaling up and replication.

36

Chapter 8

References and Appendices

a. Evaluation Terms of Referenceb. Innovation Fund Guidelinesc. Topic Guide (initial UK interviews)d. Topic Guide (Country Visits)e. People consultedf. Inception Report

37

Appendix a: Evaluation Terms of Reference

SightsaversInnovation Fund Round 1 Independent Progress Review

1. Background

Sightsavers is an international development agency and its overall goal is to contribute to the achievement of the MDGs by eliminating avoidable blindness, and promoting equality of opportunity for disabled people. Currently, Sightsavers is supporting projects in 34 countries across Africa, Asia and the Caribbean.

In 2011 Sightsavers received 3 year funding from Department for International Development (DFID) under its Programme Partnership Arrangement (PPA) funding stream which runs from April 01, 2011 to March 31, 2014. Innovation is one of our core principles and PPA funding has been used specifically to establish a new initiative in the form of the Sightsavers Innovation Fund. This was launched internally in 2011 with £789,364 allocated to projects which can demonstrate and provide evidence of impact and replication/scalability, providing a contribution to global learning at the organisational level at Sightsavers, in most cases with the requirement to work in partnership with a research institute.

In this first round, through a rigorous selection process, 5 projects demonstrating new and creative initiatives were chosen for funding. These projects include innovations building on health and education sector reforms, adopting cross-sectoral approaches to fill current gaps and duplication in services, using existing infrastructure in new ways, cross-sector NGO collaboration on eye health and sanitation, amongst others.

The Innovation Fund also required that projects clearly demonstrated and provided evidence of not only why the project is innovative but also how success and shortcomings will be documented and disseminated, along with its impact and a clear proposed plan for replicating and scaling-up the successes documented during implementation.

The following projects five were funded under Round 1:

i. Bangladesh: a project examining the effects of climate change on the lives of people with disabilities, and how to ensure that disaster preparedness activities are responsive to their particular needs

i. Zanzibar: a project to take advantage of the health and education sector reforms by adopting a cross-sectoral approach to gap filling, including the integration of child eye health into maternal and child health.

ii. Pakistan: a project to promote social inclusion through community-managed, accessible and affordable water and sanitation in three districts of Pakistan.

iii. Pakistan: a project focusing on community based diabetes prevention and control iv. Nigeria: research on the impact of integrating primary eye care and primary ear

care into work practices of Patent Medicine Vendors.

38

In the first year of the Round 1 projects it has been reported that significant progress has been made in one of the above projects while in the others, a number of concerns have been expressed as progress has been slower than anticipated. This independent review is being commissioned to validate these reports and concerns and to draw out recommendations on how to ensure that funded projects deliver on the expected results.

2. Scope of work

This review is expected to provide an independent assessment of the Round 1 Innovation Projects inNigeria (1), Zanzibar (1), Pakistan (2) and Bangladesh (1), looking at their progress so far, the technical needs of the projects and providing a realistic assessment on what project outcomes and applications of learning might be achievable within the PPA period.

2.1 Overall PurposeThe overall purpose of the review is to provide an independent assessment of the Round 1 Innovation Projects in Nigeria, Zanzibar and Bangladesh and Pakistan.

The review will be guided by the following questions:

2.1.1 Structure a. What are the established structures for the implementation and coordination of

the projects, both within Sightsavers and at the partner level? a.1 How can/should these structures be strengthened to ensure effective and

efficient project implementation?a. What systems in place for documentation and dissemination of success/failure?b. What monitoring mechanisms are in place to ensure the projects remain on

course to achieving their outcomes and how can/should these be strengthened?

2.1.2 Processc. What factors have promoted or impeded the implementation of the activities and

the achievement of the outcomes/objectives? d. What new technical directions should be adopted so the projects maximize their

impact? e. What additional technical support (if any) is required and what forms should this

support take? f. What project revisions need to be considered, if any? (targets, strategies,

activities, budget etc.)g. How is risk managed and mitigated in these projects both within Sightsavers

and at the partner level?g.1 Do projects regularly monitor and manage their risks and what specific

actions are being taken to do so.

2.1.3 Outcomes/Resultsa) To what extent are the projects on course to achieving their anticipated outcomes

and objectives? b) Is the proposed plan for replication and scale up realistic?

39

3. Review Team

The evaluation shall be conducted by an external evaluation team or consultant, selected through competitive proposal submission process. The team or consultant should be suitably-qualified and experienced individual/s, who can provide the following competencies and experience:

• Experience in measuring innovation, including a detailed understanding results-based monitoring and evaluation for international development project/programmes, with strong skills in both qualitative and quantitative approaches

• Thematic expertise/comprehensive understanding of policy issues at global and local level in health, education and/or social inclusion sectors

• Experience of carrying out large scale evaluations and or reviews• Up to date knowledge of trends in innovation in international development. • Ability to produce concise, readable and analytical reports • Excellent communication skills in written and spoken English (and in French would

be desirable).The consultant/team must be independent of involvement in any Sightsavers’ partner programmes so that they can assure impartiality and present no risk of a conflict of interests.

4. Methodology

The following key steps are envisaged for this assessment;i. Develop a concise scope of work to undertake a review that meets the overall

purpose of this Terms of Reference.ii. Secondary review of relevant documents including Innovation Fund proposals

and reports.iii. Production of an inception report clearly articulating the review design,

methodology, tools, work plan schedule and budget to carry out the assignment in each country.

iv. Semi-structured interviews with key stakeholders in each country.v. Observation of project sites. vi. Production of draft and final report.

5. Reference Materials

The following materials will be made available to the consultant: Innovation Fund Round 1 Guidelines, Concept notes, Proposals, Budgets and internal reports (financial and narrative).

6. Timeframes

The consultancy will run from December, 2012 to March, 2013. The following proposed plan will be used to guide the review.

40

Phase 1: Initial Review: Review of documentation and inception report (December, 2012)This phase will involve initial review of documents, and the presentation of an inception report. The inception report should include an elaborate plan, methodology and sampling strategy of the data collection for evaluation study. The review will only proceed to the next stage upon approval of this inception report. An appropriate inception report format will be made available to the team as part of this TOR.

Phase 2: Field study (January to February 2013)This main activity of this phase will be the country level project reviews in line with the agreed plan and methodology from phase 1.

Phase 3: Analysis and Reporting (March, 2013)Following country visits, this last phase will involve analysis of information, report writing and packaging of all collected information in accessible format for future use by Sightsavers.

The successful review team is expected to provide a detailed schedule of activities covering each of these above periods as part of their expression of interest. This should be linked to an appropriate budget, a template of which will be provided as part of the expression of interest package.

7. Outputs/Deliverables and Reporting Formats

7.1 Outputs/DeliverablesFour main outputs are expected from this review as follows:

7.1.1 Inception ReportAn inception report shall be produced no later than 17 December, 2012. The report should detail a concise review design, methodology, tools, work plan. It should also include a detailed task description for consultants who will be undertaking local level project review.

7.1.2 Draft ReportA draft report shall be produced no later than 28 February 2013. The report should conform to the Sightsavers reporting format, which will be made available to the consultant on signing the contract.

7.1.3 Final ReportA final report shall be produced no later than 31 March 2013. The report should conform to the agreed reporting format, and should incorporate all feedback received.

7.1.4 Summary Findings On submission of the final report, the team is expected to submit a PowerPoint presentation (maximum 12 slides), summarizing the methodology, challenges faced, key

41

findings under each of the evaluation criteria and main recommendations. This should be submitted together with the final report by 03 May 2012.

7.2 Reporting FormatDetailed guidelines on how to structure the review report will be provided to the evaluation team prior to commencement of the activity. The team should conform to this format.

Please note that penalties up to 10% of total agreed fees will be imposed for non-compliance with the requirements 7.1 to 7.5 and reporting format provided above.

8. Administrative and Logistic Support

8.1Budget As part of the expression of interest, interested consultants/teams should submit their competitive bid to undertake this piece of work in line with these Terms of Reference, Sightsavers understands that budget forecast could vary according to the methodology of the study but expect consultants to provide a strong rationale for their forecast. Sightsavers will cover the following costs:

i. In country transportation ii. Visa and vaccination costsiii. Travel and accommodation expenses. These will be reimbursed (economy class

airfares, mid-class hotel) upon production of receipts or at subsistence costs according to established rates per country (to be agreed in advance with Sightsavers).

All other costs not covered above should be factored in the consultancy fees proposed by the team.

8.2Schedule of Payment The following payments will be made to the successful team using and agreed mode of payment.

On contract signing: 10% After acceptance of inception report: 10% After acceptance of First draft: 30% After acceptance of Final Report: 50%

8.3LogisticsAll the logistics support will be provided by Sightsavers and should not be included in the budgets. Specifically the following aspects will be covered by Sightsavers:

i. In-country accommodation bookings (Bed and breakfast only).ii. In-country transportation arrangements.

42

Appendix b: Innovation Fund Guidelines

Programme Partnership Arrangement: Innovation Fund

Guidelines

The Innovation Fund requires projects that clearly demonstrate and provide evidence of why the work is innovative, how success will be documented and disseminated, and its impact. Projects will havea clear plan for replicating and scaling-up the successes documented during implementation. Therefore new approaches must be married with a rigorous design approach that allows for the innovation to be appropriately evaluated, documented and lessons learned used to guide future Sightsavers programming and contribute to global learning. It is for these reasons that it is expected that most proposals seeking support through the Innovation Fund will involve a partnership with a research institution.

Purpose

The Innovation fund has been created to specifically respond to the urgent need to explore innovative ways of addressing eye health, inclusive education and social inclusion priorities at scale. The purpose of the fund is to improve programme capacity and the quality of programmes by providing new ways of working and strategies to increase coverage/scaling up in Sightsavers programme areas. The fund will allow for a degree of risk-taking necessary when developing new approaches.

Without being too prescriptive, the Innovation Fund will be looking to award grants that can clearly demonstrate one of the following areas of innovation:

Innovative practices - Innovative practice refers to the methods and modes of operation adopted to achieve lasting development results. Approaches to particular issues which have proven development outcomes in pockets of practice, and where opportunities for implementation exist in other development contexts. It also involves support for niche areas of poverty reduction which have a direct and transformative impact on the most marginalised in poor communities. It may involve working on programmes that have the potential to scale up significantly in the region, or be working on a particularly innovative practice which may be replicated elsewhere.

Innovative partnerships – In order to achieve lasting results, it is necessary to establish robust partnerships with a range of development actors including local communities, businesses, and the private sector, institutions, universities, governments and NGOs. To pursue complementary programmes and collaborative approaches to achieve lasting change

43

Innovative approaches - Innovative approaches may include new or improved ways of working on locally led accountability mechanisms, or capturing beneficiary perspectives in monitoring, reporting and evaluation approaches. This also includes innovation to help strengthen local government systems and processes.

FIT with DFID’s Priorities and Values

Whilst it is essential that innovative projects fill the above criteria, it is also desirable that projects fit with DFID’s priorities and values – projects should demonstrate value for money, promise results for poor people and be rooted in strong evidence. Projects should have clear links with international commitments, such as the MDGs and UNCRPD.

Why is this approach considered to be innovative?

When putting together a proposal for the innovation fund, it is useful to consider some of the following: Is a new approach to be applied? Are new ways of partnership to be developed? Who or what is the inspiration behind this project? Will the project involve experimenting or piloting new ideas or approaches? What risks are being considered? How does this approach differ from work being done by others? What is the potential for future replication or larger-scale implementation? How will lessons be learned from the results of this initiative? How will this initiative contribute to global learning?

DefinitionsThere are a range of definitions of the term ‘innovation’ – for the purposes of this fund, the following definitions have been selected as most representative of what Sightsavers means by innovation:

“Innovation must be credible, relevant, observable, cost effective, easy to adapt, and compatible with existing policies – if not compatible (i.e. when working with stakeholders outside of the government) then advocacy for change must be built in to the innovation”(Dr Ashock Dyalchand – Sightsavers Programme Meeting, Mumbai, 2010)

Innovation does not necessarily mean ‘brand new’ but could be an approach applied for the first time in a particular country or countries; or new ways of applying/adapting/developing an existing technique or initiative. We also understand innovation to mean experimentation, with the risk of failure (as long as lessons are clearly learned and the implications of failure are appropriately considered); inspiration which is the creative process of looking at entrenched problems with fresh eyes; genuine participation of people most affected by a problem to release more energy for ideas; and from looking more at what works and why.

(DFIDs GPAF innovation fund guidelines, http://www.dfid.gov.uk/Documents/funding/gpaf/glob-pov-act-fnd-innov-gnt-guide-appls.pdf)

44

Resource allocation

Sightsavers expects to allocate £750,000 per year (A total of approximately £2.25 million by 2014) to the innovation fund, through the Programme Partnership Arrangement.

Format(Please see Sightsavers Commitments Management Process – approved concept note form to be used)

All Concept Notes should follow Sightsavers’ approved internal project concept note format, and follow the same procedures outlined in Sightsavers’ Commitments Management Process.

Application process (Please see Sightsavers Commitments Management Process for details and timeframe)

Concept notes (CNs) should be submitted to Diana Washer to then be distributed to the Commitments Management Panel 1 (CMP1). If the CN is considered a possible candidate for the Innovation Fund, it will be sent for review to Commitments Management Panel 2 (CMP2). CMP2 includes external experts who will only review CNs which are considered possible candidates for the innovation fund.

45

Appendix c: Topic Guide (initial UK interviews)

Tiger Consulting: Questions for Sightsavers UK staff pre-visits

As part of the evaluation of the Innovation Fund Round 1 projects which Tiger Consulting is currently conducting for Sightsavers we would be grateful if you could answer the following questions: The evaluation will look at how the projects are progressing against performance targets to date.

1 How long have you been working with Sightsavers?

2 What is your role within Sightsavers?

3 How would you describe the focus/mission of Sightsavers?

4 What is your involvement with the Innovation Fund, and what do you see as its aim?

5 Do you have any day-to-day/operational contact with any of the Innovation Fund Round 1 projects? If Yes, what?

6 What do you think Sightsavers will gain from the Innovation Fund?

7 What is your view of the Round 1 projects? Which do you especially like, and why? Which do you consider strongest/weakest?

8 Concerns were expressed about some of the projects; was this something you knew about?

9 What would you have changed, if anything, in the selection process for Round 1 projects?

10 Round 2 projects have now been selected. What did you learn from the Round 1 process and projects in selecting Round 2 projects?

11 Do you have anything else you want to share about the Innovation Fund Round 1? (process, aims, projects, reporting, etc)

46

Appendix d: Topic Guide for Country Visits

Topic Guide for Project Visits: Country Office staff and lead partners

Background Who are you? What is your role generally? What is your role in the project?

Project What are its aims? Who are the partners? What is the budget? What structures are in place for implementation and coordination of the project? What M+E systems are in place for monitoring progress? What are the reporting requirements and how easy are these for them to meet? What is the “Theory of Change” of this project?

Innovation What do you understand by “innovation”? Why is this project innovative; what are the innovative aspects?

Application process Who wrote the application? Was it written specifically for the Innovation Fund? When was funding approved? When was funding available? Did you know the Innovation Fund is part of the PPA from DFID?

Partners Who is the lead partner in the project? Who are the other partners? Why did you choose these partners? Is the partnership itself ‘innovative’? Are there other partners you could be working with?

Beneficiaries Who are the main beneficiaries of this project? How were the beneficiaries identified? How are the beneficiaries involved in the project?

Challenges What factors have promoted/impeded the project? What actions have you taken to meet challenges?

47

How can the project be improved? What project revisions, if any, need to be made? How is risk managed/mitigated in the project? Are the project aims realistic? (e.g., within the project time limits)

Learning and Dissemination What data are you collecting? How useful is it? What does the data show? How will the data add to Sightsavers knowledge? How are you documenting the learning from this project? What learning are you sharing, how, and with whom? What support/interest has been expressed from the Regional/UK offices?

Sustainability/replicability How sustainable does the project seem to be in terms of local benefit? How sustainable do you think the changes the project seeking to impact might

be? Is there any evidence that this work is replicable?

Disability How are you working with and involving blind and disabled people in this project

and others? How do you view disability and disabled people; are you working for them or

with them? What accessibility or other adaptations havebeen made in this project, or other

projects, or the country office, to include disabled people? Where particular challenges do you think blind/disabled people face in your

country? How could you improve the inclusion of disabled people in your work?

Performance/areas for Improvement How is this project performing against its goals? How could performance of the project be improved? What other technical support do you need? What might you do differently?

Value for Money and other DFID criteria How do you assess the VfM/cost effectiveness of this project? How could this aspect be improved in this project? What views do you have about the efficiency/effectiveness of this project, and

how do you assess these factors?

Support from UK What support is received from UK with this project? What do you think of it?

48

What else would you like?

Other work What other projects are you working on? What percentage of your time is spent on the ‘innovation’ project(s)?

49

Appendix e: People consulted

(1) People Consulted in Tanzania

Project Coordinator

Director General, Zanzibar Ministry of Health

Acting National Eye care Coordinator, Zanzibar Ministry of Health

Optometrist, MnaziMmoja Vision Centre

Ophthalmic assistant, MnaziMmoja

Optometrist (Paediatric), MnaziMmoja

Low therapist, MCBI

Primary Health Care workers, FGD Kivunguni, Unguja Island, ZanzibarKivunge Nursing Officer

Nungwi Psychiatric Nurse

Kivunge Nursing Officer, Tumbatu

Clinical Officer, Tumbatu (Jongowe)

Nursing Officer, Kidoti

PEC ZNZ

FGD with Health Workers, Kitope, North B, UngujaDistrict Public Health Worker

Shehe, Mbaleni Village

In charge of the health centre

Optometrist, Vision Centre, ChakeChake

FGD Msuka Primary School, WilayayaMichiweni, Pemba KaskaziniAssistant Medical Officer (Cataract Surgeon), Focal person for Eye Care, Ministry of Health

Primary Eye Care Co-ordinator

Inclusive education Co-ordinator

Inclusive Education Teacher

Trained IE teacher

50

PHC: Primary Eye Care FGD at Micheweni Cottage Hospital, Micheweni DistrictPublic Health Unit, PUS Konde

Public Health Co-Ordinator, District Health and Medical Team

Hospital Ophthalmologist

Muhimbili Hospital

Project Administrative Lead, Muhimbili

Paediatric Ophthalmologist, MCBI

Tanzania Country Office

Country Director

Programme Manager

Programme Officer

(2) People consulted in Bangladesh BCO staff: Acting Country Directors (2), others South Asia Regional Director Disabled Rehabilitation and Research Association (DRRA) (partner) project staff at

office in Shyamnagar, Satkhira Self Help Group members at Manik Khali, Ramjan Nagar UP, in Shyamnagar

Upazila Chair of the Union Parishad and other elected members at Padmapukur Union

Parishad (UP), Shyamnagar, Satkhira Union Disaster Management Committee (UDMC) in Satkhira Pot Song singers and audience in Satkhira DRRA volunteers Self-help group members’ trained on Income Generating Activities (IGAs) at

Iswaripur in Shyamnagar Upazila and at Datinakhali, Shyamnagar Observation of ongoing self-help group meeting (SHG) at Shinghortoli, Shyamnagar

Upazila Sharing meeting with Upazila level Advocacy Group members at Tala Upazilla Sharing meeting with multi stakeholders (govt. officials, people’s representatives,

SHG members) at Shyamnagar Upazila Observation of Joint Planning meeting at Sovnali Union Parishad, Ashasuni Upazila Sharing discussion with Ashasuni Sadar UP Dakhinpara SHG, Ashashuni Upazila

re IGA linkages for securing livelihoods, skills training and marketing entities Sharing meeting with local Union Parishad Chairman and Members at Islam kathi

UP, Tala Upazila

51

Meeting with by GMark Consulting Limited to discuss findings of scoping study to analyse potential and feasible market system for persons with disability and recommend suitable Income Generating Activities in Satkhira and Khulna

Various other individual disabled people and their families

(3) People consulted in Pakistan

PCO staff: Director, Programme Managers (2), other Programme and Admin staff South Asia Regional Director (in Bangladesh/UK) Director and various staff of Al-Ibrahim Eye Hospital Karachi (implementing partner

of DRB project) CEC Sindh Director and other staff Meeting with the core team of DRB project and beneficiaries in Gadaap Town and

Malir Representatives of Diabetic Association of Pakistan Project site visit and meeting with the core team, beneficiaries, Community

Volunteers trained under the project, Rep of the DHQ Hospital, Social Welfare Dept and any other stakeholder

Director and various project workers of HRDS (Implementing partner of Accessible WATSAN project) at head office in Islamabad

70 Lady Health Workers in Karachi region Participation in 3 Village Health Communities in Sukkur to Shahdadkot Director and staff of STEP (DPO) Seminar with Comsats staff and research students to discuss action research Various other disabled people and their families, notably for WatSan project

(4) People Consulted in UK Director of Policy & Strategic Programme Support Head of Research Head of Monitoring and Evaluations Systems Head of Strategic Government Partnerships DFID PPA Manager Institutional Funding Officer UK Head of Programme Development Programme Reporting and Control Manager

52

Appendix f: Tiger Inception ReportContents Page

TIGER CONSULTING

Evaluation of Sightsavers Innovation Fund Round 1

17/12/12IntroductionApproachMethodology and Data Collection PlanExample evaluation matrixKey challengesIn-country supportReferenced reports

1. Introduction [max 250 words]

This inception report sets out the scope and objectives of the evaluation of the Sightsavers International (SI) Innovation Fund Round 1, and describes the conceptual framework that we will use. It sets the key questions to be asked, and the evaluation methodology; the data collection methods, data sources, sampling and indicators that will be used to answer the key evaluation questions. There is also a workplan which indicates the phases in the evaluation with key deliverables and milestones.

Our Aims in carrying out this evaluation:

Follow the Terms of Reference (TOR)(see Appendix 1) Apply a rigorous and objective methodology Give a clear, concise picture of performance of the projects against outcomes Provide a clear picture of expenditure to date against forecasts and budget

allocation Make recommendations to adjust and improve performance, including

maximising learning and other benefits from the projectsWe will ask direct and challenging questions to assist Sightsavers in understanding what options are open improve programme implementation of the Round 1 Innovation Fund projects if off track, with particular emphasis on the innovation aspects.

Our demonstrable commitment and practical contribution to disabled peoples’ rights will help to ensure that our review of these innovations provides useful learning and examples toensure these projectsimprove the lives of disabled people, a key indicator for us. We will do this by providing an honest assessment of progress and useful advice and support to Sightsavers staff in country offices and the UK.

53

2. Approach [max 500 words]

Overall approach

Comparing plans to reported results and verifying results Giving overall assessment of performance (done/on track/not on track) Identifying appropriate adjustments to enhance performance Reporting objectively and clearly on what we find, with recommendations for

improvement where appropriateWe see three linked but distinct purposes, assessing:

Innovation Fund Round 1 projects against their targets and logframes, and expenditure (including validating the expressed “concerns”). We will use the DFID scoring system as part of this assessment, reflecting SI’s need to demonstrate to DFID the effectiveness of the Fund – see Appendix 9)

How the projects are delivering innovation (e.g., what was identified as innovative, is it being achieved and communicated?)

If Sightsavers has systems and procedures necessary to capture, reflect on and communicate this learning on innovation within Sightsavers and the wider sector, with the aim of replication and scaling up (recent interviews with Sightsavers staff have stressed this aspect)

The TOR for the evaluation set out questions to be answered under the headings of structure, process and outcome/results (see Appendix 5). As part of assessing the projects under these headings, we will ask:

Innovation

What do you understand by Innovation? (checking/comparing/contrasting with definitions from Innovation Fund guidelines)

What innovation was the project intended to demonstrate? Has that innovation been demonstrated? If not, why not; and how the project get back on track to deliver within the

remaining PPA period? Other than innovation, what were the specific aims and targets, and how has it

performed in terms of successes, impact and other relevant criteria (e.g., effectiveness, efficiency, value for money – key DFID PPA criteria)

Learning, dissemination and uptake

Have successes, shortcomings and lessons learned been documented and communicated as originally specified?

If Yes, how and what has been the outcome? If No, why not, and how could this be improved?

The TOR provided two core pieces of information which we have used in framing our questions above, and will bear in mind:

The five projects were chosen specifically to demonstrate “innovation”. Innovation involves risk.

“Concerns” have been voiced on four of the projects. Discussions with Sightsavers UK staff indicate these concerns were primarily about pace of expenditure, an issue raised in the recent Independent Progress Review (IPR) of SI’s PPA (Innovation Fund is allocated 20% of the PPA funding). We understand concerns may be less now with more information available, especially financial;

54

but we will pay special attention to expenditure against forecasts to ascertain actual spending against budget, using documented financial records and verbal updates from country staff, as well as a range of other key indicators of performance.

What is Innovation?Critical to success will be clarifying amongst stakeholders what is innovation. Two related definitions of innovation are set out as part of the Fund (appendix 3); DFID also talks of incremental and radical innovation. We will test if these concepts are understood by project implementers and relevant stakeholders.

3. Methodology and Data Collection Plan[max 1000 words]

Our research will produceevidence to answer all main evaluation questions. Our basic methodology will examine comprehensively the background documents and data on these projects before embarking on field work to test and validate any ideas formed at inception stage. The field work stage remains vital, giving us a chance to discuss aims and progress with both country office programme managers and beneficiaries, and to observe for ourselves. We have accepted the advice of Sightsavers that an e-questionnaire will not be necessary prior to field trips, given the richness of documentation available. We will remain flexible in terms of methodology, since each project is very different in terms of design, aims, country and cultural context (e.g., approach to interviews may need to be varied).

Our methodologies and toolsInception and Initial Review Stage (Output = Inception report and relevant data for country visits) (December 2012)

Document and desk review: taking in all the Innovation Fund Round 1 project documents, logframe, reports, financial information, M&E systems, etc, plus the TOR for the Innovation Fund and other relevant information about aims of the fund, allocated resources within PPA, etc (see Appendix 6)

Semi-structured interviews with UK-based Sightsavers staff: to derive information to shape in-country evaluation focus and methodology, and to gain data on the Innovation Fund background, aims and Round 1 projects, including knowledge of project performance. A template will aid consistency, objectivity and comprehensiveness. We will open up key issues and fine tune our approach but not enable UK staff views to direct our findings (for template, see Appendix 4)

Planning with relevant stakeholders for visits to the four country offices (Nigeria, Pakistan, Bangladesh, Tanzania): including country/regional staff, development of country visit TORs, aims and objectives, itineraries, etc

Data Collection Stage (including field work) (output = data) (January-February 2013)

Semi-structured interviews with relevant Sightsavers country office staff and other key stakeholders, depending on specific aims/objectives of each project (e.g., community representatives, local leaders, partners, policy makers, government representatives targeted for influencing). A topic guide will assist our interviews and consultations (see Appendix 5)

Direct Observation of on-the-ground project progress, ensuring we take an independent perspective as well as being guided by country office staff and beneficiaries

55

Group discussions/consultations in-country at the end of each field visit, with a cross-section of stakeholders (country/regional Sightsavers staff, partners, etc).

Participatory consultations in each target country where possible with end-user beneficiaries (e.g., blind people, recipients of preventative treatment, etc)

Data Analysis (output = draft report) (February-mid March 2013)

Further semi-structured interviews on UK return, to validate evidence and impressions from field visits

Group discussions with UK-based staff, with focus depending on what emerges from all of the above

Tiger Consulting team workshop, to analyse and compare results and outcomes so far with reasonable expectations of where the projects should be at this stage

Use of a traffic light indication of progress (RED/AMBER/GREEN); Green = basis for learning and case studies of success, Amber = areas that are satisfactory, Red = areas that are off track and need to be explored further with appropriate stakeholders. Red areas will be basis for Recommendations for change to ensure the Innovation Fund criteria are met.

Submission of draft report, meeting the Sightsavers requirements and standards for evaluation reports

Review Stage (output = presentation + final report) (mid/late March 2013)

Presentation of our draft report and findings to Sightsavers staff if requested/required

Feedback to participants in each of the four countries, to elicit their comments Revision as appropriate and production of final report and Powerpoint

presentationQuality and nature of the dataMuch of our data will be qualitative, collected through interviews and focus groups with staff, partners and beneficiaries. This data will be analysed using a coding system to identify the most common themes before drawing general conclusions. The changing lives case studies and stories which we collect will be included in this process.

We will collect quantitative data relating to both beneficiaries circumstances (e.g. age, number of family members etc) and their views about the services provided and changes created by the projects, enabling us to understand relative satisfaction levels.

We propose to apply 'Most Significant Change' techniques during the focus groups that we will conduct in the three countries where work is on-going. This is particularly useful at the early stages of a project and will help us to identify what is being achieved at present. The techniques used in Nigeria will be adapted slightly because the project is of a different nature and is almost complete.

We will useaction research techniques to help identify potential strategies to deal with challenges and gain an understanding of theories of change (TOC) that underpin each project and its interventions. We will triangulate understanding of TOC amongst stakeholders, and validate TOC against interventions for suitability, effectiveness and likelihood of delivering target outcomes.

After data collection we will hold a team workshop to share experiences and plan the analysis. Data analysis will be led by JE, using her extensive evaluation experience to ensure a rigorous process. LW will ensure we apply inclusive practice to tool

56

development and data collection and analysis.

Data analysis will compare results, measured by both quantitative and qualitative data, with actual performance; from this analysis we will draw objective conclusions about what has worked, what has not, and why, which will form the basis of further checking and recommendationsStructure of final report

We will be compare results and outcomes so far with reasonable expectations of where the project should be, based on our own experience, and also of Sightsavers' staff.

We will detail findings, conclusions and recommendations in order of priority We will use a traffic light indication of progress (RED/AMBER/GREEN), as

outlined above Our report will be compatible with Sightsavers standards for evaluation reports Our report will take account of DFID’s interest in the Innovation Fund, including

usingDFID’s scoring system for assessing projects

57

3.1 Example Evaluation Matrix (*see also Additional Evaluation Matrix in appendices)

Key Evaluation question to be addressed

Data Collection Technique

Primary Data Tools

Secondary Data Tools

Data Source

1.What monitoring mechanisms are in place to ensure the projects remain on course to achieving their outcomes and how can/should these be strengthened?

Documentation Interviews Country staff, partners

2.What factors have promoted or impeded the implementation of the activities and the achievement of the outcomes/objectives?

Documentation Interviews, observation

Country staff. Partners

3How do the projects rate against standard DFID criteria? (relevance, effectiveness, efficiency, value for money, etc)

Documentation Analysis Reports, interviews, DFID

4.To what extent are the projects on course to achieving their anticipated outcomes and objectives?

Documentation Interviews, analysis

Country staff, Tiger team

5. What innovation was the project intended to demonstrate? Documentation Interviews UK+country staff

6.Has that innovation been demonstrated? If not, why not; and how the project get back on track to deliver within the remaining PPA period?

Documentation Interviews, observation

UK+country staff, Tiger team

7. Have successes, shortcomings and lessons learned been Documentation Interviews UK+country

58

documented and communicated as originally specified? If Yes, how and what has been the outcome? If No, why not, and how could this be improved?

staff

8.What project revisions need to be considered, if any? (Targets, strategies, activities, budget etc.)

Interviews Analysis UK+country staff, Tiger team

9.

What would be the most effectiveimprovements to implementation and coordination of the Innovation projects, in Sightsavers and at each country level, to maximize replicable learning?

Interviews, documentation

Interviews, observation

UK+country staff, Tiger team

59

4. Key Challenges [max 100 words]

Conflicting views amongst those managing and implementing these projects, primarily Sightsavers country office staff and local partners, and UK-based programme managers, especially if there are performance issues

Differences in scale, budget, content and duration of the projects makes any comparison difficult (including relatively short implementation period so far)

Context of “concerns”, which may result in some resistance to scrutiny

Some of the projects have complex/large geographical reach, with consequent challenges in sampling and assessing progress on the ground

Adjustments will be necessary for the full inclusion of disabled beneficiaries in the evaluation; country offices will advise.

5. In-Country Support [max 100 words]

Planning and agreeing field visit itineraries, including meetings with beneficiaries and other relevant stakeholders

Country office/regional staff time for interviews/discussion

Accommodation recommendation/reservation

In-country travel to projects

Temporary office space

Airport transfers

Identification of key non-Sightsavers stakeholders to interview

Advice re security, visas and other travel arrangements for project countries

6. Referenced reports

Sightsavers Innovation Fund Round 1 Terms of Reference

Sightsavers Programme Partnership Arrangement (PPA): Innovation FundGuidelines

Sightsavers Innovation Fund Round 1 funded project documents (i.e., concept notes, full proposals, feedback to authors, logframes, budgets, and progress reports)

Sightsavers PPA document and logframe

Sightsavers PPA Year 1 Annual Report

Sightsavers PPA Independent Progress Review 2012

Sightsavers website

Sightsavers Annual Report and Accounts 2011-12

60