Final Evaluation Report - UNICEF · 1 Evaluation Terms of Reference, page 4. 2 2. Provision of gender equitable support to the most vulnerable people in host communities (in all 12

Final Evaluation Report

Final Evaluation Report

Evaluation of UNICEF’s response to the Water,

Sanitation and Hygiene Needs in Jordan as a

result of the Syrian refugee crisis (July 2012 to

July 2017)

PREPARED BY

INTERNATIONAL SOLUTIONS

GROUP

An IMC Worldwide Company

PREPARED FOR

UNICEF Jordan

16 April 2019

1

Evaluation Timeframe

This evaluation covers UNICEF’s WASH programme activities between July 2012 and July

2017. The evaluation was conducted between November 2017 and September 2018. Data

collection activities took place during July and August 2018.

Geographic Scope

This evaluation covers UNICEF Jordan WASH activities in the four camps of Za’atari,

Azraq, Cyber City (closed in September 2016) and King Abdallah Park, as well as Rukban

and Hadalat (abandoned in September 2017) at the Syrian/Jordanian border. Although there

are Syrian refugees registered in all 12 Governorates of Jordan, the evaluation focused on the

refugees registered in Zarqa, Irbid, Mafraq and Amman Governorates, which represent 90

percent of registered refugees.

Organizations and Consultants conducting the evaluation

This evaluation was conducted and led by International Solutions Group, a division of IMC

Worldwide, and its partner, the Interdisciplinary Research Consultants, a research firm based

in Amman, Jordan.

UNICEF Jordan was the evaluation’s commissioning organization.

1

Table of Contents THE WASH PROGRAMME AND CONTEXT ...........................................................................6 DESCRIPTION OF THE WASH PROGRAMME ..................................................................................6 PROGRAMME HISTORY ..................................................................................................................... 6 WASH PROGRAMME PURPOSE, COMPONENTS, AND MANAGEMENT ........................................................ 8 WASH PROGRAMME LOGIC MODEL AND RESULTS FRAMEWORK ............................................................. 9 PROGRAMME CONTEXT......................................................................................................... 12 MANAGING AN INFLUX OF REFUGEES FROM A RELATIVELY WATER-SUFFICIENT AREA .................................... 12 PRE-EXISTING POLITICAL ATTITUDES TOWARD REFUGEES ........................................................................ 13 LEGAL AND INSTITUTIONAL FRAMEWORK OF THE WASH SECTOR IN JORDAN ............................................ 13 ECONOMIC TENSION ...................................................................................................................... 13 FUNDING ENVIRONMENT ................................................................................................................ 14 DESCRIPTION OF THE EVALUATION ................................................................................. 15 PURPOSE AND OBJECTIVE ...................................................................................................... 15 SCOPE .............................................................................................................................. 15 EVALUATION METHODOLOGY......................................................................................... 15 DESK REVIEW ..................................................................................................................... 15 QUANTITATIVE ANALYSIS....................................................................................................... 16 PRIMARY RESEARCH: QUALITATIVE DATA COLLECTION METHODS AND SAMPLING ................................ 16 COVERAGE IN INTERVIEWS AND FOCUS GROUPS .................................................................................. 16 RAPID ASSESSMENTS AND USER INTERVIEWS ...................................................................................... 17 WASH MESSAGING AND PRACTICES ................................................................................................ 17 PARTNER INTERVIEWS .................................................................................................................... 17 DUTY BEARERS AND STAFF .............................................................................................................. 17 STAKEHOLDER INTERVIEWS.............................................................................................................. 17 ANALYSIS METHOD........................................................................................................................ 17 LIMITATIONS TO THE EVALUATION............................................................................................ 18 ETHICAL PRINCIPLES IN CONDUCTING THE EVALUATION ................................................................. 18 FINDINGS ....................................................................................................................... 19 RELEVANCE ........................................................................................................................ 19 EVALUATION QUESTION 1 ............................................................................................................... 19 EVALUATION QUESTION 2 ............................................................................................................... 26 EVALUATION QUESTION 3 ............................................................................................................... 27 EFFECTIVENESS ................................................................................................................... 29 EVALUATION QUESTION 4 ............................................................................................................... 29 EVALUATION QUESTION 5 ............................................................................................................... 53 EFFICIENCY ........................................................................................................................ 63 EVALUATION QUESTIONS 6 AND 7 .................................................................................................... 63 EVALUATION QUESTION 8 ............................................................................................................... 69 EVALUATION QUESTION 9 ............................................................................................................... 70 EVALUATION QUESTION 10 ............................................................................................................. 72 SUSTAINABILITY .................................................................................................................. 72 EVALUATION QUESTION 11 ............................................................................................................. 72 EVALUATION QUESTION 12 ............................................................................................................. 74 EVALUATION QUESTION 13 ............................................................................................................. 74 EVALUATION QUESTION 14 ............................................................................................................. 75 EVALUATION QUESTION 15 ............................................................................................................. 75 EVALUATION QUESTION 16 ............................................................................................................. 76

2

COVERAGE ......................................................................................................................... 77 EVALUATION QUESTION 17 ............................................................................................................. 77 EVALUATION QUESTION 18 ............................................................................................................. 79 COORDINATION .................................................................................................................. 79 EVALUATION QUESTION 19 ............................................................................................................. 79 EVALUATION QUESTION 20 ............................................................................................................. 80 CONCLUSIONS ................................................................................................................ 80 LESSONS LEARNED AND RECOMMENDATIONS ................................................................ 84 LESSONS ........................................................................................................................... 84 RECOMMENDATIONS ............................................................................................................ 86 WORKS CITED ................................................................................................................. 89 ANNEX A: EVALUATION TERMS OF REFERENCE................................................................ 92 ANNEX B: ANONYMIZED LIST OF INTERVIEWS ............................................................... 104 ANNEX C: LIST OF SITES VISITED .................................................................................... 105 ANNEX D: DOCUMENTS CONSULTED ............................................................................. 106 ANNEX E: DATA COLLECTION TOOLS ............................................................................. 112 ANNEX F: TEAM DESCRIPTION....................................................................................... 119 ANNEX G: EVALUATION MATRIX ................................................................................... 120 ANNEX H: UNICEF WASH PROGRAMME RESULTS FRAMEWORK..................................... 149 ANNEX I: UNICEF PROVIDED FINANCIAL ESTIMATES ...................................................... 155 ANNEX J: ADDITIONAL FINANCIAL INFORMATION ......................................................... 162

3

Table of Figures Figure 1 WASH Programme Implied Logic Model ................................................................ 11 Figure 2 Funds budgeted vs mobilised by year (USD in millions).......................................... 14 Figure 3 Public Latrine at Azraq camp .................................................................................... 23 Figure 4 The stand-by power generation increases the ability of the water supply and

distribution system to function................................................................................................. 24 Figure 5: Za'atari Camp 2012 (Source:Times of Israel, 29 August 2012) ............................... 33 Figure 6 Very clean household latrine observed in Za'atari camp. .......................................... 54 Figure 7 Rubbish Bin in Azraq ................................................................................................ 55 Figure 8 Power generators and fuel storage at Rukban water system. Extra power generation

and fuel storage capacity minimizes the potential for water system down time. .................... 58 Figure 9 Unimproved pit latrine cover at an ITS near Amman ............................................... 61 Figure 10 Cost of providing a cubic meter of water by year based on partner budgets .......... 65 Figure 11 Water provision cost savings in Za'atari (Source: UNICEF WASH Programme) .. 66 Figure 12 Membrane bioreactor wastewater treatment system functioning properly at Za'atari

Camp. ....................................................................................................................................... 67 Figure 13 Division of labor among UNICEF Partner organizations. (Source: WaSH in

Za'atari Snapshop, Jan-Feb 2014) ............................................................................................ 69 Figure 14 Percent of UNICEF Total Expenditure by Beneficiary Type (August 2012 – July

2017) ........................................................................................................................................ 71 Figure 15 WASH Programme expenditure by activity category (August 2012 – July 2017) . 71

Acronyms and Abbreviations ACF Action Contre la Faim

ACTED Agency for Technical Cooperation and Development

BPRM The Bureau of Population, Refugees, and Migration

CEDAW Convention on the Elimination of All Forms of Discrimination against Women

CRC Convention on the Rights of the Child

DAC Development Assistance Committee

FGD Focus Group Discussion

GIS Geographic Information System

GTZ Gesellschaft für Internationale Zusammenarbeit

IEC Information, Education, and Communication

IMF International Monetary Fund

INGO International Non-Governmental Organization

ISG International Solutions Group

ITS Informal Tented Settlements

JEN Government of Japan supported development agency

JHCO Jordanian Hashemite Charity Organization

JRP Jordan Response Plan

JVA Jordan Valley Authority

KAP Knowledge, Attitudes, and Practices

4

KII Key Informant Interview

MBBR Moving Bed Biofilm Reactor

MOE Ministry of Education

MWI Ministry of Water and Irrigation

NGO Non-Governmental Organization

NRC Norwegian Refugee Council

NRW Non-Revenue Water

OECD Organization for Economic Cooperation and Development

RAM Results Assessment Module

TDS Total Dissolved Solids

TOR Terms of Reference

UNHCR Office of the United Nations High Commissioner for Refugees

UNOPS United Nations Office for Project Services

UNRWA United Nations Relief and Works Agency

USAID US Agency for International Development

VIP Ventilated Improved Pit

WAJ Water Authority of Jordan

WASH Water, Sanitation, and Hygiene

WIS WASH in Schools

1

Executive Summary In March 2012, when a relatively small number of Syrians were fleeing the civil war into

Jordan, UNICEF responded by setting up Water, Sanitation, and Health (WASH) services

and facilities at border crossings, called transit centers. The initial numbers of refugees were

in the thousands rather than the much higher numbers that UNCHR was predicting, and some

members of the international press described the international community’s response and

predictions of a massive refugee influx alarmist and misleading (Seely, 2012).

By August UNICEF’s and UNHCR’s preparations paid off. One thousand refugees per week

crossing the border quickly became over 10,000 per week (UNHCR, 2012). Some refugees

found sponsorship and entered host communities, where they had access to household and

government WASH services. Others, unable to return home and without other options,

entered the Za’atari refugee camp that opened on 30 July 2012. By December, over 66,000

refugees were living in the camp (UNICEF, December 2012).

In July 2012, UNICEF was the only organization in Jordan that had the resources, capacity,

and institutional commitment to take leadership of the WASH response related to the Syrian

refugee crisis. UNICEF accepted responsibility for providing WASH services in a country

that was exhausted by previous refugee crises, water scarcity, a complex political

environment, and funding uncertainty. From July 2012 through July 2017, UNICEF provided

life-saving water and sanitation resources under these difficult conditions for approximately

400,0001.

The programme that UNICEF developed as its WASH response to the crisis was to become

one of the largest programmes in UNICEF Jordan’s portfolio. The scope of the programme

was initially a mandate to provide services to refugees in camps. It expanded to include

services for water-deprived Jordanian host communities that were resource constrained

before the crisis and were further burdened by the crisis. The WASH programme’s scope also

grew to provide urgent WASH services for refugees living in the border settlements at

Hadalat and Rukban. The emergency response required quick and flexible decision making

and efficient utilization of scarce resources, which UNICEF successfully performed.

The WASH programme was not without missteps or shortcoming, which the evaluation team

discusses in the full report. Overall however, the programme provided an effective response

in line with its core mission, providing life-saving WASH services in response to the Syrian

refugee crisis. It has also increased efficiency and worked to ensure coverage of vulnerable

populations. Two difficult achievements in a dynamic and complex environment.

By 2017, The WASH programme had evolved in response to its changing environment,

adopting a framework that worked at four levels:

1. Provision of equitable WASH services and the dissemination of WASH messages in

the four camps (Za’atari, Azraq, King Abdullah Park and Cyber City) and two

settlements in Rukban and Hadalat, to the most vulnerable people namely children

and women.

1 Evaluation Terms of Reference, page 4.

2

2. Provision of gender equitable support to the most vulnerable people in host

communities (in all 12 Governorates) namely to children and women.

3. Support to the WASH sector through technical support to the Ministry of Water and

Irrigation.

4. Coordination of the sector at camps and national level.

The WASH programme’s objective, its highest-level target, is that2 “Quality WASH facilities

are sustained and utilised, and hygienic behaviours practiced, by the most vulnerable while

ensuring the protection of the environment.”

The programme measures the achievement of this objective through four outcome indicators:

Indicator 1: Existence of strategic planning capacity in MoWI and sector and other

relevant institutions in Emergency preparedness and response.

Indicator 2: Proportion of population at camps and host communities including schools

using improved drinking water as per Jordan standards.

Indicator 3: Proportion of population at camps and host communities including schools

using improved sanitation facilities.

Indicator 4: Proportion of population practicing hygiene promotion at targeted camps,

host communities and schools.

Background and Context The WASH programme has faced complex external social, environmental, political, and

institutional challenges from its inception in 2012. Among the first was managing

expectations among the Syrian and Jordanian populations, particularly regarding the

provision of water to refugees in view of a population that had to pay for the resource. Other

challenges included Jordan’s economic crisis in 2012, managing perception of the refugee

population, and making do with financial resources that fell short compared to need

throughout the programme. UNICEF dealt with these obstacles while implementing a

complex WASH programme and serving as a conscientious steward of its donors’ resources

and interests.

Purpose of the Evaluation The purpose of this evaluation was to assess the degree to which UNICEF’s WASH

programme had achieved its intended results from its beginning in July 2012 through July

2017. The audience for the evaluation is current programme stakeholders, and future WASH

programme implementers that may benefit from its lessons learned. The evaluation’s

objective was to independently assess the degree to which the WASH programme’s design

and implementation was relevant, effective, efficient, and will lead to some definition of

programme sustainability. The evaluation also sought to discover if the programme had

covered all relevant populations, especially the most vulnerable, and the degree to which the

programme coordinated well with other similar initiatives and government strategy. UNICEF

excluded the impact criterion from the evaluation’s scope, stating that “due to the relatively

short time period since the start of the interventions.3

This evaluation covers WASH programme activities from July 2012 to July 2017.

Geographically, it covers programming in four camps (Za’atari, Azraq, Cyber City, and King

2 Results frameworks from 2013 – 2017 are included as Annex H

3 Evaluation Terms of Reference – page 4.

3

Abdullah Park), host communities, informal tented settlements, and Rukban and Hadalat,

the settlements on the Syrian/Jordanian border. It also covers refugees and Jordanians living

in host communities with high numbers of refugees.

Methodology ISG conducted a summative evaluation of UNICEF’s WASH programme. As described in

the evaluation’s terms of reference, the team used a mixed method approach that included the

review of UNICEF’s data and documentation, and the collection of qualitative data through

interviews and Focus Group Discussions (FGD). In addition, the team reviewed WASH

programme initiatives through on-site assessments.

The document review established the programme’s implied logic model and delineated its

goals and objectives. The review was also used to create interview and focus group guides.

As described in the evaluation’s ToR, quantitative analysis relied on a review of the

programme’s documents and other secondary sources.

Primary, qualitative data collection was carried out through interviews and focus groups with

stakeholders, beneficiaries, programme partners, UNICEF staff, donors, government officials,

and other key informants. The evaluation team used a rapid assessment methodology to

evaluate specific WASH programme interventions intended to improve water and sanitation

for targeted populations. The evaluation team interviewed UNICEF partner organizations that

were involved in the construction or rehabilitation of WASH facilities, the provision of

WASH services, and promotion of hygiene initiatives. The purpose of these interviews was to

understand partner perspectives on the degree to which interventions met evaluation criteria

and to understand ways in which management of the programme excelled and would benefit

from improvement4.

Findings The full evaluation report organizes findings according to the Organization for Economic

Cooperation and Development – Development Assistance Committee (OECD-DAC) criteria

included in the evaluation’s ToR, which include relevance, efficiency, effectiveness, and

sustainability as well as relevant humanitarian criteria. Findings are extensive as they cover

five years of programme operations across a variety of geographic contexts and the range of

programme activities. Here we provide a summary of the important findings.

Relevance: In camps and settlements interventions were relevant and appropriate in terms of

meeting rapidly changing WASH needs of targeted populations under difficult circumstances.

The programme’s context was particularly difficult in its early phase. UNICEF maintained

programme relevance during the winter storms, outbreaks of illness, and other obstacles to

smooth management (such as trucker strikes) that arose throughout the 2012-2017 period.

During 2012 and 2013, UNICEF determined needs by estimating the quantity of water,

sanitation services capacity, and numbers of facilities that people displaced by the Syrian

civil war required at transit centers and in Za’atari, King Abdullah Park and Cyber City. In

this period UNICEF designed and implemented water and sanitation service options and level

of services based on its analysis of the dynamic emergency response context, opinions of

partners and other stakeholders, and the resources it had available. Beginning in 2014,

UNICEF used an assessment process to maintain the relevance of its interventions. Examples

of these assessments include a multi sector assessment of ITSs conducted by REACH in

2014, and assessments in the Za’atari and Azraq camps in 2015 and 2017. Other means of

4 A list of partners that the team interviewed is included in Annex B.

4

maintaining relevance were implemented throughout the programme and are discussed in the

full report.

UNICEF has made crucial improvements to water and wastewater infrastructure in host

communities and informal tented settlements. However, it is difficult to judge the relevance

of these interventions as many other international organizations provide services in these

communities and the degree to which the UNICEF programme addresses needs generated by

the refugee crisis is unclear.

Effectiveness: In the provision of equitable WASH services and dissemination of WASH

messages in the four camps, the WASH programme achieved high levels of effectiveness for

the targeted population across all years. The programme also very effectively supported the

WASH sector through technical support to the MoWI, as well as National WASH in School

Standards. Government officials at the MoWI and WAJ reported satisfaction with the WASH

programme’s contribution to planning documents, strategies, and policies. UNICEF’s

achievement included successful coordination of the WASH sector from the camp to the

national level. The programme met its targets under each of these activity areas. The WASH

Programme struggled to meet its targets in its activities focused on supporting host

communities, including WASH in schools.

Efficiency: The WASH programme worked continuously to improve the efficiency of its

operations. As an example, the evaluation team estimates that between 2013 and 2016, the

cost of delivering water to refugees in the camps dropped from 3.34 Jod/m3 to 0.24 Jod/m3.5

The programme made similar efficiency gains in sanitation services and solid waste

collection. Other Innovations led to more efficient coverage and management of services,

such as introducing a voucher system for contractors performing desludging that made them

accountable to camp residents and reduced potential corruption.

Sustainability: UNICEF has done everything possible to ensure that its interventions are

long-lasting and that mechanisms are in place for their operation. Examples include

organizing WASH committees to oversee water access and hygiene promotion, constructing

systems out of durable materials, and implementing messaging campaigns for hygiene

promotion and water conservation. However, the ultimate sustainability of the programme is

a complex issue. It includes determining if and when UNICEF will hand WASH

management off to another entity, managing the transition of refugees either back to their

place of origin or a permanent settlement elsewhere, the dismantling or integration of camp

WASH infrastructure into Jordan’s municipal systems, and dealing with Jordan’s medium-

term water scarcity issues.

Coverage: The degree to which the programme is focused on the most vulnerable people is

influenced by the WASH programme initial mandate, which is to maintain WASH service

levels and standards in the camps. UNICEF maintains initiatives that monitor coverage in the

camps and works with partners to ensure vulnerable populations receive necessary services.

These initiatives work specifically to monitor services to people with disabilities, identify

people who meet UNICEF’s vulnerability criteria, and ensure that interventions target

vulnerable people and households.

While people entering the camps may have been the most vulnerable at the programme’s

outset, the evaluation team believes that in later years occupants of ITSs were more

vulnerable and received less attention from the WASH programme. However, the necessity

5 Figures based on estimates for the cost of water delivery included in partner organization budgets from 2012 –

2017.

5

of maintaining service in the camps combined with budgetary shortfalls has prevented

UNICEF from identifying and reaching that population extensively.

Coordination: UNICEF greatly minimized duplication of services and service gaps in

camps. Its role as WASH sector lead allowed it to assign sectors of Za’atari to the

responsibility of partner organizations and to set standards among those organizations. It

created similar standards and management arrangements in other camps. As sector lead,

UNICEF implemented mechanisms for coordination in the face of emergencies, such as

winter storms, and established third party monitoring systems that allow for quality and

security incident reporting.

UNICEF partners and GoJ officials that the evaluation team interviewed stated satisfaction

with UNICEF as sector lead. One partner said that, “UNICEF … always (has) staff in

Azraq.” A donor said that they were very happy with UNICEF’s coordination and

partnership. A staff member of one partner said, “UNICEF does a very good job of

controlling the work.”

Lessons Learned and Recommendations The full report includes sixteen lessons for the reader’s consideration. Most important among

them are the following:

Quick decision making, and decisive action saves lives. The decisions and actions UNICEF

took in the early days of the programme were crucial for initial inhabitants of Za’atari. In the

settlements, UNICEF provided water and sanitation services to respond to the settlements’

exigencies. UNICEF moved quickly and effectively, with clear goals, such as ensuring that

each person had at least 15 liters of water per day. The WASH programme also recognized

that its intervention would have an impact on the larger Ruwaished host community. To

address that impact, UNICEF rehabilitated the community’s water and sanitation facilities

and provided support to identified vulnerable HHs.

Installing quality water treatment and delivery equipment is crucial for the long-term

sustainability and use. UNICEF’s installed higher cost and higher quality water infrastructure

at Za’atari, Rukban and Azraq, including the treatment plant for Zaatari. In each case, a

robust design was followed by very good installation and excellent post construction

operation and maintenance. This was borne out through the evaluation team’s on-site

investigations. In a key-informant interview (KII) with the contractor in Rukban, the

operators noted that UNICEF always purchases high quality equipment. The result is

operational efficiency, equipment that is long-lasting in the harsh environments, and reduced

maintenance costs, downtime, and interruptions to supply.

When direct consultation is not possible, other means of assessment can ensure programme

relevance. In addition to direct consultation with beneficiaries, UNICEF put a system of

assessments and Knowledge, Attitudes, and Practices (KAP) surveys in place. These studies

kept UNICEF staff and sector participants informed of changes in the camps and other

locations and assisted the programme in staying relevant and responding quickly to

eventualities.

Based on the documentation that was shared with the evaluation team as well as its

interviews with UNICEF staff, partners, and other stakeholder, the programme appears to

lack analytical rigor for determining the programme’s objectives, indicators or resource

allocation, which has led it to overcommit to initiatives for which it did not have resources or

capacity, primarily in host communities. The WASH programme should implement a

monitoring and evaluation system, and comply with the strategic programme development

protocols and reviews that are detailed in the Jordan Country Programme 2013-2017.

6

The WASH Programme and Context The object of this evaluation is UNICEF’s response to the Water, Sanitation and Hygiene

(WASH) needs in Jordan as a result of the Syrian refugee crisis (July 2012 to July 2017).6

The response took the form of a programme of activities that began as an effort to provide

WASH facilities and services to refugees entering Jordan as a result of the Syrian civil war.

At its peak, the programme served refugees in six camps and settlements, host communities

and informal tented settlements (ITS) across Jordan, and national government ministries. The

programme’s main activities included:

• Provision of water, sanitation, and hygiene facilities and services in the camps and

settlements the programme serves.

• Promulgation of hygiene and water conservation messages to refugees and host

community members.

• Provision of WASH infrastructure support in host communities to benefit host

community members, refugees, and inhabitants of ITSs.

• Promotion of Wash in School (WIS) improvement, including infrastructure support, a

nationwide assessment, message promulgation, and capacity building for government

ministries to better support WIS.

• Delivery of technical support to the Ministry of Water and Irrigation (MoWI) to

develop strategies and accompanying documents, and leading sector planning.

• Coordination of the WASH sector in camps and at the national level.

Description of the WASH programme

Programme History UNICEF Jordan launched its WASH programme in Jordan in response to the influx of Syrian

refugees during the first half of 20127. Prior to the programme’s launch in July 2012,

UNICEF provided WASH services at Jordanian transit centers as an activity under

UNICEF’s health programme8. The centers, which acted as Syrian refugees’ entry point into

Jordan, predated the refugee camps UNICEF served as part of the WASH programme. Syrian

refuges at transit centers that could prove that they had a sponsor in a Jordanian host

community stayed at a transit center for one week, and then were permitted into Jordan.

UNICEF’s WASH activities at transit centers included providing toilets, water tanks, water

trucking to fill the tanks, and sanitation services to these centers9.

As the number of refugees at transit centers increased, the Government of Jordan, the United

Nations High Commissioner for Refugees (UNHCR), and the Jordanian Hashemite Charity


7 UNICEF did not participate in the WaSH sector before the refugee crisis (Source: Evaluation Terms of

Reference page 4.)

8 Evaluation Terms of Reference, page 5

9 Source: Interview with UNICEF Staff- 26 July 2018

besmith

Sticky Note

7

Organization (JHCO) obtained a tract of land owned by Jordanian’s armed forces to design

and build the Za’atari refugee camp (Ledwith, 2014).

UNHCR and its partner agencies cooperated to determine how to delegate the work for

managing the crisis and the camp10. In the delegation of responsibilities, UNHCR asked

UNICEF Jordan to take responsibility for leading the WASH sector. UNICEF was quick to

rally expertise and funding for WASH programme management11.

UNICEF’s primary role as WASH sector lead in July 2012, the start date of the scope of this

evaluation, was to coordinate WASH activities for the Za’atari refugee camp as well as the

Cyber City and King Abdullah Park camps, which by January 2013 housed approximately

1,500 people (UNICEF Jordan, 2013). Coordinating WASH during this phase was an

expensive endeavor, further complicated by the Government of Jordan’s moratorium on

building permanent structures in Za’atari. The response in this phase was expensive because

UNICEF had to rely on contractors to provide water and mobile sanitation facilities quickly

during the fast-moving crisis. UNICEF’s WASH team had five days to prepare for the

opening of Za’atari camp12. The WASH team’s first objectives were to provide contingency

supplies, supply water, acquire mobile toilets, and engage a contractor to provide desludging

services.13

The WASH team also sought to engage partner NGOs to undertake WASH infrastructure

construction, operations, and hygiene promotion in the camp. In Jordan, two organizations

were available and had the necessary resources on hand to serve as partner organizations.

UNICEF signed an agreement with one of the partners. However, that partner backed out of

the agreement two days before the camp opened, forcing UNICEF to request that the

remaining partner, the German Federal Agency for Technical Relief (THW) mobilize

quickly. THW mobilized in 24 hours and was ready to serve the camp when it opened14.

By September of 2012, the pace of new arrivals slowed from 100 per day to 50 per day15.

UNICEF advocated with the Water Authority of Jordan to transition to more efficient means

of serving the refugee population living in Za’atari. This advocacy resulted in UNICEF

building WASH blocks to serve refugees living in Za’atari. WASH block construction started

in the part of the camp where the first arrivals lived and moved to the newer areas. UNICEF

also transitioned sanitary facilities away from mobile toilets to sealed pits and steel tanks

made for sewage collection, commissioned Mercy Corps to drill and install two boreholes in

the camp, and partnered with Japan Emergency NGO (JEN) and Relief International (RI) to

provide hygiene promotion. In late 2012, UNICEF began serving host communities, starting

with water system construction in Ramtha and Mafraq (UNICEF Jordan, 2012).

10 The evaluation team did not have access to programmatic documentation, such as proposals from UNICEF or

agreements with donor agencies, that would have described why UNICEF was awarded responsibility for

WaSH activities, or what UNICEF proposed. The presented information comes from evaluation interviews.

11 Source: UNICEF Staff Interview – 26 July 2018

12 Ibid.

13 Ibid

14 ibid

15 ibid

8

As it became apparent at the end of 2012 and beginning of 2013 that the refugee crisis would

not soon subside, UNICEF Jordan developed a results framework that included output level

WASH results for its 2013 and 2014 commitments. These included ensuring that relevant

core commitments on WASH were met for Syrian refugees and affected Jordanians in all

humanitarian settings (Output 1.3)16. The core commitments provided for:

• Establishing a working group to coordinate the WASH sector,

• Providing safe access to water for drinking and domestic use,

• Improving sanitation facilities,

• Providing hygiene items for those that needed them,

• Promoting hygiene messages,

• Providing for WASH services in schools, and,

• Working with the MoWI and sector actors to address the impact of the influx of

refugees on the sector’s longer-term chronic needs.

UNICEF’s targeted WASH results changed between 2014 and 2015 as the programme

aligned its objectives with government policies and sought to run its WASH operations more

efficiently. In Za’atari UNICEF found that camp inhabitants preferred private WASH

facilities to community WASH blocks and converted WASH infrastructure to meet that

demand, resulting in the phasing out of 400 WASH blocks and working with the community

to set standards for private facilities17. In 2014, a new camp, Azraq, was completed and

UNICEF led its WASH activities as well. By 2016, UNICEF’s targeted results included

working with the national government to finalize and disseminate the new policies, conduct

research on water conservation, create standards for WASH in schools, promote water

conservation and hygiene among the most vulnerable populations in host communities,

camps, ITSs, and to provide for the WASH needs of vulnerable women and children through

contracted tankering and also water networks in the camps and host communities.

WASH programme Purpose, Components, and Management The WASH programme’s purpose evolved as the programme’s context changed from

emergency response to longer-term operations. From 2012 to early 2014, the programme’s

purpose was to provide for the immediate WASH needs of refugees arriving from Syria, and

to respond to contingencies, such as the tensions that developed in host communities, as they

arose18. In 2014 UNICEF began the process of planning for more efficient, longer-term

operations in refugee camps. An example of an initiative developed through this process is

the construction of a water network in Za’atari camp, which reduced the cost of water

provision, improve the sustainability of the system, and make access to water more

equitable19. In 2014 UNICEF also contributed to the Jordan Response Plan (JRP) 2015,

which sought to “bridge the divide between resilience and humanitarian systems,” (Ministry

of Planning and International Cooperation, 2014). UNICEF’s WASH programme coordinated

with the JRP by formally planning for host community interventions, leading on the refugee

response and supporting the government on the resilience component. The UNICEF WASH

programme also aligned itself with the GoJ National Water Strategy 2016 - 2025, assisting

16 UNICEF’s WaSH results framework is included as Annex H.

17 Source: UNICEF Staff Interview – 26 July 2018

18 Source: UNICEF Staff Interview – 6 August 2018

19 Additional examples are discussed in the Efficiency section of the Findings chapter of this report.

9

the government to meet its objectives regarding efficiency and the National WASH in

Schools Assessment (Ministry of Water and Irrigation, 2016).

By 2017, the WASH programme divided its activities into four activity components:

1. Provision of equitable WASH services and the dissemination of WASH messages in

the four camps (Za’atari, Azraq, King Abdullah Park and Cyber City) and two

settlements in Rukban and Hadalat, to the most vulnerable people, namely children

and women.

2. Provision of gender equitable support to the most vulnerable people in host

communities (in all 12 Governorates) namely to children and women and including

ITSs.

3. Support to the WASH sector through technical support to the Ministry of Water and

Irrigation.

4. Coordination of the sector at camps and national level.

As July 2017, the end date for the scope of this evaluation, UNICEF continued to implement

the WASH programme along the four components described above.

The WASH programme’s budget was approximately $355 million USD from 2013 to 2017. It

averaged approximately $71 million USD per year.

WASH Programme Logic Model and Results Framework The WASH programme has operated without an explicit strategy document or logic model to

guide its implementation. The evaluation team also found no evidence of a programme

document that would have detailed donor expectations. However, each phase of the

programme demonstrated an implicit logic model that has evolved20.

In the first phase of the programme, July through December 2012, the evaluation team found

no evidence of a planned structure to the programme, such as a results framework or

workplan21. The first results framework set objectives, outcomes, and output indicators for

the years 2013 and 2014. Outcome level indicators in this framework were not entirely

focused on WASH goals. The WASH programme still used a general UNICEF outcome

framework that included indicators such as “existence of a comprehensive national

monitoring system for school readiness to inform policy decisions on child disparities,” and

“existence of a child and neonatal information system in support of child and maternal death

audits.” The 2013/2014 output indicators seek to measure results from general activities and

do not disaggregate achievement by targeted population. For example, they measure the

cumulative “number of people provided with safe access to sufficient water for drinking and

domestic use.” An output indicator from the 2013/2014 framework states “Relevant Core

Commitments on WASH are met in all humanitarian settings.”

In 2015, the WASH programme set the objective level indicators that it used at least through

December 2017. In this framework, outcome indicators disaggregated by target population

are specific to the WASH programme, such as “Proportion of population at camps and host

communities including schools using improved drinking water as per Jordan standards.”

Output targets and indicators are also adapted to the WASH sector. Examples include,

“children, parents and community members are aware of and practice hygiene and water

conservation,” and, “Number refugees provided with safe access to sufficient water for

drinking and domestic use through water trucking.” While those indicators are not

20 The WaSH programme’s results frameworks from years 2012 to 2017 are attached as Annex H.

21 After the conclusion of the evaluation, UNICEF provided a workplan for the 2016/2017 period.

10

disaggregated by target population, the baseline data collected in support of them was, as was

the end of year reporting that the WASH programme completed.

In 2016, the results framework kept the same outcome level indicators, and evolved to

provide an indicator for each population that the programme targeted. Examples include

“Number of people in camps supplied with water (tankering or network),” and, “Number of

people with access to improved WASH facilities in institutions.”

By the most recent results framework that falls within the scope of this evaluation, which

covers the period from 2015 to 2017, an implicit logic model had evolved. The WASH

programme’s documentation implies a logic model that seeks to provide an effective response

to the refugee crisis while protecting Jordan’s peace, prosperity, and resources. The

programme has sought to achieve that objective by attempting to either implement or

coordinate WASH activities in four areas:

• Building capacity at the national level to equitably provide WASH services,

• Promoting hygiene and water conservation practices in camps, host communities,

schools, and ITSs,

• Providing WASH services directly for vulnerable people, and,

• Building the capacity of institutions, utilities, and other organizations to provide

equitable access to water and sanitation services.

The WASH programme states its objective, its highest-level target, as follows22: “Quality

WASH facilities are sustained and utilised, and hygienic behaviours practiced, by the most

vulnerable while ensuring the protection of the environment.”

The programme measures the achievement of this objective through four outcome indicators.

• Indicator 1: Existence of strategic planning capacity in MoWI and sector and other

relevant institutions in Emergency preparedness and response.

• Indicator 2: Proportion of population at camps and host communities including

schools using improved drinking water as per Jordan standards.

• Indicator 3: Proportion of population at camps and host communities including

schools using improved sanitation facilities.

• Indicator 4: Proportion of population practicing hygiene promotion at targeted camps,

host communities and schools.

Each outcome indicator’s achievement is in turn defined by output-level indicators. The

implied logic model is illustrated in figure 1.

22 Results frameworks from 2013 – 2017 are included as Annex H

11

Figure 1 WASH Programme Implied Logic Model

:

12

The programme’s outcome statement and output statements, as articulated in the above logic

model, come from UNICEF Jordan’s 2016 annual report (UNICEF Jordan, 2016)23. The

activities reflect the reported activities and accomplishments of the WASH programme. The

results framework reflects a broad measurement of achievement under three of the four

programme components; briefly, 1) support to the camps and settlements,24 2) support to host

communities and ITSs, and 3) support to the WASH sector through technical support to the

MoWI. The results framework does not include a direct measurement of the fourth

component - Coordination of the humanitarian WASH sector at a camp and national level.

However, the degree of the programme’s achievement under the coordination component is

closely related to the degree of achievement under the other three components.

Programme Context The WASH programme faced difficult social, political, and institutional challenges from its

inception in 2012.

Managing an influx of refugees from a relatively water-sufficient area

There is a large disparity in access to water resources among countries in the Middle East.

Regionally, the average per capita total renewable water resources is 645 m3, however, the

median country, Palestine, has 179.3 m3 for each of its residents25. Jordan and Syria tend

toward opposite ends of the water access spectrum. Jordan has access to 123.4 m3 per capita,

ranking the country below the median and well below the average for the region. Syria has

greater than average access to renewable water resources, amounting to 908 m3 per capita26.

The difference in access to resources sets the scene for the challenges that UNICEF would

face in supplying a population used to ample water resources who had relocated to a region

with far less water (UNICEF, 2012). Jordan’s largest refugee camp is the Za’atari camp

located in Mafraq governorate. In 2013, 90 percent of the population of the Za’atari refugee

originated in Daraa Syria. Daraa is a well-irrigated agricultural region, sourcing water from

groundwater (wells), or the ample (but steadily declining) lake Muzayrib (Leestma, 2017),

(Salman & Mualla, 2008) People from Daraa fled the Syrian civil war and began relocating

to Za’atari camp in the Mafraq governorate in 2012. UNICEF’s challenging task was to meet

the daily water needs of the refugee population, which fell short of the quantity of water

refugees perceived that they needed, and balancing that against local Jordanian’s perspective

that their scarce resources were being extracted and given away (REACH, 2014).

Za’atari is located in the Mafraq governorate and has grown to be the 4th largest urban

settlement in Jordan. In Mafraq, like much of Jordan, nearly all households are covered by a

piped water network. However, water is only supplied one or two days a week into household

storage containers. While the frequency of delivery is common for urban areas in the region,

water is delivered less frequently to Mafraq than Za’atari. When that water is consumed,

residents must purchase water privately. Jordanian’s living in Mafraq had stretched declining

resources before the new influx of refugees increased water demand by 21 percent

23 The annual report can be found at the link below. Note that while the report lists 5 output statements, the

activities under output 1 and output 4 are nearly identical, and the output statements are quite similar:

https://www.unicef.org/about/annualreport/files/Jordan_2016_COAR.pdf

24 Settlements such as Rukban are not explicitly mentioned in the results framework.

25 FAO Aquastat: http://www.fao.org/nr/water/aquastat/data/query/results.html

26 Ibid.

13

(International Monetary Fund, 2017). By mid 2013, tensions over those resources would lead

to local animosity and protest (Wildman, 2013).

Pre-existing political attitudes toward refugees Within two years of Jordan’s independence in 1946, the Kingdom had to dedicate resources

to host the approximately 100,000 refugees fleeing the 1948 war that established Israel as a

nation. In 1967, war broke out again and 140,000 additional people fled into Jordan (Bocco,

2010). Over the subsequent decades, further inflows of refugees included Iraqi refugees in the

1970s, 80s, and 90s; 300,000 Jordanian expatriates returning unexpectedly in the 90s, and

with the second war in Iraq in the early 2000’s approximately 750,000 returnees and refugees

(Al Wazani, 2014).

As a result of previous experiences hosting refugees, the GoJ has made decisions that

influence the current refugee crisis and particularly the WASH programme. These decisions

are discussed throughout this document. Examples of decisions that influenced the program

were the GoJ’s initial decision to not allow the construction of permanent structures in

Za’atari refugee camp, and the GoJ’s steadfast decision that they will not take ownership over

the refugee camps, including the water network and supply of WASH materials and services.

Legal and Institutional Framework of the WASH sector in Jordan One benefit of having low water usage per capita is that Jordan has achieved nearly universal

coverage in water services (International Monetary Fund, 2017). Achieving strategic results

such as universal coverage is a result of the water sector’s institutional and policy framework.

The central institution responsible for the water management in Jordan is the Ministry of

Water and Irrigation (MoWI). Under the MoWI, there are 15 units that manage

administrative and technical functions. Included among these units are two agencies that deal

with water issues and policies in Jordan and report directly to the Minister; The Water

Authority of Jordan (WAJ) and the Jordan Valley Authority (JVA). The WAJ manages

municipal water supply and wastewater services. It also manages operations, maintenance,

planning and construction. Municipal water utilities fall under WAJ and are responsible for

managing subscription issues and their own facilities. The JVA is responsible for the “social

and economic development of the Jordan River Valley” including the development and

protection of water resources (Kis, 2016).

Several other institutions are important to Jordan’s water management. There are three water

companies, The Yarmouk Water Company, The Aqaba Water Company, and the Miyahuna

Water Company. Other ministries that play a role in water management are the Ministry of

Agriculture and the Ministry of the Environment.

The Ministry of Health regulates drinking water quality in Jordan. It monitors sources for

water quality and inspects potential sources of pollution. The Ministry of Health issues

permits for all produced or imported potable water.

Economic Tension In 2012, as Syrian refugees were entering into Jordan and the newly constructed Za’atari

camp. initially, patient Jordanians watched as free water and other resources, carried by a

large number of trucks flowed into the camp (UNICEF, 2012).

Meanwhile, Jordan was confronting an economic crisis. In March of that year, the

International Monetary Fund (IMF) determined that the country’s debt levels were

dangerously high given its economic turndown as a result of the 2008 global financial crisis

(International Monetary Fund, 2017). The IMF recommended a series of measures meant to

14

stabilize the economy, including cutting subsidies for fuel.27 By November, protests erupted

in Amman and Irbid.28 The image of Syrians receiving daily water deliveries extracted from

depleted Jordanian resources while poor Jordanians were made even poorer exacerbated

tensions in camps and in communities (Luck, 2013).

Funding Environment In 2018, international donors have only provided approximately 43 percent (Financial

Tracking Service, 2018) of what is required to respond to the regional Syria Refugee Crisis,

and about 34 percent of the cost of providing a WASH sector response (Financial Tracking

Service, 2018). In 2017 donors funded 52 percent of the requirements of the regional

response plan for the Syrian Crisis (ReliefWeb, 2017). The international community’s

difficulty in coming up with sufficient funds to sustain an effective response to the crisis has

created a heavy burden for UNICEF’s WASH programme in Jordan. The lack of funding has

particularly reduced UNICEF’s ability to serve host communities and ITSs (UNICEF Jordan,

2013)29. Figure 2 below shows the amount UNICEF budgeted for interventions across

programs for the Jordan Response Plan against the amount that was raised and spent by

year30.

Figure 2 Funds budgeted vs mobilised by year (USD in millions)31

27 https://www.imf.org/external/pubs/ft/scr/2012/cr12119.pdf 28 https://uk.reuters.com/article/uk-jordan-protests-idUKBRE8AD00F20121114 29 Lack of funding was also the reason given for missing host community targets even in 2016, when UNICEF

Jordan exceeded its fund raising targets. 30 UNICEF Jordan notes that the table includes rolled over funding and earmarked amounts totaling $30 million

from German donors for Za’atari. 31 The 2015 Annual report included funds mobilized through November. The chart adds an additional 1/12 to

funds mobilized to estimate funds utilized in December.

0

50

100

150

200

250

300

2013 2014 2015 2016 2017

Budget Mobilised

15

Description of the Evaluation

Purpose and Objective The purpose of this evaluation was to assess the degree to which UNICEF’s WASH

programme had achieved its intended results from its beginning in July 2012 through July

2017. UNICEF intends to use the lessons that the evaluation process generates to inform

future WASH programmes that occur in a similar context.

The Evaluation’s objective was to independently assess the degree to which the WASH

programme’s design and implementation were relevant, effective, efficient, and will lead to

some definition of programme sustainability. The evaluation also sought to discover if the

programme had covered all relevant populations, especially the most vulnerable, and the

degree to which the programme coordinated well with other similar initiatives and

government strategy. UNICEF excluded the impact criterion from the evaluation’s scope,

“due to the relatively short time period since the start of the interventions.32”

The evaluation questions are included in the findings section of this report, and in Annex A:

Evaluation Terms of Reference.

Scope This evaluation covers UNICEF’s WASH programme from July 2012 to July 2017. This

evaluation assessed WASH programme activities, including water provision, sanitation

facilities and services, hygiene promotion, solid waste management, desludging, WASH in

Schools initiatives, WASH sector coordination, and support to the Government of Jordan

through the Ministry of Water and Irrigation (MoWI). Geographically, it covers

programming in four camps (Za’atari, Azraq, Cyber City, and King Abdullah Park (KAP),

host communities, informal tented settlements, and Rukban and Hadalat, the settlements on

the Syrian/Jordanian border. It also covers refugees and Jordanians living in host

communities with high numbers of refugees.

Evaluation Methodology ISG conducted a summative evaluation of UNICEF’s WASH programme. The team used a

mixed method approach that included the review of UNICEF’s qualitative and quantitative

data, and the collection of qualitative data through interviews and Focus Group Discussions

(FGD). In addition, the team reviewed WASH programme initiatives through on-site

assessments. The Evaluation Matrix, included as Annex G, provides the evaluation questions,

detailed answers to those questions, the key judgement criteria that was used in constructing

the evaluation team’s response to evaluation questions. It also includes the sources of

information that provided evidence for the response to each question. The matrix is

organized according to the evaluation questions in the evaluation’s ToR and by evaluation

factors, namely; relevance, effectiveness, efficiency, sustainability, coverage, and

coordination.

Desk Review The desk review established the programme’s implied logic model and delineated its goals

and objectives. UNICEF provided the evaluation team with an initial set of documents at the

start of the evaluation. The evaluation team reviewed these documents to understand how the

WASH programme evolved as the refugee crisis developed, estimate the WASH

32 Evaluation Terms of Reference – page 4.

16

programme’s achievements related to goals and objectives, understand how programme

management determined strategies, set targets, designed work, and the degree to which this

planning led to successful outcomes. For the inception phase of the evaluation, the desk

review was also used to create interview and focus group guides. The evaluation team also

noted where information was incomplete or missing and provided requests to UNICEF for

additional documentation to fill in information gaps over the course of the evaluation.

Document collection and review occurred throughout all phases of the evaluation. The list of

documents that the evaluation team reviewed is included in Annex D, with additional

documents listed in the Works Cited section.

Quantitative Analysis As described in the evaluation’s ToR, quantitative analysis relied on a review of the

programme’s documents and other secondary sources. The review identified the

programme’s goals, objectives, and output targets for each year of implementation and

sought to understand the degree to which targets were met. The indicator framework that

informed this analysis is included as Annex H to this report33.

The evaluation team also used expenditure reports to attempt to understand the programme’s

financial management and generate analysis for the questions related to efficiency in the

evaluation’s TOR. UNICEF’s financial information provided to the evaluation team was

incomplete and not well documented. UNICEF does not track expenditure by year,

programme component, or result. It also does not track indirect costs associated with the

programme’s implementation. These limitations prevented a detailed analysis of efficiency.

The evaluation team estimated costs from an incomplete list of the WASH programme’s

expenditures and estimates included in partner organization’s budgets. Documents that

included quantitative data are also included in Annex D.

Primary Research: Qualitative Data Collection Methods and

Sampling Qualitative data collection was carried out through interviews with stakeholders,

beneficiaries, programme partners, UNICEF staff, donors, government officials, and other

key informants. The evaluation team also conducted FGDs among programme beneficiaries.

The goal of interviews and FGDs was to capture information representing the range of

experiences among each targeted population group, with a focus on the most vulnerable.

Thus, for interviews, the evaluation team utilized a purposive sampling strategy in camps,

host communities and ITSs. The number of interviews for each targeted population sought to

fully answer each research question to the point that each source of information is saturated.

The number of interviews location and type is included in Annex B34. Interview and focus-

group guides are included in Annex E.

Coverage in Interviews and Focus Groups The evaluation team sought to ensure that the views of all different groups covered by the

WASH programme were represented through interviews and FGDs. Interviews and focus

groups included women and men, people living near resources and more distant, people new

33 Annex H details information extracted from UNICEF’s annual workplan and End of Year Reports.

34 The number of interviews assumes that the team is able to access each camp and host community and that

appropriate respondents are available and willing to participate in interviews.

17

to their areas of inhabitation and people that have lived in the area for a long time. However,

the numbers of people interviewed does not represent all groups equally across topic areas.

Rapid Assessments and User Interviews The WASH programme carried out rehabilitation and new construction of WASH

infrastructure, and organized WASH services to improve water access and sanitation. The

evaluation team used a rapid assessment methodology to evaluate specific projects intended

to improve water and sanitation. Infrastructure evaluations included facility inspections and

interviews with key informants about the infrastructure, including how it was and is being

utilized. Through this approach the evaluation team developed a clear picture of the

construction quality and operational effectiveness of the infrastructure.

The team evaluated UNICEF interventions in two parts. The first part was the observation

component. In this component, the evaluator inspected the intervention and determined where

critical control points or bottlenecks are apparent, while at the same time observing how

people interact with the intervention. The second component comprised qualitative data

collection. During this component the evaluator interviewed people who had varying levels

of access to the intervention. The evaluator sought out people that should have had access to

the intervention but don’t to understand why access is limited.

WASH Messaging and Practices The evaluation assessed the WASH programme’s messaging and behaviour change activities

according to the evaluation criteria. The evaluation explored the degree to which messages

were absorbed and converted into new practices by reviewing the data provided for desk

review and analysing the information gathered in stakeholder interviews.

Partner Interviews The evaluation team interviewed UNICEF partner organizations that were involved in the

construction or rehabilitation of WASH facilities, the provision of WASH Services, and

promotion of hygiene initiatives. The purpose of these interviews was to understand partner

perspectives on the degree to which interventions met evaluation criteria and to understand

ways in which management of the programme excelled and would benefit from

improvement. A list of partners that the team interviewed is included in Annex B.

Duty Bearers and Staff The evaluation team coordinated with UNICEF Jordan to interview appropriate duty bearers

in government ministries, utilities companies, municipalities, and communities. These

interviews included, the MoWI, the Yarmouk Water Company, and the Water Authority of

Jordan. These duty bearers have assisted UNICEF in program delivery and/or benefited from

UNICEF’s capacity building and sector coordination activities. The evaluation team also

interviewed UNICEF administrative, programmatic, and senior staff.

Stakeholder Interviews Where applicable, the evaluation team interviewed stakeholders that had an overview of the

WASH programme’s implementation in their community. These stakeholders include WASH

committee members, camp or host community duty bearers, and other stakeholders that

UNICEF staff thought would offer insight into the programme.

Analysis Method Durng the desk review and after field visits, the ISG Team coded qualitative data to ensure

that themes matched evaluation criteria and that issues of gender, vulnerability, and

protection were highlighted. Coding the data facilitated identifying clear themes across

primary and secondary research.

18

Much of UNICEF’s quantitative data is maintained in unsearchable .pdf documents. To

extract as much of the data as possible, the evaluation team used optical character recognition

software to recover as much quantitative information as possible. This data was particularly

useful in answering questions related to efficiency and effectiveness.

Limitations to the Evaluation There are several factors that may limit the evaluation’s analysis. These are listed below

• Quality of data provided by UNICEF – The evaluation team’s ability to verify

indicator targets and select appropriate respondents for qualitative data collection

depends on the quality and organization of the WASH programme’s data. Populations

that have participated in the programme, but for which no records were kept, or

records are incomplete, may have received insufficient treatment by the evaluation.

These populations include host communities benefiting from infrastructure

improvements and residents of camps that have closed. Records for the early years of

the programme may have been incomplete and/or poorly organized.

• Financial and Procurement Data Provided by UNICEF – Efficiency and value for

money calculations depend on the reliability of the financial and procurement data

that UNICEF provides. UNICEF does not track financial information by result,

activity, programme component, or year. UNICEF also does not track indirect costs

allocable to a specific programme. This evaluation provides an analysis of the data

available to the evaluation team, but cannot audit its accuracy or reliability.

Evaluation findings regarding efficiency and value for money are estimated.

• Sample Selection – The qualitative data collection employed by this evaluation relied

on purposeful sampling rather than utilizing random sampling. While the qualitative

data collection creates a narrative around the evaluation’s criteria, it also suffer from

particular biases, including UNICEF’s ability to provide information about

appropriate project sites, time restrictions in data collection, and access to small

numbers of beneficiaries. Additionally, the evaluation was conducted during summer

months, when most schools were closed, which limited the evaluation team’s

collection of data related to Wash in Schools.

• Availability of Respondents – Many of the WASH programme’s activities were

carried out four or more years ago. Many interventions may have been carried out

without the knowledge of direct beneficiaries, especially those that involved general

infrastructure or were implemented through partners or contractors. The lack

availability of respondents limited the number of interviews and focus groups the

evaluation team carried out.

• Reliance UNICEF for guidance – The evaluation team relied on UNICEF staff for

access to stakeholders and guidance for examples of project successes and failures.

Even with the best of intentions this type of reliance may create biases in the final

report, which the evaluation team will identify and describe.

Ethical Principles in Conducting the Evaluation ISG’s work is based on four core principles:

1. Participatory evaluation design and implementation: To the extent possible, all

people who have a stake in the outcome of a study or evaluation must have a chance

to identify risks in conducting the study and opportunities to suggest ways to reduce

those risks. Following this principal, we solicit comments and input from our clients

on inception and design documents, as well as data collection and research tools.

When clients have created terms of reference, we review the terms for input from

stakeholders and ask for a representative stakeholder review when possible.

19

2. Respect the rights, privacy and dignity of evaluation stakeholders: ISG aims to

minimize risk in evaluation management and outcomes. The primary aim of our work

is to benefit the people who are most affected by its outcomes. We put the safety,

dignity, and privacy of those that participate in our projects above the rewards that we

hope to achieve for ourselves or our firm. As such, we ensure that survey, interview,

and focus group participants are fully informed of the nature and purpose of the

research that we are conducting, obtain their consent before asking any questions or

engaging them in any other research, and allow them the opportunity to deny or

remove consent at any point in the process. We do not use names or identifying

information in reports, except in specific circumstances and then only if the

participant is fully informed and in agreement. We minimize risk to participants,

including carefully designing questions that may recreate traumatic or harmful

feelings. Finally, ISG believes that participants in our work have the right to benefit

from it. We work with our clients to produce multiple versions of documents and

materials to facilitate the distribution of results.

3. Informed and reasonable judgements: The work that ISG conducts often influences

the distribution of resources and activities in vulnerable communities. We consult

with our clients to ensure that conclusions are drawn from rigorously vetted evidence,

and that following actions are based in reliable findings. ISG’s evaluators detail the

strengths and weaknesses of our methodology and the limitations of the study given

available resources and contextual barriers.

4. Secure Data Storage Protocol: ISG utilizes secure data protocols to ensure that

respondents’ information is not used in any way beyond that which they have

provided permission. At the end of each research study, evaluation, or project that

ISG concludes, project team members submit hard copies of records, including but

not limited to research notes, photographs, or portable recordings to ISG’s designated

project manager. The project manager ensures that notes, recordings, and all other

records are moved onto ISG’s secure servers and deleted off of laptops and other

portable storage devices. ISG stores our clients’ data, as well as data collected through

primary research on secure servers through which only the project team has access. If

requested, ISG destroys all data related to a project, including paper records, or

records kept through all other means. If not request is made, ISG’s director of

programs oversees the maintenance of data on the secure server in perpetuity, or for

the time period each client or agreement mandates.

Findings This section details the evaluation team’s findings. The section is generally organized by

evaluation criteria, question, and then geographic intervention target. The sections discuss

findings by programme year, where possible. A summary of the judgement criteria,

indicators, sources of documentary information, and data collection tools used to derive each

finding is included in Annex G: The Evaluation Matrix.

Relevance Evaluation questions 1 through 3 related to the programme’s relevance.

Evaluation Question 1 • To what extent were affected people consulted on their preferences with regards to

water and sanitation service options and level of services?

• Were interventions appropriate in terms of meeting their basic needs?

20

• Was there a feedback loop and monitoring system in place for reflecting the learning

and evidence to improve programming, especially for the humanitarian response?

The emergency response period (2012 – 2013)

UNICEF and its partners were assigned responsibility for providing WASH services in the

newly opened Za’atari refugee camp in July 2012. Construction of the camp began on July

20th and the camp was opened on July 30th (Wilkes, 2012). During those ten days, the WASH

programme had to create and implement a response plan for the 10,000 refugees that would

shortly arrive, and the approximately 100,0000 more that would come over the following two

years.

UNICEF and partner organization staff that the evaluation team interviewed recalled the

urgency of operations in that time period. One staff member recalled that once he was

notified, “we had five days to respond. All we had was some contingency supplies and orders

from Copenhagen.” A UNICEF partner team member explained. “At the beginning, it was an

emergency response. We were not worrying about vulnerable populations. I don’t think there

was a lot of consultation at the time35.”

UNICEF’s determined how to best provide for refugees’ ‘WASH needs through talks with

partners such as WAJ, Mercy Corps, ACTED, Oxfam, and Relief International, and based on

the assessments and information that they’d gathered. The tactics that they employed took

into consideration the Convention of the Rights of the Child (CRC), UNICEF’s Core

Commitments to Children in Emergencies, and the Convention on the Elimination of All

Forms of Discrimination against Women (CEDAW). Through these means, UNICEF created

estimates of the best way to provide WASH services for the newly arriving refugees.

UNICEF also consulted humanitarianresponse.info for guidance on issues such as

accountability and feedback mechanisms (WaSH Sector Working Group, 2012).

UNICEF utilized institutional resources in this phase. A regional WASH Specialist was

mobilised and a Regional WASH Advisor from South Asia was sent to Amman to coordinate

the response. Also, staff reported that experts from other offices assisted the effort as did

UNICEF headquarters. The evaluation team did not find documentary evidence of support

during this phase, and it is unclear which internal resources UNICEF consulted for guidance

on WASH response in an emergency situation. For example, UNICEF may have consulted

the Global WASH Cluster, an organization that UNICEF leads, published the “WASH

Cluster Coordination Handbook” in 2009. The handbook includes specific guidance and tools

for managing WASH in emergency situations, including a schedule of activity and

methodology for conducting assessments.

UNICEF’s first concern was providing a steady flow of water and adequate sanitation

facilities. Once minimum water and sanitation standards were met, UNICEF began to focus

on providing facilities for women and people with disabilities. A UNICEF staff member who

worked in Za’atari recalled that “At the beginning, water supply was the main focus. There

was no electricity, no toilet access for night, especially for women. There were no proper

toilets for people with disabilities at the beginning. Some NGOs tried to have a chair and a

small pit for people with disabilities36.”

35 World Vision Staff Interview – 30 July 2018

36 UNICEF Staff interview – 26 July 2018

21

To determine the needs of vulnerable people, including people with disabilities, UNICEF

relied on the advice of other organizations operating in Jordan and Sphere standards. For

example, in November of 2012, Handicap International provided the WASH sector with a

document titled. “Accessible WASH Facilities to promote inclusion of persons with

disabilities, injuries, and other vulnerabilities; Za’atari Refugee Camp.” The document details

standards for communication, entrances, fixtures, toilets, and showers. Facilities that met

these criteria were eventually installed, although it is unclear from the documentation when.

Other documents from the 2012-2013 period illustrate the criteria the WASH programme

used to determine service options and levels. The Jordan RRP5 Update in August 2013 notes

that WASH services were to be designed and delivered in camps with “households living in

temporary settlements, female headed households and the needs of children, disabled, and

elderly (The United Nations, 2013).” The next priority listed is “host communities with the

largest concentration of refugees." The degree to which these priorities were addressed in a

timely fashion is unclear from the documentation. However, the referenced documentation

from 2013, the earliest to which the evaluation team had access, shows that providing water

supply and basic sanitation was the first priority, followed by ensuring that it covered

vulnerable populations.

In the emergency response phase of the programme, hygiene was poor and waterborne

diseases were common. In 2013 World Vision, a WASH programme partner, estimated that

there were 110 cases of waterborne disease for every 10,000 people; a very high rate (World

Vision Programme Cooperation Agreement 19-13 ). UNICEF and its partners would address

this issue in the next phase of the programme.

Achievements and Results (2013 – 2015)

In the emergency response phase, the WASH programme had some remarkable

achievements. Principal among them, the programme supplied water and sanitation for all

inhabitants of the camp including those that arrived newly each day. The 2015 KAP survey

for Za’atari revealed UNICEF’s responsiveness to refugees needs regarding sanitation.

During the planning phase for the waste water network, 94 percent of residents were aware of

the planned network and 96 percent of those felt it would improve life in the camp (Za'atari

Hygiene Promotion Working Grouip, 2016). A clear indication that the WASH programme

was working in concert with Za’atari’s residents. These achievements are discussed in detail

in the effectiveness section of this report.

However, the unavoidable process of quick decision making and implementation that

occurred in 2012/2013 had some consequences. Locating Za’atari over one of Jordan’s major

aquifers led to expensive measures to protect wastewater leakage. The materials of the initial

septic tanks were not adequate to protect the groundwater, and so they had to be replaced.

The WASH blocks in Za’atari were difficult to maintain because they were prone to

vandalism and theft, and unpopular with the camp residents. As a consequence, camp

residents improvised private sanitation facilities, leading to pools of greywater throughout the

camp37. Working with local partners, UNICEF developed a plan to drain the greywater and

provide a sufficient drainage solution, including desludging on a daily basis. Eventually,

37 WaSH Sector Coordination; Minutes of weekly Za’atari camp WaSH sector coordination group. Sunday 24

November 2013

22

UNICEF followed the lead of camp residents and installed household sanitation facilities for

vulnerable households that did not already have facilities, leading to another round of

removing and installing environmentally effective septic tanks. The WASH blocks would be

decommissioned. Though recycling happened where possible, it was an expensive process38.

Another consequence of the need for quick decision making was that, in the initial emergency

response phase, the camp’s effect on Jordanians living near the camp and in other host

communities with high concentrations of refugees was not taken into consideration

adequately. As an official at the MoWI said, “The more high-quality services you provide

inside the camp, the more people outside the camp will care.” As described in the context

section of this report, Mafraq, where Za’atari camp is located is a poor area and per capita

receives less water than the refugees were accustomed to in their home of origin. In addition,

Mafraq residents pay for inconsistently available water, while refugees inside Za’atari

received regular shipments of free water. In October 2012, tensions between refugees and

Jordanians living in Mafraq were erupting (Seely, 2012). This led to engagement in host

community projects, discussed in the next section.

Stability and Efficiency (2014 – 2017)

In April 2014, the Azraq refugee camp was opened. The WASH programme used a number

of lessons learned from Za’atari in designing WASH services for Azraq. For example, water

supply and sanitation facilities were less centralized and located closer to smaller clusters of

households, household clusters were composed of people that came from similar areas of

Syria, cleaning was organized by cluster with no incentives for cleanliness provided,

community mobilization and hygiene promotion was initiated as soon as refugees moved into

the camp, refugees were more involved in the day to day operation of WASH facilities, and

WASH committees were set up earlier than in Za’atari camp (World Vision Programme

Cooperation Agreement 19-13 ).

However, the public toilets and showers in Azraq remained unpopular as in Za’atari. Focus

group discussions that the evaluation team held in Azraq indicated that, as in Za’atari camp,

women resist using public facilities due to lack of privacy.39 However, In 2017, a KAP study

conducted by Action Against Hunger indicated that 100 percent of survey respondents said

that they used WASH blocks in Azraq and felt safe (ACF, 2017).

In 2014, UNICEF began seeking other means to understand needs and preferences of

refugees living inside and outside of the refugee camps, and of Jordanians living in

communities that were hosting large numbers of refugees. In January 2014, a WASH Sector

Gender Analysis was published, to guide future programme implementation. Broad

assessments were also carried out to understand targeted populations’ needs and preferences,

with a focus on the most vulnerable populations. The first such assessment, undertaken in

August 2014, was a multi-sector assessment by REACH, an international initiative to develop

information tools. This assessment covered 125 ITSs and included a household survey as

well as semi-structured ad hoc interviews and focus group discussions. The assessment was

used to demonstrate how the most vulnerable in the ITSs could be engaged in defining their

38 UNICEF Staff Interview – 26 July 2018

39 Focus Group Discussion number 1– Azraq

23

needs (REACH, 2014). In addition, UNICEF conducted a number of Comprehensive Child

Focused Assessments in Azraq and Zaatari camps in June 2015. The process was repeated in

Azraq in February 2017. These assessments were also carried out in partnership with

REACH. During this evaluation’s focus groups and interviews, residents of Za’atari and

Azraq indicated that they are asked for their suggestions and opinions during awareness

raising sessions. Residents in Azraq said that at the beginning of their residency, they

participated in the process of improving the camp40.

After the emergency response phase, UNICEF worked with its partners to adapt WASH

facilities and services to the needs and behavior of camp inhabitants. These needs and

behaviors were identified through observation and through feedback and requests from

community members. For example, in Za’atari UNICEF observed residents creating private

facilities inside their homes rather than using the public

WASH blocks. Private bathrooms were provided in some

homes, while in may people obtained equipment and

installed their own bathrooms. In response, UNICEF

adapted its intervention to provide communal tanks for

households. Each tank served up to five households. This

adjustment better suited the customs and privacy needs

of the refugees. It also responded to the realization that

the camp’s residents would live in Za’atari for an

undetermined period rather than the initially planned for

six-month period41. In Azraq, UNICEF increased the

number of water tap stands and reduced the distance

from housing units. As of July 2017, water was

distributed from multiple points within the camp (tap

stands) in two shifts, from 7am to 11 am and again 2pm to 6 pm. It was then carried to

homes via containers and then stored and used over 24 hours or more. Usually these tap

stands were provided with 4 faucets, where each faucet served 5 to 6 neighborhoods. Each

neighborhood has 12 households with an average 5 individuals. Participants in this

evaluation’s focus groups stated that the increase in number water tap stands was an

important improvement. They said that when the number of tap stations was limited, access

to water was difficult and water was scarce. The process of getting water was not well

controlled as people had to arrange themselves in queues, which made the waiting time

longer and resulted in conflicts. These issues have been reduced since the number of taps has

increased. Interview respondents said that walking home with the heavy weight was

burdensome, though not as much as previously42.

Focus groups in Azraq also indicated lack of satisfaction with sanitation facilities. The

number of toilets is limited and insufficient43. They expressed their concern regarding the

40 Azraq FDGs 1 - 4

41 Source: Interviews and on-site assessments in Za’atari camp.

42 Source: Interviews and on-site assessments in Azraq camp.

43 UNICEF notes that there is 1 male toilet and shower and 1 female toilet and shower per 12 households. The

facilities are on opposite ends of a 30-meter plot.

Figure 3 Public Latrine at Azraq camp

24

ratio of users to toilets, uncleanliness of toilets, and cultural taboos regarding women using

public showers. They also explained that the public showers are not used by women or girls.

UNICEF staff explained that in Za’atari, most households installed their own toilets. There

are fewer building and financial resources in Azraq, so households are less able to construct

their own toilets44.

Along with adopting facilities to the requirements of camp inhabitants, the WASH

programme focused on efficiency, such as reducing the cost of water provision through the

development of boreholes, pumping stations, pipelines, and a wastewater network. The

programme also sought to control the supply and standards of services, and to depend less on

contractors incase of an increase in price, a reduction of quality, or a strike.

Settlements

In Rukban and Hadalat, as in the camps, UNICEF served a population with dire needs. Time

pressure and the desperation of the Settlements’ inhabitants made consulting them about their

preferences impractical. Hygiene services for the people in the settlements were not managed

as the population rose rapidly. In January 2016 the population reached 17,000 in the

settlements, with 1,400 in Hadalat and 15,600 in Rukban. By March 2017, the population had

grown to over 77,500 in habitants (Jordan and the Berm Rukban and Hadalat 2017-2018,

2017). The WASH Programme also assisted with transitioning 28,000 people from the

settlements to Azraq camp. The urgency of the situation is detailed in UNICEF’s six-month

emergency plan for these settlements. The emergency plan notes that there were a

“disproportionate number of pregnant women” among the residents of the settlements, that

the risk of disease because of unsanitary conditions was growing, that the difficult winter

weather posed risks for children, and that there

were ongoing protection concerns for

vulnerable populations. Between January 2015

June 2016, the WASH programme

installed mobile sanitation trailers and

distributed general hygiene kits , jerrycans,

and buckets. UNICEF also provided training of

health staff and community workers. By

December of 2016, The WASH programme

was operating more stable and cost-effective

solutions to provide water and sanitation.

The evaluation team’s assessment reviewed

Rukban’s borehole, pumping stations, and

treatment system. The relevance of the water

system was demonstrated by its strong design,

needed in harsh and resource scarce areas such

as Rukban. The water system has capacity for multiple contingencies including leaks, power

outages, or effluent quality issues (figure 4). There is 800 cubic meters of storage at the

borehole site with tanks with the ability to operate independently in case of leakage. The

system appears easy to operate as well. Gama Engineering and contracting Corporation is the

44 UNICEF’s explanation was provided as a comment to the first draft of this report.

Figure 4 The stand-by power generation increases the ability

of the water supply and distribution system to function

25

operator of the system. They maintain staff on-site 24 – 7. Staff performs daily O&M

including: i) pumping from the borehole to the tanks, ii) checking the chemical

concentrations every 2-3 hours, iii) routine cleaning of the sand filter daily, and iv) repair and

replacement of parts as needed45.

UNICEF provided water and sanitation services to respond to the settlements’ exigencies.

They moved quickly and effectively, with clear goals, such as ensuring that each person had

at least 15 liters of water per day. The WASH programme also recognized that its

intervention would have an impact on the larger Ruwaished host community, so they

undertook to rehabilitate sanitation facilities there. UNICEF also implemented projects in a

hospital, schools, and vulnerable households chosen through a vulnerability assessment.

The WASH programme has maintained weekly updates since the initiation of work. The

updates monitor operations and needs. The programme’s monitoring includes regular surveys

and assessments. These tools ensure that WASH programme activities remain relevant and

effective.

Host Communities and ITSs

It is difficult to judge the degree to which the WASH programme is relevant to host

communities and ITS needs. The WASH programme’s purpose was to respond to “Water,

Sanitation, and Hygiene needs in Jordan as a result of the Syrian refugee crisis.” Eighty

percent of the WASH programme’s funds and resources have gone into the management and

construction of the refugee camps, which host 20 percent of the refugees in Jordan, while 80

percent of Syrian refugees living in Jordan are in host communities46. Many other

international organizations and agencies provide services in these communities. In the

emergency response phase of the programme, the WASH programme focused on the camps

because the programme was the only source of water sanitation for that population, whereas

refugees in host communities and ITSs presumably were availing themselves of other

sources.

UNICEF’s improvement of the water and wastewater infrastructure in host communities is

significantly important. However, the degree to which it addresses the needs generated by the

refugee crisis and the needs of the most vulnerable is uncertain. UNICEF’s work in ITSs is

guided by vulnerability maps which indicate areas of greatest need and targets vulnerable

households. UNICEF also focused on areas where government has indicated a specific need.

Even with UNICEF’s improvements, some communities continue to only get water every 7

to 9 days and just for few hours, as was the case prior to UNICEF’s involvement47. Also,

Non-Revenue Water (NRW) in many communities is higher than 50%. More relevant and

strategic initiatives may have involved promoting wide scale NRW reduction, rainwater

water harvesting, water conservation, and groundwater recharge programs as real strategic

interventions that would yield long standing results.

45 Onsite assessment and interview with Gama Engineering

46 UNICEF notes that donors earmarked much of the funds for specific infrastructure interventions.

47 Host community interviews in Irbid

26

UNICEF reports that they have served 114 ITSs, but the total number of ITSs is unknown,

and the level of that service is unclear.

Monitoring System for Learning

The WASH programme has had a monitoring system set up in the camps since the beginning

of the program. A contractor initially managed the monitoring system.. However, the

contractor’s performance was found to be unsatisfactory, so the programme decided to

directly manage the system. As a demonstration of the improvement in services, logged

points of contact increased from 7 per week at the beginning of the program to 20 per week

shortly after UNICEF took over management. At its peak, the programme managed 60 points

of contact per day48.

The evaluation team could not find evidence that the WASH programme is structured as a

learning organization. It does not appear to have taken guidance from UNICEF’s experience

in WASH programming at the outset of the programme. While it has certainly made

improvements to its activities in terms of efficiency and responsiveness to people’s needs, it

did not build monitoring networks or feedback loops that fed into a system that would allow

for learning or informing strategic management. The evaluation team found no documentary

evidence that UNICEF’s WASH programme adhered to its Monitoring, Evaluation and

Programme Management policy or annual reviews, a mid-term review in 2015, and specific

programme performance assessments (UNICEF Jordan, 2012). It also did not use its

experience or other information to create programme strategies. A 2016 report, “UNICEF

WASH Actions in Humanitarian Situations: Synthesis of Evaluations 2010 – 2016,” The

report found that there is a “need for a more consistent process of learning, disseminating,

and applying knowledge and good practices generated through experience (Hūls, 2017).”

This evaluation finds that this conclusion also applies to the UNICEF Jordan WASH

programme. This theme is further developed in the efficiency, effectiveness, and

sustainability sections of this report.

Evaluation Question 2 • To what extent did the interventions consider the needs and the priorities of the MWI

and were they coherent with global references and the regional and national response

to the Syria Crisis?

UNICEF’s work in WASH is in line with the priorities of the National Water Strategy, for

which the key reference document is the National Water Strategy 2016 – 2025. UNICEF is

focused on the sustainable operation of boreholes to supply water and create resilience.

UNICEF also recognizes the government’s concern that Jordan’s northern governorates have

not benefited from the government’s efforts to increase water supply (Ministry of Water and

Irrigation, 2016). In response, UNICEF has focused much of its host community

interventions on improving WASH in the northern governorates. Those efforts are based

UNICEF’s vulnerability mapping results.

48 Interview with UNOPS Staff – 1 August 2018

27

Both the MoWI and WAJ reported in interviews that the GoJ-UNICEF partnership is

successful. The WAJ praised UNICEF’s success at sector coordination despite budget cuts49.

The MoWI appreciated UNICEF’s role in coordinating the sector50.

Interviews at MoWI and WAJ demonstrate the Government of Jordan’s struggles with its

strategy and UNICEF’s response. Senior staff at the MoWI praised UNICEF’s

responsiveness inside the camps but sees that project’s host communities are more dependent

on available funding51. MoWI said that when the WASH programme first took on host

community projects, there was a miscommunication regarding the meaning of the protocol

UNICEF established with the MoWI. The protocol included a mandate and budget for

UNICEF to improve infrastructure in governorates. The ministry didn’t realize that the

budget was prospective and moved those items off of its budget for the year. When

fundraising fell short, those governorates were underserved52.

An interview with senior staff at MoWI indicated two concerns. The GoJ has made it clear

that the government will not take over responsibility for WASH services in the camps in any

form. The MoWI has prevented studies of potential government involvement. The MoWI

wonders what UNICEF’s plan is for sustainability or handover in the camps. Secondly, the

MoWI wondered about UNICEF’s medium to long-term role in the sector and was desirous

of closer collaboration if UNICEF were seeking to look at medium to long term

investments53.

Evaluation Question 3 • How responsive was the UNICEF WASH programme over time to changes in the

external environment?

UNICEF has responded to changes in the external environment, while maintaining standards

in its programs.

In the initial phase of the programme, the GoJ and INGOs in Jordan prepared for a temporary

camp. This preparation included procuring tents, portable sanitary facilities, and trucked in

water supplies. While these interventions were effective, their expense made them

unsustainable. When it became apparent that the need to house refugees was more than

temporary, UNICEF devised solutions that required investment, but would be cheaper and

more sustainable to operate in the medium to long-term. These interventions included WASH

Blocks, borehole construction, water network construction and operations, establishment of a

treatment plant, and established hygiene campaigns. UNICEF also made these adjustments in

the settlements, establishing water infrastructure that was both resilient enough to withstand

difficult conditions and use, but also reduced operating funding needed for maintenance54.

49 WAJ Senior Staff Interview – 30 July 2018

50 MoWI Senior Staff Interview – 30 July 2018

51 UNICEF notes that in 2018, MWI asked UNICEF inflate budgetary figures so that they include all host

community projects.

52 MoWI Senior Staff Interview – 30 July 2018.

53 Ibid.

54 UNICEF Staff Interviews – 10 October 2017

28

As senior UNICEF staff noted, “Transitioning from humanitarian response to systems and

infrastructure…is incredibly challenging, particularly in environments like the berm

(Rukban).55”

One type of transition in which the WASH programme struggled was in implementing host

community projects. As an interview respondent at the MoWI suggested, UNICEF did not

originally plan on implementing host community projects. UNICEF incorporated host

community projects into the programme when faced with external pressure from

municipalities where refugees were putting stress on local water resources56. The WASH

programme boldly took up the challenge and set high goals for itself, such as increasing the

percentage of the population in Jordan that has access to adequate sanitation facilities (2014),

or increasing the number of inhabitants with safe water and storage facilities by 1.4 million

(2015)57. While UNICEF missed these targets in each year, it continued to set high targets in

this component. As mentioned in the Funding Environment section of this report, UNICEF

blamed shortfalls in fundraising for missing host community targets in each year of the

programme. However, the evaluation team did not find evidence that the WASH programme

considered its funding outlook when planning these targets, or, given the regular shortfall in

funding, how it might better spend these funds. We discuss this topic further in the

Effectiveness and Efficiency sections.

In this evaluation’s focus groups, camp residents expressed what they felt were the most

important improvements that had taken place in WASH services, and which issues had not

yet been addressed. In Za’atari, focus group participants said that the installation of private

toilets was the most important improvement. They said that public toilets had cleanliness and

security issues, and the ad hoc pit latrines that many households dug next to their housing

units were unsanitary and dangerous. Private toilets were an enormous improvement. As far

as issues that have gone unaddressed in Za’atari, participants mentioned that they have

observed water supply and sewage disposal pipes overlapping in the same trench, and they

fear sanitation threats. They also mentioned households receiving the same quantity of water

regardless of the number of household members as an issue that requires a solution58.

In Azraq, participants said that the increase in tap stands was an important improvement.

Participants in Azraq had a number of complaints, mainly focused on lack of private

facilities, high chlorine content in water59, wasted water through misuse of some residents,

and lack of hygiene products such as detergent and soap60.

In King Abdullah Park, focus group respondents felt that improvements were particularly

made in the hygiene of the latrines and solid waste collection. They credited prompt and

frequent maintenance61.

In the current phase, the WASH programme is moving to address longer-term, nationwide

issues of sustainability. These include, climate change, depleting water resources, and

assisting communities that are becoming water scarce62.

55 UNICEF Senior Staff Interview – 6 August 2018

56 MoWI Senior Staff Interview – 30 July 2018 57 See results framework, Annex H. 58 Source: Focus Group Discussions – Za’atari Camp 59 UNICEF Jordan reports that chlorine content is within Jordanian standards. 60 Source: Focus Group Discussions – Azraq Camp 61 Source: Focus Group Discussions, KAP Camp 62 Source: Interviews with UNICEF senior staff – 6 August 2018

29

Effectiveness

Evaluation Question 4 • To what extent did UNICEF’s response achieve its intended outcomes?

• To what extent did UNICEF achieve equity results, especially for children and

women specific interventions?

The WASH programme’s achievements were tracked against different outcome statements,

output targets, and indicators in three phases; 2013/2014, 2015, and 2016/201763.

In 2012, when the programme launched, it operated without a results framework64. The

programme’s main objective was to respond to refugee needs in a quickly changing, resource

constrained context.65 In 2013 the programme drafted indicators and partially reported

achievement compared to them. Details were provided for of IR 1.3 indicators, but not IR

1.6. indicators66.

The full results framework for each year is included as Annex H. The next section provides

the programme’s objective and/or intermediate results and output statements under each

programme phase67. Following that, the report provides an overall comment on the

evaluations measure of effectiveness, and then the report presents evaluation’s findings

against indicators under each of the programme’s four components for each year.

Objective and Output Statements

2012

UNICEF and the WASH programme were operating as an emergency response programme

in 2012. In the second half of 2012, the WASH programme’s main focus was serving

refugees in the newly opened Za’atari refugee camp and at transit centers. Later in the year,

UNICEF began work in host communities, KAP and Cyber City.

The evaluation team used available documentation to assess achievements under each

component. Much of the information in this section came from Results Assessment Module

(RAM) reports, which UNICEF uses to capture achievement against indicators. The WASH

programme did not provide a workplan, grant agreement from donors, or RAM report from

2012. The programme’s documentation of its activities and achievements begins on October

4,, 2012 with the publication of UNICEF’s weekly “Syrian Refugees Response Overview

(UNICEF Jordan, 2012) .” Below are the objective and output statements for 2013 – 2017:

63 See results framework, Annex H

64 The evaluation team found no evidence of a results framework from this year.

65 Source: UNICEF, World Vision, and Mercy Corps staff interviews

66 IR 1.3 was Relevant Core Commitments on WASH are met in all humanitarian settings. IR 1.6. was Support

the Ministry of Water and Irrigation and Sector partners to address the impact of refugee influx as well as

chronic sector needs

67 UNICEF’s indicator frameworks used different language to describe result levels in 2013 and 2014.

30

2013/2014

Objective

National Institutions provide improved health and neonatal care

services and quality early childhood care with a focus on

disadvantaged groups.

Intermediate Result

1.3

Relevant core commitments on WASH are met in all

humanitarian settings.

Intermediate Result

1.6

Support the Ministry of Water and Irrigation and Sector partners

to address the impact of refugee influx as well as chronic sector

needs.

2015

Objective

Quality WASH facilities are sustained and utilised, and hygienic

behaviours practiced, by the most vulnerable while ensuring the

protection of the environment.

Output 4.1 Government and WASH sector partners plan and optimally utilize

WASH resources and services for equitable access.

Output 4.2 Children, parents and community members are aware of and

practice hygiene and water conservation.

Output 4.3 Urgent needs of vulnerable people especially children are met

through quality and lifesaving WASH services.

2016/201768

Objective

Quality WASH facilities are sustained and utilised, and hygienic

behaviours practiced, by the most vulnerable while ensuring the

protection of the environment.

Output 4.2 Institutions, utilities and organisations provide equitable access to

sustainable water and sanitation services for the most vulnerable.

Output 4.3 The most vulnerable women and children are aware of and

practice key hygiene behaviours and practice water conservation.

Output 4.4

The urgent WASH needs of the vulnerable women and children

are met through safe, predictable and sustainable water and

sanitation services and facilities.

68 The evaluation team did not have access to the 2017 RAM report. We assume targets were the same for 2016

and 2017. Results are based on UNICEF’s public reports and any internal records that the WaSH Programme

provided to the evaluators.

31

Overall comment on effectiveness

This evaluation’s ToR asks that the evaluation team judge the WASH programme’s

effectiveness, in part, by the degree to which the programme achieved its intended

outcomes69. This question is difficult to answer. First, in the emergency response phase of the

programme the intended outcome was singular, provide sufficient WASH services to the

refugees who so direly needed them70. In subsequent phases, the evaluation team found no

evidence that the WASH programme conducted strategy development or work planning

processes, which would have produced specific, measurable targets that performance could

be judged against.

The WASH programme has filed annual RAM reports since 2013. These reports include

UNICEF’s estimate of its achievements against a set of outcomes. However, the intended

outcomes are often vague in that they do not describe a means of addressing specific

population’s need that has been identified as strategically important. An example of an

intended outcome as described by RAM for 2017 is:

Outcome 4 Sustainable and equitable WASH system and key behaviours practiced.

Output 4.2 Institutions, utilities and organizations provide equitable access to

sustainable water and sanitation services for the most vulnerable.

Indicator 2 3.2 million people with access to improved municipal water services.

While those targets are certainly worthwhile, they don’t indicate the specific behaviours,

institutions, organizations, and people that the programme has identified as important to

responsiveness to the Syrian refugee crisis. Also unclear is whether the 3.2 million people is

the annual target or the cumulative programme target. These objectives do not meet the

requirements for specificity, measurability, relevance, or time-boundedness usually required

in results frameworks71.

Secondly, the WASH programme maintains limited monitoring data. Through UNOPS, the

programme monitors activity in the camps to ensure coverage and equity. The programme

collects reports from its partner organizations that include accomplishments against specific

targets defined in work plans, and budgets disaggregated by activity and result. The WASH

programme also collects water monitoring reports, and regularly completes its own reports

that include an accounting of the programme’s outputs and outcomes. However, UNICEF

does not aggregate the information in a format that would allow for its analysis. Partner

reported data is maintained in a format such as Microsoft Word, or an unsearchable .pdf

document, making it impractical for the evaluation team to extract and use the data as a

source of support for the claims made in the RAM and other reports.


70 Source: Multiple interviews with UNICEF and partner staff.

71 See, for example, The WaSH Cluster Coordination Handbook, pg 258; or UNHCR’s Practical Guide to the

Systematic Use of Standards and Indicators, pg. 57.

http://www.unhcr.org/40eaa9804.pdf

http://www.unhcr.org/40eaa9804.pdf

32

Lastly, the WASH programme’s donors do not require that UNICEF report results or

financial information disaggregated by year, achievement, activity, or other programme

category72. The lack of requirement on the part of donors provides a disincentive for UNICEF

to implement a standard financial reporting or monitoring and evaluation system.

72 Source: UNICEF Senior Staff and PMU Interviews

33

Component 1: Provision of equitable WASH services and dissemination of WASH messages in the four camps

Achievement of Intended Outcomes Overall Findings73:

Under this component, the WASH programme achieved high levels of effectiveness for the

targeted population across all years.

Details

2012 Component 1 results:

• Provision of between 1 and 1.3 million liters of

water per day to Za’atari through water

trucking.

• The completion of two boreholes to provide a

lower cost and more secure source of water for

Za’atari camp.

• The completion of 122 WASH Blocks

completed with construction of another 160 in

progress. The WASH blocks provided 662

latrines and 546 showers. The WASH units are

equitably provided for women and men, and

facilities are included for people with

disabilities.

• 151 mobile latrines and 68 mobile showers installed.

• 9,931 hygiene promotion sessions held by UNICEF’s partner organization, ACTED.

• Employment of 262 beneficiaries in a cash-for-work program in Za’atari, 38 in KAP,

and 20 in Cyber City.

• Removal of between 400 and 650 m3 per day of solid waste from Za’atari.

• Partners ACTED and THW monitor water quality arriving in Za’atari through

residual chlorine tests.

• Removal of 300 to 500 m3 per day of waste water from Za’atari.

• Maintenance of water and sanitation facilities at transit centers.

2013

In 2013 the WASH programme successfully achieved its targets under this component. 100

percent of refugees in Za’atari camp provided with drinking water. (IR1.3, Indicator 2)

• 100 percent of refugees in Za’atari provided with soap and benefited from hygiene

promotion activities. (IR 1.3, indicator 3 and Indicator 5)

• 100 percent of refugees in Za’atari have access to and use secure sanitation facilities.

• 2,340 toilets made available in Za’atari. (IR 1.3, indicator 4)

• 2,000 toilets and showers made available to serve the refugees in Azraq camp. (IR

1.3, indicator 4)

• 2,379 septic tanks installed in Azraq, or one tank for every 12 residents. (IR 1.3,

indicator 4)

73 Results in this section come from UNICEF SitRep reports, Annual Reports, and RAM Reports

Figure 5: Za'atari Camp 2012 (Source:Times of

Israel, 29 August 2012)

34

In addition to achieving its targets, the WASH programme contributed to the design and

construction of Jordan’s second largest refugee camp, Azraq. The programme incorporated

lessons learned from Za’atari and made improvements to its activities and infrastructure in

the camps that increased overall effectiveness. UNICEF made these strides under difficult

circumstances. In 2013, Jordan experienced extreme winter weather conditions, including its

heaviest rainfall in 20 years. The WASH programme struggled with wastewater removal

services provided by a contractor, and drilled a new borehole for Azraq that failed to meet

national water standards. Meanwhile, the inflow of refugees nearly doubled over the previous

year.

2013 Component 1 results not measured in UNICEF’s indicator framework include:

• Winterization of 72 WASH blocks completed by the end of March.

• Za’atari services increased to providing a daily average of 3,322,000 litres of water,

or up to 25 liters per inhabitant.

• Removing 1,250 m3 of waste water and 1,400 m3 of solid waste by the end of the

March.

• Water storage facilities/tanks are built in Za’atari, which are operational at the end of

May.

2014

In 2014 the WASH programme reported that it successfully achieved most of its targets

under this component74. It met its targets to provide water and sanitation services to residents

of the refugee camps. The WASH programme failed to achieve its targets under the

distribution of hygiene items, providing items to 107,000 of the 525,000 targeted

beneficiaries75. It is unclear what proportion of hygiene items were distributed to people in

camps versus other locations. In 2014, Azraq camp opened, a waste water treatment plant

was constructed, and new hore holes at Za’atari and Azraq were drilled.

Code Indicator Target76 Result

Output

1.3, Ind

2

Number of people provided with

safe access to sufficient water for

drinking and domestic use

(cumulative beneficiaries

reached)

100 percent

of refugees

in camps

Target met including 98,000

refugees in four camps provided

with at least 35 liters per person

per day.

Output

1.3, Ind

3

Number of persons benefiting

from improved sanitation

(cumulative beneficiaries

reached)

100 percent

of refugees

in camps


refugees in four camps

benefiting from desludging of

communal, household and

institutional latrines

74 2013 & 2014 RAM Report

75 UNICEF reports that one reason for underdistributing items was that the market in Za’atari had become more

efficient at meeting demand than distribution activities.

76 The target figure represents the target for the population specified under each component when possible. For

example, some indicators report provision of a service for all refugees, whereas the component describes

provision to those in camps, host communities, or elsewhere. In each component the target indicator is

reported for the beneficiary group described under that component.

35

Output

1.3, Ind

4


hygiene items

100 percent

of refugees

in camps

Target met including UNICEF

reports 107,000 of the 525,000

targeted beneficiaries were

reached. It is unclear what

portion were in camps and which

were in other locations.

Output

1.3, Ind

5

Number of people involved in

hygiene promotion activities

100 percent

of refugees

in camps


refugees in four camps reached

through direct messaging.

Output

1.3, Ind

8


access to solid waste

management (cumulative

beneficiaries reached)

100 percent

of refugees

in camps


refugees in four camps benefited

from of 800 m3/ day on average

of solid waste removal

2015

In 2015, the WASH programme reported that it met its targets for water provision and

sanitation in the camps. A key achievement in 2015 was a general improvement in health in

Za’atari camp. In both the 2013 and 2014 KAP surveys in Za’atari, 58 percent rated their

health as good or excellent. The percentage increased to 69 percent in 2014 and 81 percent in

2015 (Za'atari Hygiene Promotion Working Grouip, 2015).

While the WASH programme reported 100 percent water coverage in Zaatari, the 2015 KAP

survey showed that only 61 percent of residents accessed water from the public tanks.

Support for the piped water network was high with 63 percent rating this project as “very

positive”.

The 2016 Azraq KAP Survey reported that 95 percent of people felt the water supply was

adequate or very adequate77. Access was good with 46 percent collecting water within 15

minutes, well within the 30-minute Sphere standard (The Sphere Project, 2011). Ninetly-six

percent accessed water for free (from public tapstands and not private vendors). Ninety

percent were happy with the water quality (ACTED, World Vision, ACT, RI, and Save the

Children, 2016).

The third indicator under this component, “proportion of population practicing hygiene

promotion (sic)”, went unmeasured by UNICEF in 2015. The programme reported that

monitors had observed increasing rates of key hygiene practices, such as handwashing with

soap. UNICEF reported that 100 percent of refugees in camps were provided with safe solid

waste management. In 2015, the quality of that management may have not been sufficient. In

2014 88 percent said their area was generally clean but this dropped to 71 percent in 2015.

No reason was given for the drop.

Accomplishments in 2015 include Za’atari’s waste water treatment plant becoming

operational, work on Za’atari’s water network starting, and Azraq’s borehole becoming

operational. Comprehensive child focused assessments were carried out in Azraq and

Za’atari, and KAP surveys were conducted. UNICEF also managed to provide services

77 The 2016 Azraq Kap survey was completed in January, representing 2015 results.

36

during a five-day truck driver strike in Za’atari and implemented activities that prevented any

further major strikes.

Note that for the output indicators in the table below, UNICEF estimated that 140,000

refugees would reside in the camps. However, the population never grew that large, so the

number of refugees reached represents a higher proportion that represented using the 140,000

figure.

Code Indicator Target Result

Outcome

4, Ind 2

Proportion of population at camps

using improved drinking water as

per Jordan standards.

100% of

refugees in

camps

100% of refugees in camps

were provided with safe water

(with the tankered quantities in

excess of the minimum standard

of 35l/p/d)78.

Outcome

4, Ind 3


using improved sanitation

facilities.

100% of

refugees in

camps

100% of refugees in camps used

improved sanitation facilities in

either communal WASH blocks

or their own constructed

facilities in their shelters.

Outcome

4, Ind 4

Proportion of population

practicing hygiene promotion at

camps.

100% of

refugees in

camps

No result reported for camps

Output

4.2, Ind

1

Number refugee in camps through

Hygiene messages.

140,000 100,000 reached,

Output

4.3, Ind

1

Number refugees provided with


drinking and domestic use through

water trucking.

140,000 122,055 provided with 35 litres

per day

Output

4.3, Ind

2

Number of refugees provided

with safe waste water disposal

through dislodging of septic tanks.

140,000 122,055 provided with

wastewater disposal

Output

4.3, Ind

3


with safe waste water disposal

through provision of Waste Water

Network in Zaatari Camp.

140,000 No result reported. The

wastewater network was not yet

complete.

Output

4.3, Ind

4


with safe solid waste management

facilities.

140,000 122,055 provided with solid

waste management.

78 Statement appears exactly as written in UNICEF’s 2015 RAM report

37

Output

4.3, Ind

5


with safe access to sanitation

facilities by operating and

maintaining the existing WASH

Blocks through WASH

Committees and/or community

involvement

140,000 122,055 provided with WASH

blocks in four camps.

Output

4.3, Ind

6

Number of refugee Boys and Girls

in camps provided with WASH in

schools

45,000 50,691 benefited from WASH

in schools.

2016

In 2016, the WASH programme reported that it met or exceeded its targets under component

1.

Definitive measurements were not available for Outcome 4, Indicator 4 related to hygiene

practices. However, a KAP study conducted in Za’atari and the output figures for hygiene

activities in settlements imply that the target was met (UNICEF Jordan, 2016).

Other WASH programme achievements in 2016 include managing the large population

increase in the Rukban and Hadalat settlements, managing the transfer of refugees to Azraq,

and continuing management of Azraq and Za’atari water networks.


Outcome

4, Ind 2




100% of

refugees in

camps and

settlements





of 35l/p/d).

In the camps and at the border,

244,052 people were provided

with water over the course of

2016

Outcome

4, Ind 3



facilities

100% of

refugees in

camps

100% of refugees in camps and

settlements used improved

sanitation facilities in either

communal WASH blocks or

their own constructed facilities

in their shelters.

38

Outcome

4, Ind 4



targeted camps, host communities

and schools

100% of

refugees in

camps

Za’atari 2015 KAP indicated

between 80-96% of females

with childcare responsibility

washed their child’s hands at

key times and 92% said that

they did so using soap and

water.

Hygiene products and services

delivered to the settlements.

2017

In 2017, achievements included high levels of satisfaction in Azraq. In the 2017 Azraq KAP

survey, 68 percent of the participants replied the water was very adequate compared to 44

percent in the 2016 survey. In 2016, water storage was a concern, but residents felt it had

been resolved. 88 percent of respondents in the 2017 survey said they need less than 15 mins

to collect water compared to 46 percent in 2016. Satisfaction with water quality dropped in

2016 to 75 from 90 percent in 2016. The reported reason was poor taste, smell, and colour

(UNICEF, ACF, 2017).

In 2017, the programme completed the waste water system at Rukban. The programme also

ceased tankering, at Azraq realizing cost savings.


Outcome

4, Ind 2




100% of

refugees in

camps and

settlements





of 35l/p/d).

In the camps and at the border,

19.7 litres per person per day

were provided to refugees in

settlements.

Outcome

4, Ind 3



facilities

100% of

refugees in

camps

100% of refugees in camps and

settlements used improved

sanitation facilities in either

communal WASH blocks or

their own constructed facilities

in their shelters.

Outcome

4, Ind 4




and schools

100% of

refugees in

camps

Za’atari 2015 KAP indicated

between 80-96% of females

with childcare responsibility

washed their child’s hands at

key times and 92% said that

39

they did so using soap and

water.

Hygiene products and services

delivered to the settlements.

Component 2: Provision of support in the host communities including WASH in schools

Overall Findings: As detailed with citations below, the WASH programme consistently

struggled to achieve targets under this component. Two issues created difficulties: funding

and lack of strategic planning

As described in the Funding Context section of this report, UNICEF Jordan struggled with

funding issues. Maintaining services in the camps necessarily received first priority for funds

at the expense of achieving targets in host communities.

That said, after several years of missing targets, programme management documentation

indicates a lack of adjustment to strategy to make the programme more effective or realistic79.

For example, from 2013 to 2017 the programme included an indicator intended to set goals

for provision of WASH services to students80. The target was missed each year of the

programme by a wide margin (table 1). Other indicators following this example include those

regarding hygiene promotion, are illustrated below.

Table 1 Students benefiting from improved WASH facilities in the learning environment, Target vs

Achievement

Year Target Achieved Percent Achieved

2013 230,633 72,985 31.65%

2014 188,000 100,000 53.19%

2015 200,000 91,757 45.88%

2016 60,000 37,693 62.82%

2017 114,000 7,980 7.00%

Additionally, the indicators aren’t sufficiently specific under this component, so it’s difficult

to know what the programme is trying to achieve. Using the above example again, in 2016,

the indicator was stated as “Number of people with access to improved WASH facilities in

institutions,” and the year’s achievement was stated as “37,693 students benefited from

provision of support to 45 schools.” It is unclear if the initial intention was that “people” in

institutions should mean students, or if the achievement was retroactively fitted to the

indicator. UNICEF’s Outcome 4, Indicator 2 target is that 1.5 million host community

79 UNICEF reports that the outcome level was fixed in 2014 for the period of 2015 – 2017. The target was

increased in 2016/2017 because of an optimistic fundraising outlook in 2015, which did not materialize.

80 The indicator was stated differently over the course of the programme: 2013/2014 – IR 1.3, Ind 6: “Number

of children (girls and boys) provided with access to safe water, sanitation and hygiene facilities in their

learning environment and in child friendly spaces.”, 2015 - Output 4.1, Ind 3, “Number of students provided

by (sic) WASH services.”, 2016/2017 (Target and baseline were general, but achievement reported in terms

of students) – “Number of people with access to improved WASH facilities in institutions.”

40

residents use improved drinking water as per Jordanian standards. In its 2015 RAM report,

the WASH programme stated that “In Host Communities, 41% (634,858) of the target

population of 1,550,500 benefitted from improved access to water facilities and systems as a

result of rehabilitation of water infrastructure, as well as support to vulnerable households.”

In its 2016 RAM report, for the same indicator the WASH programme reported, “In Host

Communities in four governorates (Mafraq, Madaba, Balqa and Irbid), 237,360 additional

people were reached in 2016. Since 2013, an estimated 1,482,402 people have improved

access to water services …which equates to 96 percent of the Country programme target.” It

is unclear if the target is that UNICEF serve 1.5 million per year under this outcome as the

2015 report implies, or 1.5 million cumulative people as the 2016 report implies. It is also

important to consider the improvement. For example, in Irbid UNICEF made improvements

to the water distribution system. This should not imply that everyone served by the system

received better water services as a result of the intervention. Water services are still very

limited, and there are still serious problems with the systems distribution network, water

safety, and availability.

Achievement of Intended outcomes

2012

2012 results:

• The first work on host community water systems, in Ramtha and Mafraq, was

completed.

• Relief International hygiene promoters complete house-to-house visits in March with

hand-washing messaging and soap distribution to 2,000 households in Mafraq and

Ramtha.

Other evidence of component effectiveness in 2012:

Za’atari opened in Mafraq governorate in July, 2012. Tensions began to rise both within the

camp and between the camp and communities outside of the camp. In response, the UN

identified serving vulnerable Jordanian’s as an important component of the Syrian crisis

response (Seely, 2012). The WASH programme expanded its scope to serve community

needs as well as camp needs.

2013

In 2013, the WASH programme met its commitments to refugees, but underachieved in terms

of the targets it set for serving Jordanians, as the results against targets below demonstrate:

41


IR 1.3,

Ind 2

Number of emergency-affected

population (male and female)

provided with access to drinking

and domestic water

Approximately

215,00081 non-

camp refugees

and 180,000

Jordanians

150,000 refugees in HC’s

benefit from rehabilitation of

existing boreholes82.

Projects in host communities

benefit 400,000 people (UNICEF

Jordan, 2013)

IR 1.3,

Ind 3

Indicator 3: Number of

Emergency affected population

(male and female) provided with

access to soap and other hygiene

items

Approximately

215,000 non-

camp refugees

and 180,000

Jordanians

201,920 refugees provided with

soap and hygiene kits83.

IR 1.3,

Ind 4

Number of Emergency affected

population (male and female)

with access to appropriately

designed toilets and sanitation

services

180,000

Jordanians

No results outside of camps

reported achieved.

IR 1.3,

Ind 5

Number of population (male and

female) covered through face to

face (including HH and Group

sessions) hygiene promotion

messages

Approximately

215,000 non-

camp refugees

and 180,000

Jordanians

201,920 women, men, boys and

girls have been participating in

face to face hygiene promotion

activities including focus group

sessions and house to house

visits

IR 1.3,

Ind 6

Number of children (girls and

boys) provided with access to

safe water, sanitation and hygiene

facilities in their learning

environment and in child friendly

spaces

230,633 72,985 benefit from 91

rehabilitated WASH facilities

in schools.

WASH facilities were

rehabilitated in over 100

Jordanian schools benefiting

23,000 Jordanians and 8,000

Syrian children.84

81 Target is estimated. UNICEF described its target as “345,000 refugees (in camps and off camp) and 180,000

affected Jordanians.” The evaluation assumes the most conservative estimate of refugees in camps and

subtracts that from the total number.

82 2013 RAM Report

83 2013 RAM report.

84 Ibid.

42

IR 1.3,

Ind 7

Number of children (girls and

boys) not practicing open

defecation in host communities.

100% No measurement reported. The

programme reported, “host

communities (have) improved

including hygiene awareness

messages on risks of open

defecation.

2014

The WASH programme struggled to meet its targets under this component in 2014, as the

table below demonstrates. The exception was in the area of WASH in Schools (indicators 6

and 7). The programme significantly exceeded its target promoting hygiene messages to

students.


Output

1.3,

Ind 2



drinking and domestic use

(cumulative beneficiaries reached)

Approximately

255,000 non-

camp refugees

and 180,000

Jordanians

approximately 7,000 people in

vulnerable households benefited

from household water

improvements.

an estimated 450,000 people

(refugees and host community)

benefitted from increased

access to water as a result of

rehabilitation of existing water

systems.

Output

1.3,

Ind 3

Number of persons benefiting

from improved sanitation

(cumulative beneficiaries reached)

Approximately

255,000 non-

camp refugees

and 180,000

Jordanians

10,000 people benefited from

removal of sewage network

blockage.

7,000 people in vulnerable

households benefited from new

or rehabilitated facilities

Output

1.3,

Ind 4


hygiene items

Approximately

255,000 non-

camp refugees

and 180,000

Jordanians

No results reported for Non-

camp refugees or Jordanians.

Output

1.3,

Ind 5

Number of people involved in

hygiene promotion activities

Approximately

255,000 non-

camp refugees

and 180,000

Jordanians

10,000 people were reached

through mobilisation activities

on key hygiene messages

43

Output

1.3,

Ind 6

Number of children benefitting

from improved water, sanitation


environment (cumulative


100,000 188,000 WASH facilities in 164

schools85

Output

1.3,

Ind 7

Number of children benefitting

from improved water, sanitation


environment (cumulative


100,000 116,788 students reached in

host communities

Output

1.6,

Ind 3.

Percent increase of targeted host

population accessing sufficient

using appropriate sanitation

facilities

5% increased

specifically in

the Northern

Governorate

No result reported86

2015

The WASH programme underperformed against most of its targets under this component in

2015. The closest target the programme came to meeting was Output 4.1, Indicator 4,

Number of population at host communities benefiting from sewage systems improvement,

where the programme improved sewerage for about 83 percent of its target population.

Note that in the table below, student targets were set before UNICEF conducted the

Nationwide WASH in schools assessment. UNICEF reports that it anticipated a higher need

for interventions before the assessment.


Outcome

4, Ind 2

Proportion of population in host

communities using improved

drinking water as per Jordan

standards.

1,550,000

host

community,

200,000

students

634,858 HC residents benefited

from improved water

infrastructure.

91,757 students benefited from

school WASH facility

rehabilitation and construction

Outcome

4, Ind 3

Proportion of population at host

communities and schools using

improved sanitation facilities

545,000 host

community,

200,000

students;

390,065 benefited from

improved access to sanitation

systems and facilities.

85 This figure, reported by UNICEF, implies 1,150 students per school, which is possible considering that large

classes sizes and double shifts common at many Jordanian schools (Middle East and North Africa Out-Of-

School Children Initiative, 2014))

86 In the 2014 RAM report, the WaSH Programme reported that 10,000 people benefit from removed blockages

in the sewage network. However, this result does not apply to this indicator, and was already reported as the

result of Output 1.3, Indicator 3 in that year.

44

91,757 students benefited from

school WASH sanitation

facility rehabilitation and

construction

Outcome

4, Ind 4



targeted host communities and

schools

150,000 host

community,

200,000

students

100,000 people reached through

mobilization sessions.

73,268 students received

hygiene kids.

Output

4.1, Ind

2

Number of population benefiting

from safe water and storage

facilities improvement.

150,000

refugees and

1,400,000

residents in

HCs

619,450 people benefitted from

improvements to the municipal

water network


provision of key infrastructure

items

5,387 people in Vulnerable

Households were supported

with improvements to

household wash facilities

Output

4.1, Ind

3

Number of students provided by

(sic) WASH services

200,000

students in

HC

91,757 students served by

WASH facilities

Output

4.1, Ind

4

Number of population at host

communities benefiting from

sewage systems improvement

150,000

Syrians in

HC, 395,000

in HC


support to improvements to

sewerage systems.

8,255 people benefited from

mobile latrines.


provision of key infrastructure

items.

Output

4.2, Ind

1

Number refugee in host

communities and students reached

through hygiene messages

150,000 at

host com

and 200,000

students

100,000 people reached through

mobilization sessions.

73,268 students received

hygiene kids.

2016

In 2016 the programme did not meet targets under this component, falling short more

drastically than in previous years. The evaluation team believes that at this point it becomes

apparent that the issue was more of a lack of strategic design process than underperformance

given programme resources and objectives.

45


Outcome

4, Ind 2



improved drinking water as per

Jordan standards.

1,550,000

host

community,

200,000

students

237,360 reached

37,593 students benefitted from

improved WASH facilities

Outcome

4, Ind 3




545,000 host

community,

200,000

students;

35,000 people in Irbid have

improved sanitation access as a

result of the rehabilitation of the

sanitation system.

37,593 students

Outcome

4, Ind 4




and schools

150,000 host

community,

200,000

students

No result reported

Output

4.2, Ind

1

Number of people living in

vulnerable Households with

access to improved household

WASH facilities and services

18,000 No direct result reported

Assessment carried out.

Output

4.2, Ind

2

Number of people with access to

improved municipal water

services

600,000 237,260 benefited from

improvements to water

infrastructure.

Output

4.2, Ind

3


improved municipal sewerage

services

400,000 35,000 people gained access to


services.

Output

4.2, Ind

4


improved WASH facilities in

institutions

60,000 37,593 students benefitted from

the provision of support to 45

schools.

Output

4.3, Ind

3

Number of people in the host

community reached through

social/community mobilisation on

key WASH messages and water

conservation.

18,000 No direct result reported.

Assessment carried out in

preparation.

Output

4.3, Ind

4

Number of people in institutions

in the host community reached

through social/community

mobilisation on key WASH

messages and water conservation

60,000 37,593 students

Output

4.3, Ind

5

Number of people in informal

settlements reached through



conservation

15,000 7,298 vulnerable people in ITSs

received key WASH messages.

46

2017

It is difficult to tell how UNICEF performed against WASH targets from January – July of

2017. The difficulty arises because the WASH programme does not maintain a monitoring

database that tracks progress against indicators on a regular basis. Below we use the 2017

Annual Report, where possible, to show the WASH programme achievement over the course

of the year. Though much information is missing, UNICEF reports that the WASH

programme underperformed against targets in this component. One important

accomplishment in 2017 was that the national WASH in Schools Standards were released.


Outcome

4, Ind 2



improved drinking water as per

Jordan standards.

1,550,000

host

community,

200,000

students

258,000 people in host

communities (UNICEF Jordan,

2017)

14,000 students

Outcome

4, Ind 3




545,000 host

community,

200,000

students;

258,000 people in host

communities (UNICEF Jordan,

2017)

14,000 students

Outcome

4, Ind 4




and schools

150,000 host

community,

200,000

students

No measurement provided

Output

4.2, Ind

1

Number of people living in

vulnerable Households with access

to improved household WASH

facilities and services

44,000 No measurement provided87

Output

4.2, Ind

2


improved municipal water services

3.2 million 258,000

Output

4.2, Ind

3



services

2.1 million No measurement provided.

87 In the 2017 Annual Report, UNICEF states, “Little progress was made toward the target for sanitation. This

was due to sectoral lack of funds, as well as prioritization of water projects for the Government, due to the

enormous pressure to deliver water. UNICEF continued to work with the Government to advocate for

appropriate allocation of resources to meet sanitation needs. “

47

Output

4.2, Ind

4


improved WASH facilities in

institutions

114,000 7,98088

Output

4.3, Ind

3

Number of people in the host

community reached through



conservation.

44,000 No measurement provided.

Output

4.3, Ind

4

Number of people in institutions

in the host community reached

through social/community

mobilisation on key WASH

messages and water conservation


Output

4.3, Ind

5

Number of people in informal

settlements reached through



conservation


Component 3: Support to the WASH sector through technical support to the MoWI, as

well as National WASH in School Standards

Overall Findings: Government officials at the MoWI and WAJ reported satisfaction with the

WASH programme contribution to planning documents, strategies, and policies89. The MoWI

said that, as head of the WASH Taskforce, UNICEF always has a good plan, and staff that

serve together on technical committees have good partnerships with UNICEF. The Ministry

also praised the work done by the consultant that UNICEF seconded to Ministry, and wished

he could have stayed longer.


2013

In 2013 the WASH programme had one target under this component; IR 1.6, Indicator 5:

“Increased institutional capacity in the MWI for undertaking planning and coordination

leadership role in various Sector Coordination platforms (WG/TF). Development of Sector

M&E sytem/MIS. Evidence based Advocacy to increase funding flow to the sector. The

target was the development and effective dissemination of hygiene messages. “ It is unclear

whether this target was achieved in 2013. However, the WASH programme did arrange for a

seconded staff member to work with the ministry in the following year.

88 In the 2017 Annual Report, UNICEF states, “The achievements for WaSH in Schools were also significantly

lower than planned, at 7 per cent of the target. This was primarily due to reduced funding and a focus on

smaller scale and higher impact interventions through the connection of schools to the municipal water

network, with vulnerable households connected en route.

89 Source: Interviews at MoWI and WAJ

48

2014

In 2014, WASH programme support together government included:

• Conducting a water vulnerability assessment, including data on per capita water usage

and refugee distribution.

• Ensured that the sector prioritized areas of greatest need for investment.

• Secondment of a senior advisor to lead in the development of the National Water

Strategy.

2015

The WASH programme main target under this component in 2015 was to “build strategic

planning capacity in MoWI and sector and other relevant institutions in Emergency

preparedness and response (Outcome 4, indicator 1). In 2015, the WASH programme

achieved this target by supporting the development of several strategies, regulations, and

policies. They were:

• The Jordan Response Plan (2016 – 2018)

• The National Water Strategy (2016 to 2025)

The WASH programme also developed assessments and tools to assist the government,

including:

• Water and Sanitation Vulnerability Maps, which were upgraded using internal WASH

capacity and more complex criteria.

• An assessment of the Geographic Information System (GIS) structure and capacity of

the government.

• A project monitoring database.

• A Nationwide WASH in schools assessment.

2016

In 2016, the WASH programme worked with the government to achieve the following

results:

• The National Water Strategy (2016 - 2025) was finalized, with direct support from

UNICEF, and released in April.

• The Water and Sanitation Vulnerability maps were updated in September using data

generated from the 2015 census.

• The Jordan Response Plan was updated to cover the period 2017 to 2019.

• A number of result-based tools were developed which have facilitated the generation

of systematic data, including the Comprehensive Vulnerability Assessment.

• The Water Authority of Jordan coordinates amongst the different water companies to

ensure they have sufficiently planned for emergency responses and has Long Term

Arrangements with borehole owners and suppliers which can be activated, as needed.

2017

UNICEF supported the MoWI to review and finalize the baseline values for access to safely

managed water and sanitation services under the SDGs. This included UNICEF working with

the Ministry to strengthen the decision-making framework that ensures accurate baseline

measurements. The transition from MDGs to SDGs for water entails a small reduction of 5.3

per cent. The change for sanitation is significantly higher, at 24.6 per cent. The dramatic

reduction in Jordan’s sanitation rating is expected to redress the imbalance of the sector on

49

water, and to scale-up sanitation interventions, particularly for the most vulnerable. To

increase the percentage of the population using safely managed sanitation services, key

activities were included in UNICEF’s new country programme, and will be progressively

scaled up over the programme period. In 2017, UNICEF also led the publication of National

WASH in Schools Standards.

Component 4: Coordination of the sector at camps and national level

UNICEF began leading as sector coordinator in 2014. Coordination happens at three levels;

national level, camp level, and working group level.


Overall Findings: Stakeholders and UNICEF partners felt that UNICEF was effective as

WASH sector lead. One contractor that serves Rukban said “UNICEF staff are super

cooperative and available anytime of the day or night”.90

The evaluation team also spoke with two of the WASH programme’s donors, The Bureau of

Population, Refugees, and Migration (BPRM) at the United States State Department, and the

German owned development bank Kreditanstalt für Wiederaufbau (KfW). BPRM

commented that UNICEF was effective in leading the sector, and excellent to work with from

a donor perspective. BPRM felt that they had benefited from UNICEF’s guidance in setting

their own strategy for addressing the crisis. KfW appreciated the guidance UNICEF could

provide in constructing infrastructure in the camps and assistance in host communities.

As sector lead, UNICEF has led WASH activity coordination in the Camps, including

coordinating responses to the major storms and flooding that occur in the camps, particularly

Za’atari each year. UNICEF leads contingency planning for these storms and the emergency

reporting structure that allows for tracking storm impact. UNICEF also manages other

emergencies such as the outbreak of Hepatitis A in Azraq that occurred in 2015 and 2016.

As part of camp coordination, UNICEF leads a bi-weekly Za’atari coordination meeting that

has been effective in setting standards and avoiding duplication of effort. UNICEF also

improved the MoWI’s use of data to create water management strategies, and led the sector’s

WASH in schools initiatives.

One area in which UNICEF could improve its coordination is in strategic planning and

communication with partner organizations. Several partner organizations, particularly

INGOs, mentioned that they were surprised by UNICEF’s decisions in the camps, often

requiring that they make major adjustments on short notice, or conduct work for which they

hadn’t planned or budgeted91. Examples included sudden decisions to hand operations over to

other organizations or contractors with short notice, without guidance on the handover

process. Other partners mentioned contracting processes that took a long time only to have a

requirement that initiatives start days after a contract is signed. Decisions such as these, that

do not appear as part of a strategically designed plan, may force risk and costs on to partner

organizations. It may also create risks for UNICEF that its monitoring and accounting

systems cannot bear. These risks could also be mitigated by an annual strategic planning

90 Source: Interview with Gama Engineering

91 Interviews with Mercy Corps, World Vision, ACF, and Oxfam Staff

50

process that would set expectations and guidelines based on clear analysis and reduce last

minute decision making92.

2014

• Met output 1.3, indicator 1 target: established a WASH working group that guided the

sector response.

• Met output 1.6, indicator 1 target: Created a comprehensive WASH master plan

aligned to the National Plan (2022).

• Led the coordination of WASH activities with meetings conducted on a monthly basis

at a national level, in addition to weekly meetings at camp levels. Key achievements

of the coordination are summarised as:

- Incorporation of key WASH issues in the Regional Refugee and Resilience

Plan (3RP) and the Jordan Response Plan (JRP).

- standardisation of hygiene promotion messages and materials.

• Five Task Forces were established;

- WASH in Schools;

- Hygiene Promotion;

- Water distribution network in Zaatari;

- Wastewater in Zaatari; and,

- Development of Minimum Standards.

• Provided support to the development of the Strategic Framework to implement the

National Resilience Plan (2014 to 2016).

• Led partners in an exercise to jointly identify the WASH sector’s needs.

• Facilitated sector coordination that enabled partners to jointly identify sectoral needs.

• Provided technical assistance for a water vulnerability assessment, which identified

areas in need of support to improve water access. The assessment criteria were agreed

to with the Government, which ensured the use of the vulnerability map by the

Government to plan and prioritise areas of intervention.

2015 - 2017

During 2015 - 2017, UNICEF continued its coordination of the sector in the camps, including

the development of contingency plans for a number of scenarios and emergency possibilities.

UNICEF also coordinated the response to the Hepatitis A outbreak in Azraq and undertook

hygiene campaigns on important topics, such as effective handwashing. UNICEF also

coordinated interagency response at the Berm, coordinated task forces on Solid Waste and

Community mobilization, oversaw the transition of 28,000 refugees to Azraq camp, oversaw

the response to the May 2015 contractors strike, and coordinated the Azraq water system

redesign.

Extent that UNICEF Achieved Equity Results

The UNICEF WASH programme operates in a rapidly changing dynamic environment,

particularly in its first two years. Achieving a permanent state of equitable delivery of

services is a very difficult task. The UNICEF WASH programme continuously strove to

92 The UNICEF Jordan Country Programme Document 2013-2017 mentions a two year rolling workplan which

is measured through annual component reviews and a mid-term review in 2015. The evaluation team did not

see evidence that the WaSH Programme had complied with this mandate.

51

provide equitable results. In response to the quickly changing environment, the programme

established comprehensive feedback systems to rapidly respond to problems and complaints

as quickly as possible. Infrastructure was rapidly assessed, and when deemed appropriate,

implemented to get as near equity as possible.

Providing WASH services equitably is an enormous challenge and a moving target. Initially,

all people entering the camps were by definition the most vulnerable and UNICEF’s

challenge was to provide as many as possible a minimum acceptable level of water per day

and sanitary facilities93. As the situation stabilized, UNICEF has more resources to focus on

equitable access and focusing on women and children. In WASH sector coordination

meetings from 2013, the issues of gender, children, and equity are not discussed. The first

documentation of a focus on gender is a brief WASH Sector Gender analysis dated January

2014. The document focuses on safety for women and girls, as well as providing for their

hygiene and sanitation needs.

By March, 2014 the WASH working group began to focus on the safety of women and girls

in relation to WASH services.94 That meeting also reported on an assessment done of access

for people with disabilities.

In 2014 and 2015, UNICEF and its partners made sure equitable results were factored into

the design of its activities in the camps. ACTED recruited female Hygiene Promoters and

Social Mobilisers and Cash For Work personnel to promote equitable allocation of the

resources, incorporate women’s feedback. ACTED also tailored its hygiene promotion

activities in camps. It developed approaches that targeted young children and mothers in

Za’atari and Azraq (ACTED Programme Cooperative Agreement 23-13). UNICEF supported

vulnerable households with WASH facilities and ensured they met standards. A household

sanitation assessment was used to guide the household level improvements.

Action Contre la Faim (ACF) implemented a women-centered model for community

mobilization in Azraq called the Lead Mother Model. The model involves training a lead

mother, who is responsible for disseminating messages on hygiene, sanitation, and water for

12 households. The lead mother works with the other 12 mothers in her area to promote

hygiene, sanitation, and water conservation in their area. As of this evaluation, the initiative

had worked with 732 mothers95.

In 2016, Oxfam paid particular attention to ensuring WASH services for women, teenage

girls, and boys and girls under the age of 12. They noted that large percentages of these

groups felt threatened when seeking WASH services in the camps in previous years. They

indicated that the water and wastewater network, when completed, could solve this problem.

In the meantime, they recruited female staff and held weekly gender segregated feedback

sessions with the community. They also employed a dedicated gender advisor and worked

with UNICEF to provide protection staff.

Improvements in camp infrastructure improved results for women and children. Completion

of the water network in Azraq included a reduction in the distance from households to tap

stands, which made access less burdensome and safer. Access to private sanitary in facilities

93 Source: Interview with UNICEF Staff – 6 August 2018

94 WaSH Working Group Meeting # 3, Minutes of Meeting: March 2014

95 Source: Interview with ACF - 1 August 2018

52

has improved safety in Za’atari, though women in Azraq still struggle with using public

showers due to cultural norms around privacy and immodesty.

The WASH programme has struggled with equity in water distribution throughout the

programme and has made a continuous effort to improve equitable access. ACTED was

responsible for water delivery in Za’atari, and developed a computer based application to

monitor distribution in the camp. In 2015, ACTED highlighted that the network of tanks

around Za’atari provided an opportunity for corruption of the water system. Drivers of water

trucks were pressured to deliver water first to some tanks, leading to other tanks not getting

replenished as often as required96. Oxfam noted that water drivers incompletely filled public

water tanks, diverting water to wealthier households’ private tanks for pay. Oxfam also noted

in 2016 that wastewater drivers illegally charged households to desludge pits97. Hotlines were

established and there were follow-ups to complaints.

Both organizations hoped that the construction of the water network would solve this

problem. However, the water network also struggles with equitable delivery. In interviews

and focus groups conducted for this evaluation, residents of Za’atari noted that each housing

unit is delivered the same amount of water, regardless of the number of people that live in the

unit. Household size in Za’atari ranges from one member to more than eight members.98

Thus, smaller households receive more water per person than larger households. Also, some

residents game the system by removing floats that trigger a shut off mechanism from their

tanks or using suction to increase the flow of water to their units. Both mechanisms result in a

shortage of waters to others, further along the network. One focus group participant in

Za’atari stated that he needs to call the hot line daily and complain of water shortage so that

he can get enough water for his 30+ family members.

It is difficult to point to ways that UNICEF has promoted equitable results in host

communities. While its host community projects are important, and help municipalities with

severe water and sanitation issues, the projects tend to cover a wide area and don’t

particularly target women and children99. Improvements to household facilities did focus on

vulnerable households, but they were relatively small in number. Regarding hygiene

promotion, in the Jordan Response Plan for the Syria Crisis 2017 – 2019, the evaluation team

notes that the overall budget for Result 6.1, to Improve Hygiene Practices in Zaatari, Azraq

and KAP camps, is nearly the same as the budget for Result 3, improving hygiene in host

communities, despite 80 percent of refugees living in the host communities.100

96 Source: Interview with ACTED staff

97 Source: Interview with Oxfam staff

98 WaSH Infrastructure and Services Assessment in Za’atari; March 2017

99 Source: Site inspections of host community projects in Irbid.

100 Jordan Response Plan pg 124.

53

Evaluation Question 5 • What were the strengths and weaknesses of the interventions in achieving results in

the camps/settlements, ITSs and host communities on the following:

o Water (Litre/day, supply, quality, consistency, reliability, accessibility, and

equity)?

o Sanitation (adequacy, appropriateness, accessibility)?

o Hygiene promotion (appropriateness of the messages, timeliness, behaviour

change)?

o WASH in schools?

o National level support?

Camps

Water

In 2012, such as 67 percent of KAP survey respondents in Za’atari saying that they did not

have enough water (Za'atari Hygiene Promotion Working Groups). However, this high

percentage is likely the result of refugees moving from a relatively high-water access area in

Syria to a resource constrained environment. The shortage may also have been due to

restrictions on water delivery put in place by the Government of Jordan. By the 2013

iteration of the KAP survey, 80 percent said that they had enough water, a clear improvement

in perception as well as reality. Service levels were strained when the population numbers

rose to a peak in 2013/2014 and have since improved as population numbers declined and

steadied. An evaluation focus group participant in Za’atari commented that “before there

were complaints about the quantity of water supply. Up to 30 people would share water from

one tank101. There are fewer complaints now except from those not connected yet to the

network or those who suffer from low pressure.”

For both Za’atari and Azraq, the interventions were effective. Residents of both camps

received at least 35 liters of water per day, well in excess of the 15 liters per day that the

Sphere handbook defines as sufficient (The Sphere Project, 2011). The evaluation team

inspected water tanks outside a shelter in Za’atari and found the tanks to be. easily accessible,

directly outside the home, and water was present in the tank. Azraq’s two boreholes (2 and 3)

combined produce 120 cubic meters per hour. The water is treated, disinfected and sent to

the network. The treated water is pumped to storage/pumping structures and then to the

network tap stands. The evaluation team believes that this structure represents an appropriate

level of service given available funds. The system is a lower cost option for delivering safe

water as close to households as possible. UNICEF reports that some households use hoses

from the tap stands directly to their residence.

In Za’atari, the evaluation team’s opinion is that the installation of the water network is both

needed and appropriate. Za’atari is the size of a city and could exist for some time to come.

Water improvements such as network water connections to dwellings reduce NRW costs and

improve customer satisfaction and willingness to pay for the service. It makes water and

sanitation services more accessible and equitable to all of the camp residents.

101 One tank holds 1,000 litres.

54

Sanitation

While camp sanitation systems may not be ideal in

terms of cost efficiency, given political realities and

physical limitations, evaluation team site inspections

revealed that they are appropriate and adequate. The

exception is the wastewater management in Azraq

camp, where the treatment plant is non-operational

and septage is still being hauled vast distances ant a

huge expense.

The system in Za’atari, described below, is an

innovative approach that works well and is cheaper to

install than a traditional sewer system. The boreholes

and water systems at both Zaatari and Azraq produced high quality, and are well designed

systems are being run by high quality, experienced Jordanian contractors with the assistance

of beneficiaries working on a cash-for-work basis. These arrangements contribute to

sustainability.

In Za’atari, UNICEF initially constructed WASH blocks that were primarily used in the early

days upon refugee’s arrival to the camp. Za’atari’s residents were uncomfortable with public

facilities, which were necessary to accommodate the large numbers of arrivals. As the camp

evolved, residents progressively took responsibility for constructing their own WASH

facilities inside their shelters. Leading to over thousands of open pits throughout the camp.

Residents that the evaluation team interviewed said that, before connection to the sewerage

network they had access to septic tanks which they could have emptied upon request102.

Although, residents reported that sometimes it was difficult to get a response103.

UNICEF addressed the shortcoming in infrastructure by constructing communal tanks and

connected them, which doubled the volume of wastewater collected in the camp. The

evaluation team observed a household toilet, which was a pour flush latrine within a

household compound. The toilet was fully functioning and connected to the sewage network.

Household members stated that the toilet had functioned this way for 6 months.

Currently, UNICEF is connecting the communal household tanks to a constructed treatment

plant via the wastewater network.

An issue that came up frequently in focus groups and household interviews was the quantity

of dust that is produced by water system construction. Interview respondents and focus group

participants stated that the dust results in health issues such as triggered allergies and

respiratory infections.

For Za’atari, the evaluation team believes that the common septic tank and small diameter

sewer is an appropriate wastewater collection model. It has enabled the camp to reduce its

septage trucking from a fleet of 39 trucks to 5 trucks, which will no longer be needed once

102 Source: Interviews with ACTED staff

103 Household interview in Za’atari.

Figure 6 Very clean household latrine observed in

Za'atari camp.

55

the network is completed. The gravity flow from common septic tanks to the sewer trunk

line, and then to the lift stations where it is pumped to the treatment plant is a major

infrastructure improvement. These are both appropriate and necessary for the number of

people served. The treatment system composed of both membrane bio reactors and trickling

filters is effective. The system functions well and provides the level of treatment required by

national law, which is a key requirement. Obtaining labor for cash from camp residents also

helps reduce costs and add value for money104. It also contributes to sustainability as workers

have a sense of pride and ownership.

The biogas plant, a project implemented by the Food and Agriculture Organization, is also an

appropriate technology that should yield positive benefits to the energy balance. Mixing food

waste with septage biosolids is a recognized method of maximizing the volume of biogas

produced from anaerobic digestion.

In Azraq, the WASH programme has implemented gradual improvements of basic service. In

2015 shelters in Azraq were not connected to a greywater network, meaning greywater and

mud tends to accumulate in the ditches surrounding the shelters. UNICEF with partners

constructed a plot-level grey water network, and provided income generating opportunities

within the camp contributing to camp cleaning and sustainable waste management practices.

Communal toilets remain unpopular in Azraq. Women that the evaluation team interviewed

in Azraq reported that they did not like having to fully dress and cover their hair every time

they go to a toilet; children get dirty on their way to and from bathrooms, and pit toilets have

holes that are too big for young children. As a solution, refugees often construct bathroom

inside their kitchen space, but this leads to problems related to odors and hygiene. Also, there

is no space for drying laundry, and it is culturally inappropriate to dry underwear in a place

where it can be seen by others. Respondents also commented that the latrines for women and

men, though separate, are very close to each other and perceived as not safe105. That said, the

2017 Azraq KAP survey indicated that 100 percent or respondents said that they used the

WASH blocks and that they felt safe, compared to 96 percent in 2016.

For Azraq, the installation of the Moving Bed Biofilm Reactor (MBBR) system, which

provided by a donor turned out to be a poor choice, (although the

rationale for selecting it was at the time thought to be valid) and did

not meet the needs of the people served or the program in general.

Wastewater strength was higher than anticipated and after lots of

money spent, the system was eventually abandoned. Wastewater is

now extracted from tanks and transported also at great expense.

Until there is some centralized wastewater treatment option,

septage will require hauling some 90 KM away106.

The evaluation team believes it is surprising that the Azraq camp

would have more difficulty in establishing a wastewater system as

104 UNICEF reports that it has supported skills training initiatives in Za’atari to upgrade from low skilled to

higher skilled labor.

105 Comments from interviews and focus groups in Azraq

106 Source: Azraq site assessment

Figure 7 Rubbish Bin in Azraq

56

it is neatly laid out, compared to Za’atari, which has developed an elegant and lower cost

solution and has evolved into something that could become sustainable over the long term as

economic development in the camp increases.

Solid waste collection in the camps is adequate. The evaluation team observed rubbish bins,

which appeared substantial in terms of their volume and appropriateness. Large bins on

wheels that help minimize vectors such as flies and rats. Camp residents reported that rubbish

is collected once per day. Focus groups that the evaluation team held in Za’atari reported that

solid waste collection had improved over time.107

Hygiene Promotion

The absence of assessment data with respect to the early stages of the response made it

difficult to determine what the key public health issues were in relation to hygiene, and the

behaviours that needed to be targeted, at least in the programme’s initial stages. The first

Azraq camp KAP was conducted 21 months after the camp opened, making it difficult to

establish an initial baseline. This contrasts to the initial Za’atari KAP survey conducted only

4 months after opening. The Za’atari baseline showed considerably lower perceived health

status levels.

A general observation by the evaluation team was the lack of a consistent connection to the

health status of the population and the infectious disease risks prevailing at various times of

the year. It is acknowledged that the health sector would house such data also but the

connection to health status and infectious disease risk was not always present in the KAP

surveys. For example in the 2013 KAP survey for Za’atari, reference is made to frequency of

respiratory problems, and a suggestion is made to conduct messaging in 2014 (at least 18

months after the camp opened) on methods of preventing respiratory infections. Despite this

suggestion, the only disease related questions in the 2014 KAP survey relate to diarrhoea. A

notable exception was the Hepatitis A outbreak that afflicted Azraq.

A notable feature for Za’atari and Azraq was the strong commitment to repeating KAP

surveys in both camps on an annual basis enabling one to measure change over time.

The evaluation team did not have access to hygiene data with respect to Host Communities,

those in the ITSs, or the settlements (Hadalat and Rukban). However, the evaluation team

understands that given the circumstances it was not feasible to gather hygiene data in those

areas.

In Za’atari, the baseline KAP survey (November 2012) indicated that hygiene knowledge was

low. Only 50 percent named handwashing before eating as a diarrhoea prevention

mechanism, and 36 percent named handwashing after using the latrines. This result was

explained by UNICEF staff, who indicated that the residents of Za’atari at this time were

coming from rural areas of Syria where levels of education were low. The percentage of

people measured through observed practice though was high (an unusual result) with 55

percent of adults observed handwashing. As a point of comparison, an estimated global

average for the number of people who wash their hands with soap after going to the toilet is

just 19 percent (Freeman & al, 2014).

Over time the KAP survey results for Za’atari showed an improvement. In terms of

knowledge for example the baseline indicated that only 50% knew that handwashing before

eating helped to prevent diarrhoea which rose to 76% in the 2015 survey. In 2013 for

107 FGD 1, 5 August 2018.

57

example the KAP survey reported an improvement in handwashing, with 98 percent claiming

to wash hands with soap and water - up from 68 percent in 2012. This high level was

supported/triangulated by observed data where 95 percent of assessors reported seeing

evidence of soap in the house. This particularly high reported result should be viewed with

caution and suggests potential bias in the results, as the results are exceptionally good.

Nevertheless they do show an upward trend. (Za'atari Hygiene Promotion Working Groups).

Some direct observations by the evaluation team also demonstrated high hygiene standards in

areas such as water storage, cleanliness of toilets and general cleanliness of the household.

The exception to continuous improvement was in 2015 when there was a drop in General

Cleanliness from the 2014 results from 88 percent reporting their area was generally clean to

71 percent reporting in 2015. The KAP survey reports don’t provide much in the way of

analysis to explain results but in 2015 this might have been explained by the increase in the

camp’s population which was at its height in 2015 at a reported 120,000 (Za'atari Hygiene

Promotion Working Grouip, 2015).

In the area of hygiene supplies, interview respondents in Za’atari indicated that they did

receive hygiene items, but not enough of certain items. For example in the 2015 KAP survey

88 percent reported receiving soap in the distribution but 88 percent also reported it was not

enough. Other items commonly felt lacking in 2015 were laundry detergent and shampoo in

addition to soap. From a gender perspective only 21 percent of respondents reported

receiving sanitary pads in the 2015 survey, an indication that perhaps the contents of hygiene

kits were not gender proofed. Again the results of the KAP survey do not offer a rationale

behind such a low result.

Note that of the 88 percent who answered that soap received was not enough, 86 percent still

managed to access extra soap in the market. UNICEF indicated that the people of Za’atari

had more means to access hygiene items in the market and therefore did not need “free”

distributions of such items.

This result was supported by the evaluation team’s interviews and focus groups. During the

evaluation, respondents provided feedback on hygiene related items that indicated that

people did receive the items they needed but sometimes not enough of certain items like soap

powder and sometimes at irregular intervals. Hygiene was communicated in a number of

different ways including through community level hygiene promoters, children’s hygiene

clubs, campaign days linked to key global days such as global handwashing day and use of

the mass media. A disappointing aspect for the evaluation team was the paucity of

Information, Education, and Communication (IEC) materials provided for review108.

In Zaatari, one of the useful indicators to highlight effectiveness of the WASH programme

overall was reported general health. In the 2013 survey 58 percent answered their health was

good or excellent which rose to 81 percent by 2015.

Azraq opened in April 2014 quite some time after Zaatari. The first survey reviewed was

from January 2016, 21 months after the camp opened which is a significant time lag before

undertaking a KAP survey . The fact that the survey was undertaken in winter indicates some

results, such as satisfaction with the quantity of water supplies could be very different if

undertaken in the summer months when the demand for water is much higher. The baseline

108 The UNICEF WASH Programme provided materials used for hygiene promotion after the evaluation period.

These included flyers, posters, and other materials promoting practices including dental hygiene, water

conservation, sunstroke prevention, maintaining good health in winter, and other key practices.

58

survey in 2016 does indicate a high level of satisfaction with the water supply, with 95

percent expressing the supply was adequate or very adequate. Access was good, with 46

percent collecting water within 15 minutes. Quality also scored highly with 90 percent

responding it “satisfies” them. Results in 2017 compare favourably, with a higher percentage,

up from 44 percent in 2016 to 68 percent in 2017 satisfied. Satisfaction with quality dropped

from a high of 90 percent in 2016 to 75 percent in 2017 with the issue of bad taste and/or bad

smell cited as the reason. No explanation is given for this result.

The Border Settlements (Rukban and Hadalat)

Water

The intended results at the settlements were to

provide water to the Rukban and Hadalat

Refugee Camps as effectively as possible while

achieving value for money. The evaluation team

inspected the equipment and operations of the

water system at Rukban. The borehole and

related equipment are of high quality, the design

is appropriate, the construction is of high quality,

and the operations and maintenance activities are

performed properly and on schedule. Staff

working for the contractor that, UNICEF hired to

supply and operate the system at Rukban

reported in an interview that UNICEF always

purchases high quality equipment.

The borehole in Rukban was drilled in December

2016 with an estimated yield of 80 m3/hour. The system contains: i) the production well

(borehole), ii) chlorination, iii) sand filtration, iii) de chlorination, iv) reverse osmosis water

filtration, and v) post processing chlorination. At the time of the inspection, chlorine residual

was being maintained at between .6 and .8 mg/l, which meets or exceeds international

standards. The throughput of the system is estimated at 56 m3/hour post-treatment. This

water treatment plant has a capacity of 800 cubic meters of post-treatment water storage. In

addition to the borehole at Rukban, UNICEF has constructed two pumping stations; the old

pumping station (with two sets of storage tanks at the old station) and the new pumping

station (with a storage capacity of 760 m3). “The old pumping station is located

approximately 3km west of the Rukban borehole/reverse osmosis plant and the new pumping

station is located 10km further west of the old pumping station and is approximately 13km

west of the Rukban borehole/reverse osmosis plant. The two pumping stations pump across

No Man’s Land to water distribution systems along the Syrian border and into the camp

(Terms of Reference for Service Contracting - Rukban).” The new borehole came on line in

September 2017. It has a capacity of 85 cubic meters per hour, which is greater than the

expected 70 cubic meters per hour, and is 572 meters deep. Today the new borehole and

treatment unit are connected with the old infrastructure by pipeline as the best way to secure

the most water possible for the refugees.

This treatment system is sophisticated and costly, but warranted and necessary due to the

concentration of certain constituents of the raw water, and the fact that the system has to

work properly and consistently with no down time. Water quality of the groundwater is poor

with TDS at 3,230 us/cm and high levels of fluoride and iron. In consideration of the very

high cost of the trucking operation, full cost recovery will be achieved from the savings alone

within a matter of months. So far, the investment has saved almost 1 year of trucking, which

cost 18.5 JOD per cubic meter. At 600 cubic meters per day, that is a cost of 4 million JOD

Figure 8 Power generators and fuel storage at Rukban

water system. Extra power generation and fuel storage

capacity minimizes the potential for water system down

time.

59

per year that is not being spent on trucking. The UNICEF team reports that the system

implementation is 85% less expensive than trucking operation and delivers 30% more water.

Other important reasons for installing the water system include:

• increased volume of water resulting from no longer needing to comply with Army

restrictions;

• eased management resulting from reduced tension with the host community in

Ruwayshed;

• improved water quality and more consistent chlorination; and,

• improved consistency of water delivery resulting from removal of truck and trucker

management.

Therefore, the value for money from this investment is high.

The design of the system is strong with a capacity for multiple contingencies including leaks,

power outages, or effluent quality issues (figure 2). There is 800 cubic meters of storage at

the borehole site with tanks with the ability to operate independently in case of leakage. The

system appears easy to operate as well. The UNICEF hired contract that manages the system

maintains staff on-site 24 – 7. Staff performs daily O&M including: i) pumping from the

borehole to the tanks, ii) checking the chemical concentrations every 2-3 hours, iii) routine

cleaning of the sand filter weekly, and iv) repair and replacement of parts as needed.

The evaluation team held one focus group at Rukban. Participants claimed that water has

high levels of chlorine that is solely detected by smelling the water itself. One participant

elaborated that she believed the chlorine caused intestinal inflammation and diarrhea in two

of her family members. This could refer to enteritis (an intestinal inflammation that is usually

accompanied by diarrhea, or perhaps other medical cases that are related to gastrointestinal

diseases. All participants agreed that they were not willing to wait for the chlorine to fully

evaporate before using their water.

The team observed a chlorine test of the water coming out of the tank that goes through the

pipe that is directly connected to the one of the tap stands. The indicator showed chlorine

levels are at 0.5-0.7 mg/l.

Participants in the FGDs indicated that they also have experience multiple water cutoffs and

sometimes water was supplied 2 hours a day, sometimes more, sometimes less. They believe

that cut-offs take place when there is a high demand on the tap stands. While this may have

been the case during the earliest days of the water operations, it does not appear to be the case

now, or at least since April of 2016 when UNICEF staff created a log book for operational

issues, which indicates very minimal disruptions.

UNICEF invested close to $1 million for the water system at Hadalat but it was only

operational for 6 months. At the time when the investments were made, Hadalat was served

by trucking operations that originated at Ruwayshed. The trucking was very expensive and

unreliable with truck drivers going on strike demanding more money. The decision to

improve the borehole at Hadalat and develop the water treatment system was sound, and

actually saved 6 months of trucking expenses. It is also noted that water from Hadalat was

used to supplement the water in Rukban to avoid increasing the load on Ruwayshed.

It was impossible to tell at the time that the camp would be abandoned, so it is difficult to

criticize the decision to invest in new infrastructure. UNICEF reports that the cost for the RO

unit and the new pumps/gensets were recovered by the operation of the system, as well as

supplementing water to Rukban. The system remains operational should a viable water

supply be required for renewed trucking operations or should the camp become populated

60

again. UNICEF also reports that the system was handed over to WAJ in November 2018. The

system remains operational should a viable water supply be required for renewed trucking

operations or should the camp become populated again.

Sanitation

The evaluation team could not enter the camp to visibly inspect. However, FGDs on these

points offered the following:

• Numbers of toilets, privacy, adequacy and functionality is currently acceptable.

• Vandalism of public and communal toilets early on was an issue that helped drive the

movement towards individual toilets, one per family which is mostly the current case

• Solid waste management is adequate

• Desludging is being conducted on a regular basis. Some odors are observed but

overflows are rare.

While the FGDs were interesting, it should be noted that they were conducted under the

observation of government minders, which may have influenced the comments of the

participants. For example, while the FGD participants say they are receiving desludging

services which are just fine, UNICEF notes that in actuality, no formal desludging has been

accomplished since June of 2016.

Hygiene Promotion

Hygiene promotion in the settlements seems to be limited to the distribution of hygiene kits

on a periodic basis. Initially, the plan was to distribute once a month but focus group

discussions revealed that gaps in distributions may have lasted as long as four months. The

dynamic situation and insecure environment limits, and often prevents, UNICEF’s ability to

access the area, which explains gaps in distribution. Contents of the kits included collapsible

jerrycans, detergent, sanitary pads, solar lights, soap, toothpaste, shampoo, and diapers. In

addition, UNICEF have delivered training to 20 – 25 Community Health Volunteers for

delivery of face to face hygiene promotion in Rukban. There appeared to be a lack of

Information, Education and Communication(IEC) materials provided to the settlement

population. IEC materials could have been included in the hygiene kits distributed

periodically.

UNICEF reports that hygiene promotion efforts were carried out through ACF until the June

2016 attack, when activities were suspended. Following the attack, UNICEF distributed

flyers and soap through its clinic, as well as carrying out training to health workers. Activities

were limited and difficult to arrange due to extreme security measures imposed by the

military.

ITSs and Host Communities

It is difficult to gauge the WASH programme’s success in ITSs and host communities. The

evaluation team had no access to numbers and locations of refugees living in host

communities and the percentage of those served by the programme. It is also difficult to

know if conducting projects in host communities serves UNICEF’s WASH mission in

Jordan. Large donors like USAID and GIZ work with the MoWI to do large infrastructure

projects. UNICEF’s projects are small and general.

The projects that the WASH programme implements in host communities and ITS are

effective. For example, in Irbid, where the evaluation team observed WASH programme

initiatives, the needs of the host community are extreme regarding water and a key focus of

UNICEFs efforts. While just a fraction of the amount spent on camps is going to the host

community, interventions that UNICEF does implement are well planned and executed. In

61

Irbid, the interventions were sewer line improvement projects in city center, the rehabilitation

of two boreholes, and a pumping station improvement project. The interventions were

appropriate given the level of funding. Most of Irbid receives water every 5 to 8 days, so it is

extremely scarce. However, the water utility has a NRW of almost 50 percent. In addition,

more boreholes appear to be needed. It is assumed that the aquifer is well mapped in Irbid,

however, operators report that static water levels are down almost 24 meters in one year.

Operators don’t know if this is aquifer wide or a localized phenomenon because proper

monitoring is not conducted. Modeling the aquifer through drawdown testing would be

extremely useful. It would indicate if drilling a new borehole and connecting it to the system

would be useful.

The evaluation team visited two ITSs; camp 702 and camp 709. Life in camp 709 is harsh but

basic services of shelter, toilets, water and sanitation appear to be met. The people that live

there are extremely vulnerable and UNICEF aid is reaching them. One-hundred and twenty

people live in the camp and work for low wages in agricultural endeavors. They have one

latrine for every 20 people. There is a school with a UNICEF trained teacher on site. People

pay 3 JOD for potable water per cubic meter from a vendor, or .9 JOD per cubic meter for

piped water meant for agricultural use.

Respondents to evaluation interviews

reported that they find the water storage

volume and number of toilets lacking.

They would prefer to have one of each per

family instead of having to share with

other families. It was reported that there is

a fear that toilets and water tanks will be

stolen if too many are present. However,

the family head said he would take

personal responsibility for these if

UNICEF would just provide more. There

is no indication of waterborne disease, and

the health of the children seems good

according to the residents. When asked if UNICEF related services have been improving over

the years, the residents said yes, very much. UNICEF reports that it would like to do more in

ITSs, and is encouraged by the benefits of innovations such as the modular collapsible toilet,

which UNICEF developed. However, lack of donor interest, and earmarking of funding for

WASH in ITSs, limits their ability to make progress in the area.

Camp 702 is a cluster of 12 houses with 72 people. It is a nomadic agricultural camp similar

to camp 709. Residents move seasonally between this site and a similar site in the Jordan

Valley. Residents have refugee status, but it is still considered illegal to live in the ITS.

People fear they will sent be to the formal camps, even though they have employment and

livelihood opportunities here.

Sanitation

In the ITSs, UNICEF considered the need for on-site wastewater systems and determined that

pour flush toilets and offset pit latrines would be most effective in terms of cost and function.

This is the sanitation scheme in camps 702 and 709, where such latrines are shared by 3

families. For the most part, the pits were in poor shape, not having fly-proof or safe covers.

Venting was also absent, so while the need for sanitation was considered, at least in this

regard the implementation was poor. Upgrading latrines from single pit systems to proper

Ventilated Improved Pit (VIP) systems would be a good next step.

Figure 9 Unimproved pit latrine cover at an ITS near

Amman

62

WASH In Schools

Between December 2014 and April 2015, the WASH programme assessed 3,681 schools,

identifying needs and prioritizing schools for receiving WASH services. Of this number 400

have been reached. There were significant gaps to correct for all schools to reach the National

WIS standards, which were only finalized and released in December 2017. For example, 35

percent of schools (1,297) failed to meet the standard for student to latrine seat ratio.

It is important to note that the Ministry of Education requires all school interventions,

including new classrooms and WASH facilities, to be delivered as a package. This adds to the

challenge of delivering WASH in schools as it must be combined with non-WASH

components.

The evaluation team observed the WIS project implemented by WASH programme partner,

the Norweigan Refugee Council (NRC). The quality of the project was clear from NRC’s

report dated November 2017. The programme had a clear plan to both address infrastructure

deficiencies (WASH blocks built in 3 of the 5 schools NRC targeted), and also build capacity

(169 teachers trained on classroom activities on cleaning and hygiene. NRC produced a

Manual with guidance for schools on how to carry out the “soft” part of the WASH in

Schools programme). The evaluation team noted that NRC’s KAP survey was very high

quality and its report was objective and informative.

For WASH in schools, there is a complex regulatory environment when it comes to

connecting new WASH infrastructure to existing sewer systems, as was found out in Irbid,

where UNICEF invested in WASH blocks, only to have them still not connected to a sewer

after more than a year. The lesson learned is to secure the permits prior to beginning

construction.

Coverage with respect to WIS in schools would appear to be low at roughly 10 percent of the

total assessed (3,681). This finding should be put into context for a number of reasons.

Firstly, UNICEF are not the only agency working on WIS. Secondly, not all schools were

below standard and therefore did not require targeting. As mentioned above 35 percent did

not meet the standard for toilet seat to student ratio. Lastly, in the schools targeted water

supply was extended into the community adding to the per student cost and therefore limiting

the coverage possible with limited funding.

UNICEF did oversee several successful WIS pilots, such as the NRC project mentioned

above. However, the successful results often didn’t receive follow-up for a variety of reasons,

including the lack of suitable partners, lack of capacity within UNICEF to conduct

community mobilization109, and lack of donor support. Partners also reported that the

coordination of WASH in schools was a sub-group under the WASH Sector and only “ran

sporadically.110” NRC also said that coordination engaged WASH in Schools Implementing

Partners but not the Ministry of Education itself, who it is reported were invited but did not

attend. NRC records indicate that UNICEF did not promote or try to expand results, possibly

indicating a lack of focus on the WIS portion of the WASH programme.

UNICEF decided to carry out the WASH in Schools KAP survey in December 2016 and

repeated again in May 2017. This was too short a time frame to adequately assess changes in

knowledge and behavior.

109 Statement made in UNICEF staff interviews with those that managed WIS efforts.

110 Source: Interview with NRC Staff – 29 July 2018

63

National Level Support

UNICEF’s approach to national level support is solid at a basic level. Despite the highly

politicized nature of the WASH sector in Jordan, UNICEF has provided assistance and

guidance that the government appreciates. Examples include assisting with the development

of the National Water Strategy and then disseminating it to promote its implementation.

UNICEF also commissioned and developed the National WASH in Schools Standards in

cooperation with the government and facilitated the distribution of the standards. UNICEF

also conducted national vulnerability mapping, developed project databases, and developed

GIS hubs that will serve the Government of Jordan for many years. UNICEF’s challenge will

come in future phases as it works toward sustainable solutions for the camps that the GoJ

might resist.

Efficiency

Evaluation Questions 6 and 7 • To what extent did the programme use available resources in an economic manner to

provide WASH services and facilities in camps/settlements and host communities, to

ensure child rights, equity and gender equality?

• To what extent were the costs to deliver water and wastewater services, as well as

solid waste management, rationalized and optimized in the camps and settlements to

ensure Value for Money? How do these compare to similar situations at the local or

regional level? Could more cost effective operations/interventions have been

undertaken at an earlier stage?

The evaluation ToR asks that the evaluation team assess the WASH programme’s economic

use of resources and the extent to which costs to deliver water and services optimized.

Assessing the programme’s use of resources is difficult for several reasons. First, UNICEF

Jordan’s financial systems are organized to manage the organization’s risk, manage

compliance issues, ensure operational solvency, and protect the organization against fraud111.

Neither UNICEF Jordan nor that WASH programme track expenditures for management

purposes112. The evaluation team could not obtain documentation that demonstrated

expenditure by year, activity, programme component, or beneficiary group. Also, the

programme does not track its indirect costs or general and administrative expense rates

related to the programme, making it difficult to know the resources required to manage and

execute the programme or to compare that to other similar programmes. UNICEF Jordan

reports that UNICEF deducts 8 percent of all raised funds for HQ support and limits

international partners to 7 percent allocation to partners headquarter costs. Additionally, a

maximum of 7 percent is used to pay for Salaries of WASH staff and 5 percent is taken for

cross sectoral support costs. The evaluation team did not see year by year information on

these costs however, which would have indicated whether efficiencies were realized as the

programme progressed from year to year and as new innovations were implemented.

111 UNICEF Finance staff interviews.

112 After the evaluation period, the WASH Programme provided estimates of its efficiency gains and cost

savings. Those estimates are attached to the document as Annex I.

64

The WASH programme was able to provide some documentation regarding contract

expenditure and Programme Cooperative Agreements (PCA)113. The evaluation team was

provided information that accounted for $268,614,178114 of the approximately $355 million

that programme managed between 2012 and 2017. However, the evaluation team notes that

some significant agreements were missing from the documentation, such as PCAs with

Mercy Corps, which financed Za’atari’s boreholes among other activites, and the PCA with

the Norwegian Refugee Council, which implemented WASH in Schools projects on behalf of

UNICEF115.

Extent to which the programme used available resources in an economic manner to provide WASH services and facilities in camps/settlements and host communities, to ensure child rights, equity and gender equality

The evaluation team understands this question to ask for evidence of the extent to which the

programme used its resources in an economic manner, and the extent to which child rights,

equity, and gender equity were part of the resource use equation.

Broadly, the WASH programme’s resource allocation has five drivers in order of

importance116:

1. WASH service and facility coverage for people living in the camps and settlements.

2. Improvements that increase efficiency in coverage in camps and settlements, as well

as continuously seeking to ensure equity of service coverage and quality of services.

3. Improving water systems in host communities and ITSs.

4. Wash in Schools

5. Sector support for the MoWI.

This decision-making hierarchy developed as a consequence of the programme’s evolution,

the earmarked nature of its funding, allocations of areas of responsibility in response to its

programme management structure. Camps and settlements were allocated resources first

because, in the early days of the crisis, they were seen as having had no other options for

WASH services other than UNICEF, as opposed to refugees who could prove sponsorship

and had government or private options in host communities. After UNICEF was certain basic

needs in camps and settlements were covered, resources were invested in efficiency

improvements in the camps and settlements. Budget that was available after the effort in the

camps and settlements was allocated to host communities, ITSs and Schools117.

Expenditure and Efficiency - Camps

In July 2012, Syrian refugees had two legal options. Refugees that could provide proof of

sponsorship could live in a host community. Refugees that did not have sponsorship accepted

a tent in Za’atari. At this stage, the international community mobilized and assigned degrees

of urgency to the populations that needed assistance. The population that was under the direst

circumstances was clearly the population moving into the Za’atari camp. Believing that this

113 PCA’s are the form of agreement UNICEF uses with non-profit partner organizations.

114 This figure includes the amount that UNICEF reported on each contract, and the value of the PCAs’ budgets

that the evaluation team were provided. The evaluation team could not verify the degree to which partner

organizations had spent or exceeded the budgeted amount included in each PCA.

115 UNICEF Jordan shared these documents after the evaluation period (29 November 2018)

116 This hierarchy is evident in UNICEF Jordan Annual Reports and it its allocation of resources.

117 Source: UNICEF Staff Interview – 6 August 2018

65

population had the most urgent needs, and that the crisis would not last long, UNICEF and

other international actors in Jordan quickly mobilized to provide WASH services and

facilities however they could be obtained.

There is no indication that the cost of providing water services in the early stages of the

intervention was rationalized. UNICEF was operating in an emergency situation and cost

was secondary to providing lifesaving water, which involved expensive trucking operations,

and renting mobile sanitation facilities until more permanent arrangements could be made.

Design mistakes were made that would later prove costly, such as the WASH Blocks

constructed in Za’atari that would eventually have to be dismantled because they didn’t meet

Syrian cultural standards for cleanliness or privacy, and the location of Za’atari over one of

Jordan’s largest aquifers, which provides 37 percent of Jordan’s water.118

UNICEF took steps to bring down the unit cost of providing water to refugees in the camps.

As figure 10 below shows, the evaluation team estimates that between 2013 and 2016, the

cost of delivering water to refugees in the camps dropped from 3.34 Jod/m3 to 0.24 Jod/m3.119

Figure 11 presents UNICEF’s estimate of cost savings realized in Za’atari. The drastic drop

between 2014 and 2015 was a result of UNICEF reducing Za’atari’s dependence upon

external water trucking by over 50 percent through new storage facilities and the

operationalization of a third borehole.120 Another contributing factor was the operalization of

a borehole at Azraq camp which led to external tankering. The drop in cost between 2015 and

2016 represents phase 1 of the water network coming into operation.121

The expense and inefficiency of the

emergency phase of the response was a

natural result of the urgency, limited

capacity, limited options, and limited

resources that were available. As soon

as the camps were stabilized, the

WASH programme began taking steps

to reduce costs and improve delivery.

It also mitigated risks by reducing

vulnerability and inconsistency in

water access caused by trucker strikes.

In Zaatari and Azraq, the investments

in borehole construction and

upgrading, water treatment and

pumping equipment, and eventually

network development, were all made

with the specific goals of improving

quantity, quality, access and the

efficiency of the entire water service delivery operation, while reducing the high costs of

118 Source: UNICEF Jordan WaSH Specialist interview

119 Financial data estimated from UNICEF partner budgets included in PCAs, RAM reports, Annual Reports,

and other sources of demographic data. Costs of delivering water come from partner organization budgets

and may not accurately reflect actual expenses.

120 https://www.unicef.org/appeals/files/UNICEF_Syria_Regional_Crisis_SitRep_Sept_2015.pdf

121 UNICEF Annual Report 2016 - Jordan

Figure 10 Cost of providing a cubic meter of water by year

based on partner budgets

66

trucking. To a great extent, UNICEF accomplished this goal through these investments. The

investments were optimized by: i) utilizing high quality equipment and components, ii)

training local operators, and iii) utilizing local labor for network construction.

Figure 11 Water provision cost savings in Za'atari (Source: UNICEF WASH Programme)

There is some indication that trucking water into Za’atari camp went on longer than it should

have, draining the programme’s resources. Construction of the network began in 2014 and

was not completed until 2017. The delay was related to securing funding for the pipeline, and

a long process to approve the design of the network. However, in Azraq, UNICEF

transitioned there in there in May/June 2017 and it appeared to be very smooth with no

subsequent issues or need to supplement with external water. There, the system is run by

gravity, while simultaneously reducing the walking distance for the camp residents. UNICEF

didn’t have the funds for these improvements, but instead advocated across the sector

successfully for others to mobilize, which they did. This is an important indicator of success

of UNICEF's leadership role, not only in addressing the immediate crisis, but the broad array

of stakeholders as well.122

Sanitation (WASH facilities, solid waste management and desludging)

Initially wastewater management was inefficient. Issues such as contractors claiming to

remove up to 40 percent more sludge than they actually removed burdened the programme.

Drivers claimed trucks were capable of holding more volume than they actually did and

sometimes carrying waste in to the camp from outside, which was subsequently charged to

the UNICEF operation123. Also, on some occasions, it was learned that drivers claimed for

the official licensed volume of the trucks, but they had been modified to carry less. This

inefficiency was fixed by UNICEF introducing a comprehensive 3rd party monitoring system

whereby trucks were physically inspected entering and leaving the camp, and the dispatch

point of the trucks was controlled through the voucher system.

UNICEF developed the Waxi app to ensure that wastewater was collected on a priority basis

and to reduce the risk of full tanks (people put blankets and other materials into the network

and tanks) – this allowed the efficient and optimal collection of wastewater across the camp

and ensured a near continuous flow of wastewater into the tanks at the Za’atari Wastewater

Treatment Plant, which ultimately kept the drivers happy.

122 Source: Interviews with UNICEF senior staff.

123 Source: Interview with UNOPS

67

The wastewater system in Za’atari

evolved from unimproved pit latrines to a

system of shared septic tanks, a

wastewater collection network, and

eventually a wastewater treatment and

reuse system. The program realized

value for money by: i) utilizing a lower

cost and innovative wastewater

collection scheme, ii) utilizing cash for

work in network construction, and iii)

utilizing appropriate wastewater

treatment and reuse technologies (MBBR

and Trickling filter) which appear to be

currently working reasonably well. The

network and the internal system allowed

UNICEF to have leverage with the

desludging contractor. UNICEF also rebid services and infrastructure regularly to reduce the

ultimate cost.

In Azraq, the wastewater treatment plant is not functioning and has been abandoned. The

anecdotal evidence from stakeholder interviews indicates that this was a MBBR system

(moving bed biological reactor) that was a donation and sized based on wastewater

characteristics from Zatari, and before Azraq was even inhabited. The wastewater strength

turned out to be very strong, and also toxic, which killed the media regularly. UNICEF was

assured that the treatment facility would be appropriate and it was a donation and so they did

not pay for it (they paid for the installation and configuration) and it was a better alternative

than expensive long distance tankering. UNICEF is currently looking at local alternatives

In the area of Solid Waste Management, there were initially 180 waste pickers employed in

Za’atari. This was rationalized down to 40, who only picked waste in neighborhoods once per

week in an attempt to improve attitudes and practices with regard to waste among residents.

The reduction is staff improved efficiency124. Unicef was the driver of this, but experienced

resistance from the other NGOs, something that ultimately reduced the efficiency and

effectiveness of the activity.

Another innovation that UNICEF developed was the “Waste Taxi”, which served as an on-

demand removal system. It was a predictive model designed to address with the various

different volumes of wastewater tanks that were introduced when people began constructing

their own private washing and toilet facilities in Za’atari. This system predicted when tanks

were full or close to full and ensured tanks were emptied at just the right time thereby saving

money125.

Rukban and/or Hadalat

Water

UNICEF was paying for the water tankering, and it was causing tension with the host

community. In Rukban, and to a lesser extent in Hadalat, the issue was not just the cost, but

the consistency of the service, strikes, and water quality. Providing water to Rukban

124 Ibid.

125 Source: Intervuew with UNOPS Staff Member – 24 August 2018

Figure 12 Membrane bioreactor wastewater treatment system

functioning properly at Za'atari Camp.

68

required: i) initial emergency water hauling, ii) improvements to the water sources at

Ruyashed and Hadalat, iii) conducting the hydrogeologic study, installing the new borehole,

treatment and pumping equipment, plus a 13km pipeline and two parallel water distribution

systems. Delivering water by truck across the desert 90 KM one way was very expensive but

necessary for the function of the camp during the initial stages of the intervention.

Investments to improve the water source at Ruyashed were necessary to ensure water quality

met the applicable standards and was a necessary expenditure. The hydrogeologic study and

subsequent new source development represented value for money as it defrayed the cost of

trucking. UNICEF estimates that the solution was 85 percent less expensive than trucking.

The rationalization was that these investments directly reduced the cost of the trucking, and

were therefore appropriate, which in retrospect was sound.

Water trucking to Rukban (from Ruwaished) used to cost 380,000 JOD per month. It cost

18.5JOD as it included 15.5 JOD plus 3JOD/m3 as a tariff for the Govt, for both) per cubic

metre) and went on for 15 months. After an investment of 500,000 JOD (for borehole)

UNICEF paid for the RO / sand filtration unit, and generators, fuel and O&M costs at

Rukban, resulting in the supply cost being reduced to 100,000 JOD per month – a reduction

of 74 percent of the original cost for 30 percent additional water.126.

Costs associated with the Hadalat water system, were rationalized at the time. However, the

sudden evacuation of the camp significantly reduced the return on investment. The system

still has value as a viable but unused water source in an arid region that is short of water. The

situation at Hadalat, as well as the timing for making the new source investments at Rukban,

illustrate the difficulty of making decisions on infrastructure expenditures in a climate of

unknowns. The evaluation team believes that that UNICEF invested appropriately for water

infrastructure, was timely in their investments, and received value for money related to the

new source in Rukban and Hadalat.

Host Communities and ITSs

In the beginning of the emergency response phase of the programme, Jordanians were

supportive of efforts to make arrangements for Syrian Refugees. Tensions quickly developed

however between Jordanians that lived close to Za’atari in Mafraq governorate, and the

refugees that lived in Za’atari. These tensions developed largely over the demands that

refugees put on local resources (REACH, 2014)

The WASH programme has allocated funds for host community projects based on

availability and procured contractors or INGOs to carry out the projects through UNICEF’s

procurement system and in consultation with the government.

An example is HC projects carried out in Irbid. There was little funding available for water

improvements in the host city of Irbid. Consultations with the water utility, city government,

and ministry were conducted as well as an evaluation of the water network. Settling on

investments to improve the existing boreholes and pump station significantly improved the

efficiency of the water source network. The project represents great value for money as these

were sorely needed interventions that in the end did not cost that much compared to the

added volume and efficiency of the water system that resulted.

126 Source: Interview with UNICEF WaSH Officer

69

Evaluation Question 8 • To what extent did UNICEF try to minimize duplication/gaps in WASH interventions

in camps? And how early did UNICEF move to more cost-efficient interventions?

In this section we describe how UNICEF minimized duplication in camps. Please see the

evaluation team’s answer to Evaluation Question 7 regarding UNICEF’s evolution to cost

efficient interventions.

UNICEF minimized duplication or service gaps in camps by assigning different partner

organizations responsibility for WASH services in each sector127. UNICEF’s role as sector

lead allowed it to unify service standards among the different partners.

From July through December

of 2012, UNICEF worked

with Mercy Corps128 to

establish and manage WASH

services in Za’atari, and also

partnered with ACTED, who

conducted hygiene

promotion. In 2012, UNICEF

also began working with

Oxfam followed by ACTED

to provide WASH services

Cyber City, KAP, and “the

stadium,” three transit centers

as they were known at that

time129.

Between January and June of

2013, the refugee population in Za’atari grew from 32,000 to over 116,000 (OXFAM PCA

32-13 Executive Summary). To address the task of managing WASH services for the huge

population growth and to avoid duplication of services in the camp, UNICEF worked with

UNHCR’s division of Za’atari into twelve districts and three sectors. UNICEF partner

organizations each managed one sector and coordinated activities at weekly Za’atari

Coordination meetings. ACTED managed districts 1,2,9,10,11, and 12; JEN managed 3,4 and

5; and Oxfam managed 6, 7, and 8.

In Azraq, a similar coordination structure was followed as Za’atari. ACTED, World Vision,

and THW provided WASH services with Relief International providing hygiene promotion

and WASH in schools. However, coordination among the three entities appears to have been

difficult. There is far less documentation on coordinated efforts in Azraq. UNICEF reports

that there were weekly coordination meetings in Azraq. UNICEF also comments that partner

organizations often brought in activities funded from other donors, which added to the

challenge of coordination in Azraq.

127 Some UNICEF partners were assigned responsibility for camp-wide services, such as solid waste collection.

128 WaSH Sector Coordination meeting minutes indicate that Mercy Corps worked with UNICEF through 2016.

The evaluation team had no documentation of Mercy Corps’ involvement in the WaSH Programme.

However, the team was able to interview two Mercy Corps staff who participated in WaSH activities.

129 The source of this information is the PCA’s signed with each organization.

Figure 13 Division of labor among UNICEF Partner organizations. (Source:

WaSH in Za'atari Snapshop, Jan-Feb 2014)

70

In June 2015, UNICEF decided to simplify operations in Azraq by partnering with one

organization to handle WASH services. It asked partners to submit proposals and awarded

WASH operations to ACF. With the assistance of World Vision, ACF took over WASH

operations in October 2015. While these transitions were handled abruptly, causing some

discomfort for UNICEF’s partner organization, ACF reports that they reduced operations and

management costs bringing efficiency to the camp. Recently, UNICEF has transitioned

programme management in camps again from ACF to a private contractor. Moving from an

International Non-Governmental Organization (INGO) to a local business is a new direction

for the programme and carries some benefits, such as reduced indirect costs. The new

approach also carries some new risks, as a private contractor may not have the same

incentives to manage depreciation of equipment and infrastructure or project cost overruns as

an INGO, and UNICEF does not currently have systems capable of monitoring and

accounting for those items.

Likewise, in Zaatari, the WASH programme has reduced costs and realized efficiency by

reducing the numbers of partners that they are managing. In Za’atari, UNICEF has reduced

the number of partners to ACTED and Oxfam.

Evaluation Question 9 • To what extent were resources (financial and human resources) allocated for the range

of intervention types, with particular attention paid to the respective allocations for

humanitarian response and resilience, to achieve the intended results?

The WASH programme has four components, briefly; WASH in the camps, WASH in host

communities and schools, support to the national government, and coordination of the

humanitarian sector. During the period covered by this evaluation, the great majority of the

WASH programme’s resources went to humanitarian response as represented by WASH

management in the refugee camps, which was their direct responsibility. UNICEF’s

responsibility for financing the WASH services in the camps constrains its ability to address

more strategically important issues, such as the approximately 80 percent of refugees that live

in outside of the camps, or the high levels of NRW in Jordan.

UNICEF Staff estimated that approximately 5 percent of UNICEF’s budget goes to host

community interventions. The evaluation team’s estimate, which as mentioned is based on

incomplete information130, is that from January 2013 - -July 2017, about 63 percent of project

funds went to camp operations, about 15 percent is spent on host communities, and another

15 percent is spent in settlements, Rukban and Hadalat (figure 13).

130 The financial numbers used in this report were derived from UNICEF’s estimates of expenditure on contracts

and PCA agreements provided by UNICEF, which were often in a format that was difficult to read. PCA

numbers were budgeted amounts, not actual expenditure. These figures also do not include any indirect costs

that may not have been allocated proportionally by beneficiary group.

71

Figure 14 Percent of UNICEF Total Expenditure by Beneficiary Type (August 2012 – July 2017)

Figure 14 represents 97 percent of expenditure over the life of the programme, according the

data the evaluation team was able to access. In terms of types of intervention, and water

infrastructure construction and sanitation infrastructure construction are the largest

expenditure category (WatCon and WatProv), together describing about 47 percent of the

programme’s resources since its launch. Next is water provision (WatProv), which accounts

for another 20 percent. The remainder is Sanitation Mobilization (SanMo), Desludging, Solid

Waste Collection (SWCol), distributed kits, and funds that went to activities to support

MoWI.

Figure 15 WASH Programme expenditure by activity category (August 2012 – July 2017)

Note that following the evaluation period, the WASH Programme did provide some

additional financial information regarding the breakdown of expenditure by activity

categories. That information is summarized in Annex J.

72

As regards human resources, the evaluation team could not find documentation of how

human resources were utilized, such as organization charts or job descriptions.

Evaluation Question 10 • To what extent did UNICEF advocate with the Ministries to mobilise their available

resources to address the identified needs? And how efficient was UNICEF’s support

to ministries (i.e. MWI and MOE) to reaching most vulnerable people? Did the

WASH programme maximise the potential collaboration with other UNICEF

programme sections and partners?

Over the time frame that this evaluation covers, UNICEF worked with the MWI to develop

and update vulnerability maps, design the Jordan Response Plan, and provide financial

support to GoJ priorities. UNICEF’s engagement with the WI included a consultant UNICEF

hired to assist the government in developing strategies and response plans. UNICEF also

interacted with the government to support host communities.

The evaluation team saw no documentation of WASH programme advocacy with the MoE

during the period described by the scope of the evaluation. UNICEF reports that it proposed

the WASH in Schools Standards to the MoE and led their development, as well as

coordinating the review. UNICEF Jordan has education programmes that may have had

primary engagement with that government ministry. Still, a UNICEF partner reported that

there is more demand for education services than the MoE can handle. UNICEF could work

more to support the capacity of the MoE and should consider building those activities into its

future strategy in this area.

Sustainability

Evaluation Question 11 • Was the infrastructure in the camps constructed in such a manner to ensure long term

functionality, and will continue to service the most vulnerable people even when

UNICEF is no longer directly supporting? This should be investigated in terms of

Water and Wastewater infrastructure.

Following the emergency response phase of the programme, the infrastructure in the camps

was constructed to ensure long-term, sustained functionality. The GoJ prohibited the

construction of permanent structures at the beginning of the crisis, forcing UNICEF to make

use of mobile sanitation and trucked in water. When it became apparent that the refugee crisis

would not abate quickly, the government recognized the need for longer-term solutions and

UNICEF provided lasting, sustainable solutions.

UNICEF was provided with highly mechanized treatment systems for both Za’atari (trickling

filter and MBR) and Azraq (the MBBR system, which eventually failed). These systems are

expensive, but the decision to use these systems was sound and valid. The national laws of

Jordan specify stringent effluent quality from municipal wastewater sources. Even though

camps are surrounded by vast unused parcels of government land, wastewater systems must

comply with environmental standards. Also, the 4-month rainy season that makes it more

difficult for passive wastewater treatment systems to produce quality effluent year-round.

As written in the Efficiency section of this report, UNICEF has worked to maximize

sustainability in Za’atari with innovative wastewater collection models, elimination of

trucking, and the use of local labor. UNICEF has also installed dataloggers in boreholes to

measure water levels and monitor water level changes. UNICEF also points to the location of

73

Za’atari and that the GoJ’s involvement in Za’atari’s WASH service design will facilitate

turning the system over to the government if/when the refugees leave the camp.

For Azraq, though the MBBR system failed, the rationale for selecting it was valid.

UNICEF’s plan for sustainability in Azraq is difficult to discern. As UNICEF reports, “It is

very unsustainable to build a massive wastewater network across (an area) the size of Azraq,

in the middle of the desert, with no alternative use for it, and that the Syrian situation

suggests that peope in Az would not stay.”

For Azraq, the installation of the MBBR system, translocated from Afghanistan and provided

by, turned out to be a poor choice, although the rationale for selecting it was valid.

Unfortunately, the system did not meet the needs of Azraq’s inhabitants or the program in

general. Wastewater strength was higher than anticipated. Following an expensive attempt to

make it work, the system was eventually abandoned. Wastewater is now pumped and

transported at great expense. There is a greywater management scheme that has yet to come

on line fully, and until there is some centralized wastewater treatment option, septage will

require hauling some 90 KM away. It is surprising that the Azraq camp would have more

difficulty in establishing a wastewater system as it is neatly laid out, compared to Za’atari,

which has developed an elegant and lower cost solution, which has evolved into something

that could become sustainable over the long term as economic development in the camp

increases.

The water systems in the camps are designed, installed, operated and maintained to provide

long term functionality, even if UNICEF were to leave, which is unlikely. The quality of the

equipment is high, which adds to the longevity of the systems. The design provides

maximum operational flexibility, with multiple redundancies. The system is relatively easy

to operate, which also adds to the long-term sustainability. In Azraq, local camp residents are

performing the operations tasks with oversight provided by a local company, which is a

strong indication of sustainability. However, actual operational costs are high. The high costs

are largely due to the energy it takes to run the systems, but also the cost of consumables,

spare parts, water fee payments to the WAJ and generator costs in Azraq. UNICEF also

reports that it prefers paying more for high quality contractors, as lower capacity contractors

become much more expensive if system disruptions require a return to tankering. If money

is not available for these, the systems will not achieve long term sustainability.

The wastewater system at Za’atari is functioning well and should continue to do so given the

required operation and maintenance. Wastewater systems have design lives of 20 years or

more. This system is very robust and should provide long term and compliant service, again,

given proper O&M. The system itself, MBR and Trickling filter technology is relatively easy

to operate, which adds to its long-term sustainability. The collection system utilizing

common septic tanks for between 5 and 8 families requires a camp-wide organized

desludging program. This program is an integral component of the collection system that

will require long term funding. The use of cash for labor on the installation of the sewer

network built a sense of ownership of the infrastructure for some camp residents and also

adds to long term sustainability.

The water system at Rukban is constructed of high quality components that are well designed

and relatively easy to operate. A local engineering firm has been contracted to operate the

system and they are doing so effectively. As long as funding is available for O&M, the

system should function sustainably for many years. During KIIs of the water operators, it

was learned that UNICEF tends to purchase high quality equipment and replacement parts,

which minimizes down time and should add to long term functionality.

74

The upgrades to the boreholes and pump station in the host community of Irbid are well

designed and constructed and should continue to operate give the required O&M for many

years. While the UNICEF interventions are sustainable, the overall water situation in Irbid

however, is not. The extreme water scarcity, depleting aquifer, population growth, and high

NRW rates are all causes for concern. They suggest areas for next steps should donor

funding be available.

Evaluation Question 12 • To what extent did UNICEF take sustainability into account at the early stage for

water, wastewater and solid waste management services (e.g. transition to local

government institutions and local capacity)?

The evaluation team has no evidence that UNICEF has a transition plan to government or any

other institution in place. The GoJ at all levels has made it clear that they will not take

ownership or spend public funds on the crisis. The government understands that it is

providing an international public service in using internal resources such as water to sustain

the refugees and expects the international community to provide for the rest of their needs131.

UNICEF’s approach to sustainability has three aspects132:

• Build high quality infrastructure: Building systems and infrastructure that are

consistent with national standards and have long term potential to serve Jordanians

and the national government.

• Leverage other, non-governmental resources to contribute to ongoing operations

of the camps: This approach includes building the capacity of refugees to manage

their own WASH issues and partnering with national entities to engage in

management.

• Become as efficient as possible: UNICEF continues to find ways to bring down costs

so that resources are used as efficiently as possible, and available as long as refugees

need WASH services.

A question remains of whether running a municipal water utility, which is essentially

UNICEF’s role in the camps, is an appropriate one for the organization. Strategically,

UNICEF will have to decide whether it wishes to orient itself toward permanently providing

WASH services in the camps or should develop a strategy to hand it over to an organization

that may be better suited for those types of long-term administrative activities. A lack of

alternative organizations and guaranteeing payment for a replacement organization are

complicating factors. UNICEF reports that they investigated options for transferring solid

waste and desludging management to municipalities but lacked the resources and interest on

the part of municipalities required to make the transition.

Evaluation Question 13 • To what extent did UNICEF’s interventions increase the resilience of the Government

and respective target populations? To what extent will UNICEF-supported projects

(schools, ITSs, water and wastewater infrastructure, mobilisation) continue to be

operational after the end of UNICEF’s financial support

131 Source: Interviews at the MoWI, WAJ, and UNICEF.

132 Source: Interviews with UNICEF executive staff.

75

This evaluation question is difficult to answer because it appears that target populations will

require UNICEF’s financial support well into the future. UNICEF increased the resilience of

supported populations and the GoJ in the following ways.

• Designed projects in accordance with the JRP, linking refugee assistance in Za’atari

and Azraq with resilience-oriented projects that rehabilitate or extend systems in

urban areas with large refugee populations.

• Highlighting need through vulnerability maps and mobilizing for funds for resilience

projects described in the JRP to assist the GoJ.

• Instituted consistent standards and policies around water access in camps to promote

equitable access and reduce conflict.

• Worked with the MWI to design the National Water Strategy.

• Ensured that infrastructure was built using high quality materials and that construction

was carried out according to national standards.

• Built capacity of stakeholders in camps to manage water, sanitation and hygiene

within their communities.

• Created mechanisms for hygiene and water conservation promotion that operate under

the leadership of local stakeholders, such as WASH committees and the Lead Mother

model in Azraq.

• Improved water and sanitation infrastructure and services in some host communities

with high refugee populations to reduce tensions.

• Worked with the government to create Wash in School Standards to set a bar for

water, hygiene and sanitation in Jordan’s often overburdened schools. The standards

included a nation-wide assessment.

• Raised awareness regarding water conservation and promoted a leakage detection

app.

• Transferred a process for creating response and water strategies to the GoJ that they

can replicate in the future.

• Promoted the use of data for strategy development at the MoWI.

Evaluation Question 14 • Does the MWI have the necessary mechanisms in place and capacity to ensure the

water and sanitation needs of the most vulnerable people will be met as a result of

UNICEF’s support and advocacy, if another large scale humanitarian crisis occurs in

the future?

The evaluation team did not find evidence that MWI has the necessary mechanisms in place

for a future, similar crisis. One reason for this is that the government does not want to own

these crises or been seen as a refuge for at-risk populations. Additionally, UNICEF reports

that public support has dropped considerably compared to the situation in 2012.

Evaluation Question 15 • To what extent did UNICEF document its WASH interventions and make available to

the sector learning on water, wastewater, solid waste management and mobilisation?

Documentation of interventions for the purpose of sector learning was limited. Lessons

learned were shared at coordination meetings. The evaluation team did not find evidence that

the WASH programme documented its experience for the purpose of sector or institutional

learning. This evaluation is the first attempt to document UNICEF’s experience

implementing the WASH programme in Jordan. UNICEF reports that it created a new

position in January 2018 to document sectors learning and the position is working well.

76

Evaluation Question 16 • To what extent were environmental protection and sustainability taken into account in

the design and implementation of programmatic interventions?

Early in the programme, environmental protection was not taken into account sufficiently.

Lack of focus on environmental issues led to Za’atari’s construction over an aquifer that

accounts for 34 percent of Jordan’s water supply without proper facilities for managing

wastewater and preventing it from leaching into the ground. As described in earlier sections

of this report, the problem was exacerbated when open pits filled with graywater developed

as camp residents began to create private sanitation facilities in their housing units. UNICEF

corrected this issue in Za’atari by installing wastewater collection tanks, reaching 2,000 m3133

of collected wastewater per day, utilizing backwash from borehole water filters for

construction and cleaning purposes, and constructing a water treatment plant. UNICEF also

backfilled over 11,000 open pits. Water usage was controlled through UNICEF’s

enforcement of regulations and daily volume collection and distribution. Seasonal water

allocations are used to ensure sufficient levels, and in Azraq, the community has been

engaged to prevent wastage.

UNICEF applied the lesson it learned in opening a site location and water system for Azraq.

Prior to the construction of Azraq, UNICEF worked with the MoWI to conduct an

environmental impact assessment. The assessment identified a high risk of underground

water contamination. Also, the GoJ was concerned about increased abstraction from the well

field near Azraq. UNICEF incorporated protections in its design of Azraq’s WASH services.

It supported installation and operation of containerized wastewater units. As seen earlier in

this report, the treatment plant ultimately did not work out, but the evaluation team

recognizes that the decision to include it was appropriate at the time. The septage receival

and dewatering units would have permitted the system to receive wastewater from trucks and

bring down the cost of desludging operations. In 2016, a pond was also constructed at Azraq

camp to store wastewater that could be used for agriculture.

UNICEF also assisted the government in improving environmental protections as regards

WASH services. For example, UNICEF built the capacity of the MoWI to conduct

groundwater monitoring more effectively.

Other initiatives that UNICEF implemented that took environmental protection and

sustainability into account include:

• Working with the Royal Scientific Society to study the most environmentally

sensitive way to provide heating for camps. Providing heating with gas was chosen.

• Promoting water conservation in camps and host communities through

implementation partners.

• Celebrating world events, such as World Environment Day.

• Promoting recycling where feasible.

• Leading the construction of household connections in Azraq to reduce potential

contamination.

133 UNICEF reports this represents a doubling of wastewater volume collected.

77

Coverage

Evaluation Question 17 • Did UNICEF’s WASH response identify and reach the most vulnerable people? To

what extent did specific interventions strike an equitable balance between

interventions in camps/settlements and in host communities, to reach the most

vulnerable people?

Identification and outreach to the most vulnerable people

It is difficult to know if the response identified and reached the most vulnerable people over

the course of 2012 – 2017. At the initiation of the project it was assumed that refugees

housed in Za’atari camp were the most vulnerable, as refugees taking up residence elsewhere

had other options for WASH services, such as private sources or the government.134 The

evaluation team did not find documentary evidence of an attempt to identify the most

vulnerable people and reach out to them following the emergency response phase of the

project through July 2017. For example, while UNICEF has served inhabitants of some ITSs,

there is no comprehensive understanding of the number of ITSs or their population.

That said, UNICEF described its efforts to the evaluation team. These included:

• Sanitation assessments in Za’atari to identify households that needed the most

assistance. These households were supported by partner organizations.

• Identification of households that included people with a disability, and a subsequent

construction of ramps to meet their needs.

• Vulnerability assessments conducted in Ruwayshed followed by a prioritization of the

households which met vulnerability criteria for assistance.

• Identification of households in extreme in partnership with ACF, and conducting

interventions for these households.

• Continued work with especially vulnerable schools, ITSs, and host communities.

Following the emergency response phase, the WASH programme continued to devote most

of its resources to camps throughout the life of the programme. The programme did not

methodologically study the number and status of refugees outside the camps, and the

proportion of programmatic resources that went to people outside of the camps was small.

Though approximately 80 percent of the Syrian refugees in Jordan live outside of the camps,

only between 5 and 15 percent of the WASH programme budget went to host communities.

Finding resources to serve host communities more equitably would have meant reducing

standards in the camps.

Within the camps, UNICEF’s WASH programme has had a focus on ensuring access for the

most vulnerable. This includes, women, children, and people with disabilities. Please see the

section titled “Extent that UNICEF Achieved Equity Results” under evaluation question 4 of

this report.

Equitable balance between interventions in camps and host communities

The degree to which the programme is focused on the most vulnerable people is influenced

by the WASH programme first mandate, which is to maintain WASH service levels and

standards in the camps. While people entering the camps may have been the most vulnerable

134 Source: Interviews with UNICEF senior staff.

78

at the programme’s outset, the evaluation team believes that in later years occupants of ITSs

were more vulnerable and received less attention from the WASH programme. However, the

necessity of maintaining service in the camps combined with budgetary shortfalls has

prevented UNICEF from identifying and reaching that population.

Until July 2012, UNICEF Jordan did not have a WASH programme. It had WASH activities

within its health unit, but not a stand-alone programme. When the Syrian refugee crisis

began, the Jordanian offices of UNHCR and UNICEF came to an agreement that UNICEF

would provide WASH services in the camps, and that UNICEF would lead the WASH sector.

Thus UNICEF became tied to managing WASH in the camps as a first priority. As executive

staff at UNICEF Jordan put it “The camps had to be a starting point for funding allocation

because there was no other option for those people. 80,000 in Za’atari had no other way to

survive. Others had government and other options. The starting point is how do we maintain

services in the camps.”

As the number of refugees quickly increased, tensions between host communities and refugee

populations grew. Three months after Za’atari opened, Mercy Corps conducted a study to

document the growing tensions between Jordanians and the refugees. UNICEF began

allocating some resources to projects in the host communities, particularly Mafraq and

Ramtha, but its focused remained, appropriately, on the camps, which provided resources to

people with no other alternative. In the 2012 UNICEF Jordan Annual Report, UNICEF

Acknowledged, “here has been an additional challenge with the identification of the most

vulnerable children within the host communities,”

That challenge continued through 2013 and 2014 as the population of the camps continued to

grow. UNICEF implemented projects in host communities as well, but they were for the

purpose of quelling tension rather than serving the most vulnerable. REACH’s June 2014

report, “Access to Water and Tensions in Jordanian Communities Hosting Syrian Refugees,”

illustrates why UNICEF chose the projects it did in host communities, and why reducing

tensions was a worthwhile goal. A host community project in Irbid serves as a good example.

In Irbid, the WASH programme improved sewer lines and rehabilitated two boreholes. The

improvements were hugely important in Irbid, a governorate with a high refugee population

and degraded water infrastructure. The evaluation team’s inspection found that the

interventions added volume and efficiency to the water system.

As early as 2014 the WASH programme began to target indicators improving water access

for Jordan as a whole in its indicator framework. By 2016, though it was clear that the large

majority of refugees were outside the camps and that those living in ITSs were existing under

dire circumstances, still only 4 percent135 of the WASH programme’s budget went to host

communities.

Serving the most vulnerable in the camps

Within the context of the camps, the WASH programme has been conscientious of serving

vulnerable inhabitants. This evaluation report details initiatives to serve vulnerable

population in the camps in response to Evaluation Question 4.

135 Based on incomplete information submitted to the evaluation team.

79

Evaluation Question 18 • Did UNICEF as Sector Lead ensure that the needs of those in most need were

addressed in a coherent manner, particularly in comparison to alternative existing

systems at the local or regional level (i.e. UNRWA camps)? If not, how?

As described elsewhere in this report, the evaluation team found that UNICEF ensured the

needs of those in camps and at the settlements were addressed in a coherent manner.

UNICEF organized sector coordination efforts and coordination efforts at the camp and

settlement levels to design activities, assign responsibilities to avoid duplication, and set

standards to ensure equity of results as far as that was possible. Following that, UNICEF

worked in host communities and ITSs to the extent resources allowed. The evaluation team

does not have evidence that UNICEF used its position as sector lead to make sure those in

need were identified and that their needs were addressed. UNICEF reports that it shared

vulnerability mapping widely across the sector and encouraged partners to raise funds to

serve vulnerable people.

Coordination

Evaluation Question 19 • To what extent did UNICEF role as WASH sector lead enhance response processes

and results to avoid duplication and gaps in interventions?

Please see the evaluation’s response to question evaluation question 8 regarding avoiding

duplication.

Regarding enhancing response processes, a sample of WASH sector meetings from May

2016 through April 2018 show 13 organizations regularly attending. Action points and focal

points are thoroughly covered, and cover issues from the National Water Strategy to Schools

Assessments and issues in the camps.

UNICEF also established a number of reporting mechanisms through third party monitors

including water quality reporting, and security incident reporting. UNICEF established the

use of vulnerability Maps for Water and wastewater. The maps revealed Azraq governorate

as the most vulnerable place in both water and sewage. UNICEF has also worked with

partners to submit proposals for sources of funding to address specific issues.

UNICEF has also put tremendous effort toward preparation for winter storms including

stockpiling supplies, coordinating partner roles, and negotiating contingency plans. UNICEF

also provides ongoing monitoring during storms as well as clear communication with

partners.

UNICEF partners that the evaluation team interviewed revealed partner satisfaction with

UNICEF as sector lead. One partner said that, “UNICEF are very good coordinators. They

always have staff in Azraq.” A donor said that they were very happy with UNICEF’s

coordination and partnership. A staff member of one partner said, “UNICEF does a very

good job of controlling the work. 95 percent approval rating for UNICEF.”

80

Evaluation Question 20 • Were partnerships effective and leveraged to the maximum extent to assist the most

vulnerable population in camps/settlements, ITSs and in host communities?

The evaluation team believes that partnerships were effective and leveraged to the maximum

extent in the camps and settlements. As described in this report, UNICEF utilized its partners

to ensure full coverage in the camps and compliance with standards. Based on the limited

financial information that the evaluation team was provided, the team estimates that UNICEF

also leverages over $27, 250,000 in in-kind contributions from the partners it worked with136.

UNICEF also effectively worked with partners in host communities. Partners were relied on

to identify and manage key interventions. UNICEF had clear communication with partners to

identify ITS and school needs and to mobilize appropriate funds. WASH in schools programs

were also managed effectively through partners.

One aspect of working with partners that UNICEF could have improved was communicating

strategic decisions and medium-term vision for the project. The UNICEF partners that the

evaluation team interviewed reported that they often heard about drastic changes to plans or

operations with little notice or time to prepare. One partner noted “UNICEF take quick

decisions – the change from Mercy Corps to ACF in Azraq could have been done in a better

way. It was not a transparent process.”

Conclusions In July 2012, UNICEF was the only organization in Jordan that had the resources, capacity,

and institutional commitment to take leadership of the WASH response related to the Syrian

refugee crisis. UNICEF accepted responsibility for providing WASH services in a country

that was exhausted by previous refugee crises, water scarcity, a complex political

environment, and funding uncertainty. From July 2012 through July 2017, UNICEF provided

life-saving water and sanitation resources under these difficult conditions for the

approximately 400,000 people that benefited from WASH services137.

The programme that UNICEF developed as its WASH response to the crisis was to become

one of the largest programmes in UNICEF Jordan’s portfolio. The scope of the programme

was initially a mandate to provide services to refugees in camps. It quickly expanded to

include services for the water-deprived Jordanian host communities. These communities were

resource constrained before the crisis and were further burdened by the crisis. The WASH

programme’s scope also grew to include providing urgent WASH services for refugees living

in the border settlements at Hadalat and Rukban. The emergency response required quick and

flexible decision making and efficient utilization of scarce resources, which UNICEF

successfully performed.

The quick acceptance of responsibility in July 2012 and rapid action that followed led to

several missteps. The evaluation team could not find evidence that UNICEF WASH staff

utilized UNICEF’s internal resources that provide guidance in managing WASH during the

first phase of an emergency response. For example, UNICEF has led the Global WASH

Cluster for over a decade. In 2009 the organization published a handbook that described how

136 Estimate based on a review of PCA’s during the life of the WaSH Programme


81

to WASH services in an emergency like the Syrian Refugee Crisis (redrUK, 2009).

Utilization of the handbook might have assisted the WASH programme team in contributing

to the decision regarding the location of Za’atari camp, assessing refugee needs and

preferences, conducting ongoing monitoring, and allocating responsibilities to various

partners were made.

The UNICEF WASH programme was a conscientious steward of its donors’ contributions. In

the emergency response phase of the programme, UNICEF staff worked under difficult

conditions to bring WASH services to refugees on short notice. After the situation stabilized,

UNICEF was innovative in its analysis of WASH operations. This analysis led to consistent

reductions in operating costs, which allowed financial resources to serve many more people

at a higher standard than was otherwise possible. Examples include committing to build new

infrastructure that could be operated at a lower cost than emergency interventions, and the

“waste taxi” predictive model, which allowed WASH managers to anticipate desludging

needs and overflowing tanks.

Regarding hygiene, the evaluation notes an absence of reference to key infectious diseases

such as diarrhea and respiratory infections in the WASH programme’s documentation, with

the exception of the Hepatitis A outbreak in Azraq and the background information provided

in ACF’s PCAs. A notable feature of Za’atari and Azraq’s management was the strong

commitment to regularly implementing KAP surveys in both camps on an annual basis,

enabling the measurement of change over time. However, the evaluation team did not find

hygiene data with respect to host communities or those in the ITSs.

The information that the evaluation team collected through interviews and focus groups on

hygiene related items indicates that people received the items they needed. Sometimes,

however, quantities of certain items like soap powder were insufficient and at times arrived in

irregular intervals.

Hygiene was communicated in a number of different ways including through community

level hygiene promoters, children’s hygiene clubs, campaign days linked to key global days

such as global handwashing day and use of the mass media.

The evaluation found that consultation regarding preferences for WASH service options was

not always possible, and that meeting preferences was also not always possible. Nonetheless,

the WASH programme’s interventions remained highly relevant. The WASH programme set

standards early and used assessments to identify the best way to address needs given

available options and resource constraints. When the situation stabilized, the WASH

programme consulted more closely with camp residents to discover their preferences and

identify ways that operations could improve. The positive press that Za’atari has received,

(Jaafar, 2017) and Azraq camp residents’ reports of consistent improvement in services

speaks to the relevancy and effectiveness of the WASH programme’s main operations.

UNICEF’s work in host communities has reduced tensions and brought relief to hundreds of

thousands of people. Its work in host communities was well coordinated and of high quality.

One improvement UNICEF could make is rethinking its approach to serving host

communities and ITSs. In each year of the project, UNICEF estimated the number of people

they would need to serve to effectively achieve the WASH programme’s goal in host

communities, and every year the programme fell well short of the target. The reasons for

falling short certainly include lack of funding, as UNICEF stated in its annual reports. This is

certainly true as the international community has underfunded the humanitarian response

regionally, and much of UNICEF’s funding was earmarked for provision of services in

camps and settlements. However, missing the target several years in a row should have

triggered a strategy review and a reorienting of this programme component. In fact, though it

82

falls outside of the scope of this evaluation, senior staff interviews indicated that this strategic

reorientation may come in 2019, as UNICEF applies its innovative capacity and technical

knowledge to assist other donors that are more capable of funding large, municipal systems.

Serving refugees in ITSs should be an immediate concern for UNICEF. While in the early

days of the response, the most vulnerable people were in the camps, by December 2013 it

was clear that people living in ITSs had dire needs that were going unaddressed (REACH,

2013). Despite the vulnerability of people, especially children, living in ITSs, they have

remained remarkably underserved. UNICEF reports that it has worked with partners to raise

resources for ITSs and Schools. The evaluation team could find no recent information

documenting the number, locations, or status of ITSs in Jordan. The ITSs that the evaluation

team visited, which UNICEF serves, suffered from unsanitary and dangerous conditions,

such as unsatisfactorily covered latrine pits138. One can imagine that ITSs not benefitting

from UNICEF’s services are far worse.

Equity of coverage remains a challenging issue that the WASH programme consistently

seeks to address. The programme struggled early on with equity as public facilities in Za’atari

camp were difficult for women and children to securely access, and water trucking and

sanitation services were corruptible leading to some households receiving more water and

better desludging service than others. Building the water and wastewater networks in the

camps, and their management with innovative apps and third-party monitoring, solved the

initial problems, and improved the situation greatly. However, other new issues arose, such as

households in Za’atari receiving equal quantities of water regardless of the number of

household members. The water network in Za’atari was also misused as some households

rigged their water tanks to receive more of their share of water. UNICEF remains vigilant in

the camps and will continue to work to address these challenging issues.

Equity of coverage for women and children in the camps is also challenging. UNICEF

manages this challenge through ensuring that partners that provide camp services have

sufficient numbers of female hygiene promoters and social mobilizers on staff. Partners also

come up with innovative, woman centered mechanisms for programme management, such as

the Lead Mother model implemented by ACF in Azraq. UNICEF’s network expansion to

address graywater was also implemented to address inequity.

The WASH programme’s work at the national level has greatly assisted the government in

forming and implementing policy. National level capacity building has also assisted the

government in using data more effectively. The GoJ values UNICEF’s partnership and sector

coordination. UNICEF could further enhance its role in consulting with the government.

Areas of focus could include reducing NRW, advocating for an improved legal framework,

improving the efficiency of operations, and assisting with strategy implementation. The

evaluation’s interviews with senior UNICEF staff indicate that UNICEF may already have

taken steps in that direction.

The WASH programme completed a comprehensive nationwide assessment of WASH in

schools and created a plan for bringing those schools up to standard. However, achievement

against that plan was limited, and strong pilot projects carried out were not followed up on.

The limited achievement in the WIS area, as in host communities in general, reflects a lack of

strategic consideration when committing to objectives and setting targets.

138 See figure 9.

83

UNICEF’s stakeholders agree that that it has done a solid job in coordinating the sector.

UNICEF has established systems for avoiding duplications or service gaps in the camps. The

government appreciated UNICEF’s leadership in contributing to sector efficiency and

making it more data driven. Partner organizations that the evaluation team interviewed said

that UNICEF was always available to trouble shoot issues or provide useful technical input.

The WASH programme should improve its strategic management. Strategic management

includes the setting of objectives based on the consideration of the specific needs of a well-

defined population compared to an organization’s internal assets, internal shortcomings,

external opportunities, and external challenges. Strategic management during the course of

the evaluation period suffered because the programme’s rapid start-up meant that the WASH

programme team forwent standard assessments, strategic analysis, and programme design

activities. Still, the evaluation team was surprised that over the course of the programme,

monitoring and evaluation protocols and a monitoring data system were not established. The

WASH programme team noted that they do not document their activities adequately139.

Included in strategic management is the implementation a standard monitoring and evaluation

system, which would track results at all levels. Implementing an adequately managed

monitoring system is especially important given the regular, planned turnover among

UNICEF staff. Good record keeping prevents important knowledge from disappearing with

staff as they move to new assignments. It also allows for catching trends in data that may not

otherwise be apparent.

Also importantly, UNICEF’s WASH programme lacks an activity based accounting system,

which would allow it to track spending by year and programme component. A review of the

proposals and reports that UNICEF’s NGO partners demonstrate the standards required to

monitor and financially account for programmatic activities. Improving accounting standards

will increase in importance as UNICEF moves away from working through partner

organizations to directly managing contractors. Contractors will have a greater profit

incentive than NGOs and may recommend replacing equipment more frequently than is

necessary or other such profit generating activities. UNICEF will have to track depreciation

of assets, closely monitor costs, and have the ability to assign indirect costs to programme

activities to properly manage resources.

UNICEF has done everything possible to ensure that its interventions are long-lasting, and

that mechanisms are in place for their operation. Examples include organizing WASH

committees to oversee water access and hygiene promotion, constructing well designed

systems out of durable materials which meet the national standards, and implementing

messaging campaigns for hygiene promotion and water conservation. However, the ultimate

sustainability of the programme is a complex issue. It includes determining if and when

UNICEF will hand WASH management off to another entity, managing the transition of

refugees either back to their place of origin or a permanent settlement elsewhere, the

dismantling or integrating camp WASH infrastructure into Jordan’s municipal systems, and

dealing with Jordan’s medium-term water scarcity issues. Addressing these sustainability

issues will require deeper engagement from multilateral organizations and the agreement of

the Government of Jordan.

UNICEF’s WASH programme accomplishments are impressive and future challenges

complex. The programme’s experience offers much that future programme’s could learn

139 Source: WaSH programme staff interview – 10 October 2017

84

from. This report presents these lessons and recommendations for future programmes in the

following section.

Lessons learned and Recommendations This section presents lessons learned, followed by recommendations. At the end of the

section we include further recommendations based on the evaluation’s findings.

Lessons Lesson 1. Quick decision making, and decisive action saves lives. The decisions and

actions UNICEF took in the early days of the programme were crucial for initial inhabitants

of Za’atari. In the settlements, UNICEF provided water and sanitation services to respond to

the settlements’ exigencies. UNICEF moved quickly and effectively, with clear goals, such as

ensuring that each person had at least 15 liters of water per day. The WASH programme also

recognized that its intervention would have an impact on the larger Ruwaished host

community. To address that impact, UNICEF rehabilitated the community’s water and

sanitation facilities and provided support to identified vulnerable households.

Lesson 2. Installing quality water treatment and delivery equipment is crucial for the

long-term sustainability and use. UNICEF’s installed higher cost and higher quality water

infrastructure at Za’atari, Rukban and Azraq, including the treatment plant for Zaatari. In

each case, a robust design was followed by very good installation and excellent post

construction operation and maintenance. This was borne out through the evaluation team’s

on-site investigations. In a KII with the contractor in Rukban, the operators noted that

UNICEF always purchases high quality equipment. The result is operational efficiency,

equipment that is long-lasting in the harsh environments, and reduced maintenance costs,

downtime, and interruptions to supply.

Lesson 3. Innovative approaches to inefficiencies are scalable: Initially wastewater

management was hugely inefficient. For example, some contractors claimed to remove up to

40 percent more sludge than they actually removed. This stemmed from issues such as

carrying waste in to the camp from outside, which was subsequently charged to the UNICEF

operation. This inefficiency was fixed through the introduction of a voucher system where

communities in receipt of services had to sign for them. Another innovative approach was the

development of a third-party monitoring system, which paid off in improved service, greater

coverage and increased efficiency.

Lesson 4. When direct consultation is not possible, other means of assessment can

ensure programme relevance. In addition to direct consultation with beneficiaries, UNICEF

put a system of assessments and KAP surveys in place. These studies kept UNICEF staff and

sector participants informed of changes in the camps and other locations and assisted the

programme in staying relevant and responding quickly to eventualities.

Lesson 5. The division of labor system, which UNHCR determined and UNICEF

followed, in the camps worked well to determine roles, prevent duplication of services,

ensure that standards were set and enforced, and protect against gaps in coverage. The system

also assisted in understanding cost drivers and making improvements to efficiency.

Lesson 6. Setting strict standards for private sector engagement and holding contractors

to those standards improves efficiency. Emergency situations require engagement with the

private sector, but also present opportunities for gaming the system. Evidence of this

dilemma in the Jordan WASH programme includes some contractors water delivery and

desludging services giving preference to some people’s needs over others and claiming more

85

volume was moved than was accurate. It may also have occurred in ways that weren’t

spotted.

Lesson 7. Documents, tools, and methodologies are available to quickly assist UNICEF

staff in designing and managing WASH programmes in emergency contexts. There are many

lessons learned from the early days of the project regarding issues such as camp site

selection, means of conducting assessments, and methods of determining beneficiaries’

preferences. Most of these have been recorded and addressed in UNICEF guidebooks and

lessons learned documents. Many of these predate the WASH programme's initiation.

Lesson 8. A complaint hotline should not be run by the same organization that is

responsible for the delivery of services. Early in the programme, ACTED managed service

delivery and the complaint hotline. This arrangement was problematic. Complaints were

better addressed when delivered to the implementing partner via UNICEF’s third-party

complaint response mechanism.

Lesson 9. The programme lacks analytical rigor for determining the programme’s

objectives, indicators, or resource allocation, which has led it to overcommit to initiatives for

which it did not have resources or capacity, primarily in host communities. Examples include

the programme’s overestimation of the degree to which it could serve host communities, and

its funding of successful WIS pilots that did not receive follow up or resources to scale. The

WASH programme has operated without strategic planning, and operating theory of change,

or specific, measurable, timebound indicators. The lack of strategic programme design

protocols may have prevented the programme from leading strategically more appropriate

initiatives. These initiatives may have involved promoting wide scale NRW reduction,

rainwater water harvesting, water conservation, and groundwater recharge programs as real

strategic interventions that would yield long standing results.

Lesson 10. One of the greatest challenges faced by the programme was managing the

expectations of refugees and host community members. Refugees came from a relatively

water sufficient region that valued privacy and cleanliness. Host community members lived

in a water scare region and were concerned about newly arriving refugees depleting their

water supply, as well as receiving a free and higher standard of service than locals were

receiving. Creating an information campaign to keep both of these groups aware of the work

that UNICEF was doing, where sacrifices were required, and what they could expect in the

future might have prevented unrest and other difficulties

Lesson 11. The clearest path to sustainability for a programme like the UNICEF Jordan

WASH programme is to transfer activities to the government. However, in the Syria Refugee

Crisis, handing the programme to the government of Jordan was never an option. In taking

lead in the sector, UNICEF has indefinitely taken ownership over the equivalent of a

municipal utility in the form of water and sanitation service provision to Za’atari and Azraq.

Lesson 12. When it initially established protocols with the GoJ, particularly the MoWI,

UNICEF presented a budget and estimation of contributions to infrastructure for which it

hadn’t yet secured funding. The consequence was a misunderstanding with the government

that led to UNICEF not covering project costs because it didn’t meet fundraising targets, and

the GoJ taking infrastructure projects out of its portfolio because it thought they were

covered.

Lesson 13. International NGOs have different risk factors, management structures, and

commitments than UNICEF and other UN agencies. UNICEF often asked NGOs to make

rapid changes, perform out of scope work, or quickly transition in and out of camps. These

requests often felt like last minute decisions to UNICEF’s partners, and didn’t allow them to

prepare adequately in terms of notifying staff, changing commitments, or transitioning assets.

86

Lesson 14. The emergency response phase of the programme required quick decision

making and action. As the report mentioned, UNICEF staff said that there was five days

between UNICEF taking WASH sector lead and the opening of Za’atari. As a result,

UNICEF staff were not aware of UNICEF’s institutional resources that would have helped

response staff plan and manage the organization’s activities in the early days of the

programme.

Lesson 15. Decisions in implementing WASH programmes require trade-offs. For

example, deciding to locate a camp close to an urban population means easier access to

municipal infrastructure, but more tension between the camp and community; whereas

locating a camp in a distant location means less civil tension, but more expensive services

like desludging. Camp residents and community members see the effect of these trade-offs

and speculate as to why they were made.

Lesson 16. Beginning in 2012, UNICEF clearly desired to implement a comprehensive

programme that responded to all refugees needs. As the programme evolved, it has provided

excellent infrastructure and services for people in camps, and struggled to serve refugees

outside of the camps. Thus, commitments to the GoJ and host community and WIS fell short.

This struggle is partially due to a very difficult funding environment, but also a result of a

missing strategic development process.

Recommendations

Reference Lesson Recommendation

Lesson 6 - Private

sector engagement

and contractors

UNICEF should develop a system for contracting WASH services and

monitoring those services. The system should include regular report

objective measures back to the contractor as a means of increasing

efficiency.

Lesson 7 –

Utilizing

UNICEF’s tools

and methodologies

UNICEF should have a global or regional WASH manager that is

responsible for ensuring that emergency responders have the guidance

and tools they need at the beginning of a response. These could include

tools like the WASH Cluster Coordination Handbook.

Lesson 8 – Hotline

management

Ensure that future programmes set up a mechanism for beneficiaries to

report issues/complaints directly to UNICEF, not through an

implementing partner. If UNICEF is the direct implementer, an

accountability mechanism unconnected to the implementation of the

programme should be established.

Lesson 9 –

Improving

analytical rigor

and planning

The WASH programme should implement and monitoring and

evaluation system and comply with the strategic programme

development protocols and reviews that are detailed in the Jordan

Country Programme 2013-2017 document. The WASH programme

should hold itself to the same standard it holds its partners. UNICEF’s

PCA agreements require that partner organizations provide detailed

analysis of the reasoning behind proposed interventions and directly

87

connect activities to that analysis. It also requires that partners set

specific, carefully designed objectives that are strategically coherent.

UNICEF’s partners put monitoring systems in place to track progress

toward these indicators. Monitoring systems are also useful for making

decisions during programme implementation, as they allow managers

to detect initiatives that go astray.

Lesson 10 –

Managing

stakeholder

expectations

Future WASH emergency response efforts would benefit by

understanding how refugees’ standards in camps differ from what

they're accustomed to. When camps are near urban populations,

programmes should seek to understand how host community members

perceive WASH operations. Information campaigns and rallying

resources to support both populations could head off future tensions.

Lesson 11 –

Sustainability

UNICEF should decide if it has the capacity to serve as the manager of

WASH services to Za’atari and Azraq indefinitely, and if that is a

proper role for the organization. If it determines that it is not a proper

role, it must explore options for transfer. If it decides to stay in the role

for as long as required, it should upgrade its management and

accounting systems appropriately.

Lesson 12 –

Managing

government

expectations

In the future, UNICEF should make sure that it communicates clearly

with government partners the likelihood of protocol commitments

occurring. This issue is also tied to shortfalls in strategy and planning.

Lesson 13 –

Managing NGO

partners

UNICEF should seek to better understand their partners business

models and capacities. Forming an understanding would smooth

management of projects implemented through these partners.

Lesson 14 –

Managing

institutional

resources

UNICEF should pre-load a computer with the software and manuals

that emergency response staff need in the early days of a project. The

preloaded information should include up-to-date WASH emergency

response manuals or tools, a pre-developed results indicator

framework and logic model, and other strategic tools that will assist in

decision making and goal setting. It should also include management

tools. Such as a pre-set up version of QuickBooks or similar software

with classes and accounts need to track expenditures, and any contract

templates, human resources documents, financial information, or other

required management tools.

If UNICEF has these resources already available, a system for

ensuring that deployed staff have downloaded the materials would be

helpful. The staff should also have a conversation with someone in the

emergency so that they are prepared, and that there is HQ follow up

support for staff once deployed staff arrive at their post.

88

Lesson 15 –

Negotiating trade-

offs among

beneficiaries

Clear communication about why one decision was made over another

would help diffuse tensions in camps and communities. Where

possible, UNICEF should seek to explain why decisions were made,

such as the decision to put public latrines in Azraq, but private latrines

in Za’atari, so that people feel informed and their expectations are

managed.

Lesson 16 –

Achieving some

programme targets

while missing others.

UNICEF should formulate a realistic strategy for the services it can

provide given the needs of the population it serves and the resources it

has access to. An important part of any organization’s strategy is

knowing what it cannot provide. Once that is identified, UNICEF

could serve in an important leadership role to assist the international

community in rallying resources and capacity to address the uncovered

and vulnerable populations.

Other Recommendations

• UNICEF should address the disparity in water allocation in Za’atari. The water

network in Za’atari provides the same quantity of water to each household regardless

of the number of people in the household. Households can have from one person to

more than eight people. If UNICEF strives for 35 liters per person per day on average,

the wide variation in household size means that some people are getting very little,

while others receive much more than they are entitled.

• The trend toward cutting funds for hygiene promotion is disconcerting. UNICEF

should avoid cutting those funds if possible, or if they must, plan for the

contingencies that could result.

• Partner interviews revealed that UNICEF’s role in the Wash in Schools initiatives

may not be clear. UNICEF should clarify its role in this category of activity and

coordinate a stakeholder strategy.

89

Works Cited 1. ACF. (2017). Azraq Camp Knowledge Attitudes and Practices Survey Report 2017. Amman:

ACF.

2. (n.d.). ACTED Programme Cooperative Agreement 23-13.

3. ACTED, World Vision, ACT, RI, and Save the Children. (2016). Azraq Camp Knowledge

Attitudes and Practices Survey Report 2016.

4. Al Wazani, K. W. (2014). The Socio-Economic Implications of Syrian Refugees on Jordan.

Amman: Konrad Adenauer Stiftung.

5. Bocco, R. (2010). UNRWA And The Palestinian Refugees: A History Within A History.

Oxford Journals.

6. Financial Tracking Service. (2018, October). Appeals and response plans 2018. Retrieved

from

https://fts.unocha.org/appeals/overview/2018/sectors?f%5B0%5D=destinationPlanIdName%3

A%22629%3ASyria%20Humanitarian%20Response%20Plan%202018%22&f%5B1%5D=de

stinationPlanIdName%3A%22658%3ASyria%20regional%20refugee%20and%20resilience%

20plan%20%283RP%29%2020

7. Financial Tracking Service. (2018, October). Financial Tracking Service. Retrieved from

Syrian Arab Republic - Civil Unrest (from 2012) 2018:

https://fts.unocha.org/emergencies/600/summary/2018

8. Freeman, M., & al, S. M. (2014). Hygiene and health: Systematic review of handwashing

practices worldwide and update of health effects.

9. Hūls, V. (2017). UNICEF WASH Action in Humanitarian Situations; Synthesis of Evaluations

201-2016. New York: UNICEF.

10. International Monetary Fund. (2017). Jordan, Selected Issues. Washington, DC: The

International Monetary Fund.

11. Jaafar, S. (2017, April 5). This refugee camp in Jordan has turned into a frontier town for Syrians escaping war. Retrieved from PRI: https://www.pri.org/stories/2017-04-05/refugee-

camp-jordan-has-turned-frontier-town-syrians-escaping-war

12. (n.d.). JEN Programme Cooperative Agreement 25-14.

13. Kis, A. e. (2016). Water demand management in the context of water services: Jordan.

Regional Centre for Energy Policy Research.

14. Ledwith, A. (2014). Zaatari: The instant City. Boston: The Afforable Housing Institute.

15. Leestma, D. (2017, October 1). Draining a lake in Daraa: How years of war caused

Muzayrib to dry up. Retrieved from Middle East Eye:

https://www.middleeasteye.net/news/draining-lake-daraa-how-years-war-caused-lake-

muzayrib-largely-disappear-1202640340

16. Luck, T. (2013, April 21). In Jordan, tensions rise between Syrian refugees and host

community. The Washington Post.

17. Middle East and North Africa Out-Of-School Children Initiative. (2014). Jordan Country

Report on Out-Of-School Children. UNICEF.

18. Ministry of Planning and International Cooperation. (2014). Jordan Response Plan 2015 for

The Syria Crisis. Amman: The Hashemite Kingdom of Jordan.

90

19. Ministry of Water and Irrigation. (2016). National Water Strategy 2016 - 2025. Amman:

Hashemite Kingdom of Jordan.

20. OXFAM PCA 32-13 Executive Summary. (n.d.).

21. REACH. (2013). Informal Tented Settlements in Jordan: A Multi-Sector, Baseline

Assessment". UNICEF.

22. REACH. (2014). Access to Water and Tensions in Jordanian Communities Hosting Syrian

REfugees. Amman: REACH.

23. REACH. (2014). Syrian Refugees Staying in Informal Tented Settlements in Jordan. Amman:

REACh.

24. redrUK. (2009, January). Water, Sanitation, and Hygiene (WASH) Cluster Coordination

handbook.

25. ReliefWeb. (2017). Humanitarian Funding Update; December 2017. ReliefWeb.

26. Salman, M., & Mualla, W. (2008). Water demand management in Syria: centralized and

decentralized views. Water Policy, 549-562.

27. Seely, N. (2012, March 1). Jordan’s “open door” policy for Syrian refugees. Foreign Policy.

28. Seely, N. (2012, October 1). Most Jordanians say no to more Syrian refugees. The Christian

Science Monitor.

29. Terms of Reference for Service Contracting - Rukban. (n.d.).

https://www.unicef.org/jordan/RUKBAN_ToR.pdf.

30. The Sphere Project. (2018). Water supply standard 1: Access and water quantity. Retrieved

from Sphere Handbook: http://www.spherehandbook.org/en/water-supply-standard-1-access-

and-water-quantity/

31. The United Nations. (2013). Syria Regional Response Plan; January to December 2013. The

United Nations.

32. UNHCR. (2012, August 28). Syria: Doubling of refugees fleeing to Jordan. Retrieved from

UNHCR USA: http://www.unhcr.org/en-us/news/briefing/2012/8/503ca1c99/syria-doubling-

refugees-fleeing-jordan.html

33. UNICEF. (2012, October). In Jordan, huge water delivery and testing operation meets the life-saving water and sanitation needs of Syrian refugees. Retrieved from UNICEF Jordan:

https://www.unicef.org/wash/jordan_66157.html

34. UNICEF. (December 2012). Syria Crisis Bi-Weekly Humanitarian Situation Report.

UNICEF.

35. UNICEF Jordan. (2012). Jordan Country programme document; 2013 - 2017, pg 11.

Amman: UNICEF JOrdan.

36. UNICEF Jordan. (2012). SitRep 20 December 2012. Amman: UNICEF.

37. UNICEF Jordan. (2012). Syrian Refugee Response Overview. Amman : UNICEF.

38. UNICEF Jordan. (2013). SitRep 25 January 2013. Amman: UNICEF.

39. UNICEF Jordan. (2013). UNICEF Annual Report 2013 - Jordan. Amman: UNICEF.

40. UNICEF Jordan. (2016). UNICEF Jordan Annual Report 2016. Amman: UNICEF.

41. UNICEF Jordan. (2017). UNICEF Jordan Annual Report 2017. Amman: UNICEF Jordan.

42. UNICEF, ACF. (2017). Azraq Camp Knowledge and Attitudes and Practicies Survey Report

2017. Amman.

91

43. WASH Sector Working Group. (2012). Water, Sanitation and Hygiene (WASH) Sector Working Group Strategy for the Syrian Refugees Response in Jordan; December 2012 - June

2013. Amman.

44. Wildman, T. (2013). Water Market System in Balqa, Zarqa, & Informal Settlements of

Amman & the Jordan Valley - Jordan August - September 2013. Oxfam.

45. Wilkes, S. (2012). Jordan opens new camp for Syrian refugees amid funding gaps. Retrieved

from UNHCR: Jordan opens new camp for Syrian refugees amid funding gaps

46. World Vision Programme Cooperation Agreement 19-13 . (n.d.).

47. Za'atari Hygiene Promotion Working Grouip. (2015). Knowledge Attitudes and Practices

Survey; Za'atari Refugee Camp 2015. Amman.

48. Za'atari Hygiene Promotion Working Grouip. (2016). Knowledge, Attitudes and Practices

Survey Za’atari Refugee Camp 2015. Amman.

49. Za'atari Hygiene Promotion Working Groups. (n.d.). 2012 Baseline KAP Survey, Za'atari.

92

Annex A: Evaluation Terms of Reference STATEMENT OF WORK/TOR

Terms of Reference for the

Evaluation of UNICEF#s response to the Water, Sanitation and Hygiene needs in Jordan as a

result of the Syrian refugee crisis (July 2012 to July 2017)

Table of Contents

1. Background and context 1

2. Evaluation purpose and objective 2

3. Evaluation scope 3

3.1 Coverage and level of results 4

3.2 Geographical coverage 4

3.3 Time-period of the evaluation 4

4. Key evaluation questions 4

4.1 Relevance 4

4.2 Effectiveness 5

4.3 Efficiency 5

4.4 Sustainability 5

4.5 Coverage 6

4.6 Coordination 6

5. Evaluation Stakeholders 6

6. Methodology 6

6.1 Desk Review 7

6.2 Meetings with WASH section 7

6.3 Visits 7

6.4 Key informant interviews (Kll) 8

6.5 Focus Group Discussions (FGD) 8

6.6 Participation in at least 2 coordination meetings in Za#atari, Azraq and the border, and the

sector 8

7. Work assignment 8

7.1 Phase 1. Desk review 8

7.2 Phase 2. Prepare an inception report with project delivery plan 8

7.3 Phase 3. Data Collection 9

7.4 Phase 4. Draft evaluation report 9

7.5 Phase 5. Finalize the Final Evaluation Report 9

7.6 Phase 6. Prepare and present two PowerPoint presentations 9

8. Management of the Evaluation 9

9. Expected deliverables & delivery dates 10

10. Official travel involved 10

11. Desired qualifications 10

11.1 Team leader 11

11.2 Two WASH Experts 11

12. Estimated duration of the contract and payment schedules 11

13. Roles and responsibilities in the evaluation process 11

13.1 The Evaluation Manager 11

13.2 Evaluation Reference Group (ERG) 12

13.3 The Evaluation Team 12

14. Performance indicators for evaluation of results 12

15. Frequency of performance reviews 12

18. Equity, gender, human rights, including child rights 15

19. Ethical principles and premises of the evaluation 15

20. UNICEF recourse in case of unsatisfactory performance 15

1. Background and context

Jordan, is described as being an arid to semi-arid country with average annual rainfall of 200

mm/year over the country. On this basis, it is one of the most water scarce countries in the world,

and has been facing chronic challenges to ensure the provision of clean water and sanitation

services to its population for a number of years. The chronic water situation has been

compounded by successive waves of refugees from neighbouring areas, the most recent being

the Syrian crisis. Since the onset of the Syrian crisis in 2011, more than 659,000 Syrian refugees

have crossed the border into Jordan to seek assistance and refuge from the atrocities underway

in Syria. Since then, the population in some areas has risen dramatically, particularly those in the

93

north of the country (by more than 60% particularly in the Northern Governorates e g. Irbid). The

2015 Census reported that the total population of Jordan as 9.5 million, showing a considerable

increase over the recent years, and 30.6% of the population is non-Jordanian, including Syrian

refugees.

While an estimated 15% of the registered Syrian refugees live in refugee camps, the remaining

85% are living in host communities, which has caused tension between Jordanians and Syrians,

due to a perception that the Syrian refugees are absorbing the limited resources. This tension has

resulted in conflicts and disputes, particularly in the northern governorates, where the additional

pressure on resources has been most acutely felt.

According to the Joint Monitoring programme (JMP) 2015, access rates to a piped water network

in urban areas is estimated at 93%, and 80% in rural areas. However, in reality, most areas have

access to water once a week (in the main cities) reducing dramatically in rural areas, with some

areas reported to receive water once a month. Due to the limited resources, and associated

limited access, average household consumption is 71 l/p/d (National Water Strategy 2016 #

2025). As a result, most households have significant amounts of water storage facilities and when

necessary, supplement these with private water tankering services. For wastewater systems, the

situation is even more critical. Prior to the Syrian crisis, an estimated 62% of the population was

connected to a sewerage system nationally, however this figure is much lower in rural areas. In

areas where there has been a significant recent increase in the population, the existing

infrastructure has been unable to support the increase in the volume of wastewater which has led

to overflows in urban areas, causing significant environmental health concerns.

As a result of the influx, in addition to the growing expansion of the Jordanian economy, as well

as natural population growth, the demand for water and wastewater services has increased

significantly, which has contributed to over expansion of groundwater from aquifers. The Ministry

of Water and Irrigation (MOWI) estimates that average rates of over-pumping exceed 200

Mm3/year resulting in seven of the twelve groundwater basins being over-pumped at between

135 to 225% of the estimated safe yields of the aquifers. The critical situation is further

compounded by high levels of leakage, estimated at more than 60% in some of the northern

governorates, in the existing systems as well as non-revenue water.

In July 2012, UNICEF commenced its Water, Sanitation and Hygiene (WASH) programme,

initially focusing on refugee response and providing WASH services in Za#atari camp, and

expanding to a total of six settlements (currently five are active) over time. The initial focus on

refugee response, developed in line with the Syria Regional Response Plan (RRP), has gradually

expanded to cover resilience in line with the National Resilience Plan which was developed in

2013 to cover the period of 2014-2016 . Also since 2012, UNICEF has led the WASH Sector for

the humanitarian response.

UNICEF continues to be a key player in providing services to refugees living in the four main

camps of Za#atari, Azraq, King Abdullah Park and Cyber City; two settlements in Rukban and

Hadalat; as well as in Informal Tented Settlements (ITSs) and host communities. Currently, the

population of the five refugee camps/settlements is estimated to be about 170,000 people. It is

understood that the camp population peaked to 120,000 in Za#atari camp in late 2013 and the

population in Za#atari declined to approximately 80,000 thereafter, with Azraq opening in April

2014. The two settlements at the border, Rukban and Hadalat, expanded rapidly in 2016, to a

current estimated combined population of 55,000. As the movement to/from the camps has been

very fluid, there is no actual figure of the number of people who have benefitted from WASH

services since 2012, however, this is estimated to be more than 400,000.

Although UNICEF has had an active presence in Jordan since 1952, UNICEF was not active in

the WASH sector prior to the refugee crisis. As a result, the UNICEF Jordan Country programme

of Cooperation for 2013-2017, when originally developed in 2012, did not include WASH

programme. Seeing the rapid scaling-up of the refugee response in 2013, UNICEF Jordan revised

its results framework for 2013-2017 to include an output-level result focusing on Core

Commitments for Children in Humanitarian Action. This was further reviewed in 2014 to reflect

resilience more prominently, at the time of the Strategic Moment of Review and Reflection and in

light of the National Resilience Plan. The rapid onset and scale-up of the crisis has meant

revisions in the UNICEF programme Results Framework.

UNICEF WASH programme has evolved to work at four different levels as follows:

- Provision of water, sanitation and hygiene services, as well as the dissemination of key

messages to refugees on a daily basis in four camps and two settlements;

- Provision of support in the host communities (infrastructural support, support to

94

vulnerable households, informal tented settlements) including WASH in schools;

- Support to the WASH sector through technical support to the Ministry of Water and

Irrigation in the development of key planning documents (National Resilience Plan 2014, Jordan

Response Plan 2015, 2016-2018, 2017-2019), strategies (National Water Strategy 2016 to 2025)

and policies, as well as the nationwide WASH in Schools Assessment (2014/5) and the National

WASH in School Standards;

- Coordination of the humanitarian WASH sector at a camp and national level.

In terms of UNICEF Country programme in Jordan, the WASH programme has been one of the

largest in terms of scale and scope, constituting about 53% (USD 78 million) of the total budget in

2013, 54% (USD 100 million) in 2014, 31% (USD 56 million) in 2015,

27% (USD 61 million) in 2016 and 30% (USD 60 million) in 2017. The costs and logistics

required, as well as the monitoring associated with the provision of services in the camps, have

been enormous. To address this, and to improve the consistency, reliability, quality and equity of

services in the camps, UNICEF has been implementing a transitioning strategy through the

construction and operation of five boreholes and two wastewater treatment plants, with water and

wastewater networks currently under construction since 2015.

2. Evaluation purpose and objective

The purpose of the evaluation is to assess whether UNICEF WASH programme met its intended

results, and to generate lessons learnt which can inform WASH programmes in similar context.

The overall objective is to conduct an independent and critical assessment to the WASH

programme and its support to the sector and government institutions, using evaluation criteria of

relevance, effectiveness, efficiency and sustainability as well as coverage and coordination in the

target communities (camps, ITSs and host communities). The Impact criterion is excluded due to

the relatively short time period since the start of the interventions. Additionally, the evaluation is

anticipated to collect evidence and lessons learnt from

UNICEF#s WASH programme implementation to inform future programming, and to develop

recommendations to further strengthen delivery and planning of WASH services in similar

protracted humanitarian crisis in view of building resilience and sustainability. More specifically, this

evaluation will aim for the following key specific objectives:

a) To assess the relevance of the WASH programme (emergency response and resilience

components) in the context of Syrian Crisis in Jordan and its national priorities. To assess the

relevance of WASH programme design to achieve the expected results, taking into account

the appropriateness of the interventions to targeted communities.

b) To assess the effectiveness of the WASH programme and to measure to what extent the

programme has achieved its set results in an equitable manner.

c) To assess the efficiency of the WASH programme and to what extent the programme

has used resources (human, financial and others) and coordinated in an efficient manner over

time. To assess the cost of the response per unit of aggregation as compared to the cost being

incurred by similar WASH programme implemented in other countries.

d) To assess the sustainability of the WASH programme and its results, taking into account

the institutionalization and capacity building of national counterparts as well as a potential exit

strategy.

e) To assess the coverage of the most vulnerable population by the WASH programme and

provision of quality services in a coherent manner.

f) To assess the coordination of UNICEF as a sector lead to avoid duplication and gap and

enable effective partnerships to reach the most vulnerable population.

g) To document lessons learnt and good practices that will inform future programming,

replication in other countries in the region, suggesting different options for UNICEF and partner

agencies.

It is anticipated that the evaluation, and the expected Lessons learned and good practices, will

also benefit the Ministry of Water and Irrigation, implementing partners in addition to donors of

Jordan, as well as future UNICEF WASH programming in the region and globally.

3. Evaluation scope

This evaluation will be summative in nature. It should focus mainly on UNICEF#s WASH

programme as it evolved from refugee response to include resilience building from July 2012 to

July 2017. This includes the initial response which aimed to meet the urgent needs of Syrian

refugees, primarily residing in refugee camps, as well as resilience programming to support host

95

communities and government bodies to respond to the impact of the crisis. This evaluation will

assess UNICEF#s WASH programme which includes water, sanitation (WASH facilities, solid

waste management and desludging) and hygiene promotion implemented by UNICEF#s partners

and contractors. While UNICEF started providing life-saving WASH services at refugee transit

centres in March 2012, this evaluation will cover UNICEF#s WASH programme from the opening

of the Za#atari refugee camp in July 2012, and will look at programming in the four camps of

Za#atari, Azraq, Cyber City (currently closed) and King Abdallah Park, in addition to host

communities (and informal tented settlements), as well as Rukban and Hadalat at the

Syrian/Jordanian border. Although there are Syrian refugees registered in all 12 Governorates of

Jordan, 90% of the refugees are registered in Zarqa, Irbid, Mafraq and Amman Governorates,

with 99% of the refugees registered in ten Governorates.

In addition, this evaluation will assess support to the sector through technical support provided to

the Government, and UNICEF role in leading the sector at the camps and at the national level.

Stakeholder participation is important for this evaluation, and their views and inputs should be

reflected through means such as interviews and focus group discussions. Special emphasis will

have to be made on Ministry of Water and Irrigation (MWI), Ministry of Education (MoE), UNHCR,

WASH sector, local community leaders and International Partners (IPs) who were directly

involved in the UNICEF WASH response since its early stages. In addition, the voices of

caretakers, parents (both mothers and fathers) and children who benefited from UNICEF#s

WASH intervention should be captured during stakeholders consultations. Representatives from

the Ministry of Water and Irrigation will comprise a key component of the Evaluation Reference

Group.

3.1 Coverage and level of results

The evaluation will focus on four components of the UNICEF#s WASH programme in Jordan and

will include:

a) Provision of equitable WASH services and the dissemination of WASH messages in the four

camps (Za#atari, Azraq, King Abdullah Park and Cyber City) and two settlements in Rukban

and Hadalat, to the most vulnerable people namely children and women

b) Provision of gender equitable support to the most vulnerable people in host communities (in

all 12 Governorates) namely to children and women

c) Support to the WASH sector through technical support to the Ministry of Water and Irrigation

d) Coordination of the sector at camps and national level

3.2 Geographical coverage

The geographical scope of the evaluation will cover the following:

e) The four major camps of Za#atari, Azraq, King Abdullah Park and Cyber City; and two

settlements of Rukban and Hadalat. In addition, the evaluation will cover a number of Informal

Tented Settlements (ITSs) (in 2017, 111 ITSs are provided with WASH interventions) and will be

based on discussions with the consultant;

f) Host community interventions in all 12 Governorates. 3.3 Time-period of the evaluation The period July 2012 to July 2017 will be considered as the timeframe for this evaluation.

4. Key evaluation questions

The evaluation will apply Organization for Economic Cooperation and Development Assistance

Committee (OECD-DAC) criteria of relevance, efficiency, effectiveness, and sustainability as well

as relevant humanitarian criteria, taking into consideration the purpose, objectives and scope of

the evaluation. The Evaluation team should answer the following key questions, but not limited to.

They are requested to unpack them and should propose more detailed ones as part of the request

for proposal and inception report.

The humanitarian criteria of coverage and coordination are directly addressed and should be

investigated throughout the evaluation process.

4.1 Relevance

a) To what extent were affected people consulted on their preferences with regards to water and

sanitation service options and level of services? Were interventions appropriate in terms of

meeting their basic needs? Was there a feedback loop and monitoring system in place for

reflecting the learning and evidence to improve programming, especially for the humanitarian

response?

b) To what extent did the interventions consider the needs and the priorities of the MWI and

were they coherent with global references and the regional and national response to the Syria

96

Crisis?

c) How responsive was the UNICEF WASH programme over time to changes in the external

environment?

4.2 Effectiveness

a) To what extent did UNICEF#s response achieve its intended outcomes (as stated in

UNICEF#s Country programme Outcome and Outputs as given in Annex A)? To what extent

did UNICEF achieve equity results, especially for children and women specific interventions?

b) What were the strengths and weaknesses of the interventions in achieving results in the

camps/settlements, ITSs and host communities on the following:

- Water (Litre/day, supply, quality, consistency, reliability, accessibility, and equity)?

- Sanitation (adequacy, appropriateness, accessibility)?

- Hygiene promotion (appropriateness of the messages, timeliness, behaviour change)?

- WASH in schools?

- National level support?

4.3 Efficiency

a) To what extent did the programme use available resources in an economic manner to provide

WASH services and facilities in camps/settlements and host communities, to ensure child

rights, equity and gender equality?

b) To what extent were the costs to deliver water and wastewater services, as well as solid

waste management, rationalized and optimized in the camps and settlements to ensure Value for

Money? How do these compare to similar situations at the local or regional level? Could more

cost effective operations/interventions have been undertaken at an earlier stage?

c) To what extent did UNICEF try to minimize duplication/gap in WASH interventions in

camps? And how early did UNICEF move to more cost-efficient interventions?

d) To what extent were resources (financial and human resources) allocated for the range of

intervention types, with particular attention paid to the respective allocations for humanitarian

response and resilience, to achieve the intended results?

e) To what extent did UNICEF advocate with the Ministries to mobilise their available

resources to address the identified needs? And how efficient was UNICEF#s support to ministries

(i.e. MWI and MOE) to reaching most vulnerable people? Did the WASH programme maximise

the potential collaboration with other UNICEF programme sections and partners?

4.4 Sustainability

a) Was the infrastructure in the camps constructed in such a manner to ensure long term

functionality, and will continue to service the most vulnerable people even when UNICEF is no

longer directly supporting? This should be investigated in terms of Water and Wastewater

infrastructure.

b) To what extent did UNICEF take sustainability into account at the early stage for water,

wastewater and solid waste management services (e.g. transition to local government institutions

and local capacity)?

c) To what extent did UNICEF#s interventions increase the resilience of the Government

and respective target populations? To what extent will UNICEF-supported projects (schools, ITSs,

water and wastewater infrastructure, mobilisation) continue to be operational after the end of

UNICEF#s financial support

d) Does the MWI have the necessary mechanisms in place and capacity to ensure the

water and sanitation needs of the most vulnerable people will be met as a result of UNICEF#s

support and advocacy, if another large scale humanitarian crisis occurs in the future?

e) To what extent did UNICEF document its WASH interventions and make available to the

sector learning on water, wastewater, solid waste management and mobilisation?

f) To what extent were environmental protection and sustainability taken into account in the

design and implementation of programmatic interventions?

4.5 Coverage

a) Did UNICEF#s WASH response identify and reach the most vulnerable people? To what

extent did specific interventions strike an equitable balance between interventions in

camps/settlements and in host communities, to reach the most vulnerable people?

b) Did UNICEF as Sector Lead ensure that the needs of those in most need were

addressed in a coherent manner, particularly in comparison to alternative existing systems at the

local or regional level (i.e. UNRWA camps)? If not, how?

97

4.6 Coordination

a) To what extent did UNICEF role as WASH sector lead enhance response processes and

results to avoid duplication and gap in interventions?

b) Were partnerships effective and leveraged to the maximum extent to assist the most

vulnerable population in camps/settlements, ITSs and in host communities?

5. Evaluation Stakeholders

The stakeholders that were found relevant to this evaluation by the WASH programme are listed

in the below table 1. They were divided into 2 categories in terms of interest and influence.

6. Methodology

The evaluation methodology will mainly utilise qualitative methods, with a combination of tools for

data collection and analysis. The participatory approach is suggested to ensure participation of

various social groups including the most marginalized. Quantitative analysis will be carried out

through the desk review of the following major data sources, but not limited to:

Camps:

- Monthly, quarterly and final reports from partners

- Third party monitoring data (UNOPS) on beneficiary interviews, wastewater volumes, water

delivery and water quality

- Knowledge Attitude and Practice surveys (Za#atari 2013 & 2014/2015; and Azraq 2016 &

2017)

- Comprehensive Child Focussed Assessment (Za#atari 2015; Azraq 2015 and 2017)

- Partnership agreements and budgets

- Field visit reports

- Monthly Donor updates and Situation Reports

- Assessment reports

- Needs assessments -Transitioning Strategies

- Contracts

- Terms of Reference for the Groundwater Assessment for Za#atari and Azraq

- Water level and quality monitoring data

Settlements (border):

- Monthly Border updates and Situation Reports

- Partnership agreements

- Partner reports

- Contracts

- Third party monitoring data

- Security incident reports

- Distribution data


Host Communities:

- Nationwide WASH in Schools assessment


- Partner reports


- Contracts

- Situation of Syrian children in host communities in Jordan (Frequent monitoring)

Table 1. Potential stakeholders Interest Influence

- MWI

- MoE

- Donors

- UNICEF

- UNHCR

- Sector

- NGOs - Women

- Children (girls and boys)

- Local Communities

- WASH Committees

- Schools (including parents, teachers and students)

98

ITS:


- Partner reports

- Field visit reports -Transitioning Strategy

National-level support:

- Vulnerability assessments for water and wastewater (2014/2015; and 2016)

- Jordan Response Plan (2014; 2015; 2016/2017; 2017/2019)

- Regional Response Plans

- National Water Strategy (2016 # 2025)

The evaluation team is expected to propose a relevant set of tools and methods, as appropriate

to answer the evaluation questions. This should be detailed in the form of an evaluation matrix,

showing methods against proposed evaluation questions. Methodologies proposed by the

consultant will be subject to consultation with the UNICEF WASH and Planning Monitoring and

Evaluation (PME) sections, and WASH Working Group as appropriate. As part of methodology, it

is essential to articulate how ethical considerations will be taken into account in the design of the

evaluation and throughout the evaluation process. Methodology should include, but not limited to

the following:

6.1 Desk Review

Undertake a Desk Review: review of similar evaluations such as the recent Regional assessment

of UNICEF#s response, review of WASH Annual Reports, partner programme Cooperation

Agreements and programme Documents, contracts, funding tables, KAP surveys, third party

reports, UNOPS and partner reports, minutes and resources of Sector meetings, consultation of

UNHCR information platforms, review of Outputs (Water Strategy, vulnerability maps) and other

documents relevant to the evaluation.

6.2 Meetings with WASH section

At the start of the evaluation, there will be a telephone call for a briefing with key WASH and PME

sections personnel for an overview of the activities undertaken and the timeline of events.

Arrangements for additional meetings with the WASH section will be arranged as needed,

throughout the evaluation process.

6.3 Visits

Conduct visits to some of the main interventions in the camps and host communities:

a) Visits to the six camps/settlements of Za#atari, Azraq, King Abdullah Park, Cyber City, Rukban

and Hadalat;

b) Visits to the host communities.

6.4 Key informant interviews (Kll)

Conduct interviews with relevant stakeholders and beneficiaries as follow:

a) Duty bearers: This would include WASH section personnel, UNICEF Management, UNICEF

WASH Regional team for the Middle East and North Africa Regional Office, Ministries, NGO

partners (main partners in camps, WASH in Schools and ITSs) and UNHCR representatives.

b) Right holders: Camps, settlements, host community and ITS representatives

6.5 Focus Group Discussions (FGD

Conduct Focus Group Discussions with:

Beneficiaries in the five major active camps of Za#atari, Azraq, King Abdullah Park, Rukban and

Hadalat and the host communities, ITSs and schools. The FGDs should be disaggregated by age

and gender (women, men, children and youth);

6.6 Participation in at least 2 coordination meetings in Za#atari, Azraq and the border, and the sector

7. Work assignment

The key tasks to be undertaken under this evaluation are as follows (see summary in Table 2):

Table 2: Key tasks of the evaluation

Phase Activity

Phase 1 Desk review

Phase 2 Prepare an Inception Report

Phase 3 Data collection

99

Phase 4 Draft evaluation report

Phase 5 Final Evaluation Report

Phase 6 PowerPoint presentations

7.1 Phase 1. Desk review

Undertake a desk review of all relevant documents (e.g. the recent Regional assessment of

UNICEF#s response), review of WASH Annual Reports, programme Cooperation Agreements,

funding tables, consultation of UNHCR information platforms, review of Outputs (Draft Water

Strategy) etc.

7.2 Phase 2. Prepare an inception report with project delivery plan

The Inception Report should be developed in line with UNICEF global standards and should

include: (1) purpose and scope, (2) evaluation criteria and questions, (3) evaluation methodology,

(4) evaluation matrix, (5) a detailed evaluation delivery plan within the designated timeframe

(seven months), and (6) outline of the final report. Methodology should also include the evaluation

approach, and theory of change. The outline of the evaluation report should be in line with the

UNICEF-Adapted UNEG Evaluation Reports Standards, and UNICEF Global Evaluation Report

Oversight System.

The inception report will be used as an initial point of agreement and understanding between the

evaluation team and the evaluation managers and should be approved by the evaluation

Reference Group. Field visits and data collection can#t start before finalization of the inception

report.

7.3 Phase 3. Data Collection

Conduct the necessary interviews, field visits, consultations as agreed in the Inception Report,

which are expected to include:

a) Meetings with WASH section;

b) Conduct Field Visits to the six camps/settlements, randomly selected schools and ITSs, in

addition to host communities;

c) Kll with current and historic duty bearers (including ex-UNICEF and partner staff);

d) Kll with right holders;

e) FGD with rights holders and WASH partners;

f) Attend Coordination meetings in the camps.

7.4 Phase 4. Draft evaluation report

Perform the analysis and draft the evaluation report. It must be compliant with UNICEF-adapted

UNEG evaluation report standard (http://www.uneval.org/). The Evaluation report should

systematically answer the key evaluation questions included in the objectives sections of this

TOR. It should fairly and clearly represent the views of the different actors/stakeholders. It should

clearly give the findings, conclusions and recommendations in a way that is substantiated by

evidence. All recommendations included in UNICEF#s

evaluation require management response. It is pertinent that all recommendations are clear and

specific to what UNICEF Jordan could do or influence.

Validation process of the inception report will be as follow:

a) Disseminate the Evaluation Report to the WASH and Planning Monitoring and Evaluation (PME) Sections;

b) Collate feedback and integrate comments where agreed;

c) Finalize the draft evaluation report and submit to WASH and Planning Monitoring and

Evaluation (PME) Sections.

7.5 Phase 5. Finalize the Final Evaluation Report After review of the draft report, the

consultant will:

a) Collate final feedback made on the draft report and integrate all comments;

b) Finalize the final evaluation report and submit to WASH and Planning Monitoring and

Evaluation (PME) Sections.

The final report will be in English and Arabic of 60 to 80 pages in length (excluding annexes). It

will also contain an executive summary of no more than 4 pages that includes a brief description

of the programme, its context and current situation, the purpose of the evaluation, its methodology

and its major findings classified as per the evaluation objectives, conclusions and

recommendations.

http://www.uneval.org/

100

7.6 Phase 6. Prepare and present two PowerPoint presentations

Prepare two PowerPoint presentations detailing the findings of the evaluation. They will be

presented during:

a) An open session for the Jordan Country Office;

b) And a second one for key sector partners.

8. Management of the Evaluation

To ensure independence, this evaluation will be co-managed by WASH Specialist and Monitoring

& Evaluation Officer at UNICEF Jordan Country Office. A reference group will be formed,

engaging subject matter experts from inside and outside of UNICEF, including the UNICEF

Regional Office for Middle East and North Africa. Members of the reference group will be engaged

and consulted at key milestones of the evaluation process such as review of the TORs, inception

report, and draft reports.

Selection of the evaluation team will be made through an open and competitive bidding process.

Review of technical proposal will be done by at least three members.

WASH Specialist will be responsible for technical oversight in relation to the WASH programing,

support to the evaluation process by providing necessary background information, data, contact

information, as well as management of contractual issues once the bidding process completes.

Monitoring & Evaluation Officer will oversee and quality assure the evaluation, ensuring that it

complies with the UNICEF global standards and norms.

The evaluation team will report to WASH Specialist and Monitoring & Evaluation Officer at

UNICEF Jordan Country Office who will serve as key contact points.

9. Expected deliverables & delivery dates The assessment team will be expected to submit the following deliverables as per the start date of 02.OCTOBER.2017 (table 3):

Table 3: Deliverables and timeframe # Deliverables Type and language Delivery Date 1 Inception

Report with Project Delivery Plan Soft

(within 3 calendar weeks of start date, or 21 working days)

2 Draft Evaluation Report Soft copy (in English) 04.MARCH.2017

(within 5 calendar months of the start date, or 126 working days)

3 Final Evaluation Report accepted Soft copy (in English and Arabic) 02.MAY.2018

(within 7 calendar months of the start date, or 180 working days)

4 Two PowerPoint presentations developed and presented at two sessions Soft copy (in

English and Arabic) 09.MAY.2018

10. Official travel involved

Travel to Amman and field locations in Jordan should be anticipated as per the evaluation

methodology and deliverables. All travel costs should be planned properly in financial proposal.

UNICEF will not provide transport support or field travel arrangements, thus the estimated cost of

travel should be included in the financial proposal. Please note that if selected, the contract can

be a supporting document to obtain entry visa (if necessary). UNICEF will be unable to secure

travel visas. Any applicable per diems should be included as part of the lump sum price proposal.

11. Desired qualifications

Given the broad scope, it is expected that the evaluation will be carried out by an institution or a

team of technical specialists, overseen by the Evaluation Team Leader. The evaluation team will

consist of at least one international expert as the Evaluation Team Leader, as well as two WASH

experts preferably with Arabic speaking skills. Any other capacities, as deemed required for the

evaluation such as an expert on Value for Money should be proposed by the evaluation team. It is

optional to include more than one expert to ensure that all aspects of water, sanitation and

hygiene (WASH) are adequately covered.

11.1 Team leader

101

- Desirable 15 years of experience in conducting evaluation with proven experience in leading

evaluations of similar scope, such as WASH evaluations

- Demonstrated experience undertaking comprehensive programme evaluations in

emergency contexts;

- Understanding of Cluster/Sector Coordination. Experience in supporting or working with

clusters and/or sectors an asset;

- Experience in working in refugee/IDP camps;

- Proven experience in projects related to gender equality and human rights, including child

rights;

- Proven experience in institutional and policy reviews in the field of water and sanitation;

- Excellent writing skills;

- Fluency in English;

- Preferable to have experience working in the Middle East and North African region;

- Knowledge of Arabic a considered advantage.

11.2 Two WASH Experts

- At least five years of experience implementing water supply interventions in development or

emergency contexts (at least one expert should have this level of experience);

- At least five years of experience implementing sanitation interventions in development or

emergency contexts (at least one expert should have this level of experience);

- At least two years of experience implementing hygiene promotion/community mobilisation

interventions in development or emergency contexts (at least one expert should have this level of

experience);

- Experiences in evaluating WASH programmes an advantage;

- Preferable to have experience working in the Middle East and North African region;

- Fluency in Arabic would be a considered advantage

If the team members do not speak Arabic, then arrangements should be made by the successful

bidder to ensure translation/interpretation services are provided, and are included in the costs in

the financial proposal

12. Estimated duration of the contract and payment schedules

The contract is expected to commence on 02.OCTOBER.2017 and will be undertaken over a

maximum of seven months, and should be completed by 09.MAY.2018.

Payment is contingent upon approval by the UNICEF Evaluation Manager and will be made in four

instalments. The following payment schedule is proposed:

- 30% of the total contract value will be paid upon acceptance of the Inception Report

with Project Delivery Plan;

- 40% of the total contract value will be paid upon

acceptance of the Draft Evaluation Report;

- 30% of the total

contract value will be paid upon finalisation and acceptance of the final evaluation report

and two PowerPoint

presentations (one to the Jordan Country Office and a second one for key sector partners) # the

presentations will take place in the UNICEF Jordan Country Office.

13. Roles and responsibilities in the evaluation process

13.1 The Evaluation Manager

- Manage the evaluation process throughout the evaluation (design, implementation and

dissemination and coordination of its follow

up);

- Convene the evaluation reference group meetings;

- Facilitate the participation of those involved in the evaluation design;

- Coordinate the selection and recruitment of the evaluation team;

- Safeguard the independence of the exercise and ensure the evaluation products meet

quality standards;

- Connect the evaluation team with the wider programme unit, senior management and key

102

evaluation stakeholders, and ensure a fully inclusive and transparent approach to the evaluation;

- Facilitating the evaluation team#s access to all information and documentation relevant to

the intervention, as well as to key actors and informants who should participate in interviews, focus

groups or other information-gathering methods;

- Provide the evaluators with overall guidance as well as with administrative support;

- Oversee progress and conduct of the evaluation, the quality of the process and the

products;

- Approve the deliverables and evaluate the consultant#s/team#s work in consultation with

Evaluation reference group and will process the payments after submission of the deliverables that

respond to the quality standards;

-Take responsibility for disseminating and learning across evaluations on the various programme

areas as well as the liaison with the National Steering Committee;

- Disseminate the results of the evaluation.

13.2 Evaluation Reference Group (ERG)

The ERG will comprise the representatives of the major stakeholders including Evaluation

Manager, Chief of the WASH Section, Planning, Monitoring and Evaluation Officer, and a

representative from the Ministry of Water and Irrigation. The Regional WASH Advisor and the

Regional Monitoring and Evaluation Advisor will also provide a quality assurance of ToR,

inception report, draft and final evaluation report and participate at ERG meetings as necessary.

The ERG will:

- Provide clear specific advice and support to the evaluation manager and the evaluation

team throughout the whole evaluation process;

- Review the ToR, inception report and draft evaluation report and ensure final draft meets

the UNICEF-Adapted UNEG Evaluatio

103

Reports Standards; - Review and provide comments and feedback on the quality of the evaluation process as well as on the evaluation products (comments and suggestions on the TOR, draft reports, final report of the evaluation).

13.3 The Evaluation Team

The evaluation team will report to Evaluation Manager and conduct the evaluation by fulfilling the

contractual arrangements in line with the TOR, UNEG/OECD norms and standards and ethical

guidelines. This includes developing of an evaluation plan as part of the inception report, drafting

and finalising the final report and other deliverables, and briefing the commissioner on the

progress and key findings and recommendations, as needed.

14. Performance indicators for evaluation of results

Proposed timelines for completion of activities are met and deliverables submitted on time with

good quality and as per the standards described in the TORs as well as UNICEF/UNEG global

standards. Overall performance at the end of the contract will be evaluated against the following

criteria: timeliness, responsibility, initiative, communication, and quality of the products delivered.

15. Frequency of performance reviews

Performance review will be conducted after completion of each deliverable. Bi-weekly meeting or

meeting at a frequency mutually agreed will be held between the evaluation team and managers.

Equity, gender, human rights, including child rights

The evaluation process will need to follow -but not limited to- guidelines and procedures on equity,

gender and human/child rights made by UNICEF and the United Nations found in the Zip File

"Annex 1: Equity, gender, human rights, key DOCs".

Main Instruments or policies on human rights, including child rights and gender equality, that should

guide the evaluation process are:

- Child Rights and International Legal Framework, https://www.unicef.org/tdad/index_56373.html

- Convention on the Rights of the Child (CRC), convention and subsequent comments,

guidelines, declarations: https://www.unicef.org/crc/

- Sustainable Development Goal (SDG), namely SDG 5 "Achieve Gender Equality and

Empower All Women and Girls",

https://sustainabledevelopment.un.org/post2015/transformingourworld, and

http://www.unwomen.org/en/news/in-focus/women-and-the-sdgs/sdg-5-gender-equality

- Transforming our world: the 2030 Agenda for Sustainable Development,

http://www.un.org/ga/search/view_doc.asp?symbol=A/RES/70/1&Lang=E

- UNICEF strategic 2014-2017; https://www.uniceforg/strategicplan/index_68123.html

- CEDAW, http://www.un.org/womenwatch/daw/cedaw/

- UNICEF Gender Action Plan 2014-2017

- Security Council Resolution 2122 on Sexual Violence in Armed Conflict (2013),

http://unscr.com/en/resolutions/doc/2122

- International Conference on Population and Development (1994), http://www.unfpa.org/icpd

- Security Council Resolution 1325 on Women and Peace and Security (2000),

http://www.un.org/womenwatch/osagi/wps/

Ethical principles and premises of the evaluation

The evaluation process will need to adhere to the United Nations evaluation norms and standards

(https://www.unicef.org/supply/files/ATTACHMENT_IV-

UNICEF_Procedure_for_Ethical_Standards.PDF), and the UNICEF ethical standards found in the

Zip File "Annex 2: UNICEF Ethical Standards". Inception report, data collection tools, and final

report will be reviewed on ethics by Institutional Review Board and need to be cleared.

https://www.unicef.org/tdad/index_56373.html

https://www.unicef.org/crc/

https://sustainabledevelopment.un.org/post2015/transformingourworld

http://www.unwomen.org/en/news/in-focus/women-and-the-sdgs/sdg-5-gender-equality

https://www.uniceforg/strategicplan/index_68123.html

http://www.un.org/womenwatch/daw/cedaw/

http://unscr.com/en/resolutions/doc/2122

http://www.unfpa.org/icpd

http://www.un.org/womenwatch/osagi/wps/

https://www.unicef.org/supply/files/ATTACHMENT_IV-UNICEF_Procedure_for_Ethical_Standards.PDF

https://www.unicef.org/supply/files/ATTACHMENT_IV-UNICEF_Procedure_for_Ethical_Standards.PDF

104

Annex B: Anonymized List of Interviews Code GENDER ORGANISATION Role Date

01 Male Unicef Jordan Management July 24

02 Female Unicef Jordan Technical Specialist July 24

03 Female Unicef Jordan Technical Specialist July 25

04 Male Unicef Jordan Technical Officer July 25

05 Male Unicef Jordan Information

Management Officer July 25

06 Male Mercy Corps Technical Specialist July 25

07 Male Mercy Corps Technical Specialist July 25

08 Male UNICEF Jordan Technical Specialist July 26

09 Male Unicef Jordan up to

Sept. 2017 Technical Specialist July 26

10 Male Oxfam Jordan Management July 26

11 Male Oxfam Jordan Project Leader July 26

12 Female Norwegian Refugee

Council Education Specialist July 29

13 Male Norwegian Refugee

Council Project Manager July 29

14 Female ACTED Management July 29

15 Male UNICEF Monitoring and

evaluation July 30

16 Female UNICEF Management July 30

17 Male Ministry of Water and

Irrigation Official July 30

18

Male

Water Authority of

Jordan Official July 30

19 Male World Vision Project Manager July 30

20 Male UNICEF Quality Assurance July 31

21 Female ACF Management July 31

22

Male ACF Technical Officer July 31

23 Male Unicef Ruwaished,

Jordan Teachnical Officer July 31

105

24 - 27 4 Males Contractor – four

participants July 31

28 Male UNICEF Management July 31

29 Male Kfw Management Aug 1

30 Male Unicef Regional

Office, Amman Technical Advisor Aug 1

31 Male US State

Department/BPRM Official August 1

32 - 34

2 males

and 1

female

Future Pioneers (local

NGO)

Management

Monitoring and

evaluation officer

Technical Specialist

Aug 2

35 Female Unicef Officer Aug 5

36 Female Unicef Technical Officer Aug 5

37 Male Unicef Technical Officer Aug 5

38 - 43

2 males

and 4

females

Oxfam Project Staff Aug 5

44 - 46 3 males Oxfam Project Staff Aug 5

47 Male UNHCR Management Aug 6

48 Male Unicef Technical Specialist Aug 6

49 Male UNICEF Management August 6

50 Male UNOPS Management Aug 24

51 Female ACTED Management

52 Male Mercy Corps

Management

Sept 4

Annex C: List of Sites Visited

1. Irbid Governorate Host Community programme

2. Ramtha

3. ITS site 702

4. ITS site 709

5. Ruyaished Host Community

6. Rukban Settlement

7. Za’atari Camp

106

8. Azraq Camp

9. King Abdullah Park Camp

10. WIS Schools Visit in Irbid

Annex D: Documents Consulted

Camp Assessments

UNICEF_REACH_Comprehensive Child Focused Assessment Azraq Refugee Camp

UNICEF_REACH_Comprehensive Child Focused Assessment Za'atari Refugee Camp

107

Camp Contracts

6.2.2 Aref & Mohamed Lafi Contract for repair septic tanks

f6.2.3 Bab El-Amood Contracts

6.2.7 IMDAD

6.2.8 Irbid District Electricity Company contract 43141693

t6.2.9 Open Hands - Monitoring of Refugee Camp wastewater Disposal Contract

6.2.10 Team Connect - Contract # 43143909

6.2.12 Safe Gaz PO # 43133530

6.2.13 Royal Scientific Society contract #43132050

6.2.18 GIS Contract - Ghazi Gsouss

6.2.21 EEC

6.2.25 Site Group

6.2.34 Loyality

6.2.35 New Day

6.2.37 Engicon

6.2.38 ACE

6.2.39 GITEC

6.2.40 ITC

6.2.41 Aquatreat

6.2.42 Health Bureau

6.2.44 Nimer al lawzi

6.2.47 Desluding Zaatari 2015-16

6.2.51 Specalized company for trade service - Hygien Kits - LLTA 42407750

6.2.53Bakhos

6.2.59 JEA

6.2.60 Al Mehwar

6.2.61 GAMA

6.2.64 Al Faris

6.2.65 Al Manaseer

Field Trip Reports

ACF Report - March2015.docx

programme visit and trip report - ACF-Dec 2016-0&M.

programme visit and trip report - ACF-Dec 2016-soft

programme visit and trip report - ACF-dec-O&M.docx

108

programme visit and trip report - ACF-July 2017-0&M

programme visit and trip report - ACF-June 2017-0&M

programme visit and trip report - ACF-June-V2.

programme visit and trip report - ACF-Nov-GTD

programme visit and trip report - ACF-Oct-GHWD

Trip report - Q4 - ACF1.

Trip report - Q4 - ACF2

Trip report - Q4 - ACF3

15 06 ACTED Prog Visit.pdf

16 01 20 Za programme visit and trip report.docx

16 06 08 Prog Review visit to Azraq.pdf

16 07 27 programme visit and trip report-ACTED-28.07.2016

Prog Review visit to Azraq

17 0411 Prog Review visit to Zaatari

17 06 20 Prog Review visit to Azraq.pdf 83 ACTED Report April 2017

Jen Reports

Q1 2015

Q1 2015

Q1 2016

Q1 2017

Q2 2015

Q2 2015

Q2 2016

Q2 201

Q3 2014

Q3 2014

Q3 2015

Q3 2015

Q3 2016

Q4 2014

Q4 2014

Q4 2015

Q4 2016

Oxfam Reports

programme visit_trip report-0xfam20140917

109

programme visit_trip-report-Oxfam20141231

OXFAM-Q1-Prog-Visit_Trip-Report-20150323

Notes from programme Visit Meeting_Oxfam_290615

OXFAM-Q2-Prog-visit-trip-report-20150629

Notes from programme Visit Meeting.Oxfam 300915



World Vision

Prog-Visit_Trip-Report-World_Vision-20141109

programme visit and trip report - World.Vision and IMC-November -SAL

programme visit and trip report - World.Vision-December -SAL

programme visit and trip report - World_Vision-Feb-SAL

programme visit and trip report - World.Vision-January -SAL

programme visit and trip report - World_Vision-Mar-SAL

programme visit and trip report -WVI-july-SAL

programme visit and trip report -WVI-June-SAL

programme visit and trip report -WVI-sep

Trip report - Q2 - WVI

Trip report - Q3 – WVI

Program Cooperative Agreements

WV PCA19-13 Prog visit report Azraq - 9 Nov 2014

WV PCA 19-13 Prog visit report Azraq - 9 Nov 2014

WV PCA 19-13 Prog visit report Azraq -11 Sep 2014

WV spot check 2014 0119.pdf

WVI PCA 19-13 Prog visit report Azraq -19 Feb 2015

WVI PCA 19-13 Prog visit report Azraq -19 Feb 2015

lWVI PCA 19-13 Prog visit report Azraq - 25 Mar 2015

WVI PCA 19-13 Prog visit report Azraq - 25 Mar 2015

WVI PCA 19-13 Prog visit report Azraq V5 & J stations - 25 Jan 2015

WVI PCA 19-13 Prog visit report Azraq V5 & J stations - 25 Jan 2015

l§J WVI PCA 19-13 Prog visit report Azraq-23 June 2015

L WVI PCA 19-13 Prog visit report Azraq-23 June 2015

KAP Surveys

2017 AZRAQ Camp Knowledge Attitudes and Practices Survey Report Final.

110

KAP 2013 Final Report

KAP 2014 Final Report 15.03.15

KAP 2015 Final Report 21.03.16

KAP 2015 Executive Summary 21.03.16

ACF

ACF PCA14-PD 1-2015-Azraq

ACF PCA 26-14-Houshold-(Ajloun-lrbid)

ACF-PCA-14-PD 02 Irbid

ACTED

ACTED PCA 02 PD 02 2017

ACTED PCA 02-PD 01-2015

ACTED PCA 23-13

JEN

JEN PCA 01-PD 01-2015

JEN PCA 01-PD 03-2015 (amend to PCA 25-14)

JEN PCA 01-PD 04 - 2017 Za'tari and Mafraq

JEN PCA 9-13

JEN PCA 18-15

JEN PCA 25-14

JEN PCA 34-13

Oxfam

OXFAM PCA 13-PD 1-2015



OXFAM PCA 32-13

SRIPCA 33-13

World Vision (WV) May 2013

Factsheets on Syrian Refugees October 2012 - December 2017

Monthly, Quarterly, and Annual Reports from Partners

Partnership Agreements, Budgets, and Ammendments, 2012 - 2017

Other Documents

Sitrep Reports December 2012 - August 2018

Third Party desludging, water and solid waste Monitoring data (UNOPS)

Water monitoring data, Yarmouk; 2015-2017

Jordan Nationwide Assessment in Public Schools for Strategic Planning

111

Running on Empty: The situation of Syrian Children in Host Communities in Jordan

ITS Field Visit Reports

Host Community Field Visit Reports

AZRAQ Camp Knowledge Attitudes and Practices Survey Report 2016_HJ_HP unicef

comment

KAP Report 2017(non-public) REACH_JOR_factsheet_CCFA_Feb2017_WASH

REACH_JOR_UNICEF Zaatari Camp Assessments List_Oct 2017

REACH_UNICEFJTS_MS_AUGUST2014_FINAL

UNICEF_REACH_Comprehensive Child Focused Assessment Azraq Refugee

Camp_J.._.pdf p UNICEF_REACH_Comprehensive Child Focused Assessment Za'atari

Refugee Camp UNICEF_REACH_Factsheet_Zaatari_CCFA_WASH

2017 UNICEF Jordan Audit

Access to water and tensions 2014

ACTED Gender Task Force

UNICEF Jordan Annual Reports 2012 - 2017

Controling Groundwater Thesis.html L' Coordination Study.pdf & Cyber City Map

EVALUATION_OF_THE_WASH_PROGRAMME_WITHIN_THE_UNICEF

COUNTRY_PROGRAMME_IN_LEBANON_2013-2015_2017-00

Manual for maintenance cleaning and hygiene promotion in schools - English

Explanation of Gama Contract

Groundwater Jordan

Handicap International Advice

Market Systems 2013

History of Za'atari

History of the Palestinian Refugees in Jordan

IT Humanitarian Funding Update_GHO_31DEC2017

IMF Jordan Report 2017

Syrian refugees in Jordan, A Protection overview January 2018

C Jordan, 2012 Article IV Consultation; IMF Country Report 12_119; March 23, 2012

Jordan_Water_Strategy_2008-2022

Jordanian attitudes toward refugees 2012

MoWI Annual Report 2016

National Water Strategy( 2016-2025)-25.2.2016

Equity, scalability and sustainability in UNICEF WASH programming: A thematic meta-

analysis of UNICEF’s WASH evaluations 2007-2015

Water, Sanitation, and Hygiene (WASH) Cluster Coordination Handbook; January 2009

112

WASH Sector Coordination; Minutes of weekly Za’atari camp WASH sector coordination

group.

Annex E: Data Collection Tools

Key Informant and Key Stakeholder Interview Guide

Introduction for the Interviewer ISG is conducting an evaluation of a programme managed by UNICEF. The objective of the research

is to understand how UNICEF has helped people get better access to water and sanitary facilities, and

the results of their hygiene activities where you live. ISG will conduct its research via key informant

interviews, key stakeholder interviews, and focus group discussions. The interviews will include male

and female stakeholders to ensure differences in uses and needs are captured. This interview guide

serves as a manual for data collectors conducting the key informant interview portion of the research.

ISG’s researchers will conduct face to face interviews WASH stakeholders. While this guide is

intended to assist the interviewer in setting key areas for research and orienting the discussions, the

interview should happen as a conversation, allowing the interviewer to ask follow up questions that

may lead to discovering new and important information.

INSTRUCTIONS FOR THE INTERVIEWER

• Follow the informed consent process below.

• You should ensure that you cover all the major issues noted in this guide. However, some

questions may elicit long discussions and others short answers depending on the relevance of

individual topics for discussion to individual respondents. Exploring the most relevant areas

(and others that may be important to the respondent, but not necessarily covered in this guide)

is more important than covering every question in the guide.

• Ensure that women participate in the interview, especially if the interview is taking place in a

group.

• The WASH programme’s activities operate in a dynamic and complex environment. It is

likely that there are key issues that the author of this guide did not anticipate. The interviewer

should feel free to ask questions that the interviewer feels are fruitful whether or not they are

included in the guide.

• Please take notes in a separate document.

113

• Make sure to record your notes at the end of each day. Too much time between the interview

and recording notes can lead to loss of salient information that will have a negative impact on

the rest of the research.

Informed Consent Process This section outlines the consent process for the UNICEF Jordan WASH programme Evaluation.

While this document provides illustrative language, it is important to tailor both language and

presentation to each stakeholder to ensure understanding of the study and his/her involvement.

The people that the evaluation team will interview have the right to know that they are participating in

a study, that the evaluation team is collecting their opinions as part of a research project, and to be

told about the purpose of the research and the possible risks and benefits of the project and their

participation. The interviewer will introduce himself/herself and explain the purpose of the interview

as detailed in the Informed Consent Form below. Following the explanation, the researcher will ask

the respondent to sign the Informed Consent Form:

INTERVIEWER INTRODUCTION

Hello! Thank you for participating in this interview. My name is _________ and I am working on a

project for International Solutions Group, a research organization based in the United States. ISG is

conducting research on behalf of UNICEF.

The purpose of our research is to assess the effectiveness of UNICEF’s Water, Sanitation, and

Hygiene related activities where you live. You may not know about UNICEF’s specific activities.

However, UNICEF is interested in understanding if the program has achieved its intended goals,

which include improved access to water and sanitation and improved hygiene for people who live in

your area. I will ask you questions about your life and experience in relation to UNICEF’s water,

sanitation, and hygiene promotion activities. Although you may not directly benefit from participating

in this study, we hope that the findings of this study will contribute to the improving UNICEF’s

Water, Sanitation, and Hygiene programme’s for people like you.

The interview today will last about an hour. During the interview, I’ll take notes so I can remember

what you said.

I want to mention a few important points before we start.

• There are no ‘right’ or ‘wrong’ answers to any of the questions. We are interested in hearing

your opinions, whatever they are.

• Your participation in this interview is voluntary. If you don’t want to answer any question,

you don’t have to. The responses that you provide are confidential: we won’t mention your

name in any report we write and we will never connect your name with anything you say.

Your responses will be combined with responses from other people.

Do you have any questions before we begin?

Now that I’ve told you about the topics we’ll discuss and answered your questions, are you

comfortable proceeding with the interview?

1. IF NO: ASK IF YOU COULD PROVIDE ANY OTHER INFORMATION THAT WOULD

MAKE THE RESPONDENT MORE COMFORTABLE. IF RESPONDENT PREFERS NOT

TO CONTINUE, TERMINATE INTERVIEW.

2. IF YES: CONTINUE TO THE NEXT SECTION.

114

Household Level Questionnaire

Basic Information [NOTE TO INTERVIEWER: Fill out before the interview begins.]

1) Consent form number:

2) Date of Interview:

3) Gender (if apparent):

4) Primary and Secondary location of interview:

5) Interview Number:

Water Supply 6) Is water supply sufficient for your needs? [If no Goto 6a]

a) Which needs are unmet [for example: drinking, cleaning, latrine, other]

b) How much more water you would need?

7) Where do you most often access water [if outside the home, Goto 7a]

a) How often do you fetch water?

b) How far do you go?

c) How long does it take?

d) Are there alternative sources from where you fetch or buy water?

8) Do you buy additional water from other providers? [If yes, Goto 8a. If no, Goto 9]

c) Why do you buy additional water? Is it expensive?

d) Does the water have good quality?

9) Have you ever experienced problems accessing water? [if yes, Goto 9a]

a) What caused the problem?

b) How was the problem solved?

c) Has it reoccurred?

Water Quality 10) Is the quality of the water sufficient for all of your needs [for example: drinking, cleaning, latrine,

other]? [if no, Goto 10a]

a) For which activities is in insufficient (for example: drinking, cleaning, latrine, other)?

11) Have you ever become sick because of the water? [If yes, Goto 11a. If no, Goto 12]

a) What was the illness?

b) Was it just one person or many people in your family?

c) How often did it happen?

115

d) How was it treated?

Facilities 12) Does your family have access to a latrine and bathing facility?

a) Is the bathing facility and latrine private or communal?

b) Is it satisfactory?

c) Are there any improvements you would make to it (if communal, is distance reasonable, is it

safe, is it lighted sufficiently, does it lock)?

13) Is there enough privacy for women and children?

a) Are sanitation and bathing facilities segregated by gender?

b) Is there enough distance between female and male WASH facilities?

14) [if latrines are inside in the house Goto 14a; if outside of the house Goto 15]

a) Does the household have access to latrines/septic tanks or mainline sewerage? [if septic tanks,

go to 14b]

b) Is there any service provider that comes to empty the latrines/septic tanks? Is the service

timely and reliable?

e) Have you ever paid for the maintenance of your WASH facilities? If so, whom did you pay?

Is it the money worth the service?

15) Have you ever experienced problems accessing latrines or bathing facilities? [if yes, Goto 15a]

a) Have they ever been closed or not usable for short or long periods of time? If so, why? Was

the problem solved? How often did it happen?

16) If people generally have trouble with water or sanitary facility access, or the facilities don’t work,

what do they do?

17) Have you or any other member of your community experienced any threat while using the WASH

facilities? [if yes, Goto 17a]

a) Was it at day or night?

b) What do you think can be done to avoid similar situations?

c) Has security improved since the event?

18) Does the camp/community have latrines and bathing facilities accessible to people with limited

mobility? Where are they? (follow observation)

a) Are they sufficient? If not, what could be done differently?

19) Is there any area of this camp/community that has a harder time accessing WASH facilities?

a) If yes, which areas? Why?

20) Are there any vulnerable people that have a more difficult time accessing WASH facilities?

a) If no, which areas? Why?

Operations and Maintenance 21) Who cleans the WASH facilities that you use? Have you ever cleaned them? Is there a system in

place to maintain the facilities clean?

22) Is there any committee in charge for the maintenance of the water or sanitation facilities or

infrastructure?

23) Is there anyone in your community or immediate circle able to fix minor issues with the WASH

facilities? [if yes, Goto 23a]

a) If so, is this person paid?

b) Does he/she do a good job?

c) Is he/she reliable?

24) Do you know if this person was trained by UNICEF or by other organizations? Which

organization?

Final Questions 25) Were you consulted about your preferences regarding water and sanitation services? If so, how?

a) Were the different options clearly explained to you?

116

b) Did you have the opportunity to provide feedback on the services provided? If so, how? Did

the feedback you provided have any outcome? (Were the services

adjusted/corrected/improved according to your feedback?)

26) Have you noticed major changes related to access to water in your camp since you have been

here? [If yes, Goto 26a]

a) If so, what kind of changes?

b) How did they happen?

c) Who made the changes?

d) Did the changes improve your situation?

27) Is there anything else that you would like us to know about Water, Sanitation, or Hygiene where

you live?

Focus Group Discussion Guide

Introduction for the Focus Group Leader ISG is conducting research on behalf of UNICEF. The objective of the research is to understand the

impact of the UNICEF’s Water, Sanitation, and Hygiene programme. Particularly, we want to know

how the program has worked to meet you expectations, where it has succeeded, and where it could

improve. ISG’s research will be conducted via a survey, focus group discussions, and key informant,

key stakeholder interviews. This focus group guide serves as a manual for the focus group leader and

note taker conducting focus groups.

ISG will conduct focus groups with residence of camps, ITSs, host communities and stakeholders in

schools:

1. The degree to which project activities served the needs of beneficiaries.

2. The attitude of community members toward ownership and maintenance of infrastructure and

asset building projects

3. The effectiveness of program in achieving water, sanitation, and hygiene results for all

stakeholders, including vulnerable groups.

4. The sustainability of the programme’s activities and facilities.

The purpose of these focus groups is to present broad issues and to let participants generate ideas and

build on each other’s ideas. The focus group leader’s goal is to encourage participation and ask

questions that build on participants thoughts. This guide provides follow up questions that the leader

may use. The leader may ask specific participants to compare their opinion or thoughts to those of

other participants through prods such as “Participant B, how does your experience compare to the

experience of participant A that we just heard?” or “Participant C, what would you add to participant

B’s comments?” The interaction that the focus group leader encourages will provide depth to the

information we are gathering through the survey and interviews.

Preparation

• Make sure that informed consent forms are available for each participatng.

• Make sure the room is set up with flip charts or whatever materials you need.

117

• Set up refreshments if appropriate.

• If available, set up small gifts to thank the focus group participants.

Agenda

Introduction, Ground Rules, Logistics and Sign-in Sheet

[FOCUS GROUP LEADER: Introduce yourself and the note taker.]

Thank you all for participating in this focus group. My name is _________ and I will lead the group,

and this is ____________ who will take notes. Our discussion will last no more than an hour and a

half. Before we get started, let me mention that there are drinks and refreshments available (point

them out), and the bathroom is (indicate location of bathroom). Please feel free to get up and move

around or help yourself to refreshments during the discussion.

The purpose of this focus group is to understand the impact of the UNICEF Water, Sanitation, and

Hygiene programme in your community (or school). Particularly, we’d like to understand:

• How project has served your community’s needs.

• Your opinion of the project’s facilities and services.

• What you like best about the project.

• What you would most like to improve about the programme

Infront of you is an informed consent form. This form let’s you know about the purpose of the

research and the types of questions we will ask. Please read the form and let me know if you have any

questions. If you agree to participate in this focus group, please sign the form at the bottom where

indicated.

I wanted to mention a few important points before we start.

• The purpose is for us to learn from you, both your positive and negative experiences and

opinions.

• We’re gathering information, not trying to achieve consensus. Please contribute your

experience and opinion. It’s ok for there to be disagreement or different ideas.

We have a few ground rules that we hope you’ll agree with:

• Please participate. Your contribution is valuable and will improve the program’s activities in

the future.

• Please agree to keep the information provided in the focus group confidential. We won’t

discuss any sensitive topics. Still, people may share ideas that they’d rather not be discussed

privately.

• Stay with the group and please don’t have side conversations.

• Turn off cell phones if possible.

Are there any questions before we get started?

FOCUS GROUP LEADER: Go around the table and ask people to introduce themselves.

118

Relevance 1. Do you think the interventions responded to your basic needs related to water, sanitation and

personal hygiene?

a. What’s the most important way your community has improved in terms of water

access, sanitation, or hygiene practices?

b. What needs does the community have that aren’t being addressed? Why do you think

they haven’t been addressed?

Effectiveness 2. If you could improve your access to water or sanitation facilities, what would you improve?

3. Is there any group in your community that have not been reached or that has not benefitted

from UNICEF like the others?

4. What do people do if they have trouble accessing water, or the sanitary facilities don’t work?

a. What authorities are responsible? Are they responsive?

WATER

5. How has access to water improved for this community recently? What is the most important

improvement?

6. What changes would you make to the community’s ability to access water?

SANITATION

7. Do you use toilets or latrines?

a. Are they public or private?

b. How are they kept clean?

c. Do they work all the time?

d. Are they safe?

8. What changes would you make to sanitary facilities?

HYGIENE

9. Has the hygiene knowledge or practices changed in the community in the last five years?

How?

a. Do people ever get

SUSTAINABILITY

10. What would happen if UNICEF would stop its activities tomorrow? Do you think your

community/school would be able to maintain the water and sanitation infrastructures as they

are?

11. Is the wastewater and solid waste handled properly? Do you have any concern for the

environment?

119

Annex F: Team Description

Mr. Jason Wares; Team Leader

Role: Mr. Wares led the team l, organized the completion of deliverables, conducted data

collection activities, and arranged team logistics. He also interfaced with the UNICEF at all

stages of the assignment.

Mr. Luca Palazotto, Mr. David Robbins, and Mr. Niall Roche; Co-Evaluators

Role: The Co-Evaluators will assist the Team Leader during the tools development,

sampling, desk review, data collection, and reporting aspects. Dr. Bassam will lead in the

translation of tools and reporting, as he will serve as the Country Expert for this assignment.

interdisciplinary research consultants (id:rc)

Role: id:rc organized logistics for the evaluation in Jordan and provided local WASH and

monitoring and evaluation consultants. Id:rc participated in data collection and analysis, and

ensured that the evaluation was grounded in the Jordanian context.

120

Annex G: Evaluation Matrix

EQ1: To what extent were affected people consulted on their preferences with regards to water and sanitation service

options and level of services? Were interventions appropriate in terms of meeting their basic needs? Was there a

feedback loop and monitoring system in place for reflecting the learning and evidence to improve programming,

especially for the humanitarian response?

Judgement Criteria Indicators Sources of Documentary

Information

Methods and tools for data

collection

JC 1.1: The needs of the

population, in particular

those of vulnerable groups,

were well considered during

the programming process

and adjusted during

programme implementation.

- Evidence of an process to

identify needs prior to the

design and implementation

of interventions.

- The extent to which chosen

interventions were consisted

with the needs that

assessments identified.

- The choice of interventions

prioritized activities that

would serve vulnerable

groups.

- The existence of a feedback

loop and monitoring system

that evolved with the

programme.

- Starting at WASH Working

Group paper

- WASH Cluster Coordination

Handbook

- Jordan Regional Response

Plan 5

- 2015 KAP survey for

Za’atari

- WASH Sector Coordination;

Minutes of weekly Za’atari

camp WASH sector

coordination group. Sunday

24 November 2013

- 2017 Azraq KAP Survey

Report

- 2014 WASH Sector Gender

Analysis Document

- Specific data analysis

- World Vision Staff Interview

30 July 2018

- UNICEF Staff interview – 26

July 2018

- Focus Gropu Discussion

number 1 through 4– Azraq

- Official Interview – MoWI

- Mercy Corps Staff Interview

– 4 September 2018

- On site assessments and

interviews in Za’atari, Azraq,

KAP, and Rukban.

- Interview with Gama

Engineering

- Host community Interviews

in Irbid and Ruwashaed

121

- Evidence that UNICEF used

evidence for learning.

- REACH 2014 ITS

Assessment

- World Vision PCS 19-13

- ACF 2017 Azraq KAP

Survey Report

- Rukban emergency plan

- UNOPS Staff Interview

- UNICEF Jordan Country

Programme Document

- UNICEF WASH Actions in

Humanitarian Situations:

Synthesis of Evaluations,

2010 - 2016

In the intial phase of the Programme, UNICEF determined refugee’s WASH needs through talks with partners such as The WAJ,

Mercy Corps, Acted, and Relief International, and based on the assessments and information that they’d gathered. The tactics that

they employed took into consideration the Convention of the Rights of the Child (CRC) and the Convention on the Elimination of

All Forms of Discrimination against Women (CEDAW). Through these means, UNICEF created estimates of the best way to

provide WASH services for the newly arriving refugees. UNICEF also consulted humanitarianresponse.info for guidance on issues

such as accountability and feedback mechanisms.

The evaluation team did not find evidence that UNICEF consulted internal resources for guidance on WASH response in an

emergency situation.

Another consequence of the need for quick decision making was that, in the initial emergency response phase, the camp’s effect on

Jordanians living near the camp and in other host communities with high concentrations of refugees was not taken into

consideration adequately.

In later phases, UNICEF used assessments and consultations to understand the needs of beneficiaries inside the camps. Azraq

residents benefited from lessons learned in the establishment of WASH services in Za’atari and contributed their preferences to

WASH set up.

After the emergency response phase, UNICEF worked with its partners to adapt WASH facilities and services to the observed needs

and behavior of camp inhabitants.

122

Time pressure and the desperation of the Settlements’ inhabitants made consulting them about their preferences impractical.

UNICEF provided water and sanitation services to respond to the settlements’ exigencies. They moved quickly and effectively,

with clear goals, such as ensuring that each person had at least 15 liters of water per day. The WASH programme also recognized

that its intervention would have an impact on the larger Ruwaished host community, so they undertook to rehabilitate sanitation

facilities there.

The WASH programme has maintained weekly updates since the initiation of work. The updates monitor operations and needs. In

2017, surveys and assessments became part of the monitoring mechanism. These tools ensure that WASH programme activities

remain relevant and effective.

UNICEF’s improvement of the wells and infrastructure in host communities is significantly important. However, the degree to

which it addresses the needs generated by the refugee crisis and the needs of the most vulnerable is uncertain.

UNICEF reports that they have served 114 ITSs, but the total number of ITSs is unknown, and the level of that service is unclear.

The evaluation team could not find evidence that The WASH programme is structured as a learning organization. It does not appear

to have taken guidance from UNICEF’s experience in WASH programming at the outset of the programme. While it has certainly

made marginal improvements to its activities in terms of efficiency and responsiveness to people’s needs, it did not build

monitoring networks or feedback loops that fed into a system that would allow for learning or informing strategic management

EQ2: To what extent did the interventions consider the needs and the priorities of the MWI and were they coherent with

global references and the regional and national response to the Syria Crisis?

123


Information


collection

JC 2.1: Degree to which

UNICEF took government

policy and strategy into

consideration when

determining programme

priorities.

JC2.2: Degree to which the

MWI and WAJ were

satisfied that UNICEF

responded to GoJ needs

- Evidence that MWI priorities

were considered in

intervention design and

implementation.

- Government officials feel

that UNICEF’s interventions

were in line with policy and

regulations.

- National Water Strategy

2016 – 2025

- Jordan Response Plans

- Senior Staff Interview –

WAJ 30 July 2018

- Senior Staff Interview –

MoWI 30 July 2018

- Specific focus groups

UNICEF’s work in WASH is in line with the priorities of the National Water Strategy, for which The key reference document is the

National Water Strategy 2016 – 2025. UNICEF is focused on the sustainable operation of boreholes to supply water and create

resilience. UNICEF also recognizes the government’s concern that Jordan’s northern governorates have not benefited from the

government’s efforts to increase water supply (Ministry of Water and Irrigation, 2016). In response, UNICEF has focused much of

its host community interventions on improving WASH in the northern governorates.

Both the MoWI and WAJ reported in interviews that the GoJ-UNICEF partnership is successful. The WAJ praised UNICEF’s

success at sector coordination despite budget cuts140. The MoWI appreciated UNICEF’s role in coordinating the sector141.

140 WAJ Senior Staff Interview – 30 July 2018


124

Interviews at MoWI and WAJ demonstrate the Government of Jordan’s struggles with its strategy and UNICEF’s response. Senior

staff at the MoWI praised UNICEF’s responsiveness inside the camps, but sees that project’s host communities are more dependent

on available funding. MoWI said that when the WASH programme first took on host community projects, there was a

miscommunication regarding the meaning of the protocol UNICEF established with the MoWI. The protocol included a mandate

and budget for UNICEF to improve infrastructure in governorates. The ministry didn’t realize that the budget was prospective, and

moved those items off of its budget for the year. When fundraising fell short, those governorates were underserved.

EQ3: How responsive was the UNICEF WASH programme over time to changes in the external environment?


Information


collection

JC 3.1: UNICEF

demonstrated flexibility in

- Evidence of adjusting

programme design and

- Jordan Annual Reports - Senior Staff Interview – 6

August 2018

125

decision making and the

ability to reorient resources

as the external environment

changes.

operations to changing

conditions

-

- Terms of Reference for Service

Contracting - Rukban.

- Focus group discussions in

Azraq, Za’atari, and KAP

In the initial phase of the programme, the GoJ and INGOs in Jordan prepared for a temporary camp. This preparation included

procuring tents, portable sanitary facilities, and trucked in water supplies. While these interventions were effective, their expense

made them unsustainable. When it became apparent that the need to house refugees was more than temporary, UNICEF devised

solutions that required investment, but would be cheaper and more sustainable to operate in the medium to long-term. These

interventions included WASH Blocks, piped in water from nearby boreholes, and established hygiene campaigns. UNICEF also

made these adjustments in the settlements, establishing water infrastructure that was both resilient enough to withstand difficult

conditions and use, but also reduced operating funding needed for maintenance142. As Senior UNICEF Staff noted, “Transitioning

from humanitarian response to systems and infrastructure…is incredibly challenging, particularly in environments like the berm

(Rukban).143”

One type of transition in which the WASH programme struggles was in implementing host community projects. As an interview

respondent at the MoWI suggested, UNICEF did not originally plan on implementing host community projects, but turned to them

in the face of external pressure from municipalities that felt that refugees were putting stress on local water resources144. The

WASH programme boldly took up the challenge and set high goals for itself such as increasing the percentage of the population in

Jordan that has access to adequate sanitation facilities (2014), or increasing the number of inhabitants with safe water and storage

facilities by 1.4 million (2015)145. While UNICEF missed these targets in each year, it continued to set high targets in this

component. As mentioned in the Funding Environment section of this report, UNICEF blamed shortfalls in fundraising for missing

host community targets in each year of the programme. However, the evaluation team did not find evidence that the WASH

programme considered its funding outlook when planning these targets, or, given the regular shortfall in funding, how it might

better spend these funds. We discuss this topic further in the Effectiveness and Efficiency sections.

142 UNICEF Staff Interviews – 10 October 2017

143 UNICEF Senior Staff Interview – 6 August 2018


145 See results framework, Annex H.

126

EQ4: To what extent did UNICEF’s response achieve its intended outcomes (as stated in UNICEF’s Country

Programme Outcome and Outputs as given in Annex A)? To what extent did UNICEF achieve equity results, especially

for children and women specific interventions?


Information


collection

JC 4.1: The degree to which

UNICEF set specific,

measurable and realistic

targets.

JC4.2: The degree to which

the evaluation team could

find evidence to support

UNICEF’s achievement

measurements.

JC4.3 The degree to which

targets were met.

- Indicators based on results

frameworks.

- RAM reports 2013- 2016

- UNICEF Jordan Annual

Reports

- Za’atari Hygiene Working

Group KAP reports

- The Sphere handbook

- Interviews with partner

organizations

- Interviews with UNICEF

staff

- Reviews of PCAs

Component 1

Under this component, the WASH programme achieved high levels of effectiveness for the targeted population across all years.

127

Component 2

As detailed with citations below, the WASH programme consistently struggled to achieve targets under this component. Two issues

created difficulties: 1) funding, 2) lack of strategic planning

As described in the Funding Context section of this report, UNICEF Jordan struggled with funding issues. Maintaining services in

the camps necessarily received first priority for funds at the expense of achieving targets in host communities.

That said, after several years of missing targets, UNICEF doesn’t appear to adjusted its strategy to make it more effective or

realistic. Also, indicators aren’t specific or timebound under this component, so it’s difficult to know what the programme is trying

to achieve. For example, UNICEF’s Outcome 4, Indicator 2 target is that 1.5 million host community residents use improved

drinking water as per Jordanian standards. In its 2015 RAM report, the WASH programme stated that “In Host Communities, 41%

(634,858) of the target population of 1,550,500 benefitted from improved access to water facilities and systems as a result of

rehabilitation of water infrastructure, as well as support to vulnerable households.” In its 2016 RAM report, for the same indicator

the WASH programme reported, “In Host Communities in four governorates (Mafraq, Madaba, Balqa and Irbid), 237,360

additional people were reached in 2016. Since 2013, an estimated 1,482,402 people have improved access to water services

…which equates to 96% of the Country programme target.” It is unclear if the target is that UNICEF serve 1.5 million per year

under this outcome as the 2015 report implies, or 1.5 million cumulative people as the 2016 report implies. It is also important to

consider the improvement. For xample, In Irbid UNICEF made improvements to the water distribution system. This should not

imply that everyone in Irbid had better water services as a result of the intervention. Water services are still very limited, and there

are still serious problems with the systems distribution network, water safety, and availability.

Component 3

Government officials at the MoWI and WAJ reported satisfaction with the WASH programme contribution to planning documents,

strategies, and policies146. The MoWI said that, as head of the WASH Taskforce, UNICEF always has a good plan, and has a good

partnership with UNICEF on technical committees. The Ministry also praised the work done by the consultant that UNICEF

seconded to Ministry, and wished he could have stayed longer.

146 Source: Interviews at MoWI and WAJ

128

The ministry staff interviewed by the evaluation team mentioned that there were some unintended consequences to the work that the

WASH programme has done in host communities. A specific example was the downstream effects of increasing water supply

through a network. Increased water supply means quicker degradation on infrastructure, leading to the Ministry having to replace

equipment sooner than expected. One interview respondent mentioned the opportunity cost of providing water to refugees, in that

the water could have gone to Jordanian workers or industry. He estimated that for each refugee the government spends $650 on

water and $7,700 on indirect costs included lost opportunities.

Component 4

Stakeholders and UNICEF partners felt that UNICEF was effective as WASH sector lead. One contractor that serves Rukban said

“UNICEF staff are super cooperative and available anytime of the day or night”.147

The evaluation team also spoke with two of the WASH programme’s donors, The Bureau of Population, Refugees, and Migration

(BPRM) at the United States State Department, and the German owned development bank Kreditanstalt für Wiederaufbau (KfW).

BPRM commented that UNICEF was effective in leading the sector, and excellent to work with as a donor. BPRM felt that they had

benefited from UNICEF’s guidance in setting their own strategy for addressing the crisis. KfW appreciated the guidance UNICEF

could provide in constructing infrastructure in the camps and assistance in host communities.

As sector lead, UNICEF has led WASH activity coordination in the Camps, including responses to the occasional emergency, such

as the 2015 outbreak of Hepatitis A in Azraq. As part of camp coordination, UNICEF leads a bi-weekly Za’atari coordination

meeting that has been effective in setting standards and avoiding duplication of effort. UNICEF also improved the MoWI’s use of

data to create water management strategies, and led the sector’s WASH in schools initiatives.

One area in which UNICEF could improve its coordination is in strategic planning and communication with partner organizations.

Several partner organizations, particularly INGOs, mentioned that they were surprised by UNICEF’s decisions in the camps, often

147 Source: Interview with Gama Engineering

129

requiring that they make major adjustments on short notice, or conduct work for which they hadn’t planned or budgeted148.

Examples included sudden decisions to hand operations over to other organizations or contractors with short notice, without

guidance on the handover process. Other partners mentioned contracting processes that took a long time only to have a requirement

that initiatives start days after a contract is signed. Decisions such as these, that do not appear as part of a strategically designed

plan, may force risk and costs on to partner organizations. It may also create risks for UNICEF that its monitoring and accounting

systems cannot bear. These risks could also be mitigated by an annual strategic planning process that would set expectations and

guidelines based on clear analysis and reduce last minute decision making149.

EQ5: What were the strengths and weaknesses of the interventions in achieving results in the camps/settlements, ITSs

and host communities on the following:

1. Water (Litre/day, supply, quality, consistency, reliability, accessibility, and equity)?

2. Sanitation (adequacy, appropriateness, accessibility)?

3. Hygiene promotion (appropriateness of the messages, timeliness, behaviour change)?

4. WASH in schools?

5. National level support?


Information


collection

148 Interviews with Mercy Corps, World Vision, ACF, and Oxfam Staff

149 The UNICEF Jordan Country Programme Document 2013-2017 mentions a two year rolling workplan which is measured through annual component reviews and a mid-

term review in 2015. The evaluation team did not see evidence that the WaSH Programme had complied with this mandate.

130


UNICEF met its own and

international standards.

JC 5.2: Beneficiary opinions

on stregths and weaknesses

of interventions.

- International WASH

standards

- WASH in Schools targets vs

accomplishment

- Jordan response plan

- Global WASH Cluster

handbook

- Sphere handbook.

- UNICEF Reports

- Camp Assessments

- Hygiene Promotion Working

Group reports.

- Interviews with Partner Staff


staff

- Site inspections of UNICEF

intervenions

Water

For both Za’atari and Azraq, the interventions were effective. Residents of both camps people at least 35 liters of water per day,

well in excess of the 15 liters per day that is the Sphere standard

Sanitation

The system in Za’atari, described below, is an innovative approach that works well and is cheaper to install than a traditional sewer

system. The same is true for the boreholes and water systems at both Zaatari and Azraq, where high quality, well designed systems

are being run by local operators from the camps, which contributes to sustainability.

In Azraq, The WASH programme has implemented gradual improvements of basic service. In 2014, shelters in Azraq were not

connected to a greywater network, meaning greywater and mud tends to accumulate in the ditches surrounding the shelters.

UNICEF with partners constructed a plot-level grey water network, and provided income generating opportunities within the camp

contributing to camp cleaning and sustainable waste management practices.

Communal toilets remain unpopular in Azraq.

The absence of assessment data with respect to the early stages of the response makes it difficult to determine what the key public

health issues were in relation to hygiene, and the behaviours that needed to be targeted. As noted earlier in the report, WASH

131

programme partner ACF noted a high occurrence of waterborne diseases in Za’atari in 2012/2013. A general observation from the

evaluation team is that the no official documents referenced these key infectious diseases. The exception was the Hepatitis A

outbreak that afflicted Azraq. The first reported measurement on the issue of Hygiene was the Baseline KAP survey in Za’atari in

November 2012, four months after the camp opened. A notable feature for Za’atari and Azraq was the strong commitment to

repeating KAP surveys in both camps on an annual basis enabling one to measure change over time.

The evaluation team did not have access to hygiene data with respect to Host Communities, those in the ITSs, or the settlements

(Hadalat and Rukban). However, the evaluation team understands that given the circumstances it was not feasible to gather hygiene

data in those areas.

Over time the KAP survey results for Za’atari showed an improvement.

In the area of hygiene items, interview respondents indicated that they did receive hygiene items, but not enough of certain items.

The intended results at the settlements were to provide water to the Rukban and Hadalat Refugee Camps as effectively as possible

while achieving value for money. The evaluation team inspected the equipment and operations of the water system at Rukban. The

borehole and related equipment are of high quality, the design is appropriate, the construction is of high quality, and the operations

and maintenance activities are performed properly and on schedule. Staff working for Gama, the UNICEF hired to supply and

operate the system at Rukbank reported in an interview that UNICEF always purchases high quality equipment.

At Rukban, the treatment system is a sophisticated and costly, but warranted and necessary due to the concentration of certain

constituents of the raw water.

Hygiene promotion in the settlements seems to be limited to the distribution of hygiene kits on a periodic basis.

132

EQ6: To what extent did the programme use available resources in an economic manner to provide WASH services and

facilities in camps/settlements and host communities, to ensure child rights, equity and gender equality?

EQ7: To what extent were the costs to deliver water and wastewater services, as well as solid waste management,

rationalized and optimized in the camps and settlements to ensure Value for Money? How do these compare to similar

situations at the local or regional level? Could more cost effective operations/interventions have been undertaken at an

earlier stage?


Information


collection


the programme could

demonstrate it had used

resources in an economic

manner.

JC 6.2: Documentary

evidence that costs were

rationalized and optimized.

- Demonstrated declining

operations costs.

- Demonstration of value for

money.

- Existence of an adequate

system.

- Review of PCA agreements.

- Review of cost reports

-

- Compilation of budget data.

- Complication of cost data.

- Interviews with financial

management staff.


staff

There is no indication that the cost of providing water services in the early stages of the intervention was rationalized.

UNICEF took steps to bring down the unit cost of providing water to refugees in the camps. As figure 10 below shoes, the

evaluation team estimates that between 2013 and 2016, the cost of delivering water to refugees in the camps dropped from 3.34

Jod/m3 to 0.24 Jod/m3.

133

There is some indication that trucking water into Za’atari camp went on longer than it should have, draining the programme’s

resources.

Initially wastewater management was inefficient. Issues such as contractors claiming to remove up to 40 percent more sludge than

they actually removed burdened the programme. Drivers claimed trucks were capable of holding more volume than they actually

did and sometimes carrying waste in to the camp from outside, which was subsequently charged to the UNICEF operation

The wastewater system in Za’atari evolved from unimproved pit latrines to a system of shared septic tanks, a wastewater collection

network, and eventually a wastewater treatment and reuse system.

In Azraq, the wastewater treatment plant is not functioning and has been abandoned.

An innovation that UNICEF developed was the “Waste Taxi” also known for a time as Uber for Waste. It was a predictive model

designed to address with the various different volumes of wastewater tanks that were introduced when people began constructing

their own private washing and toilet facilities in Za’atari.

EQ8: To what extent did UNICEF try to minimize duplication/gap in WASH interventions in camps? And how early did

UNICEF move to more cost-efficient interventions?


Information


collection

134

JC 8.1: The existence of a

management plan to

minimize duplication and

gaps.


the plan was effectively

implemented

- Evidence of full coverage in

beneficiary locations

- Satisfaction with coverage

from beneficiary populations.

- 3

- REACH Assessments

- UNICEF camp management

documents.

- Working group meeting

minutes

- Focus groups among

beneficiary populations

- Interviews with UNICEF and

Partner staff.

-

UNICEF minimized duplication or service gaps in camps by dividing the camps into sectors and assigning different partner

organizations responsibility for WASH services in each sector. UNICEF’s role as sector lead allowed it to unify service standards

among the different partners.

In Azraq, a similar coordination structure was followed as Za’atari. ACTED, World Vision, and THW provided WASH services

with Relief International providing hygiene promotion and WASH in schools. However, coordination among the three entities

appears to have been difficult. There is far less documentation on coordinated efforts in Azraq.

in Zaatari, the WASH programme has reduced costs and realized efficiency by reducing the numbers of partners that they are

managing.

135

EQ9: To what extent were resources (financial and human resources) allocated for the range of intervention types, with

particular attention paid to the respective allocations for humanitarian response and resilience, to achieve the intended

results?


Information


collection


UNICEF could demonstrate

allocation of resources

- Financial information

showing distribution of

resources

- Partner organizations PCAs

and budgets.

-


management staff.

UNICEF Staff estimated that approximately 5 percent of UNICEF’s budget goes to host community interventions. The evaluation

team’s estimate, which as mentioned is based on incomplete information150, is that from January 2013 - -July 2017, about 63

percent of project funds went to camp operations, about 15 percent is spent on host communities, and another 15 percent is spent in

settlements, Rukban and Hadalat (figure 13).

EQ10: To what extent did UNICEF advocate with the Ministries to mobilise their available resources to address the

identified needs? And how efficient was UNICEF’s support to ministries (i.e. MWI and MOE) to reaching most

vulnerable people? Did the WASH programme maximise the potential collaboration with other UNICEF programme

sections and partners?

150 The financial numbers used in this report were derived from UNICEF’s estimates of expenditure on contracts and PCA agreements provided by UNICEF, which were

often in a format that was difficult to read. PCA numbers were budgeted amounts, not actual expenditure. These figures also do not include any indirect costs that may

not have been allocated proportionally by beneficiary group.

136


Information


collection

JC 10.1: Existence of

documentation

demonstrating advocacy

efforts.


GoJ officials agreed that

UNICEF’s support assisted

them in meeting most

vulnerable people.

JC 10.3 Degree to which

maximum potential

collaboration was

demonstrated

- Satisfaction levels of

government officials

- Existence of reports detailing

support initiatives.

- Partners report positive and

maximum collaboration

- Programmatic reports.

- GoJ strategy documents

- Interviews with partners


staff

- Interviews with government

officials.

Over the time frame that this evaluation covers, UNICEF’s primary engagement with the MWI was through a consultant UNICEF

hired to assist the government in developing strategies and response plans. UNICEF also interacted with the government to support

host communities.

The evaluation team saw no documentation of WASH programme advocacy with the MoE during the period described by the scope

of the evaluation. UNICEF Jordan has education programmes that may have had primary engagement with that government

ministry. Still, a UNICEF partner reported that there is more demand for education services than the MoE can handle. UNICEF

could work more to support the capacity of the MoE and should consider building those activities into its future strategy in this area.

137

EQ11: Was the infrastructure in the camps constructed in such a manner as to ensure long term functionality, and

will continue to service the most vulnerable people even when UNICEF is no longer directly supporting? This should

be investigated in terms of Water and Wastewater infrastructure.


Information


collection

JC 11.1: Demonstration that

infrastructure in camps was

constructed with long term

functionality as a priority.


UNICEF balanced quality

and price in construction

decisions.

- Evidence of quality material

use.

- Evidence that long term

sustainability was taken into

account.

- Review of Partner PCAs

- Review of Contractor

documentation

- Onsite inspections

- Review of construction plans

and reports.

- Interviews with engineering

staff


contractors

Following the emergency response phase of the programme, The infrastructure in the camps was constructed to ensure long-term,

sustained functionality. The GoJ prohibited the construction of permanent structures at the beginning of the crisis, forcing UNICEF

to make due with mobile sanitation and trucked in water. When it became apparent that the refugee crisis would not abate quickly,

the government recognized the need for longer-term solutions and UNICEF provided lasting, sustainable solutions.

UNICEF chose highly mechanized treatment systems for both Za’atari (trickling filter and MBR) and Azraq (the MBBR American

system, which eventually failed). These systems are expensive, but the decision to use these systems was sound and valid. The

national laws of Jordan specify stringent effluent quality from municipal wastewater sources. Even though camps are surrounded

by vast unused parcels of government land, wastewater systems must comply with environmental standards. Also, the 4-month

rainy season that makes it more difficult for passive wastewater treatment systems to produce quality effluent year round.

For Za’atari, the common septic tank and small diameter sewer is an appropriate and innovative wastewater collection model.

138

For Azraq, the installation of the MBBR system, translocated from Afghanistan and provided by US donors, turned out to be a poor

choice, although the rationale for selecting it was valid. Unfortunately, the system did not meet the needs of Azraq’s inhabitants or

the program in general. Wastewater strength was higher than anticipated. Following an expensive attempt to make it work, the

system was eventually abandoned. Wastewater is now pumped and transported at great expense. There is a greywater management

scheme that has yet to come on line fully, and until there is some centralized wastewater treatment option, septage will require

hauling some 90 KM away. It is surprising that the Azraq camp would have more difficulty in establishing a wastewater system as it

is neatly laid out, compared to Za’atari, which has developed an elegant and lower cost solution, which has evolved into something

that could become sustainable over the long term as economic development in the camp increases.

The water systems in the camps are designed, installed, operated and maintained to provide long term functionality, even if

UNICEF were to leave, which is unlikely. The quality of the equipment is high quality, which adds to the longevity of the

systems.

The upgrades to the boreholes and pump station in the host community of Irbid are well designed and constructed and should

continue to operate give the required O&M for many years. While the UNICEF interventions are sustainable, the overall water

situation in Irbid however, is not. The extreme water scarcity, depleting aquifer, population growth, and high NRW rates are all

causes for concern. They suggest areas for next steps should donor funding be available.

EQ12: To what extent did UNICEF take sustainability into account at the early stage for water, wastewater and solid

waste management services (e.g. transition to local government institutions and local capacity)?


Information


collection

JC 12.1: Evidence of

UNICEF’s Jordan

- Strategic plans for

sustainaibility.

- Workplanning

documentation.


staff.

139

promotion of Sustainability

Initiatives.

- - Interviews with GoJ officials.

The evaluation team has no evidence that UNICEF has a transition plan to government or any other institution in place. The GoJ at

all levels has made it clear that they will not take ownership or spend public funds on the crisis. The government understands that it

is providing an international public service in using internal resources such as water to sustain the refugees and expects the

international community to provide for the rest of their needs.

UNICEF’s approach to sustainability has three aspects.

• Build high quality infrastructure: Building systems and infrastructure that have long term potential to serve Jordanians

and the national government.

• Leverage other, non-governmental resources to contribute to ongoing operations of the camps: This approach includes

building the capacity of refugees to manage their own WASH issues and partnering with national entities to engage in

management.

• Become as efficient as possible: UNICEF continues to find ways to bring down costs so that resources are used as

efficiently as possible, and available as long as refugees need WASH services.

EQ13: To what extent did UNICEF’s interventions increase the resilience of the Government and respective target

populations? To what extent will UNICEF-supported projects (schools, ITSs, water and wastewater infrastructure,

mobilisation) continue to be operational after the end of UNICEF’s financial support


Information


collection

140

JC 13.1: Evidence of planned

resilience in UNICEF

Interventions

- - Jordan Response Plan

- Partner descriptions of

projects in PCAs

- Interviews with beneficiaries

- Interviews with partner staff

- Focus group discussions

This evaluation question is difficult to answer because it appears that target populations will require UNICEF’s financial support

well into the future. UNICEF increased the resilience of supported populations and the GoJ in the following ways.

• Designed projects in accordance with the JRP, linking refugee assistance in Za’atari and Azraq with resilience oriented

projects that rehabilitate or extend systems in urban areas with large refugee populations.

• Instituted consistent standards and policies around water access in camps to promote equitable access and reduce conflict.

• Ensured that infrastructure was built using high quality materials.

• Built capacity of stakeholders in camps to manage water, sanitation and hygiene within their communities.

• Created mechanisms for hygiene and water conservation promotion that operate under the leadership of local stakeholders,

such as WASH committees and the Lead Mother model in Azraq.

• Improved water and sanitation infrastructure and services in some host communities with high refugee populations to reduce

tensions.

• Worked with the government to create Wash in School Standards to set a bar for water, hygiene and sanitation in Jordan’s

often overburdened schools.

• Transferred a process for creating response and water strategies to the GoJ that they can replicate in the future.

• Promoted the use of data for strategy development at the MoWI.

EQ14: Does the MWI have the necessary mechanisms in place and capacity to ensure the water and sanitation needs of

the most vulnerable people will be met as a result of UNICEF’s support and advocacy, if another large scale

humanitarian crisis occurs in the future?

141


Information


collection


mechanisms in place at the

Ministry level.

-

The evaluation team did not find evidence that MWI has the necessary mechanisms in place for a future, similar crisis. One reason

for this is that the government does not want to own these crises or been seen as a refuge for at-risk populations.

JC 14.2:

EQ15: To what extent did UNICEF document its WASH interventions and make available to the sector learning on

water, wastewater, solid waste management and mobilisation?


Information


collection

JC 15.1: Evidence of the

documentation of

interventions and their use

for learning.

- Documentary evidence of

lessons learned

- Evidence of propogation of

materials.

- Reports from annual reviews

as per the Jordan Programme

Document.

- - Interviews with UNICEF

staff.

- Interviews with partner

organizations.

- Interviews with sector

coordination group

participants.

142

Documentation of interventions for the purpose of sector learning was limited. Lessons learned were shared at coordination

meetings. The evaluation team did not find evidence that the WASH programme documented its experience for the purpose of

sector or institutional learning. This evaluation is the first attempt to document UNICEF’s experience implementing the WASH

programme in Jordan.

EQ16: To what extent were environmental protection and sustainability taken into account in the design and

implementation of programmatic interventions?


Information


collection

JC 16.1: Evidence that

environmental protection

assessments were carried

out.


environmental protection

- Completed environmental

assessments.

- Ongoing environmental

protection efforts.

- Programme Reports

- PCA Environmental

protection descriptions

- Interviews with partner staff.

- Interviews with donors.

- Onsite reviews

143

was built into programme

designs.

Early in the programme, environmental protection was not taken into account sufficiently. Lack of focus on environmental issues

led to Za’atari’s construction over an aquifer that accounts for 34 percent of Jordan’s water supply without proper facilities for

managing wastewater and preventing it from leaching into the ground. As described in earlier sections of this report, the problem

was exacerbated when open pits filled with graywater developed as camp residents began to create private sanitation facilities in

their housing units. UNICEF corrected this issue in Za’atari by installing wastewater collection tanks, reaching 2,000 m3 of

collected wastewater per day, utilizing backwash from borehole water filters for construction and cleaning purposes, and eventually

constructing a water treatment plant.

UNICEF applied the lesson it learned in choosing a site location and water system for Azraq. Prior to the construction of Azraq,

UNICEF worked with the MoWI to conduct an environmental impact assessment. The assessment identified a high risk of

underground water contamination. UNICEF incorporated protections in its design of Azraq’s WASH services. It commissioned

construction and operation of containerized wastewater units. It also attempted to install a treatment plant and septage receival and

dewatering units. As seen earlier in this report, the treatment plant ultimately did not work out, but the evaluation team recognizes

that the decision to include it was appropriate at the time. The septage receival and dewatering units permitted the system to receive

wastewater from trucks and bring down the cost of desludging operations. In 2016, a pond was also constructed at Azraq camp to

store wastewater that could be used for agriculture.

UNICEF also assisted the government in improving environmental protections as regards WASH services. For example, UNICEF

built the capacity of the MoWI to conduct groundwater monitoring more effectively.

Other initiatives that UNICEF implemented that took environmental protection and sustainability into account include:

• Working with the Royal Scientific Society to study the most environmentally sensitive way to provide heading for camps.

Providing heating with gas was chosen.

• Promoting water conservation in camps and host communities through implementation partners.

• Celebrating world events, such as World Environment Day.

• Promoting recycling where feasible.

144

EQ17: Did UNICEF’s WASH response identify and reach the most vulnerable people? To what extent did specific

interventions strike an equitable balance between interventions in camps/settlements and in host communities, to reach

the most vulnerable people?


Information


collection

JC 17.1: Sector satisfaction

with UNICEF’s performance

as sector lead.

JC17.2: Evidence that

UNICEF took the lead in

sector coverage and ensured

the most vulnerable’s needs

were addressed.

- Degree to which the most

vulnerable’s needs were

covered.

- Sector satisfaction with

UNICEF’s achievements as

sector lead.

- - Interviews with WASH

sector actors.

As described elsewhere in this report, the evaluation team found that UNICEF ensured the needs of those in camps and at the

settlements were addressed in a coherent manner. UNICEF organized sector coordination efforts and coordination efforts at the

camp and settlement levels to design activities, assign responsibilities to avoid duplication, and set standards to ensure equity of

results as far as that was possible. Following that, UNICEF worked in host communities and ITSs to the extent resources allowed.

The evaluation team does not have evidence that UNICEF used its position as sector lead to make sure those in needs were

identified and that their needs were addressed.

145

EQ18: Did UNICEF as Sector Lead ensure that the needs of those in most need were addressed in a coherent manner,

particularly in comparison to alternative existing systems at the local or regional level (i.e. UNRWA camps)? If not,

how?


Information


collection


UNICEF enhanced

addressed needs in a

coherent manner

-

As described elsewhere in this report, the evaluation team found that UNICEF ensured the needs of those in camps and at the

settlements were addressed in a coherent manner. UNICEF organized sector coordination efforts and coordination efforts at the

camp and settlement levels to design activities, assign responsibilities to avoid duplication, and set standards to ensure equity of

results as far as that was possible. Following that, UNICEF worked in host communities and ITSs to the extent resources allowed.

The evaluation team does not have evidence that UNICEF used its position as sector lead to make sure those in needs were

identified and that their needs were addressed.

EQ19: To what extent did UNICEF role as WASH sector lead enhance response processes and results to avoid

duplication and gap in interventions?


Information


collection

146


processes that UNICEF

enhanced.

- Identification of process

improvements through

documentation and

interviews.

- Vulnerability maps

- National WASH inSchools

strategy.

- Interviews with WASH

programme stakeholders.

Regarding enhancing response processes, a sample of WASH sector meetings from May 2016 through April 2018 show 13

organizations regularly attending. Action points and focal points are thoroughly covered, and cover issues from The National Water

Strategy to Schools Assessments and issues in the camps.

UNICEF also established a number of reporting mechanisms through third party monitors including water quality reporting, ad

security incident reporting. UNICEF established the use of vulnerability Maps for Water and Sewage, though these have only been

in use since 2017. The maps revealed Azraq as the most vulnerable place in both water and sewage.

UNICEF partners that the evaluation team interviewed revealed partner satisfaction with UNICEF as sector lead. One partner said

that, “UNICEF are very good coordinators. They always have staff in Azraq.” A donor said that they were very happy with

UNICEF’s coordination and partnership. A staff member of one partner said, “UNICEF does a very good job of controlling the

work. 95 percent approval rating for UNICEF.”

EQ20: Were partnerships effective and leveraged to the maximum extent to assist the most vulnerable population in

camps/settlements, ITSs and in host communities?


Information


collection


partners provide in kind

contributions.

- Amount of partner in kind

contribution.

- Degree to which reports

show effective collaboration.

- Financial documents.

- PCAs

-

- Partner interviews

- Host community interviews

147


vulnerable populations were

served.

l

The evaluation team believes that partnerships were effective and leveraged to the maximum extent in the camps and settlements.

As described in this report, UNICEF utilized its partners to ensure full coverage in the camps and compliance with standards. Based

on the limited financial information that the evaluation team was provided, the team estimates that UNICEF also leverages over

$27, 250,000 in in-kind contributions from the partners it worked with151.

UNICEF also effectively worked with partners in host communities. Partners were relied on to identify and manage key

interventions. WASH in schools programs were also managed effectively through partners.

One aspect of working with partners that UNICEF could have improved was communicating strategic decisions and medium-term

vision for the project. The UNICEF partners that the evaluation team interviewed reported that they often heard about drastic

changes to plans or operations with little notice or time to prepare. One partner noted “UNICEF take quick decisions – the change

from Mercy Corps to ACF in Azraq could have been done in a better way. It was not a transparent process.”

1.

151 Estimate based on a review of PCA’s during the life of the WaSH Programme

148

149

Annex H: UNICEF WASH programme Results Framework Outcome Indicators

Objective Objective

Outcome Indicators Baseline Target Outcome Indicators Baseline 2015 - 2017 Target

Indicator 1: Existence of a

comprehensive national

monitoring system for school

readiness to inform policy

decision on child disparities

Monitoring system

needs improvement

Monitoring

system upgraded

Indicator 1: Existence of

strategic planning capacity in

MoWI and sector and other

relevant institutions in

Emergency preparedness

and response

no clear strategy strategy in place

Indicator 2:Existence of a

child and neonatal

information system in

support of child and

maternal deaths audits

No information

system existent

Information

system in place

Indicator 2: Proportion of

population at camps and

host communities including

schools using improved

drinking water as per Jordan

standards

100,000 at camps, 225,000 at host

communities and 188,000 students

Target: 100% of 2015

target 140,000 camps,

1,550,000 host

community, 200,000

students


updated and functional MoH

guidelines to focus on neo-

natal mortality and high risk

pregnancies

Not updated Updated


population at camps and

host communities including

schools using improved

sanitation facilities

100,000 at camps, 575,000 at host communities and 180,000 students

100% of 2015 target:

140,000 camps, 545,000

host community,

200,000 students;

Indicator 4: % of health

facilities providing neonatal

care services with the new

MoH guidelines

0 30%


population practicing

hygiene promotion at

targeted camps, host

communities and schools

Baseline: 100,000 camps, 35,000 host communities, and 188,000 students

Target: 100% of 2015

targets 140,000 camps,

150,000 host

community, 200,000

students

2013/2014 2015-2017

National Institutions provide improved

health and neonatal care services and

quality early childhood care with a focus

on disadvantaged groups

Quality WASH facilities are sustained and utilised, and

hygienic behaviours practiced, by the most vulnerable while

ensuring the protection of the environment

150

Output Indicators

151

2013/2014 2015 2016/2017

Output Indicators Baseline Target Output Indicators Baseline Target Output Indicators Baseline Target


comprehensive WASH

Master plan aligned to

National priority, NRP and

integrate Syrian Crisis

Impact

Fragmented

WASH donor

plans, National

plan in place

(2022 strategy)

not include

Syrian refugee

impact

Comprehensive

WASH master

plan in place

aligned to

National plan

(2022) and

incorporate

Syrian crisis

Impact

Indicator 1: Availability

of effective planning,

monitoring and

coordination

mechanism for

engagement of

government

WASH

working

group,

scattered

concept and

preliminary

idea

Master

Plan/Strategy on

Humanitarian for

development in

WASH, GIS/MIS


people living in vulnerable

Households with access to

improved household

WASH facilities and

services

2016:

18,000

2017:

44,000

Indicator 2 : % Reduction

of water wastage/leakage

to ensure targeted host

population accessing

sufficient quantity of

water of appropriate

quality for drinking,

cooking and personal

hygiene (2 cubic)

50 percent of

the water

supplied by the

Public water

systems is Non-

revenue water.

Rreduce water

waste/leakage

by 20%


population benefiting

from safe water and

storage facilities

improvement

150,000

refugees in

HC and

225,000

people in

host

communities

150,000 refugees

in HC and

1,400,000 host

communities


people with access to

improved municipal water

services

2016:

600,000

2017: 3.2

million

Indicator 3: % Increase of

targeted host population

accessing sufficient using

appropriate sanitation

facilities

Sanitation

Coverage in

Jordan overall is

around 62 %

while in

Northern

Governorates it

is at the level of

37 %

5% increased

specifically in

the Northern

Governorate


students provided by

WASH services

188,000

students in

HC

200,000 students

in HC



improved municipal

sewerage services

2016:

400,000

2017: 2.1

million

Indicator 4: No of targeted

population accessing

hygiene promotion and

hygiene education in

schools

KAP survey

available for

Northern

Governorate

and camps

Host community

(Nationally start

in 2

Governorates)

and 324 schools


population at host

communities benefiting

from sewage systems

improvement

575,000 at HC 150,000 Syrians in

HC, 395,000 in HC



improved WASH facilities

in institutions

2016:

60,000

2017:

114,000

Indicator 5: Availability

of Cost analysis Study

conducted (Y/N)

None Evaluation study

Output 1.6: New Output: Support the Ministry of Water and

Irrigation and Sector partners to address the impact of

refugee influx as well as chronic sector needs

Output 4.1: Government and WASH sector partners plan

and optimally utilize WASH resources and services for

equitable access

Output 4.2: Institutions, utilities and

organisations provide equitable access to

sustainable water and sanitation services for the

most vulnerable

152

2013/2014 2015 2016/2017

IR Indicators Baseline Target Output Indicators Baseline Target Output Indicators Baseline Target

Indicator 1:Availability

of effective working

group coordination

mechanism

WASH

Working

group

functional

and ToRs

developed

Formation of a

working group

for the sector

response plan

Indicator 1: Number refugees

provided with safe access to

sufficient water for drinking

and domestic use through

water trucking

100,000

refugees

140,000

refugees


people in camps

reached through

social/community

mobilisation on key

WASH messages and

water conservation

2016:

50,000

2017:

50,000

Indicator 2:Number of

emergency affected

population (male and

female) provided with

access to drinking and

domestic water

250,000

345,000

refugees (in

camps and off

camp) and

180,000

affected

Jordanians.


refugees provided with safe

waste water disposal through

dislodging of septic tanks

100,000

refugees

140,000

refugees


people in institutions

in camps reached

through social/

community

mobilisation on key

WASH messages and

water conservation

2016:

20,000

2017:

20,000


Emergency affected


female) provided with

access to soap and

other hygiene items

345,000

refugees (in

camps and off

camp) and

180,000

affected

Jordanians.



waste water disposal through

provision of Waste Water

Network in Zaatari Camp

0 refugees 140,000

refugees


people in the host

community reached

through

social/community

mobilisation on key

WASH messages and

water conservation

2016:

18,000

2017:

44,000

Output 4.3: The most vulnerable women and

children are aware of and practice key hygiene

behaviours and practice water conservation

IR 1.3: Relevant Core Commitments on WASH are met

in all humanitarian settings

Output 4.3: Urgent needs of vulnerable people

especially children are met through quality and

lifesaving WASH services

153

IR Indicators Baseline Target Output Indicators Baseline Target Output Indicators Baseline Target


Emergency affected


female) with access to

appropriately designed

toilets and sanitation

services

345,000

refugees (in

camps and off

camp) and

180,000

affected

Jordanians.



solid waste management

facilities

0 refugees 140,000

refugees


people in institutions

in the host community

reached through

social/community

mobilisation on key

WASH messages and

water conservation

2016:

60,000

2017:

114,000



female) covered

through face to face

(including HH and

Group sessions)

hygiene promotion

messages

19,300

345,000

refugees (in

camps and off

camp) and

180,000

affected

Jordanians.



access to sanitation facilities by

operating and maintaining the

existing WASH Blocks through

WASH Committees and/or

community involvement

100,000

refugees

140,000

refugees


people in informal

settlements reached

through

social/community

mobilisation on key

WASH messages and

water conservation

2016:

15,000

2017:

45,000


children (girls and boys)

provided with access

to safe water,

sanitation and hygiene

facilities in their

learning environment

and in child friendly

spaces

72,985 230,633

Indicator 6: Number of refugee

Boys and Girls in camps

provided with WASH in schools

19,108

students

45,000

students


children (girls and boys)

not practicing open

defecation in the camp

settings and in host

communities

NA 100%

154

2016/2017

Output 4.4: The urgent WASH needs of the vulnerable women and children are met through safe, predictable and sustainable water and sanitation services and facilities

Output Indicators Baseline Target

Indicator 1: Number of people in camps supplied with water (tankering or network)

2016: 140,000 2017: 140,000

Indicator 2: Number of people in camps receiving wastewater services (tankering or network)

2016: 140,000 2017: 140,000

Indicator 3: Number of people in camps receiving solid waste collection services

2016: 140,000 2017: 140,000

Indicator 4: Number of people in informal settlements with improved access to WASH facilities and services

2016: 15,000 2017: 45,000

Indicator 5: Number of people in institutions benefitting from WASH facilities and services

2016: 20,000 2017: 20,000

155

Annex I: UNICEF Provided Financial Estimates Note: The following documents were provided after the evaluation period.

Overview of examples of increased efficiency in the

UNICEF Jordan WASH programme

(June 2016, updated July 2016, updated October 2016)

Ongoing WASH services in Camps

UNICEF has been providing WASH services to camps since July 2012. Once critical WASH

services were established, UNICEF has consistently worked to reduce the cost associated

with the provision of these life-saving interventions through the construction of infrastructure

to reduce external tankering, establishment of third party monitoring mechanisms to ensure

that only actual services delivered were paid for (verification of truck volumes, reduced

recycling of wastewater etc) and the progressive rebidding of services to ensure that the unit

costs are progressively reduced.

- Unit costs have been progressively reduced as outlined in the table (since 2012 or

from the start of the service):

Price

Difference

(JD/m3) % Reduction

Water Tankering - Za - internal 0.17 14%

Water Tankering - Za - external 0.41 16%

Water Tankering - Az - internal 0.12 8%

Water Tankering - Az - external 0.2 6%

Water Tankering - KAP -

external 0.78 31%

Water Tankering - CC - external 0.78 31%

Desludging - Za - internal 0.02 1%

Desludging - Za - external 0.55 17%

Desludging - Az - internal

Desludging - Az - external 0.61 10%

Desludging - KAP - internal 0.73 29%

Desludging - CC - internal 0.73 29%

Solid waste unit costs - Za 2.9 67%

Solid waste unit costs - Az 8.46 85%

156

Solid waste unit costs - KAP 2.7 63%

Solid waste unit costs - CC 2.7 63%

- Transitioned from external water sources to internal tankering (three wells in Za’atari

and one well in Azraq)

As the four boreholes are now operational, the water is supplied for most of the year

from the internal boreholes with no need for external tankering. In Za’atari, Borehole

1 became operational in 2012, Borehole 2 became operational in 2013 and Borehole

3 in May 2015. In Azraq, the internal borehole started providing water to the camp in

October 2015. Through these key infrastructural projects, the cost of supplying water

has been reduced considerably For Za’atari, the difference between the highest

external water tankering unit cost and the current price is 1.48 JD/m3. Likewise, for

Azraq, the price difference is 2.05 JD/m3. On average, this equates to a monthly cost

difference of 150,000 JD for Za’atari and 73,800 for Azraq (excluding the operational

costs of the boreholes)

- Improved Operation and Maintenance of the three boreholes has resulted in reduced

downtime of the boreholes and their more consistent operation due to regular

monitoring and maintenance (there was no daily external water trucking from

November 2015 to May 2016).

- Over the period July to November 2014, the average daily demand (including

institutions) was 3,704 m3, of which 780 m3 (21%) was from Borehole 1, 1,160 m3

(31%) from Borehole 2 and the remaining 1,757 m3 (48%) was from external

boreholes. Prior to 2015, there were no seasonal allocations and issues were

regularly faced with Borehole 1 and turbidity. As a result, it was often necessary to

further supplement the water supply with external water. Over 2015, the water needs

at a block level were regularly recalculated (based on updated population numbers as

well as a seasonal alteration to take account of increased needs in summer and lower

needs in winter), and as a result, the volume of water required was recalculated and

the volume reduced, over an annual period

- UNICEF’s positive response to the water strike in May 2015 (alternative modes of

delivery and the accelerated operation of BH3, as well as the construction of a mini

network within a few days) sent a very strong signal to the Contractors and their

ability to negotiate

A mini-network was installed and this enabled 200 to 300m3/day to be delivered

directly by pumping and as a result, there has been a reduction (of approx. 270

JOD/day, 8,100 JOD/month, 129,600 to date – the network cost less than 5,000 JOD

to construct)

- Construction of a connecting line to the main pumping distribution line in Za’atari as

a Contingency supply (with the permission of the Minister of Water and irrigation)

which can be activated in the event of an emergency

- Mobilisation on water conservation to parents and children (to reduce children

playing with the water) and reducing the losses – refugees have been engaged to

disseminate messages at tapstands in Azraq where children often played with the taps,

which has resulted in a considerable reduction in the water consumed, as well as the

amount of standing water

- Replacement of items which were prone to vandalism e.g. the ventilation pipes (triple

bolted) and taps (taps removed and a basic design installed) which has significantly

reduced the cost of replacement

157

There was a reduction of 58% in the costs for the replacement of taps, and 24% for

the replacement of the ventilation pipes in Azraq

- Transitioned from external desludging in Za’atari to the internal treatment plant and

nearing completion of the Azraq wastewater treatment plant (unit transportation costs

will reduce to less than one third of the external)

The cost to manage 1 m3 of wastewater in Za’atari Camp has decreased from an

estimated with the WWTP completion in February 2015 (the figures are currently

being updated)

- Extensive third party monitoring to accurately determine the volume of wastewater

reported (physical inspection of the truck when it arrives and before it is offloaded)

and to reduce the scope for ‘recycling’ of the waste

See the figure below:

- Development of an Operational Plan to ensure the most efficient use of the desludging

trucks (to ensure a smooth supply of wastewater to the treatment plant and to reduce

the scope for external desludging)

See the figure below:

158

- Expansion of the storage capacity at the Za’atari Wastewater Treatment to reduce the

scope for external tankering

The works to expand the storage facility (by 400 m3 storage + 200 m3 treatment

capacity) cost approximately 250,000 JOD – this led to an additional 19,500 m3 of

wastewater/month which could be treated at the plant (and not transported

externally) which has led to a saving of c20,800 JOD/month on the tankering costs

- Independent physical verification of the volume of the wastewater trucks – brought to

a weighing station as often modifications have been made to the trucks (the registered

volume was different to the actual measured volume)

The actual measured figure was 27% but a compromise figure of 23% was agreed

with the Contractor – this equates to a 23% saving in the monthly desludging costs

for Za’atari

- Construction of a weighbridge for the volume of the compacted solid waste in Azraq

- Regular rebidding of contracts for improved unit rates as well as UNICEF’s Long

Term Arrangements (as given in the first table)

- Progressive reduction in Cash For Work (gradually reduced and in consultation with

all agencies) while supporting alternative job opportunities (recycling)

The first scale down of CFW took place in April 2016, which reduced the cost by

about 20% with a second (20%) scheduled to take place from 9 October 2016 – these

two reductions will reduce the monthly cost by approximately JOD 10,000

- Progression from partner implementation to UNICEF’s direct engagement of

Contractors (networks, desludging, borehole operation and maintenance, construction

etc)

- The decommissioning of public WASH blocks has significantly reduced the cost of

repairs, operation and maintenance. The installation of private toilets with no O&M

costs for UNICEF and much appreciated by the refugees (significant cost reduction

(over 50%) in the installation of private toilets due to the recycling/or reuse of

materials from decommissioned public WASH blocks)

159

As of early October, 346 WASH Blocks have been decommissioned (out of 355) in the

camp – the materials from the blocks are being recycled for Household toilets for

vulnerable families, where possible

- The phasing out of O&M of WASH facilities in schools and the subsequent

engagement of another agency (by Education section) have led to reduced operational

costs for WASH

- Reduced cost of water quality monitoring due to internalized water supply

- Functional water supply contingency plan have led to price stability; prices could

have been most likely increased if we were more vulnerable

It is hard to quantify this but the fact that alternative modes of transport and delivery

were up and running within one day of the strike sent a very strong message, as did

the construction of the tanks for the network – we consider that this ensured that the

prices for the internal and external water did not increase in the subsequent bidding

period

- Reduced power consumption due to the installation of VFDs at the boreholes

(installed on all four boreholes and already making a difference)

This has just recently been introduced but is anticipated to have two impacts; a) on

the energy consumed (c 7.5% for the reduction in the output) and b) more continuous

water supply with less interruptions and shutting down/restarting

- Construction of water reuse pond where contractors can access water for construction

works at no cost. Otherwise, UNICEF would have been charged for construction

water. The water was also used for cleaning of the water tanks, again contributing to

reduced water needs

During the peak of wastewater network construction, an estimated 56m3 (six trips of

8m3 truck) of water was used for construction, as well as for other purposes – on

average, approximately 60m3 per day of water is collected and used, which has saved

approximately 2,700 JOD per month

- Reuse of water from the Wastewater treatment plant for agricultural purposes in

Za’atari and under construction for Azraq

Approximately 1000 to 1200 m3 of treated wastewater is produced by the Za’atari

wastewater treatment plant, each day – if this had to be tankered to the site, the cost

would be 25,000 to 49,500 JD/month, depending upon the source

- Efficiency review of the energy requirements of all of the infrastructure (an initial

review of the energy consumption of the Za’atari Wastewater Treatment Plant has

been carried out)

- Partner contribution to cooperation agreements (up to c10% but up to 70% in some

agreements)

Za’atari Network Construction:

UNICEF will directly engage the Contractors for second phase of Water and Wastewater for

the Za’atari Networks

- This will significantly reduce the number of staff maintained by partners and

associated indirect costs and overheads, as well as reduce duplication in roles and

responsibilities

- UNICEF will combine the phase II water and wastewater network design and

construction tender documentation, and will directly engage construction supervision

consultants and construction contractor(s). This strategy will improve cost

effectiveness through streamlining of the project management structure and reduction

160

in construction contractor costs through joint delivery of the networks. Joint delivery

of the networks will also offer significant benefits to beneficiaries through reduced

construction impact on communities, reduced construction safety and coordination

risk, and improved construction quality outcomes

It is estimated that by revising the project management structure and combining the

water and wastewater work into distinct packages, the cost will be further reduced by

an estimated 15-22% due to construction and management efficiencies.

- The works have been designed to facilitate the operation of the water network in

districts which will facilitate a gradual reduction in the costs of tankering

It is planned that three districts will be connected per quarter from June 2017 (25%

reduction per quarter) from when the pumping stations start to be commissioned –

this will reduce the Za’atari costs by over 87,000 JD per quarter (for each three

districts)

- The costs to supply water across the camp via the network will eliminate the need for

water tankering and as a result dramatically reduce the cost of water supply – directly

(no costs for tankering) as well as indirectly (reduced monitoring, support staff,

transportation, water quality monitoring etc)

The elimination of tankering from Za’atari and Azraq will result in savings of over

US$ 4M a year (over US$ 7M compared to external tankering)

- Although there will still be a need for desludging of the wastewater network, the

frequency and scale will be significantly reduced

It is anticipated that there will be a reduction in the frequency of desludging from

household tanks from once per week to once every 6 months (desludging from the

communal tanks of the WASH blocks was daily)

- Procurement of desludging trucks for the network desludging (once or twice a year)

which can be done using our own trucks which we can procure

The cost of desludging will be reduced by over US$ 1.5M/year

- Bulk procurement of tanks for Phase 1 of the wastewater network (across all three

partners)

- Pipeline for Azraq – elimination of trucking

- UNICEF’s Engineers oversight of the designs and revision of the BoQs

Reduced reliance on partners and increased oversight of the technical requirements

and strategic planning of progressive construction, commissioning and

operationalization of the networks, will ensure that ongoing operational costs are

progressively reduced as the camp transitions to a longer term sustainable operations

and maintenance framework

- Reduce the cost of the chemical used to treat the wastewater at Za’atari Wastewater

Treatment Plant (ZWWTP)

Review of the ZWWTP hydraulic and process efficiency in coordination with WAJ

will ensure that plant energy and chemical usage requirements are optimised while

meeting camp wastewater treatment requirements. UNICEF Engineers are working

collaboratively with other agencies to investigate the potential for ZWWTP sludge to

be used in a trial biogas generation facility that will process generated sludge

(removing the need for external trucking and disposal of the sludge) as well as

generate renewable energy for export to the local power grid and reduction in camp

operational power costs.

Support to Host communities:

161

- UNICEF support the Water and Sanitation Vulnerability mapping to ensure that areas

in most need were prioritised (based upon agreed criteria)

- UNICEF supports the replacement of leaking water lines and inefficient equipment to

reduce the operational costs in water systems

- UNICEF supports the capacity building of operators of the utility companies to

undertake maintenance and strengthened oversight of the equipment

- UNICEF supports the repair, unblocking and replacement of sewer lines to reduce

sewerage overflows

162

Annex J: Additional Financial Information

The table below summarizes additional financial information provided by UNICEF in

December 2018, after the evaluation period concluded.

The table below summarizes UNICEF WASH programme reported expenditure by category

for the years 2012 through 2017. The table include categories that accounted for at least 1

percent of total expenditure. These categories account for 96 percent of total expenditure. The

top expenditures were for the categories “Sanitation and Solid Waste Management, and

“Provide Access to Safe Drinking Water,” accounting for 35 percent of the total, though there

are many categories with similar titles on the list.

Items USD Percent

SANITATION AND SOLID WASTE MANAGMENT

$47,113,983 18%

PROVIDE ACCESS TO SAFE DRINKING WATER $44,568,288 17%

PREPAREDNESS & CONTINGENCY PLANNI $22,224,107 9%

PROVIDE REFUGEES IN CAMPS WITH SAF $15,547,325 6%

CONSTRUCT/EXTEND WATER NETWORK IN $13,650,381 5%

PROVIDE DRINKING WATER-CAMPS $10,147,397 4%

CONSTRUCT WASTEWATER NETWORK IN ZA $11,457,662 4%

SEWAGE NETWORK-CAMPS $7,409,768 3%

HYGIENE EDU AND MATERIALS $6,011,048 2%

SOCIAL MOB IN CAMPS (GENERAL) $5,590,211 2%

WASTEWATER DISPOSAL-CAMPS $4,210,105 2%

SOLID WASTE MANAGEMENT-CAMPS $4,418,779 2%

WASTEWATER COLLECTION (DESLUDGING) $4,806,943 2%

COLLECT AND DISPOSE SOLID WASTE $4,997,696 2%

TECH ASSIST FOR SUSTAINABLE W $4,732,505 2%

PROVISION OF SANITATION AND HYGIENE $2,074,047 1%

MERCY CORPS PCA/0/12 $1,625,658 1%

ACCESS TO SAFE WATER $1,637,291 1%

WASH SUPPLIES $3,408,290 1%

WASH IN SCHOOLS $3,561,058 1%

WATER SYSTEM REHABILITATION $1,715,156 1%

SEWAGE NETWORK REHAB-HOST COMM $1,615,982 1%

HYGIENE PROMOTION-CAMPS $3,152,868 1%

WATER SUPPLY NETWORK-CAMPS $3,607,696 1%

WASTEWATER TREATMENT-CAMPS $1,322,810 1%

OPERATE/MAINTAIN WASH FACILITY $2,732,411 1%

O&M WASTEWATER TREATMENT PLANTS IN $2,788,510 1%

PROVISION OF WASH SERVICES IN ITS $3,037,358 1%

WASH FACILITIES IN INSTITUTIONS I $2,339,079 1%

WATER INFRASTRUCTURAL SUPPORT $3,060,999 1%

WASH FACILITIES IN SCHOOLS, CFS, C $1,627,130 1%

Total $246,192542 96%

163