Annex C: ECD Kit Mission Report - Home page | UNICEF · 1 Mission Objectives 1. Consult with relevant stakeholders to plan for the evaluation activities (e.g., evaluation governance,

Annex C: Terms of Reference – Two-Country Impact Evaluation on UNICEF’s Early Childhood Development Kit (ECD Kit)

UNICEF Evaluation Office 2016

Annex C: ECD Kit Mission Report

1

Mission Objectives

1. Consult with relevant stakeholders to plan for the evaluation activities (e.g., evaluation governance, timeline, expectations, required approvals, etc.) and to finalize the interventions (e.g., objectives, delivery method, targeted beneficiaries, expected outputs, outcomes, etc.);

2. Discuss lessons learned from previous experiences using the kits within the existing ECD humanitarian and development programming in light of the draft Terms of Reference;

3. Analyse the relevance and applicability of the preliminary theory of change for the evaluation and potential

modifications for the South Sudan and Senegal context; 4. Visit a sample of ECD centres and child friendly spaces (CFS) that will be included in the evaluation and conduct

focus groups or interviews with ECD facilitators to: 1) better understand ground realities for the evaluation (e.g., use of existing monitoring tools, physical settings where the toys will be used, how often the toys are used, etc.); and 2) to learn beneficiary perspectives about the most significant changes observed from using the kits with children and caregivers; and

5. Gather relevant documentation to support the evaluation (e.g., maps of ECD kits with CFS and ECD Centres

identified, relevant programming and policy documents related to ECD, etc.).

Main Findings on Evaluation Feasibility

UNICEF County Offices (CO) and Regional Offices (RO) are committed to the proposed work, with strong buy-in from senior leadership and ECD, Education and Child Protection Specialists who agreed to provide support for the implementation of the interventions and the evaluation in both countries.

UNICEF, government and NGO partners in both countries see great value in evaluating the ECD kit interventions; the results will feed into and advance existing ECD policy and programming efforts in South Sudan and Senegal.

Each country presents a range of opportunities and strengths for the proposed work (namely UNICEF capacity, partner and facilitator interest, motivation to do the study, and rich community assets to leverage), as well as a variety of unique and common challenges (such as the potential for renewed conflict and high human mobility in South Sudan, and incomplete or lacking data on the kit distribution and children’s dosage to the materials in both countries).

Additional data collection and validation is required in both countries to finalize the intervention concept note and terms of reference for the evaluation. In particular, specific information on where the ECD kits have been distributed at the local level are required to determine an appropriate sample for the evaluation.

Both common and unique outcomes have been identified in each country for children, facilitators and parents. These outcomes can be used as a basis for planning the interventions and their associated theories of change. These outcomes largely reflect the results documented in the ECD Kit Global Assessment Report, namely, improvements in children’s early learning, development and psychosocial well-being, as well as improved caregiver attitudes and practices towards children, and more inclusive, supportive and caring relationships among children, among parents, and between children and their caregivers.

The impact evaluation design for South Sudan needs to accommodate the high human mobility of the population, a lack of trust in outsiders and among different social groups, a lack of a control group, and a lack of control over the multiple delivery methods for the ECD kit intervention by various UNICEF partners. The recommended design is a comparative case study impact evaluation. The planned quasi-experimental impact evaluation design with a control group is operationally feasible in the Senegal context.

2

Proposed Location of Interventions and Evaluations

The interventions will take place in the Casamance region of Senegal (slides 2-4), and in Malakal and Maban in the Upper Nile State in South Sudan (northeast region of the country). Both regions are long distances from the country capitals; transport to Maban requires support from the World Food Program; transport to the Casamance region requires ground transport. These regions were selected because they are targeted by UNICEF for the ECD kit distribution. In South Sudan, the Upper Nile State provides an opportunity to work with the ‘most stable’ population living in an emergency context.

Summary of Fieldwork

Meetings with UNICEF Staff Members o South Sudan – Country Office Deputy Representative, Officer in Charge of Education, former Chief of

Education, Chief of Child Protection, Education/ECD Specialists, Child Protection Specialists, and Planning, Monitoring and Evaluation Specialist

o Senegal – Country Representative, Chief of Field Office, Chief of Education, Education Specialists, ECD Consultant, Regional ECD Advisor

Meetings with Partners o South Sudan – 16 participants (9 female, 7 male) attended an all day workshop on the ECD kits, including

international and national NGO partners, one government representative from the Ministry of General Education and Instruction, and two UNICEF staff members.

o Senegal –MoE Central Office, IF/IAF Kolda (Regional and District MoE), IF/IAF Sédhiou (Regional and District MoE)

Observations of ECD Settings/Semi-Structured Interviews with Beneficiaries o South Sudan – A total of 5 site visits were conducted (4 ECD centres and 1 CFS), including interviews with 18

facilitators and 22 other caregivers (parents and/or PTA members), and observations of more than 600 children between the ages of 3 and 8 years.

o Senegal – A total of 8 site visits were conducted in community preschools, including interviews with 20 teachers and school administrators and 5 other caregivers (parents and/or PTA members), and observations of more than 400 children between the ages of 3 and 5 years.

http://news.bbc.co.uk/2/hi/africa/1896798.stm

3

South Sudan Site Visits

Location Population Characteristics

Setting/Partners Year ECD Kit

Received/# Kits

Observed

# Facilitators Present

#/Age of Children

Observed

# Parents/PTAs Present

Juba Protection of Civilians (POC) camp of internally displaced people (IDP) who speak Arabic and English

ECD Centre, Intersos (NGO)

2015 – 15 kits

17 facilitators total; we observed 5 facilitators (2 male; 3 female)

1,255 children total; about 150 observed during visit, ages 3-7

7 (2 women, 5 men) were interviewed, but there are a total of 14 PTA members for the ECD Centre

Juba Older children who are survivors of the first conflict; children living on the streets; abandoned children, Arabic speaking

CFS + residential dormitory for children and adolescents, Confident Children Out of Conflict (national NGO)

5 kits received in 2014 4 ECD received in June 2016

2 teachers 5 social workers All female We interviewed 3 of these staff members

60 children under 18, about 25 are under the age of 8; we observed 6 children (5 ECD and 1 adolescent)

None available during our visit

Maban, Upper Nile State

Refugee families from Sudan who speak Arabic, Batil Refugee Camp

Malul ECD Centre, Lutherans World Federation INGO (LWF), MoE

2012 – 1 new kit opened for the day

5 teachers (2 males; 3 females) 5 water carriers 3 security guards

479 children -235 boys -244 girls -ages 3-7




Gortes ECD Centre, Lutherans World Federation INGO (LWF), MoE

No ECD kits 1 facilitator for the CFS, no ECD Centre teachers available

Estimated number of children 50 – the space was visited during the afternoon when it becomes a CFS

7 (3 women, 4 men)



Pora ECD Centre, Lutherans World Federation INGO (LWF), MoE

2012 - # of kits not confirmed

4 facilitators (2 females and 2 males)

3-7 years old 8 (4 males, 4 females)

NOTE: ECD Evaluation Specialist and ECD Specialist from HQ conducted field visits in Juba. Education and Child Protection Staff from the South Sudan CO conducted field visits in Maban after receiving training from HQ specialists.

4

Senegal Site Visits

Location Population Characteristics

Setting/Partners Year ECD Kit Received/#

Kits Observed

# Facilitators Present

#/Age of Children

Observed

# Parents/PTAs Present

Kolda, Senegal

Poor, rural families who speak French and Poular

Ecole Maternelle de Dabo, MoE

2014 – 1 kit 3 (2 male teachers, 1 female director)

125 in school, ages 3-5, a group of about 25 observed

4 PTA members (2 female, 2 male)

Kolda, Senegal

Poor, rural families who speak French and local language

Ecole Maternelle de Joseph Baldé, MoE

No ECD kits 7 (1 male teacher, 6 male interns being trained by the MoE), we spoke with one male teacher

One classroom - 37 children, age 3


Kolda, Senegal


Ecole Maternelle de Gadapara, MoE

2014 – 1 kit 3 teachers (2 female, 1 male)

106 children (none present during our visit)

1 PTA member (female)

Kolda, Senegal


Ecole Maternelle de Municipale, MoE

No ECD kits 3 teachers (all female)

Ages 3-5, about 25 per class, split into 3 groups


Sédhiou, Senegal

Poor, rural families who speak Mandeng

Ecole Maternelle de Karcia, MoE

2014 – 1 kit 3 teachers (2 female, 1 male)

91 children ages 3-5, we observed about 15


Sédhiou, Senegal


Community preschool (name not confirmed), MoE

March 2016 – 1 kit

2 teachers (both male)

140 children, ages 3-5, none were available when we visited


Sédhiou, Senegal

Poor, rural families who speak French and Mangcai

Classe communautaire de Sitaba, MoE

March 2016 – 1 kit

1 male director 1 female teacher

50-60 children ages 3-5


Sédhiou, Senegal


Ecole Maternelle de Sédhiou, MoE

2011 – 1 kit 6 teachers (3 women, 3 men), we spoke with one male teacher

206 children, ages 3-5, 2 classes per age group


NOTE: ECD Evaluation Specialist, HQ and ECD Regional Advisor conducted the field visits in Senegal. A representative from the Ministry of Education’s Central Office and Chief of UNICEF’s Field Office in the Casamance region, participated in the field visits.

5

Relevance of the Foundational ECD Kit Interventions

Existing ECD Kit Interventions - According to the ECD kit guidebooks and theory of change, the foundational intervention should ideally consists of: 1) a targeted supply of the kits to reach the most marginalized children; and 2) basic training for kit facilitators in play based early learning and development. South Sudan more closely resembled the comprehensive intervention envisaged, while Senegal had only supplied the kits to date. The kits were well received by partners as a way to improve the quality of ECD/ECE services in Senegal, and as an entry point for delivering ECD/ECE in emergencies programming in South Sudan.

South Sudan - The ECD kits represent the primary strategy for UNICEF to support young children and their caregivers who have been impacted by the crisis in South Sudan and the Sudan since 2014. The ECD kits are being used through two delivery models in South Sudan. The Child Protection Section has been deploying the ECD kits for psychosocial support (PSS) in child friendly spaces and other community-based response settings that are supported by international and national NGOs. The Education Section has been supporting the use of ECD kits for early learning and school readiness within ECD centres supported by NGOs and the Ministry of Education. To date, ECD kits have been distributed nationally in states with active humanitarian response (such as the Upper Nile), as well as in states that have not witnessed conflict.

Senegal - In Senegal, the ECD kits have provided an opportunity for UNICEF to further enhance its partnership with the Ministry of Education to improve the quality of early learning within the country since 2012. The UNICEF CO is using the kits to support community-based preschools in partnership with the Ministry of Education. The kits were provided as an enrichment supply to improve the quality of the learning environment in these preschools. To date, ECD kits have been distributed in the Casamance and Dakar regions. The UNICEF CO is also considering the use of the kits in nutrition/feeding centres; however this initiative will not be considered for the evaluation as it is still in early stages of development.

Evaluation Context

South Sudan - The evaluation is viewed by UNICEF colleagues as a way to support the CO in developing an integrated ECD programme with Education and Child Protection Sections (in particular, but also Health, Nutrition, WASH). There is also great interest from the UNICEF CO, MoE and Ministry of General Education and Instruction to revive earlier efforts from 2008 at developing a national ECD policy and curriculum, which the evaluation results can also inform. NGO partners are interested in improving the delivery of the ECD kits through the evaluation process, and in improving the overall quality of their ECD in emergency programmes. The evaluation is considered timely, as the CO is transitioning from emergency response into recovery and the (re)establishment of services in collaboration with government and NGO partners. However, the socio-political situation is highly fragile given the peace agreement was only recently reached and put into action. There is great concern that the country will erupt into conflict again, with the proposal to split the country from 10 into 28 states, and because of an overall lack of trust among different social groups. Therefore, the evaluation will take place within an uncertain and evolving humanitarian context, with a high potential for localized conflict and subsequent human mobility within and outside of the country.

Senegal – The evaluation is viewed by UNICEF colleagues and government partners as a mechanism for improving the quality of ECD centres and early learning opportunities for young children. There is great interest by the Ministry of Education in community-based preschools as a replicable model with potential for scaling, especially among the hardest to reach families living in the poor, rural Casamance region. The evaluation is considered timely because the results can help inform the strengths and weaknesses of this approach to quality early learning provision. Communities are increasingly demanding preschool services; however, the government is not able to provide an adequate supply (0.1% of the national education budget is currently devoted to preschool), and parents struggle to pay teacher salaries and to provide basic learning materials. Therefore, the evaluation will take place within a development context to support the preschool education sub-sector, with a high potential to influence the quality of ECE provision within the country.

6

Summary of Internal and External Factors Relevant to the Evaluation Context

Strengths (Internal to UNICEF) Opportunities (External to UNICEF)

South Sudan Senegal South Sudan Senegal

Strong support from CO Deputy Representative for ECD and for the evaluation

Support from RO ECD Specialist

Committed support of staff from Child Protection and Education sections to execute the intervention and evaluation

Other support from CO staff in GIS mapping and M&E

Interest in creating an integrated ECD CO programme

MICS data and recent Situation Analysis available

Strong support from CO Representative for ECD and for the evaluation

Technical leadership from RO ECD Specialist

Committed support of staff from ECD and Education in CO and Field Office to execute the intervention and evaluation

Other support from CO staff involved in innovations, GIS mapping and M&E

Strong interest from NGO and government partners in the evaluation and how to improve the delivery of the ECD kits

Interest among NGOs to improve upon ECDiE programming

Earlier efforts by UNICEF, government and NGOs to develop a national ECD policy and curriculum

Partner program monitoring data available (quality not yet validated)

Potential for community engagement

Strong support from the Ministry of Education at all levels (central, regional and district levels) for the interventions and evaluation

Desire of the government to improve the quality of its preschool services

National ECD curriculum and policies that can be leveraged/improved

Strong M&E capacity within the country through academic networks

Government preschool monitoring data available (quality not yet validated)

Potential for community engagement

Weaknesses (Internal to UNICEF) Threats/Risks (External to UNICEF)

South Sudan Senegal South Sudan Senegal

Arranging in-country travel may require more time and additional oversight by CO staff

Attempts at contracting with a third party monitoring firm to support CO M&E needs has not worked well (poor quality deliverables)

Staff turnover during the evaluation is likely

A recent Situation Analysis is not available

No MICS ECD data available

Staff turnover during the evaluation is likely

High likelihood of resurgence in localized conflict

High potential for human mobility among potential evaluation participants

Long distance from capital to locations where ECD kits are being used

Low M&E capacity in the country

Participation of the Ministry of Education in the evaluation needs to be carefully considered to ensure independence

Long distance from capital to locations where ECD kits are being used

No existing maps of preschools

Community Participation in the Management of ECD Settings

The Role of Parent-Teacher Associations (PTAs) – PTAs are active in both Senegal and South Sudan and primarily provide a mechanism for community-based management of the ECD centres. In South Sudan, when asked about the role of the PTA in the ECD Centre in Juba, the PTA chairperson explained that in a ‘normal’ situation, the PTA would be comprised of doctors, minsters and influential community members who are able to help support the school through cash donations and other types of support because they are influential community members. However, in the current situation, the PTA members are not working, so they cannot contribute financially to the ECD centre. According to one PTA chairperson, “the PTA tries to help to observe the children and to identify which children do not have clothes or who are barefoot and try to get them clothes and shoes to wear. They try to make sure that the teachers who are working with the children are prepared and that the materials from NGOs are being used properly. They also try to pay attention to whether the space is a conducive environment for children.” For example, it is the PTA in Juba that helped to mobilize support from the UNHCR to ensure children had play opportunities before the ECD kit was available.

The Role of Parents – In Senegal and in South Sudan, the expectation of parents is that they will contribute to the financing of their children’s education. In South Sudan, many parents struggle to sustain their families; and thus, they rely upon NGOs to support and maintain the ECD Centres. In Senegal, parents are expected to support teacher salaries in the community-based preschool model. Teachers remarked that parents started sending their

7

younger children to school with their older children, creating a community-driven demand for ECD services on the primary school sites. However, because parents cannot afford the school fees, several teachers noted that they were not paid for several months. In South Sudan, parents reported helping to mobilise the community to enrol their children in the ECD centres. This was done through parent meetings in the community in which they emphasized the ECD centres as a “good environment for the children that will make them forget what they experienced during the crisis.” Parents also reported sending their children to play in the ECD centres to give them free time to do other activities to support their families. The engagement of parents has strong potential for the planned interventions, and could be strengthened in terms of their participation in their children’s

learning and playing activities (discussed in greater detail in other sections of the report). Because only one CFS was visited in South Sudan, it is not

possible to comment on the role of parents in these ECD settings.

Children’s Participation – Children’s participation in the observed ECD centres was minimal and tokenistic in both countries. For example, in one school in Senegal, we observed children participating in the transport and storage of the ECD kit materials, demonstrating to some extent their ownership over the toys (slide 6). While teachers felt children should have some responsibility for the ECD kit toys, their participation did not go beyond this form of engagement (for example, to decide which toys or play activities to do). In South Sudan, when asked how children participate in the management of their play activities, or in decisions about their ECD centres, the PTA responded by

explaining the children are ‘informed’ about school decisions, rather than being invited to participate in the decision-making process.

Conditions of the ECD Settings

ECD Centre Construction – Most of the ECD centres observed in both countries were temporary learning spaces (TLS) made of non-durable materials such as wood and straw (see slides 11-15 and 46-50). In Senegal, these TLS are often located adjacent to primary schools. Primary schools tend to be more permanent in nature, constructed from cement and more durable materials (slide 9). We only observed one ECD centre made out of durable, cement and wood; this was a community-based preschool in Senegal that was constructed by World Vision (slide 16). In general, the TLS are of poor quality, but help to protect the children from sun, wind and rain to a certain extent. In one case, teachers reported shifting children from one side of the room to the other to avoid sunlight during certain parts of the day. In one ECD centre in Senegal, the TLS had fallen on top of the teachers and children right before we arrived (thankfully, no one was hurt), raising potential safety hazards. In the Batil Refugee Camp in Maban, two of the ECD centres met in an open area; there is no construction of any kind (slide 57). Classes are conducted under trees in an open space without a roof and usually in the morning when the sun is not very hot. Other ECD centres in Maban consisted of three classrooms made out of a basic structure with metallic sheets and no lateral coverage (slide 58).

ECD Centre Infrastructure - The physical environment immediately surrounding the ECD centres generally included large plots of land with dirt and sand with minimal vegetation (slide 46, 57), and wooden or cement

walls to “keep other children and adults out” (slide 5). It was clear that measures were taken to ensure children’s safety. However, in the open air ECD centres, challenges reported by facilitators included “donkeys and cows passing through the space and scaring the children.” All of the ECD centres lacked sufficient water for the children and teachers. In South Sudan, we observed one or two latrines in some ECD centres, but there was no clear differentiation by sex. Washing facilities are available for the children to use but are insufficient in quantity. None of the ECD centres had a school canteen or ways of providing meals to children. For this reason, parents, teachers, facilitators and community elders requested basic improvements to the infrastructure of the ECD

centres in both countries.

“The location of the centre is safe. The centre is located within the camp and the neighbourhood is the families where children come from. Passers-by go through the compound and it’s likely that learning is interrupted in some instances. The surrounding is not noisy.”

UNICEF field notes from ECD Centres, Batil Refugee Camp, South Sudan

“If there is a day set aside for cleaning up the school, the children are informed that on this day there will be a classroom/school clean up and that they should come prepared to help clean the school. If there is a holiday, the children are told they should not come to school because it is a public holiday; if there is an issue around construction of the school, they tell them not to come to school; if a teacher is sick they let the children know.”

Male PTA member, Juba PoC site, South Sudan

8

Existing Play and Recreation Opportunities - In Senegal, many ECD centres had outdoor play and recreation equipment available (mostly tailored to primary school children) that we observed younger children using (slides 20-22). In South Sudan, the ECD Centres also provide some outdoor recreation equipment (such as merry-go-rounds, volleyball court, and swings), including some locally made playgrounds (slides 41-43, 57). In both countries we observed the equipment was broken, no longer operational, or requiring repair (slide 21). For example, in one ECD centre in Senegal, we noted a slide turned on its side, and a merry-go-round not secured into the ground. In South Sudan, UNICEF staff noted broken down swings that had been provided by UNHCR. UNICEF recreation kits were also prevalent in both countries, which younger children are using outside of the ECD centres, including jump ropes and footballs (slides 49-53).

Classroom Environment – The quality of the classroom environment within the observed ECD centres also varied within and across countries. In South Sudan, we observed classrooms consisting of open-air spaces under trees with no matting for children to sit on, as well as classrooms with rows of benches that were secured into the dirt so they could not be stolen or rearranged (slides 47-48). In open air classrooms, children are easily distracted during the lessons (slide 57). Furthermore, writing material for the children and chairs for the teachers are insufficient. The staff lack a clear identification badge. In Senegal, we observed open spaces with matting for children to sit on inside the classrooms (slides 14-15, 18), as well as UNICEF tables and chairs designed for small children (slides 17, 18 and 28). In South Sudan, no interior decorations were observed within the classrooms (slide 47-48). In Senegal, we observed the use of coloured paper and recycled chip bags to make decorations, as well as art work made by local painters (slides 18-19, 22). In other classrooms in Senegal, we observed minimal or no decorations or furniture. However, murals drawn by local artists could be observed on primary school walls (slides 5, 7-9), including one in Senegal that says, “First I do my homework, then I play” (slide 7).

Storage of the ECD Kits – The ECD kits were stored in different places. In the PoC site in South Sudan, at first the facilitators stated that they stored the kits in their home camp because they do not have storage space at the ECD centre. This presented an interesting possibility of facilitators potentially using the kits with children near their home. However, when asked follow up questions, the facilitators changed their story and stated the kits were stored with security guards ‘near their homes’. In other sites in South Sudan, teachers noted that the materials were stored in the staff or director’s office (slide 39, 61). These spaces are often multi-purpose and are used by multiple groups and agencies for meetings and storage. In Senegal, the newer ECD kits (those received within the last 3 months) were generally stored within the classrooms or staff offices in their carton or box (slide 24). In other cases where the kits had been used since 2014, we observed loose toys inside existing shelving (slide 23); in other cases the toys were available and scattered throughout the room (slide 28-29). For this reason, where and how the kits are stored may offer an indicator on their usage, assuming that with more frequent usage the toys become more integrated into the classroom environment.

Presence of Other Didactic and Play Materials – In South Sudan, the ECD Centres received the ECD kits alongside other UNICEF kits, such as the school-in-a-box and recreation kits (slides 49-53). This is because the ECD Centres are used by children up to the age of 10 years. The same situation was also true in Senegal, with the presence of multiple UNICEF kits. In addition, many of Senegal’s ECD Centres had other didactic and play materials, such as those donated by NGOs or philanthropists (slides 30-32). Often these other play materials were not age or culturally appropriate and stood in sharp contrast to the carefully selected ECD kit toys. For example, many toys included small parts that could be swallowed by young children, or included dolls that do not look like the children (slide 32). Some toys donated to ECD centres in Senegal also reflected religious themes, such as learning the alphabet through characters or passages from the Bible. Because other play and learning materials are available in many ECD centres, it may be challenging to isolate the unique attribution of the ECD kits for their impact on children.

“The children sit on mats. The mats are damaged and not enough for the children. Some of them sit on the dusty floor. Children can spot anything on the floor, and whatever it is, it will probably go in their mouths, especially the very young children who come with their siblings.”


9

Supply Dimensions of the ECD Kit

Confusion over Multiple UNICEF Kits – In both countries there was great confusion among partners and beneficiaries about the kits provided by UNICEF. Because UNICEF has more than one kit – a recreation kit, a school-in-a-box kit, a hygiene kit, and an ECD kit, etc. – it is difficult for facilitators to understand the different objectives of these kits. In particular, the NGO and government partners, as well as the facilitators of ECD settings described receiving the kits and using them regardless of what they contained. Some of this is caused by the labelling on the box, which is in English, as well as the packing slip, which is also in English. There is also no clear indication of the age groups each kit is designed to address on the outside of the box, so some contents of the school-in-a-box kit end up erroneously inside preschool classrooms, especially in Senegal where the preschools are located adjacent to primary schools. We observed some teachers, therefore, teaching young children with primary school materials from the school-in-a-box kit. Further complicating matters, older versions of the ECD kit contain different items than newer versions of the ECD kit, which can cause confusion among facilitators.

Tracking the ECD Kit Distribution – One of the most challenging aspects of the supply of the ECD kit is tracking its distribution at the local level. The data kept by UNICEF at the CO level only monitors which partners have received the kits, how many, and in which regions. However, data on the actual distribution of the ECD kits by UNICEF partners within preschools, ECD Centres or CFS have proven difficult to obtain. In Senegal, the MoE does not even have a map of all its schools – it can only provide a list of schools that received the ECD kits and their general geographic location (e.g., about 30km from the city of Kolda). But this information is often erroneous – in fact we encountered several preschools that claimed to have ECD kits, but because of the confusion described earlier, what they actually had were school-in-a-box kits, not ECD kits. Similarly, there are no maps of the ECD kit locations in South Sudan at the local level (only at the state level), often because there are no maps that exist or because of the ‘temporary’ nature of the refugee and IDP camps. Understanding the exact locations where the ECD kits have been distributed is a basic need for an impact evaluation in order to determine an appropriate sample. Data on the local distribution of the kits will need to be updated and validated by both CO before the evaluation can take place.

Capacity of the ECD Kit Facilitators/Caregivers

Description of the ECD Kit Facilitators – In South Sudan, the ECD kit facilitators tended to be young adults who were teachers prior to the conflict, NGO staff members, or trained social workers, with a mix of both men and women. Some facilitators knew a little bit of English, but most speak Arabic. In addition, older children were observed as facilitators of the ECD kits with younger children, demonstrating a child-to-child approach to the materials within child friendly spaces. In Senegal, preschool teachers are the facilitators of the ECD kits, with more men than women serving in this capacity. The facilitators tend to be able to speak French, although reading French may be more challenging. For example, one facilitator claimed that she could read the ECD kit facilitator guide in French, but she was not able to use some toys. The facilitators also speak the local language (such as Poular) which varies by region.

Facilitator Training on the ECD Kits – In Senegal, no formal training on the ECD kits has taken place to date. Since 2012, the ECD kit intervention in this country has consisted of the supply function only, rather than the comprehensive approach recommended by the ECD kit guidebooks, which includes the supply of the kits and basic training. In this regard, Senegal presents an interesting ‘natural experiment’ for understanding how and to what extent facilitators were able to use the kit materials on their own. In general, facilitators reported using their own intuition and experimentation to figure out how to use the toys. For example, one preschool teacher demonstrated how he supports children in replicating objects they observe in their environment. He did this by sketching an outline for a car in the dirt, and then encouraged a young boy to use the coloured blocks to trace the outline (slide 24-25). Some facilitators also recalled having a unit on the role of play in early learning as part

“When we asked the facilitators to take the ECD kits out for the children, they don’t know which one to pick among the many piled up. We understand that they are not familiar with the distinction among ECD, Recreational Kits and School-in-a-Box kits.”

“The Education Assistant confirmed that the office doesn’t open the boxes. They just distribute what they are told to give out.”


10

of their formal training to become preschool teachers, which is also mentioned in their curriculum guides (slide 35-36). Some were able to make linkages between this training and the use of the toys to teach early numeracy and literacy skills, such as counting and vocabulary for shapes and colours. However, facilitators also reported struggling to understand the use and function of certain toys, especially those that require more cognitive skills. In South Sudan, partner NGOs and facilitators reported receiving training on the ECD kits. These trainings consisted of a training of trainers (ToT) among different stakeholders involved with managing and using the ECD kits. At a central level (in Juba), partners reported that UNICEF trained them on broad principles of ECD in emergencies over a period of a two-day workshop, during which 1 hour each day was devoted to learning about the ECD kits. The training on the ECD kits was reportedly very general, explaining what each toy is designed to do and how to form groups by age to ensure children are using the appropriate toys. These ‘master trainers’ (who largely consist of well-educated NGO staff members) then provide training to frontline facilitators of the ECD kits, including community members, social workers, youth and teachers working in the ECD centres or child friendly spaces. However, there is consistent evidence from partners, facilitators and parents that the cascade approach is not sufficient for frontline facilitators. For example, at one ECD centre in Batil Refugee Camp, out of the 5 facilitators, only 1 had received training on the ECD kits. In other cases, facilitators reported trying to train each other on using the toys, especially if one facilitator had the opportunity to attend the training while others could not.

Use of the ECD Kit Guidebooks - In both countries, the use of the ECD kit guidebooks was problematic for several reasons. First, older versions of the ECD kit did not include any guidebooks. Thus, since 2012, most of the facilitators in Senegal did not receive a guidebook, nor did they receive training. In South Sudan, only 9 out of 16 NGO partners reported using the ECD Kit Activity Guide, and none reported ever seeing the Coordinator’s Guide or Facilitator’s Guide. However, in Senegal, even facilitators with later versions of the kits who had received the guidebooks in French could not read easily in French. In South Sudan, partners reported that the use of the ECD Kit Activity Guide does not work well, because of the skills level of the facilitators (some ideas are reportedly ‘too complex’), and because their time is limited to use the materials for self-learning. In both countries, stakeholders questioned the logic of including the ECD Kit Activity Guide in three languages (English, French and Arabic), given not all languages are relevant for their local context.

Observations on Toy Conditions/Usage – In general, the ECD kits appeared to be well used in Senegal, with the exception of those kits that were delivered in the last three months. In South Sudan, usage appeared to be less consistent across the sites; in some cases the toys were brand new (the packaging had not yet been opened), and in other sites some toys looked well used (as evidence by dirt or marks on the toys). Some toys are used more consistently than others in both countries, in particular, the stack-and-sort, the shape sorter, moulding clay, the puppets, and the block puzzles (slides 26-27, 44-45, 54-56, 61-62, 64-65). Other materials that require greater cognitive skills (such as the domino puzzle and chain matching blocks) were too complex and not well used because the facilitators did not understand what to do with them (slide 60, 64). In Senegal, materials that require replenishment (such as crayons, coloured markers, and soap) were no longer available, with teachers reporting they were all used. In South Sudan, children and facilitators requested more small balls, moulding material or clay, exercise books and pencils.

Drawing upon Traditional Forms of Play – Before the ECD kits arrived, facilitators reported that they encouraged children’s play with local material, by digging holes in the ground, using stones and building balls (slide 36, 59, 62, 63). For example, “the children use to make their own football from used socks, we also used local clay to mould toys.” There is great potential to improve adult capacities to play with children using the ECD kit materials, through the local production of toys, by leveraging existing community assets and artistic skills, and by drawing upon traditional forms of play. In both South Sudan and Senegal, when asked how to improve the ECD kits and their effective use, NGO and government partners recommended local toy production (using the ECD kit as a model), in addition to focused trainings for frontline users using guidebooks that are adapted for the local context. Local toy production was viewed as a way to make the ECD kit intervention more sustainable,

“When I was a child I used to play football; at our time we did not have ECD kits. We had a hard time having toys and nice materials like this. We used to mould mud. We made toy cars from old cans.”

Father and PTA member, Batil Refugee Camp, South Sudan

11

as an opportunity for community members to become more engaged in their children’s learning and development (and in the ECD centre activities, more broadly), as a mechanism to ensure traditional forms of play are retained, as a way to teach young children about the environment and recycling, and as a livelihood strategy for adults and children to learn new skills that might support their future employment or wage earning potential.

Promoting Intergenerational Empathy through Play Recall – Related to the last point, we also noticed a high degree of engagement by parents and other caregivers when asking about their own play experiences as a child, suggesting memory recall as a potential method to include in future trainings to build intergenerational empathy around the value of play. In particular, the use of mud or local clay to mould objects from the environment was found to be very important for both parents and children over time (slide 63). Interestingly, the use of the clay that comes with the ECD kits received mixed reviews. On the one hand, some partners in South Sudan mentioned that the clay should be removed from the kits because “children eat the clay.” However, frontline facilitators remarked upon the significance of the clay as a way to connect with the more traditional forms of play they were familiar with (slide 64). Currently the parents we spoke with reported having a very limited engagement in play or learning activities at the ECD centres, and specifically with the ECD kits. For example, one parent in South Sudan stated, “I do not use ECD kit materials - first of all they are not enough and I just thought it is meant for children, not for us. But I believe that I should be using it to play with my children.” Parent engagement in playtime activities at home was also mixed in both countries. Only a handful of the parents we interviewed reported playing games with their children at home. These games consisted of crafting objects out of mud, singing, telling stories, playing hide-n-seek, or using objects from the home (such as dishes) to create games that stimulate children.

Forms of Play Observed with ECD Kits – In line with the results documented in the ECD Kit Global Assessment Report, we observed children and adults engaging in different forms of play with the toys and materials. For example, adults were observed engaging in guided play or teaching through play in which the facilitator has a direct role in supporting children to use the toys to achieve a specific objective (such as using coloured blocks to teach children colours) (slide 27). We also observed free play among children, or play in which the children direct their own games or uses of the toys (slide 26). More specific forms of play observed during the site visits included: 1) small-motor play, in which children manipulate small objects with their hands and fingers to sort and stack toys (slide 24, 45. 61-62); 2) construction play, in which children build objects seen in their local environment (63-64); 3) symbolic play, such as when children take one object and change it into another prop using their imaginations (e.g., this block is a house, this block is a tree, etc.) (slide 64); and 4) competitive play, in which one child was encouraged by an adult facilitator to compete with another child on a task (such as to see who could build a tower first with the stacking toys) (slide 55).

Facilitator Capacity to Manage Large Group Sizes – Aside from facilitator training, two other determinants appeared to influence children’s exposure (or dosage) to the ECD kit materials, including: 1) the child-to-facilitator ratio, and 2) the number of kits supplied to each ECD setting. Partners in South Sudan and Senegal repeatedly remarked that they are overwhelmed by the number of children coming to the ECD centres and CFS each day. For example, while the UNICEF Senegal CO reported that the ECD centres were comprised of 3 teachers and 100 students on average (or a ratio of 1 facilitator for every 33 children, 1:33), during our field visits, we noted ratios of up to 1:70, or 1 facilitator for every 70 children. In South Sudan, partners and facilitators of the ECD centres and CFS reported even higher ratios up to 1:100, or 1 facilitator for every 100 children. These overcrowded conditions are further complicated by the size of the classrooms, where structures were present.

Facilitators have developed some skills for managing large group sizes, such as splitting the children into small groups. However, when split into small groups, the facilitator can only manage to ‘teach’ one group at a time, which necessarily leaves the other groups to their own devices if no other adults are present. Facilitators do not appear to have the skills to use the toys with such large groups. For example, during a site visit in Juba, facilitators brought a minimal amount of toys from one ECD kit (out of the 15 they received) to use with more than 150 children (slide 54). This caused children to push and fight each other to play with the toys. Large group sizes combined with limited facilitator capacity means that many children do not have access to the toys each day. This situation also leads many facilitators, parents and children to complain that they do not have enough toys. For these reasons, the number of kits provided to each ECD centre should be carefully matched with the

12

number of children attending or enrolled to ensure an appropriate dosage or exposure to the materials during the interventions that will be evaluated. Ideally, the number of trained ECD kit facilitators in each setting should also be increased to ensure children’s routine exposure to the toys and learning materials.

Current Monitoring Approaches

South Sudan – NGO partners reported monitoring a range of indicators on their programme beneficiaries, including: 1) attendance data by age and sex, 2) individual child progress, and 3) interactions between children, and between children and caregivers (slide 66). Partners suggested this data was of ‘good’ quality, although this has not been verified. Partners do not currently monitor interactions between caregivers, nor do they collect data on children with disabilities, the child friendliness of the ECD settings, the type of level of vulnerabilities children and families face, and their socio-economic or religious/ethnic/cultural group. In addition, NGO partners often must collect monitoring data from a random sample of children, given the sheer number attending ECD centres or CFS. Further complicating matters are the different programming approaches employed by the range of NGOs involved in the ECD kit distribution. Monitoring tools are not consistent or systematic across partner organizations, nor do they focus upon the use of the ECD kits.

Senegal – Government partners provide minimal monitoring of the community-based ECD centres where the ECD kits are being used. Each year the Ministry of Education creates a plan to inspect a specific number of schools, which is largely determined by budget and human resources. The reality is that many schools are never inspected because of their remote location and a lack of funding. When inspections do occur, the inspector examines the general conditions of the school and classroom, teacher capacities, the learning environment and available materials. The UNICEF CO proposed adding something about the ECD kit to be monitored as part of this process. However, given the infrequent nature of the inspection visits, this information will not be useful for the evaluation. The MoE does have attendance data by sex and age for each community-based preschool, but it is not clear how valid or up-to-date this data is. Some teachers reported that they do not keep track of attendance on a daily basis or report increases or decreases in attendance to the MoE, or this information (if collected) is out-of-date (slides 33-34). Thus, existing monitoring approaches for the ECD kits are not currently in practice.

Reported Child Outcomes Associated with the ECD Kits

Dosage – Before reporting on the most significant changes observed in children and caregivers through their exposure to the ECD kit toys and learning materials, it is worth repeating that the exact dosage of the materials is not currently known because this data is not being collected through routine programme monitoring practices in either country. Instead, what is available are proxy indicators, such as the observations on toy usage and wear and tear discussed previously, the degree to which the toys appear to be integrated into the classroom environment, and self-reporting by facilitators, parents and children. However, self-reporting is also problematic, as it was difficult for beneficiaries to say how many times they used the toys each week. What beneficiaries are able to comment upon are the hours of operation for the ECD centres and CFS, and how frequently they come each week. Most facilitators, parents and children claimed that the ECD centres and CFS were open each day, Monday through Friday, from 8:30 until 11:30. Each group reported consistent attendance – or that they came to the centres each day for about three or four hours. However, it is not clear how many of those hours are devoted to using the ECD kit toys, as the centres do not appear to have a general work-plan or timetable of activities. In addition, during the rainy season in both countries, attendance in the ECD centre often decreases because it is difficult for the children to travel, and because of the poor quality of the school environment to protect children from the harsh weather. A better understanding of the dosage children receive with the ECD kits will be required to appropriately assess how long it will take for impacts to be observed (e.g., it may take longer than a 6-9 months to observe impact).

Use of Toys by Different Child Groups – When asked about differential use of the toys among children, most beneficiaries responded by explaining that certain toys are more attractive or preferred by girls and boys. This finding was consistent across countries. Girls reportedly like the puppets and small blocks that can be strung into a necklace more than boys (slide 65); boys reportedly like the small balls that come with the ECD kits more than the girls. In addition, the objects that children mould when using the clay varies by sex – boys make cars, cows and donkeys, while girls make houses and ‘future partners’ they will marry. Overall, facilitators tended to allow mixing among girls and boys during their playtime activities. However, one facilitator remarked that some

13

parents do not want them to mix girls and boys during sessions. Children with disabilities were noticeably absent from the ECD centres we visited, with few exceptions. In the Juba CFS (slide 38), we observed two children with visual and physical disabilities; they were able to use the play materials on their own, but often were provided support to use the toys from an adult or adolescent helper (slide 44-45). In Senegal, we noted children with visual impairments, but could not determine if or how these children played with the toys. Another important point to raise is the lack of understanding among facilitators on the age appropriateness of the toys. For example, facilitators often mixed children of different ages when playing with the materials, even if the toys were designed for younger or older children.

Children’s Early Learning – Consistent with the ECD Kit Global Assessment Report, early learning was mentioned as one of the most significant changes that facilitators and parents observed in children after their exposure to the toys and learning materials. In both countries, the most frequently reported early learning outcomes (mostly for older children) included: 1) counting 1 to 10 using the toys; 2) identification of, and vocabulary for colours and shapes from the puzzles and shape sorters; 3) identification and replication of objects from the environment; 4) identification of animals from the blocks, puppets and storybooks; and 5) improved creativity and imagination through drawing, moulding clay and block construction. In the Juba PoC, several older children could identify

shapes in English and parents reported children who attended the ECD centres were more prepared for primary school than those who did not attend.

Children’s Psychosocial Well-being - Consistent with the ECD Kit Global Assessment Report, changes in children’s psychosocial well-being as a result of the ECD kits were reported in both countries. The most frequently reported outcome by parents, facilitators and children was that ‘children feel happy/happier’ when using the toys. In South Sudan, parents and facilitators also indicated that children’s stories, drawings and games changed from war and violence to peaceful and positive representations. Facilitators noted improvements in children’s self-regulation; they were found crying less and more able to regulate their emotions, especially among children who suffered severe loss, such as the death of their parents.

Children’s Development - Consistent with the ECD Kit Global Assessment Report, changes in children’s development as a result of the ECD kits were reported in both countries. The most frequently reported outcomes by facilitators and parents were: 1) children’s improved fine motor skills from manipulating the toys; 2) improved prosocial skills, such as friendship formation through group games, and children’s sharing and coordination of the toys; and 3) more inclusive and supportive child-to-child relationships, such as teaching other children how to use the toys, and including children with disabilities in playtime activities.

Reported Caregiver Outcomes Associated with the ECD Kits

Caring and supportive relationships – Adult facilitators of the ECD kits reported that they improved their capacity to care for and support children in both countries. For example, in South Sudan, one facilitator remarked that she developed love for the children while playing with the toys, and a handful reported that they continue to play with the children near their own homes because it makes them feel happy and loved.

“I have learned how to build a house using the blocks.” “I have learned to make a cow from the clay.”

Children, Batil Refugee Camp, South Sudan

“After the crisis the children had more fighting on their minds, now after the ECD kits have been used, they changed their mind and are less aggressive.”

“They learn how to socialize and stop fighting, peaceful conflict resolution.”

“Children are more responsive, for example, before they might cry because they see a toy and now they think they are interesting.”

“Before they started the program they would make drawings about war, but now after the ECD kits they are drawing flowers, family - now you can see them drawing the toys.”

“Before these play materials, our children used to talk about war, playing horrible games like shooting games and moulding different types of war weapons using clay, but now they tell us about the play stories at school and talk about their other friends at school. This is a very big change compared to the time when they came from Blue Nile and children only talk of war, guns and killings.”

“They spend most of their time playing with the materials in the school here. Some of these children lost both of their parents. Playing with this materials helps them to forget bad memories about the death of their parents.”

Comments made by facilitators, various ECD centres, South Sudan

14

Some reported becoming friendlier with children and are more in tune with children’s needs, or overall improvements in their child-adult relationships.

Improved ECD facilitation skills – Facilitators reported becoming more aware of the importance of play for child development in both countries, such as learning the psychology of the child and their emotional regulations. Other facilitators reported becoming better lesson planners, how to be more flexible during lessons, and that they “think about planning play activities all the time now.” One NGO partner also noted that facilitator treatment of children with disabilities had improved while playing with the toys, and this in turn had an impact on the way children treated each other.

Changes in parent attitudes towards ECD centres – In South Sudan, where we had more interactions with the parents, there were positive changes in their attitudes about the ECD centres and their activities. For example, parents remarked that the program is “very important” and “good” for their children, that they “noticed the benefits of bringing the children to the centre.” Some of these benefits include the fact that playing at the centres “keeps children good” or “out of trouble,” such as “stealing from the market.” One parent noted, “Children no longer go and run around the streets now that they are confined to the centre.” Other parents see the benefits of bringing the children to the centre because it “frees up their time and helps them cope with stress,” and addresses their need for childcare. In Senegal, parents clearly value the ECD centres, as the MoE cannot keep up with the demand from communities to provide adequate service coverage.

Enhanced social cohesion – In South Sudan, NGO partners reported that they see that ECD kit as a tool for peace, because caregivers have the opportunity to have more interactions with each other; the kit is a means for peacebuilding through interactions between different community members. For example, in the IDP communities, the parents used to be alone, but now there is “good interaction between the parents within the IDP camps and between parents in the host communities – you see the way they chat with each other.”

Unanticipated Outcomes Associated with the ECD Kits

Improved children’s attendance in ECD programs – In Senegal and South Sudan, partners, teachers and facilitators reported an increase in children’s attendance in the ECD centres as a result of seeing and using the ECD kit toys and games. However, this information could not be validated because no attendance records are kept. In Senegal, the toys were attractive to young children, helping them to cope with leaving their parents at home and transitioning into a school-like environment. Parents in South Sudan reported that their children give them “a hard time because they want to go to the centre and play.” In some of the ECD centres in Maban, facilitators reported that if there are no games or toys to play with, the children become bored and go home.

Inappropriate uses of the ECD kit materials – In South Sudan, the UNICEF CO reported that the box was used to store bullets during the conflict. NGO partners also reported that some teachers steal the toys or they misuse the materials (such as the soap) for their own purposes.

Potential loss of traditional forms of play – In South Sudan, two facilitators reported that the local clay used before to mould objects from the environment is now far less used. With the distribution of the clay in the ECD kits, it consequently discouraged the use of local material, as it is less convenient than the moulding material of the kits (because it makes children dirty and needs to be obtained and made on site).

Conclusion

The ECD kit impact evaluation in Senegal and South Sudan is technically and operationally feasible. However, a different impact evaluation design is recommended for South Sudan given the complexity and challenges of this setting. A comparative case study impact evaluation design is recommended because it allows for the analysis of multiple delivery models of the ECD kits, mitigates risks associated with the potential mobility among the population due to a resurgence in conflict, and allows for longitudinal analysis of qualitative and quantitative impacts. In Senegal, a quasi-experimental design with a control group is possible, and the necessary buy-in for this approach has

“We have observed that when the

facilitators treat the children with

disabilities (such as autism) the same

as other children with the kits, we can

see that their peers also treat them

better and they feel more included,

they feel they are treated equally.”

NGO Partner, Batil Refugee Camp, South Sudan

15

been obtained from the MoE. Careful consideration should be given to the recommended dosage for the interventions that will be evaluated (and how to increase it with better training and more kits if needed), and how to isolate the unique effects of the ECD kits for the evaluation.


Do not distribute photographs; they are provided for illustrative purposes only for the ECD kit evaluation

Senegal

1 2

3 4

5 6



7 8

9 10

11 12



13 14

15 16

17 18



19 20

21 22

23 24



25 26

27 28

29 30



31 32

33 34

35 36



South Sudan

37 38

39 40

41 42



4344

45 46

47 48



49 50

51 52

53 54



55 56

57 58

59 60



61

62

62

6363 64

65 66

Documents

Annex C: ECD Kit Mission Report - Home page | UNICEF · 1 Mission Objectives 1. Consult with relevant stakeholders to plan for the evaluation activities (e.g., evaluation governance,