FINAL REPORT I

© UNICEF

II FINAL REPORT

© UNICEF

FINAL REPORT This report was written by Universalia Management Group. The report represents the views of the consultant and should not be attributed to the UNICEF or to any other organization.

FINAL REPORT III

© UNICEF

Independent Evaluation of UNICEF Nigeria Training Investments

Final Report © United Nations Children’s Fund, Nigeria, 2017. United Nations Children’s Fund Plot 617/618, UN House, Diplomatic Zone, Central Area District Abuja, Nigeria

UNICEF’s Nigeria Country Office manage evaluation and evaluative studies on a wide range of topics for the purpose of contributing to learning about what makes for effective development, as well as supporting accountability for results in Nigeria. These evaluations aim at identifying what works and what does not in terms of achieving sustainable and equitable development results, and to throw light on how and why interventions succeed or not under various circumstances. In assessing UNICEF support to Nigeria government at Federal and State level and other development partners, these evaluations consider where, how and why progress is being made and the difference it is making in the lives of children, women in Nigeria. By publishing evaluation reports, the UNICEF Nigeria Country Office makes evaluation findings, lessons and conclusion available to a wide audience. Lessons learned from evaluation are expected to inform operational improvements and, ultimately, to support the achievement of better results. The publication of evaluation reports also supports accountability, by providing all interested parties with independently determined evidence relating to UNICEF’s performance. This provides a basis for informed dialogue and discussion, and helps to assure all stakeholders that the organization operates in an open and transparent manner. The content of this report do not necessarily reflect the policies or views of UNICEF. The text has not been edited to official publication standards and UNICEF accepts responsibility for error. The designations in this publication do not imply an opinion on the legal status of any country or territory, or of its authorities, or the delimitation of frontiers. The copyright for this report is held by the United Nations Children’s Fund – Nigeria Office. Permission is required to reprint/reproduce/photocopy or in any other way to cite or quote from this report in written form. UNICEF has a formal permission policy that requires a written request to be submitted. For non-commercial uses, the permission will normally be granted free of charge. Please write to the UNICEF Nigeria Country Office at the address below to initiate a permission request.

For further information, please contact: UNICEF Nigeria Country Office United Nations Children’s Fund Plot 617/618, UN House, Diplomatic Zone, Central Area District Abuja, Nigeria djobin@unicef.org

IV FINAL REPORT

© UNICEF

Forword

UNICEF Nigeria is a learning organization, at the heart of the work we do, is ensure that, at the round of Country Program Document (CPD) 2018-2021 development, we address and take lessons as well as incorporate these into next programmme. Since training interventions are a major component of our work in Nigeria, the Country Office commissioned this evaluation to inform discussions on the next CPD. The findings show that, as an organization, we have invested considerably in Nigeria for the delivery of trainings to facilitate the implementation of UNICEF’S Country Programme in Nigeria. Trainings which are delivered by UNICEF in Nigeria have been appreciated by partners and participants for the relevance and quality of the trainings. However, the planning process is not part of an overarching country strategy, which leads to a diversity of trainings that runs the risk of dispersal. The absence of an explicit Theory of Change and the lack of a strong results measurement framework also do not permit adequate monitoring and learning. Some recommendations are; 1) UNICEF in Nigeria should develop a comprehensive country training strategy to design trainings proactively, as opposed to focusing on individual punctual training activities that support specific aspects of programme implementation; 2) UNICEF in Nigeria should further monitor and evaluate its trainings by adopting monitoring and evaluation mechanisms and developing M&E instruments, guidelines and templates for training activities that systematically collect data on all training activities, expenditures and results and ; 3) UNICEF in Nigeria should ensure appropriate data management through the establishment and management of a database on trainings. I wish to express my personal thanks to all sections within UNICEF Nigeria and especially to the Monitoring and Evaluation team for managing this evaluation and supporting Universalia Management Group to give an in-depth insight that is well presented in this report on the effectiveness of the UNICEF Nigeria training investment.

FINAL REPORT V

© UNICEF

Acronyms CD Capacity Development

CPD Country Programme Document

DAC Development Assistance Committee

DSA Daily Subsistence Allowance

FAO Food and Agriculture Organization

FGD Focus Group Discussion

FO Field Office

GM General Manager

HACT Harmonized Approach to Cash Transfers

IMEP Integrated Monitoring and Evaluation Plan

KM Knowledge Management

LGA Local Government Area

M&E Monitoring and Evaluation

NCO Nigeria Country Office

NGO Non-Governmental Organization

OECD Organization for Economic Cooperation and Development

RAM Results Assessment Modules

RBM Results Based Management

RWP Rolling Work Plan

ToC Theory of Change

TORs Terms of Reference

ToT Training of Trainers

UMG Universalia Management Group

UNDAF United Nations Development Assistance Framework

UNDAP United Nations Development Assistance Programme

UNDG United Nations Development Group

UNESCO United Nations Educational, Scientific and Cultural Organization

UNICEF United Nations Children’s Fund

WASH Water, Sanitation and Hygiene

WASHCOM Water, Sanitation and Hygiene Committee

WASHIMS WASH Information Management System

FINAL REPORT VII

© UNICEF

Contents

EXECUTIVE SUMMARY ................................................................................................. 1

1 INTRODUCTION ..................................................................................................... 5

1.1 Evaluation Objectives .................................................................................................................... 5

1.2 Evaluation Context ........................................................................................................................ 6

2 METHODOLOGY .................................................................................................... 8

2.1 Evaluation Phases .......................................................................................................................... 8

2.2 Approach to Data Collection ......................................................................................................... 8

2.3 Theoretical Approaches .............................................................................................................. 12

2.4 Methodological Limitations......................................................................................................... 12

3 EVALUATION FINDINGS ....................................................................................... 14

3.1 Evaluability Assessment .............................................................................................................. 14

3.2 Relevance .................................................................................................................................... 17

3.3 Effectiveness................................................................................................................................ 22

3.4 Efficiency ..................................................................................................................................... 39

3.5 Sustainability ............................................................................................................................... 40

3.6 Cross-Cutting issues .................................................................................................................... 41

4 CONCLUSION AND RECOMMENDATIONS ............................................................. 42

4.1 Conclusion ................................................................................................................................... 42

4.2 Recommendations ...................................................................................................................... 42

VIII FINAL REPORT

© UNICEF

Figures

Figure 2.1 Phases of the Evaluation .......................................................................................................... 8

Figure 2.2 Mapping of consulted UNICEF Offices ..................................................................................... 9

Figure 2.3 Number of respondents disaggregated by sector and region ............................................... 11

Figure 2.4 Respondents’ participation in a UNICEF training between 2014 and May 2016 ................... 11

Figure 2.5 Respondents’ participation in a UNICEF training before 2014 .............................................. 11

Figure 2.6 Gender of respondents .......................................................................................................... 11

Figure 2.7 Occupation of respondents at the time of the training ......................................................... 11

Figure 3.1 Participants’ needs assessment prior to the training ............................................................ 19

Figure 3.2 Respondents’ needs assessment ........................................................................................... 20

Figure 3.3 Training response to respondents’ needs .............................................................................. 20

Figure 3.4 Relevance to participants’ work/ main occupation ............................................................... 21

Figure 3.5 Training’s response to respondents needs to do their job better ......................................... 21

Figure 3.6 Respondents’ selection to attend the UNICEF training ......................................................... 21

Figure 3.7 Clarity of Respondents about the purpose of the training prior to their attendance ........... 21

Figure 3.8 Reasons why respondents attended the UNICEF training ..................................................... 22

Figure 3.9 Quality of presentation and lecturing .................................................................................... 26

Figure 3.10 Usefulness of training material during the training sessions ................................................. 27

Figure 3.11 Usefulness of training material after the training sessions ................................................... 27

Figure 3.12 Extent to which objectives of the training were clearly outlined at the start of the session 31

Figure 3.13 Respondents level of engagement was during the training .................................................. 31

Figure 3.14 Participants’ perception of whether the training number of participants was optimal to ensure everyone’s participation ............................................................................................ 32

Figure 3.15 Participants’ perception of whether the training was worth the time invested ................... 32

Figure 3.16 Participants’ perception of whether the training was satisfactory ....................................... 32

Figure 3.17 Duration of trainings attended .............................................................................................. 32

Figure 3.18 Participants’ perception that they were given adequate opportunity to practice learning .. 33

Figure 3.19 The extent to which content of the training met respondents needs ................................... 33

Figure 3.20 Respondents perceptions whether they acquired the knowledge, skills and capacities they had expected .......................................................................................................................... 34

Figure 3.21 Participants’ overall feeling that respondents’ capacities have increased ............................ 34

Figure 3.22 Feeling of commitment by respondents to apply what they learned in their work / life ..... 34

Figure 3.23 Use of newly acquired knowledge/skill by respondent ......................................................... 36

Figure 3.24 UNICEF training changes to respondents’ ways of doing things ........................................... 36

Figure 3.25 Respondents’ perception of adequacy of support from their organizations to apply newly acquired knowledge ............................................................................................................... 36

Figure 3.26 Respondents organizations’ openness to changes suggested as a result of participation in the training ............................................................................................................................. 36

FINAL REPORT IX

© UNICEF

Figure 3.27 Use of training learning to the benefit of participants’ communities ................................... 38

Figure 3.28 Use of learning to the use of participants’ work .................................................................... 38

Figure 3.29 Respondents formally train other people on the acquired skills / knowledge ...................... 41

Figure 3.30 Respondents share informally what the acquired skills / knowledge with other people ..... 41

Figure iv.1 Five Key UNDG Steps for Capacity Development .................................................................. 52

X FINAL REPORT

© UNICEF

Tables

Table 2.1 Key stakeholders consulted and approach to data collection ............................................... 10

Table 2.2 Methodological challenges and mitigation strategies ........................................................... 12

Table 3.1 Compliance of the actual ToT model with international practices ........................................ 24

Table 3.2 Availability of Information: Rick Davies Checklist .................................................................. 29

Table 3.3 Availability of M&E data on trainings at the field level, aggregated by sector...................... 30

Table 3.4 Changes in training participants’ behaviour captured in the survey ..................................... 37

Table 4.1 Table of recommendations .................................................................................................... 43

Table v.1 Interviews conducted during the inception phase ................................................................ 54

Table v.2 Interviews conducted during data collection: Training Planners/Implementers interviewed ................................................................................................................................................ 54

Table vi.1 Basic Information on the Training Activity ............................................................................. 57

Table vi.2 Persons to Consult ................................................................................................................. 58

Table vi.3 Documents to Consult ............................................................................................................ 58

Table vi.4 Key Questions by Phases of a Training Activity ...................................................................... 59

Appendices

Appendix I List of Findings ............................................................................................................ 44

Appendix II Evaluation Matrix ....................................................................................................... 45

Appendix III Roles and Responsibilities of Team Members .......................................................... 50

Appendix IV Theoretical Frameworks ........................................................................................... 52

Appendix V List of Persons Consulted .......................................................................................... 54

Appendix VI Training Evaluation Template ................................................................................... 57

Appendix VII Data Collection Tools ............................................................................................... 60

Appendix VIII Bibliography ............................................................................................................ 77

Appendix IX List of Trainings ......................................................................................................... 92

Appendix X Table of Changes to the Original Evaluation Questions in the TORs......................... 97

Appendix XI The Training Cycle: From Planning to Result .......................................................... 100

Appendix XII Terms of Reference................................................................................................ 101

FINAL REPORT 1

© UNICEF

Executive Summary This evaluation of UNICEF Nigeria’s training investments was commissioned by the UNICEF Nigeria’s country monitoring and evaluation unit to determine the extent to which individual training activities conducted by UNICEF in Nigeria have been relevant, effective, efficient and sustainable in supporting the implementation of the Country Programme Documents (CPD) 2014-2017.

This evaluation covers Investments by UNICEF Nigeria into ‘Individual Training Activities’ between January 2014 and May 2016 in the five CPD ‘sub-components’ (also called ‘sections’ or ‘sectors’) of Health, Nutrition, Education, Child Protection, and WASH.

The objectives of the evaluation are:

▪ To strengthen UNICEF Nigeria’s capacity development strategy with clear recommendation on how to improve future trainings and its implementation;

▪ To provide information on the quality of trainings provided by UNICEF: content, delivery, timeliness, felt need, etc.; and

▪ To contribute to knowledge management and organizational learning, by providing the evidence to support best practices on trainings.

The evaluation adopted mixed methods for the data collection and was conducted in six phases:

▪ Launch call (August 2016),

▪ Evaluability Assessment (September 2016)

▪ Inception mission to UNICEF Nigeria’s offices in Abuja and visits to six field offices to establish data sources (September 2016)

▪ Data collection in eight States in Nigeria (November 2016-January 2017)

▪ Data analysis and report writing (February-April 2017)

▪ Validation of findings and submission of final report (April 2017)

Evaluation findings were mixed in terms of relevance. Beyond the various notable efforts to meet the national training needs, UNICEF Nigeria’s Training Investments were not designed as a coherent programme and lack an explicit theory of change. In the absence of a national capacity gap analysis, trainings are scattered, which undermines relevance of the portfolio of training investments as a whole.

Also on relevance, UNICEF Nigeria’s individual training activities respond to a certain extent to the needs of participants with opportunities for some possible improvements. In that sense, surveyed training beneficiaries were overall satisfied with the training response to their needs and to their jobs. However, the current training needs identification and participant selection processes give no insurance as to how targeted its portfolio of trainings is in terms of selection of beneficiaries.

In terms of effectiveness, the evaluation the evaluation noted the absence of a comprehensive national training strategy where trainings are strategically planned to respond to national training needs. Also, the evaluation observed the absence of a robust results framework for its trainings, in order to assess whether trainings have been delivered properly, and to whom. In the absence of a country training strategy and results framework, the overall effects of the training portfolio’s impact are jeopardized and run the risk of dilution of potential results.

2 FINAL REPORT

© UNICEF

From a design perspective, all sectors except the Child Protection sector adopt the same approach of delivering trainings through Training of Trainers (ToT). As ToT activities, the effectiveness of the training activities is mixed. Training participants revealed that respondents were satisfied with the quality of the technical trainings received. However, in the actual model, trained trainers are expected to return to their respective State to develop and deliver training courses to support implementation. As it stands, UNICEF Nigeria’s individual training activities limit themselves to delivering a solid technical component. The trainings do not include any modules on training process, or how to deliver trainings, nor does it provide training participants with the opportunity to practice their teaching skills prior to delivering trainings. Indeed, at the moment, the training is based on the assumption that participants will be able to deliver training after attending a technical workshop. This assumption could be true in some cases, but needs to be verified.

In addition, the current training design does not have a quality assurance mechanism, which represents a risk to training effectiveness. It does not proceed to any qualification test or ‘certification’ process of its trainees to identify the extent to which they can deliver trainings effectively, nor does it provide additional learning opportunities on the subject matter for the trainees to update their knowledge. In that sense, cascaded trainings run the risk of quality degradation.

In terms of monitoring and evaluation, some efforts are made to monitor training participants’ learning at the field level. However, the monitoring of participants’ reaction, learning, changes in behaviour and impact is not systematic. One sector was found to be pro-active in monitoring of trainings: WASH was noted for the availability of monitoring data during the Evaluability Assessment and individual interviews.

Based on all data collected and analyzed training activities, there is a high potential for effectiveness in terms of reaction and changes in behaviour. Survey respondents found the training to be favorable and engaging where they acquired useful knowledge. The evaluation also collected positive feedback of the stories of changes in participants’ behaviour, mainly in terms of personal confidence, technical work, and work with community, planning, M&E and reporting, management and training/class delivery.

In terms of efficiency, the evaluation concluded that the ToT approach is a contributing factor to efficiency. However, the very limited and incomplete data on the UNICEF training activities seriously undermines UNICEF’s ability to ensure that trainings are performing best. According to self-reported evidence, the potential for the sustainability of training results is strong. There is however some room for improvement, especially in terms of quality assurance to ensure that cascading trainings remain of quality. Trainings sustainability was also observed to be dependent on Programme delivery. If a Programme stops, its training components end.

In terms of gender, on the dimensions examined in this evaluation, there is not enough evidence or indications as to whether the gender component was taken into account in the training design, nor whether specific actions were taken in order to ensure equal opportunities for women and men to participate in the trainings, in terms of access, participation and community engagement. This evaluation did not analyze the content of the training to assess whether the gender component was taken into consideration.

In conclusion, a considerable effort is put forth by UNICEF in Nigeria for the delivery of trainings to facilitate the implementation of UNICEF’S Country Programme in Nigeria. Trainings which are delivered by UNICEF in Nigeria are appreciated by partners and participants for the relevance and quality of the trainings. However, the planning process is not part of an overarching country strategy, which leads to a diversity of trainings that runs the risk of dispersal. The absence of an explicit Theory of Change and the lack of a strong results measurement framework also do not permit adequate monitoring and learning.

FINAL REPORT 3

© UNICEF

While some efforts are made to monitor training participants’ learning at the field level, the monitoring and reporting on trainings is not systematic, accompanied by the absence of a centralized knowledge management system.

From a training design perspective, the training of trainers can be a good model if appropriately designed. Survey of past participants revealed positive effects in terms of reaction, learning, changes in behaviour and knowledge transfer. However, the evaluation observed that the current design might affect effectiveness, being based on a false assumption that delivering technical trainings without training on training process would allow participants to become effective trainers. Follow-up and quality assurance mechanisms are also subject to improvement, to reduce risks to efficiency and sustainability.

The evaluation team developed a total of three prioritized and sequenced recommendations (outlined below) that are based on its findings and conclusions. These recommendations are the following:

▪ Recommendation 1: UNICEF in Nigeria should develop a comprehensive country training strategy to design trainings proactively, as opposed to focusing on individual punctual training activities that support specific aspects of programme implementation.

▪ Recommendation 2: UNICEF in Nigeria should further monitor and evaluate its trainings by adopting monitoring and evaluation mechanisms and developing M&E instruments, guidelines and templates for training activities that systematically collect data on all training activities, expenditures and results.

▪ Recommendation 3: UNICEF in Nigeria should ensure appropriate data management through the establishment and management of a database on trainings

FINAL REPORT 5

© UNICEF

1 Introduction 1.1 Evaluation Objectives

1. The purpose of this evaluation is to determine the extent to which individual training activities conducted by UNICEF in Nigeria have been relevant, effective, efficient and sustainable in supporting the implementation of the Country Programme Documents (CPD) 2014-2017. To the extent that the expected findings contribute towards enhancing UNICEF’s readiness to reliably measure, monitor, evaluate and report on progress, these also contribute towards UNICEF’s organizational and development effectiveness. This is a formative evaluation whose objective is to inform future work carried out by UNICEF in Nigeria. Primary users of this evaluation include UNICEF Nigeria’s Deputy Representative, Field Offices, programme managers and UNICEF Nigeria country monitoring and evaluation unit.

2. Specifically, and as cited in the TORs, the objectives of the evaluation are:

▪ To strengthen UNICEF Nigeria’s capacity development strategy with clear recommendation on how to improve future trainings and its implementation;

▪ To provide information on the quality of trainings provided by UNICEF: content, delivery, timeliness, felt need, etc.; and

▪ To contribute to knowledge management and organizational learning, by providing the evidence to support best practices on trainings.

3. This evaluation covers Investments by UNICEF Nigeria into ‘Individual Training Activities’ between January 2014 and May 2016 in the five CPD ‘sub-components’ (also called ‘sections’ or ‘sectors’) of Health, Nutrition, Education, Child Protection, and WASH.

4. This is an evaluation led by and for the UNICEF Nigeria Country Office (NCO) as a whole. Although the evaluation recognizes important differences between CPD sections and took these into account throughout the assignment, the scope of this evaluation does not include a full section-by-section review of training activities, but rather an overall assessment of UNICEF NCO training investments.

Terminology

5. Below, are defined key terms as understood in the context of this evaluation:

▪ By Capacity, it means “the numbers of people (staff) and resources within an organization for the performance of specific functions, as well as the capability and ability of the staff and resources to perform those functions.” (cf. TORs, p.3). It is worth noting that the Capacity to carry out a task does not necessarily imply the willingness, practice, habit or behaviour of actually doing it.

▪ By Capacity Development (CD), it means increasing the ability of individuals, communities or organizations to execute certain tasks. CD can either communicate the same skills to a wider set of actors, or instruct a given set of actors on how to better deploy and employ their existing skills; for instance, by modifying the institutional, societal and environmental context.

▪ By Individual Capacity Development, it means ‘training,’ that is, to develop, through specialized instruction and practice, the skills and abilities of individuals to carry out certain tasks.

6 FINAL REPORT

© UNICEF

▪ ‘Training,’ ‘Training Session,’ ‘Training Activity,’ and ‘Individual Capacity Development’ are henceforth used interchangeably.

▪ The Country Programme Document’s (CPD) sub-components (e.g. ‘Nutrition’) are referred to as ‘sectors,’ and the UNICEF teams responsible for managing these are referred to as ‘sections.’

6. The present report presents evaluation findings that respond to questions presented in the TORs, as presented in the Evaluation Matrix (see Appendix II).

Structure of the Report

7. The report is organized into four sections:

▪ Section 1 – Introduction, including evaluation objectives and evaluation context

▪ Section 2 – Methodology, including evaluation phases, approach to data collection, theoretical approaches and methodological limitations

▪ Section 3 – Evaluation Findings on Evaluability, Relevance, Effectiveness, Efficiency, Impact and Sustainability of the Project

▪ Section 4 – Conclusions and Recommendations.

8. The appendices contain additional documents, including the terms of reference, the evaluation matrix, the list of stakeholders consulted, the list of documents reviewed, the evaluation frameworks and the data collection tools.

1.2 Evaluation Context

9. Based on conversations and documents reviewed, it is understood that the rationale for this evaluation is grounded in the following considerations:

▪ From the perspective of the UNICEF Nigeria Country Office (NCO), training investments are substantial and cut across most (CPD) sectors, but have never been evaluated as such. The Monitoring and Evaluation (M&E) unit of the NCO would like to better understand the relevance, effectiveness, efficiency, sustainability, and impact of these training investments, for both accountability and learning purposes, and especially in view of the development of the next CPD.

▪ From the perspective of middle-income countries, Capacity Development is becoming the primary and preferred contribution of international partners, including UNICEF (cf. the UNICEF Programme, Policy and Procedure Manual on “Programming in Middle-Income Countries”, pp.61-62). Middle-income countries like the Federal Republic of Nigeria aspire to autonomously provide for the rights and needs of their citizens, and are working with partners to identify and address capacity gaps.1

▪ From a global perspective, Capacity Development is increasingly becoming a hallmark of development programming, and is aligned with the Paris Declaration on Aid Effectiveness, as well as with the 2030 Agenda for Sustainable Development. At the level of UNICEF, Capacity Development is one of the seven “implementation strategies” of the UNICEF global Strategic Plan

1 As of July 2016, the World Bank classifies Nigeria as a “Lower middle income” country. See siteresources.worldbank.org/DATASTATISTICS/Resources/CLASS.XLS, accessed September 21st, 2016.

FINAL REPORT 7

© UNICEF

2014-2017 (p.11); at the level of the United Nations Development Group, CD is one of the five “principles” of Country Programming.2 Thus, understanding whether CD works and how it could be improved is of high importance for future UNICEF programming in Nigeria, and elsewhere.

2 Cf. UNDG Guidance Note: Application of the Programming Principles to the UNDAF.

8 FINAL REPORT

© UNICEF

2 Methodology 2.1 Evaluation Phases

10. The evaluation included five main phases, as presented in Figure 2.1 below.

Figure 2.1 Phases of the Evaluation

11. Following (a) two launch conference calls with UNICEF Nigeria’s M&E division (August 2016); (b) an assessment of information available and missing, presented in the Evaluability Assessment (September 2016); (c) a three-day inception mission to UNICEF Nigeria’s offices in Abuja, including phone conversations with field offices (September 2016); (d) visits to six field offices to establish data sources and data collection in eight States in Nigeria3 (November 2016-January 2017); (e) data analysis was conducted and followed by report writing (February-April 2017). (f) A debrief presentation was conducted in UNICEF Nigeria country office to validate the evaluation’s findings and conclusions and collect stakeholders’ perceptions and feedback. The report was then revised to its final version (April 2017).

12. This evaluation is aligned with the obligations of evaluators relating to independence, impartiality, credibility, conflicts of interest, and accountability.

13. The following section outlines the approach adopted to complete this assignment.

2.2 Approach to Data Collection

14. The data-collection phase constitutes the core of the work for this evaluation. In order to gather information on training investments, four data-collection strategies were adopted: document review, interviews, focus group discussions, and a questionnaire survey.

15. The first – document review – involved desk-based work, whereas the three remaining strategies involved site visits.

Sites Sampling

16. Key “sites” for this evaluation are 10 of UNICEF Nigeria’s Field Offices,4 from which the majority of training investments are managed. Sampling of the Field Offices included seven States.

17. At the inception and evaluability assessment phases, sampling was limited to six Field Offices hoping that training activities in the five sectors under review will be covered. However, during the first field visit to establish the data sources, the evaluation team discovered that there were States with high training concentration. States sampling was therefore modified to include the following 10 States:

3 (Appendix V lists the persons consulted) 4 Field Offices are sometimes called ‘Zonal Offices’ in UNICEF Nigeria documentation. We use these interchangeably.

InceptionEvaluabilityAssessment

Data Collection

Data Analysis

Validation of Findings

Reporting

FINAL REPORT 9

© UNICEF

▪ Abuja, where the NCO is located, where the team conducted high-level interviews and focus group discussions (FGDs) with senior management.

▪ Enugu, Lagos5, Bauchi, Kaduna, Sokoto, Katsina, and Akure, where the team conducted interviews and FGDs with staff, training participants and beneficiaries, and administered survey questionnaires to training participants.

▪ Rivers and Borno were not included in the data collection 6for two reasons: in the case of Rivers because of its proximity to Enugu FO, and in the case of Borno due to the ongoing security situation in the area.

▪ Niger and Anambra, which are not Field Offices, where the team conducted interviews and FGDs with training participants and beneficiaries, and administered survey questionnaires to training participants.

18. The following map (Figure 2.2) presents the sample of the visited States.

Figure 2.2 Mapping of consulted UNICEF Offices

Desk Review

19. The desk review served two purposes. First, a review of general, strategic, and guidance documentation helped further understand and refine the context of UNICEF’s training investments, and in particular (a) their relevance in the face of capacity gaps in Nigeria; (b) their alignment with UNICEF Headquarters and United Nations System Capacity Development Guidance; and (c) their alignment with NCO Country Programme Strategies. This included documents such as UNDG Capacity Development Guidelines, UNICEF’s Global Strategic Plan, UNICEF’s Programme Policy and Procedure Manual, the Nigeria UNDAF and UNDAP, the NCO Country Programme Document, NCO Rolling Work Plans and Rolling Management Plans, IMEP data, RAM data, and so forth. The full bibliography consulted is presented in Appendix VII. Second, the desk review provided key information with regards to the evaluation of individual training activities.

5 The Lagos FO has recently been turned into an NCO Zonal Office, and all of its former FO responsibilities have been transferred to the new Akure FO. Interviews/FGD for Lagos were conducted at the Akure FO except for child protection, where the interview, FGD and questionnaire survey were done in Lagos. 6 The two remaining ‘new’ Field Offices (Rivers and Borno) were not included for two reasons: in the case of Rivers because of its proximity to Enugu FO, and in the case of Borno due to the ongoing security situation in the area.

10 FINAL REPORT

© UNICEF

Individual interviews

20. Individual interviews were used for stakeholders who were likely to have substantive strategic or operational insights on training investments (in general, or related to a specific training activity), such as UNICEF Programme Specialists, Chiefs of Field Offices, Sector Chiefs, and Government Officials. Interviews were conducted in a semi-structured fashion, on the basis of questions which aimed to contribute information to key evaluation questions. A sample interview protocol is presented in Appendix VII.

Survey Questionnaire & Focus Group Discussions

21. Survey questionnaires and focus group discussions primarily targeted training participants and training beneficiaries.

a. The first category includes persons who actually attended training sessions organized, facilitated or financed by UNICEF Nigeria; these include, depending on the kind of trainings, various government officials, civil servants, service providers (such as teachers and nurses), third parties (e.g. journalists, NGO representatives), and community members, workers, volunteers and leaders.

b. The second category includes stakeholders who are expected to benefit from the conduct of the training: beyond participants themselves, this includes managers of capacitated institutions, recipients of capacitated services, and other relevant affected organizations and partners.

22. In order to administer questionnaires and conduct focus group discussions, training participants and beneficiaries were sampled prior to being reached and successfully consulted. In order to address the absence of a definitive list of training (including by FO), the absence of complete training participant and beneficiary lists, and the unsure ability to contact respondents, a first round of field visits were conducted to establish data sources for training participants from which respondents were randomly selected for sampling.

23. Table 2.1 below summarizes the general outlay of data collection.

Table 2.1 Key stakeholders consulted and approach to data collection CATEGORY OF

STAKEHOLDERS KEY PERSONS MAIN APPROACH

Planners of Training

UNICEF staff: section chief, section specialists (including at FO level), M&E specialists, financial specialists Government officials at national / state / LGA level

Interviews

Designers of Training

UNICEF section specialists, M&E specialists, financial specialists Relevant experts (UNICEF, partners, consultants, contractors)

Interviews

Deliverers of Training

UNICEF section specialists Relevant experts (UNICEF, partners, consultants, contractors)

Interviews

Recipients of Training

Training participants from government, partners, communities (both ToT and Step-down, if applicable)

Survey Questionnaires (Paper and Focus Group Discussions)

Beneficiaries of Training Programme

Training participants Managers of capacitated institutions Affected citizens, communities Affected organizations, partners

Survey Questionnaires (Paper) Focus Group Discussions

24. Five focus groups discussions were held with average of 5-7 participants, 34 respondents were interviewed (26 UNICEF Staff and 8 government officials). 506 respondents completed the survey. The

FINAL REPORT 11

© UNICEF

following Figures (Figure 2.3 to Figure 2.7) outlay the Descriptive Statistics of the survey respondents’ sample. Figure 2.3 Number of respondents disaggregated by sector and region

Figure 2.4 Respondents’ participation in a UNICEF training between 2014 and May 2016

Figure 2.5 Respondents’ participation in a UNICEF training before 2014

Figure 2.6 Gender of respondents Figure 2.7 Occupation of respondents at the time of the training

42

2

40

0

28

149

75

88

82

0

Health

Nutrition

WASH

Education

Child Protection

South North

11%

15%

27%

6%

41%

2014

2015

2016

Do notremember

Between 2014to 2016

25%

42%

25%

8%Yes, one

Yes, Several

No, Never

Not sure/ Donot remember

67%

33%

Male

Female

86%

3%7%

1%3%

Employed inpublic sector

Employed inprivate sector

Self-employed

Unemployed

In school/studying

12 FINAL REPORT

© UNICEF

2.3 Theoretical Approaches

25. The adopted approach to assessing UNICEF Nigeria’s training investments is underpinned by the following approaches and standards: ▪ OECD-DAC criteria7 to assess the relevance, effectiveness, efficiency and sustainability of the NCO’s

training investments; ▪ United Nations Development Group’s Guidance (UNDG) on Capacity Development (CD)8, defining

Partner Engagement, Needs Assessment, Design of CD Strategy, Implementation of CD Activities, and Evaluation of CD Activities as the five main steps to CD;

▪ Kirkpatrick Model9 for evaluating the effectiveness of training, building on four consecutive levels of training ‘results’: reaction, knowledge and skills, change in behaviour and results; and

▪ Cross-Cutting Issues: In addition to these three frameworks, the evaluation also followed four cross-cutting considerations in its approach: differences by programme section, differences by type of training, and gender and equity were taken into consideration.

26. For further information on each of the frameworks, please refer to Appendix IV. These frameworks are all integrated into the Evaluation Matrix (see Appendix II), and reflected in data-collection tools (see Appendix VII). 27.

2.4 Methodological Limitations 28. The following key observations, highlighted in Table 2.2. are helpful to understand how the evaluation team approached this evaluation.

Table 2.2 Methodological challenges and mitigation strategies

OBSERVATIONS IMPLICATION FOR THE EVALUATION METHODOLOGY

There are no overarching objectives for training investments (or, these cannot be differentiated from the overarching objectives of the CPD as a whole).

Therefore, many evaluation findings relied on an aggregation of findings related to individual training activities. UNICEF’s training investments are evaluated as a set of activities, rather than as one coherent programme.

Lack of data to triangulate findings on effectiveness and to assess efficiency Available documentation does not provide any indication of the training targets (e.g., expected number of trainings to be delivered, number of training participants, duration of trainings, participants, etc.). Also, information on the nature, type, and results of delivered training activities is not collected systematically.

In the absence of planned explicit targets and monitoring data, effectiveness of trainings cannot be evaluated, including the extent to which planned training activities were delivered or the extent to which the expected outputs were reached (number of trainings, number of training participants, etc.). Also, survey results regarding effectiveness in terms of learning, changes in behaviour and results cannot be triangulated.

7 The DAC Principles for the Evaluation of Development Assistance, OECD (1991), Glossary of Terms Used in Evaluation, in ‘Methods and Procedures in Aid Evaluation’, OECD (1986), and the Glossary of Evaluation and Results Based Management (RBM) Terms, OECD (2000).

8 These Capacity Components are described in the UNDG Capacity Assessment User Guide, Feb 2008: “The eight capacity components are: human resources; public sector accountability; access to information, development knowledge and technology; inclusion, participation, equity and empowerment; financial resources; material resources; environmental resources; and external/international relations” (p.5) 9 For a basic introduction to the Kirkpatrick model, and for the source of the Kirkpatrick sub-questions in this table, see http://www.kirkpatrickpartners.com/OurPhilosophy/TheKirkpatrickModel (accessed September 22nd, 2016).

FINAL REPORT 13

© UNICEF

OBSERVATIONS IMPLICATION FOR THE EVALUATION METHODOLOGY

In order to mitigate this challenge, the evaluation took note of self-reported data and did not take its reliability for granted. Whenever possible, the evaluation team used its own judgement and information collected through interview and focus group discussions, to filter the information received.

Lack of data to assess efficiency Financial data on trainings is unavailable.

Cost efficiency could not be assessed due to the lack of financial data.

14 FINAL REPORT

© UNICEF

3 Evaluation Findings

3.1 Evaluability Assessment

Finding 1: UNICEF Nigeria’s Training Investments were not designed as a coherent Programme and lack a theory of change. These shortcomings are problematic and allow limited opportunities for making an informed judgment on the portfolio overall

Training investments were not planned as a Programme

29. The first step in determining the evaluability of an intervention or set of investments is to clearly delineate its scope. During the evaluation exercise, it was observed that Capacity Development (and its ‘individual-level’ counterpart, training) has not been planned as an integrated or cross-cutting project or Programme. Instead, every Section requests training as it relates to the implementation of a particular intervention where training is almost exclusively conducted at the activity level. This impacted the way the evaluation was conducted, where many evaluation findings relied on an aggregation of findings regarding individual training activities. This has implications on the feasibility for UNICEF’s training investments to be evaluated as one coherent Programme, rather than a set of activities.

Absence of a comprehensive database of training activities

30. It was also observed that a comprehensive database for training activities conducted does not exist. UNICEF was not able to provide a specific or an estimate of the number of training activities that have been conducted in the period under review (2014–May 2016).

31. During the evaluability assessment and data collection phases, the evaluation team attempted to reconstruct a list of training activities conducted in Nigeria, based on document review (Work Plans and Annual Reports), interviews and survey results. (Please refer to the full list of trainings captured in Appendix IX).

32. The evaluation included in this universe of training all individual capacity development activities (i.e., ‘training’) listed in the 2014-2015 or 2015-2016 Rolling Work Plans of the five sectors named in the TORs: Health, Education, Nutrition, Child Protection, and WASH.

33. Although Rolling Work Plans (RWP) were used as a basis for mapping out the universe of training for this evaluation, they are not ideal sources of information for the following reasons: (a) the format and information provided in RWPs varies from sector-to-sector, and from year-to-year even within a sector; (b) RWPs often describe ‘training’ indirectly, instead of referring to general ‘capacity development’ or ‘technical assistance’; (c) RWPs often group together various activities under one heading. Consequently, it was not always possible to identify with precision what kinds of training were planned, in terms of objectives, types of participants, numbers, region, and budget.

34. To mitigate information gaps, the evaluation cross-referenced the list of training drawn from RWPs with three further sources: (a) survey results; (b) training reported in the Annual NCO Reports for 2014 and 2015. However, the narrative style of annual reports is not systematic in the information it reports about training.; and (c) other training documents shared by the Evaluation Unit for the purpose of this evaluation.

FINAL REPORT 15

© UNICEF

35. This evaluation, in particular:

▪ Includes training of various types, sizes, geographical coverage, partners and objectives.

▪ Includes training conducted between January 2014 and May 31st, 2016 (as specified by the TORs); excludes training conducted before or after these dates.

▪ Includes training on emergency operations where these training are classified by the Country Programme Document as activities under the five aforementioned sectors; but excludes training on emergency operations conducted under Outcome 13 (Emergency) of the CPD.

▪ Excludes UNICEF internal training (such as UNICEF Staff training).

▪ Excludes outreach, C4D, advocacy and awareness-raising activities which do not involve a sustained teaching component, with well-defined materials, trainers, and participants. For instance, ‘Global Handwashing Day’ and similar activities are not considered “training” for the purposes of this evaluation; although these activities share certain aspects with “training” (educational message, targeted behaviour change), they do not offer the active support and lessons usually provided in training.

Absence of a Theory of Change

36. The evaluation was able to confirm that there was no overarching causal chain that connects UNICEF training investments with intended results, other than their incorporation as ‘activities’ in the CPD, and corresponding sectional RWPs.

37. This evaluation attempted to draft a Theory of Change (ToC) that would reflect the current understanding of training interventions based on documents and interviews carried out.

38. The proposed ToC is based on Kirkpatrick’s model to evaluate training. Its four levels are shown on the left-hand side, e.g., Level 1 – Reaction, Level 2 – Learning, etc. For each level, a result was formulated that should be expected from UNICEF training investments. Results for the first three levels were drawn from the evaluation understanding of the Kirkpatrick model and UNICEF’s interventions, while the results for the fourth level are based on the UNICEF’s capacity development strategy.10 The final impact or goal to be achieved was extracted from the UNICEF Nigeria Country Programme Document.

39. Whenever relevant, the evaluation identified assumptions linked to results and also included examples of external influences which may apply to any of the four levels described above.

40. This draft ToC should be discussed internally, refined and finalized by the UNICEF Nigeria CO. The evaluation made a conscious decision not to include additional, embedded ToCs in order to avoid too much complexity (e.g., an embedded ToC for the training of trainers).

10 “Capacity development aims to (a) strengthen delivery of supplies and services; (b) develop the enabling environment needed to guarantee key rights, promote behaviour change and demand for services; and (c) strengthen the ability of rights-holders to claim key rights.” (p.11, UNICEF Strategic Plan 2014-2017).

16 FINAL REPORT

© UNICEF

Exhibit 3.1 Draft Theory of Change

FINAL REPORT 17

© UNICEF

3.2 Relevance

41. The evaluation adopted the OECD DAC’s Glossary of Key Terms in Evaluation and Results-Based Management to define Relevance as being “The extent to which the objectives of a development intervention are consistent with beneficiaries’ requirements, country needs, global priorities and partners’ and donors’ policies.”11

42. Overall, UNICEF Nigeria’s individual training activities are seen as relevant by training attendees. However, their relevance of the overall portfolio, in the absence of country training strategy and country capacity gaps assessments, is questionable.

Finding 2: Beyond the various notable efforts to meet the national training needs, in the absence of a national capacity gap analysis, trainings are scattered, which undermines relevance of the portfolio of training investments.

43. There have been various notable efforts to meet the national training needs, with the objective of facilitating the implementation of the UNICEF Nigeria Country Programme.

44. Capacity Development (and training) is a priority in UNICEF’s (2014-2017) strategic plan where “Capacity development at individual, community and government levels has always been one of the main UNICEF implementation strategies. Support to individual and community capacity development is often provided through ‘communication for development’ to strengthen the capacity of government at all levels, UNICEF focuses on increasing capacity to address gaps of rights-holders and duty-bearers. Measures include training and technical assistance for reforms of government institutions; piloting models for scaling up; and using national and local systems and processes to strengthen capacities and empower children,

11 Glossary of Evaluation and Results Based Management (RBM) Terms, OECD (2000) Page 32.

18 FINAL REPORT

© UNICEF

families and communities. Strengthening management systems, results-based management and planning capacities are key components of a comprehensive approach to capacity development.”12

45. UNICEF Nigeria’s trainings are also aligned with UNICEF’s priorities, where capacity development is mentioned in the Country Programme (2014-2017) as being one of the four interlinked strategies to reduce disparities and reach the most disadvantaged children: “Developing technical capacities at State and LGA levels to coordinate the delivery of quality social services”13. In addition, all UNICEF training activities in Nigeria are conducted as part of programme delivery which makes them fit for purpose. In terms of alignment with national strategic orientations, Nigeria produced in December 2009 a national development strategy entitled Vision 20:2020, where “investing in human capacity development” is a key target to support the Vision 20:2020 aspirations. UNICEF’s Country Programme is aligned with Vision 20:2020 national priorities: “The (UNICEF) Country Programme was developed through close collaboration with the Federal Government of Nigeria, led by the National Planning Commission. Its preparation was synchronized with the United Nations Development Assistance Framework (UNDAF-III) 2014-2017, for which the United Nations country team chose the One Programme modality. The preparation was guided by Vision 20:2020.14”

46. In order to identify training needs, UNICEF engages in a consultative process with its implementing partners. UNICEF consults national institutions and agencies to plan, design and deliver its training programmes. Prior to training planning and delivery, each sector systematically agrees with the State implementing partners on specific training that would be required to deliver a particular programme.

47. The process through which training is programmed can be described as follows15: If a particular intervention requires training to be delivered, then a draft proposal is submitted by an implementing partner to UNICEF for approval. If the proposal is approved by UNICEF, training planning is conducted by the latter through a wide consultation with the relevant stakeholders (at the national, state and ward levels) in order to define the training objectives and targeted participants.

48. In addition to the process of engaging partners described above, specific training needs are then identified using one of the following approaches:

▪ Consultation with relevant stakeholders: Meetings/workshops with national stakeholders are held to agree on what training is needed in each sector.

▪ Observation of errors made by the health workers in the execution of their duties.

▪ Pilot surveys: some trainings are informed by findings and recommendations of pilot surveys conducted by UNICEF to collect internal and/or partner’s staff training needs. It is a trend common to all sectors but with greater emphasis within sectors like WASH, Health, and Nutrition.

▪ Changes in global standards: some trainings are driven by changes in concepts, technology, international protocols, global standards, concepts and strategies. This is very common in the health and nutrition sector.

12 The UNICEF Strategic Plan, 2014-2017. United Nations Children’s Fund, Executive Board Second regular session 2013. 3-6 September 2013. Item 4 of the provisional agenda. E/ICEF/2013/21. Pages 11-12. 13 UNICEF, 2013. Nigeria Country Programme Document 2014-2017. Page 8 14 UNICEF Nigeria Country Programme Document 2014-2017. Page 8.

15 All the sectors except the Child Protection follow the same approach to developing and delivering training activities. The Child Protection sector mostly works primarily at the state level unlike the other four sectors that has national coverage.

FINAL REPORT 19

© UNICEF

49. According to survey respondents, 65% indicated that they were consulted on their training needs prior to the training (Figure 3.1).

50. These consultative processes allow UNICEF to successfully engage with partners, to ensure that trainings offered respond to their needs for the implementation of programmes.

Figure 3.1 Participants’ needs assessment prior to the training

51. These training activities supported by UNICEF sectors can be relevant in developing or strengthening the capacity of State professional staff to deliver and to implement its programmes.

52. However, UNICEF does not conduct systematic capacity gaps analysis at the country level to identify priorities in terms of training needs, targeted participants and regions. As a result, the design and delivery of individual trainings by UNICEF in Nigeria has been reactive rather than proactive. Trainings run the risk of dispersion which undermines the relevance, effectiveness and efficiency of the training portfolio as a whole.

Finding 3: UNICEF Nigeria’s individual training activities respond to a certain extent to the needs of participants with opportunities for some possible improvements.

53. Surveyed training beneficiaries were overall satisfied with the training response to their needs. Indeed, 65% of survey respondents indicated that they were consulted on their training needs prior to the

training (Figure 3.2) and 85% felt that the training they participated in responded to needs they had before the training (Figure 3.3).

54. Respondents also found the training to be relevant to their jobs where 92% found that the UNICEF training they participated in was relevant to their work/main occupation (Figure 3.4) and 94 % found that the UNICEF training they participated in was needed to better do their job (Figure 3.5).

55. However, in the absence of M&E data to triangulate this self-reported data, there is limited indication on the extent to which the trainings actually helped beneficiaries better fulfill their role in supporting programme implementation and enhanced the partner organization’s performance overall.

56. Moreover, in terms of participants’ selection, according to UNICEF staff interviewed, the nomination of participants is generally done by State Implementing Partners. Survey results indicate that 53% of respondents were selected by their supervisor/local authority to attend the training while 38%

12%

15%

40%

26%

7%

I was consulted on my training needs prior to the training

Strongly disagree

Disagree

Agree

Strongly agree

Don't know

20 FINAL REPORT

© UNICEF

were selected by UNICEF (Figure 3.6). Due to the absence of data, the evaluation has not assessed the extent to which participants were qualified to attend the trainings.

57. As a result, the evaluation team observed that the current selection process does not allow UNICEF to ensure that training beneficiaries are strategically selected. The survey results show that 89% of participants were clear on why they were selected to attend the training. However, only 70% were clear about the purpose of the training before they attended, while 23% were not clear about its purpose (Figure 3.7). Survey results also show that while 56% indicated that the main reason that drove them to actually attend the UNICEF training was career enhancement, only 20% attended for their interest in the subject matter (Figure 3.8). In that sense, the relevance of the training recipients’ selection can be questionable, especially since trainings are not delivered by UNICEF for career enhancement purposes.

58. Consequently, the current training needs identification and participant selection processes give no insurance as to how strategic UNICEF is in responding to the training needs and how relevant and targeted its portfolio of trainings is in terms of selection of beneficiaries.

Figure 3.2 Respondents’ needs assessment Figure 3.3 Training response to respondents’ needs

12%

15%

40%

26%

7%

I was consulted on my training needs prior to the training

Stronglydisagree

Disagree

Agree

Strongly agree

Don't know

2%

7%

41%45%

5%

The UNICEF training I participated in responded to

needs I had before the training

Stronglydisagree

Disagree

Agree

Strongly agree

Don't know

FINAL REPORT 21

© UNICEF

Figure 3.4 Relevance to participants’ work/ main occupation

Figure 3.5 Training’s response to respondents needs to do their job better

Figure 3.6 Respondents’ selection to attend the UNICEF training

Figure 3.7 Clarity of Respondents about the purpose of the training prior to their attendance

2%

4%

37%

56%

1%

The UNICEF training I participated in was relevant to

my work/ main occupation

Stronglydisagree

Disagree

Agree

Strongly agree

Don't know

3%

2%

32%

62%

1%

The UNICEF training I participated in was needed to do my job better

Stronglydisagree

Disagree

Agree

Strongly agree

Don't know

1%

53%38%

1% 7%

Please indicate how you were selected to attend the UNICEF training

I self-enrolled

My supervisor/ local authority told me to attendthe trainingUNICEF invited me to participate in the training

I replaced someone else who could not attend

I don't know

7%

16%

41%

29%

7%

I was clear about the purpose of the training before I attended

Strongly disagree Disagree

Agree Strongly agree

Don't know

22 FINAL REPORT

© UNICEF

Figure 3.8 Reasons why respondents attended the UNICEF training

3.3 Effectiveness

59. We adopt the OECD DAC’s Glossary of Key Terms in Evaluation and Results-Based Management to define effectiveness as being “The extent to which the development intervention’s objectives were achieved, or are expected to be achieved, taking into account their relative importance.” 16

60. In this section, effectiveness is examined at two levels. First, we examine the extent to which the overall portfolio of UNICEF training is effective, namely the extent to which it has a clear strategy, is underpinned by a clear Theory of Change and includes appropriate feedback mechanisms allowing UNICEF decision-makers to make appropriate changes to the nature of the portfolio. Second, we review the effectiveness of the individual training activities, focusing specifically on training design, training effects (with the caveat that such effects are self-reported), and the existence of appropriate M&E systems.

16 Glossary of Evaluation and Results Based Management (RBM) Terms, OECD (2000). Page 22.

56%

2%

20%

20%

2%

Please indicate the main reason that drove you to actually attend the UNICEF training

Career enhancement

Networking

Had to attend because I was sent by my supervisor

Interest in the subject matter

Possibility of obtaining a certificate

FINAL REPORT 23

© UNICEF

Reviewing the actual quality of the training content is beyond the scope of this exercise and was not explored beyond anecdotal evidence provided by participants.

61. Overall, as will be described in the section, while the effectiveness of the training activities leads to some positive and less positive findings, the effectiveness of the portfolio as a whole is subject to improvement.

Finding 4: In the absence of a country training strategy and results framework, the overall effects of the training portfolio’s impact are jeopardized.

62. During inception and data-collection mission consultations, the evaluation noted the absence of a comprehensive national training strategy where trainings are strategically planned to respond to national training needs.

63. In that sense, the current approach to training design and delivery leads to a large diversity of ad-hoc uncoordinated trainings. For example, the list of trainings that the evaluation team captured through document review, interviews and survey results comprises a large diversity of training themes17.

64. A comprehensive national training strategy outlines the following elements18:

▪ The overall strategic training objectives and the underlying Theory of Change;

▪ The process by which UNICEF’s training is identified, and prioritized;

▪ Tactics and actions required for the development, budgeting, and delivery of trainings;

▪ A comprehensive results framework;

▪ The process by which training standards are developed;

▪ Follow-up and monitoring mechanisms to ensure training quality; and

▪ The evaluation feedback loop to inform and report on the training performance overall.

65. Also, UNICEF Nigeria Country Programme’s monitoring and evaluation (M&E) system does not allow nor require the monitoring of training activities. The M&E Unit of UNICEF monitors the various sectors performance through independent monitoring of programme components; each section is responsible to monitor its own programme including the progress and result of its activities. Each sector is expected to submit an annual report for Results Assessment Modules (RAM) and Milestone Data for outputs. However, such reports are highly aggregated in the Annual Report and do not report on individual training activities.

66. Thus, in the absence of a robust results framework for its trainings, UNICEF is not assessing whether trainings have been delivered properly, and to whom. It does not have information on the outcomes of the trainings, and is not in a position to emit an informed judgment on the effectiveness of its training portfolio. Actually, this very evaluation is the first effort to perform such an assessment.

67. In the absence of national training capacity gaps assessments, strategic planning and prioritization of trainings, a comprehensive results framework, and feedback and learning loops, the trainings portfolio runs the risk of dilution of potential results.

17 Please refer to Appendix IX to read the full list of training themes captured through data collection

18 FAO, 2001. Asia-Pacific Forestry Commission - Regional training strategy: supporting the implementation of the code of practice for forest harvesting in Asia-Pacific - Food and Agricultural Organization of the United Nations, Regional Office for Asia and the Pacific. Bangkok, Thailand. (And) UNESCO (unknown year). Training of Trainers in Teacher Education for a Sustained Quality Education - External Evaluation Report.

24 FINAL REPORT

© UNICEF

Finding 5: From a design perspective and as Training of Trainer activities (ToT), the effectiveness of the training activities is mixed.

68. All sectors except the Child Protection sector adopt the same approach of delivering trainings through Training of Trainers (ToT). The training approach to Child Protection is somewhat different given the high technicality and the coverage of the programme: it mostly works at the state level, unlike the other four sectors that have national coverage. It consists mostly of direct delivery of training to those who need it to perform their duties, such as police officers, judges, juvenile courts, lawyers and child welfare officers.

69. To achieve an optimal multiplier effect, the actual model relies on the training of trainers, where trained trainers are expected to return to their respective State to develop and deliver training courses to support implementation. In that sense, trainings cascade down to the State and sometimes to the ward levels.

70. Benchmarking on international practices in terms of ToT design, Training of Trainers' design includes (or at least partially includes) the following features:19: 1) engaging with partners and conducting needs assessments; 2) including a focus on both content (in the case of UNICEF, this refers to the technical content) and on process (for instance, all notions related to group dynamics, principles of adult learning); 3) having a quality assurance system as the training cascades down; and 4) measuring results at four levels of the Kirkpatrick chain of training results.

71. Based on the evaluation's analysis, Table 3.1 summarizes the extent to which the design of UNICEF training complies with these features.

72. While Engagement with partners and needs assessment are covered in Finding 2, each of the following features is further discussed in subsequent findings (See Findings 6 to 8).

Table 3.1 Compliance of the actual ToT model with international practices

TRAINING OF TRAINERS DESIGN COMPLIANCE

Engagement with partners and needs assessment

Balance between technical training and training process (andragogy)

Quality assurance system

Measuring results at four levels

Legend: Not compliant; Somehow compliant; Compliant

19 Jean C. Murphy, Ed. D. and Carol O. Carson-Warner, Ed. D. (unknown date). Train-the-Trainer Manual - Chicago State University, Chicago, IL. And http://www.fao.org/docrep/W8088E/w8088e03.htm page consulted February 20th 2017. And UNDG Capacity Assessment User Guide, Feb 2008.

FINAL REPORT 25

© UNICEF

Finding 6: The trainings do not balance technical training with delivery process. Training activities are designed with the dual purpose of transferring technical knowledge to participants and to enable them to, in turn, transfer their knowledge to others. From a design perspective, the technical content of the training is of good quality, but insufficient time is dedicated to building participants' training skills.

73. A ToT design should include a balanced transfer of learning on content (technical) and on process (andragogy).20 While the content component of a training allows participants to understand and master the technical material, the process component intends to prepare trainees (and trainers-to-be) to become effective trainers, as to the method and practice of teaching. According to survey results, the technical training is strong, whereas no attention is paid to enhancing the skills of participants in terms of training delivery.

Technical training

74. The review of the quality of the training technical modules was beyond the scope of this evaluation. As such, evidence provided as to the quality of the training relies solely on feedback obtained from participants.

75. This being said, training participants revealed that respondents were satisfied with the quality of the technical trainings received. Survey data reveals that 97% of respondents found the quality of the

20 http://www.fao.org/docrep/X5665E/x5665e04.htm - page consulted February 20th 2017.

26 FINAL REPORT

© UNICEF

presentation and lecturing to be high, and agreed the content was presented in a way that was easy to understand21 (Figure 3.9).

76. Interview respondents also indicated that training materials, such as training guidelines, training manual existed–assertion verified by the evaluation team–and were distributed at the onset of each training session.

77. Ninety-six percent of survey respondents also confirmed the strong relevance of the training materials which they felt useful to support their learning. Similarly, 91% of respondents confirmed that training materials were adapted to their needs and prior knowledge (Figure 3.10) and 96% of respondents indicated that they still refer to the training materials as a useful source of information (Figure 3.11).

78. However, some concerns were shared regarding the level of complexity and timeliness of training guidelines and materials. Some of the training materials were reported to be complex and difficult to teach at the community level, which might affect the quality of trainings delivery by trained trainees at the state and ward levels. In that sense, the survey results revealed that 31 respondents recommended that improvements be made on training materials.

Figure 3.9 Quality of presentation and lecturing

21 Data disaggregation indicates that results do not vary significantly between the different sectors.

1%1%

40%

57%

1%

The quality of the presentation and lecturing was high and content was presented in a way that was easy to understand

Strongly disagree

Disagree

Agree

Strongly agree

Don't know

FINAL REPORT 27

© UNICEF

Figure 3.10 Usefulness of training material during the training sessions

Figure 3.11 Usefulness of training material after the training sessions

Training on Training Process

79. The assumption that, by having been exposed to a technical workshop, any given participant will be able, in turn, to deliver the same workshop to another audience is a weak assumption. Therefore, effective ToT design complements the technical training modules with some notions of a training process. This can include a wide range of topics, such as training content design, classroom management, effective training delivery, group dynamics, providing feedback to participants, all in all, a series of themes that allow any given trainer to deliver his/her content effectively.

80. As it stands, UNICEF Nigeria’s individual training activities limit themselves to delivering a solid technical component. The trainings do not include any modules on training process, or how to deliver trainings, nor does it provide training participants with the opportunity to practice their teaching skills prior to delivering trainings. Indeed, at the moment, the training is based on the assumption that participants will be able to deliver training after attending a technical workshop. This assumption could be true in some cases, but needs to be verified.

Finding 7: The current training design does not have a quality assurance mechanism, which represents a risk to training effectiveness.

81. Once the training of trainers completed, quality assurance becomes an important step in the design and delivery of ToTs. Currently, there is no framework ensuring that cascaded trainings delivered through the ToT approach remain of quality.

Guidance of trained trainers

82. A field-based trainer (Supervisor) who is the primary liaison between UNICEF and the future trainer is necessary to facilitate the future trainer’s transition into a competent trainer. This includes evaluating the trainer’s competence, providing support and clarifications on contents to the trainee, and submitting observations and recommending the trainee for readiness to serve as a trainer.

83. UNICEF provides additional support to training participants; 82% of survey respondents indicated that UNICEF carried out follow-up activities (contact, further support) after the training. However, there

1%1%

40%

57%

1%

Training materials (handouts, documentation) were useful and

assisted my learning

Stronglydisagree

Disagree

Agree

Strongly agree

0%2%

43%

53%

2%

I still refer to the training materials as a useful source of

information

Stronglydisagree

Disagree

Agree

Stronglyagree

28 FINAL REPORT

© UNICEF

is no evidence of follow-ups in terms of evaluating the trainer’s competence and readiness to serve as a trainer.

Updating knowledge

84. According to adult learning theory, people who train others remember 90 percent of the material they teach22. In the absence of a mechanism where trainers’ knowledge is updated, the training quality can degrade after only a few repeated trainings (from the national level to the state and ward levels). In that sense, of survey respondents, 45% agreed with the statement: “I have an overall feeling that, unless more training is provided, I will not retain my new knowledge/capacities” and 48% disagreed. Therefore, additional refresher trainings could be necessary to ensure an adequate delivery of trainings. Also, if any changes are brought to the modules, due to changes to the global standards in the subject matter, updating knowledge ensures proper transition of trainers to the new modules.

85. The current ToT approach does not proceed to any qualification test or ‘certification’ process of its trainees to identify the extent to which they can deliver trainings effectively, nor does it provide additional learning opportunities on the subject matter for the trainees to update their knowledge. In that sense, cascaded trainings run the risk of quality degradation.

Finding 8: Some efforts are made to monitor training participants’ learning at the field level. However, the monitoring of participants’ reaction, learning, changes in behaviour and impact is not systematic.

86. The programme sections supporting the training do not have complete information on the nature, type, duration and outcomes of training activities that have been conducted from 2014 to June 2016 - the period being evaluated. In the eventuality where the Evaluation Unit requests information from different sectors on the training activities, the availability of data depends on the willingness of the leadership of each sector to provide information. Data, if available, remains concentrated at the Field-Office level but somewhat uncoordinated and unsystematic23.

87. The table below summarizes the availability of information by Rick Davies Checklist.

22 http://www.urban.org/research/publication/assessing-train-trainer-model-evaluation-data-democracy-ii-project - Page consulted February 20th 2017. 23 A new Harmonized Approach to Cash Transfer (HACT) was introduced in 2014. It requires the submission of an Activity Report as part of requirements for financial reporting. The new approach proved to be challenging, where nearly all the Activity Reports were mostly descriptive and did not include any analysis or any feedback on challenges met and lessons learned on training design and delivery.

FINAL REPORT 29

© UNICEF

Table 3.2 Availability of Information: Rick Davies Checklist

INFORMATION AVAILABILITY QUESTIONS

ANSWERS

Is a complete set of documents available?

No.

Documentation is insufficient for the evaluation of most training, thematic sectors, evaluation questions, and Kirkpatrick levels. Additionally, documentation that is available is partly inconsistent in content and format across thematic areas and time.

Do baseline measures exist?

No (for training)/Yes (for CPD outputs relevant to training activities).

The evaluation found a limited number of pre- and post-tests carried out before and after training. However, these are not available for all training interventions.

At the UNICEF CPD output-level, baseline information is usually available.

Is data being collected for all the indicators?

No.

There are no specific set of indicators for training interventions, which are considered ‘activities’ in UNICEF’s Work Plans. Only in certain sectors (WASH, nutrition, and education) do individual training activities have indicators attached to them (almost always in terms of number of persons to be trained). Training interventions are usually expected to more broadly contribute to output-level indicators (for which data is usually available).

Is participant, beneficiary, and stakeholder data available?

Not systematically.

The evaluation collected some participant lists for the vast majority of training activities. However, such information was not systematically available either at the UNICEF NCO or at the Field Office level.

Is gender disaggregated data available?

Not systematically.

It may be possible to disaggregate training participants by men and women as some training participants’ lists keep track of this information. Documentation does not provide such gender-disaggregated data consistently.

If reviews or evaluations have been carried out, are the reports available?

No

No formal evaluation has previously been carried out on training investments as a whole. Reports are usually written at the level of the individual training activity; however, as with other documentation, the evaluation did not have access to these reports for most training activities.

Do existing M&E systems have the capacity to deliver, and to collect presently missing data in the future? (budget, management, capacity)

No.

A major limiting factor in the context of this evaluation is that no M&E system was built for training as such. The PME relies only on the information provided by section chiefs to track performance. The information is highly aggregated and often lumped together with other intervention implementation. As such, a comprehensive data base for training does not exist.

88. Evidence collected through document review and interviews suggests that there is a discrepancy between sectors in the systematization and depth of monitoring of training activities. The following table presents a summary of the data on trainings currently collected by sectors at the Field-Office level.

30 FINAL REPORT

© UNICEF

Table 3.3 Availability of M&E data on trainings at the field level, aggregated by sector

KIRKPATRICK LEVELS HEALTH NUTRITION EDUCATION CHILD

PROTECTION WASH

Reaction

Knowledge and skills

Change in behaviour

Results

Legend: Not monitored; Not systematically monitored; Systematically monitored

89. The evaluation observed that there had been some effort, however not systematic, to test training participants’ learning at all sectors, where 88% of survey respondents indicated that they were tested on their knowledge at the start and/or at the end of the training. In cases where trainings delivered captured the list of trainees and their affiliations, the level of attainment of training objectives, lessons learned, and participants’ feedback remain unavailable in most of the documentation consulted.

90. As for the changes in behaviour and impact, one sector was found to be pro-active in monitoring of trainings: WASH was noted for the availability of monitoring data during the Evaluability Assessment and individual interviews. The WASH sector conducts its internal review, research, needs assessment, and impact assessment of programmes. This can be explained by two potential factors:

▪ Availability of resources: UNICEF-Nigeria WASH programme is one of the largest development programmes for UNICEF globally.24

▪ WASHCOMs: Within the WASH programme, “the normal process of engagement in a community includes the establishment of a WASH Committee (WASHCOM) and training of its members on their roles and responsibilities, record keeping and management of WASH facilities to ensure ownership and sustainability of interventions.25” The existence of these WASHCOMs can be a facilitating factor for the delivered training monitoring.

Finding 9: Based on all data collected and analyzed training activities, there is a high potential for effectiveness in terms of reaction and changes in behaviour. Results are difficult to assess due to the insufficiency of M&E data.

91. The evaluation of training activities’ achievements was based on the Kirkpatrick model where the effectiveness of training builds on four consecutive levels: reaction, knowledge and skills, change in behaviour and results.

92. According to survey responses received, training activities seem to be effective. In the absence of pre- and post-training monitoring data, the only source of data we have to evaluate effectiveness is the survey results, which consists of self-reported data that cannot be triangulated. As a result, the data presented in the following paragraphs should be read with great reserve.

24 Kannan Nadar, George Ubong; UNICEF-Nigeria, 2016. Case Study -2: WASH Committees drive birth registration and immunization of children in their communities in Benue & Jigawa States in Nigeria 25 Kannan Nadar, George Ubong; UNICEF-Nigeria, 2016. Case Study -2: WASH Committees drive birth registration and immunization of children in their communities in Benue & Jigawa States in Nigeria. Page 1.

FINAL REPORT 31

© UNICEF

Reaction

93. Survey respondents found the training to be favorable. In that sense, 95% of survey respondents indicated that the objective(s) of the training were clearly outlined at the start of the session (Figure 3.12).

94. Survey respondents also found the training to be engaging. 96% indicated they fully engaged with what was going on during the training (Figure 3.13) and 93% of them found the number of training participants to be optimal to ensure everyone’s participation (Figure 3.14).

95. Eighty percent indicated that the training was worth the time they put into it (Figure 3.15), and 97% indicated that, overall, they found the training to be satisfactory (Figure 3.16). An interview respondent stated that “most of the functions of an average health worker are dependent on the knowledge and skills from UNICEF-supported training.”

96. However, training participants’ appreciation of the training revealed their duration is mostly less than five days long (Figure 3.17). Several respondents during interviews and survey results suggested that the training duration is not sufficient. In that sense, in response to the open-ended question “Based on your experience, what are key suggestions for improving UNICEF’s training activities going forward?” a total of 74 respondents suggested increasing the number of days for training (26 from the Health sector, 11 from the Nutrition sector, 20 from the Wash sector, 14 from the Education sector, 3 from the Child protection sector).

Figure 3.12 Extent to which objectives of the training were clearly outlined at the start of the session

Figure 3.13 Respondents level of engagement was during the training

2%1%

42%

53%

2%

The objective(s) of the training were clearly outlined at the start of

the session

Stronglydisagree

Disagree

Agree

Strongly agree

1%

1%

42%

54%

2%

I fully engaged with what was going on during the training

Stronglydisagree

Disagree

Agree

Stronglyagree

Don't know

32 FINAL REPORT

© UNICEF

Figure 3.14 Participants’ perception of whether the training number of participants was optimal to ensure everyone’s participation

Figure 3.15 Participants’ perception of whether the training was worth the time invested

Figure 3.16 Participants’ perception of whether the training was satisfactory

Figure 3.17 Duration of trainings attended

Learning

97. Survey results show that participants had a good learning experience during the delivery of the training. 95% indicated that, during the training, they were given adequate opportunity to practice what they were learning (Figure 3.18). 97% indicated that, overall, the content of the training met their learning needs (Figure 3.19).

98. In addition, survey results suggest that the training participants acquired the intended knowledge, skills, attitude, confidence and commitment. 98% responded that, overall, they acquired the knowledge,

1%3%

53%

40%

3%

The number of training participants was optimal to ensure

everyone’s participation

Stronglydisagree

Disagree

Agree

Strongly agree

Don't know

3%

12%

40%

40%

5%

The training was worth the time I put into it

Stronglydisagree

Disagree

Agree

Stronglyagree

Don't know

1%0%

36%

61%

2%

Overall, I found the training to be satisfactory

Stronglydisagree

Disagree

Agree

Strongly agree

Don't know

25%

51%

14%

0%

10%

Duration of Training Attended

1- 2 days

3-5 days

6-10 days

11- 14 days

Do notremember

FINAL REPORT 33

© UNICEF

skills and capacities they had expected (Figure 3.20). And 96% indicated that they have an overall feeling that their capacities have increased (Figure 3.21).

▪ For example, one of the trainees interviewed stated that “the training has helped to increase my knowledge about what I should be doing with regard to issues concerning children, and it has inculcated the skills needed to handle child cases.”

99. 94% indicated that, after the training, they felt committed to applying what they learned in their work/life (Figure 3.22). And in a response to an open-ended question on what they do differently as a result of the skills, knowledge and capacities taught during the UNICEF training they participated in, 90 respondents’ responses included confidence and effectiveness in carrying out their job. Additionally, observation during interviews revealed that the knowledge of the subject matter by WASH trainees interviewed was an indication that the training instilled some level of confidence in the discharge of their duties.

Figure 3.18 Participants’ perception that they were given adequate opportunity to practice learning

Figure 3.19 The extent to which content of the training met respondents needs

0%4%

40%

55%

1%

During the training, I was given adequate opportunity to practice

what I was learning

Stronglydisagree

Disagree

Agree

Strongly agree

Don't know

1%

1%

44%

53%

1%

Overall, the content of the training met my learning needs

Stronglydisagree

Disagree

Agree

Strongly agree

Don't know

34 FINAL REPORT

© UNICEF

Figure 3.20 Respondents perceptions whether they acquired the knowledge, skills and capacities they had expected

Figure 3.21 Participants’ overall feeling that respondents’ capacities have increased

Figure 3.22 Feeling of commitment by respondents to apply what they learned in their work / life

Change in behaviour

100. In the absence of a robust set of monitoring data, following-up with training participants, following their participation in the training, it is not possible to systematically assess the extent to which changes in behaviour occurred and the effectiveness of the trainings in improving practices in the five sectors areas of work. However, survey results and interviews with respondents captured elements of proof and actual changes in behaviour reported by training participants.

0%

1%

36%

62%

1%

Overall, I acquired the knowledge, skills and capacities I had expected

Stronglydisagree

Disagree

Agree

Strongly agree

Don't know

0%

1%

34%

62%

3%

I have an overall feeling that my capacities have increased

Stronglydisagree

Disagree

Agree

Strongly agree

Don't know

2%2%

36%

58%

2%

After the training, I felt committed to applying what I learned in my

work/ in my life

Stronglydisagree

Disagree

Agree

Strongly agree

Don't know

FINAL REPORT 35

© UNICEF

101. The following are illustrative examples of the stories of changes in participants’ behaviour as a result of their participation in the training:

▪ In the words of a respondent “Seeing women attending Antenatal care and practicing exclusive breastfeeding as well as speaking about the relevance of the first milk was an indication of utilization of knowledge gained from the training.”

▪ According to another interviewee, “exclusive breast feeding was an issue in the (Gombe) State, and there was an increase among malnourished children. UNICEF supported training and sensitization on the importance of exclusive breast feeding has resulted in more men, traditional rulers and the governor of Gombe state advocating for breast feeding in public”. This statement is an indication of effective training and sensitization of the public.

▪ One of the high-level stakeholders from Kaduna State said “UNICEF liaised with the implementing partners and supported training based on our needs, which has enabled me to plan as a policy maker, to have a better picture of what is required and to convince other policy makers. It helps in my supervision too.”

▪ A participant at an FGD session in Kaduna State reported that training has helped to enhance the effectiveness and proficiency of service delivery, knowing that if service providers have a better understanding of what they do, they will be able to deliver appropriately.

▪ According to a Nutrition Officer interviewed from Enugu State “I cannot really quote the year but if you go to our nutrition indices you have it that in the South East our state is doing well. Prior to 2014, we were not doing so well especially within Nsukka Zone, Igbo-etitei to be precise; they usually get highest numbers of mal-nourished children. It was after the training at Igbo-etitei we now learnt how to measure nutrition status using the tape. We were able to record and track a very high number of children that were mal-nourished. With the used of Report at Site (RAS) in the place, they had to call me when they discovered two malnourished cases. We went to see the two cases, but we had six (6). The discovery of 4 additional cases can be attributed to the training”.

▪ Kachia community was one of the communities in Kaduna state where open defecation was a normal practice and most preferred method. At the very beginning, the process of finding people to work with in this community was a very big issue, but when the training started and the community was engaged, the willingness of the community members to participate in the programme continued to increase. Currently, all the toilets in this community are water closet and communities close to Kachia are requesting UNICEF for similar intervention with their communities.

102. As for the survey results, 97% of respondents indicated that, they use their new knowledge/skill on a daily basis (Figure 3.23). 85% confirmed they do things differently as a result of their participation in the UNICEF training (Figure 3.24) and 84% received adequate support from their organization after the training to apply their new knowledge (Figure 3.25). 88% added that the organization they work for is open to the changes they suggested as a result of their participation in the UNICEF training (Figure 3.26).

36 FINAL REPORT

© UNICEF

Figure 3.23 Use of newly acquired knowledge/skill by respondent

Figure 3.24 UNICEF training changes to respondents’ ways of doing things

Figure 3.25 Respondents’ perception of adequacy of support from their organizations to apply newly acquired knowledge

Figure 3.26 Respondents organizations’ openness to changes suggested as a result of participation in the training

103. Table 3.4 illustrates the responses of survey respondents to the open-ended question: “What I do differently as a result of the skills, knowledge and capacities taught during the UNICEF training I participated in” (only responses that accounted 10 or more respondents are presented in this table). Following a categorization of the responses (bolded rows in the table), it can be observed that survey respondents observed changes in their behaviour in terms of personal confidence (90 respondents),

2%

47%50%

1%

I use my new knowledge/skill on a daily basis

Disagree

Agree

Strongly agree

Don't know

4%

9%

42%

43%

2%

I do things differently as a result of my participation in the UNICEF

training

Stronglydisagree

Disagree

Agree

Strongly agree

Don't know

2%

11%

51%

33%

3%

I received adequate support from my organization after the training

to apply my new knowledge

Stronglydisagree

Disagree

Agree

Strongly agree

Don't know

1% 5%

57%

31%

6%

The organization I work for is open to the changes I suggested as a result of my participation in

the UNICEF training

Stronglydisagree

Disagree

Agree

Strongly agree

Don't know

FINAL REPORT 37

© UNICEF

technical work (176 respondents), in their work with community (127 respondents), their planning, M&E and reporting (95 respondents), management (69 respondents) and training/class delivery (62).

Table 3.4 Changes in training participants’ behaviour captured in the survey

SURVEY QUESTION: WHAT I DO DIFFERENTLY AS A RESULT OF THE SKILLS, KNOWLEDGE AND CAPACITIES TAUGHT DURING THE UNICEF TRAINING I PARTICIPATED IN.26

Categories (Occurrence)

Sub-categories (Occurrence)

Personal/general (134) Confidence and effectiveness in carrying out my job (90)

On job training (44)

Technical

(176)

Teaching people on personal hygiene (47)

Vaccine Management (26)

Appropriate position for breastfeeding (19)

Preventive maintenance of the cold chain equipment (12)

How to construct toilet and wash hand basin (10)

Work with Community

(127)

Community engagement (64)

Sensitization of communities (28)

Better way of relating with people (11)

Teacher community relationship (10)

Planning, M&E and Reporting

(95)

Monitoring and evaluation (27)

Data collection and data entry into WASHIMs is now more regular and up-to-date (20)

Proper record keeping and how to send data (10)

Proper record keeping of students (10)

Management

(69)

Management of personnel (21)

Improved communication skills (14)

Participatory approach to resolving issues and bottom-up decision-making process (13)

Supportive supervision (12)

Training delivery

(62)

Training facilitation (32)

Better method of teaching (15)

Classroom management and control (10)

Results

104. Due to the absence of a sound Theory of Change (ToC) for training investments that is commonly agreed upon within UNICEF in Nigeria, expectations in terms of results of trainings are unclear, and results attainment is hard to assess.

26 Only the Sub-categories that accounted 10 responses or more are presented in this table.

38 FINAL REPORT

© UNICEF

105. In the absence of systematic monitoring and evaluation data on results, we asked survey respondents questions about the use of their learning. 96% indicated that they were able to use what they learned in the training to the benefit of their community (Figure 3.27) and 96% indicated they were able to use what they learned to the benefit of their work (Figure 3.28). In the absence of additional data, the above-mentioned figures cannot be triangulated nor explored further.

Figure 3.27 Use of training learning to the benefit of participants’ communities

Figure 3.28 Use of learning to the use of participants’ work

On factors affecting effectiveness

106. Interviews revealed that two main challenges affect training effectiveness. In addition to the absence of the training process component in the ToT design and the lack of a follow-up mechanism as the training cascades down (Findings 6 and 7), training funding modality constitutes another factor to training effectiveness: while trainings funded fully by UNICEF were delivered as planned, those co-funded were not fully implemented, mainly due to the fact that funding counterparts did not provide the required funding and if provided, it may be inadequate or untimely.

107. The following are examples of trainings that were not fully delivered:

▪ According to a respondent in Akure in Ondo State, in the education sector, where the funding for Annual School Census to strengthen education management information system at the State level requires states to contribute 35% and UNICEF 65%, the training was not fully implemented in States that fail to contribute their share.

▪ In the WASH sector in Kaduna State, a project agreement expected all Local Government Agencies (LGAs) to contribute the sum of 250,000 Naira each for training monitoring activities. Most LGAs delayed the contribution, which hindered facilitation of such activities.

▪ Similarly, in health and nutrition that are implemented in all UNICEF states of intervention, planned trainings are implemented when UNICEF is the only funder.

0%

1%

35%

62%

2%

I was able to use what I learned in the training to benefit of my

community

Stronglydisagree

Disagree

Agree

Strongly agree

Don't know

1%2%

38%

58%

1%

I was able to use what I learned in training to the benefit of my work

Stronglydisagree

Disagree

Agree

Strongly agree

Don't know

FINAL REPORT 39

© UNICEF

3.4 Efficiency

108. In the absence of itemized costs of the training portfolio or of specific activities, the evaluation of the training efficiency is extremely limited.

109. OECD DAC Criteria’s definition of efficiency is “a measure of how economically resources/inputs are converted to results.”27 Efficiency of trainings can be assessed by the extent to which inputs (usually financial, human, technical and material resources) were converted to outputs in an economic way.

110. The current financial report system (HACT) does not contain information on how much was spent on individual training activities. This is because the training activities are not reported on their own merit, but as part of a wider programme. For example, there is no data on the total cost of a particular course, UNICEF spending on training materials, instructor fees and facility costs, UNICEF expenditure on travel and lodging, meals, administration and other miscellaneous expenses, etc.

111. In such a context, cost per student, partners’ contribution to cost, and more importantly, an assessment of return on investment is not possible to calculate. Nor is it possible to make a comparative analysis (with other countries or between sectors).

Finding 10: The ToT approach is a contributing factor to efficiency. However, the very limited and incomplete data on the UNICEF training activities seriously undermines UNICEF’s ability to ensure that trainings are performing best.

112. The evaluation team can only provide an independent judgment on the efficiency of the design. Indeed, training for trainer design, if well-conceived and well implemented is, in theory, an efficient approach to generating the multiplier effect. The evaluation’s appreciation of the quality of the design suggests that there is room for improvement. Nevertheless, with a training model that cascades to states, ward and community levels, with potential transfer of learning to a large audience, the ToT approach is an efficient training delivery approach.

113. However, with the information on trainings not centralized and the reporting format of trainings not harmonized, UNICEF is not in a position to compare its training activities and to assess their efficiency. As part of the Evaluability Assessment conducted during the early phases of this evaluation mandate, the evaluation team noted that UNICEF is not in a position to produce a list of all training courses delivered in the 2014 to June 2016 period. The document review also revealed that the reporting on training is uneven between the different sectors that are within the scope of this evaluation. It was noted that Rolling Work Plans (RWPs) reported unevenly on trainings. For example, the nutrition sector’s RWP (2016) reported on the number of people trained, while other sectors reported on capacity development at large.

114. The absence of a data management system, and therefore the very limited data available, limits the capacity of UNICEF to learn from past experiences and enhance the optimum performance of trainings. For example, it limits UNICEF’s capacity to avoid duplication of training and participants. Also, the inability to compare training costs, approaches and results reduces learning opportunities between sectors and within the organization overall.

27 Glossary of Evaluation and Results Based Management (RBM) Terms, OECD (2000) Page 21.

40 FINAL REPORT

© UNICEF

3.5 Sustainability

115. Sustainability is defined by OECD DAC Criteria as being “the continuation of benefits from a development intervention after major development assistance has been completed” and “The probability of continued long-term benefits. The resilience to risk of the net benefit flows over time.”28

116. The following assessment of sustainability covers the durability of knowledge acquired following the trainings’ delivery, and the continuity of training delivery beyond funding and programme completion.

Finding 11: According to self-reported evidence, the potential for the sustainability of training results is strong. There is however some room for improvement.

117. Beyond the training activities delivered by UNICEF, the ToT model is one that has a great potential for sustainability. This is because the trainees are perceived to acquire knowledge that is useful over the long term and transfer of knowledge to larger audiences is a strong contributor to training investments’ sustainability. In that sense, 95% of survey respondents indicated that they expect their new capacities (acquired through training) to remain relevant and useful for a prolonged period of time and 87% of survey respondents indicated that they have gone on to formally train other people on the skills/knowledge they acquired. Of whom, 23% have formally trained 1 to 10 persons, 35% trained 11 to 50 persons, 11% trained 51 to 100 persons, 6% trained 101 to 200 persons, and 24% trained 200 persons and above.

118. Also, 80% responded that they have gone on to informally share with other people the skills/knowledge they acquired, of whom, 35% indicated that they shared them with community members, 34% with professional colleagues and the other 30% includes GM & other RUWASSA officers, gas service providers, friends and neighbors, their household, health workers, and care givers.

119. Also, staff rotation does not seem to be a major threat to sustainability. In general, the majority (76%) of survey respondents are still in the same work position since they attended the training (Figure 3.29). However, some disparities can be observed between sectors. Most sectors have similar figures, ranging between 75% and 80%, except for the child protection sector where only 53% of survey respondents are still in the same position since they attended the UNICEF training (Figure 3.30).

120. Therefore, it can be observed that sustainability of the knowledge transfer is positive. However, inadequacy of some training design components (as mentioned in Finding 7 on quality assurance) where the absence of resources allocated to support transfer of training and conduct quality assurance checks limits UNICEF’s capacity to ensure sustainability of training quality.

121. In addition, the absence of a country training strategy is another threat to training sustainability, where the current trainings are financially dependent on UNICEF funding and programming. According to government officials and implementing partners interviewed, in the absence of UNICEF’s technical and financial support to the various programmes, the training would stop. Also, beyond funding, since trainings are part of programme delivery, if a programme is ended, its training components end too.

28 Glossary of Evaluation and Results Based Management (RBM) Terms, OECD (2000) Page 36.

FINAL REPORT 41

© UNICEF

Figure 3.29 Respondents formally train other people on the acquired skills / knowledge

Figure 3.30 Respondents share informally what the acquired skills / knowledge with other people

3.6 Cross-Cutting issues

122. In order to ensure flow of text, differences between sectors were highlighted throughout the report. This section focuses on the extent to which gender was included as a cross-cutting theme in training, more specifically, the extent to which gender considerations were taken into account in the design and delivery of trainings.

Finding 12: There is limited evidence that specific actions were adopted to ensure that women and men have equal opportunities.

123. This evaluation did not analyze the content of the training to assess whether the gender component was taken into consideration.

124. On the dimensions examined in this evaluation, mainly the level of design and participation, there is not enough evidence or indications as to whether the gender component was taken into account, and whether specific actions were taken in order to ensure equal opportunities for women and men to participate in the trainings, in terms of access, participation and community engagement. However, survey data reveals that 94% of respondents found that men and women were given equal opportunity to participate in the training. Disaggregated data does not reveal any disparities between women’s and men’s responses, where 96 % of women and 93% of men agreed to the statement “Men and women were given equal opportunity to participate in the training.”

At the design levels, we have not come across any documentation indicating that UNICEF in Nigeria did an assessment of women’s needs prior to training planning to ensure that women have equal opportunities to attend the trainings (security, babysitting, social factors, etc.). Interviewees’ responses indicate that most training activities did not take gender considerations into account when planning for training, during participants’ selection or during training delivery. In that sense, there is lack of evidence of specific actions undertaken by training planners and trainers in order to ensure that women’s specific needs are taken into account in the planning and delivery of trainings.

1%

9%

48%

39%

3%

I have gone on to formally train other people on the

skills/knowledge I acquired

Stronglydisagree

Disagree

Agree

Strongly agree

3%

13%

49%

31%

4%

I have gone on to informally share with other people the

skills/knowledge I acquired.

Stronglydisagree

Disagree

Agree

Strongly agree

42 FINAL REPORT

© UNICEF

4 Conclusion and Recommendations

4.1 Conclusion

125. A considerable effort is put forth by UNICEF in Nigeria for the delivery of trainings to facilitate the implementation of UNICEF’S Country Programme in Nigeria. Trainings which are delivered by UNICEF in Nigeria are appreciated by partners and participants for the relevance and quality of the trainings. However, the planning process is not part of an overarching country strategy, which leads to a diversity of trainings that runs the risk of dispersal. The absence of an explicit Theory of Change and the lack of a strong results measurement framework also do not permit adequate monitoring and learning. While some efforts are made to monitor training participants’ learning at the field level, the monitoring and reporting on trainings is not systematic, accompanied by the absence of a centralized knowledge management system.

126. From a training design perspective, the training of trainers can be a good model if appropriately designed. Survey of past participants revealed positive effects in terms of reaction, learning, changes in behaviour and knowledge transfer. However, the evaluation observed that the current design might affect effectiveness, being based on a false assumption that delivering technical trainings without training on training process would allow participants to become effective trainers. Follow-up and quality assurance mechanisms are also subject to improvement, to reduce risks to efficiency and sustainability.

4.2 Recommendations

127. The evaluation team developed a total of three prioritized and sequenced recommendations (outlined below) that are based on its findings and conclusions. These recommendations are actionable and are derived from the findings and conclusions of this evaluation. These recommendations have been validated during the presentation of findings on the Independent Evaluation of UNICEF Training Investment that was held in UNICEF office in Nigeria on 11th April 2017. Please refer to the Table 4.1 for recommendations, priority, timeframe and responsibility.

FINAL REPORT 43

© UNICEF

Table 4.1 Table of recommendations

RECOMMENDATION RELATED FINDINGS

PRIORITY, RESPONSIBLE, TIMEFRAME

Recommendation 1: UNICEF in Nigeria should develop a comprehensive country training strategy

UNICEF in Nigeria should develop a country training strategy to design trainings proactively, as opposed to focusing on individual punctual training activities that support specific aspects of programme implementation.

A comprehensive country training strategy identifies country level needs and capacity gaps, target clientele based on needs, sets out overall objectives and the underlying ToC, defines a comprehensive results framework (where gender is mainstreamed), identify risks and appropriate mitigation strategy in implementing the strategy. Based on the country training strategy, UNICEF Nigeria will be able to determine the range of training and training of trainers activities to be provided.

Based on: Finding 1, 2, 3, 4, 5,6, 7, 9, 11 and conclusion

Priority: Very high

Responsible: Deputy representative office in consultation with section chiefs, Zonal Offices, and key national stakeholders.

Timeframe: Starting immediately upon approval of this evaluation.

Recommendation 2: UNICEF in Nigeria should further monitor and evaluate its trainings

Monitoring and evaluation are critical for building a strong evidence base to assess whether programme efforts have had a measurable impact on expected outcomes and have been implemented effectively.

UNICEF in Nigeria should adopt monitoring and evaluation mechanisms and develop M&E instruments, guidelines and templates for training activities that systematically collect data on all training activities, expenditures and results.

This action recommendation is to be embedded in the overall training strategy mentioned under Recommendation 1.

Based on: Findings 3, 4, 7, 8, 10, 11, 12, and conclusion

Priority: high

Responsible: UNICEF Nigeria country Chief of monitoring and evaluation in consultation with Deputy representative, Field Offices, and programme managers to develop guidance and templates to capture the training efforts.

The section chiefs and Chief of Zonal office (CZO) to adhere the guidance

Timeframe: two months following the development and adoption of the country training strategy.

Recommendation 3: UNICEF in Nigeria should ensure appropriate data management through the establishment and management of a database on trainings

UNICEF in Nigeria should systematize its data management system on training activities to allow the Country Office, sectors and Field Offices to store data on all training activities. Actively managing knowledge is important to an organization’s success because it facilitates decision-making capabilities and builds learning organizations by creating a learning routine.

Based on: Findings 10, 11 and conclusion

Priority: high

Responsible: Programme section chiefs in consultation with, Zonal Offices, and programme specialists.

Timeframe: Following the development and adoption of the country training strategy.

44 FINAL REPORT

© UNICEF

Appendix I List of Findings Finding 1: UNICEF Nigeria’s Training Investments were not designed as a coherent Programme and lack

a theory of change. These shortcomings are problematic and allow limited opportunities for making an informed judgment on the portfolio overall

Finding 2: Beyond the various notable efforts to meet the national training needs, in the absence of a national capacity gap analysis, trainings are scattered, which undermines relevance of the portfolio of training investments.

Finding 3: UNICEF Nigeria’s individual training activities respond to a certain extent to the needs of participants with opportunities for some possible improvements.

Finding 4: In the absence of a country training strategy and results framework, the overall effects of the training portfolio’s impact are jeopardized.

Finding 5: From a design perspective and as Training of Trainer activities (ToT), the effectiveness of the training activities is mixed.

Finding 6: The trainings do not balance technical training with delivery process. Training activities are designed with the dual purpose of transferring technical knowledge to participants and to enable them to, in turn, transfer their knowledge to others. From a design perspective, the technical content of the training is of good quality, but insufficient time is dedicated to building participants' training skills.

Finding 7: The current training design does not have a quality assurance mechanism, which represents a risk to training effectiveness.

Finding 8: Some efforts are made to monitor training participants’ learning at the field level. However, the monitoring of participants’ reaction, learning, changes in behaviour and impact is not systematic.

Finding 9: Based on all data collected and analyzed training activities, there is a high potential for effectiveness in terms of reaction and changes in behaviour. Results are difficult to assess due to the insufficiency of M&E data.

Finding 10: The ToT approach is a contributing factor to efficiency. However, the very limited and incomplete data on the UNICEF training activities seriously undermines UNICEF’s ability to ensure that trainings are performing best.

Finding 11: According to self-reported evidence, the potential for the sustainability of training results is strong. There is however some room for improvement.

Finding 12: There is limited evidence that specific actions were adopted to ensure that women and men have equal opportunities.

FINAL REPORT 45

© UNICEF

Appendix II Evaluation Matrix

EVALUATION FOCI

QUESTIONS SUB-QUESTIONS PROXY INDICATORS SOURCES OF

DATA

1. Relevance & Design

To what extent did training activities respond to Nigeria’s National and sub-national capacity gaps?

• Were training activities aligned with national / regional needs and priorities?

• Is individual capacity development a priority in national development strategies?

• Were training activities aligned with UNICEF’s country programme in Nigeria?

Evidence of training objectives being reflective of and tailored to needs of consulted stakeholders

Degree of alignment between training activities and national and regional needs and priorities

Degree of priority of capacity development programming in national development strategies

Training documents

Stakeholders’ perceptions

Were the five key UNDG steps in integrating capacities (training) into the planning process followed?29

• Were partners and communities engaged to build consensus and promote national and local ownership?

• Were capacities (training) assets and needs assessed?

• Were capacity development strategies designed?

• Did UNICEF support the implementation of capacity development strategies as planned? (answered under ‘effectiveness’)

• Were systems set up to adequately monitor capacity development efforts? (answered separately below)

Evidence of partner / community consultations and engagement

Evidence of training being designed in a participatory manner, and with trainees’ and beneficiaries’ interests being taken into account

Evidence that CPD sectors (including relevant partners and communities) were assessed for capacity gaps

Existence and quality of gaps/needs analysis for training under review

Existence of a Capacity Development Strategy (general or by sector)

Documents

Stakeholders’ perceptions

29 These “five steps” are described in the UNDG Technical Brief on Capacity Development, March 2009. They are as follows – Step 1: Engagement with partners and building consensus; Step 2: Assessing capacity assets and needs; Step 3: Designing capacity development strategies; Step 4: Implementation of capacity development strategies; Step 5: Evaluation of capacity development efforts.

46 FINAL REPORT

© UNICEF

EVALUATION FOCI

QUESTIONS SUB-QUESTIONS PROXY INDICATORS SOURCES OF

DATA

What other guidance, if any, was used? Was it consistently applied?

• What guidance, if any, was used to design individual training activities?

• What minimum standards, if any, did training activities comply with?

• To what extent were capacity components as defined by UNDG incorporated in training, as relevant?30

• What are the underlying assumptions in the logic of intervention? Do they continue to be valid? Are any missing?

• How were training participants (organisations or individuals) recruited / selected?

Existence and quality of written strategy, guidance, standards

Existence and spread of unwritten norms, assumptions, expectations

Degree to which standards are explicit vs. implicit

Evidence that UNDG capacity components are systematically incorporated in training planning

Existence and quality of theory of change, logic model, assumptions

Assessment of criteria for recruiting participants

Documents

Stakeholders’ perceptions

To what extent were training activities consistently monitored?

• How were training activities monitored (baselines, targets, indicators)?

• Were the objectives of individual training activities clearly delineated?

• What data, if any, was systematically collected / reported from training activities?

• Were data collection / monitoring instruments adequate?

Share of individual training activities with clear near- and medium-term objectives

Degree to which indicators comply with S.M.A.R.T. standards

Share of training activities for which baseline / target data is available

Qualitative assessment of data collection instruments

Documents

Stakeholders’ perceptions

30 These Capacity Components are described in the UNDG Capacity Assessment User Guide, Feb 2008: “The eight capacity components are: human resources; public sector accountability; access to information, development knowledge and technology; inclusion, participation, equity and empowerment; financial resources; material resources; environmental resources; and external/international relations” (p.5)

FINAL REPORT 47

© UNICEF

EVALUATION FOCI

QUESTIONS SUB-QUESTIONS PROXY INDICATORS SOURCES OF

DATA

2. Effectiveness Were training activities delivered as planned?

• Were planned training activities delivered?

• Were the expected numbers of participants reached?

• To what extent did training activities demonstrate progress set objectives?

• To what extent did training activities contribute to the overall UNICEF Nigeria Country Programme objective?

Proportion of planned training activities implemented

Proportion of planned training activities budget disbursed

Proportion of expected training participants who started the training

Proportion of trainees who successfully completed their training

Degree of progress of training activities under review towards their set objectives

Documents

Stakeholders’ perceptions

To what extent was training effective, with respect to the four levels of the Kirkpatrick model?31

• LEVEL 1 - REACTION: Did participants find the training favorable, engaging and relevant to their jobs / lives?

• LEVEL 2 - LEARNING: Did participants acquire the intended knowledge, skills, attitude, confidence and commitment?

• LEVEL 3 - BEHAVIOUR: Did participants apply what they learned during training when they are back on the job / in their communities?

• LEVEL 4 - RESULTS: Did targeted outcomes occur as a result of the training and the support and accountability package?

L1: Degree of participant satisfaction with training (various indicators based on feedback questionnaires)

L2: Difference in scores between pre and post tests

L3: Evidence that trainees report applying training from lessons on their job / in their lives; Frequency and significance of the changes reported

L4: Illustrative stories of change that can be plausibly traced back to contributions from UNICEF training activities

L1: Training evaluation questionnaires

L2: Pre/post test scores

L3: Survey questionnaires / Stories of change from trainees

L4: Stories of change from trainees / Stories of change from beneficiaries

31 For a basic introduction to the Kirkpatrick model, and for the source of the Kirkpatrick sub-questions in this table, see http://www.kirkpatrickpartners.com/OurPhilosophy/TheKirkpatrickModel (accessed September 22nd, 2016).

48 FINAL REPORT

© UNICEF

EVALUATION FOCI

QUESTIONS SUB-QUESTIONS PROXY INDICATORS SOURCES OF

DATA

What are the strengths and weaknesses of the training programmes?

• What good/best practices emerged across training activities?

• What major or recurrent issues were encountered across training activities?

• Did training activities have any unintended results (good or bad)?

Evidence of notable successes / challenges / trends / lessons learned

Evidence of unintended positive or negative results, if any

All available sources

3. Efficiency How efficient is UNICEF Nigeria at providing training activities?

• How much did training activities cost? Average training cost per session, per trainee, per type

Main cost drivers

Documents (financial data)

• Were training activities conducted in the most cost-effective way?

Comparison of training costs and results achieved across Sections

Degree to which stakeholders are satisfied with cost-effectiveness

Documents (financial data)

Stakeholders’ perceptions

• To what extent do partners contribute to training cost?

Average funding provided by partners per training

Documents (financial data)

4. Sustainability To what extent are UNICEF Nigeria training interventions likely to produce continued benefits? (by design)

• What mechanisms were set up to foster sustainable training results? (in the face of issues such as turnover, knowledge retention, etc.)

• What, if any, follow-up on training activities was undertaken or planned?

• To what extent did training activities generate results that are sustainable?

Evidence of planning for sustainability during the design and implementation stages

Evidence of efforts to sustain achievement of training objectives post-training delivery

Percentage of training that indicate evidence of actual / potential sustainability

Documents

Stakeholders’ perceptions

To what extent are UNICEF Nigeria training interventions likely to produce continued benefits? (external factors)

• Is there favorable organisational support for the sustainability of the capacity development agenda and programme?

• Is training in the short-run consistent with a long-run approach to address structural and systemic issues, including changes in the incentive and support systems to

Extent to which trainees’ organizations support the application of learning acquired

Time lag between training and participant’s ability to apply skills/knowledge

Documents (incl. financial data)

Stakeholders’ perceptions

FINAL REPORT 49

© UNICEF

EVALUATION FOCI

QUESTIONS SUB-QUESTIONS PROXY INDICATORS SOURCES OF

DATA

facilitate the practice of newly acquired skills?

• Is there a long-term budget to support training efforts and strategies?

• To what extent are government authorities increasingly able to provide for their own Capacity Development needs?

Extent to which participants are provided with the required tools to apply skills/knowledge acquired

Trends in financial support

Percentage of training interventions that have UNICEF resources to be continued / repeated / scaled-up

Evidence of alternative sources of funding from government and/or partners

5. Cross-Cutting Issues

Are there systematic differences in findings with respect to key evaluation criteria for given sub-groups of training activities?

• How do findings compare across the five programme sections under review?

• How do findings compare across different types of training activities?

• How do findings compare across different regions of the country?

Level of congruence / divergence in findings relative to these sub-groups

Existence of factors that would explain or justify differentiated performance along these sub-groups

Documents

Stakeholders’ perceptions

To what extent have gender and equity considerations been mainstreamed into training investments?

• To what extent are training activities designed with these considerations in mind?

• To what extent do training activities convey these considerations to participants?

• To what extent is gender and equity-related data tracked?

Level of stakeholder awareness and familiarity with gender and equity considerations

Evidence of planning for the integration of gender and equity considerations in training activities

Availability of gender and equity related monitoring data

Documents

Stakeholders’ perceptions

50 FINAL REPORT

© UNICEF

Appendix III Roles and Responsibilities of Team Members 128. An updated list of Team Members’ roles and responsibilities is provided below.

PROPOSED STAFF

POSITION IN THE FIRM

LOCATION ROLE AND RESPONSIBILITIES

Dr. John Adeboye ADEYEMO

Associate Ibadan, Nigeria Team Leader

• Team management in Nigeria

• Chief liaison with NCO IMEP

• Participates in the evaluability assessment

• Participates in the design of methodology and tools

• Leads and conducts the inception mission to Abuja

• Leads and manages data collection and analysis process

• Conducts the portfolio review

• Oversees the design of the survey and training of the enumerators

• Conducts data collection missions in three field offices + FCT

• Conducts interviews with senior stakeholders

• Co-lead author for report

Christian T. IDOKO

Associate Abuja, Nigeria Senior Evaluation Expert

• Participates in the evaluability assessment

• Conducts the inception mission to Abuja

• Participates in the design of methodology and tools, including the design of the survey

• Conducts the portfolio review

• Trains the enumerators and oversees the implementation of the survey

• Plans the field missions (schedule interviews)

• Conducts data collection missions in four field offices + FCT

• Conducts interviews with stakeholders

• Performs data analysis

• Contributes to report drafting

Dr. Marie-Hélène ADRIEN

Quality Assurance Advisor

• Overall team management

• Provides quality assurance advisory

FINAL REPORT 51

© UNICEF

PROPOSED STAFF

POSITION IN THE FIRM

LOCATION ROLE AND RESPONSIBILITIES

Mariane ARSENAULT

Employee Montreal, Canada

Senior Evaluation Expert

• Participates in the evaluability assessment, including the reconstruction of the theory of change

• Participates in the design of methodology and tools, including the design of the survey

Rima SLAIBI Employee Montreal, Canada

Evaluation Consultant

• Contributes to data analysis

• Co-lead author for report

Lorenzo DAÏEFF Employee Montreal, Canada

Research Assistant

• Participates in the evaluability assessment, including the development of the report

• Conducts the portfolio review

• Contributes to data analysis

Three Research Assistants

Associates Nigeria Enumerators

• Collect data with training participants based on data collection tools and instructions provided by Dr. Adeyemo and Mr. Idoko, in six field offices (two per enumerator).

52 FINAL REPORT

© UNICEF

Appendix IV Theoretical Frameworks 129. The first, and most general, framework consists of the OECD-DAC criteria outlined in the TORs questions and the evaluation matrix. These internationally accepted evaluation standards – as defined by the OECD-DAC – define key areas to be assessed in the review of a project, programme or organization. As such, the present assignment assessed the relevance, effectiveness, efficiency and sustainability of the NCO’s training investments. In particular, the evaluation matrix stipulates relevant questions, sub-questions, indicators, and sources of information related to each evaluation criteria. A fifth criteria, impact, was not evaluated: as outlined in the evaluability assessment, the unavailability of data makes it impossible to coherently and validly assess the ‘impact’ of the NCO’s training investments.

130. The second framework guiding this assignment is the United Nations Development Group’s Guidance (UNDG) on Capacity Development (CD), and in particular the ‘five key steps’ in designing CD activities, illustrated in Figure iv.1 below: Partner Engagement, Needs Assessment, Design of CD Strategy, Implementation of CD Activities, and Evaluation of CD Activities. This framework, cited in the TORs and integrated into the evaluation matrix, was used to ensure that every step of planning is adequately taken into account and evaluated.

Figure iv.1 Five Key UNDG Steps for Capacity Development

131. Finally, the third framework behind our approach is known as the ‘Kirkpatrick Model’, a worldwide standard for evaluating the effectiveness of training. As outlined in our technical proposal, the model identifies four consecutive levels of training ‘results’: (1) the reaction of participants to the training; (2) the knowledge and skills participants acquire during training; (3) the change in behaviour of participants once they return to their work and lives; and (4) the results that actually occur (in terms of performance, service-delivery, intended objectives, etc.) as a consequence of the training activity. As outlined in the evaluability assessment as well as in the evaluation matrix, different kinds and amounts of data are required in order to assess these different levels. The evaluation, did not have the data for this assignment

FINAL REPORT 53

© UNICEF

to formally satisfy the high informational demands of level 4 (“results”). The evaluation was however able to trace individual, qualitative stories of changes resulting from training, and drew on these for a cautious assessment of actual and potential training results.

Cross-Cutting Issues

132. In addition to these three frameworks, our evaluation team also payed attention to the following four cross-cutting considerations in our approach, which informed sampling, questions, and final analysis:

▪ Differences by Programme Section: the five CPD sections under review operate in distinct environments, and plan training investments largely independently from each other. As such, the evaluation was attentive to relevant differences in the context, management or performance of training investments by section.

▪ Differences by Type of Training: the mapping of training activities points to a diversity of training types: day-long, week-long, and month-long training events; UNICEF-provided training vs contractor-provided training, etc. Whilst the evaluation was not expected to systematically report on the relative performance of each sub-type, its remained alert to any systematic trends reported in documentation and conversations.

▪ Gender and Equity: although they are not noted in the TORs for this assignment, gender and equity are guiding principles of UNICEF worldwide and as such extensively noted in the organizations’ Strategic Plan (2014-2017). The evaluation team consulted stakeholders and documentation to identify the degree to which these considerations have been mainstreamed in the NCO’s training investments.

54 FINAL REPORT

© UNICEF

Appendix V List of Persons Consulted Table v.1 Interviews conducted during the inception phase

NAME POSITION Organization

Kannan Nadar Chief, WASH UNICEF

John Egbe Agbor Chief, Health UNICEF

Ophilia Karumuna Child Protection Specialist (Child Justice)

UNICEF

Charles Avelino Education Specialist UNICEF

Bamidele Davis Omotola Nutrition Specialist UNICEF

Gloria Ugochi Nwulu Health Specialist (Immunization) UNICEF

Mirza Mohammad Human Resource Specialist UNICEF

Binta Isah-Ismail M&E Specialist UNICEF

Denis Jobin Chief, M&E UNICEF

Raymond Akor M&E Specialist, Kaduna FO UNICEF

Maureen Zubie-Okolo M&E Specialist, Enugu FO UNICEF

Adeniyi Olaleye M&E Specialist, Akure/Lagos FO UNICEF

Danjuma Almustafa M&E Specialist, Bauchi FO UNICEF

Table v.2 Interviews conducted during data collection: Training Planners/Implementers interviewed

NAME POSITION Organization

Mr. Nebo Fabian State Health Educator State Ministry of Health, Enugu

Mrs. Amu Hilda State Cold Chain Officer State Ministry of Health, Enugu

Mrs. Mary Manu Reproductive Health and Save Motherhood Coordinator

State Ministry of Health, Enugu

FINAL REPORT 55

© UNICEF

NAME POSITION Organization

Mrs. B. P. Enenwali Baby Friendly Initiative Coordinator State Ministry of Health, Enugu

Mrs. Martina Eze Integrated Management of Child and Illnesses(IMCI)

State Ministry of Health, Enugu

Mrs. HenrietaUguwu Nutrition Officer and Focal person for Maternal New Born and Child Health

State Ministry of Health, Enugu

Joy Anorve EPI Consultant

Enugu

UNICEF

Saaondo Anom WASH Specialist

Enugu

UNICEF

Toyin Adisa WASH Consultant

Enugu

UNICEF

Mr Ogbodo, Obinna Mark WASH Consultant Anambra UNICEF

Dr Willy Mamah Child Justice Consultant

Lagos

UNICEF

Bisi Orebiyi WASH Consultant

Akure

UNICEF

Ada Ezeogu Nutrition Specialist

Akure

UNICEF

Murtala Mohammed Education Specialist

Akure

UNICEF

Adebola Hassan Health Specialist

Akure

UNICEF

Theresa Pamma WASH Specialist, Kaduna UNICEF

Dr. Amina Mohammed Baloni Health Specialist, Kaduna UNICEF

Daniel Ali Deputy Director, PHC, State Ministry of Health Kaduna

State Ministry of Health Kaduna

Danladi Adamu Education Specialist, Kaduna UNICEF

Dr. Drissa Yeo WASH Specialist, Bauchi UNICEF

Dr. Huba Lawal WASH Officer UNICEF

Ladi Alabi Child Protection Specialist, Bauchi UNICEF

Abdulai Kaikai Chief of Field Office, Bauchi UNICEF

Philomena Nutrition Specialist, Bauchi UNICEF

Dr Halima Abdu Health Specialist, Bauchi UNICEF

56 FINAL REPORT

© UNICEF

NAME POSITION Organization

Ishaku, A, Nda Deputy Director PRS, Ministry of Education Science & Technology, Niger State.

Ministry of Education, Science &Technology, Niger State

Kannan Nadar Chief WASH, Abuja. UNICEF

Ophilia Karumuna Child Protection Specialist (Child Justice), Abuja

UNICEF

Charles Avelino Education Specialist, Abuja UNICEF

Gloria Ugochi Nwulu Health Specialist (Immunization), Abuja

UNICEF

Assaye Bulti Nutrition Officer, Abuja UNICEF

Elfriede Kormawa Information Management Officer, Abuja.

UNICEF

Ranganai Matema Health Specialist (Immunization/Polio), Abuja.

UNICEF

Arjan de Wagt Chief Nutrition, Abuja UNICEF

FINAL REPORT 57

© UNICEF

Appendix VI Training Evaluation Template

Section: Date: / / 2016 Author:

Name/Short Description:

(who is trained on what?)

General Information

133. This information will be collected during individual interviews with relevant stakeholders, especially training designers and implementers. It will also be complemented by information from the desk review.

Table vi.1 Basic Information on the Training Activity

INFORMATION PLANNED REALISED

Outcomes/Outputs UNDAF Outcome/Output:

CPD Output:

Key Indicator:

Objectives Baseline:

Target:

Narrative:

Result:

Narrative:

Year(s)

Responsible Office (NCO, FOs, ...)

# of sessions (and location of each)

# of participants (total / per session)

Cost/Budget (total)

Main Cost Points

Cost Contributions

Main Partners

58 FINAL REPORT

© UNICEF

Table vi.2 Persons to Consult

RESPONSIBILITY PERSONS KEY QUESTIONS CONSULTED?

Planners of Training

UNICEF staff: section chief, section specialists (including at FO level), M&E specialists, financial specialists

Partners, co-financiers, government officials at national / state / LGA level

How was the need for this training identified? Who was consulted?

What level of priority is (this) training? What budget is assigned to it?

Designers of Training

UNICEF section specialists, M&E specialists, financial specialists

Relevant experts (UNICEF, partners, consultants, contractors)

Is the design new or old? And the materials?

If new, who designed the training? On what basis?

Were M&E aspects built in?

Were Gender & Equity considered?

Deliverers of Training

UNICEF section specialists

Relevant experts (UNICEF, partners, consultants, contractors)

Who delivered the training?

What expectations from the training were achieved?

Recipients of Training

Training participants How, if at all, has the training affected the behaviour of participants in their job/lives?

Which positive or negative effects has the training had?

Beneficiaries of Training Programme

Training participants

Managers of capacitated institutions

Affected citizens, communities

Affected organizations, partners

Which changes can be traced back, at least partially, to the training?

Which services improved as a (partial) result of the training?

What roadblocks, if any, remain?

Table vi.3 Documents to Consult

DOCUMENT KEY QUESTIONS REVIEWED?

Needs Assessment Was a clear need for this training activity identified? How?

Narrative Report(s) Did the training go according to plan? Were objectives met?

Are there any generalizable lessons?

Training Evaluation Questionnaire Are the questions clear, adequate?

Pre/Post Questionnaires Are the questions clear, adequate?

Training Evaluation Data What was participants’ feedback?

Pre/Post Data What did participants learn?

Participant Lists How many men/women?

How many regions?

Do we have contact details?

FINAL REPORT 59

© UNICEF

DOCUMENT KEY QUESTIONS REVIEWED?

Training Agenda How long was the training?

Was it logically structured?

Training Materials: PowerPoint Presentations; Trainer Manual; Trainee Manual; etc.

Are the materials of high quality?

Do they cover the required themes?

Are they in line with objectives?

Financial Data How much did the training cost? (total/per participant)

What were the major cost points?

Table vi.4 Key Questions by Phases of a Training Activity

STAGE KEY QUESTIONS ASKED?

Needs Assessment Was a needs assessment conducted? How? By whom?

Design of remedial Training Who designed the training? With whom? On what basis?

Delivery of Training How was the training delivered? By whom? To whom?

Did training delivery proceed according to plan?

Results of Training What effects did the training have? (Kirkpatrick approach)

What factors (internal or external) influenced training results?

Evaluation of Training Was the training evaluated? How?

Additional Notes:

Overall Assessment of the Training:

60 FINAL REPORT

© UNICEF

Appendix VII Data Collection Tools

Questionnaire used during Inception Mission

INTERVIEW GUIDE

1) What is the name of training?

2) What are the objectives of the training?

3) Brief description of the training?

4) Duration of training?

5) Where was the training delivered?

6) Cost of training

7) Total number of participants?

8) Is there a Participants list?

9) Key contacts to meet in the field (phone and email)

10) Who designed the training (UNICEF, other partners, consultant)?

11) Who delivered the training (UNICEF staff from field offices, country office, WCARO, HQ, outsourced to vendors)?

12) Was there a need assessment?

13) Were there pre and post tests?

14) Any evidence of training effectiveness and or results in terms of learning and behavioural change?

15) Is there documentation with respect to:

– Learning

– Behavioural change

– Results

FINAL REPORT 61

© UNICEF

Interview Protocol for High-Level Stakeholders

134. Note: This sample protocol is meant for stakeholders that may not have been closely involved with planning, designing or attending training activities, but who have a stake as partners or indirect beneficiaries. This includes federal and local governments, and relevant technical and financial partners.

135. We have been asked by UNICEF Nigeria to review its training interventions for individuals conducted between January 2014 and May 2016 in the following five sectors: Education, Health, Child Protection, WASH, and Nutrition.

136. You have been identified as a relevant stakeholder for this discussion of about 30-45 minutes in order to better understand the lessons from UNICEF Nigeria’s recent training investments. Your contribution will help us to identify (1) what did and did not work, and (2) areas of improvement for the future.

137. This isn’t a test – no right or wrong answers; your opinions count and we’ll listen respectfully. Please note that information provided will remain anonymous, and that you may refuse or interrupt participation at any time. We thank you in advance for your time and cooperation in this critical exercise.

138. Before we begin, do you have any general questions?

BASIC INFORMATION

Last Name, First Name:

Title, Unit, Organization:

Date, Location of Interview:

Introduction

1) Can you tell me your exact position and how long you have been in it?

2) What is your or your organization’s relationship to UNICEF training activities & investments?

3) Are there any particular UNICEF training activities you have been directly involved with?

Relevance & Design

4) In your opinion, to what extent do UNICEF training activities respond to Nigeria’s capacity development needs?

5) In your opinion, to what extent are UNICEF Nigeria’s training activities relevant to you / your organization’s work and mandate?

6) In your opinion, to what extent do training activities fit in with the capacity needs and learning requirements of target trainees?

7) To what extent, if at all, were you / your organization consulted regarding the planning and design of, UNICEF Nigeria’s training activities?

8) What are your expectations vis-à-vis UNICEF Nigeria’s training investments?

62 FINAL REPORT

© UNICEF

Effectiveness

9) In your experience, are UNICEF’s training activities delivered as planned?

10) In your experience, do UNICEF’s training activities attain their intended objectives?

11) What factors have facilitated/hindered the achievement of training objectives? (factors can be internal, e.g. training design or management, or external)

12) Do you have any stories of changes (positive or negative) that can be traced back to UNICEF training activities?

13) What do you think of the value-added of capacity-development activities, such as training, relative to other plausible investments?

Conclusion

14) Overall, what would you say are main strengths and weaknesses of UNICEF’s training investments?

15) What, in your opinion, are suggestions to improve UNICEF’s training investments, going forward?

16) Is there any additional information you would like to share, or any further comments?

Thank you for your time and cooperation.

FINAL REPORT 63

© UNICEF

Interview Protocol for Training Planners

139. Note: This sample protocol is meant for stakeholders that may not have been closely involved with a particular training activity, but who contribute to the overall planning and direction of such investments. This includes high-level UNICEF staff (e.g. Section Chiefs, Heads of Field Offices), and, if relevant, partners involved in training planning. Training design and delivery are covered in a separate questionnaire.

140. We have been asked by UNICEF Nigeria to review its training interventions for individuals conducted between January 2014 and May 2016 in the following five sectors: Education, Health, Child Protection, WASH, and Nutrition.

141. You have been identified as a relevant stakeholder for this discussion of about 30-45 minutes in order to better understand the lessons from UNICEF Nigeria’s recent training investments. Your contribution will help us to identify (1) what did and did not work, and (2) areas of improvement for the future.

142. This isn’t a test – no right or wrong answers; your opinions count and we’ll listen respectfully. Please note that information provided will remain anonymous, and that you may refuse or interrupt participation at any time. We thank you in advance for your time and cooperation in this critical exercise.

143. Before we begin, do you have any general questions?

BASIC INFORMATION

Last Name, First Name:

Title, Unit, Organization:

Date, Location of Interview:

Introduction

1) Can you tell me your exact position and how long you have been in it?

2) What are your current responsibilities with regards to UNICEF training activities & investments?

3) Are there any particular UNICEF training activities you have been directly involved with?

Relevance & Design

4) In your opinion, to what extent do UNICEF training activities respond to Nigeria’s capacity development needs?

5) In your opinion, to what extent do training activities fit in with the capacity needs and learning requirements of target trainees?

6) Please describe the process of how a training activity is usually planned:

– Who at UNICEF is in charge of planning capacity development / training activities?

– What guidelines, guidance, or minimum standards are followed?

– How is the need for a training identified?

– Who is consulted?

64 FINAL REPORT

© UNICEF

7) How, if at all, are the following issues accounted for in planning a training activity:

– Monitoring and Evaluation

– Cost-efficiency

– Sustainability Considerations

– Gender and Equity

Effectiveness

8) In your experience, are UNICEF’s training activities delivered as planned?

9) In your experience, do UNICEF’s training activities attain their intended objectives?

10) What factors have facilitated/hindered the achievement of training objectives? (factors can be internal, e.g. training design or management, or external)

11) Do you have any stories of changes (positive or negative) that can be traced back to UNICEF training activities?

Efficiency

12) How adequate are human and financial resources for training activities?

13) How are training activities assigned resources and budgeted for?

14) How, if at all, were costs shared with partners or funds mobilized from partners?

15) What do you think of the value-added of capacity-development activities, such as training?

Conclusion

16) Overall, what would you say are main strengths and weaknesses of UNICEF’s training investments?

17) What, in your opinion, are suggestions to improve UNICEF’s training investments, going forward?

18) Is there any additional information you would like to share, or any further comments?

Thank you for your time and cooperation.

FINAL REPORT 65

© UNICEF

Interview Protocol for Training Designers and Implementers

144. Note: this interview protocol is written with interviewees in mind who have worked on a specific training (as managers, designers, implementers, trainers).

145. We have been asked by UNICEF Nigeria to review its training interventions for individuals conducted between January 2014 and May 2016 in the following five sectors: Education, Health, Child Protection, WASH, and Nutrition.

146. You have been identified as a relevant stakeholder for this discussion of about 30-45 minutes in order to better understand the lessons from UNICEF Nigeria’s recent training investments. Your contribution will help us to identify (1) what did and did not work, and (2) areas of improvement for the future.

147. This isn’t a test – no right or wrong answers; your opinions count and we’ll listen respectfully. Please note that information provided will remain anonymous, and that you may refuse or interrupt participation at any time. We thank you in advance for your time and cooperation in this critical exercise.

148. Before we begin, do you have any general questions?

BASIC INFORMATION

Last Name, First Name:

Title, Unit, Organization:

Date, Location of Interview:

Training Activity:

Introduction

1) Can you tell me your exact position and how long you have been in it?

2) What UNICEF training activity/activities have you been involved in? In what capacity?

3) To the best of your knowledge, please help us to fill in basic information on the(se) training(s) (cf. our “training assessment template” – location, duration, participants, objectives, etc.)

Relevance & Design

4) Please describe the process of how the training activity was designed / delivered:

– Who was in charge of designing / delivering the training? (Did the training and its materials already exist, or were the designed specifically for this activity?)

– Which guidelines, guidance, or minimum standards were followed?

– What needs were addressed through the training? How were these training needs identified?

– Who was consulted for the needs assessment and training design?

5) Regarding the needs addressed by the training:

– What needs were addressed through the training intervention?

– What were the practices of trainees before the intervention?

– What baseline skills, knowledge or assets of trainees did the training build on?

6) How, if at all, were the following issues accounted for in designing / delivering the training:

66 FINAL REPORT

© UNICEF

– Monitoring and Evaluation

– Cost-efficiency

– Sustainability Considerations

– Gender and Equity

Effectiveness

7) Was the training delivered as planned?

8) Were an evaluation questionnaire and pre/post tests administered?

9) Did the training reach its intended objectives? How do we know?

10) What factors facilitated/hindered the achievement of training objectives? (factors can be internal, e.g. training design or management, or external)

11) Can you narrate any stories of changes (positive or negative) that can be traced back to this training?

12) What follow-up (monitoring, other training sessions, step-down training), if any, has occurred since?

13) How sustainable/lasting are the results of the training, if support is removed?

Efficiency

14) How adequate were the financial and human resources for the training?

15) What did you think of the value-added of this training? How adequate was it?

16) Were resources mobilized from partners, or other cost-efficient approaches used?

Conclusion

17) Overall, what would you say were the main strengths and weaknesses of this training?

18) What, in your opinion, are suggestions to improve UNICEF’s training investments, going forward?

19) Is there any additional information you would like to share, or any further comments?

Thank you for your time and cooperation.

FINAL REPORT 67

© UNICEF

Focus Group Protocol for Training Participants

149. We have been asked by UNICEF Nigeria to review its training interventions for individuals conducted between January 2014 and May 2016 in the following five sectors: Education, Health, Child Protection, WASH, and Nutrition.

150. You have been identified as a relevant stakeholder for this discussion of about 30-45 minutes in order to better understand the lessons from UNICEF Nigeria’s recent training investments. Your contribution will help us to identify (1) what did and did not work, and (2) areas of improvement for the future.

151. Everyone’s opinion is important and we want to hear from each of you. This isn’t a test – no right or wrong answers; your opinions count and we’ll listen respectfully. Please note that information provided will remain anonymous, and that you may refuse or interrupt participation at any time. We thank you in advance for your time and cooperation in this critical exercise.

152. Note: Respondents should be asked to list UNICEF training activities they participated in. Notice if respondents are involved in multiple activities or a few.

153. Before we start, do you have any questions?

1) First, let’s go around the group to get to know each other a little bit: please mention your name, a little bit about your background, and the training(s) you received.

2) Why did you get involved in UNICEF-supported training events?

3) If you were to explain to a friend what the training has meant to, how would you describe it?

4) Have you been able to apply what you learned from the training(s) in life or at work? (if yes – how so; if no – why not)

5) Beyond your own capacity, would you say the training(s) has increased the capacity of your organization, institution, or community? (if yes – how so; if no – why not)

6) Do you have suggestions for improving UNICEF’s training investments going forward?

7) Any other comments?

68 FINAL REPORT

© UNICEF

Focus Group Protocol for Training Beneficiaries

154. Note: Specific questions for Beneficiaries should be tailored to the kind of training at hand, the types of potential beneficiaries, and the types of potential benefits they may have derived.

155. We have been asked by UNICEF Nigeria to review its training interventions for individuals conducted between January 2014 and May 2016 in the following five sectors: Education, Health, Child Protection, WASH, and Nutrition.

156. You have been identified as a relevant stakeholder for this discussion of about 30-45 minutes in order to better understand the lessons from UNICEF Nigeria’s recent training investments. Your contribution will help us to identify (1) what did and did not work, and (2) areas of improvement for the future.

157. Everyone’s opinion is important and we want to hear from each of you. This isn’t a test – no right or wrong answers; your opinions count and we’ll listen respectfully. Please note that information provided will remain anonymous, and that you may refuse or interrupt participation at any time. We thank you in advance for your time and cooperation in this critical exercise.

158. Before we start, do you have any questions?

1) First, let’s go around the group to get to know each other a little bit: please mention your name and a little bit about your background.

2) FOR ALL: What major hurdles do you face in your life / your work? How could or should these be resolved?

3) FOR CITIZENS: How do you feel about the services government provides (health, education, etc.)?

4) FOR ALL: What in your view are the main actions to be taken to strengthen children’s wellbeing?

5) FOR MANAGERS / COMMUNITY LEADERS: Would you say the training activity attended by your staff/members has increased the performance of your organization, institution, or community? (if yes – how so; if no – why not)

6) FOR MANAGERS / COMMUNITY LEADERS: How satisfied are you with the new knowledge and skills you have acquired through the UNICEF training interventions? What have you learned, gained, accomplished?

7) FOR MANAGERS / COMMUNITY LEADERS / PARTNERS: Can you narrate stories of change (in work, life) that can be traced back to UNICEF training interventions?

8) FOR MANAGERS / COMMUNITY LEADERS / PARTNERS: How sustainable are the results gained from the training interventions if UNICEF support is removed?

9) Any other comments?

FINAL REPORT 69

© UNICEF

UNICEF TRAINING PARTICIPANTS QUESTIONNAIRE

159. The consultancy company Universalia is currently undertaking an evaluation on behalf of UNICEF Nigeria regarding its training investments since 2014. The main objective of this evaluation is to understand how and why training activities carried out by UNICEF Nigeria are useful or not useful to training participants.

160. We would be very grateful if you could take a few minutes of your time to answer some questions. The questions were designed to get some insights on the UNICEF training in which you participated.

161. As you go through the questionnaire, if you have any doubts or concerns, please ask for clarifications from the consultant or research assistant who gave or sent you this questionnaire. Some questions may be unclear to you since the UNICEF training activities may have taken place a long time ago. Do not hesitate to ask for additional explanations.

162. If you want more information about this overall evaluation, please get in touch with its leader, Dr. Adeboye Adeyemo: boyeadeyemo@yahoo.com. Please rest assured that all your answers will remain confidential. UNICEF Nigeria will not be made aware of your individual responses. They will only see the combined answers from all participants.

163. We thank you very much for your time and interest in this evaluation.

BACKGROUND INFORMATION

Did you participate in a training facilitated by UNICEF between January 2014 and May 2016? Please mark your responses in this questionnaire with an ‘X’.

Yes No

If you answer NO to Question 1.1, please do not continue this survey.

Was it clear to you that the training you participated in was organized by UNICEF?

Yes, it was clear since before the training.

Yes, it was clear as of the training itself.

No, it was not clear.

Briefly, what training session did you participate in? Describe it by its title, if you remember, or by briefly indic ating who was trained on what topic.

70 FINAL REPORT

© UNICEF

Please indicate when you participated in a training facil itated by UNICEF. If the training overlapped several years, please cross all the relevant boxes.

2014 2015 2016 Do not remember

Had you ever participated in a UNICEF training before 2014? Yes, one Yes, several No, never Not sure / Do not

remember

How long (hours or days) was the training? If you attended several sessions (separated in time, and/or in various locations), please indicate the duration of each.

Please indicate where the training was held (city and state). If you attended several sessions, please indicate the location of each.

Please indicate your gender:

How old are you? 18-35 36-50 51-70 70 +

What was your occupation at the time of the training? Employed in public sector

Employed in private sector

Self-employed Unemployed In school / studying

Are you still in the same occupation? Yes, same position and organization.

Same organization but different position.

Same position but different organization.

No, my occupation has changed. Please specify:

What is the highest level of education that you completed? None Primary education Secondary

education OND/NCE/Degree Other, please specify:

FINAL REPORT 71

© UNICEF

QUESTIONS ON UNICEF TRAINING

164. The following questions are designed to help us understand your perceptions, satisfaction, and the overall changes that you noticed in your skills and capacities following the UNICEF training you participated in. Please select the answer that best reflects your perception on the statement based on the following scale where 1 means “strongly disagree” and 4 means “strongly agree”. If for any reason you cannot respond to a given question, please select “don’t know”.

165. The questions are structured around three periods: before, during, and after the training event.

BEFORE THE TRAINING 1-

STRONGLY DISAGREE

2-

DISAGREE

3- AGREE

4-STRONGLY

AGREE

DON’T KNOW

I was consulted on my training needs prior to the training.

I was clear about the purpose of the training before I attended.

I was clear on why I was chosen/offered to attend the training.

The UNICEF training I participated in responded to needs I had before the training.

The UNICEF training I participated in was relevant to my work / main occupation.

The UNICEF training I participated in was needed to do my job better.

Please indicate how you were selected to attend the UNICEF training:

▪ I self-enrolled

▪ My supervisor / local authority told me to attend the training

▪ UNICEF invited me to participate in the training

▪ I replaced someone else who could not attend

▪ I don’t know

▪ Other, please specify:

Please indicate the main reason that drove you to actually attend the UNICEF training:

▪ Career enhancement

▪ Networking

▪ Had to attend because I was sent by my supervisor

▪ Interest in the subject matter

▪ Possibility of obtaining a certificate

▪ Other, please specify:

72 FINAL REPORT

© UNICEF

DURING THE TRAINING 1-

STRONGLY DISAGREE

2-

DISAGREE

3- AGREE

4-STRONGLY

AGREE

DON’T KNOW

The objective(s) of the training were clearly outlined at the start of the session.

The way the training related to UNICEF’s overall objectives in Nigeria was clearly explained.

The quality of the presentation and lecturing was high and content was presented in a way that was easy to understand.

The number of training participants was optimal to ensure everyone’s participation.

I fully engaged with what was going on during the training.

During the training, I was given adequate opportunity to practice what I was learning.

Training materials (handouts, documentation) were useful and assisted my learning.

Training materials were adapted to my needs and prior knowledge.

I connected and interacted in a valuable way with other training participants.

Men and women were given equal opportunity to participate in the training.

Overall, the content of the training met my learning needs.

Overall, I found the training to be satisfactory.

Overall, I acquired the knowledge, skills, and capacities I had expected.

I was tested on my knowledge at the start and/or at the end of the training.

Participants were offered adequate opportunity to provide feedback at the end of the training.

RESOURCES SPENT TO ATTEND 1-

STRONGLY DISAGREE

2-

DISAGREE

3- AGREE

4-STRONGLY

AGREE

DON’T KNOW

The training was worth the time I put into it.

I / my organization would be willing to pay to obtain more of such training in the future.

FINAL REPORT 73

© UNICEF

RESOURCES SPENT TO ATTEND 1-

STRONGLY DISAGREE

2-

DISAGREE

3- AGREE

4-STRONGLY

AGREE

DON’T KNOW

I feel the resources UNICEF spent on the training could have been better spent on something else / in another way.

If yes, how/on what? ____________________

AFTER THE TRAINING 1-

STRONGLY DISAGREE

2-

DISAGREE

3- AGREE

4-STRONGLY

AGREE

DON’T KNOW

After the training, I felt committed to applying what I learned in my work / in my life.

I was able to use what I learned in training to the benefit of my community.

I was able to use what I learned in training to the benefit of my work.

I use my new knowledge/skills on a daily basis.

I still refer to the training materials as a useful source of information.

I do things differently as a result of my participation in the UNICEF training.

I received adequate support from my organization after the training to apply my new knowledge.

The organization I work for is open to the changes I suggested as a result of my participation in the UNICEF training.

I have an overall feeling that my capacities have increased.

I have an overall feeling that, unless more training is provided, I will not retain my new knowledge/capacities.

I am expecting more positive results from the training in the future.

I expect my new capacities to remain relevant and useful for a prolonged period of time.

I have gone on to formally train other people on the skills/knowledge I acquired.

If so, how many? _______________________

I have gone on to informally share with other people the skills/knowledge I acquired.

If so, who with? _______________________

74 FINAL REPORT

© UNICEF

AFTER THE TRAINING 1-

STRONGLY DISAGREE

2-

DISAGREE

3- AGREE

4-STRONGLY

AGREE

DON’T KNOW

I have actively stayed in touch with (at least some) other participants since the training.

If so, how? _______________________

UNICEF carried out follow-up activities (contact, further support) after the training.

If yes, what kind? _______________________

FINAL REPORT 75

© UNICEF

What do you do differently as a result of the skills, knowledge and capacities taught during the UNICEF training you participated in? Please provide examples. This could be from your work or personal environment.

Please select the main reasons for being able to apply what you learned during the training. Select all that may apply.

▪ My past experience

▪ The quality of the training itself

▪ Support from my supervisor

▪ Support from my community

▪ Support from co-workers

▪ My own efforts or discipline to apply what I learned

▪ Referring back to the course materials

▪ Additional training that I voluntarily enrolled in

▪ Other, please specify:

Please select the main reasons for NOT using what I learned during the training. Select all that may apply.

▪ What I learned is not useful for my job

▪ I have been told not to use it

▪ I don’t remember what I learned

▪ I have too many other things to do

▪ I got stuck and did not know how to find help

▪ It’s too difficult to apply

▪ I have not been encouraged to apply it

▪ There are no incentives for me to apply it

▪ Other, please specify:

Please indicate the main changes that you have been able to observe following the training when you went back to your principal occupation. Select all that may apply.

▪ I had the ability to make more informed decisions

▪ I got increased responsibility

76 FINAL REPORT

© UNICEF

▪ I got increased recognition from persons around me

▪ I felt increased motivation to perform my duties

▪ I found greater satisfaction in what I do

▪ I was able to perform new duties

▪ I was able to better perform my duties

▪ I did not find there was a significant change

▪ Other, please specify:

In your view, what are the three most important strengths of the training you participated in?

1.

2.

3.

In your view, what are the three most important weaknesses of the training you participated in?

1.

2.

3.

Based on your experience, what are key suggestions for improving UNICEF’s training activities going forward?

Is there any additional information you would like to share, or any further comments?

Thank you for your cooperation.

FINAL REPORT 77

© UNICEF

Appendix VIII Bibliography

SHORT REFERENCE

CITATION

Akinsolu 2016 Akinsolu, Abiodun Olatoun. 2016. School Based Management Committee Effectiveness Monitoring Report for Bauchi, Katsina, Niger, Sokoto and Zamfara States. July 2016.

Ayoub 2015 Ayoub, Maie. 2015. Programme Planning Process Workshop Report. September 2015.

Baser et al 2008 Baser, Heather and Peter Morgan. 2008. Capacity, Change and Performance: Study Report. April 2008.

Davies 2013 Davies, Rick. 2013. Planning Evaluability Assessments: A Synthesis of the Literature with Recommendations. Working Paper 40. DFID. August 2013.

Faokunla 2015 Faokunla, Tolu. 2015. Report of SBMC Refresher Training for Civil Society – Government Partners Sokoto State. April 2015.

FAO 2001 Food and Agricultural Organization of the United Nations Regional Office for Asia and the Pacific. 2001. Asia-Pacific Forestry Commission Regional Training Strategy: Supporting the Implementation of the code of practice for forest harvesting in Asia Pacific.

FAO Food and Agricultural Organization of the United Nations Regional Office for Asia and the Pacific. [Date unpublished]. Training of Trainers Framework.

Federal Government of Nigeria

Federal Government of Nigeria. [Date unpublished]. Key Messages Booklet: The Community Infant and Young Child Feeding Counselling Package.

Federal Government of Nigeria

Federal Government of Nigeria. [Date unpublished]. National Operational Guidelines: For Community Management of Acute Malnutrition.

Federal Government of Nigeria 2011

Federal Government of Nigeria. 2011. Developing School Based Management Committees: Trainers’ Manual. September 2011.

Federal Government of Nigeria 2011

Federal Government of Nigeria. 2011. Management of Acute Malnutrition. October 2011.

Federal Government of Nigeria 2011

Federal Government of Nigeria. 2011. Participant Materials: The Community Infant and Young Child Feeding Counselling Package. August 2011.

Federal Government of Nigeria 2012

Federal Government of Nigeria. 2012. Integrated Infant and Young Child Feeding Counselling: A Training Course. February 2012.

Federal Government of Nigeria 2013

Federal Government of Nigeria. 2013. Guidelines for Implementing Maternal Newborn and Child Health Week in Nigeria. July 2013.

78 FINAL REPORT

© UNICEF

SHORT REFERENCE

CITATION

Federal Ministry of Health 2016

Federal Ministry of Health. 2016. Integrated Management of Childhood Illness: Integrated Community Case Management (iCCM): Manual. June 2016.

Federal Ministry of Health 2016

Federal Ministry of Health. 2016. Integrated Management of Childhood Illness: Integrated Community Case Management (iCCM): Facilitator’s Guide. June 2016.

Georgewill 2014 Georgewill, Theodore. 2014. Rural Water Supply and Sanitation Project of the Niger Delta Support Programme in Akuku Toru Local Government Area, Rivers State, Nigeria. March 2014.

Hill et al 2010 Hill, Ian et al. 2010. Assessing the Train-the-Trainer Model: An Evaluation of the Data & Democracy II Project. July 2010.

Liverpool School of Tropical Medicine 2015

Liverpool School of Tropical Medicine. 2015. Independent Evaluation of the Maternal, Neonatal and Child Health Week in Nigeria. Evaluability Assessment. Final Report. October 2015.

Kirkpatrick Partners

Kirkpatrick Partners. [Date unpublished]. The Kirkpatrick Model.

NPHCDA National Primary Health Care Development Agency. [Date unpublished]. Promotion of Key Family and Maternal and Child Health Practices for Village Health Workers (VHWs).

Ministry of Health 2014

Nigeria Federal Ministry of Health. 2014. The Participant Training Manual on Life Saving Skills (LSS) for Nurses/Midwives. January 2014.

Ministry of Health 2014

Nigeria Federal Ministry of Health. 2014. The Facilitator’s Guide on Life Saving Skills (LSS) Training Manual for Nurses and Midwives in Nigeria. January 2014.

Ministry of Health 2014

Nigeria Federal Ministry of Health. 2014. Training Manual on Expanded Life Saving Skills for Doctors. January 2014.

Murphy et al Murphy, Jean and Carol Carson-Warner. [Date unpublished]. Train-the-Trainer Manual: Mentoring Adult Learners.

MSG MSG. [Date unpublished]. How Efficient Knowledge Management System Leads to Greater Productivity and Synergie.

Nigeria Police Force 2014

Nigeria Police Force. 2014. Human Rights Trainers’ Guide.

Nigeria Police Force 2014

Nigeria Police Force. 2014. Human Rights Training Manual.

Nigeria Police Force 2016

Nigeria Police Force. 2016. Nigeria Police Force Human Rights Training. Chapter Five: Children’s Rights. Facilitation Materials. 2016.

Nigeria Police Force 2016

Nigeria Police Force. 2016. Nigeria Police Force Mapping Report: Specialized Police Desks. March-June 2016.

OECD 2000 OECD. [2000]. Glossary of Key Terms in Evaluation and Results Based Management.

OECD 2006 OECD. 2006. Triggering/Ignition Moments. 2006.

UN 2013 UN. 2013. United Nations Guidance Note for Effective Use and Development of National Capacity in Post-Conflict Contexts. July 2013.

FINAL REPORT 79

© UNICEF

SHORT REFERENCE

CITATION

UN 2013 UN. 2013. United Nations Development Assistance Framework – Action Plan: UNDAP Nigeria 2014-2017. 2013.

UN 2013 UN. 2013. United Nations Development Assistance Framework: Nigeria-UNDAF III 2014-2017. 2013.

UN 2014 UN. 2014. UN Development Assistance Framework: Annual Progress Report: Nigeria. 2014.

UNDAF 2010 UNDAF. 2010. Application of the Programming Principles to the UNDAF. January 2010.

UNDP 2008 UNDP. 2008. Capacity Assessment Methodology: User’s Guide. Capacity Development Group, Bureau for Development Policy. November 2008.

UNDG 2006 UN Development Group. 2006. Enhancing the UN’s contribution to National Capacity Development: A UNDG Position Statement. Ocotober 2006.

UNDG 2008 UN Development Group. 2008. UNDG Capacity Assessment Methodology. User Guide: for national capacity development. February 2008.

UNDG 2009 UN Development Group. 2009. CA Supporting Tool. April 2009.

UNESCO UNESCO. [Date unpublished]. Training of Trainers in Teacher Education for a Sustained Quality Education.

UNECOSOC 2013 UN ECOSOC. 2013. Annex: Supporting Document to the UNICEF Strategic Plan, 2014-2017. UNICEF Executive Board, Annual Session 2013. 22 April 2013.

UNECOSOC 2013 UN ECOSOC. 2013. UNICEF Strategic Plan, 2014-2017. UNICEF Executive Board, Annual Session 2013. 22 April 2013.

UNICEF UNICEF. [Date unpublished]. Action Plan 2016 WASH in Emergency Action Plan.

UNICEF UNICEF. 2015. Advocacy: Building a Strategy, Maximizing Impact. Presentation. August 2015.

UNICEF UNICEF. [Date unpublished]. Akuku-Toru Local Government Area: Local Investment Plan for Water Supply and Sanitation Sector, 2013-2017.

UNICEF UNICEF. [Date unpublished]. Anambra State WASH Emergency Action Plan.

UNICEF UNICEF. [Date unpublished]. Assessment of the Situation of Children in Conflict with the Law in Nigeria.

UNICEF UNICEF. [Date unpublished]. Bayelsa State Disaster Risk Reduction Plan for 2016.

UNICEF UNICEF. [Date unpublished]. CPS EPR Action Plan for 2016.

UNICEF UNICEF. [Date unpublished]. DRR Pre- and Post- Test Results and Facilitator Remarks.

UNICEF UNICEF. [Date unpublished]. DDR Workshop Pre- and Post-Test.

UNICEF UNICEF. [Date unpublished]. Ebonyi State DRR Action Plan 2016.

UNICEF UNICEF. [Date unpublished]. IMAM – RapidPro Tool for the Integrated Management of Acute Malnutrition. Training Manual.

UNICEF UNICEF. [Date unpublished]. IMO EPR Profile and WASH 2015.

80 FINAL REPORT

© UNICEF

SHORT REFERENCE

CITATION

UNICEF UNICEF. [Date unpublished]. Increasing Utilization of Community Based MNCH Services in Northern Nigeria.

UNICEF UNICEF. [Date unpublished]. Management of the Sick Child age 2 months up to 5 years.

UNICEF UNICEF. [Date unpublished]. Minimum Standards for Conducting ENCC Training.

UNICEF UNICEF. [Date unpublished]. Proposal for a 3-Day Zonal Training of Trainers State Level Implementing Partners on Emergency Preparedness and Disaster Risk Reduction.

UNICEF UNICEF. [Date unpublished]. Report of Borno Intensive IPC Skills Training to Build Resilience.

UNICEF UNICEF. [Date unpublished]. Results planning for 2014 and the new PIDB coding structure.

UNICEF UNICEF. [Date unpublished]. Situation Analysis of Emergencies in States in Abia, Ebonyi and Enugu States.

UNICEF UNICEF. [Date unpublished]. Template for DDR Action Plan.

UNICEF UNICEF. [Date unpublished]. UNICEF Annual Report 2014: Nigeria.

UNICEF UNICEF. [Date unpublished]. UNICEF Annual Report 2015: Nigeria.

UNICEF 2012 UNICEF. 2012. Programme Policy and Procedure Manual, Programme Operations. March 2012.

UNICEF 2014 UNICEF. 2014. UNICEF Strategic Plan 2014-2017. July 2014.

UNICEF 2014 UNICEF. 2014. UNICEF Style Book. March 2014.

UNICEF 2014 UNICEF. 2014. Revised Supplementary Programme Note on the Theory of Change for the UNICEF Strategic Plan, 2014-2017. UNICEF, Exeuctive Board, Annual Session 2014. May 2014.

UNICEF 2014 UNICEF. 2014. CRING 2014 Guidance Note. March 2014.

UNICEF 2014 UNICEF. 2014. insight/RAM 2.0. Presentation. October 2014.

UNICEF 2014 UNICEF. 2014. Integrated Management of Childhood Illness: Counsel the Mother. November 2014.

UNICEF 2014 UNICEF. 2014. Integrated Management of Childhood Illness: Follow-Up. November 2014.

UNICEF 2014 UNICEF. 2014. Integrated Management of Childhood Illness: Identify Treatment and Treat the Child. November 2014.

UNICEF 2014 UNICEF. 2014. Feedback on RAM Draft Template by Nigeria Country Office.

UNICEF 2014 UNICEF. 2014. Updated Programme Guidance on the Result Assessment Module of the VISION Performance Management System. September 2014.

UNICEF 2014 UNICEF. 2014. Result Assessment Module (RAM) UNICEF Nigeria, Phase II – Reporting.

UNICEF 2014 UNICEF. 2014. Results Assessment Module, Phase 1 – Planning: Child Protection.

FINAL REPORT 81

© UNICEF

SHORT REFERENCE

CITATION

UNICEF 2014 UNICEF. 2014. Results Assessment Module, Phase 1 – Planning: Communication for Development.

UNICEF 2014 UNICEF. 2014. Results Assessment Module, Phase 1 – Planning: EPR DRR.

UNICEF 2014 UNICEF. 2014. Results Assessment Module, Phase 1 – Planning: Education.

UNICEF 2014 UNICEF. 2014. Results Assessment Module, Phase 1 – Planning: Health.

UNICEF 2014 UNICEF. 2014. Results Assessment Module, Phase 1 – Planning: HIV.

UNICEF 2014 UNICEF. 2014. Results Assessment Module, Phase 1 – Planning: PME.

UNICEF 2014 UNICEF. 2014. Results Assessment Module, Phase 1 – Planning: Social Policy.

UNICEF 2014 UNICEF. 2014. Results Assessment Module, Phase 1 – Planning: WASH.

UNICEF 2015 UNICEF. 2015. Child Justice Bottleneck Analysis. July 2015.

UNICEF 2015 UNICEF. 2015. Family Courts Capacity Assessment. May 2015.

UNICEF 2015 UNICEF. 2015. Programme Guidance on the Results Assessment Module of VISION/insight.

UNICEF 2015 UNICEF. 2015. UNICEF Strategic Plan, 2014-2017 Evaluability Assessment. Final Report. Evaluation Office. July 2015.

UNICEF 2016 UNICEF. 2016. 2016-Mid-Tear Review Report: FGN/UNICEF Basic Education Programme.

UNICEF 2016 UNICEF. 2016. Community-Led Total Sanitation Training Manual. June 2016.

UNICEF 2016 UNICEF. 2016. Child Protection: FGN/UNICEF Country Programme 2014-2017. 2016 Mid-year Review Report.

UNICEF 2016 UNICEF. 2016. Government-UNICEF Programme of Cooperation. Report of the Mid-Year Review Meeting of the Communication for Development Section.

UNICEF 2016 UNICEF. 2016. Output Ratings with Latest Progress as of 5 September 2016.

UNICEF Nigeria UNICEF Nigeria. [Date unpublished]. Abua Field Office Nutrition Multi-Year Workplan 2014-2015.

UNICEF Nigeria UNICEF Nigeria. [Date unpublished]. Action Plan for the Implementation of WASH M&E in Bayelsa State.

UNICEF Nigeria UNICEF Nigeria. [Date unpublished]. Analysis/Evaluation of Orientation Meeting on Strategies to Interrupt Transmission of WPV for Media Operators from Kaduna, Katsina and Niger States.

UNICEF Nigeria UNICEF Nigeria. [Date unpublished]. Analysis of Participants’ Pre-Test and Post-Test (Bauchi Workshop).

UNICEF Nigeria UNICEF Nigeria. [Date unpublished]. AWKA, ToT Workshop on Development of Water Safety Plans in Communities.

UNICEF Nigeria UNICEF. [Date unpublished]. Baseline Assessment for 10 Project LGAs.

UNICEF Nigeria UNICEF Nigeria. [Date unpublished]. Basic Concept of Water Safety Plan. Presentation.

82 FINAL REPORT

© UNICEF

SHORT REFERENCE

CITATION

UNICEF Nigeria UNICEF Nigeria. [Date unpublished]. Basic Negotiation and Interview Skills. Presentation.

UNICEF Nigeria UNICEF Nigeria. [Date unpublished]. Bauch Field Office Nutrition Multi-Year Workplan 2014-2015.

UNICEF Nigeria UNICEF Nigeria. [Date unpublished]. Bauchi, ToT Workshop on Development of Water Safety Plans in Communities.

UNICEF Nigeria UNICEF Nigeria. [Date unpublished]. Bayelsa State Existing M&E System.

UNICEF Nigeria UNICEF Nigeria. [Date unpublished]. Checklist for Verification and Certification.

UNICEF Nigeria UNICEF Nigeria. [Date unpublished]. Child Protection System Strengthening, Mapping and Assessment, Federal Capital Territory.

UNICEF Nigeria UNICEF Nigeria. [Date unpublished]. Child Safety Checklist.

UNICEF Nigeria UNICEF Nigeria. [Date unpublished]. Child Safety Guidelines.

UNICEF Nigeria UNICEF Nigeria. [Date unpublished]. Children’s Clubs. Presentation.

UNICEF Nigeria UNICEF Nigeria. [Date unpublished]. Code of Conduct for Facilitators.

UNICEF Nigeria UNICEF Nigeria. [Date unpublished]. Country Office Annual Report 2013.

UNICEF Nigeria UNICEF Nigeria. [Date unpublished]. Country Office Annual Report 2015.

UNICEF Nigeria UNICEF Nigeria. [Date unpublished]. Data Quality. Presentation.

UNICEF Nigeria UNICEF Nigeria. [Date unpublished]. Database of ALSS and MLSS.

UNICEF Nigeria UNICEF Nigeria. [Date unpublished]. Defection Area Mapping. Presentation.

UNICEF Nigeria UNICEF Nigeria. [Date unpublished]. Defection Area Transect Walk. Presentation.

UNICEF Nigeria UNICEF Nigeria. [Date unpublished]. Draft Pre-test for VHP TOT Workshop.

UNICEF Nigeria UNICEF Nigeria. [Date unpublished]. Edo State List of Activities Implemented under EU/NDSP Rural Component.

UNICEF Nigeria UNICEF Nigeria. [Date unpublished]. Enugu Field Office Nutrition Multi-Year Workplan 2014-2015.

UNICEF Nigeria UNICEF Nigeria. [Date unpublished]. Establishment of WASHCOMs. Presentation.

UNICEF Nigeria UNICEF Nigeria. [Date unpublished]. Expectations and Fears of WASHCOM training. Presentation.

UNICEF Nigeria UNICEF Nigeria. [Date unpublished]. Exploring Self in Relation to our roles as patrons. Presentation.

UNICEF Nigeria UNICEF Nigeria. [Date unpublished]. Exercises on CtC Six Steps. Presentation.

UNICEF Nigeria UNICEF Nigeria. [Date unpublished]. Facilitator’s Report: Akwa IBOM State Step Down CLTS Re-Training Workshop, Eket.

UNICEF Nigeria UNICEF Nigeria. [Date unpublished]. Facilitator’s Report: Akwa IBOM State Step Down CLTS Re-Training Workshop, Otukpo.

FINAL REPORT 83

© UNICEF

SHORT REFERENCE

CITATION

UNICEF Nigeria UNICEF Nigeria. [Date unpublished]. Facilitation of DEAL Sessions. Presentation.

UNICEF Nigeria UNICEF Nigeria. [Date unpublished]. Facilitation of DEAL Sessions: Introduction to Image Work. Presentation.

UNICEF Nigeria UNICEF Nigeria. [Date unpublished]. Facilitation of DEAL Sessions: Overview of DEALS. Presentation.

UNICEF Nigeria UNICEF Nigeria. [Date unpublished]. FCC Study Pretest. Presentation.

UNICEF Nigeria UNICEF Nigeria. [Date unpublished]. Federal Government of Nigeria –UNICEF Rolling Workplan 2014-2015.

UNICEF Nigeria UNICEF Nigeria. [Date unpublished]. Federal Government of Nigeria –UNICEF Rolling Workplan 2014-2015: Quality Basic Education Programme: Kaduna, C-field Offices.

UNICEF Nigeria UNICEF Nigeria. [Date unpublished]. Federal Government of Nigeria –UNICEF Rolling Workplan 2014-2015: Quality Basic Education Programme: Lagos, B-field Offices.

UNICEF Nigeria UNICEF Nigeria. [Date unpublished]. FGD Moderation Orientation. Presentation.

UNICEF Nigeria UNICEF Nigeria. [Date unpublished]. Flipchart for Promotion of Hygiene in the Communities.

UNICEF Nigeria UNICEF Nigeria. [Date unpublished]. Field Office Report for Cluster Level Training of SBMCs in Zamfara and Sokoto States.

UNICEF Nigeria UNICEF Nigeria. [Date unpublished]. Field Support: FCC Study. Presentation.

UNICEF Nigeria UNICEF Nigeria. [Date unpublished]. Final evaluation by facilitator.

UNICEF Nigeria UNICEF Nigeria. [Date unpublished]. Final evaluation with participants.

UNICEF Nigeria UNICEF Nigeria. [Date unpublished]. Forward Cold Chain Assessment. Presentation.

UNICEF Nigeria UNICEF Nigeria. [Date unpublished]. Gbonyin list of activities.

UNICEF Nigeria UNICEF Nigeria. [Date unpublished]. Good Facilitation Skills. Presentation.

UNICEF Nigeria UNICEF Nigeria. [Date unpublished]. Guidelines for Hygiene Promotion ‘in and through’ schools in Nigeria.

UNICEF Nigeria UNICEF Nigeria. [Date unpublished]. Guidelines for Hygiene Promotion in Community and Rural Markets in Nigeria.

UNICEF Nigeria UNICEF Nigeria. [Date unpublished]. Guidelines for Hygiene Promotion in Community and Rural Markets. Presentation.

UNICEF Nigeria UNICEF Nigeria. [Date unpublished]. Group 6 Work – Bayelsa State M&E Framework.

UNICEF Nigeria UNICEF Nigeria. [Date unpublished]. Health DCTs 2015-2016.

UNICEF Nigeria UNICEF Nigeria. [Date unpublished]. Health – DCT Monitoring Report by Implementing Partner Ondo SPHCDA.

UNICEF Nigeria UNICEF Nigeria. [Date unpublished]. Household Medical Expenses. Presentation.

84 FINAL REPORT

© UNICEF

SHORT REFERENCE

CITATION

UNICEF Nigeria UNICEF Nigeria. [Date unpublished]. Hygiene Promotion. Presentation.

UNICEF Nigeria UNICEF Nigeria. [Date unpublished]. I Deal: Training Module for Facilitators.

UNICEF Nigeria UNICEF Nigeria. [Date unpublished]. Improved Sanitation Facilities. Presentation.

UNICEF Nigeria UNICEF Nigeria. [Date unpublished]. Introduction to FGM. Presentation.

UNICEF Nigeria UNICEF Nigeria. [Date unpublished]. Introduction to the revised EPI data tools. Presentation.

UNICEF Nigeria UNICEF Nigeria. [Date unpublished]. Instruction for Shake Test.

UNICEF Nigeria UNICEF Nigeria. [Date unpublished]. Katsina, ToT Workshop on Development of Water Safety Plans in Communities.

UNICEF Nigeria UNICEF Nigeria. [Date unpublished]. Kaduna Field Office Nutrition Multi-Year Workplan 2014-2015.

UNICEF Nigeria UNICEF Nigeria. [Date unpublished]. Key Attributes of a Good Facilitator. Presentation.

UNICEF Nigeria UNICEF Nigeria. [Date unpublished]. M&E Fundamentals. Presentation.

UNICEF Nigeria UNICEF Nigeria. [Date unpublished]. Monitoring and Evaluation. Presentation.

UNICEF Nigeria UNICEF Nigeria. [Date unpublished]. Monitoring and Evaluation with RapidPro of the Management of Severe Acute Malnutrition Programme: Rapid Pro, Nutrition Training. Presentation.

UNICEF Nigeria UNICEF Nigeria. [Date unpublished]. Module evaluation with participations.

UNICEF Nigeria UNICEF Nigeria. [Date unpublished]. Module evaluation by facilitator.

UNICEF Nigeria UNICEF Nigeria. [Date unpublished]. My Own Goal Description and Line.

UNICEF Nigeria UNICEF Nigeria. [Date unpublished]. National ODF Certification Protocol. Revised Edition.

UNICEF Nigeria UNICEF Nigeria. [Date unpublished]. Nigeria Country Programme Document 2014-2017.

UNICEF Nigeria UNICEF Nigeria. [Date unpublished]. Nigeria 2014-2017 outcome outputs post-2015.

UNICEF Nigeria UNICEF Nigeria. [Date unpublished]. Nigeria 2014-2017 outcome outputs pre-2015.

UNICEF Nigeria UNICEF Nigeria. [Date unpublished]. Nigeria Results Framework.

UNICEF Nigeria UNICEF Nigeria. [Date unpublished]. Niger State SBMC Effectiveness Monitoring General Report.

UNICEF Nigeria UNICEF Nigeria. [Date unpublished]. Nutrition – DCTs 2015-2016.

UNICEF Nigeria UNICEF Nigeria. [Date unpublished]. Outcome/Output name.

UNICEF Nigeria UNICEF Nigeria. [Date unpublished]. Organogram.

UNICEF Nigeria UNICEF Nigeria. [Date unpublished]. Orientation Workshop Hygiene Promotion in and Through Schools: Components of WASH in Schools: 5-Star Approach. Presentation.

FINAL REPORT 85

© UNICEF

SHORT REFERENCE

CITATION

UNICEF Nigeria UNICEF Nigeria. [Date unpublished]. Orientation Workshop Hygiene Promotion in and Through Schools: Group Work on Stakeholder Roles and Responsibilities for WASH in Schools. Presentation.

UNICEF Nigeria UNICEF Nigeria. [Date unpublished]. Ovia South West Activity Training List.

UNICEF Nigeria UNICEF Nigeria. [Date unpublished]. Participant’s own goals and reporting form.

UNICEF Nigeria UNICEF Nigeria. [Date unpublished]. Participatory Monitoring and FU. Presentation.

UNICEF Nigeria UNICEF Nigeria. [Date unpublished]. PCRs and IRs.

UNICEF Nigeria UNICEF Nigeria. [Date unpublished]. Performance Assessment Checklist for WASH in Primary Schools in Nigeria.

UNICEF Nigeria UNICEF Nigeria. [Date unpublished]. PMTCT Costing Summary report.

UNICEF Nigeria UNICEF Nigeria. [Date unpublished]. Practical Session on Data Tool.

UNICEF Nigeria UNICEF Nigeria. [Date unpublished]. Pre-Implementation Checklist: Maternal Newborn and Child Health Week.

UNICEF Nigeria UNICEF Nigeria. [Date unpublished]. Pre-test on use of Zinc supplement on Home management of diarrhea.

UNICEF Nigeria UNICEF Nigeria. [Date unpublished]. Pre-Training Assessment for Health Workers.

UNICEF Nigeria UNICEF Nigeria. [Date unpublished]. Procurement Training Trip Report.

UNICEF Nigeria UNICEF Nigeria. [Date unpublished]. Progress Report, Issues, Challenges, Recommendation and Way Forward on Hands on Contracts on New Latrine Designs in Kolga.

UNICEF Nigeria UNICEF. [Date unpublished]. Proposal for a 3-Day Zonal Training of Trainers State Level Implementing Partners on Emergency Preparedness and Disaster Risk Reduction.

UNICEF Nigeria UNICEF Nigeria. [Date unpublished]. Proposal for Training of LGA and State WASH Staff on WASH Information Management System.

UNICEF Nigeria UNICEF Nigeria. [Date unpublished]. Protocol for Verification and Certification of Open Defection Free and Total Sanitation Communities. Presentation.

UNICEF Nigeria UNICEF Nigeria. [Date unpublished]. Psychosocial Care and Support. Presentation.

UNICEF Nigeria UNICEF Nigeria. [Date unpublished]. Report Akuku-Toru LGA Institutional Diagnosis.

UNICEF Nigeria UNICEF Nigeria [Date unpublished]. Report of Borno Intensive IPC Skills Training to Build Resilience.

UNICEF Nigeria UNICEF Nigeria. [Date unpublished]. Research Assistants’ database.

UNICEF Nigeria UNICEF Nigeria. [Date unpublished]. Research Assistants’ database – Oyo State.

UNICEF Nigeria UNICEF Nigeria. [Date unpublished]. Running a Children’s Club. Presentation.

UNICEF Nigeria UNICEF Nigeria. [Date unpublished]. Sanitation Ladder Promotion. Presentation.

UNICEF Nigeria UNICEF Nigeria. [Date unpublished]. Session 1: Workshop Introduction. Facilitation Notes.

86 FINAL REPORT

© UNICEF

SHORT REFERENCE

CITATION

UNICEF Nigeria UNICEF Nigeria. [Date unpublished]. Session 4: Operationalizing Children’s Clubs. Facilitation Notes.

UNICEF Nigeria UNICEF Nigeria. [Date unpublished]. Session 5: Running a Children’s Club. Facilitation Notes.

UNICEF Nigeria UNICEF Nigeria. [Date unpublished]. Session 6: Working with Children Through a Children’s Club. Facilitation Notes.

UNICEF Nigeria UNICEF Nigeria. [Date unpublished]. Session 7: Exploring Self in relation to our roles as club patrons.

UNICEF Nigeria UNICEF Nigeria. [Date unpublished]. Shit & Food Demonstration. Presentation.

UNICEF Nigeria UNICEF Nigeria. [Date unpublished]. SITAN-SKAP for FGM in 6 States of Nigeria: Goals and Objectives. Presentation.

UNICEF Nigeria UNICEF Nigeria. [Date unpublished]. SITAN KAP: Overview and Methodology. Presentation.

UNICEF Nigeria UNICEF Nigeria. [Date unpublished]. Situation Assessment and Analysis, Knowledge Attitude and Practical/Qualitative Study on Norms Surrounding FGM in Nigeria: Case Studies.

UNICEF Nigeria UNICEF Nigeria. [Date unpublished]. Situation Assessment and Analysis, Knowledge Attitude and Practical/Qualitative Study on Norms Surrounding FGM in Nigeria: FGD Guide.

UNICEF Nigeria UNICEF Nigeria. [Date unpublished]. Situation Assessment and Analysis, Knowledge Attitude and Practical/Qualitative Study on Norms Surrounding FGM in Nigeria: Final Qualitative Questionnaire.

UNICEF Nigeria UNICEF Nigeria. [Date unpublished]. Situation Assessment and Analysis, Knowledge Attitude and Practical/Qualitative Study on Norms Surrounding FGM in Nigeria: KII Guide.

UNICEF Nigeria UNICEF Nigeria. [Date unpublished]. Standard Implementing Procedures: Forward Cold Chain Evaluation.

UNICEF Nigeria UNICEF Nigeria. [Date unpublished]. Stakeholders Workshop on the Development of Bayelsa State M&E.

UNICEF Nigeria UNICEF Nigeria. [Date unpublished]. Step by Step Guidelines to Implement CLTS. Presentation.

UNICEF Nigeria UNICEF Nigeria. [Date unpublished]. Step-by-Step Training Manual on Polio and Immunization: A Media Reporting Guide.

UNICEF Nigeria UNICEF Nigeria. [Date unpublished]. Strengths and Difficulties.

UNICEF Nigeria UNICEF Nigeria. [Date unpublished]. Summary Results Matrix: Federal Government of Nigeria and UNICEF Country Programme, 2014-2017.

UNICEF Nigeria UNICEF Nigeria. [Date unpublished]. Three Day Training for Assessors and Social Welfare Officers Working in the Family Courts. Nigeria.

UNICEF Nigeria UNICEF Nigeria. [Date unpublished]. Training on School Sanitation and Hygiene Education. Presentation.

FINAL REPORT 87

© UNICEF

SHORT REFERENCE

CITATION

UNICEF Nigeria UNICEF Nigeria. [Date unpublished]. Training Curriculum: Integrated Health Training Package for Volunteer Social Mobilizers.

UNICEF Nigeria UNICEF Nigeria. [Date unpublished]. Training Manual on Hygiene Promotion for Volunteer Community Hygiene Promoters.

UNICEF Nigeria UNICEF Nigeria. [Date unpublished]. Training Plan for SHAWN States.

UNICEF Nigeria UNICEF Nigeria. [Date unpublished]. Training of Trainers Workshop on Water Safety Planning. Presentation.

UNICEF Nigeria UNICEF Nigeria. [Date unpublished]. Vaccine Management Training Manual for EPI Coordinators and Focal Person.

UNICEF Nigeria UNICEF Nigeria. [Date unpublished]. VCM/VWS Post-Test for Training of Trainers on Integrated Health Manual.

UNICEF Nigeria UNICEF Nigeria. [Date unpublished]. WASHCOM Functionality Checklist.

UNICEF Nigeria UNICEF Nigeria. [Date unpublished]. WSP Logic.

UNICEF Nigeria UNICEF Nigeria. [Date unpublished]. WSP Selected Communities.

UNICEF Nigeria UNICEF Nigeria. [Date unpublished]. Workplan 2014: Child Protection.

UNICEF Nigeria UNICEF Nigeria. [Date unpublished]. Working with Children through a Children’s club 1 and 2. Presentation.

UNICEF Nigeria 2010

UNICEF Nigeria. 2010. Inclusion of Zinc in the Household Management of Childhood Diarrhea in Nigeria: Training Manual. October 2010.

UNICEF Nigeria 2011

UNICEF Nigeria. 2011. Zinc and Low Osmolarity ORS in the Management of Childhood Diarrhoea in Nigeria. June 2011.

UNICEF Nigeria 2012

UNICEF Nigeria. 2012. Situation Analysis of Children and Women in Nigeria. 2011 Update. September 2012.

UNICEF Nigeria 2013

UNICEF Nigeria. 2013. Federal Government of Nigeria- UNICEF: WASH Multi-Year Work Plan 2014-15. March 2014.

UNICEF Nigeria 2013

UNICEF Nigeria. 2013. SAM Reports Training for the Integrated Management of Acute Malnutrition Training Agenda, Kano State, Nigeria. July 2013.

UNICEF Nigeria 2013

UNICEF Nigeria. 2013. Sentinel Surveillance for new vaccines. Presentation.

UNICEF Nigeria 2014

UNICEF Nigeria. 2014. Accountability Framework. December 2014.

UNICEF Nigeria 2014

UNICEF Nigeria. 2014. DSA Payment – Sitan KAP – Ekiti. October 2014.

UNICEF Nigeria 2014

UNICEF Nigeria. 2014. Field Work Manual for Quantitative Survey. October 2014.

88 FINAL REPORT

© UNICEF

SHORT REFERENCE

CITATION

UNICEF Nigeria 2014

UNICEF Nigeria. 2014. Federal Government of Nigeria –UNICEF Rolling Workplan 2014-2015: Quality Basic Education Programme: Bauch, D-field Offices.

UNICEF Nigeria 2014

UNICEF Nigeria. 2014. Federal Government of Nigeria –UNICEF Rolling Workplan 2014-2015: Communication for Development.

UNICEF Nigeria 2014

UNICEF Nigeria. 2014. Federal Government of Nigeria –UNICEF Rolling Workplan 2014-2015: Communication and Advocacy.

UNICEF Nigeria 2014

UNICEF Nigeria. 2014. Federal Government of Nigeria –UNICEF Rolling Workplan 2014-2015: Emergency and Disaster Risk Reduction.

UNICEF Nigeria 2014

UNICEF Nigeria. 2014. Federal Government of Nigeria –UNICEF Rolling Workplan 2014-2015: Children and AIDS.

UNICEF Nigeria 2014

UNICEF Nigeria. 2014. Federal Government of Nigeria-UNICEF Rolling Workplan 2014-15: Child Protection.

UNICEF Nigeria 2014

UNICEF Nigeria. 2014. Federal Government of Nigeria – UNICEF Rolling Workplan 2014-15, Health Section. February 2014.

UNICEF Nigeria 2014

UNICEF Nigeria. 2014. Federal Government of Nigeria –UNICEF Rolling Workplan 2014-2015: Quality Basic Education Programme: Enugu, A-field Offices.

UNICEF Nigeria 2014

UNICEF Nigeria. 2014. Federal Government of Nigeria –UNICEF Rolling Workplan 2014-2015: Social Policy.

UNICEF Nigeria 2014

UNICEF Nigeria. 2014. Federal Government of Nigeria-UNICEF Rolling Workplan 2014-15: Social Policy, Analysis, Research and Communication.

UNICEF Nigeria 2014

UNICEF Nigeria. 2014. Pre and Post Abia State IDPS. February 2014.

UNICEF Nigeria 2014

UNICEF Nigeria. 2014. Report on Training of Research Assistants in Osun and Ekiti States. October 2014.

UNICEF Nigeria 2014

UNICEF Nigeria. 2014. Rolling Management Plan 2014-2015. April 2014.

UNICEF Nigeria 2015

UNICEF. 2015. Advocacy: Building a Strategy, Maximizing Impact. Presentation. August 2015.

UNICEF Nigeria 2015

UNICEF Nigeria. 2015. Benue State Emergency Profile 2016. November 2015.

UNICEF Nigeria 2015

UNICEF Nigeria. 2015. Ebonyi State WASH Emergency Profile 2015 Update. October 2015.

UNICEF Nigeria 2015

UNICEF Nigeria. 2015. Enugu State WASH Emergency Profile 2015. October 2015.

UNICEF Nigeria 2015

UNICEF Nigeria. 2015. Facilitating Psychosocial Care and Support through Children’s clubs: Workshop Introduction. October 2015.

UNICEF Nigeria 2015

UNICEF Nigeria. 2015. Implementation Report to n a 3-Day Zonal Training of Trainers of State Level Workshop on Emergency Preparedness response and disaster risk reduction.

FINAL REPORT 89

© UNICEF

SHORT REFERENCE

CITATION

UNICEF Nigeria 2015

UNICEF Nigeria. 2015. IMO State WASH Emergency Profile 2016. October 2015.

UNICEF Nigeria 2015

UNICEF Nigeria. 2015. Learning and Development Strategy for UNICEF Nigeria -2016. November 2015.

UNICEF Nigeria 2015

UNICEF Nigeria. 2015. Procurement Training Agenda. November 2015.

UNICEF Nigeria 2015

UNICEF Nigeria. 2015. Rivers State WASH Emergency Profile 2016. October 2015.

UNICEF Nigeria 2015

UNICEF Nigeria. 2015. Rolling Management Plan 2015-2016. February 2015.

UNICEF Nigeria 2015

UNICEF Nigeria. 2015. Summary of Steps for Procurement Contract Implementation and Management. Presentation. November 2015.

UNICEF Nigeria 2015

UNICEF Nigeria. 2015. Tentative Agenda for a 4-day ToT on EPR and DRR. October 2015.

UNICEF Nigeria 2016

UNICEF Nigeria. 2016. 2015 Achievements by Field Offices and Targets for 2016. January 2016.

UNICEF Nigeria 2016

UNICEF Nigeria. 2016. 2016 Milestones for Monitoring Achievement of Outputs. January 2016.

UNICEF Nigeria 2016

UNICEF Nigeria. 2016. Case Study -2: WASH Committees drive birth registration and immunization of children in their communities in Benue & Jigawa States in Nigeria.

UNICEF Nigeria 2016

UNICEF Nigeria. 2016. Early Childhood Development Teaching Methodologies.

UNICEF Nigeria 2016

UNICEF Nigeria. 2016. ECD Participant’s Handout.

UNICEF Nigeria 2016

UNICEF Nigeria. 2016. Facilitator’s Guide: Three Day Training on the Family Courts, for Family Court Assessors and Social Welfare Officers. 2016.

UNICEF Nigeria 2016

UNICEF Nigeria. 2016. Federal Government of Nigeria – UNICEF Rolling Workplan 2016-2017: Child Protection.

UNICEF Nigeria 2016

UNICEF Nigeria. 2016. Federal Government of Nigeria – UNICEF Rolling Workplan 2016-2017: Communication for Development.

UNICEF Nigeria 2016

UNICEF Nigeria. 2016. Federal Government of Nigeria – UNICEF Rolling Workplan 2016-2017: Education.

UNICEF Nigeria 2016

UNICEF Nigeria. 2016. Federal Government of Nigeria – UNICEF Rolling Workplan 2016-2017: EPR and DRR.

UNICEF Nigeria 2016

UNICEF Nigeria. 2016. Federal Government of Nigeria – UNICEF Rolling Workplan 2016-2017: Health.

UNICEF Nigeria 2016

UNICEF Nigeria. 2016. Federal Government of Nigeria – UNICEF Rolling Workplan 2016-2017: HIV.

90 FINAL REPORT

© UNICEF

SHORT REFERENCE

CITATION

UNICEF Nigeria 2016

UNICEF Nigeria. 2016. Federal Government of Nigeria – UNICEF Rolling Workplan 2016-2017: Nutrition.

UNICEF Nigeria 2016

UNICEF Nigeria. 2016. Federal Government of Nigeria – UNICEF Rolling Workplan 2016-2017: Social Policy.

UNICEF Nigeria 2016

UNICEF Nigeria. 2016. Federal Government of Nigeria – UNICEF Rolling Workplan 2016-2017: Social Policy, Analysis, Research and Communication.

UNICEF Nigeria 2016

UNICEF Nigeria. 2016. Federal Government of Nigeria – UNICEF Rolling Workplan 2016-2017: WASH.

UNICEF Nigeria 2016

UNICEF Nigeria. 2016. Nutrition Programme 2016 Mid-Year Review Report Kebbi State.

UNICEF Nigeria 2016

UNICEF Nigeria. 2016. Nutrition Programme 2016 Mid-Year Review Report Zamfara State.

UNICEF Nigeria 2016

UNICEF Nigeria. 2016. Objectives of the Training (EU and SHAWN Projects). Presentation. February 2016.

UNICEF Nigeria 2016

UNICEF Nigeria. 2016. Participant’s handbook: Three Day Training on the Family Courts, for Family Court Assessors and Social Welfare Officers. 2016.

UNICEF Nigeria 2016

UNICEF Nigeria. 2016. Programme Implementation Details by Grant. September 2016.

UNICEF Nigeria 2016

UNICEF Nigeria. 2016. Re: Retirement of UNICEF Support Activities in Ekiti State. September 2016.

UNICEF Nigeria 2016

UNICEF Nigeria. 2016. State Health Programme 2016 Mid-Year Review Report Kebbi State.

UNICEF Nigeria 2016

UNICEF Nigeria. 2016. State Health Programme 2016 Mid-Year Review Report Sokoto State.

UNICEF Nigeria 2016

UNICEF Nigeria. 2016. State Health Programme 2016 Mid-Year Review Report Zamfara State.

UNICEF Nigeria 2016

UNICEF Nigeria. 2016. Submission of Documents. Child Protection. October 2016.

UNICEF Nigeria 2016

UNICEF Nigeria. 2016. Training of Teachers/Caregivers on Reggio and other Child Centered Approaches. June 2016.

UNICEF Nigeria 2016

UNICEF Nigeria. 2016. Tracking WSP Roll-out Achievement per state. June 2016.

UNICEF Nigeria 2016

UNICEF Nigeria. 2016. UNICEF WSP Agenda. February 2016.

UNICEF Nigeria 2016

UNICEF Nigeria. 2016. Updated Agenda for WSP Step down training in Bauchi State. August 2016.

UNICEF Nigeria 2016

UNICEF Nigeria. 2016. WASH Programme 2016 Mid-Year Review Report Kebbi State.

FINAL REPORT 91

© UNICEF

SHORT REFERENCE

CITATION

UNICEF Nigeria 2016

UNICEF Nigeria. 2016. WASH Programme 2016 Mid-Year Review Report Sokoto State.

UNICEF Nigeria 2016

UNICEF Nigeria. 2016. WASH Programme 2016 Mid-Year Review Report Zamfara State.

UNICEF Nigeria 2017

UNICEF Nigeria. 2017. Government/UNICEF Annual workplan for 2017 Kebbi State.

UNICEF Nigeria 2017

UNICEF Nigeria. 2017. Government/UNICEF Annual workplan for 2017 Sokotoi State.

UNICEF Nigeria 2017

UNICEF Nigeria. 2017. Government/UNICEF Annual workplan for 2017 Zamfara State.

Nwangu 2014 Nwangu, Rosemary. 2014. Review of GEP3 Head Teacher Training Intervention. December 2014.

SC and Government of Nigeria 2013

Save the Children and Federal Government of Nigeria. 2013. Workshop Report: Training of Health workers from Bwari FCT PHCs on Helping Babies Breathe and Essential Newborn Care.

Robinson et al. 2015

Robinson, Wammanda et al. 2015. Kaduna State ICCM Training of Trainer Workshop. October 2015.

Rogers 2014 Rogers, Patricia. 2014. Theory of Change. Methodological Briefs. Impact Evaluation No. 2. UNICEF Office of Research. 2014.

92 FINAL REPORT

© UNICEF

Appendix IX List of Trainings

TRAINING SESSIONS TRAININGS IDENTIFIED THROUGH SURVEY32

DOCUMENT REVIEW

ARC+GIS training 2 x

ASC monitoring 7

Baseline Training 1 x

Baby Friendly Hospital Initiative 4

Breast milk substitute x

CAD Training 3

Capacity building of care givers 55

Case Management 1

Centre Based Management Committee 20 x

Child friendly school 1

Child Investigation 2

Child Justice system for legal Aid providers 4 x

Child Protection System Strengthening Training x

Community Led Total Sanitation 96 x

Cluster training 1

Community-based Management of Acute Malnutrition 31

Cold chain training 37 x

Communication for development 2

Conduct of annual school census 10 x

Construction of latrines x

Consultative Meeting with Key stakeholders on Model of Child Protection System

x

Content of Oral Rehydration Therapy 27

COPES Training 3 x

Child Protection System Strengthening 23

32 We asked survey respondents: « Briefly, what training session did you participate in? Describe it by its title, if you remember, or by briefly indicating who was trained on what topic. »

FINAL REPORT 93

© UNICEF

TRAINING SESSIONS TRAININGS IDENTIFIED THROUGH SURVEY32

DOCUMENT REVIEW

Data entry clerks training 2 x

Data Management 45

DCT 4

Development of Sector-Wide Approach to Programme (SWAP) 2015, 2016

x

District Health Information System 1

Disaster Risk Management 1 x

Education Management Information System 4

Enrolment Drive Campaign (EDC) 10

Emergency Preparedness and Response 56

Exclusive breast feeding initiative 4

Extended guideline on WASHCOM 42 x

Facility Mapping and Tracking 7 x

Girls Child Education Issue 1

Harmonized Approach to Cash Transfer 96 x

Hard to Reach Programme 5

Head Teachers Leadership Training 33 x

Health Education 1

HIF 59

HIP Training 4

HTR x

Hygiene promotion training of lead trainer x

Hygiene Promotion volunteers 1 x

Integrated Community Case Management x

Integrated Community Case Management TOT 16

Information and Communication Technology 56

Integrated Measles Campaign x

Integrated Maternal Child Illness 74 x

Integrated Maternal Child Illness –Plus (Vaccine Management Component of IMCI)

54

Integrated management of childhood disease 1

94 FINAL REPORT

© UNICEF

TRAINING SESSIONS TRAININGS IDENTIFIED THROUGH SURVEY32

DOCUMENT REVIEW

Investment Plan 2

Inactivated Polio Vaccine 37 x

Integrated Quranic School Facilitators Trainers 6 x

Integrated Support Supervision 4

Infant and Young Child Feeding 131 x

Training on Knowledge, Attitude and Practice x

Key household practices 39

Local area machining 2

M&E Training 7

Maternal Audit 2

Measles Training 36

Menstrual hygiene Management 54

Maternal and Neo-Natal Child Health 39 x

Maternal and Neo-Natal Child Health Week 1

Maternal Neo-Natal Tetanus Elimination x

Mobilization on RI activities 4

National Immunization Plus Days 35 x

Nutrition in emergency 36

Open Defecation Free 2 x

PCV 37

PENTA 15

PHC review meeting 3

Polio eradication 2

Preparation of blended complementary foods 34

Prevention of diarrhea in children 22

Procurement 2 x

Proposal Development 42

Pump Training 2

Rapid Pro Survey 24

FINAL REPORT 95

© UNICEF

TRAINING SESSIONS TRAININGS IDENTIFIED THROUGH SURVEY32

DOCUMENT REVIEW

Refresher Training of STGS Members 0n New Protocol of ODF Certification

x

Result-based management 1

Road map on open defecation eradication 54

Rural Development on Education 1

Severe Acute Malnutrition 2

Satiation marketing x

SBCC School Management Board Committees 1

SC 3

Score card tools review 1

Self-Assessment Workshop on the implementation of the Child Rights Law of Lagos 2007

x

Social welfare costing 3

Social Welfare professionalization Bill x

SSHE 55

State level training on MNCW 37

State strategic development plan for JGSG 4

Stock Management 2

Switch (TOPV to BOPV) 35

Training for Lagos State Family Courts Judges and Magistrates

x

Training of Community Facilitators for Installation and use of Tippy Tap in Communities

x

Training of Teachers on Reggio Emillia Pedagogy and other Child Centred Approach to Training

x

Training on dehydration of a child 27

Training on how to use dashboard 4

Training on Implementation of Birth Registration during the December 2015 MNCH in Osun State

x

Training on solar refrigerator 3

Triggering Training 14 x

UNICEF staff’s logistic management 1

Using of Mid Upper Arm Circumference Tape (MUAC) tape 2

96 FINAL REPORT

© UNICEF

TRAINING SESSIONS TRAININGS IDENTIFIED THROUGH SURVEY32

DOCUMENT REVIEW

Violence Against Children 1

Training for Volunteer Community Mobilizers 4

VCT x

Training of Volunteer Hygiene Promoters 2

Violence Against Children x

VLOM 4

Vaccine Management Training 43 x

Water, Sanitation and Hygiene in schools x

Water, Sanitation and Hygiene Committee Management Training

40 x

Water, Sanitation and Hygiene Information Management Systems

59 x

Water Safety Planning 2

Work Plan 3

Refresher Training for Trainers on Whole School Development Planning

1 x

TOT for Development Community Water Safety Planning 67 x

FINAL REPORT 97

© UNICEF

Appendix X Table of Changes to the Original Evaluation Questions in the TORs

EVALUATION FOCI

ORIGINAL TORS QUESTIONS PLACE IN THE NEW MATRIX

Impact Are the long-term impact and outcomes clearly identified and are the proposed steps towards achieving these clearly defined and meet standards for Results-Based Management?

MOVED & REFORMULATED

This is not an impact question but a question about adequate M&E of training investments. It has been moved to “Relevance & Design” and reformulated as “To what extent were training activities consistently and coherently monitored?” (with sub-questions).

Efficiency How much for training cost per unit per type training?33

REFORMULATED

The new phrasing is “How much did training activities cost?”, with references to unit, type, etc. in the indicators.

Where trainings conducted in the most cost effective way?

MAINTAINED

To what extent have the investments in training led to achieving results for children?

MOVED & GROUPED

This is not an efficiency but an effectiveness question. It has been grouped with the question on Kirckpatrick Level 4 – Results of Training Activities.

To what extent do partners contribute to training cost?

MAINTAINED

Relevance To what extent do training activities respond to Nigeria National and sub national capacity gaps?

MAINTAINED

Are theories of change of the sectoral training efforts available, sound and consistent with gaps to be addressed?

Were capacity gaps identified and assessed and training strategies developed with ongoing monitoring?

DELETED

No sectoral theories of change for training efforts are available.

Sub-question: GROUPED

This sub-question was grouped with “Were capacities (training) assets and needs assessed?” and “How were training activities monitored?”

33 When fiscal years are not specified, this means that we found no differences between annual work plans from a single section.

98 FINAL REPORT

© UNICEF

EVALUATION FOCI

ORIGINAL TORS QUESTIONS PLACE IN THE NEW MATRIX

Were the five key steps in integrating capacities (trainings) into the planning process followed?

Were partners and communities engaged to build consensus and promote national and local ownership?

Were partners and communities jointly assessed for capacity (training) gaps?

Were partners and communities jointly designed training strategies?

Did UNICEF support the implementation of capacity development interventions as planned?

How capacity development efforts provided were evaluated or assessed?

MAINTAINED

Sub-questions: REPHRASED / MOVED

The sub-questions were rephrased to better align with the five key UNDG steps.

Sub-questions 4 is now covered under effectiveness; Sub-question 5 is now covered under the question “To what extent were training activities consistently and coherently monitored?”.

To what extent was the analysis of the individual, Institutional and Enabling level capacity gaps based on the immediate, underlying and basic causes of the training gaps?

GROUPED

This sub-question was grouped with “Were capacities (training) assets and needs assessed?”

To what extent, capacity need assessment was undertaken to inform the design of capacity building?

GROUPED

This sub-question was grouped with “Were capacities (training) assets and needs assessed?”

To what extent were capacity components as defined by UNDG (human resources; public sector accountability; access to information, development knowledge and technology; inclusion, participation, equity and empowerment; financial resources; material resources; environmental resources; and external/international relations have been covered in the individual trainings) incorporated in the trainings, as relevant?

MAINTAINED

(now as a sub-question)

Effectiveness To what extent were the trainings efforts and strategies effective?

To what extend does the country programme ensure that support for strengthening individual capacities was given at significant scale?

To assess to which extent the training approaches/activities contributed effectively to capacity development?

Were objectives of trainings and expected results achieved?

What may have been the barriers and bottle neck that impeded the achievement of results?

Were objectives of capacity building and expected results achieved? And Why?

REFORMULATED

This question (and sub-questions) were reorganized under the question “Were training activities delivered as planned?” (and its sub-questions), “What are the strengths and weaknesses of the training programmes?” (and its sub-questions), and “To what extent was the training effective, with respect to the four level of Kirkpatrick model?” (and its sub-questions).

FINAL REPORT 99

© UNICEF

EVALUATION FOCI

ORIGINAL TORS QUESTIONS PLACE IN THE NEW MATRIX

Were they addressing the training needs comprehensively and holistically?

MOVED & GROUPED

This is a Relevance question, not Effectiveness. It is now covered under “To what extent did training activities respond to Nigeria’s National and sub-national capacity gaps?”

To what extent was the training effective, with respect to the four level of Kirkpatrick model?

MAINTAINED

Sustainability What mechanisms were set up to foster sustainable capacity development?

MAINTAINED

(now as a sub-question)

To what extent do individual capacity development efforts generate results that are sustainable?

MAINTAINED

(now as a sub-question)

What are the results of training supporting activities at the behavioural and organizational levels (3 and 4)?

MOVED & GROUPED

This is not a Sustainability Question, but Effectiveness. It is now covered under “To what extent was the training effective, with respect to the four level of Kirkpatrick model?”.

Was there a favorable political support for the capacity development agenda and programme to be sustained?

REFORMULATED (now as a sub-question)

“Political” was changed to “Organizational”, since in our experience the main factor affecting the sustainability of training investments is the degree of support for change at the organizational (not the political/systemic) level.

Was there a long-term budget to support training efforts and strategies?

MAINTAINED

(now as a sub-question)

100 FINAL REPORT

© UNICEF

Appendix XI The Training Cycle: From Planning to Result

FINAL REPORT 101

© UNICEF

Appendix XII Terms of Reference

102 FINAL REPORT

© UNICEF

FINAL REPORT 103

© UNICEF

104 FINAL REPORT

© UNICEF

FINAL REPORT 105

© UNICEF

106 FINAL REPORT

© UNICEF

FINAL REPORT 107

© UNICEF

108 FINAL REPORT

© UNICEF

FINAL REPORT 109

© UNICEF

110 FINAL REPORT

© UNICEF

FINAL REPORT 111

© UNICEF

112 FINAL REPORT

© UNICEF

FINAL REPORT 113

© UNICEF

114 FINAL REPORT

© UNICEF

FINAL REPORT 115

© UNICEF

Time table

116 FINAL REPORT

© UNICEF

FINAL REPORT 117

© UNICEF