LAC EDUCATION SUPPORT CONTRACT QUARTERLY REPORT 3

Fiscal Year 2020 – April 1, 2020 to June 30, 2020

Phot

o C

redi

t: U

SAID

/CA

RS A

ctiv

ity

This publication was produced for review by the United States Agency for International Development. It was prepared by DevTech Systems, Inc.

LAC EDUCATION SUPPORT CONTRACT Quarterly Report Fiscal Year 2020 - Quarter 3 April 1, 2020 to June 30, 2020

Submitted to: Barbara Knox-Seith, Contracting Officer’s Representative USAID/LAC

Submitted on: July 30, 2020

Submitted by: DevTech Systems, Inc. Contract GS-10F-0048L/7200AA18M00024

Cover Photo Caption: Participant of a Community Action for Reading and Security (CARS) project activity in the Caribbean Coast of Nicaragua. Photo Credit: Ileana Lacayo, USAID/CARS Activity 2-6038

Disclaimer:

This report is made possible by the support of the American people through the United States Agency for International Development (USAID). The contents of this report are the sole responsibility of DevTech Systems, Inc. and do not necessarily reflect the views of USAID or the United States Government.

TABLE OF CONTENTS

Acronyms 1

1. Executive Summary 2

2. Introduction 3

3. Tasks Progress 3 3.1. Task 1: Sector Assessment, Design, and Implementation Support 3

3.1.1. Rapid Education and Risk Analysis (RERA) in The Dominican Republic 3 3.1.2. Rapid Education and Risk Analysis (RERA) in Colombia 3 3.1.3. Venezuela Education Diagnostic 6

3.2. Task 2: Monitoring and Evaluation of Education Programs 6 3.3. Task 3: Preparation for Education Skills-Building Workshops 7

4. Other Reports 7

5. Lessons Learned 7

Annex A. Progress Summary of Indicators 10

1 |LAC EDUCATION SUPPORT CONTRACT: QUARTERLY REPORT Q3 FY 2020 USAID.GOV

ACRONYMS CO Contracting Officer

COR Contracting Officer’s Representative

COVID-19 Coronavirus Disease 2019

IADB Inter-American Development Bank

KIIs Key Informant Interviews

LAC Latin American and the Caribbean

LAC/RSD/EDU Education Team of the Regional Sustainable Development Office of the Bureau for Latin America and the Caribbean

LRCP LAC Reads Capacity Program

RERA Rapid Assessment Education and Risk Analysis

UCAB Universidad Católica Andres Bello

UNICEF United Nations International Children’s Fund

USAID United States Agency for International Development

VRIO Venezuela Response and Integration Office

USAID.GOV LAC EDUCATION SUPPORT CONTRACT: QUARTERLY REPORT Q3 FY 2020 | 2

1. EXECUTIVE SUMMARY The LAC Education Support Contract, as intended by USAID Bureau for Latin America and the Caribbean Office of Regional Sustainable Development (LAC/RSD), is mainly demand-driven in that both LAC/RSD and the LAC Missions will request activities and services as needed (e.g., studies, evaluations, assessments, conference organizing).

This quarterly report presents LAC Education Support Contract performance and progress of the three tasks, covering the period from April 1 to June 30, 2020.

• Under Task 1, Sector Assessment, DevTech received approval from USAID on the Spanish version of the Rapid Assessment Education and Risk Analysis (RERA) report conducted in the Dominican Republic. DevTech also conducted planning, remote data collection and drafting of the Colombia RERA during this period. Also, a new Activity, the Venezuela Education Diagnostic commenced in mid-June.

• Under Task 2, Monitoring and Evaluation of Education Programs, there were no activities conducted in this period.

• Under Task 3--Preparation for Education Skills-Building Workshops, there were no activities conducted in this period.

By the end of the third quarter of the Fiscal Year 2020, one sector assessment was completed, another one is being implemented in Colombia, and a third one commenced in Venezuela. The production and distribution of dissemination and communication documents for an Activity was supported and one performance evaluation of education programs was carried out.

3 |LAC EDUCATION SUPPORT CONTRACT: QUARTERLY REPORT Q3 FY 2020 USAID.GOV

2. INTRODUCTION The LAC Education Support Contract, awarded to DevTech Systems, Inc., began implementation in October 2018, with the objective to provide on-demand support services to the Education Team in the USAID Bureau for Latin America and the Caribbean, Office of Regional Sustainable Development (LAC/RSD/EDU). The LAC Education Support Contract is funded by USAID/LAC and ends on September 2023.

The Contract is divided in three main tasks: 1) Sector Assessment, Design, and Implementation Support, 2) Monitoring and Evaluation of Education Programs, and 3) Preparation for Education Skills-Building Workshops. DevTech will work with USAID/LAC and bilateral Missions in the region to provide rapid access to cost-effective expert technical assistance and logistical services, as required, to strengthen the design, management, monitoring, and evaluation of regional and bilateral education programs in Latin America and the Caribbean.

This quarter DevTech finalized and received approval on the Spanish translation of the RERA Dominican Republic Spanish Report under Task 1. Also, for the Rapid Education and Risk Assessment (RERA) in Colombia that started last quarter, remote data collection was conducted, and a first draft of the report was submitted to USAID for review and comment. Lastly, DevTech initiated work on a new Activity, an Education Diagnostic in Venezuela, which hopes to provide an illustrative snapshot of education inputs and outputs in Venezuela, focusing on basic education (pre-K through 12th grade). Annex A contains information on Progress to Targets.

3. TASKS PROGRESS

3.1. TASK 1: SECTOR ASSESSMENT, DESIGN, AND IMPLEMENTATION SUPPORT

3.1.1. RAPID EDUCATION AND RISK ANALYSIS (RERA) IN THE DOMINICAN REPUBLIC

During this reporting period, USAID/Dominican Republic provided concurrence on the Spanish Version of Rapid Education and Risk Analysis (RERA) final report. The report is now available on the Development Clearinghouse website1.

3.1.2. RAPID EDUCATION AND RISK ANALYSIS (RERA) IN COLOMBIA

During this period, DevTech contracted the RERA Team, conducted training for a locally hired data collection firm, conducted remote data collection in Colombia, provided preliminary results and a first draft of the RERA to USAID.

1 Rapid Education and Risk Analysis Report – Dominican Republic (Spanish Version):

https://pdf.usaid.gov/pdf_docs/PA00WNWJ.pdf

USAID.GOV LAC EDUCATION SUPPORT CONTRACT: QUARTERLY REPORT Q3 FY 2020 | 4

In April, DevTech brought on board a local Education Specialist, a Conflict Specialist and a Research Assistant, all based in Bogota. DevTech also hired the Centro Nacional de Consultoria, a local firm who provided facilitators, enumerators and data entry staff, to support the core RERA team on data collection. To adapt the RERA methodology and data collection tools to a remote scenario, DevTech also hired a Technical Advisor based in Massachusetts, to provide support to the team in developing an online survey to be administered to respondents in May using KoBoToolbox2. The RERA team would operate virtually from their home locations in Bogotá, New York, Washington, D.C., and Massachusetts throughout the RERA while under lockdown. DevTech also worked with USAID/Colombia and the Colombian Ministry of Education on correspondence to the 11 education secretariats (see Figure 1) in the RERA sample, to inform them of the assessment and seek their collaboration on this Activity. Secretariats were the points of contact for the RERA team to reach out to schools and school principals, to obtain contact information for parents, teachers, students and members of the school community, to whom the team would reach out to participate in the RERA. Figure 1. Sample Secretariats

In early May, the Colombia RERA Team conducted an online training in the span of four days on Primary data collection. Around 16 local individuals were trained by the team, using PowerPoint pre-recorded presentations and the application Ring Central, for group discussions. The team also carried out surveys and interview simulations to assess the trainees' readiness and address any issues or questions they had.

The RERA team in coordination with USAID/Colombia, the Ministry of Education and local education secretariats selected 24 schools in 11 cities and peri-urban municipalities. The team collaborated intensively with local education secretariats and school principals by telephone to identify over 600 respondents and obtain their contact information. Ten secretariats provided information, but one, due to additional data protection requirements on their end, were not able to participate in the survey or

2 https://www.kobotoolbox.org/

5 |LAC EDUCATION SUPPORT CONTRACT: QUARTERLY REPORT Q3 FY 2020 USAID.GOV

interview phase of the assessment. The team conducted a separate case study for the Medellin secretariat with a reduced number of participants and KIIs.

DevTech carried out the survey between May 8th and May 15th, resulting in 457 successful responses. The phone surveys were conducted by the Centro National de Consultoria staff, using their call center, and recorded responses using KoboToolBox. The RERA team conducted qualitative interviews starting May 12th with 201 of those respondents. The interviews were conducted in two weeks, using skype between the respondent, a facilitator and a note-taker.

Table 1. Number of Surveys and Interviews completed.

Secretariats No. of Registries No. of Completed Phone Surveys

No. of Completed Interviews

Barranquilla 84 68 33 Bello 48 27 9 Bogotá 32 25 8 Cali 38 27 9 Cartagena 134 92 36 Maicao 54 35 13 Riohacha 23 15 5 Soacha 70 48 19 Soledad 92 74 43 Palmira 60 46 26 Total general 635 457 201

Given that the RERA will serve to inform future education programming by USAID/Colombia, the RERA Team presented preliminary findings on the assessment during the Juntos Aprendemos co-creation workshop on June 5th. The presentation was organized by USAID/Colombia and attended by a group of prospective implementers. On June 12th, the RERA Team presented these same findings to the Education in Emergencies Cluster, in a meeting organized by UNICEF Colombia, and attended by the Ministry of Education, Save the Children, USAID, and other organizations actively working on education and humanitarian responses in Colombia. In late June, the RERA Team submitted a first draft of the assessment report to USAID for comment and feedback.

Challenges and Solutions

Due to the travel and mobility restrictions imposed by COVID-19, the RERA team implemented all steps of the RERA planning and implementation remotely, including coordination, enumerator training, and data collection. Logistically, coordination was key between the data collection firm, the team leader and DevTech’s local consultants to ensure the training for the data collection was accessible, participatory and understood by all staff. PowerPoint Presentations were recorded and made available via google drive to Centro Nacional de Consultoria staff, and the RERA Team conducted simulations to assess the readiness of the trainees. The Team spent additional time with a few staff, who performed poorly on the simulations. Two survey implementers had to be replaced and retrained as part of the quality control measures taken by the RERA Team.

USAID.GOV LAC EDUCATION SUPPORT CONTRACT: QUARTERLY REPORT Q3 FY 2020 | 6

Communication and building trust with the Secretariats was paramount to be able to obtain the contact information for RERA participants. Protocols regarding sharing of student information vary across secretariats, so although the Ministry of Education provided a letter introducing the RERA and the Team, additional follow up was required from DevTech staff to ensure the timely receipt of the information needed. Only in one instance, the Team was unable to obtain the contact information of participants before the survey and the interviews commenced. The Secretariat required additional steps before they were able to release the data to the RERA Team. To ensure this Secretariat was represented in the assessment, their input was included as a case study in the report.

Next Steps

RERA – Colombia: During Q4, DevTech will address comments provided by USAID on the first draft of the report, and provide a more finalized version by the end of July. DevTech will also discuss with USAID how to disseminate some of the RERA findings to a broader audience.

3.1.3. VENEZUELA EDUCATION DIAGNOSTIC

In the month of June, DevTech received USAID approval on the budget and revised work scope for the Venezuela Education Diagnostic. The purpose of this Activity is to produce an illustrative snapshot of education inputs and outputs in Venezuela, focusing on basic education (pre-K through 12th grade). The completed report will inform the assistance efforts of USAID, the Inter-American Development Bank (IADB) and other international partners, in concert with the interim government of Venezuela. Two local subcontractors will support the Activity, Universidad Católica Andres Bello (UCAB) and ANOVA Diagnósticos y Propuestas Basadas en Evidencia. During this same month, DevTech received USAID approval of the subcontractors’ technical staff, and a Technical Advisor to support the implementation of the Activity. Also, DevTech worked with ANOVA and UCAB to get subcontracts to start activities in July 2020. Next Steps

During Q4, DevTech will work with subcontractors to develop a research plan for the Activity, and will submit it for review and comment to USAID and an external advisory group. The Subcontractors will also be responsible for data collection planning and the development of data collection tools ahead of the start of the school year in Venezuela in September.

3.2. TASK 2: MONITORING AND EVALUATION OF EDUCATION PROGRAMS

No activities are currently active under this task.

Challenges and Solutions

Nothing to report in this period. Next Step.

This activity does not require any further steps.

7 |LAC EDUCATION SUPPORT CONTRACT: QUARTERLY REPORT Q3 FY 2020 USAID.GOV

3.3. TASK 3: PREPARATION FOR EDUCATION SKILLS-BUILDING WORKSHOPS

No activities are currently active under this task.

Challenges and Solutions

Nothing to report in this period.

Next Steps

Nothing to report in this period.

4. OTHER REPORTS Nothing to report in this quarter.

5. LESSONS LEARNED

The following are lessons learned from the Colombia Rapid Education and Risk Assessment that emerged in carrying out the assessment remotely: Early and clear explanations of adaptation options help agile decision-making.

• DevTech drafted a concept note in March on how to adapt the RERA to remote management, with optional scenarios and budgetary implications, to inform a quick decision by USAID/Colombia and the Ministry of Education.

Intensive collaboration with local education authorities and principals is even more crucial when working remotely.

• After formal introductions from the Ministry of Education, the RERA team invested considerable time sending emails and talking by telephone iteratively with local education secretariat officials to explain the RERA’s remote process, discuss sample requirements and school selection, as well as respondent types and our connectivity requirements. This took more time given the crisis that secretariats and schools were managing, and getting through by phone was understandably challenging.

• In one area with a history of child recruitment by guerrilla groups and organize crime, a secretariat required additional assurances of data protection before it would share student contact information.

• The RERA Team also spoke with several principals, who contacted all respondents personally to explain the RERA process and enlist their participation.

Various technologies are available and effective to support remote fieldwork.

• The RERA Team used KoboToolbox for the survey, and a mixture of Skype (both Skype-to-Skype and Skype-to-telephone).

• Documents were developed, shared and discussed live using Google Drive and Google Documents.

• The RERA Team did not conduct focus groups, but platforms exist to conduct focus group discussions—which requires respondent access to technology and data, appropriate planning and team training.

USAID.GOV LAC EDUCATION SUPPORT CONTRACT: QUARTERLY REPORT Q3 FY 2020 | 8

Using a local call center capability can increase call efficiency. • Administering the survey electronically was originally anticipated, but the RERA Team opted to

use a local data collection firm’s own call center capability to administer the survey by telephone in order to reach respondents without access to data or internet.

• The call center scheduled and conducted survey interviews, recorded informed consent, and scheduled all follow-up, qualitative interviews.

• Local call center capability also optimizes a culturally sensitive approach. Conducting the quantitative survey and follow-up interview on the same call can avoid lower response rates in follow-up interviews.

• Response rates for the RERA survey were strong, but dropped off for the second, follow-up interview calls to the same respondents.

• Merging a survey and qualitative interview can be organized within a one-hour call. Working remotely can constrain efforts to randomize the selection of respondents.

• While this can occur in any in-person RERA, concerns about sharing respondent contact information electronically for the RERA team to randomize emerged as a limiting factor. Ultimately, school principals decided on respondent lists using criteria and guidance provided by the RERA Team.

Remote training can be effective—with sufficient time.

• Asynchronous and synchronous training approaches should be combined for effective training. For example, allow participants time to study materials off-line and then use platforms—such as live written or verbal question-and-answer sessions—strategically and efficiently.

• Plenary, synchronous sessions are important for specific topics that the entire team must discuss, such as ethics.

• Running simulations to assess data collectors and test the protocols themselves are a necessity, especially for open-ended interviews, but sufficient time is needed for corrections and re-runs.

• If a data collection firm is hired, maximizing the firm’s internal quality control and management systems to discuss training content, prioritizing questions, and considering running internal question-and-answer sessions can winnow and prioritize the list of questions plenary.

• The RERA Team needed more time to adequately test survey and interview questions before the launch.

Remote data collection can build in certain biases.

• Requiring respondents to have a cell phone (or internet connectivity) can bias data collection away from the most vulnerable respondents in the sample. Tackling a lack of access to technology is a challenging and complex issue beyond the scope of an assessment, especially rapid ones. Some partners doing face-to-face surveys have issued phones and solar chargers to respondents, but this approach for remote, rapid assessments may not be feasible. Local phone sharing strategies have potential, but these are also complicated by the COVID-19 reality. A common approach is to find ways to supplement their data in order to infer the situation of vulnerable populations.

• In interviews, there were instances where parents accompanied their children on the telephone and directed them on what to say. Interviewers can explain to parents the need to let their children speak freely, but this must also be carefully observed to ensure quality control of the data.

Remote management requires less funds—but may also reap less insights.

9 |LAC EDUCATION SUPPORT CONTRACT: QUARTERLY REPORT Q3 FY 2020 USAID.GOV

• Remote management does not require domestic and international travel, lodging and meal expenses.

• However, conducting conversations on sensitive issues by telephone can affect the level of confidence in the respondent. On the one hand, speaking remotely might increase a respondent’s trust and candor. On the other hand, and in contexts where child recruitment and other predatory behavior is common, speaking on the telephone can reduce trust.

• Typically, doing in-person school visits for a RERA offer opportunities for the team to observe the school environment and gain additional insights from being present. This dimension is lost in a remotely managed RERA.

RERA briefing and validation meetings can and should be held remotely.

• The RERA process involves two important moments of consultation with stakeholders—a briefing before data collection starts, and a consultation on preliminary results as data collection closes.

• These consultations were held using different online platforms and remain vital to the legitimacy of the RERA process and to the quality of the RERA conclusions and recommendations.

Working under COVID-19 social restrictions affects everyone—including the RERA Team.

• RERA is designed to be a rapid exercise, thus it is important to recognize the pressures and unique stresses that affect not only respondents and partners, but RERA Team members themselves.

• Working remotely under COVID-19 is not the same as working remotely in a normal situation. Understanding, flexibility and creativity all play important roles in managing for quality and results.

USAID.GOV LAC EDUCATION SUPPORT CONTRACT: QUARTERLY REPORT Q3 FY 2020 | 10

ANNEX A. PROGRESS SUMMARY OF INDICATORS

1. DESIGN AND IMPLEMENTATION SUPPORT ACTUAL COMMENTS

1.1 Number of sector assessments supported 1 Final Dominican Republic RERA

English and Spanish report approved

1.2 Number of education proposals and new education activities supported

1.3 Number of implementing partners who received capacity building support

1.4 Number of research and white papers produced

1.5 Number of reports analyzing education sector trends produced

2. MONITORING AND EVALUATION OF EDUCATION PROGRAMS ACTUAL COMMENTS

2.1 Number of performance evaluations of education programs carried out 1 LAC Reads Performance Evaluation

report approved

2.2 Number of impact evaluations of education programs carried out

2.3

Number of implementing partners that received support with data collection, collation, verification, and/or reliability assessment

3. PREPARATION FOR SKILLS-BUILDING WORKSHOPS ACTUAL COMMENTS

3.1 Number workshops supported for USAID staff, implementers, and/or other stakeholders

3.2 Number of individuals who participated in training/workshops

3.3 Number of dissemination and communication documents produced and distributed

1 Production of Peru Legacy book and short videos supported