Upload
others
View
1
Download
0
Embed Size (px)
Citation preview
QUARTERLY PERFORMANCE REPORT
FY2019, THIRD QUARTER REPORT
APR - JUN 2019
USAID/UGANDA PERFORMANCE AND
IMPACT EVALUATION FOR LITERACY
ACHIEVEMENT AND RETENTION
ACTIVITY (LARA)
Contract No. AID-617-C-16-00003
This publication was produced for review by the United States Agency for International Development. It was
prepared by NORC at the University of Chicago. The authors’ views expressed in this publication do not
necessarily reflect the views of the United States Agency for International Development or the United States
Government.
ID/UGANDA PERFORMANCE AND
IMPACT EVALUATION FOR
LITERACY ACHIEVEMENT AND
RETENTION ACTIVITY (LARA)
QUARTERLY PERFORMANCE
REPORT
FY2019, THIRD QUARTER REPORT
APR- JUN 2019
Prepared under Contract No.: AID-617-C-16-00003
July 30, 2019
Submitted to:
USAID/Uganda
Submitted by:
NORC at the University of Chicago
Attention: Ritu Nayyar-Stone
Phone:+1-301-634-9538
DISCLAIMER The authors’ views expressed in this publication do not necessarily reflect the
views of the United States Agency for International Development or the United States
Government.
Uganda LARA – AID Contract No. AID-617-C-16-00003
QUARTERLY PERFORMANCE REPORT 2019 QPR 2 i
TABLE OF CONTENTS
TABLE OF CONTENTS ............................................................................................................... i
ACRONYMS .................................................................................................................................. ii
INTRODUCTION ......................................................................................................................... 1
IMPACT EVALUATION ACTIVITIES ....................................................................................... 2
UPDATE ON IMPACT EVALUATION ACTIVITIES ..................................................................................... 2 MAJOR ACCOMPLISHMENTS AND CHALLENGES OF PAST QUARTER .......................................... 6
MAJOR ISSUES THAT NEED TO BE RESOLVED .......................................................................................... 6
PERFORMANCE EVALUATION ACTIVITIES ......................................................................... 6
UPDATE ON PERFORMANCE EVALUATION ACTIVITIES ...................................................................... 6 MAJOR ACCOMPLISHMENTS AND CHALLENGES OF PAST QUARTER .......................................... 6
MAJOR ISSUES THAT NEED TO BE RESOLVED .......................................................................................... 6
P&IE PLANNED ACTIVITIES FOR NEXT QUARTER ............................................................ 7
ANNEX A. MIDLINE IMPACT EVALUATION SRGBV TRAINING AGENDA........ 8
ANNEX B. RETENTION AND ATTENDANCE STUDY PRESENTATION ............. 9
ANNEX C. CLASSROOM OBSERVATION STUDY PRESENTATION .................. 17
Uganda LARA – AID Contract No. AID-617-C-16-00003
QUARTERLY PERFORMANCE REPORT 2019 QPR 2 ii
ACRONYMS
CAPI Computer Assisted Personal Interviewing (
CRO Classroom Observations
CSR Centre for Social Research
EGR Early Grade Reading
EGRA Early Grade Reading Assessment
FY Fiscal Year
FGD Focus Group Discussion
LARA Literacy Achievement and Retention Activity
MoES Ministry of Education and Sports
NORC NORC at the University of Chicago
P&IE Performance and Impact Evaluation
RTI Research Triangle Institute
RWI Research World International
R&A Retention and Attendance
SRGBV School-Related Gender-Based Violence
STS School-to-School International
TASO The AIDS Support Organization
USAID United State Agency for International Development
Uganda LARA – AID Contract No. AID-617-C-16-00003
QUARTERLY PERFORMANCE REPORT 2019 QTR 3 | 1
INTRODUCTION
NORC at the University of Chicago (NORC), in partnership with subcontractor Panagora
Group, is pleased to submit this Quarterly Performance Report covering the period April to
June 2019 under the USAID/Uganda Performance and Impact Evaluation (P&IE) of the 5-year
Literacy Achievement and Retention Activity (LARA) in Uganda.
LARA is a 5-year (April 7, 2015 to April 6, 2020) USAID-funded initiative to improve reading
skills for 1.3 million learners in 28 districts throughout Uganda. The project, which is being
implemented by Research Triangle Institute (RTI) International, is designed to support the
Ministry of Education and Sports (MoES) - Uganda in its efforts to improve early grade reading
(EGR) and retention. LARA has two main objectives:
Result 1 (R1) focuses on strengthening the capacity of MOES and other educational
stakeholders to deliver EGR. To this end, the activity focuses on improved reading skills
in three local languages (Luganda, Runyoro-Rutooro, and Runyankore-Rukiga) and
English for early primary grade learners P1-4.
Result 2 (R2) focuses on promoting a safer primary school learning environment to
prevent and reduce incidents of school-related gender based violence (SRGBV).
The activity hypothesizes that reducing SRGBV will increase learners' retention because they
will be able to focus on their lessons and feel secure in their learning environment, thereby
improving their ability to learn to read. To realize these objectives, the activity focuses its
efforts on systemic capacity building of the education system, school-level support, and
community- and household-level support and participation.
The Uganda LARA P&IE activity (April 2016 - April 2021) has two objectives: (1) to assess the
impact of LARA on learners’ literacy skills and retention rates; and (2) to assess the
performance of LARA in terms of project management, learning, design, implementation,
results, and sustainability.
Uganda LARA – AID Contract No. AID-617-C-16-00003
QUARTERLY PERFORMANCE REPORT 2019 QTR 3 | 2
IMPACT EVALUATION ACTIVITIES
UPDATE ON IMPACT EVALUATION ACTIVITIES
Preparation and training to collect Midline Impact Evaluation SRGBV data
NORC, in collaboration with Panagora and local firm Centre for Social Research (CSR), will
collect Midline Impact Evaluation SRGBV data in July 2019 in 11 districts. NORC will gather
SRGBV-related data through the following:
Female learner survey
Male learner survey
Primary caregiver survey
Teacher survey
Head teacher survey
Focus Group Discussions (FGDs) with learners
FGDs with primary caregivers
FGDs with teachers
School Safety Observation Tool
Prior to data collection, the LARA P&IE team updated the baseline evaluation instruments to
include questions and get feedback from respondents on LARA implemented activities. NORC
programmed the surveys and observations tool in the NField Computer Assisted Personal
Interviewing (CAPI) application.
In addition, the team updated manuals and the child protection referral forms, and created
introduction letters to allow the fieldwork team access to schools in the districts where data
collection takes place. In addition, CSR conducted advance school visits to gain parental consent
for learner participation in the data collection.
SRGBV expert, Marie-Celine Schulte from Panagora, joined LARA P&IE Evaluation Lead, Dr.
Ritu Nayyar-Stone and Mr. Carlos Fierros to provide training in Kampala between June 19th
and June 28th, 2019. The training agenda is included in Annex A.
Uganda LARA – AID Contract No. AID-617-C-16-00003
QUARTERLY PERFORMANCE REPORT 2019 QTR 3 | 3
CSR was able to retain about 80 percent of the enumerators, moderators, counselors, and
supervisors that had participated in the baseline data collection. This was beneficial in ensuring
continuity in the data collection, counseling, and child protection referral processes. The other
20 percent researchers were recruited on the basis of their qualifications (higher education in
social sciences), experience working with young people and completion of training on
protection of human subjects. CSR also ensured that they had some previous training on
gender-based violence. All new researchers were fairly younger (between 21 and 30 years of
age) so that they could comfortably interview children. Finally, all were confirmed to have
certificates of good conduct from International Police, which qualifies them to work with young
people.
Six of the FGD moderators and note takers from the baseline (2 female and 4 male) were
replaced due to the fact that they delayed submitting their baseline transcripts. The 6 new FGD
experts were cleared after scrutinizing some of the transcripts they prepared recently. They
also have experience working with young people.
All counselors were recruited on the basis of their experience working as counselors with
children. All have certificates in guidance and counseling as well as protection of human
subjects.
Overall, there will be 8 survey teams (4 enumerators, 1 supervisor and 1 counselor). As much
as possible, CSR ensured that there were equal number of male and female researchers
because they will be interviewing male and female respondents respectively. For FGDs, CSR
recruited 4 teams (2 Luganda and 2 Runyankore). Each pair (male and female) will handle FGDs
involving respondents of the same gender. Each of the FGDs will have a same gender counselor
to support the team in case there are children that need their attention.
Preparation for Year 3 Term 2 Retention and Attendance (R&A) data collection
NORC’s local data collection firm partner, Research World International (RWI), will conduct
R&A data collection for year 3 term 2 in 71 schools for a total sample of 3,505 learners
currently in P1 through P7 (most will be in P3 and P6) in July 2019.
In preparation of the data collection, this quarter, NORC reviewed the data collection
instruments and protocols from the last data collection round, and updated them based on
lessons learned from prior fieldwork and data cleaning. NORC’s Nfield programmer updated
the student instrument to show the correct grades in which the pupils are enrolled, and
preloaded the names of the P3 and P6 assigned teachers in the teacher instrument.
Uganda LARA – AID Contract No. AID-617-C-16-00003
QUARTERLY PERFORMANCE REPORT 2019 QTR 3 | 4
Analysis of R&A Data
NORC compiled and harmonized the R&A data from six waves1 of data collection (terms 2 and
3 of 2017; terms 1, 2 and 3 of 2018; and term 1 of 2019) in order to conduct a brief analysis
and produce a set of graphical representations of the data. NORC is including the data displays
in a brief summary report. A few key findings are:
Dropout rates are high, reaching 10.5 percent in Term 1 of 2019. Boys drop out of school
in statistically significant higher proportions than girls.
Absenteeism is high. Average learner attendance is 82 percent each school day. P4 learners
were slightly more likely to attend school than P1 students. Attendance levels are slightly
better in Runyankore/Rukiga than in Luganda dominant regions. Likewise, girls show slightly
better attendance than boys.
Repetition rates are high: around 20 percent of learners failed to pass to the next grade
when they returned to school in 2018, and 36 percent by 2019.
Seventeen percent of the classrooms visited did not have a teacher -assigned or substitute-
present. In some cases, the teacher was in the school but not present in the classroom.
During an average visit day, almost 8 percent of the teachers were absent.
The LARA P&IE Evaluation Team Lead, Dr. Ritu Nayyar-Stone, presented the results of the
analysis of the R&A data to USAID/Uganda on June-26th, 2019. Annex B includes the
presentation slides.
Analysis of 2018 Classroom Observation (CRO) Data
The P&IE Early Grade Reading CRO Expert, Dr. Ursula Hoadley, performed qualitative and
quantitative analysis on the CRO data gathered during the Term 3 of 2018, and wrote a
comprehensive report. A few key findings are:
Greater fidelity to the program structure was evident in 2018 than in 2017. However, all
learners had the LARA books in fewer classes in 2018.
1 Term 1 in 2017 was used to list the target learner population – i.e. the sample – for future retention and
attendance checks.
Uganda LARA – AID Contract No. AID-617-C-16-00003
QUARTERLY PERFORMANCE REPORT 2019 QTR 3 | 5
Classroom discourse patterns had not changed very much and, as in 2017, tended to follow
traditional patterns of highly repetitive chorused readings of syllables, words, sentences or
short texts.
Teachers tended to follow the LARA lesson plans in a very procedural way, rarely
exercising any agency in contextualizing the lesson plans in their classrooms.
The LARA P&IE Evaluation Team Lead, Dr. Ritu Nayyar-Stone, also presented the results of the
analysis of the CRO data to USAID/Uganda on June-26th, 2019. Annex C includes the
presentation slides.
USAID Approval to Drop SRGBV Scenario Approach for Midline Data Collection
NORC had raised concerns about the inclusion of scenarios in the SRGBV assessment tools to
introduce the subject of experiential violence to child respondents and establish a rapport
between enumerator and child. Based on several email discussions and sharing of several other
epidemiological best practice surveys that do not include scenarios, the USAID/Kampala
Mission supported excluding the scenarios for the LARA P&IE midline assessment. USAID
noted that their reasoning was based on the following:
1. NORC presented a credible case that there is the potential for increased harm to
children based on this approach.
2. USAID received written confirmation from NORC that there is no risk to the impact
evaluation's ability to measure LARA's program effects across the treatment arms.
LARA P&IE’s COR, Kay Leherr noted that going forward USAID plans to work with its
partners and stakeholders to continue investigating the most appropriate approaches for
researching child-related violence themes. Based on these ongoing consultations, they reserve
the right to reinstate the scenarios for the LARA P&IE endline assessment should the full
preponderance of evidence point in that direction. Given this go ahead NORC dropped all
scenarios for the pilot conducted on June 27th and field data collection.
Submission of Project Deliverables
Submission of Annual Report/Work Plan (Deliverable #21)
In May 2019, NORC submitted the 2019 Annual Report/Work Plan for AID-617-C-16-00003,
Uganda Performance and Impact Evaluation for Literacy Achievement and Retention Program
(LARA).
Uganda LARA – AID Contract No. AID-617-C-16-00003
QUARTERLY PERFORMANCE REPORT 2019 QTR 3 | 6
MAJOR ACCOMPLISHMENTS AND CHALLENGES OF
PAST QUARTER
The major accomplishments of this past quarter were:
Preparations for SRGBV data collection and enumerator training without major challenges.
The presentation of preliminary findings based on the analysis of R&A and CRO data.
Submission of 2019 Annual Report/Work Plan.
MAJOR ISSUES THAT NEED TO BE RESOLVED
None.
PERFORMANCE EVALUATION ACTIVITIES
UPDATE ON PERFORMANCE EVALUATION ACTIVITIES
The LARA P&IE team and its subcontractor, Panagora Group, prepared an Implementation
Update of the LARA activity throughout Project Years 2 and 3, which also includes a program
summary. The main sources included reports made available by USAID/Uganda, as well as
clarifications provided by Geri Burkholder, LARA Chief of Party, about training schedules, EGR
support supervision, and SRGBV referral systems and trainings.
The Implementation Update was useful during the development of instruments of the 2019
midline impact evaluation, and another update will be done by the end of 2019 to inform the
2020 final performance evaluation. NORC also intends to use midline findings of the impact
evaluation (October 2019) to inform the performance evaluation (February 2020) in such
matters as the locations of schools to select for field visits and the questions to be asked during
the key informant interviews.
MAJOR ACCOMPLISHMENTS AND CHALLENGES OF
PAST QUARTER
None.
MAJOR ISSUES THAT NEED TO BE RESOLVED
None.
Uganda LARA – AID Contract No. AID-617-C-16-00003
QUARTERLY PERFORMANCE REPORT 2019 QTR 3 | 7
P&IE PLANNED ACTIVITIES FOR NEXT
QUARTER
Midline Impact Evaluation SRGBV data collection
NORC’s local data collection firm partner, CSR, will collect data for the SRGBV component of
the Midline Impact Evaluation between July-1st and July-26th, 2019. Survey data collection will
use tablets and the Nfield CAPI application.
Training and fieldwork to collect Year 3 Term 2 R&A data
RWI will conduct R&A data collection for Year 3 Term 2 between July-15th and July-25th, 2019.
Data collection will use tablets and the Nfield CAPI application. In preparation of the R&A data
collection, RWI will hold a training on July-12th, 2019.
Preparations to collect Year 3 CRO data
The CRO Expert, in collaboration with the LARA P&IE team, will adapt CRO instruments and
protocols based on the experience and lessons learned from Year 2 fieldwork and analysis.
Year 3 fieldwork will occur in October 2019 (during Term 3).
During Year 3 Term 2 data collection, RWI will record 30-minute videos of literacy lessons of
six classes (2 treatment P1, 2 treatment P3, 1 control P1, and 1 control P3). Our Expert will
use the video recordings during the training for CRO data collection, expected in late
September/early October 2019.
Preparations and training for fieldwork to collect Midline Impact Evaluation EGRA
and Year 3 Term 3 R&A data
NORC, in collaboration with RWI and School-to-School International (STS), will train
enumerators in the second half of September to collect the EGRA data for the midline impact
evaluation in October 2019. Prior to data collection, NORC will work on the following:
Update the baseline data collection tools to include questions and get feedback from
respondents on LARA implemented activities, and to make the P3 Early Grade Reading
Assessment (EGRA) comparable to the instrument used by LARA.
Get introduction letters that will allow our data collection staff to access schools in the
districts where data collection will take place.
Update training manuals and materials.
Uganda LARA – AID Contract No. AID-617-C-16-00003
QUARTERLY PERFORMANCE REPORT 2019 QTR 3 | 8
ANNEX A. MIDLINE IMPACT EVALUATION
SRGBV TRAINING AGENDA
Time Survey Training FGD Training Counselors
Day 1 (Thursday, June 20)
AM Introduction, Overview and Sampling, The AIDS Support Organization (TASO)
PM Gender and Violence
Day 2 (Friday, June 21)
AM Gender and Violence
PM Child Protection Protocol
Day 3 (Saturday, June 22)
AM Informed Consent/Assent
PM Paper Surveys, Learner & Caregiver Best Practices for FGDs In survey or FGD training
Day 4 (Monday, June 24)
AM Introduction to Tablets Learner FGD Training In survey or FGD training
PM All Group Role Plays, English Caregiver FGD Training In survey or FGD training
Day 5 (Tuesday, June 25)
AM All Group Role Plays, By Language Teacher FGD Training Counselor Training Session
PM Small Group Role Plays Counselor Training Session
Day 6 (Wednesday, June 26)
AM Reliability Tests Counselor Training Session
PM Teacher, Head Teacher, and
School Infrastructure Surveys In survey or FGD training
Day 7 (Thursday, June 27)
AM Pilot
PM Pilot
Day 8 (Friday, June 28)
AM Debrief Debrief Debrief
PM Supervisor Training Supervisor Training
Uganda LARA – AID Contract No. AID-617-C-16-00003
QUARTERLY PERFORMANCE REPORT 2019 QTR 3 | 9
ANNEX B. RETENTION AND ATTENDANCE
STUDY PRESENTATION
Uganda LARA – AID Contract No. AID-617-C-16-00003
QUARTERLY PERFORMANCE REPORT 2019 QTR 3 | 10
Uganda LARA – AID Contract No. AID-617-C-16-00003
QUARTERLY PERFORMANCE REPORT 2019 QTR 3 | 11
Uganda LARA – AID Contract No. AID-617-C-16-00003
QUARTERLY PERFORMANCE REPORT 2019 QTR 3 | 12
Uganda LARA – AID Contract No. AID-617-C-16-00003
QUARTERLY PERFORMANCE REPORT 2019 QTR 3 | 13
Uganda LARA – AID Contract No. AID-617-C-16-00003
QUARTERLY PERFORMANCE REPORT 2019 QTR 3 | 14
Uganda LARA – AID Contract No. AID-617-C-16-00003
QUARTERLY PERFORMANCE REPORT 2019 QTR 3 | 15
Uganda LARA – AID Contract No. AID-617-C-16-00003
QUARTERLY PERFORMANCE REPORT 2019 QTR 3 | 16
Uganda LARA – AID Contract No. AID-617-C-16-00003
QUARTERLY PERFORMANCE REPORT 2019 QTR 3 | 17
ANNEX C. CLASSROOM OBSERVATION
STUDY PRESENTATION
Uganda LARA – AID Contract No. AID-617-C-16-00003
QUARTERLY PERFORMANCE REPORT 2019 QTR 3 | 18
Uganda LARA – AID Contract No. AID-617-C-16-00003
QUARTERLY PERFORMANCE REPORT 2019 QTR 3 | 19
Uganda LARA – AID Contract No. AID-617-C-16-00003
QUARTERLY PERFORMANCE REPORT 2019 QTR 3 | 20
Uganda LARA – AID Contract No. AID-617-C-16-00003
QUARTERLY PERFORMANCE REPORT 2019 QTR 3 | 21
Uganda LARA – AID Contract No. AID-617-C-16-00003
QUARTERLY PERFORMANCE REPORT 2019 QTR 3 | 22
Uganda LARA – AID Contract No. AID-617-C-16-00003
QUARTERLY PERFORMANCE REPORT 2019 QTR 3 | 23
Uganda LARA – AID Contract No. AID-617-C-16-00003
QUARTERLY PERFORMANCE REPORT 2019 QTR 3 | 24
Uganda LARA – AID Contract No. AID-617-C-16-00003
QUARTERLY PERFORMANCE REPORT 2019 QTR 3 | 25
Uganda LARA – AID Contract No. AID-617-C-16-00003
QUARTERLY PERFORMANCE REPORT 2019 QTR 3 | 26
Uganda LARA – AID Contract No. AID-617-C-16-00003
QUARTERLY PERFORMANCE REPORT 2019 QTR 3 | 27
Uganda LARA – AID Contract No. AID-617-C-16-00003
QUARTERLY PERFORMANCE REPORT 2019 QTR 3 | 28