Report of the External Evaluation of the Eureka Superkidz Program 2012-13

The Eureka Superkidz Program was implemented in 675 villages in the academic year 2012-13. Each year, an external evaluation agency, New Concepts Information Systems (NCIS), conducts a detailed evaluation of students in a sample set of villages to gauge the effectiveness of the program. This year, they evaluated over 3209 children from 30 Aid India program Villages and 1983 children from 20 Control Villages. Children from class-1 to class-8 from the same villages have been assessed over three years on Tamil and Math skills.

Summary of the Key Findings Below is a short summary of the key findings followed by detailed graphs and explanations for different classes in Math and Tamil.

• Overall (in all classes, and in both Math and Tamil), we have seen that children in the Eureka villages continue to perform much better than children from the control villages

• In the Eureka villages, the “jumps” or the improvements made in this academic year (2012-13) have been smaller than in the last academic year (2011-12) in many skills. However, the improvement still compares better to those in the control villages for most skills.

• There have been cumulative improvements for our lower class cohorts in basic skills (like addition and subtraction in Math or reading sentences and stories in Tamil). Children in the control group also pick up some of these skills over the years, but children in the Eureka villages are still significantly ahead.

• In more advanced Math skills (like multiplication, division or fraction addition), children in Eureka villages perform much better than their counterparts in Control villages. More significantly, the percentage of children acquiring these skills seems to be stagnant in control villages – i.e. children are not learning these skills even as they progress to higher classes.

• Apart from those who have been with us for 3 years, we have also seen significant improvements in both Tamil and Maths for students who joined in 2011(and have 2 years of intervention) and in 2012. Analysis and Explanation

• Overall, the results show the value of a sustained intervention over multiple years with a focus on outcomes. When children learn basic skills earlier (as they are supposed to), it translates to a sizeable learning advantage over the years.

• In this academic year (2012-13), the core team of Aid India was responding to government enquiries and audits which took up a significant portion of time. As a result, we could not visit villages frequently and identify gaps to improve on our implementation.

• In the Eureka villages, children already started at a higher level than in control villages. It was more difficult to get the same “high” level of jumps from that starting point.

• Over the years, some portion of the gains made during the academic year are often lost during the summer break when children do not attend school or after-school classes. As a result, the levels in June2012 would have been lower than in April2012 (what is shown in the graphs) and the actual improvements made in the academic year (June2012-April2013) would have been higher.

Detailed results from the External Evaluation by NCIS

I. Cohort of children over 3 years In each of the following graphs, we look at the learning improvements by a cohort of children for a specific skill. The class given (for ex: class-1 for number recognition) is the class corresponding to April 2011. The same child would be in class-3 by April 2013.

Math Evaluation Findings – 3-digit number recognition and addition

Over the past 2 years, a larger proportion of children in Eureka cohort have learnt to recognize 3-digit numbers and perform addition. The improvement in 3-digit number recognition has been higher in Eureka villages over the past 1 year. In addition, the Eureka villages have started at a much higher level than control. Due to summer loss, the levels would have dipped and the starting point would have been lower in June2012.

Maths Evaluation Findings-Multiplication and Division

The

Eureka cohort showed a marked improvement between April 2011 and April 2012 in both multiplication and division skills. There has been an improvement in the multiplication and the division skills in the past one year as well. However the magnitude of improvement is low due to the higher starting level. In the control villages, the levels in both multiplication and division are more-or-less stagnant. This shows that these are tougher skills to acquire (and retain too), and hence the difficulty in improving the figures from last year.

Maths Evaluation Findings-Like fractions & Fraction addition for Class-5

The fractions skill level charts show a clear divergence between the Eureka and the Control cohort. While the control cohort skill levels are flat, there is continuous improvement in the Eureka cohort. Competency in basic skills like addition and multiplication has in turn helped Eureka children to perform well in higher-level skills like addition of fractions.

Maths Evaluation Findings-Shapes & word problems

The Eureka children have shown additional improvements in 2012-2013 over that acquired last year. Almost 90% of class-6 Eureka cohort children are proficient in shapes and properties compared to 70% of control cohort. 74.7% of class-6 Eureka cohort children can do word problems as opposed to 46% of class 6 control cohort.

Tamil Evaluation findings-Reading sentence and stories

Cumulative progress of the children in the Tamil sentence and story reading skills has also been markedly better compared with the control group.

Tamil Evaluation skills- Reading words

Over the past 2 years, a larger proportion of children in the Eureka cohort compared to control cohort, have picked up Tamil word reading skills. Because the children in the Eureka cohort were able to learn Tamil words sooner, it has been easier for them to progress to reading sentences and stories.

II. Results for class cohorts who started in 2011 or 2012

The previous section showed the results for children who have been in the program for at least 3 years. Children who started class-1 in 2011, have had only two evaluations (April 2012 and April 2013). The more recent batch of children who entered class-1 in 2012 have been evaluated only in April 2013, and for them, we have only shown the absolute levels of both control and program in April 2013.

Maths Evaluation Findings-Addition & Subtraction

Children of class-1 who started in 2011 have benefitted considerably from the Eureka SuperKidz program last year. The Eureka group has shown an improvement of 16% in addition 20% in subtraction. The comparative numbers for control are 8% and 2% respectively.

Tamil Evaluation skills- Tamil sentences& Tamil stories The class-1 Eureka cohort has shown an improvement of 27% in reading Tamil sentences and 23% in reading Tamil stories from April 2012 to April 2013. In comparison, the control cohort has shown an improvement to around 13% and 12% respectively.

Since the starting levels were lower, we were able to show higher levels of improvement for the class-1 cohort children (who have now been with us for about 2 years) in both Tamil and Mathematics. Children who joined in 2012

Skill Evaluated for children who started class-I in June 2012

% of children with skill in program villages ( April 2013 )

% of children with skill in control villages (April 2013)

2-digit number recognition 45.6 21.0 2-digit addition 21.1 0.0 Identify Tamil letters 81.4 39.2 Reading Tamil words 57.0 10.1 Reading Tamil sentences 21.2 0.0%

For every skill, we observe that class-1 children of 2012 in the Eureka group have performed significantly better than their counterparts in the Control group.