Imran oomer sample project impact measurement in education

I M P A C T A S S E S S M E N T P I L O T S T U D Y : M E T H O D O L O G Y A N D P R E L I M I N A R Y

F I N D I N G S

B Y I M R A N O O M E R

Lend-A-Hand India: PLAN100

Background

Current measurement focuses on quantitative measures of student achievement:Student attendanceExamination pass rates Graduation rates

Lend-A-Hand India (LAHI) is developing a robust impact measurement system to track the short-, medium-, and long term

impact on the students’ lives it touches through Plan100.

New measurements will understand behavioral outcomes of its program, such as:• Increases in student confidence• Personal aspirations• Entrepreneurial spirit

Project Objectives

LAHI is at the initial stages of developing an impact measurement system. This study will lay the groundwork for it, and specifically accomplish the following:

• …a set of student outcomes expected from participation in the IBT (Introduction to Basic Technology) ProgramDefine

• …an assessment tool to test hypothesized outcomes with students through one-on-one interviewsDevelop

• …a pilot study testing the assessment tool across a sample of IBT and non-IBT students (N>100)Execute

• …preliminary findings from pilot study assessing areas of high, moderate and low impact ; make recommendations for future impact measurement activity

Review

METHODOLOGY

The Big Picture

Input OutputIndividual

Impact

Institution-Wide

Impact

System-wide

Impact

• An example of what impact assessment aims to measures and achieve

Newly revamped

teacher training program

Well-trained ,

energized and

motivated teachers

More engaged students,

resulting in higher

attendance

Improved scores on

standardized tests

Increase in the population’s

job-readiness as a result of a

higher proportion of

high school graduates

continuing on to tertiary education

Constant Feedback Loop

Impact assessment is a method of gaining constant feedback on the design of your organization’s programs with the aim of refining them in real-time to achieve the results you want

Defining Outcomes and Metrics

The following expected student outcomes (near- and long-term) as a result of participation in the IBT program were identified during the initial phase of the study (slide 1 of 2):

Educational Achievement:

• Improved school attendance• Improved performance in

regular curriculum• Improved performance in 10th

standard examination• Heightened interest in STEM-

related learning (science, technology, engineering, math)

• Heightened interest in pursuing further education

Skills Attainment:

• Writing skills• Organizational skills• Project management skills• Problem-solving skills• Basic understanding of

economics (e.g. costs, demand)• Vocational skills (e.g.

engineering, electric wiring)

Defining Outcomes and Metrics

Behavioral Changes:

• Confidence in self• Increase in self-worth / feeling of self-

reliance• Sense of accomplishment• Pride in work• Reduced stress related to school /

examinations• Increased comfort in asking questions• Increased comfort in working with

groups (including mixed gender groups)• Increase in creativity• Increase in entrepreneurial spirit

Employability / Value to Society:

• Career aspirations• Ability to apply skills outside of school• Aspirations to give back to one’s

community• Employment status after 2 /4 / 6 years• Increase in family involvement and

commitment to children’s education• Increase in community involvement and

commitment to local education institutions

• Improved local economy with better job opportunities and a well-educated community

The pilot study has illuminated preliminary findings to support several expected outcomes across the four defined categories. The analysis, however, will focus on

behavioral outcomes – the least studied outcomes of the IBT program

The Assessment

The assessment begins with each student answering the following four unaided, open-ended questions in written form, anonymously.

The purpose is to achieve: a) an unbiased view of student preferences of school subjects, specifically in favor of or against STEM-related courses, and b) students’ unbiased evaluation of their school and IBT (where applicable)

Q1: What is your favorite subject in school? Why? Q2: What is your least favorite subject in school? Why? Q3: What do you like most about your school*? Why? Q4: What would you change about your school*? Why?

*IBT students were asked specific about the IBT program in Q3 and Q4 to gain a deeper understanding of student perceptions of the program and its benefits

The Assessment

One-one-one interviews with each student and ethnographic observation are the core of the assessment. The interviews combine qualitative and quantitative measures (complete assessment tool can be found in the appendix)

The following is an excerpt from the tool where the students rate the importance of specific skills learned at school and within the IBT program. The purpose is to measure where IBT and non-IBT students differ in their perception of skills attainment, and which specific skillsets emerge as the most significant for students in the IBT program

Q1: Please read the following 14 skill areas. Which FIVE do you feel are the most important skills you are learning at school?

Study skills Creativity Reading/writing Math

Science Communication skills Computer skills Career planning skills

Time mgmt. skills Business skills Organizational skills Problem solving skills

Job-specific skills Teamwork skills

The Assessment

The tool also includes an 11-question self-esteem battery, adapted from the widely accepted self-esteem scale developed by Dr. Morris Rosenberg*. Students rate their level of agreement with each statement (below) on a 4-point scale from strongly agree to strongly disagree.

The questions aim to gain insight into key behavioral changes among IBT students.

I am proud of my projects and homework I enjoy working in groups

I am proud of my marks at school I enjoy working in mixed groups (boys and girls, combined)

I feel positive that I will pass the 10th standard examination

I like to ask questions to my teachers

I am excited about my future I feel that I make my parents / family proud

I feel that I will have great opportunities after I finish school

If there is a problem, I believe I can fix it

I feel that I am gaining many good skills at school

*Source: http://www.bsos.umd.edu/socy/research/rosenberg.htm

http://www.bsos.umd.edu/socy/research/rosenberg.htm

Sampling and Administration

During the pilot study, data was collected from 109 students across four secondary schools in rural Maharashtra. Survey counts by gender and category are below:

The surveys were administered in Marathi, the students’ local language with a duration of 5 -10 minutes.

Survey Sample: Impact Assessment Pilot Study

Total sample: 109 students

IBT: 60 students (55%)

Non-IBT (control group): 49 students (45%)

9th Standard: 57 students (52%)

10th Standard: 52 students (48%)

Male: 61 students (56%)

Female: 48 students (44%)

PRELIMINARYFINDINGS

Preliminary Findings

Most important skills learned through school / IBT

Top 5 (in order of frequency) Most Selected Skills

IBT Students Non-IBT Students

Study skills Study skills

Teamwork skills Reading/writing

Problem-solving skills Problem-solving skills

Career-planning skills Science

Business skills Computer skills

IBT students are more likely to prioritize outcomes-focused, non-academic skills such as business skills, teamwork, and career-

planning skills – all core foci of the IBT curriculum


Effect of IBT Program on Importance Placed on Skill Attainment (slide 1 of 2)

Skill Control % Selected

IBT % Selected

% Effect

Career planning skills 20% 43% 112%

Time management skills 27% 18% (-31%)

Business skills 33% 42% 28%

Organizational skills 24% 13% (-46%)

Problem solving skills 42% 43% 1%

Job-specific skills 30% 33% 9%

Teamwork skills 41% 45% 10%

Highlighted skills are those that IBT students chose more often than non-IBT students as being a Top 5 important skill learned at school.


Effect of IBT Program on Importance Placed on Skill Attainment (slide 2 of 2)

Skill Control % Selected

IBT % Selected

% Effect

Study skills 71% 70% (-2%)

Creativity 14% 22% 52%

Reading/Writing 53% 37% (-31%)

Math 41% 38% (-6%)

Science 43% 40% (-7%)

Communication skills 18% 15% (-18%)

Computer skills 43% 38% (-11%)

Negative effect does not indicate reduced impact on specific skill areas; instead, it is a reflection on the prioritization given by students on other skills


By overlaying the self-esteem question set with the proposed outcomes of the IBT program, we identified 6 key behavioral success factors to assess:

Do you want to

start your own

business?

I am proud of

my projects

I am proud of

my grades

I like to ask

questions

I have learned

good skills

I feel positive

that I will pass

I am excited

about my future

I feel like I have many

great opps.

If there is a problem, I can fix it

I like working in groups (and mixed-gender

groups)

Measures and indicators to assess factors


Entrepreneurial Spirit 65.3% 71.7%

Self-Pride / Self-Worth 3.61 3.67

Self-Confidence 3.58 3.66

Personal Aspirations 3.51 3.40

Self-Reliance 3.51 3.58

Interpersonal Skills 3.41 3.58

Sel

f-E

stee

m B

att

ery

+S

elec

t S

urv

ey Q

ue

stio

ns

Outcome Control Score

IBT Score

% Difference

9.7%

1.6%

2.1%

-3.1%

2.1%

4.9%

Fa

cto

rs M

ost

Aff

ect

ed

Am

on

g

IBT

Stu

de

nts

Measured on a 100 ptscale


Areas of High

Impact

• Pride in one’s work• Recognition of one’s

achievements• Entrepreneurial spirit• Gender equality

Areas of Moderate

Impact

• Confidence in test taking

• Confidence in asking questions

• Self-reliance

Areas of Low/No

Impact

• Family pride in child’s accomplishments

• Perception of a bright future

Learnings / Recommendations

• Group work and practicals, “learning by doing,” has had the biggest impact on students’ confidence and interpersonal skills

• Identify how these areas can be leveraged to positively influence students’ long-term view and development

• Self-reliance (the ability to solve problems and think independently) is a core expected outcome of IBT and should be emphasized more during the program; same for asking questions, where many students are still hesitant

• Connect more with parents and families about what IBT is offering their children. This will not only positively affect the families and the students’ self-esteems, but will help in getting families to contribute fees


Student Aspirations for Continuing Education

Q: After 10th standard, what are you interesting in doing?

Control IBT

Go to ITI or another technical school to earn my diploma

18% 22%

Attend junior college (11th and 12th) 78% 74%

Go work with my family 0% 2%

Go work somewhere else 0% 0%

I don’t know 4% 2%

While IBT students are slightly more interested than their peers in pursuing technical education after 10th standard, the majority are still interested in junior college.

Recommendation: LAHI should evaluate its position as a feeder into diploma programs and

how this incentive is being communicated to IBT students and families

WRAP-UP

Key Findings From Pilot Study

Transmitting the Importance of Life

Skills Learning

• IBT students are twice as likely as their peers to view career planning as a fundamental skill learned during their secondary education

• Business, problem-solving, and job-specific skills are also perceived to be of higher importance among IBT students

Inspiring Future Entrepreneurs

• Specifically, an IBT student is 10% more likely to be interested in starting his or her own shop or business

• IBT students’ ideas range from tailoring and welding shops to computer institutes and medical clinics

Facilitating Strong Interpersonal Skills

• IBT students are more likely than their peers to feel comfortable working in groups, particular mixed gender groups (7% effect)

• This is a significant achievement in schools where rote learning and individual achievement are the norms

LAHI has created positive impact for IBT students across a number of measures. In this study, the most notable outcomes are:

Recommendations For Lend-A-Hand India

Questions to consider:

What outcomes exceeded expectations?

Which measures were less impacted by the program than what was expected?

How can LAHI leverage areas of high impact (e.g. entrepreneurial drive) to positively affect areas of moderate (e.g. creativity) and low impact (e.g. family pride)

In areas of high impact…In areas of moderate and

low impact…

Identify drivers (e.g. teaching styles, level of practicals) of

student achievement that can be leveraged towards improving

on other measures

Pinpoint gaps between expected and actual outcomes (e.g. students’

comfort in asking questions) and determine programmatic changes

that can be tested to minimize achievement gaps

Recommendations For Further Impact Work

Be aware of measurement pitfalls*, such as:

Trying to assess a whole program instead of specific components

Aspiring for scientific-level design that is appropriate for large scale populations when striving for generalizable trends but not for context-specific learning where a finer, more customized and qualitative approach often yields richer results

Focusing on uncontrollable , long-term community impact metrics as a measure of a program’s success over tangible, measureable, near-term outcomes

*R&D: A New Form of Evaluation, TCC Group

Impact measurement is not a static, annual process to appease funders; it is a dynamic, continuous effort aimed at understanding, refining and improving

upon an organization’s programs and outcomes

Recommendations For Further Impact Work

Expanding on the pilot study, future steps should include: Communicating expected outcomes and measurement goals to

IBT schools to onboard key stakeholders (e.g. headmaster, instructors) to a more robust assessment system

Tracking outcomes by standard to understand when and how outcomes evolve over program duration

Scaling assessment across all IBT programs with the aim of identifying and replicating star school strategies for maximizing outcomes

Expanding the next phase of the study to include more control variables such as family involvement, family income level, academic achievement, etc.

APPENDIX

Student Verbatims – Positive Reflections on IBT

“I will always have the opportunity to do something because of IBT”

“Because of IBT, we can get jobs anywhere from nursing to engineering”

“I like IBT because we get knowledge of different things”

“I chose IBT because if I don’t get a proper job, I can start my own business”

“We get to learn many different and new things; IBT taught me how to make use of waste”

“I get to know the benefits of IBT because in our future, we can do wiring, prepare chikki, and first aid solutions on our own. IBT is going to help us a lot more in our future.”

“Even if I cannot take further education, I can do something or the other and make a living because of what I’ve learned through IBT”

With the help of IBT, you learn how to solve problems and start a small-scale business in the future. “

“We get to learn many things that can help our family.”

“It’s very beneficial and it’s the only place where girls and boys can work together”

Student Verbatims – Suggestions for IBT

“We should be taught about computers as well”

“Internet should be added as the 5th section of IBT”

“Each student should be required to do each practicals because some don’t do it”

“Practicals should be increased”

“IBT should double its syllabus and newer things should be taught”

“Sometimes materials are in shortage and when tools get broken, it takes long to get them fixed”

“An electronic switchboard should be provided in all four sections”

“There should be more developed machineries for the practicalsso we can learn more things”

IBT Impact Assessment Tool (slide 1 of 4)

Pre-screener questions:

School

Standard

Gender

IBT Status (IBT or Non-IBT)

Self-reported questionnaire:

Q1: What is your favorite subject in school? Why?

Q2: What is your least favorite subject in school? Why?

Q3: What do you like most about your school*? Why?

Q4: What would you change about your school*? Why?

*IBT students were asked specific about the IBT program in Q3 and Q4 to gain a deeper understanding of student perceptions of the program and its benefits


Interview questions: Q5: Please read the following 14 skill areas. Which FIVE do

you feel are the most important skills you are learning at school?

A) Study skills B) Creativity C) Reading/Writing D) Math E) Science F) Communication skills G) Computer skills H) Career planning skills I) Time management skills J) Business skills K) Organizational skills L) Problem solving skills M) Job-specific skills N) Teamwork skills

Q6: After 10th standard, what do you want to do? A) Go to ITI or other school to get a diploma B) Go to Junior College C) Go work with my family D) Go work somewhere else E) I don’t know


Q7: What is your dream job?__________________

Q8: Why?_______________________________

Q9: Do you ever want to start your own shop or business? A) Yes B) No

Q10: What kind of shop or business?________________________________

Q11: Please reach each statement carefully. For each statement, tell me if you: A) Strongly agree B) Agree C) Disagree D) Strongly Disagree

I am proud of my projects and homework

I am proud of my marks at school

I feel positive that I will pass the 10th standard examination


Q11 (continued): Please reach each statement carefully. For each statement, tell me if you: A) Strongly agree B) Agree C) Disagree D) Strongly Disagree

I am excited about my future

I feel that I can have great opportunities after I finish school

I feel that I have gained many good skills at school

I enjoy working in groups

I enjoy working with mixed groups (boys and girls)

I like to ask questions to my teachers

I feel that I make my parents / family proud

If there is a problem, I believe I can fix it