Development of M&E frameworkDevelopment of M&E framework

Prepared by Nyi Nyi THAUNG, UIS (Bangkok)

Capacity Building Workshop on Monitoring and Evaluating Progress in Education in the Pacific

27 – 31 October 2008Nadi, FIJI

2

Strategic sector planningBasic logical steps of the plan preparation

1. Situation analysis

2 Policy,goal and target setting

3 Formulation of priority action programs

4 Preparation of financial framework

5 Preparation of monitoring framework

6 Consolidation of draft sector plan

7 Final revision and adoption of the plan

8 Preparation of yearly operational plans

3

Strategic planningThe strategic planning cycle

Medium term strategic plan

Final evaluation& sector review

Monitoring of implementation

Yearly operational plans

+ MTEF

+ Budget

4

Key Questions

• How can we link data from MIS to information needs at the Nation Education Strategy

• How can we develop M&E framework

• What are the framework for developing indicators in education sector

• What does it mean by information and data?

5

Linkage between policy and data

Policy Statement/ Goal

Target(S)

Measurable indicator(s)

Data required to produce the

indicator

Benchmark

Baseline

Data DataData

Source of data

Frequency

Method of collection

Disaggregation

QC

InteractiveDynamic

6

M & E Framework

• A document describing all these information agreed by all stakeholders

• Usually matrix

• Linkage

• Vertical : Goal toward data

• Horizontal : collaboration among all stakeholders particularly in data providers and users

7

What is

data information indicators

data information indicators

8

Data

• Raw facts and number used for reference or further analyses

• Atomic level (basic) pieces of information that by themselves have little meaning

e.g. number 23 can mean nothing or many things

• Several data may have to be combined to achieve required meaning / understanding

• State of unprocessed

• “A collection of facts from which conclusions may be drawn; “Statistical Data”

• Sometime, defining data vs information is depend on the user

9

• Data that have been processed

• It may require several data to construct a piece of information to result to a better understanding

• Information is a collection of data that is understandable by itself

in previous example, add another piece of data, the character string “age=” so that “age=23”, and we now have better understanding

• Facts or knowledge provided or learned as a result of reasoning, research, or study (data analysis)

Information

10

Male Female Male FemaleLiteracy Programme A 150 110 140 109Literacy Programme B 165 120 144 116Literacy Programme C 132 89 130 73Literacy Programme D 145 126 130 119Literacy Programme E 110 95 98 89Literacy Programme F 133 121 122 115Literacy Programme G 80 66 69 60

Participants Completers

Data vs. Information

From a set of data, an analysis can be made and

Information can be drawn

From a set of data, an analysis can be made and

Information can be drawn

11

Indicators

One way of transforming data into information is by constructing indicators.

An indicator is a self-contained piece of information

Male FemaleLiteracy Programme A 93% 99%Literacy Programme B 87% 97%Literacy Programme C 98% 82%Literacy Programme D 90% 94%Literacy Programme E 89% 94%Literacy Programme F 92% 95%Literacy Programme G 86% 91%

% of completers

12

Male Female Male FemaleLiteracy Programme A 150 110 140 109Literacy Programme B 165 120 144 116Literacy Programme C 132 89 130 73Literacy Programme D 145 126 130 119Literacy Programme E 110 95 98 89Literacy Programme F 133 121 122 115Literacy Programme G 80 66 69 60

Participants Completers

Male FemaleLiteracy Programme A 93% 99%Literacy Programme B 87% 97%Literacy Programme C 98% 82%Literacy Programme D 90% 94%Literacy Programme E 89% 94%Literacy Programme F 92% 95%Literacy Programme G 86% 91%

% of completers

Indicators

13

An indicator: indicates a state or level acts as a gauge or meter for a specified purpose shows current situation, progress made so far

and the remaining distance towards achieving the desired target

• Like a ‘road sign’shows whether you are on the right roadhow far you have traveled, andhow far you still have to go in order to reach

your destination (or attain your objectives)

What is an indicator?

14

Why do we need to set indicators?Indicators are set:

• to serve as benchmark, guide

• to act as a “measuring rod” in comparing different population groups, localities, levels, sex, etc.

• to lead systematic and appropriate analyses

• to help defining relevant objectives and outputs

Indicators are:

• basis for measuring progress towards objectives

• prerequisite for a monitoring system

15

A GOOD INDICATOR MUST BE:

1. Direct

2. Objective

3. Adequate

4. Quantitative, where possible

5. Disaggregated, where appropriate

6. Practical

7. Reliable

What is good indicators

Indicators should be smart:

• Specific• Measurable• Achievable or

Attainable• Result oriented, and• Time-bound

16

• Enable management to monitor effectiveness and efficiency in the delivery of education services

• Enable judgments on key aspects of the functioning of the education system

• Useful tools to identify and measure changes in the education system over time, including the effect of planned interventions

• When indicators are produced on a regular basis they can reveal possible changes in response to policy actions

Education Indicators

17

Identification of Indicators of Education

The education system can be viewed as a form of “production“ which has three components:

Inputs Processes Output (and Outcome)

Then, basic education indicators can be identified and grouped under these categories.

Framework: "Education as a Production Framework: "Education as a Production Function"Function"

18

Identification of Indicators of Education

If education is viewed as a “basic social service“, basic education indicators can be identified under three areas:

Access Quality, and Management

Framework: "Education as a Basic Social Framework: "Education as a Basic Social Service"Service"

19

Monitoring Education from Various Aspects

Education System

Participation

Access

Next Level

Labour Market

Management, Quality and Equity

Drop-out

Access

20

Monitoring Education from Various Aspects

Education System

Access

Next Level

Labour Market

Drop-out

Access

Participation• Gross Enrolment

Ratios • Net Enrolment

Ratios

Access to education• Gross Intake Ratios• Net Intake Ratios• % of new entrants with

ECCE

Output• Survival rate to last

grade• Completion ratePerformance

• Promotion, Repetition and drop-out rates

• Internal efficiency

Quality• Pupil-teacher ratio• % of trained

teachers

Equity• Indicators by sex• Gender parity

Index

21

Analytical Framework

Issues

Analysis

All Education Sub-sectors

Demand Resources

Access and Participatio

n

Quality and Performanc

e

Equity Output and

Outcome

Meeting Goals, targets

Trends, changes

Gaps, Differences

Finding reasons

22

Sources of Education DataSupply-side Information

Demand-side Information

Schools• School info• Student info• Teacher info• Achievement

EMIS

Education providers• Provision to education

(finance)• Management• Policy • Curriculum

Reports

Households• Socio-economic • Health and nutrition• Relevant info for

student’s participation, achievement, drop-out

• Awareness and value to education

• External efficiency (link to labour market

Surveys

Research and case studies - Findings

23

Basic Indicators of EducationIndicator

1 Adult Literacy Rate (Aged 15+)

2 Coefficient of Internal Efficiency by Level

3 Completion Rate by Level

4 Dropout Rate by Grade

5 Gross Enrolment Ratio (GER) by Level

6 Gross Enrolment Ratio in Pre-Primary (ECCE) Programs

7 Gross Intake Rate (AIR)

8 Literacy Rate of 15-45 Year Old

9 Net Enrolment Ratio (GER) by Level

10 Net Intake Rate (NIR)

11 % of Classrooms meeting FSQL

12 % of Female Teachers by Level

13 % of Girl Enrolment by Level (Preschool, P, LS, US)

14 % of new Grade 1 intakes with Preschool experiences

15 % of Oversized Classes (41+ Pupils) by Level

16 % of Principals with School Management Training by Level

17 % of Schools Inspected during previous year by Level

18 % of Schools with Active Community Participation by Level

19 % of Schools with Adequate Sanitation Facilities by Level

20 % of Schools with both Proper Water Supply and Adequate Sanitation Facilities by Level

Indicator

21 % of Schools with Computer Lab

22 % of Schools with Library

23 % of Schools with Proper Drinking Water Supply by Level

24 % of Schools with Science Lab (Lower Sec., Upper Sec.)

25 Professionally Trained Teachers by Level

26 Promotion Rate by Grade

27 Public Exp. on Education as a % of Total Government Exp.

28 Public Expenditure on Education as a % of GDP

29 Public Recurrent Expenditure on Education as a % of Total Government Recurrent Expenditure

30 Pupil-Class Ratio by Level

31 Pupil-Teacher Ratio by Level

32 Repetition Rate by Grade

33 Retention Rate by Level

34 Survival Rate to Grade 5

35 Teacher Attrition Rate by Teaching Level

36 Teachers with Required Academic Qualification by Level

37 Transition Rate from Lower Sec. to Upper Secondary Level

38 Transition Rate from Primary to Lower Secondary Level

39 Transition Rate from Upper Secondary to Tertiary Level

40 Unit Cost (public recurrent expenditure per pupil) by Level

24

AccessSr. Indicator Function

1 Gross Enrolment Ratio in Pre-Primary / Preschool (ECCE) Programs Input

2 % of new Grade 1 intakes with Preschool (ECCE) experiences Input

3 Gross Intake Rate (AIR) Input

4 Net Intake Rate (NIR) Input

5 Gross Enrolment Ratio (GER) by Level (primary, lower sec., upper sec.) Input

6 Net Enrolment Ratio (GER) by Level (primary, lower sec., upper sec.) Input

7 % of Girl Enrolment by Level (preschool, primary, l.sec., usec.) Input

8 Public Expenditure on Education as a % of GDP Input

9 Public Exp. on Education as a % of Total Government Exp. Input

10 Public Recurrent Expenditure on Education as a % of Total Government Recurrent Expenditure

Input

11 Unit Cost (public recurrent expenditure per pupil) by Level Input

Basic Indicators of Education

25

QualitySr. Indicator Function

12 Teachers with Required Academic Qualification by Level Input

13 Professionally Trained Teachers by Level Input

14 % of Classrooms meeting FSQL Input

15 % of Schools with Library Input

16 % of Schools with Computer Lab Input

17 % of Schools with Science Lab (Lower Sec., Upper Sec.) Input

18 % of Schools with Proper Drinking Water Supply by Level Input

19 % of Schools with Adequate Sanitation Facilities by Level Input

20 % of Schools with both Proper Water Supply and Adequate Sanitation Facilities by Level

Input

Basic Indicators of Education

26

QualitySr. Indicator Function

21 Pupil-Teacher Ratio by Level Process

22 Pupil-Class Ratio by Level Process

23 % of Oversized Classes (41+ Pupils) by Level Process

24 Promotion Rate by Grade Process

25 Repetition Rate by Grade Process

26 Dropout Rate by Grade Process

27 Survival Rate to Grade 5 Process

28 Coefficient of Internal Efficiency by Level Process

29 Retention Rate by Level Process

Basic Indicators of Education

27

QualitySr. Indicator Function

30 Completion Rate by Level Output

31 Transition Rate from Primary to Lower Secondary Level Output

32 Transition Rate from Lower Secondary to Upper Secondary Level Output

33 Transition Rate from Upper Secondary to Tertiary Level Output

34 Literacy Rate of 15-45 Year Old Output

35 Adult Literacy Rate (Aged 15+) Output

Basic Indicators of Education

28

ManagementSr. Indicator Function

36 % of Female Teachers by Level Input

37 % of Principals received School Management Training by Level Input

38 Teacher Attrition Rate by Teaching Level Process

39 % of Schools with Active Community Participation by Level Process

40 % of Schools Inspected during previous year by Level Process

Basic Indicators of Education

29

Difference use of education statistics and Indicators

Planning and resource allocation Address special problems Monitoring Evaluation Accounting Management and decision making

30

Basic Steps in Selecting IndicatorsBasic Steps in Selecting Indicators

Common steps in selecting indicators for monitoring (or evaluation) purposes are:

• step 1 Setting clearly defined objective(s)

• step 2 Developing a list of possible indicators

• step 3 Assessing each possible indicator

• step 4 Selecting the “most appropriate” indicators

31

Some suggestions in selecting Some suggestions in selecting indicatorsindicators

Do Don’t• Look for signals that have

clear meaning• Make use of proxies, when

needed• Set targets that are realistic• Agree with beneficiaries and

partners• Look for data that is easily

available; avoid major data collection

• Keep data sources and monitoring responsibilities in mind

• Lose sight of the objectives• Assume that data will be

available• Set targets that cannot be

achieved• Impose or insist on any one

indicator• Over invest in attempts to

quantify• Use indicators that need

expert analysis• Use more indicators than

necessary