Presented ToUganda Evaluation Association & ECD

ProjectKampala Evaluation Talk

Thursday, 2nd October 2014

1

• Muhwana Wilberforce, MWSES CONSULT• Ntinda, Steka House, 1st Floor• Plot 3122 Ntinda – Kiwatule road• wamuhwana@gmail.com;

mwssconsult@rocketmail.com; 0772429679; 0701242652

• Social Sector Development Consultancy• Education, Health, Community Livelihoods• Policy/project formulation, Baseline surveys, Mid

and End Line Evaluation, Impact Assessments, Tracking studies

• Government ministries, Institutions, Donor agencies, NGOs

2

MUHWANA WILBERFORCE (LOCAL CONSULTANT) – Kampala, Uganda

CLAUDIA MUMMENTHEY (INTERNATIONAL CONSULTANT) – Cape Town, South Africa

Sourced by BTC separately Teamed up to harmonise the methodology for the

evaluation Delivered a single inception report Supported by 7 RAs and 3 data entrants

3

• The Business, Technical, Vocational Education and Training (BTVET) Subsector is under the Ministry of Education and Sports (MoES), BTVET Department and was established under TVET Act, 2008.

• BTVET offers training under an overlapping three- tier system:– (i) P7 leavers to enroll in technical schools &

community polytechnics,– (ii) ‘O’ leavers enroll in technical institutes &

vocational training institutes, – (iii) ‘A’ leavers enroll in technical colleges; and– (iv) recently non-formal short courses across the

various levels.

4

The sub-sector fell short of the overall goals and objectives regarding the production of a ‘critical mass’ of graduates that could readily be absorbed into the existing labour market.

Internally the sector and subsector was grappling with issues of low quality of education and irrelevant curricula (UNESCO, 2005). Externally the BTVET system was not corresponding with the needs of the private sector (ESAPR, 2010).

The private sector was not involved in curriculum development thus there was no realignment of knowledge and skills acquisition to the real needs of the labour market.

There was widespread unfavourable perception and recognition of the BTVET programs among the public

5

MoES embarked on developing a 10-year BTVET Strategic Plan in 2011

Six studies had been conducted ex- ante The Tracer Study conducted belatedly

◦ To generate data and information about the effectiveness of different BTVET programmes in Uganda as an input into the strategic planning process of identifying potential improvement areas for the overall system.

6

• Overall objective: To provide an in-depth analysis of the employment destinations of graduates BTVET programmes.– To undertake a comparative analysis of factors

affecting external effectiveness and relevance of training provided by the public and private sectors,

– To identify employment destinations/career paths of completers of BTVET programmes and factors determining post – training careers

7

◦ Accurately assess the potential significant differences in the external effectiveness of different BTVET programmes

8

Tracing of a “reachable” (i.e. within the given restrictions of data availability and time constraints of the assignment) and representative cohort of BTVET graduates from the existing BTVET system using a scientific approach. ◦ Effectiveness – ability to equip learners with the

skills and knowledge required in the labour market thus leading to employment readily.

9

• Consensus on study expectations • Methodology– Industry model– Scope– Sample Structure–Data collection tools–Data collection–Data recording and analysis –Data Quality

10

• Desk –based review of reports by strategy development group

• Debriefing by Strategy development group• Debriefing/interviews with key stakeholders

– MoES, WB, BTC, GTZ/PIVOT • On-site visits – BTVET institutions – public,

private, rural & urban – Helped to understand expectations, quality of

BTVET programmes, available data; planning for the study

11

Industry model: Input process output, with focus

on:◦ Training competence and infrastructure (input)◦ training processes (processes)◦ Training outcome (output/outcome)◦ Overall satisfaction and image/reputation of

BTVET

12

• Scope:• GRADUATES (i.e. learners who had passed final examination;

learners who had completed the training but not yet received certificate)

• Current EMPLOYERS of graduates.• Contextual elements: – Established the readiness of BTVET graduates being

absorbed into in the labour market (i.e. formally employed or self-employed) and/or accessing further education and training opportunities within the system.

– Identified and examined the factors and circumstances that enhance and/or inhibit the placement of graduates into employment and/or further education and training opportunities.

– Included the evaluation of BTVET delivery processes, interaction between BTVET institutions and employers and expectations of graduates and employers, among others.

13

BTVET Graduate Cohort◦ Multi dimensional & stratified:(i) By institutions – public + private = 650◦ arbitrary 10% representative sample – 65; ◦ Public – 33 + private – 32 = 65 (ii) By regional distribution - UGAPRIVI Classification

7 regions (i.e. Central (incl. Kampala), East, North, South, South West, West Nile and Western

14

15

Region

No. Private Inst.

% regional dstn

No. Public Inst.

Sample private inst

Sample public inst.

Central 124 24% 31 8 8

East 73 14% 18 4 4

North 73 14% 18 4 4

South 56 11% 15 3 4

South West

72 14% 18 4 4

West Nile

77 15% 20 6 9 6 5

West 43 8% 11 3 4 3

Total 518 100% 131 32 33

• BTVET Graduate Cohort– (iii) By year of completion – distribution across

different years of completion (2007-2010)– (iv) By learning pathways –formal, informal, non-formal – (v) By exit examinations- UNEB, DIT– (vi) By different occupational training areas– (vii) By gender distribution – male- 80%, female – 20%

• Target sample – telephone contacts for at least 60 graduates in each institution were compiled; 3629 contacts compiled; client’s target was 1500; 1415 graduates were interviewed by phone

16

• Employers– UBOS data on sector growth – agriculture, industry,

services (potential for employment)– UBOS data on distribution of workforce by industry– Kampala & suburbs were covered – good

concentration of businesses/companies – Formal & informal– Size of the enterprises

• Target sample – contact details for 60 employers; 100 contacts compiled (more than 60 within Kampala); 63 interviewed, 12 not responded, 25 not contacted

17

Structured questionnaires for graduates and employers◦ Inclusive of industry model focal areas◦ 5- point Likert-type scales◦ Qualitative responses

18

• 7 RAs deployed in 7 regions to collect bio data (especially phone contacts) on all target graduates in the selected institutions; used a fixed template

• 3 RAs deployed to collect profile data on employer organisations (especially phone & e-mail contacts)

• 7 RAs administered the questionnaires to the graduates using telephonic interviews (English and in local languages)– 7 weeks, 50 interviews per week/ 10 per day

• 2 RAs administered questionnaires to employers physically, collected completed questionnaires, sent via email – based on respondents’ preference – 3 weeks, 20 interviews per week/4 per day

19

• M&E web-based survey platform- Metior Solutions.

• 3 data recorders trained for ½ day • Each data recorder was given account on the

platform• Entered each questionnaire and uploaded on

the platform • Quantitative and qualitative data was analysed

using platform’s own instant analysis tools• Descriptive statistics on quantitative data and

summary reports on qualitative data provided instantly

20

• Training of RAs for 3 days including simulation & feedback

• Initial supervised/observed telephonic interviews – 5 interviews after training

• Confirmation of interviews – 5 in every 50• Paper questionnaires thoroughly checked• Instant platform-based feedback on data

entry – incomplete questionnaires, inconsistencies, errors in entries, etc

21

• Overall satisfaction and BTVET image– 59% of graduates were satisfied – 48% of employers were satisfied

• Training quality and delivery (efficiency)– 55% of graduates provided a good rating– 35% of employers rated private institutions as

excellent– 38% of employers rated public institutions as good

• Overall BTVET outcome (effectiveness)– 64% of graduates felt that skills acquired made them

ready for the labour market – 63% of employers felt that graduates were ‘prepared’

for applying the skills and knowledge but not up to the required levels of employment

22

Establish effective BTVET Management Information System

Upgrade, modernise and sufficiently resource existing training infrastructure

Intensify and strengthen cooperation/partnerships between training institutions and industry

Enhance female participation Focused BTVET programmes on local skills

demands

23

• Lack of baseline data on both graduates and employers– A base population frame of contacts details had to be

established before sampling• EMIS BTVET data on graduates not disaggregated

by training programmes – Called for prior field visit to public and private institutions

• Lack of up-to-date contact data on public and private institutions across 7 regions– individual research approaches on institutions had to be

conducted• Obtaining employer interviews in the formal sector

was cumbersome as it involved formal approval– Many were left out

24

Some employers required repetitive follow –ups and/or solicitation for nominal incentives◦ Discarded from the list after failure through

persuasion Less eagerness for participation among

graduates in the central region◦ opted for extra interviews in South and South

West to compensate

25

Tracer study periodically reaffirms quality, relevance and effectiveness of BTVET sub sector/related services

Tracer study adequately informs direction for prioritisation

Telephonic interviews minimise time consumed in ‘snowballing’

Metior platform facilitates quick delivery of analysed data particularly in restrictive ‘time bound’ situations

26