Building a Results-Based
Monitoring and Evaluation
System for the
Provincial Government Western
Cape, South Africa
CONTEXT OF THE PRESENTATION
1. Background and rationale
2. Designing of the results-based M&E system
3. Phases of the provincial-wide M&E System
4. Type of data and information produced
BACKGROUND AND RATIONALE
• The primary aim of monitoring and evaluation (M&E) in the Provincial Government
of the Western Cape (PGWC) is to provide information for decision-making focussing
on the policy context.
• Data collected for M&E purposes is often wide-ranging; fragmented ; multiple
reporting requirements
• Make sense of the complexity/ diversity of M&E data collected against indicators,
turn data into useful strategic management information, integrate the data into a
system.
• The Provincial-wide M&E system developed for the PGWC - indicator and data
management system - a Results-based M&E system
• Moving towards an integrated system -with other governance systems- and fully
automated
• Provide strategic management information based on data that meets the criteria of
international best practice.
PROVINCIAL MANDATE• The PWMES for the PGWC focuses on the following three key principles:
– ensuring that the Executive is supported in the monitoring of provincial
performance
– monitoring and reporting the outcomes of the PGWC’s policies, strategies and
more specifically the 12 Provincial Strategic Objectives [PSOs] programmes and
projects
– ensuring that there is provincial alignment with National Government-wide M&E
System ; Performance Monitoring and Evaluation System; Framework for
Managing Programme Performance Information; National Statistical System
(NSS)
– Facilitate M&E for 13 provincial government departments
• The core and supporting processes of Provincial-wide M&E for PGWC is presented in
following diagram.
Provincial- wide Monitoring and Evaluation
Ensure the executive is supported in the monitoring of provincial performance in respect of national and provincial policy, strategy, programme and project implementationMonitor and report on the outcomes of the PGWC policies, strategies and programmes and project as well as monitor and report on the progress of provincial programme and project implementationEnsure provincial alignment to Government Wide M&E System and Performance M&E System
Support Processes
• Determine data and information requirements for the outcome indicators; policy areas and national statistical themes
• Facilitate the analysis of the causal relationships between outcomes and PGWC-specific policies, strategies, programmes and/or projects
Conducting Provincial-wide M&E
Methodologies; Standards and Data
Governance
• Develop and maintain a strategic framework for provincial wide M&E
• Monitor the implementation of methodologies, standards and procedures
Maintaining/Sustaining the
Provincial-wide M&E System
• Information Architecture to support the PWMES and its integration with other governance systems
• Departmental M&E Systems
• National Statistical System
• PGWC M&E forum• Co-ordination of data
producers and data users • Network of M&E
practitioners and special interest groups
• Data Quality Assessment Team
• Facilitate the implementation ond methodologies, standards and data governance for PWMES
• Facilitate workshops on building RBME systems and approaches
Policy context for Monitoring,
Evaluation (M&E)
Automated Provincial wide Monitoring and evaluation system
Monitoring and Evaluation Expertise
Capacity BuildingBusiness Processes
• Programme and Project Management methodologies
• Spatial Information• National Statistical
System
Core Processes
• Provincial alignment with the GWMES and Performance M&E System
• Provincial executive dashboard system meets the national and provincial requirements for Performance M&E
• Design and maintain indicator frameworks for core outcome indicators for provincial policies, strategies and programmes
• Develop and maintain a core directory of common data sources
• Develop and maintain a data management system and data warehouse
Core and supporting processes of Provincial-wide M&E
Data and Information on outcome indicators for MDGs; PSOs; Provincial Strategies; policy areas; statistical themes. Inform Provincial policy development and Provincial Strategic Planning is informed with strategic information
Outputs / Outcomes
The Provincial-wide Monitoring& Evaluation System(PWMES)• The Western Cape Model provides the platform for setting up RBM&E systems for
public sector M&E [13 provincial government departments] in the Province.
• The main aim regarding the use or application of the RBM&E system is to ensure
that it provides the data and information to measure against a core set of indicators
contained within the PWMES.
• It is based on methods of collecting data to measure outcomes and their links to
specific 12 Provincial Strategic Objectives.
• It is also based on linking outputs and their measurements, as identified by each
provincial department- Annual Performance Plans [APP] , and their contribution to
each outcome of the PSOs.
• Such information enables evidence-based decision-making in line with the provincial
government policies, strategies, programmes and projects.
2. Designing of the Results- based M&E System
Principles of RBM&E applied for the PGWC • The logical steps in building a RBM&E system and international results-
based M&E models – such as the Word Bank Model (Ten steps to Results-
based Monitoring and Evaluation), the IMA Model (six-step model) and the
Malaysian Governance System – informed the design and development of
the PWMES.
• To develop the RBM&E system for the Province, a specific sequence of
seven phases interdependent phases formulated
• Seven interdependent phases form the basis on which the PWMES seven-
phase model, also known as the Western Cape Model, is built.
Phases in designing the PWMES• Each of the seven phases constitutes a sub-system.
• These subsystems are interdependent and contained within the
overarching PWMES and the PGWC mandate .
• The subsystems provide the necessary components of the system
so that it can operate as a whole through effective indicator and
data management.
• The subsystems are reviewed annually to maintain an up-to-date
and comprehensive M&E system that will function effectively.
PWMES Provincial mandatePhases (subsystems) Components Core process Supporting processes
Phase 1Readiness assessment and stakeholder engagements
Provincial auditsInternal and stakeholder engagements
Building M&E capacityBuilding M&E expertise
Phase 2Overarching frameworks Conceptual framework
M&E strategyM&E framework
Policy context for M&E
Phase 3Indicator development Indicator frameworks:
PSONational statistical themesMillennium development goals (MDGs)
Maintaining and sustaining the Provincial-wide M&E System
Phase 4Monitoring and results framework
Monitoring and results frameworks:PSONational statistical themesMillennium development goals (MDGs)
Maintaining and sustaining the Provincial-wide M&E System
Phase 5Data management and data assessment
Core Directory of Common Data Sources (official data; mainstream data; provincial administrative data; spatial data)Indicator trends
Maintaining and sustaining the Provincial-wide M&E System
Data assessments on SASQAF application Methodologies, standards and data governance
Phase 6Information architecture Systems’ life cycle
Integration with governance systems Automating PWMES
Programme and project management Spatial informationNational Statistical System
Business processes
Phase 7Planning and sustaining the PWMES
M&E on results-based levelM&E on implementation level
Conducting provincial-wide M&E
The PWMES and its components aligned with the core processes and supporting processes of the provincial-wide M&E mandate
Phase 1Readiness assessment and stakeholder engagements
Phase 2Overarching frameworks for the PWMES
Phase 3Indicator development
Phase 4Monitoring and Results Frameworks
Phase 5Data management and data assessment
Phase 6Information architecture
Phase 7PWMES process: Planning to implement and sustain the PWMES
3. Phases of the PWMES Seven Phase Model
PHASE1:Readiness Assessment & Stakeholder Engagements• Readiness assessment involved conducting provincial audits
• Audits ascertained the capacity and readiness in each department:
– build a RBM&E system;
– be aligned to this provincial-wide system
– critical challenges faced in each department in relation to building RBM&E
systems
• Stakeholder engagement involved :
– identifying relevant stakeholders at local, provincial, national and international
levels;
– institutionalising stakeholder engagement through M&E forum
– external reference group
Policy Choices
Provincial Cabinet
Auditor General
Treasury
Provides Data onCross-Cutting Policy Areas
Compute Outcome &
Output Indicators
(PSO)
National Cabinet
Strategic Directives
Provincial – Wide Performance Management, M&E - Process
Produce:Indicator Assessment Reports
Bi-Annual Report Cards Programme of Action
Programmes/Projects (EPD) & Evidence
Submit Quarterly/Annual Reports
Provincial Strategic Objectives(aligned to National Outcomes)
Data
Provincial Administrative Data(Government Department)
Data
Official Data Sources
(STATS SA)
Data
External Data Sources(Research Institutions)
Compile Indicator Frameworks(Outcomes & Output)
Department Programme Officeand Monitoring & Evaluation
Projects Dashboard (systematic data collection &analysis as projects progress)
Departments
Spatial InformationProducts
Provincial Wide Monitoring& Evaluation System
(PWMES)
Provincial Programme
Office
Informs Cabinet
Report Quarterly on Project Performance Indicators
DotP Policy & Strategy
Annual Performance Plans
Premier’s Reports
Reports
Reports
PHASE 2: Developing Overarching Strategic Frameworks
• In this phase the policy objectives and outcomes for the PGWC were formulated as
agreed upon. The policy context of this PGWC for the period to 2014 is focused on
the provincial strategic objectives (PSOs) as approved by the Provincial Cabinet.
• The development of the strategic M&E frameworks for the PWMES:
– Provide conceptual and strategic understanding: provincial–wide M&E mandate,
RBM&E system and relationship to the policy context of the provincial
government.
– Sets out the strategic approach : how to implement RBM&E; assess how well
the provincial government is doing in meeting its strategic objectives.
– Integrates various nuances of M&E : programme and budget performance and
provincial-wide M&E (results-based M&E approach); the national context of
performance M&E -outcomes-based approach of the Presidency
PHASE 3: Indicator Development• This phase was the starting point in translating the Provincial Strategic Objectives
(PSOs) into broad outcome-based themes and then subdividing these themes into
aims linked to the desired outcomes, for the period 2010 to 2014.
• In this phase a compendium of indicators was selected to measure each aim ;
desired outcome for the PSOs taking into account the national statistical production
areas and global imperatives.
• Each indicator within the compendium was defined.
• The compendium of outcome/impact indicators; reference database
• Suggested measuring instrument to monitor and report on the outcomes of the
PSOs and national outcomes
Institutional Change
Change attributable to project funding /management
OUTPUT INDICATORS
Improvement or change in lives of
citizens
APP PERFORMANCE INDICATORS
OUTCOME INDICATORS
Results based M&E
Implementation M&E
APP: Programme and Budget Performance
Incorporating lessons learned into both
operational and strategic management
BASELINE DATA / EVIDENCE FOR POLICY ISSUE
IMPACT INDICATORS
ORIGINATION OF PROBLEM
STATEMENT
E.g. TEENAGE PREGNANCY, low
levels of Numeracy in Grade 6, ETC
Reduced TEENAGE PREGNANCY, Improved Numeracy levels in Grade 6, ETC
PROVINCIAL STRATEGIC OBJECTIVES
APP
TRANSVERSAL PROGRAMMES AND
PROJECTS
MANAGED THROUGH PROGRAMME AND
PROJECT MANAGEMENT METHODOLOGY
Indicators in the policy context
Outputs
Activities
Inputs
OutcomesResultsDirect
Intermediate
Impacts What we aim to change?
What we wish to achieve?
A long term developmental result at a societal level that is a
consequence of achieving specific outcomes
Medium-term results or consequences for society that are a logical consequence of achieving
specific outputs
The final products, or goods and services
produced by the institution for delivery to the
customer
The processes or actions that convert a range of inputs into the desired
outputs
The resources that contribute to production and
delivery of outputs
What we produce or
deliver?
What we do?
What we use to do the work?
Focu
s of th
e G
WM
&E sy
stem
g
en
era
lly
Focu
s of Pe
rform
an
ce
Info
rmatio
n
Focu
s o
f Pro
vin
cia
l Wid
e
M&
E
Imp
lem
en
tat
ion
M&
EDefinition
sFocus
Resu
lts-
based
M&
E
Ou
tcom
es b
ase
d a
pp
roach
PHASE 4:Developing Monitoring &Results Frameworks• In this phase all the attributes for each indicator selected identified in
order to build a monitoring system.
• The indicator frameworks developed in phase 3 were then extended to
accommodate all the attributes.
• The indicators and its attributes culminated into a monitoring and results
framework for each PSO.
• The monitoring system for each core indicator aims to drive all
subsequent data collection, analyses and data interpretation.
• It also provides the base for the selection of appropriate monitoring
methods, evaluation techniques and reporting requirements.
INDICATOR FRAMEWORK FOR PSOsSTRATEGIC OBJECTIVE 3: INCREASING ACCESS TO SAFE AND EFFICIENT TRANSPORT OUTCOME THEME 3: Improving transport safety
Aims/ Intended Outcomes
Indicator Definition /description
Baseline Frequency of data collection
Data source * Core directory
Data producer
Disaggregation
Reduce number of fatalities on Western Cape roads
Number of road fatalities in a given year in the Western Cape as a proportion of the total deaths
Number of road fatalities in a given year in the Western Cape as a proportion of the total deaths
1 623 (Percentage to be calculated)
Annual Causes of death (Road Traffic Accident Deaths)
Stats SA gender, age
Reduce number of accidents in Western Cape roads
Number of road accidents in the Western Cape in a given year as a percentage of the total number of road accidents in the country
Number of road accidents in the Western Cape in a given year as a percentage of the total number of road accidents in the country
New collection using administrative data
Provincial traffic; Municipal traffic
geographical area
Reduce number of accidents in Western Cape roads
Percentage of road accidents involving pedestrians in the Western Cape in a given year
Number of road accidents involving pedestrians in the Western Cape as a percentage of all road accidents in a given year
New collection using administrative data
Provincial traffic; Municipal traffic
Reduce fatalities in public transport in the Western Cape
Number of deaths in a given year occuring in rail transport as a percentage of total passenger trips in the Western Cape
Number of deaths in a given year occuring in rail transport as a percentage of total passenger trips in the Western Cape
New collection using administrative data
Metrorail
PHASE 5: Data Management and Data Assessment• This phase relates to data collection on the outcome indicators; assess if targets are
being met, analysis and reporting of results, the assessment of the quality of the
data
• The use of the most relevant data source ; the compilation of core directory of
common data sources
• In each data source, identified for the compendium of indicators, profiled and
electronically stored.
• This culminates in a composite system, draws its data from contributory systems
and provides official, mainstream and administrative data sources for the PSO
indicators
• Data quality standards for provincial administrative data records - South African
Statistical Quality Assessment Framework (SASQAF)
Thematic Area: Employment and economic activity. A. Labour Force Matters
Data Source: QLFS Profiling
Purpose:The QLFS is a household-based sample survey conducted by Stats SA. It collects data on the labour market activities of individuals aged 15 years and above who live in S.A.
Type of Data:Labour Force characteristics by :•Employment•Unemployment•Inactivity •Employed by:
• Industry plus sex• Industry and province; etc
•Key labour market indicators•Section 4: Not economically active [NEA}•Section 5: Social demographic characteristics•Section 6: Involvement in non-market activities and labour market status by province
Year:2nd quarter 2010
Tenure:Quarterly
Example 1 of Mainstream Data Profiling
Data Source: QLFS Profiling
Sample size:30 000 households per quarter
Statistical Production Area:•Employment, job creation and decent work•Economic growth and transformation
PSO Link:•Increasing opportunities for growth and jobs•Reducing and alleviating poverty
Limitation:•This survey only covers labour market activities of persons aged 15 to 65 years
Strength:•Quarterly data useful, unlike data being available bi-annually which reflects labour market activities
Source: Stats SA. Statistical release PO211: Quarterly Labour Force Survey. Q2: 2010
Continued…
Data Source: DOE: Exam Results Profiling
Purpose:To manage the calculation and publication of all results for the National Senior Certificate exams
Type of Data:National Senior Certificate exam results
Year:2003 - current
Tenure:Annually
Sample Size:[Results per learning area/subject]
Statistical production area•Education
Example 2 of Provincial Administrative Data Profiling
Data Source: DOE: Exam Results Profiling
PSO Link:Strategic Objective: Improving Education outcomesOutcome theme: Improved National Senior Certificate results
PTMS Data Theme:Skills Development[16.3.f Manage the calculation and publication of all exam results]
Limitations:External development of ‘system’ and maintenance thereof.
Strengths:None indicated
Source: Department of Education: Exam Results
Continued…
PHASE 6: Information Architecture• The information architecture is designed for supporting manual processes,
collecting PWMES information and for managing data collected to measure the
indicators.
• Business Requirement Specifications
• PWMES broader computer-based relational data base.
• This data base contains data collected not only for the PWMES, but also for:
– Annual Performance Assessment System (APAS)
– Executive Provincial Dashboard (EPD).
• The PWMES, as an electronic system, draw its data by interfacing with other e-
platform systems.
• APAS, an electronic application, store output indicator results
• EPD, as an electronic application, receives and captures information on
departmental projects and tracks progress and budget utilisation.
Executive Projects Dashboard(Premier’s Provincial Dashboard)
http://www.capegateway.gov.za
Provincial-wide Monitoring and Evaluation System
PHASE 7: PWMES: Planning to implement
• In this phase one ensures that the PWMES delivers an effective indicator
and data-management system
• The annual review of the subsystems of the PWMES takes place in this
phase.
• This review is pivotal in maintaining an up-to-date comprehensive M&E
system
• Ensuring that the components in each phase adhere to the necessary
policy context, M&E elements and mechanisms for such a system.
4. Types of data and information
generated• Various reports have been produced annually through the effective
implementation of the PWMES
• Examples include:
– Six month report Card
– Annual Data Release
– MDG report
– Performance indicators for Gender, Youth and Disability
– National output indicators
– APRM provincial reports
Thank You
Project Team: Department of the PremierZeenat Ishmail [email protected] Wynford [email protected] Noordien [email protected]
Quality Assurance Faldie Esau [South Africa Labour & Development Research Unit] [email protected] Dr. Ros Hirshowitz [email protected]