World Meteorological OrganizationWorking together in weather, climate and water

WMO OMM

WMO www.wmo.int

5.8 Monitoring and Evaluation of 5.8 Monitoring and Evaluation of Capacity DevelopmentCapacity Development

WMO OMM

GSN STn ERn1KOn12

KOn11

KOn13

Del

Del

Del

Act

Act

Act

Act

Budget

Budget

Budget

Budget

SP

OP (2012-15)

SECRETARIAT RBB (2012-15)

Linkage between the next SP, OP and RBB

WMO OMM

ST 1 ER 1

KOs of TC1 … TC8

Indicators, BaselinesTargets

KOs of RA I … VIIndicators, Baselines, Targets

Secretariat DepartmentsActivities and Budget

ER 2 For every ER, eachRA and TC will define specific Key Outcomes together with Indicators, Baselines and Targets; based on these plans and in close coordination the Secretariat develops deliverables, activities and budget proposals.

Framework of the Results-based WMO Operating Plan and Secretariat Budget 2012-2015

Strategic

levelOperational

level

(5) (8)

ER 1

ER 2

WMO OMM

KO’s , KPI’s and Priority Areas

• ST2: Advancing scientific research and application as well as development and implementation of technology

• ER4:Enhanced capabilities of Members to access, develop, implement and use integrated and interoperable surface-based and space-based systems for weather, climate and hydrological observations, as well as related environmental observations, based on world standards set by WMO.

• KO 4.1: WMO Integrated Global Observing System (WIGOS) is implemented

• KPI– Completion of WIGOS milestones according to WIGOS implementation

plan., – Increased availability of observations for users

• Priority: WIGOS and WIS are priority areas

• Total of 24KOs, most of them have not more than 2KPIs.• Additional 7 KOs on GFCS and DRR as major initiatives

WMO OMM

1. WMO Strategic Plan : High-level statement of strategic directions and priorities expressed through Strategic Thrusts and organization-wide Expected Results and Performance Indicators addressing the Global Societal Needs;

2. WMO Operating Plan: converts the organization-wide ERs into specific outcomes/ deliverables, indicators and targets

3. Secretariat Implementation Plan and budget: converts the organization-wide ERs into Secretariat specific outcomes/ deliverables, indicators and targets and links them to the planned activities and the Secretariat budget;

4. WMO Strategic Executive summary: In a language appealing to those outside WMO and those making decisions to fund NMHSs.

and

Tool/System for Monitoring & Performance Evaluation – Tracks implementation and provides for control and corrective actions for achieving the ERs.

Strategic planning documents

WMO OMM

WMO Strategy for Capacity Development : Why?

• Establish a CD Strategy to ensure, in a holistic approach, all actors in Met/Hydro/Climate work towards the same overall objective :

Facilitating sustainable development of NMHS

Resolution Cg-XVI/B/WP 11.2• Congress noted that a coordinated and cohesive approach for capacity

development would be needed to enhance capabilities of NMHSs in developing countries to meet growing Societal Needs at different levels.

• Congress noted that capacity building is a crosscutting activity and contributes to all Expected Results (ER), especially ER 6 and Strategic Thrust 3 of the 2012-2015 Strategic Plan.

WMO OMM

Definitions and Logic of WMO strategic planning approach

WMO Strategic

Plan WMO OP

2012-2015

WMO OMM

WMO OMM

Strategic Thrust n3Expected Result n6

WMO OMM

Tools to Monitor and Evaluate Progress

• Types: Empirical, Qualitative, Index, breadbasket, case histories, Indicators (available economic, social or environmental)

• Example Collection tools: – Self reporting: Surveys, Visits of PR and national staff

– Country assessments– National Records– Reports: OPAGS, Regional WGs, Fellowship Evaluations, project evaluation

– Country Profile Data Base

WMO OMM

Conceptual Framework for Monitoring and Evaluation

Inputs

WMO governance decisions

TCs and RAs human resources

Financial & human resources:WMO Secretariat

WMO Joint ProgrammesMembers

Partner Organizations’ contributions

Activities

Activities asstipulated in the

WMO Operating Plan based oncontributions

of theTCs, RAand the

Secretariat

Activitiesof WMO JointProgrammes

Activities ofMembers

Deliverables

Deliverables asstipulated in the

WMO Operating Plan based oncontributions

of theTCs, RAs

and theSecretariat

Deliverables of WMO Joint

Programmes

Deliverables of Members

Expected Results

Expected Resultsas stipulated in the

WMO Strategic Plan

Expected Results of WMO Joint

Programmes

M&EM&EM&EM&E

Feedback and learning

WMO Secretariat enabling processes and coordination

Key Outcomes

Key Outcomesas stipulated in the

WMO Operating Plan

Key Outcomesof WMO JointProgrammes

WMO OMM

Capacity Development – and ER 6

Expected Result 6: Enhanced capabilities of NMHSs, in particular in developing and least developed countries, to fulfil their mandates.

Looking at Key Outcomes (Kos) and Key Performance Indicators (KPIs)

How do we measure?

WMO OMM

EC-WG/SOP Questionnaire

• The working group reviewed in 2011 a standardized questionnaire to assess the impacts of the achieved results.

• Such a survey could help create baselines and targets.

WMO OMM

Draft Survey on WMO Impacts of Achieved Results on Members

Survey Questions for Capacity Development,

ER 6

Key Outcomes 6.1-6.4

KO 6.1: NMHSs and Regional Centers are improved, particularly in developing and least developed countries

WMO OMM

KPI 6.1.1: Number of NMHSs and WMO Regional Centers with defined roles in national and regional socioeconomic development activities, in cooperation with international and regional development agencies

1. Has your NMHS been involved in national or regional socio-economic development activities in cooperation with international and regional development agencies?

Yes No 2. Does your NMHS have a defined role in the National Development Plan?

Yes No 3. If yes, please define your key role:

WMO OMM

KPI 6.1.2: Number of NMHSs with programmes and networks for collaborating on service delivery at regional levels

4. During the past two years, has your NMHS carried out programmes or become part of a network for collaborating on service delivery at regional level?

Yes No 5. Does your NMHS carry out programmes or participate in networks on service delivery at Regional level? Yes No 6. If yes, please give details:

WMO OMM

• KO 6.2: Infrastructure and operational facilities of NMHSs and Regional Centres are improved, particularly in developing and least developed countries

WMO OMM

KPI 6.2.1: Number of NMHSs with improvement in infrastructure and operational facilities to address identified deficiencies

7. During the last two years has your NMHS implemented or developed improvements to its service delivery capacity? Yes No 8. If yes,

i. In the surface observing network? Yes No

ii. In the upper-air observing network? Yes No

iii. In equipment for meteorological/environmental satellite data? Yes No

iv. In the data-processing/forecasting sector? Yes No

v. In other operational sectors of your NMHS (please specify)

WMO OMM

KPI 6.2.2: Number of NMHSs with modernized capacities to develop and disseminate products to their customers

9. What distribution systems do you use to get products and services to users?

i. Telephone of facsimile

ii. Media (printed, TV & Radio)

iii. SMS

iv. Cell Broadcast

v. Internet (web pages, ftp, email etc.)

vi. Social networking (twitter, face book etc)

vii. Dedicated telecommunication links

viii. Remote displays (electronic bill boards etc)

ix. Staff attachment to users of centres of operation

x. Remote briefing (video conference, telephone, help desk etc.)

xi. Others (please list)

WMO OMM

• KO 6.3: Education and Training Development activities at national and regional levels are improved, especially in developing and least developed countries

WMO OMM

KPI 6.3.1: The number of RTCs providing education and training support for GFCS related activities

• To be completed by WMO Secretariat using other sources.

WMO OMM

KPI 6.3.2: The degree to which Members are getting value for money from the WMO Fellowship Programme

10. Has your NMHS contributed to or had students under the WMO Fellowship Programme during the past two years?

Yes No 11. If yes,

In general terms, how would you rate the value your NMHS has been getting from the Fellowship Programme?

1= very low; 5= very high

1 2 3 4 5

WMO OMM

KPI 6.3.3: The degree to which the RTCs support the regional training demands in particular for students from developing and least developed countries

12. If you send staff to a WMO Regional Training Centre(s), please indicate by inserting ‘X’ the WMO Region(s) where the RTC(s) are located:

WMO Regions I II III IV V VI 13. From your perspective, how would you rate the degree of satisfaction with the RTC(s) you use?

1= very low; 5= very high 1 2 3 4 5

WMO OMM

• KO 6.4: Capacities of NMHSs are enhanced through cooperation and partnerships with other national and regional organizations

WMO OMM

KPI 6.4.1: Development projects and activities funded through voluntary and extra-budgetary resources

• To be completed by WMO Secretariat using other sources.

WMO OMM

Your inputs…ideas?• Questions in Survey….

– Are we measuring what needs to be measured to show the capacities of NMHS are being developed?

– Are the questions • Too simple (yes/no) to be of use?• Too difficult to answer with enough specificity to show

improvement in 4 years?• About right

– What other tools does WMO need to develop/improve (if any) to evaluate and monitor CD?

WMO OMM

Additional Sources

• Survey on the Role and Operation of Meteorological Services (2011)

• Regional Surveys• Programmatic Surveys• Information from assessments done on bilateral

basis, other development agencies or at sub-regional levels as part of project development

WMO OMM

Annex 1KPI reference

sheet to be completed

WMO Strategic Plan 2012 – 2015

Key Performance Indicator(KPI) Reference Sheet KPI:

Expected Result: Key Outcome:

UNIT OF MEASURE

PLAN FOR DATA ACQUISITION

Data collection method:

Data source(s):

Timing/frequency of data acquisition:

Estimated cost of data acquisition:

Responsible department:

Responsible individual(s):

DATA QUALITY ISSUES

Data limitations (if any):

Actions planned to address data limitations:

PLAN FOR DATA ANALYSIS, REVIEW, & REPORTING

How the data will be analyzed:

How data will be presented:

Reporting date and to whom:

WMO OMM

Annex 2Exampleof a duly

completed KPI Reference

Sheet

WMO Strategic Plan 2012 – 2015

Key Performance Indicator (KPI) Reference Sheet 5.2.1: Increased research on operations products and services

Expected Result 5: Enhanced capabilities of Members to contribute to and draw benefits from the global research capacity for weather, climate, water and environmental science and technology development Key Outcome 5.2: Research in the prediction of high-impact weather on time scales of hours to seasons is enhanced

UNIT OF MEASURE

Assessment by JSC of WWRP on progress in high-impact weather.

PLAN FOR DATA ACQUISITION

Data collection method: annual meetings of WWRP JSC

Data source(s): reports from WWRP working/expert groups/team

Timing/frequency of data acquisition: once per year

Estimated cost of data acquisition: minimal

Responsible department: RES/ARE/WWR

Responsible Individual(s): C/WWR

DATA QUALITY ISSUES

Data limitations (if any):

Actions planned to address data limitations:

PLAN FOR DATA ANALYSIS, REVIEW, & REPORTING

How the data will be analyzed: review of reports by JSC members, discussion at annual meeting

How data will be presented: report from annual JSC/WWRP meeting

Reporting date and to whom: June each year; to CAS and WMO leadership