Monitoring and evaluating progress towards becoming a more ... · one kind of system to another (ie. transformational change). ... ISET Climate Resilience Framework and characteristics

RMIT University©2015 GCP Workshop Measuring Urban Resilience Tokyo 2015

Global Carbon Project Workshop 7-10 Dec 2015‘Tools and Indicators for Measuring Urban Resilience’

Monitoring and evaluating progress towards becoming a more adaptive and resilient region: lessons from Melbourne

Dr Susie Moloney, Senior Research Fellow, Climate Change and Resilience ProgramCentre for Urban Research, RMIT University, Melbourne.

[email protected]

mailto:[email protected]

Outline

1. Definitions and Context: Adaptive Capacity Building and Urban Resilience

2. Research: low carbon transitioning? (governance, social change and urban planning); adaptation planning, capacity building across local governments

3. Project: How well are we adapting? How can we monitor and evaluate progress across the western region of Melbourne

4. Designing a Framework for Monitoring and Evaluating Climate Change Adaptation – Approaches and Indicators (Lessons from the literature)

5. Piloting: Themes and Indicators

6. Lessons/questions for measuring and assessing resilience


1. Definitions and Context: Adaptive Capacity Building and Urban Resilience Some useful definitions (O’Connell et al 2015:6)

• Adaptability (adaptive capacity): The capacity of actors in a system to respond to shocks and to trends and (if known) the proximity of the state of the system to a threshold and so to influence resilience

• Adaptation: a process of responsive change that improves the ability of a system to achieve desired goals, including by reducing vulnerability to disturbance or threats (including climate change)

• Adaptive governance: institutional and political frameworks designed to adapt to changing relationships in society and between society and eco-systems

• General resilience: capacity of all parts of the system to cope with all kinds of shocks and disturbances, and so be able to avoid crossing thresholds, known or unknown, to alternate regimes or systems. It is sometimes referred to as ‘coping capacity’ and in this report is used synonymously with adaptive capacity.

• Transformation: is physical or qualitative changes in form, structure, function or meaning.

• Transition: the course of the trajectory from one regime of a system to another or from one kind of system to another (ie. transformational change).


What does it mean to build urban climate change resilience

What is urban climate change resilience? (ACCCRN and Rockerfeller2013:

• It is the capacity of cities (individuals, communities, institutions, businesses and systems) to survive, adapt, and thrive in the face of climate related stresses and shocks and even transform when conditions require it

What is required to build urban climate change resilience?

• effective leadership,

• strong partnerships,

• inclusive processes and

• an ability to translate technical data and information into practical action


ISET Climate Resilience Framework and characteristics of resilience (Tyler and Moench 2012)


Systems:• flexibility and diversity; • redundancy and

modularity;• safe failure

Institutions:• rights and entitlements, • decision-making

structures, • access to information• processes that support

learning and change

Agents: • ability to learn, • resourcefulness• responsiveness

2. Research: adaptive capacity building across local governments in Melbourne in the context of climate change

• The necessity to build adaptive capacity at the local scale has led to alternative forms of networked and informal governance

• ‘Climate Change Alliances’ (4 metro/peri-urban, 6 rural)

• Important role in building adaptive capacity of councils

and communities

• How? (Moloney and Funfgeld 2015):– Facilitating institutional interactions horizonally across

councils and vertically between federal, state and local levels

– Regional scale risk assessments, mitigation and adaptation strategies

– Creating opportunities for reflexive learning between members and wider stakeholders, across scales through knowledge exchange, learning and advocacy

– Characteristically adaptive and integrative forms of governance


3. Project Context “How Well Are We Adapting?” (2014-2017): Developing an Monitoring and Evaluation Framework for Climate Change Adaptation (Western Alliance for Greenhouse Action and Victorian Gov’t)


Greater Melbourne: 7000+ km24.4mill (8 mill by 2050)

Western Region (8 LGAs): 4700 km2, 838,500 pop’n

• WAGA Climate Change Adaptation Strategy and Action Plan (2012):– Mainstream adaptation across councils– Embed adaptation planning processes– Review progress of adaptation work carried out by WAGA councils and

regionally

• 88 risks identified (17 severe) : covering council service areas– assets and infrastructure; transport; open space and recreation; natural

environment; emergency management; health and community; planning and building; business continuity



A Framework to Monitor Evaluate and Report (ME&R) on Climate Change Adaptation

Purpose?

This framework will:

• Help us track how councils are managing or responding to climate change

• Monitor the impacts of climate on council operations

• Communicate with the community about climate vulnerability and council action.

• Assess the effectiveness of actions and inform future actions* (i.e. if there are maladaptive actions occurring) (*Eventual goal).

It is designed to:

• Evaluate and report on actions that help manage climate risks.

• Focus on learning rather than measuring success or failure.

• Focus on areas of commonality across all WAGA councils.


Reporting to Who and Why?Two key audiences, council staff and executives and the local community.

Council objectives

• To inform future decision making and adaptation activities

• To improve learning and build capacity

• For comparative purposes

Community objectives

• Improve awareness:– What is happening– What is council’s responsibility

• Assure action is happening

• Climate change education

We want to monitor:

• The delivery of actions with concrete outcomes

• Building the capacity of the council and community to respond to and manage climate impacts.


4. Designing a Framework - Approaches and Indicators:Some key lessons from the literature


https://gallery.mailchimp.com/b38874b25e686137780eb836e/files/M_E_Lit_Review.pdf

AdaptME toolkit (UKCIP 2011)

Are we doing things right?Are we doing the right things?


Why are we doing this?Many possible reasons…..• To evaluate effectiveness

• To assess efficiency

• To understand equity

• To provide accountability

• To assess outcomes

• To improve learning

• To improve future activities or interventions

• To compare with other similar activities or interventions.

(AdaptME Toolkit, 2011)


Understand the different types of evaluation/assessment

• Formative: informs decisions to improve, assess program development

• Summative: assesses overall effectiveness at the end

Source: Learning to ADAPT (Villaneuva 2011, p. 20)


Approach: Input-output-outcome


1. Inputs 2. Activities 3. Outputs 4. Outcomes 5. Impacts What resources go into a program

What activities the program undertakes

What is produced through those activities

The short and medium term effects of an interventions outputs; change in development conditions.

Actual or intended changes that result from the program over the longer term.

Approach: Process based evaluation

• Aims to determine how a programme is being implemented.

• Seeks to define the key stages in a process that would lead to the ‘best choice of end point’ without specifying that point at the outset.

• Aims to measure and reflect on building adaptive capacity but does not define what type of outcomes will emerge.

• Allows for flexibility according to changing circumstances and encourages a learning-by-doing approach. (E.g. build climate change adaptation response into policies, establish an adaptation committee etc.)

• Process evaluation uses quantitative and qualitative data collection and analysis methods.

• The limitations of the process evaluation is that it does not evaluate outcomes.


Process-based indicators Outcome-based indicators• Allow stakeholders/sectoral experts

to choose the most appropriate adaptation action to meet an outcome

• Flexible approach – can adjust to new information as it becomes available

• Most government policy objectives/targets are outcome-based• May be possible to link adaptation objectives with objectives in other policy areas• Likely to be sector-specific


Process-based indicators Outcome-based indicators• Defining a process does not guarantee successful adaptation• A different approach from most other government targets, so often unfamiliar to practitioners• May make it difficult to integrate adaptation objectives with objectives in other policy areas• Not necessarily sector-specific

• Defining an outcome does not guarantee successful adaptation• Risk of being overly prescriptive of adaptation options (specifying suboptimal options)• May be inflexible and make it difficult to introduce new information (though great scope for flexibility in implementing specific actions to achieve outcome)

Disadvantages

Advantages

Developing adaptation related indicators


Source: Harley et al. (2009)

M & E Framework aims to measure performance

• Against objectives– Comparing outputs and outcomes against what a program intended

• Against ‘good adaptation’– Defined by guiding principles such as sustainability, collaborative, open,

effective, efficient, equitable– ADAPT Principles: Adaptive learning; Dynamic monitoring, Active,

Participatory, Thorough (Villeneuva 2011)

• Against a baseline– Conditions change, before and after interventions not always adequate – Shifting baselines….


Issues to consider in dealing with shifting or dynamic baselines?• Moving targets and scenarios

• Adaptive capacity (AC) and vulnerability (V) are dynamic variables

• 3 issues to consider (Villaneuva 2012:18)1. Distinguish between generic and specific indicators (ie. generic

indicators capture underlying causes of vulnerability while the latter target the specific measures undertaken to reduce vulnerability)

2. Evaluation processes snapshot V and AC at the end of a program but this needs to be followed up by constant monitoring over long-term

3. M & E needs to capture the existence of V and AC and those processes that may effect the distribution of V or how capacity leads to action


Baselines for 5 areas (World Bank 2010)• Climate data: climate parameters such as temperature, rainfall, humidity, and local

habitat parameters such as soil conditions and water salinity

• Socio-economic data: Indicators of economic and social well-being in a community, including income, food security, health, and the impact of climate change on these factors

• Data on institutional and policy processes: Capacity and existence of appropriate institutions (official or unofficial) and the legal framework (e.g. existence and implementation of climate change policies.)

• Ecosystem services: The extent to which ecosystem services are affected by the impacts of climate change

• Coping Strategies: What strategies the local population has so far used to cope with climate change variability.



WAGA(Draft)FrameworkNov (2015)

5. Piloting the Framework: Themes and Indicators (examples)

Regional Baselines: Indicators

1. Regional Climate variables (temperature, rainfall, water supply etc)

2. Regional Vulnerability and Resilience:– Regional heat vulnerability/canopy cover mapping– Demographic vulnerability to heatwave impact by age– Number of distributed heat related deaths and illness– Demographic vulnerability to CC (socio-economic disadvantage; recent

migrant distribution)– Understanding community cohesiveness and support structures (surveys)


Community Wellbeing and Emergency Management: Indicators

1. Service Vulnerability and Resilience:– Knowledge of vulnerable people exposed to extreme weather events– Continuity of critical home and community care services– Residents seeking refuge in official council run emergency relief centres

during severe weather events

2. Institutional capacity:– Staff capacity to address climate change in decision making (interviews

and surveys)– Emergency management framework recognises and responds to

changing risk levels with climate change

3. Resourcing and Budgeting:– Tracking long term trends in investment towards preparing

for/responding to extreme weather events

4. Community Participation:– Commuity preparedness to respond to extreme weather events– Community lifestyle and household resilience to a changing climate


Open Space and Water Security: Indicators

1. Service Vulnerability/Resilience: – Type and Volume of water supply used; – variation of gap in water supply compared to demand over time; – water efficiency

2. Institutional Capacity:

3. Resourcing: – changes in long term trend in the cost and frequency of extreme weather

event cleanups– Understanding the impact of climate change on council’s operational

budgets

4. Participation and Awareness:– Understanding community satisfaction with open space over time


Data Collection – Indicator templatePiloting with four councils – training and implementation early 2016


Finally - Challenges and limitations in M & E

• Identifying success (defining)

• Timeframes go beyond the duration of an intervention

• Measuring the impact of an intervention

• Lack of concrete definitions

• Measuring against a ‘shifting baseline’

• Avoiding maladaptation

• Dealing with uncertainty

• Diversity in adaptation scales and sectors


How to address these challenges and what to consider

• Engage broad range of stakeholders to help define what success might be, who benefits, who does not; flexibility of an intervention might be a measure of success

• Systematic M & E, iterative processes, tracking progress helps deal with long time lines

• Contribution rather than attribution of interventions

• Share learning – across organisations and beyond

• Learnings from M & E timed to inform key decisions

• Where, when and to whom do the key learning messages need to be articulated?

6. Questions for consideration: • Clarify purpose? What are we measuring and assessing resilience for

(resilience of what for what and by whom and for whom?)

• Develop a tool as a learning framework? - emphasise adaptive learning to shape and inform the course of adaptation over time, review adaptive management and DRR interventions

• Consider these principles (or something similar) to guide the choice and type of indicators?


Principles Indicators

Adaptive Reflect possibility of changing conditionsDynamic Capture the way processes are changingActive Capture actions rather that statesParticipatory Are developed by and with those affected by

interventionsThorough Include maladaptation indications and capture

how, or not, the intervention addresses the underlying causes of vulnerability

Documents

Monitoring and evaluating progress towards becoming a more ... · one kind of system to another (ie. transformational change). ... ISET Climate Resilience Framework and characteristics