Upload
others
View
2
Download
0
Embed Size (px)
Citation preview
Impact Assessment E2B
Govert Gijsbers -TNO
2
Overview Workshop
Programme
• 10.00-10.45 Introductory presentation
• 10.45-11.15 Discussion
• 11.15-11.30 Break
• 11.30-12.30 Towards indicators: Generating
ideas and indicators on long-term impact,
outcomes and outputs
• 12.30-12.45 Wrap-up
3
Introductory Presentation
• An impact case
• Some methodological issues
• The impact assessment process
I. Mini case study
Dr. Pasveer’s Ditch
5
The “Pasveer sloot”
1950s: Sewage treatment was expensive and only available for larger
cities, no cost-effective solution for smaller towns existed.
Dr. Pasveer at TNO set out to develop a low cost solution, mimicking
natural processes in rivers, in a closed system:
• An „oval‟ ditch with an inlet for sewage and an outlet for treated water
• A rotor and brushes to circulate the water and to add oxygen
• Storage and treatment for sludge
• First pilot at TNO in Delft
6
1954: The cutting edge of environmental technology!
- Fully operational plant (capacity 600 user equivalents)
- Excellent technical performance: much better than existing systems
- Cheap: 50% lower cost than existing technology
7
Scaling up and scaling out
- 400,000 user equivalents!
- World-wide adoption
8
1990s: Disadoption -Now an industrial monument
- Stricter norms for emissions (fully aerobic process, energy-
intensive, CO2 emissions)
- Huge surface area required for large scale treatment facilities
- But not the end of the story…
9
Technical improvements: Carousel
model
• Underlying technology is the same
• Used globally today
10
Huge impact – with hindsight (1)
• Initially hard to recognize as a breakthrough technology
• Long term perspective: 1954-1991 (NL)
– still in use in other countries, and the basis for current
technology
• TNO didn‟t commercialize it – never saw a penny in
return
• Dutch engineering companies built systems all over the
world and profited handsomely
• Is that a problem? Public money, public goods?
11
Huge impact – with hindsight (2)
• Huge benefits; different types: financial, environmental,
public health
• How to measure?
– Worldwide spillovers
– Impact on subsequent technology (carousel model)
• Finally: Who cares?
– Then was then and now is now!
– Past performance not a criterion for future funding
– Key issue: IA can only be done after the fact, but then
stakeholders (policy makers) have lost interest.
12
A few lessons
• Impact may take a long time to emerge
– but it may also persist for a long time
• Different types benefits/impacts
– financial, environmental, public health
• Technology adoption can be world-wide
• More actors involved: R&D, companies:
– attribution
• Subsequent technical improvements
II. Some methodological
issues
14
Why Impact Assessment?
• R&D / innovation programmes challenged to
demonstrate their impact on clients, stakeholders and
society
• Two main reasons for impact assessment (IA)
– Accountability (external)
– Learning (internal)
• Much talk about IA, but not much practical work
– Because IA is a difficult issue
• Challenge to develop a workable system
15
Eight issues in IA
1. Level of analysis– Impact of what?
2. Level of analysis– Impact on what?
3. Attribution
4. Time lags
5. Selection bias
6. Non-linearity
7. The problem of the counterfactual
8. Measurement issues
16
1. Level of analysis – Impact of
what?
– Projects, Programs, Technologies, Business Units,
Departments, Organisations?
– Is the impact of the whole greater than the sum of the
parts?
17
2. Level of analysis – Impact on
what?
– Direct outcomes for clients
– Indirect outcomes for clients and stakeholders
(intended and unintended)
– Impact on society at large
18
3. Attribution• Attribution (of impact) a key problem in IA
• With every step away from the direct outputs produced by
an organization or program, the question of attribution
becomes more pertinent
19
The Attribution Gap
20
“Contribution analysis”(to deal with attribution problems)
“Contribution analysis focuses explicitly on the issues of
attribution and contribution and aims to present a
credible story of how a program or project contributes
to solving a problem.” (John Mayne, Canadian Auditor
General Office )
• “Plausible association”: whether “a reasonable person,
knowing what has occurred in the program and that the
intended outcomes actually occurred, agrees that the
program contributed to those outcomes?”
21
4. Time lags
• New technologies may take decades to impact on
society
• Looking back 25 years at past achievements is no longer
of much policy interest
22
5. Selection bias
• “Unsuccessful” projects not well documented
• Needs an estimate of the distribution of project success
rates (representative portfolio)
• Alternatively, focus on a few “blockbusters”
– R&D inherently an uncertain business, where many
project must fail
– a few mega successes may repay the cost of R&D
many times over
23
6. Non-linearity
• The impact chain presents a linear model
– a bit like the linear model of innovation
• But we know that impact is often achieved in roundabout
and unexpected ways
24
7. The problem of the
counterfactual
• We don‟t know what would have happened
anyway (without the project / program in
place)
• With / without
• Control groups?
• (Behavioural) additionality
25
8. Measurement issues
• Quantitative and qualitative approaches– from econometric models to story telling
• Quantification limited by conceptual and data problems
• R&D produces “knowledge”, which is hard to measure, but is more than the number of patents.
• Need to develop indicators that can be measured
• Start with Identification of all outputs (not a trivial task for a major programme such as E2B)
• Numbers supported by qualitative stories
III. Developing an IA process
27
Key steps
1. Define IA requirements
• with key stakeholders
2. Build an E2B “logic model”
– inputs, outputs, outcomes, impact
3. Measurement:
– ID indicators for outcomes
4. Implementation
• responsibilities, etc.
28
1. Impact assessment system
requirements E2B
• Provides relevant information on E2B‟s performance (outputs, outcomes and impact) to different users
• Is accepted by E2B funding agencies as a credible set of instruments
– this is the political aspect
• Is cost-effective in information collection and maintenance
• Strong links to E2B mission and strategic objectives
• Builds on best practice from public and private sector, nationally and internationally.
29
2. Build a “logic model”
• A logic model connects the inputs and outputs of
a project or programme to intermediate and final
outcomes (impact)
• A.k.a.
– program logic
– intervention logic
– outcomes model
– theory of action
– impact pathway
– mental model of the program
30
InputsActivities/Process
Intermediate
Outcomes
Products /Outputs
• Finance
• People
• Infrastructure
• R&D and
innovation
projects
• Technologies
• Publications
• Business
models
• Prototypes
• Services
• Technologies
/ materials
adopted
• Houses
insulated /
energy neutral
• Construction
processes
adopted
• Reduced GHG
emission
Program logic - general
Final
outcomes /Impact
• Incomes
• Employment
• Environment
• Climate
change
• Public health
• Cultural
heritage
31
Example: Overview model
32
Sub model - training
33
Logic model
34
3. Develop indicators
• Start with outputs
– basically counting, but…
– many R&D organisations/ programs do record all
outputs
• Add indicators to outcomes
– (to be) collected by program, or
– collected by others (statistics offices)
• ID “contributions” to aggregate societal/economic impact
• Key point: indicator development comes at the end, not
the beginning
35
4. Implementation
• Assign responsibilities to project partners
• Set up (annual) monitoring, data collection
– primary, secondary
• Complement quantitative indicator-based
information with in-depth case studies on
specific topics
• Communication of impact to funders,
clients, stakeholders
36
Overview Workshop
Programme
• 10.00-10.45 Introductory presentation
• 10.45-11.15 Discussion
• 11.15-11.30 Break
• 11.30-12.30 Towards indicators: Generating
ideas and indicators on long-term impact,
outcomes and outputs
• 12.30-12.45 Wrap-up
E2B expected impacts
38
Impacts: Energy
• New zero energy, energy+ buildings
• Renovated energy efficient buildings
• Increased RES installments
• Reduced energy dependence
– reduced energy consumption
39
Outcomes: environmental
• Reduce CO2 emissions
– 65 Mt /yr
• Low energy content materials
• Reduction of waste, 75% recycling
• Municipal solid wast reduction
– 26 Mtoe/yr
40
Outcomes: social
• 90-150k jobs
• Improved indoor climate:
– CO2 content 400 ppm, humidity 30-50%
• Sound climate (noise)
• Lighting comfort & reduced energy
consumption (using ICT)
• Accessibility
41
Outcomes: Economic
• Investment in efficient & clean technologie
of 12-17,000 M €/yr
• Cost reduction of 10-15% on above
investment
• Shorten time to market
• Increased labor productivity
• Cost savings: lighting, heating, water
– 126-150,000 M€
42
Outputs, outcomes and impact
43
“Contribution analysis”
Contribution Analysis: Addressing Attribution with
Performance Measures
• Acknowledge the attribution problem
• Present the logic of the program – “theory of
action”, “impact pathways”
• Identify and document behavioral changes (clients
or beneficiaries)
• Use discriminating indicators.
• Track performance over time.
• Discuss, and test alternative explanations.
• Gather additional relevant evidence.
• Gather multiple lines of evidence.