Long term evaluation – summarizing “How-to-do-it” CIVITAS Thematic Groups, Interactive training 7 May 2014, ECOMM 2014, Florence, Italy Anders Wretstrand,

Long term evaluation – summarizing “How-to-do-it”

CIVITAS Thematic Groups, Interactive training

7 May 2014, ECOMM 2014, Florence, Italy

Anders Wretstrand, Lund University

22

« How-to-do » long term evaluation

Each measure

• has one or more objectives

• may be a combination, a bundle, of measures

Objectives may be like

• Modal shift

• Vivid city centre

• Coordinated urban freight

These have to be operationalized into indicators that are

• Linked to objectives

• Quantifiable

• Measurable

• Each objective relate to several measures

• Each measure should be linked to several indicators

33

Impact and process evaluation

Impact evaluation

• is carried out in several steps

• starting with a baseline study (before)

• after implementation (after - short term, 1st or 2nd year)

• after “a while” (after - long term, 5th or 10th year)

• what matters is what impacts the measure has on the particular area that the measure aims at, revealed by monitoring indicators

Process evaluation

• a means to facilitate dissemination and transferability of the results

• tells the story of

• planning and design

• implementation and operation

• reveals the “road to success” (or failure)

44

Short and long term

Short term evaluation

• generally covers one year or year or duration of project funding duration

• uses before and after data

Long term evaluation

• can include projecting impacts into future

• Forecasting, scenario-building

• can involve time series data for programs/measures over years

• running, ongoing surveys

• planned, repeated long-term effect surveys

Different results…

• no impact evidence on short term – but positive on long term

• impact on short term – but no “up-scaling” on long term

• in both cases: the process (the story) is important to capture!

55

About effects – impact assessment

Measure

Before After - Short term

TimeTime

Impact

Impact

Effect of measure

Long termLong term

Beware!

66

About effects – impact assessment

Measure

Before After - Short term

TimeTime

Impact

Impact

Effect of other factors

Effect of measure

Long termLong term

”Business as usual”

77

Evaluation design options

Before MeasureMeasure After I

Measure site

Baseline Impact I

After II

Impact II

Before After I

Control site

After II

Changes?Changes?

Changes?Changes?

The ideal situation with control site – nice but quite rare. To strive for!

88



Measure site

Baseline Impact I

After II

Impact II

After I

Control site

After II

Changes?Changes?

Changes?Changes?

Control site without baseline

99


MeasureMeasure After I

Measure site

Impact I

After II

Impact II

After I

Control site

After II

Changes?Changes?

Changes?Changes?

No baseline data

1010



Measure site

Baseline Impact I

After II

Impact II

Changes?Changes?

No control site to compare with

1111


MeasureMeasure After I

Measure site

Impact I

After II

Impact II

Changes?Changes?

No control site or baseline data to compare with

Common situation when asked to evaluate!

1212


MeasureMeasure

Measure site

After

Impact

Changes?Changes?

No control site, previous evaluation or baseline data to compare with

Common situation when asked to do long-term evaluation!

1313

Examples – measures from DYN@MO

Cases

• planning: SUMP

• health and accessibility: healthy walking

• public transport: more efficient trolley buses

• mobilit management in residential areas

1414

Examples – SUMP

Measure Objective Indicator Method Baseline

SUMP To (plan to) implement more “sustainable” transport measures than previously

“Sustainability” of measures previously implemented and in new SUMP; and those from SUMP that are actually implemented.

Score using checklist methodology (e.g. Ecomobility SHIFT, or a simplified version)

Analysis of recently implemented measures,

One of the objectives and indicators

Very suitable for long term evaluation!

1515

Examples – Planning for healthy walking


Healthy walking

Improve health especially amongst older people in the CIVITAS area

Levels of physical activity that target group in area undertake – self-reported

Questionnaire to 100-200 patients at each Centro de Salud; and a panel of the same 30 people each year.

First survey results before implemen-tation


Very suitable for long term evaluation! But still……difficult to assess long term effects, since people age!!!

1616

Examples – More efficient trolley buses


”Supercaps” Economic efficiency

Operating and capital costs

Calculations using records and collecting technical data

Existing records and data sets


Easy to do short term evaluation as well! Long term evaluation reveals if system e.g. requires more maintenance. Baseline is less of a problem when dealing with technical data!

1717

Examples – Electromobile living


Offer shared e-vehicles and pedelecs for tenants

Less car-dependent lifestyle

Travel behaviour, modal split

Case/control, annual travel diary

First survey


THEORETICALLY easy to do short term evaluation, but might only detect attitudes and awareness! Long term reveals if people substantially have changed their behaviour, e.g. got rid of 2nd cars

1818

Finally

The « How-to-guide on long term evaluation» from CiViTAS CAPITAL

• will be available for all participants after approval as a final document

• through us or Mobiel21

Now, as a hand-out

• Checklist: recommendations for long term evaluation of MM activities

• If you or anyone else have done a previous evaluation of the measure, or

• If no previous evaluation exists

1919

Checklist: based on existing short-term evaluation

Steps RemarksImpact evaluation

Identify policy/measure objectives Break down the policies/objectives into discrete parts

Reflect on cause and effect relations Possible long-term behavioural adaptations and changes?

Select short-term evaluation indicators They should match and measure the discrete objectives

Analyse short-term impact evaluation Good methods to be continued, poor methods to be left out

Revise and refine indicator descriptions Make use of ongoing surveys, revise some indicators if needed

Collect new data Align with short term evaluation to continue the time series

Analyse data Use proper methods for analysis, based on guidelines

Interpret data Are some indicators influenced by other measures? Bundles?

Process evaluation

Evaluation of existing documents New policies in place, upscaling of measures, new funding, etc.

Self-assessment of stakeholders Interviews, workshops: reported failure, success, lessons learned

2020

Checklist: no short-term evaluation available

Steps RemarksImpact evaluation

Identify policy/measure objectives Break down the policies/objectives into discrete parts

Reflect on cause and effect relations Possible long-term behavioural adaptations and changes?

Propose indicators They should match and measure the discrete objectives

Seek for available historical data Which technical and/or survey data sources are available?

Revise and refine indicator descriptions Change of direction may be needed, because of poor data

Collect new and historical data New = current situation; historical = available time series

Analyse data Use proper methods for analysis, based on guidelines

Interpret data Are some indicators influenced by other measures? Bundles?

Process evaluation

Evaluation of existing documents New policies in place, upscaling of measures, new funding, etc.

Self-assessment of stakeholders Interviews, workshops: reported failure, success, lessons learned

2121

Thank you!

Anders Wretstrand

Tom Rye

Contact DetailsContact Details

Lund University

P.O. Box 118, SE-22100 Lund, SWEDEN

[email protected]

[email protected]

http://www.civitas.eu

Documents

Long term evaluation – summarizing “How-to-do-it” CIVITAS Thematic Groups, Interactive training 7 May 2014, ECOMM 2014, Florence, Italy Anders Wretstrand,