2nd Transnational Co-lab Impact Definition & Measurement

REPORT WPT 1 – Social innovation for NEETs and seniors

Organiser: Fachhochschule Vorarlberg

Lab leader NEETs: Eckert Schools Lab leader seniors: PSP PACA

Location: Fachhochschule Vorarlberg, Hochschulstraße 1, 6850 Dornbirn

4th of July

14.00 – 18.00 - Training Session on Social Impact Measurement

1- Methodology & Methods , Matjaž Črnigoj (IER)

2. SROI, Juan David Rivera Acevedo (University of Erfurt)

Social Impact Bonds – measurement

Matjaž Črnigoj and Damjan Kavaš from the Institute of Economic Research provided the project partners

and observers with an overview about social impact measurement. They started with monitoring and

evaluation and why this is actually nothing new. Furthermore, it was discussed what a social impact actually

is. The theory of change was elaborated. The next point on the agenda was social impact measurement –

the current state of the art, social impact measurement in the context of SIBs. Guidelines from the SII

Taskforce for Impact measurement were presented and as a last point methodologies and approaches for

social impact measurement were illustrated.

“Monitoring is the observation of project/programme implementation and performance through a

continuous and systematic process of generating quantitative and in many cases qualtitative information.

Indicators enable the monitoring of the progress. Indicators should be clearly defined, be relevant and

measureable and clearly link the activity to expected outputs and results.”

“Evaluation examines the implementation and impact of the activities that have been delivered, to what

extent the outputs and results can be attributed directly to those activities and looks at whether the

anticipated effects and benefits have been realized”.

It was also emphasized how important it is to start with the evaluation and monitoring at an early stage to

make sure that in the end there will be sufficient data.

2

The next question about was “What is impact and social impact?”. According to O’Donohoe and colleagues

(2010) impact is about purpose, not only profits, in the social and environmental spheres.

Social impact is defined as a significant, positive change that addresses a pressing social challenge (CSI-UM,

2017). Therefore, it is essential to differentiate between the more generic term “impact” and the more

specific definition of “social impact”.

3

Furthermore, it was stated that it is the objective of many companies to have an impact beyond financial

returns and not only of charitable enterprises and investors (Florman et al., 2016).

As a consequence, more than 150 impact assessment methods evolved (Foundation Center, 2015).

Asymmetry of power distribution and information asymmetry impede the evaluation of a social impact.

Therefore, the establishment of an effective system for social impact measurement which could decrease

the information asymmetry is very critical for the development of a successful SII market.

4

It is difficult to measure social impact within a SIB, since the “consumers” of the service do not have to pay

for the service. Therefore, their willingness to pay is not visible.

The Theory of change illustrates how and why desired change is expected to happen within a certain

context. The focus lies on mapping out or filing in the missing link or the black box between an intervention,

programme, initiative and the desired results (CTC, 2017).

5

“The process generates a visual map of the relationships between interventions, outcomes and long-term

outcomes; and articulates all assumptions about how interventions will generate the desired outcomes.”

“TOC is an approach to planning and evaluating activities, projects, services or communities of service to

deliver identified long-term changes, or outcomes.”

Through backward-mapping, the possible interventions or activities will be elaborated. The starting point is

the desired long-term goal. Afterwards, the conditions or outcomes before the long-term goal are

identified, which have to be in place for the long-term objective to occur. In a las step, the activities or

interventions which are necessary for the preconditions of the longer-term objective to occur, are

identified.

This process results in better planning and a better understanding of how change happens and results in

better evaluation since it will be possible to measure not merely outputs but long-term objectives.

6

The theory of change was further illustrated with an example: “Project Superwomen” – an example of

project evaluation (CTC, 2017).

7

The next step is the development of indicators to measure the progress. It was stated that each indicator

has four parts, namely population, target, threshold and timeline. The questions that should be answered

in this regards are:

- Who is changing?

- How many do we expect will succeed?

- How much is good enough?

- By when does this outcome need to happen?

The next step is the identification of the required interventions to achieve the predefined outcome.

Social impact measurement is no standardized yet. There exist more than 150 impact measurement

methodologies (Foundation Center, 2015), like IRIS, SROI and CARS. Furthermore, it was emphasized that

the process of performance measurement is dynamic.

Accounting for Social Value: Social Return on Investment (SROI)

The last presentation on the first day was given by Juan David Rivera Acevedo about “Accounting for Social

Value: Social Return on Investment (SROI)”. During the presentation, the principles of SROI, the stages in

SROI and its merits and limitations were explained. The presentation ended with an exercise about the

theory of change and SROI.

8

The principles are that you have involve the relevant stakeholders, you have to understand what changes,

you have to value the things that matter and you can only include what is material. It is essential to avoid

over-claiming. Transparency is important as well as verification of the results.

The SROI process consists of six stages:

1. Establishing scope and identifying stakeholders

2. Mapping outcomes

3. Evidencing outcomes and then availability

4. Establishing impact

5. Calculating the SROI

6. Reporting and embedding.

9

The advantages of SROI are that it provides NGOs with legitimacy and it can help to allocate resources

efficiently and effectively. Critics can be grouped in three groups. There are the ones that criticize SROI

because of commensuration and the underlying concept of utilitarianism. Others say a limitation is that

SROI ratios have to be interpreted in regards to the methodology used and additional information is

necessary. The third group questions causality, temporality, standardization and quality assurance.

10

During the exercise, the newly obtained knowledge should be used with a practical example. We had to

think of a current situation and the impact we want to see in the end. The stages between had to be filled –

inputs, outputs, intermediary outcomes and long term outcomes.

11

12

5th of July

09.00-15.30 Lab Meetings

Workshop NEETs (Lab leader: Eckert Schools)

2.1. NEETS Co-creative lab: objectives and guidelines Objective: The workshop aims at showing how the evaluation/measurement of a SIB works. Guidelines: A co-creative and participative session, based on logical framework approach, where the participants learn about the relevant factors related to the evaluation/measurement of a SIB work by: - defining a problem - answering the question “What do I want to measure?” - discussing how to operationalize - how to prepare the contract to ensure a valid measurement Proceedings: The workshop NEETs started with the short presentation carried out by the Knut Wuhler from the Eckert School. Evaluation is an integral part of the problem-solving cycle, consisting of the following phases: 1. Identify 2. Explore 3. Set goals 4. Alternatives 5. Select 6. Implement 7. Evaluate Nevertheless, if you do not know the problems of NEETs, do not start the project. That is a case for experts. You need a team. To measure/evaluate SIBs, it is important to: - Define outcome metrics and attribution - Agree on measurement - Establish a new type of partnership between government, the social sector and the investors = Good

Governances In order to build and implement SIBs the following six criteria should be fulfilled: 1. Flexibility: successful SIBs are very flexible. Without flexibility you can’t handle new knowledge and you

won´t be innovative. 2. Rigour: Anyone who doesn’t stick to measurable data and results won´t be able to proof the impact! 3. Partnership: you cannot do by your own, but with the stakeholders. Those who don´t work in

partnership with their stakeholders won´t achieve any synergies they need for innovation. 4. Client-centred: Only those who focus on the participants will work effectively and achieve measurable

results. A system change is not a goal of a SIB. Cause and effect must be representable in a SIB.

13

5. Early intervention: The greatest lever of effectiveness lies in prevention 6. Innovation: A SIB-financed project must be interesting for investors and shouldn´t already be carried

out by the state. 3 groups (app. 5 persons each with diverse profiles) were established. Each group focus on a NEETs challenge (problem), propose interventions, define objectives and measurement/evaluation framework. Interventions are based on the theory of change approach following six criteria described above (flexibility, rigour, partnership, client-centred, early intervention, innovation). Steps: - First, groups should define the problem and objectives. - Secondly, the groups should develop intervention approach (activities) based on theory of change,

starting with objectives. - Thirdly, each group should define measurement/evaluation framework. At the end of the 3 steps, each group presented its idea. 2.2. Results Problem and goal 1. Reduced youth unemployment (more jobs trainings &placements for NEETs around Pordenone)

14

2. Reduced ESL’s rate of 2nd generation migrant students in the city of Turin

15

3. Reduced ESL’s rate (before 16)

16

2.3. Conclusions For each potential project measurement metrics was prepared and discussed. It is important to set indicator, its value and measurement method. There is a paradox: more we try to be innovative more difficult is how to measure. In order to measure results, different methodologies as focus groups, interviews, questionnaires, analysis (statistical data) could be used.

17

Workshop NEETS

(Lab leader and moderator: PSP PACA)

1.1 Introduction

The workshop aims at helping the participants to develop proposal for potential SIB project and to answer

the questions having regards to results coming from the 1st co-creative lab in Augsburg. On the basis of the

selected ideas, we will look further for a full SIB track (thematic and problems to be settled). For instance,

present the role of each actor in the creation of a SIB based on these activities to be developed.

3 topics were selected at the 1rst co-creative lab for seniors :

- Adapted physical activity - Connected housing - Shared housing

1.2. SENIOR Co-creative lab: objectives and guideline

Objectives: Impact measurement and evaluation of a SIB project activity

- Develop 3 proposals of potential SIB projects (activities) for seniors;

- Successive sessions: build on production and on results of the first co-creative workshop on “SIB –

seniors” (User profile, SIB ideas…) in Augsburg, Germany (May 2017);

- Lead participants to consider / imagine themselves as actors in an implementation of a SIB project /

activity (1st Colab : as a “senior user”/2nd Colab : as a “SIB stakeholder”)

- 2nd co-creative lab: focus on impact measurement and impact evaluation

How to evaluate a SIB project / activity? Find progressively some appropriate indicators (technical,

pragmatic and practical)

Time length: 2h30

Proceedings: participants were spit into 3 groups (with divers profiles) following 3 activities selected at the

end of the first co-lab (activities having got the highest score). In that respect, was necessary to enable an

easy space re-organization (chairs and tables). A video projector, felt pens (different colors) Post-it

(different colors) were available to participants. Participants were invited to spit themselves between

nationalities.

The two seniors ID cards (Wilhelm and Bruno) completed at the 1st colab were available for participants.

Each person, in each group, had to choose a role among the ones proposed. The different roles were

identified through a table nameplate and a card with the related stakeholder’s characteristics.

Participants were invited to assume a different role as SIB stakeholders as: Investor; Public authority;

Intermediary; Service provider; Evaluator; Beneficiary

The objective were to find outcomes: for users, for family caregivers, activities; what are the changes?

What we are expecting to change with a sib project on elderly? An increase autonomy impact; what should

change for this person (Willem) ?

The co-creative lab were divided into several successive sequences:

I. WORKSHOP PRESENTATION (15’)

1. Reminder of the seniors profile and the selected activities (created, developed and voted during

the Colab 1)

18

3 project ideas coming from the 1st Colab:

Thematic : ADAPTED PHYSICAL ACTIVITY Senior ID: Wilhelm, 86 years old, lives in a farm in a mountain village in Austria

Particular sign: Loss of appetite since he takes medicines. He is “fragile”. He has no car, but a tractor and a cat. His neighbors are helping him a lot. He likes TV shows.

He has 2 children who are far away from home (Milan and Vienna) and who don’t visit him often. Objective Actions / activities User’s benefits Collectivity’s benefits

Continue maintaining the garden

Start regular exercises More energy Longer autonomy

Lower health cost Postpone entrance in medical institutions

Thematic : CONNECTED HOUSING Senior ID: Wilhelm, 86 years old, lives in a farm in a mountain village in Austria

Particular sign: Loss of appetite since he takes medicines. He is “fragile”. He has no car, but a tractor and a cat. His neighbors are helping him a lot. He likes TV shows.

He has 2 children who are far away from home (Milan and Vienna) and who don’t visit him often. Objective Actions / activities User’s benefits Collectivity’s benefits

To see family more often

Get equipped and trained in ICT

Happiness Lower health cost

Prevent mental health Prevent fragility

Thematic : COLLECTIVE HOUSING Senior ID: Bruno, 84 years old, lives in the countryside in his own house with floors Particular sign: He broke twice last year his neck of the femur because he fell down

He is French and Italian. He has 2 children and lives in small house (with stairs). He needs some physical activity. He has bird and likes playing accordion.

Objectives Actions / activities User’s benefits Collectivity’s benefits

Stay at home 2. Find someone to live at home with the senior (refugees, students)

Help everyday live Less lonely

Housing and social integration of refugees

2. Some information about the selected social issues

Adapted physical activity / connected housing / collective housing

II. ANIMATION (2h)

1. Develop the SIB project (45’)

On the bases of the selected activities at the 1st colab -> develop the SIB project idea:

1.1. Brainstorming about the SIB project idea (15’)

o Define the objectives of your SIB project (general / common and for each role / stakeholder)

o Develop details of the activity / intervention and the project organization (duration, amount,

target group, delivery…)

These data are collectively discussed. They don’t have to be exactly precise, but realistic enough so

that they can be used as a support to co-creative work about the evaluation activity.

1.2. Potential impacts of the project (20’)

19

- Identify the financial and social costs (direct and indirect) of the problem considered

(user/collectivity/family helpers)

o Dependence or loneliness cost: hospitalization, social allowances, institutional care, helper’s

tiredness, mobility decrease, etc.

- Complete the benefits (direct and indirect) of the imagined intervention (user/collectivity/family

helpers)

o Reduced dependence or loneliness feeling benefits: decrease the number of hospitalization,

the social allowances, the entrance in institutional care, respite for the helpers, etc.

1.3. Exchange time (20’)

- 5 min of restitution for each group + exchange time with the other groups

2. Indicators and evaluation activity (1h10)

How to measure the achievement of the formulated objectives?

II.1. Indicators (20’)

On the basis of the intervention benefits list and the criteria which determine the formulation of relevant

indicators (presented during the first day?):

- Find collectively appropriate qualitative indicators

- Identify data source (for quantitative indicators) – where can we find the data to benchmark,

evaluate the impacts of our activities?

Indicator example Wihelm 1 Wihelm 2 Bruno

- Number of fractures - Dependence age - Social relations related to the activity…

- Number of relation trough ICT (use the technologic tool) - Loneliness feeling - Loved ones daily life knowledge…

- Number of fractures - Dependence age - Loneliness feeling

- Activities with the cohabitants…

II.2. Evaluation activity (20’)

- Historical data, test group

- Indicators analysis

II.3. Evaluation frequency during the experimentation (10’)

- Evaluation agenda (each 3, 6, 12 months?)

II.4. Exchange time (20’)

- Participants were invited in the plenary section to provide a 5 min of restitution for each group +

exchange time with the other groups

Results

20

Wihelm 1 Wihelm 2 Bruno

1. SIB Project : adapted physical activity

General objective: To achieve social inclusion in cost/effective manner SIB details: - 3 years - 1 627 500 euros - Target group : from 70;

people from the country side; 100 people join the programme

Details of activity: - Organise transports on

demand (small/smart solutions)

- Organise group helping domestic activities

- Organise soft physical activities (walk, physio) to socialise

Impacts for user: - Revitalized, longer autonomy - Improving of live-conditions

(cooking, cleaning) - More social contacts by

mobility

Impacts for collectivity: - Lowering cost for health

insurance Indicators: Qualitative - How does your day look like? - How often do you leave your

home? - How often do you meet other

people? - How many meals di you have

per day? / What do you eat? - How many hours a day do

you work in the garden? Quantitative - Frequency of visits at the

doctor - Age of people entering

retirement home - How many people are

2. SIB Project : Connected housing (Get equipped and trained in ICT)

General objective : To get him more autonomy and see more often his family SIB details: - 3 years - 30000euros/month Details of activity: (…) Impacts for user: Output: phone, activity, affordable accommodation for young adults Outcome: more connected with family, improve moral and physical health, less lonely Impacts for family caregivers: Output: more calls from grandfather Outcome: more connected to their grandfather, feel less guilty Impacts for collectivity: Improve mental health : low health cost Improve quality of lives Cost savings Indicators: Qualitative - Itw (health, lonely, autonomy)

Quantitative - Participants at the activities - Calls - Activities

Evaluation Activity - Ex-ante, ITW + ex-post ITW - Compare to secondary data

(gvnt, world bank)

3. SIB Project : Shared housing General objective : improving wellbeing of seniors at home SIB details: - 3 years - 500 000 euros - Target group : senior referred by

hospital for mobility problems living alone and in a rural area with low income / young people (18+) disadvantaged

Details of activity: Selection matching Training “intermediation” (monitoring of house sharing) Impacts for user: Output : 20 people living with room-mates Outcome : reduction of fragility and loneliness, increase of mobility and autonomy Impacts for family caregivers: Output : 20 young people living in a cheaper apartment Outcome : improving social skills and personal (maturity) / language skills Impacts for collectivity: Output : 20 shared houses Outcome : reducing health cost of isolated senior Indicators: Qualitative - 1) Perception senior - 2) Perception youngs

Quantitative - 3) 10 months (source : seniors) :

length of cohabitation - 4) Number of hospitalisation - 5) wellbeing assessment tools (mental

and body) Evaluation Activity - 1 et 2 : itw ante/in itinere/post - 3 : Monthly visit reporting - 4: statistical historical data on the

focus group - 5: questionnaire ante/initinere/post

21

involved in the program Evaluation Activity - Historical data for

comparison (public authority, health insurance)

- How much budget was spent before the project and after? (Data from public auth.)

Partipants had a final discussion about the typical issue they had to face:

did you fell confortable with your role ? Did you had conflict between roles ?

In particular the investor's role was very important to better define the purpose of the project.

PA: »As local authority I'm not very happy about the private capitals, question of control, to control the

private side, provide better service with less money«.