FORMATIVE AND SUMMATIVE EVALUATIONprojectsmartmap.eu/wp-content/uploads/2019/04/D_7_2_final.pdf · understood rri/RRI to be, and the challenges, opportunities benefits and costs that

1

www.projectsmartmap.eu

#euSMARTmap

Sally Randles, Mohammad Hajhashem, Monica Gonzales, Eniko Demeny, Peter Kakuk

7.2 Deliverable (D 7.2) of the

H2020 SMART-Map

Project

FORMATIVE AND SUMMATIVE EVALUATION

FINAL REPORT

https://www.facebook.com/SMART-map-Project-171652993255775

https://twitter.com/euSMARTmap

https://www.linkedin.com/groups/11900187

http://www.projectsmartmap.eu/

2


Contents

1. RATIONALE: The Role and Contribution of Formative/Summative Evaluation ................................................................................................................................4

2. METHODOLOGY......................................................................................................................12 3. FINDINGS .................................................................................................................................43 4. DISCUSSION AND CONCLUSIONS ..............................................................................55 5. REFERENCES............................................................................................................................60 6. APPENDICES............................................................................................................................62 7. ACKNOWLEDGEMENTS......................................................................................................96

List of Tables:

Table 1 – Definition of each actor group

Table 2 – Merged dataset: 6 countries, 3 technologies, and total analyzed by stakeholder category

Table 3 – Schedule of the SMART-Map Industrial Dialogues

Table 4 – Actor categories in the sample

Table 5 – Technology

Table 6 – De-facto responsible research by actor group

Table 7 – De-facto responsible research by technology area

Table 8 – De-facto responsible innovation by actor group

Table 9 – De-facto responsible innovation by technology area

Table 10 – Awareness and recall of RRI






http://twitter.com/@euSMARTmap

http://www.linkedin.com/groups/11900187/profile


























3


Table 11 – Awareness and recall of RRI by actor category

Table 12 – Awareness and recall of RRI by technology area

Table 13 – Awareness and recall of RRI by stakeholder group

Table 14 – Awareness and recall of RRI by technology area

Table 15 – Relevance of EC ‘5 keys’ of RRI by technology area

Table 16 – Benefits for individuals

Table 17 – Benefits for society – by technology area

Table 18 – Challenges for individuals – by technology area

Table 19 – Challenges for society – by technology area

Table 20 – Challenges for society – by actor group

List of Figures:

Figure 1 – Potential Stakeholders for Precision Medicine (responsible) innovation ecosystem.

Figure 2 – Potential Stakeholders for Synthetic Biology (responsible) innovation ecosystem (UK).

Figure 3 – SMART-Map Stakeholder Breakdown

Figure 4 – Stakeholder Groups per Country

Figure 5 – Coalition of the Willing

































4


1. RATIONALE: The Role and Contribution of Formative/Summative Evaluation

1.1 A brief literature review. The establishment of a specific stream within the field of Evaluation Studies seeking to provide definitional clarity between formative VERSUS summative evaluation, reaches back at least 50 years. Early origins are attributed to Scriven (1967) working in the area of education performance. Motivated by the desire to more effectively assess pupils’ learning and out-turn attainments, Scriven’s foundational work provided a framework for discussing the philosophical and practical distinction between formative and summative evaluation . Similarly, in the context of student performance in N.American Universities, Murray (1980) says that formative evaluation is developmental, with the primary objective being to positively support learning through longitudinal monitoring and systematic feed-back to the learners. Whilst summative evaluation by contrast, provides ‘judgements’ by evaluating end-point performance against pre-set criteria and has a focus on supporting out-turn decision making.

A comparison of summative and formative assessment methods in education highlights the key differences1:

Summative Assessment Formative Assessment

When At the end of a learning activity During a learning activity

Goal To make a decision To improve learning

Feedback Final judgement Return to the material

Frame of Reference Sometimes normative (comparing a student against all others); sometimes criterion

Always criterion (evaluating students according to the same critera)

1 Adapted from R. Pregent, Charting your course: How to prepare to teach more effectively, Atwood, 2000. https://en.wikipedia.org/wiki/Formative_assessment

































5


In a different area, evaluating a new and experimental medical intervention, Stetler et al (2006), provide a useful commentary on the contexts that are best suited to formative evaluation. The authors stress its appropriateness in situations where the project or programme being evaluated is novel, experimental, dynamic and unfolding. Formative evaluation helpfully accompanies such a process and is appropriate as a recursive (learn/evaluate/feedback/adapt/learn) approach. Its primary aim being to help the actors involved in the process to learn and adapt alongside the intervention. They say:

Formative Evaluation is: “… a method of judging the worth of a program while the program activities are forming or happening” It comprises: “….evaluative activities undertaken during the design and pretesting of programs to guide the design process” As an assessment method it focuses on: “…the internal dynamics and actual operations of a program in order to understand its strengths and weaknesses and changes that occur in it over time” It is well suited to situations and research settings where the evaluators are seeking:

i) Rapid-response learn and adjust processes during the length of the project

ii) Understanding the nature and significance of the local implementation setting

Stetler et al , 2006.





































6


Today, the early origins of formative and summative evaluation studies continue into contemporary scholarship and practice, with significant attention paid to education policy and practice at all levels, from Bloom’s edited volume (1971) Handbook of Formative and Summative Evaluation of Student Learning; to Guyot (1978) who focuses on business education; to Murray (1980) and Harlen and James (1997). Most recently in the education context, attention moves to different branches and disciplines within education, such as Brophy’s Ed (2019) handbook of assessment, policy and practice in music education.

Critical thinking on formative and summative evaluation as an object of study ie ‘evaluating evaluation’, has extended from its origins in Education, to new policy areas such as medicine, health and social policy (Stetler et al 2006, Greve 2017). And yet, only relatively recently have issues of comparative evaluation and impact assessment been taken up to evaluate programmes in science, research and innovation policy (Edler et al eds., 2016). This extension, and the proliferation of applications to an ever-wider array of public policy areas, can likely be attributed to greater demand for evidence-based scrutiny of policy programmes, under the narrative of increased accountability to the tax-payer. It sits against a backdrop of reductions in government spending at all policy levels: local, regional, national and international, and is accompanied by increased use of evaluation instruments at all levels. More recently, a powerful driver is the demand from policy audiences for evidence-based ex-post justification to support policy-spend in particular areas and defend budgets from being switched to alternative priorities. By demonstrating positive policy impact to the benefit of the full range of beneficiaries targeted under a particular instrument, these pressures can in-part be mitigated. This demonstrable-impact agenda has become a particularly acute policy imperative.

Interestingly, the OECD Directorate of Education has also picked up the formative/summative evaluation debate, producing a critique of the overly polarised representation of formative/summative evaluation methods (Looney, 2011). Looney argues for greater integration of formative and summative elements within evaluation study designs. The author shifts attention to declare that formative evaluation captures bottom-up learning whilst summative evaluation is appropriate for assessing system-level impact. Summative evaluation addresses questions of accountability, value for money, and effectiveness in meeting the original policy objectives. Thus both formative and summative approaches are

































7


better considered in terms of their ‘fitness for purpose’ under different project/programme/policy contexts and conditions. Here, the motivation and aims of the evaluators and the funders of the projects and programmes is an important consideration. Under this interpretation, formative and summative elements can be combined or blended into study-designs, in a complementary rather than polemic fashion.

It is this more ‘blended’ approach that characterises the SMART-Map evaluation.

1.2 Formative and Summative Evaluation Applied to the SMART-map project.

WP7 of the SMART-Map project implemented a blended formative and summative evaluation process, designed to ‘track’ the learning journey of the project across six countries , three emerging technologies (Precision Medicine, Synthetic Biology, and 3D biomed); and longitudinally across the full 30 months length of the SMART-Map project (1st May 2016 to31st October 2018).

In line with the ethos of summative evaluation, the project’s own objectives and goals provide the opening criterion against which to evaluate the project’s performance and achievements:

In addition, the evaluation team were keen to establish a ‘Baseline’ position at the beginning of SMART-Maps project. One important aim of the Baseline (quantitative) survey was to

The SMART-Map goal:

…..to define and implement concrete roadmaps for the responsible development of technologies and services in three key game-changing fields: precision medicine, synthetic biology and 3D printing in biomedicine.

http://projectsmartmap.eu/about/ accessed 10 October 2018





































http://projectsmartmap.eu/about/

8


capture respondents’ understandings and interpretation of responsible research and innovation before they engaged in the SMART-Map longitudinal process. Lets call this rri, or ‘little rri’ as it captures respondents de-facto position on what responsibility in research and innovation means to them before they are exposed to the European Commission’s definition of RRI through the instrument of the SMART-Map Industrial Dialogues (IDs). Let’s call this EC policy definition of RRI, and indeed other policy-defined frameworks2 of Responsible Research and Innovation ‘Big RRI’. The dynamic relationship between the participants de-facto understanding of what it means to practice research and innovation in a responsible way (rri); and their awareness, views and experiences of top-down policy framework of RRI according to the European Commission (or the different interpretation of RRI in the UK and elsewhere) we can call rri/RRI.

The Baseline survey was implemented with all those who had agreed to attend their respective country- level SMART-Map Industrial Dialogues (IDs), with the Baseline interview taking place around a week before they actually did so. The primary purpose of this Baseline was to establish the opening views of the full range of SMART-Map participants, across the different actor groups (Industry, University researchers, Civil Society, Intermediaries, Funders, and Regulatory & Political), in order to ascertain whether actor-group made a difference to participant’s awareness, views, and disposition towards rri/RRI. Likewise, analysing the Baseline survey enabled the evaluators to test whether the working context of engagement with the three technologies made a difference to either the understanding of rri/RRI or their predisposition to engage with rri/RRI; comparing opinions before and after the IDs. The results of the ‘before’ Baseline survey was reported back to the consortium members and the ID participants at the start of each ID event. This before/after approach thus enabled a formative

2 In the UK, RRI is implemented by the UK Engineering and Physical Sciences Research Council (EPSRC) under the AREA framework: Anticipate, Reflect, Engage and Act. This framework was influenced by the work of UK academics Jack Stilgoe, Richard Owen and Phil Macnaghten who proposed the four-dimension framework Anticipation, Reflexivity, Inclusivity, Responsiveness (Stilgoe et al 2013) under the term Responsible Innovation (rather than Responsible Research and Innovation). A longer history of reflection on Responsible Innovation (and the originator of the term) can be attributed to David Guston and colleagues in the USA, sustained today under the international network funded by the US National Science Foundation, the Virtual Institute of Responsible Innovation (VIRI). Within Europe, leading academic thinkers, hail from the direction of philosophy in Science/Society relations, colleagues at the Universities of Twente (Arie Rip, Stefan Kuhlmann), Delft (Jeroen Van Den Hoven and colleagues), and Waggeningen (where Vincent Blok and colleagues pay particular attention to the connection with Business Ethics ) and Germany’s Karlsruhe Institute of Technology (Armin Grunwald and colleagues). This work saw a particular take-off in 2010, and at the same point saw an extension in technology scope, from origins in Nanotechnology. An important actor working from a more practitioner perspective is the Bassetti Foundation in Italy.

































9


(learning) step, as the ID participants were able to reflect on what, as a collective, they understood rri/RRI to be, and the challenges, opportunities benefits and costs that rri/RRI presented. In so-doing, we were able to feed these collective views back to the ID participants as part of the opening/introductory steps of the ID deliberative process. This process also enabled the participants to begin to see, and appreciate, how participants different pre-ID actor-group positions informed the ID deliberations.

By returning to a selection of the ID participants after the Industrial Dialogues the Baseline survey enabled the evaluation team to undertake ‘before/after’ quantitative telephone or skype interviews in order to evaluate how the experience of participating in the IDs influenced their views on rri/RRI and the extent to which the immersive co-constructio experience of the IDs increased awareness and reflection on RRI (partly in terms of highlighted what activities they currently undertook which came under the banner of RRI even if they hadn’t come across the term RRI or RI before).

The ‘after’ interviews also ascertained whether respondents were more, or less, positively disposed towards rr/RRI after the IDs than they had been before the event, in terms of awareness and knowledge of the concept, and whether they would be more (or less) likely to think and act with an eye to RRI pronciples, after the ID workshops than before. The post-ID call-back interviews therefore aimed to gain some insight into the extent to which concrete and practical immersive participation in the IDs changed participants views and their disposition towards thinking and acting in a way that was more positively oriented towards RRI, than they had before, where this criterion of increase in positive orientation to RRI was considered an intermediate positive impact of the project. The ‘after ID’ Baseline interviews were therefore more summative in orientation.

This example from the two stages of the SMART-Map evaluation, ‘before/after the Baseline’, shows how formative and summative elements were blended into the first stage of the SMART-Map evaluation. A similar approach was taken to the qualitative interviews undertaken before/after the 2nd stage of SMART-Map, ie to evaluate the learning and experiences of the much smaller cohorts who participated in the 5- month ‘Tool development pilots’ which took place in three of the six countries which had implemented the IDs (Spain on precision medicine; Italy on 3D printing in biomed; and UK on synthetic biology). The two-staged SMART-Map design (the six Industrial Dialogues which prototyped three RRI tools per country,

































10


followed by the 5-month development of one tool in each of 3 countries) produced the systematic methodology and process which became the basis of the three ‘ concrete roadmaps for the responsible development of technologies and services in three game-changing fields ‘ (ie the primary objective and success-criterion of the SMART-Map project.

Legacy and multiplier effects

In addition to the primary goal of the SMART-Map project, to define and implement concrete roadmaps for the responsible development of technologies and services in three game-changing fields; the project was also keen, as a secondary objective to affect, and provide evidence of transformative change; and to provide a lasting legacy of impact, scale-up and multiplier effects.

Legacy and multiplier effects were anticipated as a project outcome, in the original SMART-Map Description of Action, p.24. as:

a) Creating learning material which can be shared, reused and capable to penetrate teaching expertise of other people as well as the SMART-Map participants themselves.

b) Training involving the SMART-Map Champions.

In total, the SMART-Map evaluation uses a rich blend of formative and summative approaches, to ‘track’ the experiences, learning, outputs and outcomes, legacy and impact of the SMART-Map project, across 6 countries, 3 technologies and multiple participant actor-categories, longitudinally through the 30month journey of the SMART-Map project.

It comprises formative evaluation elements to assist learning, and summative elements to enable the comparative evaluation of outputs, outcomes, challenges and benefits, legacy and impact, to aid decision making. Together, the formative and summative evaluation of the SMART-Map projects enables us to assist the following audiences:

































11


In sum our approach can be defined as:

The SMART-Map consortium : to provide early/recursive learning, feedback and adjustment, by highlighting successes, but also highlighting signs of ambiguity or misunderstandings from the participants perspective in the projects aims, methods, processes, to enable the project consortium to improve reflexive clarity along the SMART-map journey. The SMART-Map participants : to provide a mode of collective learning and feedback during the course of the project and to support participants expressing themselves through anonyous ‘voicing’. The funder : to facilitate systematic longitudinal/learning and evaluation of publicly funded projects by tracking the progress of an EU-funded RRI project, in order to assist policy learning and decision making. The researcher : to demonstrate the use of blended formative and summative evaluation design and techiques, applicable to the contexts of publicly funded, new and experimental interventions across a range of reseach and innovation policy areas.

Baking recursive learning into the fabric of the project.

(Randles et al 2017)









































12


2. Methodology

The SMART-Map evaluation used multiple quantitative/qualitative methods across 5 stages of the evaluation process:

Title Brief Description of Research Objectives and Method

Timeframe

Stage 1

Creating a Data-set of multiple actors eligible for participation in the SMART-Map project

To produce a combined data-set of 100+ actors per country/technology, using social networks, directories, snowballing and other methods to collect contact information. Quota: no less than 50% of the contacts provided by each country to be industry contacts. This data-set was used to survey ID participants for Stage 2; Baseline ‘Before’.

Sept-Oct 2016

Stage 2

Baseline ‘Before’

To design and undertake a quantitative ‘Baseline’ survey of participants, 1-2 weeks before each Industrial Dialogue in order to capture participants awareness and views of rri/RRI before attending the ID.

Dec 2016 – June 2017 ie two weeks before the first ID (Aarhus 16-17 Jan 2017) to two weeks before final ID (Hungary, 12-13 June 2017). The Baseline survey methodology comprised a programme of ‘rolling’ telephone and skype Interviews process ie undertaken in each country with the ID participants 1-2 weeks before the ID.

Stage 3

Baseline ‘After’

This comprised a smaller/shorter ‘call-back’ interview with a selection of participants in order to capture their experiences of attending the ID, and any changes in their predisposition to rri/RRI as a result of attending the ID.

Jan 2017- end June 2017. Telephone and skype ‘rolling’ interview programme, undertaken in each country with ID participants one week after

































13


the first Aarhus ID to one week after the Hungary ID.

Stage 4

Pilots ‘Before’ This comprised qualitative interviews undertaken with ID participants and relevant consortium members in the three countries (Spain, Italy, UK) about to embark on the ‘RRI Tool selection and piloting’ stage of SMART-Map. It sought to establish participants’ expectations of the pilot’s process, so that before/after views on the pilots process could be gleaned.

Oct 2017- April 2018. Ie commencing at the start of the 5-month pilots (pilots took place Nov 2017-March 2018).

Stage 5

Pilots ‘After’ A smaller set of qualitative interviews were undertaken with key ‘Champion’ participants in the three countries (Spain, Italy and UK3) at the end of the piloting process in order to ascertain their views on the process and outcomes, (ie what did they expect and did the piloting experience meet or exceed their expectations?). Key participants in each country/technology were also asked whether and how, the pilots experience had produced, or had the opportunity to produce, lasting transformative change, legacy and impact, beyond the immediate life-span of the SMART-Map project.

April 2018-May 2018. Ie spanning before/after the 5 month long pilots

3 For the Synbio pilot in UK, two practitioner participants from the Hungary ID were also interviewed under Stages 4 & 5 (a small biotech company, and the head of a research institute in Hungary), in order to maintain the cross-national knowledge exchange and learning dimension in the UK study. In the Italian pilot, the focus of piloting attention was training undertaken through the conduit of the EU project network SYMBIONICA, which wasn’t limited to Italy, therefore respondents under Stages 4 & 5 on 3D Printing in Biomed included respondenst from beyond Italy.

































14


Stage 1 Creating a Data-set of multiple actors eligible for participation in the SMART-Map project The SMART-Map consortium leaders for each country were asked to provide an Excel spreadsheet giving contact information for the multiple stakeholder groups (Industry, University researchers, Civil Society, Intermediaries, Funders, Regulators and Political actors) relevant to the particular country/technology focus under their jurisdiction. By way of guidance, given the industry focus of the SMART-Map project, a quota of 50% industrial actors was requested of those compiling the list in each country.

Table 1, below, provides a working definition of each actor group.

Table 1 Definition of each actor group: Industry New products and services development and

sales organisations, with a commercial and/or market interest

Multi-nationals; Nationals; and SMEs, inc university spin-outs.

Intermediary Interdisciplinary actors who their main task and role liaises between other actors

Covers both formal intermediaries (industry and trade associations) and de-facto intermediaries.4

University University scientists, both natural and social scientists

Funding Entities in charge of providing finance for different projects. They can be public, private or semi-private. Seed funds, VCs,

Eg. Research Funding Councils; Charities providing funds to science and research; Venture Capitalists; Business Angels; Banks; Ethical investment;

Regulatory National Quality infrastructure (NQI) actors Eg. Testing labs, Quality assurance bodies, Standards development organizations

Political Political and policy actors, at regional, national and European levels.

Eg. MPs, MEPs, National government departments and civil service.

4 We need to crucially consider individuals who sit astride (and indeed in both) of two or more categories: ’two-hatted’ individuals, eg scientists who also work in spin-out companies; actors in one category (scientists, companies) who are also have policy-influencing or political roles; who have high reputation and legitimacy to bridge two communities. For the purpose of our Stakeholder boxes we have put them in a ‘first/main’ category according to their self-description. But this is conceptually flawed and in one or two examples directly flies in the face of how they wish to be categorised (as sitting within two or astride two or more categories). The asymmetric influence of these individuals in the system also makes it conceptually inappropriate to put them in one category. A ‘network’ mapping would be a better way to capture their intermediary role and its significance as boundary-spanning actors. See the classic text by Star and Griesemer (1989) on Boundary Objects. These observations are material to our conceptual work questioning the existence and conceptualisation of ‘Responsible Innovation (eco)systems’ (See SMART-Map deliverable report D7.1).

































15


Civil Society Civil Society and societal representative organisations.

Eg- NGOs, charities, lobbying and campaigning for consumers, environment, health and patients’ groups, public participation in science, civil society expertise in responsible governance of new and emerging technologies etc.

By way of a guide, each country first produced a ‘mapping’ of the actor-landscape relevant to their country/technology. Below provides the actor mapping for Precision Medicine (responsible) innovation ecosystem, ie all actors, including civil society actors.

Figure 1 - Actor Mapping for Precision Medicine (Source: Kaisa Granquist, ZSI)

A further depiction of a potential actor-mapping for Synbio (responsible) innovation ecosystem in UK is provided below (Figure 2, source: Ros Le Feuvre, University of Manchester).

































16


Using these guidance mappings as templates, the amalgamated contact lists provided by each country translates into the total SMART-Map dataset used in Stage 1 of the SMART-Map evaluation.

































1


The combined Stakeholder dataset Below are presented tables and graphs which display the key descriptive statistics of the merged 6 country dataset analysis (where the total number of entries N=738). Table 2 Merged dataset: 6 countries, 3 technologies, and total analyzed by Stakeholder category

Numbers UK HU Synthetic Biology DK SP

Precision Medicine GE IT

3D Printing in Biomed Total

Industry 53 41 94 40 41 81 87 45 132 307 Intermediary 7 6 13 17 18 35 20 20 40 88 University 17 24 41 3 20 23 25 20 45 109 Funding 6 5 11 5 5 10 15 18 33 54 Regulatory 5 7 12 0 4 4 1 6 7 23 Political 7 19 26 13 8 21 8 4 12 59 CSO 22 14 36 16 5 21 20 21 41 98 Total 117 116 233 94 101 195 176 134 310 738

Percentage UK HU Synthetic Biology DK SP

Precision Medicine GE IT 3D printing in Biomed

Industry 45.3 35.3 40.3 42.6 40.6 41.5 49.4 33.6 42.6 Intermediary 6.0 5.2 5.6 18.1 17.8 17.9 11.4 14.9 12.9 University 14.5 20.7 17.6 3.2 19.8 11.8 14.2 14.9 14.5 Funding 5.1 4.3 4.7 5.3 5.0 5.1 8.5 13.4 10.6 Regulatory 4.3 6.0 5.2 0.0 4.0 2.1 0.6 4.5 2.3 Political 6.0 16.4 11.2 13.8 7.9 10.8 4.5 3.0 3.9 CSO 18.8 12.1 15.5 17.0 5.0 10.8 11.4 15.7 13.2


18


Fig 3, below provides a breakdown of the SMART-Map Combined Dataset by actor type.

Finally, Fig 4 provides the dataset analysed by country, technology and actor groups in numbers where N= 738.

Industry42%

Intermediary12%

University 15%

Funding7%

Regulatory3%

Political 8%

CSO13%

Fig 3 : SMART-Map Stakeholder Breakdown

0,0

10,0

20,0

30,0

40,0

50,0

60,0

UK HU DK SP GE IT

Fig 4 : Stakeholder Groups per Country (numbers). N= 738

Industry

Intermediary

University

Funding

Regulatory

Political

CSO


19


Stage 2 – the Baseline ‘Before’ survey Recruiting strategy for survey participants

The Baseline survey was ‘wrapped’ around the Industrial Dialogues (IDs). The first of the two Baseline surveys, ie Baseline ‘Before’ took place on a ‘rolling’ schedule, one to two weeks before each ID took place. The six IDs took place at approximately monthly intervals, from January 2017 to June 2017. The survey comprised telephone or skype interviews with a selection of the ID participants (those who responded to our requests to participate in an interview). Interviews were conducted in the respondents’ mother-tongue in Italy and Spain ie Italian for the Milan ID; Spanish for the Valencia ID. Responses were translated into English by the interviewer. The other four IDs were conducted in English.

The same applied to Stage 2: Baseline ‘After’ interviews.

Each ID lasted 1.5days, to the schedule detailed below:

Table 3 – Schedule of the SMART-Map Industrial Dialogues

City Technology Focus Dates

Aarhus (DK) Precision Medicine 16-17 January 2017

Munich (GE) 3D Printing in Biomedical field

6-7 Feb 2017

Manchester (UK) Synthetic Biology 9-10 March 2017

Valencia (SP) Precision Medicine 5-6 April 2017

Milan (IT) 3D Printing in Biomedical field

10-11 May 2017

Budapest (HU) Synthetic Biology 12-13 June2017

In fact, this was a different recruitment methodology to that which was originally conceived. Our original plan had been to undertake interviews following a quota sampling procedure from the ‘stakeholder mapping’ contact list produced at Stage 1 above. However the rate of

































20


refusal to participate in an interview was very high indeed. In addition, we felt the reasons given for refusal were not reliable (lack of interest, lack of time, or the view that the project was not relevant to them). In addition, as the Hungarian case exemplified, a number of potential industry respondents reported that their organisation was a subsidiary of an HQ located in a different country. Decision making was therefore not done locally and for this reason respondents felt it was not helpful or appropriate for them to participate. The high refusal rates were particularly true of industry respondents, and was our first indication that we were struggling with the concept ‘Responsible Innovation Ecosystems’ since only a small minority of those approached from the datasets were willing to participate in an interview; let alone commit to a 1.5 day workshop. Approximately, the rate of attrition at this stage was from around 100 contacts listed on the dataset, to around 25 acceptancse per ID, thus 1 in 4 acceptance rates.

We therefore changed our strategy, working closely with colleagues in the country which was organising each ID, to ‘piggy-back’ the Baseline ‘Before’ interview into the ID recruitment process, by interviewing those who had already agreed to participate in the ID. Therefore, we were interviewing those already positively inclined to engage with the idea of Responsible Innovation, by agreeing to participate in the ID, and therefore happy to participate in the pre-ID interview. In some cases, a different strategy was enlisted where respondents who declined to participate in the ID or were not available on the day the ID was scheduled, agreed to participate in a short skype/telephone survey instead. Thus, the consortium member organising the ID; and the consortium member undertaking the Baseline interview worked very closely together. This included (especially for the opening Aarhus ID) the consortium member organising the ID briefing the Baseline researcher in some depth, giving reasons why contacts approached to participate in the ID had agreed to do so, or had declined to do so. We already then, began to see evidence at this early stage, that we were struggling with the concept ‘Responsible Innovation (eco)systems’ and instead were witnessing something more akin to a small minority willing and interested to engage with us. Hence, we coined the term ‘Coalitions of the Willing’ to better describe the phenomenon we were witnessing. We felt this term to be a more accurate empirically-informed label than ‘Responsible Innovation (Eco) systems’, an idea we had proposed in D7.1 (See section 4 and 5 below and SMART-Map Deliverable D7.1).

































21


The survey instruments The questionnaire used for the Baseline ‘Before’ and Baseline ‘After’ interviews, which combines Stages 2 and 3 of the evaluation, is presented as Appendix 1 of this report.

Sample: Baseline ‘Before’ For the Stage 2 Baseline ‘Before’ survey, a total of 109 interviews were completed across the 6 countries and 3 technologies, and within multiple actor-groups. The following table provides the sample profile, analysed by actor groups (Table 4) and technology domain (Table 5). These two categories proved the most interesting for indicating differences in views according to actor category, and technology focus, within the sample. Therefore, it is these two categories that we focus on in presenting the survey findings in this report.

Table 4 – Actor categories in the sample Industry University

(researcher) CSO Other:

(Intermediary =11; Funding Body =3; regulatory=3; political =2)

Total

32 34 21 29 109

Table 5 – Technology

Technology

Total Medical 3D printing Precision Medicine Synbiotech

35 35 39 109

































22


The analysis of findings reported in the following sections, follow the questionnaire sections chronologically.

Questionnaire Section 1 - The interviewee and the organisational context This section collected and/or validated, basic data about the interviewee’s organisation (size and role of the organisation, job title, role, and level of experience/number of years the interviewee has been working within the organisation).

Questionnaire Section 2 – ‘De-facto responsibility in research and innovation’

.

The tables of responses are the result of multi-coding open-ended replies. The approach was therefore inductive, ‘letting the data speak’ to us, rather than applying pre-selecting variables.

Table 6 – De-facto responsible research (by actor group)

Stakeholder Group

Personal View in

Research CSO

funding body

industry intermediary political regulatory university Total

Compliance with

regulations & standards

0 0 3 0 0 1 2 6

Education 0 0 0 1 0 0 2 3

Questions

Q3.1 Talking about different interpretations and practices of ‘responsibility’ in research in different situations and contexts, what does responsibility in ‘research’ mean to you? Can you give examples to help us understand how you ‘apply’ or ‘do responsible research in your daily work/role?

Q4.1 Talking about different interpretations and practices of ‘responsibility’ in innovation in different situations and contexts, what does responsibility in ‘innovation’ mean to you? Can you give examples to help us understand how you ‘apply’or ‘do’ responsible innovation in your daily work/role





































23


Enhance public

awareness-engagement

6 0 7 1 1 0 6 21

Follow the best

practice in research

8 1 18 4 2 1 22 56

Meet societal needs

2 0 4 3 0 2 3 14

Open access-IPR protection

1 0 3 1 0 0 5 10

Risk assessment

& anticipation

4 1 2 4 0 2 8 21

Shareholders & ethical concerns

3 2 3 0 1 0 4 13

Sustainability &

environment 0 0 3 0 0 0 2 5

Thinking of endusers' benefits

2 0 5 5 0 0 1 13

Total 26 4 48 19 4 6 55 162

Table 7– De-facto responsible research – by technology.

Technology

Medical 3D printing

Precision Medicine

Synbiotech Total

Personal View in Research

Compliance with regulations & standards

1 2 3 6

Education 1 1 1 3

































24


Enhance public awareness-engagement

6 4 11 21

Follow the best practice in research 23 14 19 56

Meet societal needs 1 6 7 14

Open access-IPR protection 4 4 2 10

Risk assessment & anticipation 4 5 12 21

Shareholders & ethical concerns 4 5 4 13

Sustainability & environment 0 0 5 5

Thinking of endusers' benefits 8 5 0 13

Total

52 46 64 162

Table 8 – De-facto responsible innovation (by actor group)

Stakeholder Group

CSO funding

body industry inter-

mediary political regula-

tory univer-sity

Total

Personal View to Innovation

Compliance with regulations & standards

3 0 4 0 0 1 3 11

Education 0 0 0 1 0 0 0 1

Enhance public awareness-engagement

3 0 5 1 0 0 3 12

Follow the best practice in research

6 1 14 4 1 0 14 40

































25


Meet societal needs

3 1 4 2 0 0 4 14

Open access-IPR protection

1 0 2 0 0 0 2 5

Risk assessment & anticipation

0 0 1 0 0 0 0 1

Shareholders & ethical concerns

1 1 1 0 2 2 1 8

Sustainability & environment

1 0 3 0 0 1 1 6

Thinking of end-users' benefits

4 0 5 3 0 0 7 19

22 3 39 11 3 4 35 117

Table 9 – De-facto responsible innovation (by technology)

Technology

Medical 3D printing

Precision Medicine

Synbiotech Total

Personal View to Innovation

Compliance with regulations & standards 1 4 6 11

Education 0 1 0 1

Enhance public awareness-engagement 6 1 5 12

Follow the best practice in research 14 13 13 40

Meet societal needs 4 6 4 14

Open access-IPR protection 2 3 0 5

Risk assessment & anticipation 0 0 1 1

Shareholders & ethical concerns 0 3 5 8

Sustainability & environment 2 1 3 6

Thinking of end-user’s benefits 10 7 2 19

39 39 39 117

































26


COMMENTARY

De-facto responsibility in research and innovation

i) The WP7 team were struck by the degree of stability in the responses, across actor groups and technology. These findings suggest a higher level of convergence in de-facto rri than was expected.

ii) This convergence is apparent whether we ask about the respondents’ own individual perspective, or that of the organisation for whom the respondent works. This convergence was also surprising to the research team. The tables of findings on organisation perspective have not therefore been included in the presentation of findings, since they mirror the individuals’ view.

iii) However there were some interesting differences between de-facto understandings and practicing of responsibility in research; compared to de-facto responsibility in innovation

iv) De-facto responsibility in research. The top de-facto descriptors, in order of mentions, regardless of actor group or technology were: a. 1) Following best practice research b. 2) Enhancing public awareness/engagement c. = 2) Risk assessment and anticipation d. Meeting societal needs

v) We do see some concentration on the descriptor of ‘following best practice in research’ as a primary consideration shared across industry and university respondents, where other actors give less primacy to this descriptor.

vi) We do see some signs of difference by technology area, with 3D printing highlighting the dimension ‘Thinking of end-users benefits’ than the other two technologies (possibly becasue this technological field operates ‘closer’ to end-users; and Synthetic biology highlighting the dimension ‘Sustainability and Environment’ than the other two technologies, possibly because this technological field is broader in scope than the other two technologies, spanning environmental as well as medical applications.

vii) De-facto responsibility in innovation. The top de-facto descriptors, in order of mentions, regardless of actor group or technology were: a) 1) Following best practice research b) 2) Thinking of end-users benefits c) 3) Meeting societal needs d) 4) Enhancing public awareness /engagement

viii) We see greater attention to the descriptor ‘Compliance with regulations and standards in the context of responsible innovation, than responsible resaerch, giving a first signal that responsible innovation is asscociated in respondents minds with regulations andstandards. Likewise we see a higher ranking of the descriptor ‘thinking of end-users benefits ‘ and ‘meeting societal needs’ under de-facto responsible innovation, than de-facto responsible research, suggesting that from the respndents perspective, ‘responsible innovation’ is aprocess which is closer to the end-user/society than is ‘responsible research’.

ix) Finally, there is some indication that the descriptor ‘thinking of end-users benefits’ resonates more with 3D printing in Biomed and Precision Medcine respndents than in Synthetic Biology, suggesting that 3D printing and Precision Medicine are considered closer to end-users/applications than is Synthetic Biology.





































27


Responsible Research and Innovation (RRI) – from the European Commission’s perspective.

Questions

5.1 Are you aware of/Have you heard of the term Responsible Research and Innovation?

6.1 The European Commission has a particular interptetation of RRI, are you aware of what it is?

7.1 The European Commission’s defintion of RRI is (read out)..

“Responsible Resaerch and Innovation is an approach that anticipates and assesses potential implications and societal expectations with regard to research and innovation, with the aim to foster the design of inclusive and sustainable research and innovation”

Does this ring any bells for you?’

8.1 5 keys (unprompted). The European Commission has a particular interpretation of the content of RRI as comprising 5 Dimensions. Are you aware or can you recall what they are?

i) Public engagement

ii) Gender

iii) Ethics

iv) Science Education

v) Open Access

8.2 5 keys (prompted). The European Commission has a particular interpretation of the content of RRI, as comprising 5 Dimensions.... (READ OUT 5 KEYS, RECORD WHICH ONES THE RESPONDENT HAS HEARD IF OR MENTIONS)

i) Public engagement

ii) Gender

iii) Ethics

iv) Science Education

v) Open Access





































28


Table 10 – Awareness and recall of RRI

Technology

3D printing in Biomed

Precision Medicine

Synbiotech Total

RRI

Dialogue and communication 0 0 1 1

Good practice 0 0 2 2

Grant writing 4 1 5 10

Long-term impact 1 0 2 3

Measure 0 1 1 2

Project collaboration 4 4 0 8

Public consciousness 0 0 1 1

public engagement 0 0 1 1

Sustainability 0 0 1 1

Workshop 1 1 3 5

10 7 17 34

Table 11 – Awareness and Recall of RRI - by actor category

Stakeholder Group

CSO funding



tory university Total

Dialogue and communication

0 0 1 0 0 0 0 1

Good practice 0 0 1 0 0 0 1 2 Grant writing 0 1 3 1 0 0 5 10 Long-term impact

2 0 0 0 0 0 1 3

Measure 0 0 0 0 0 0 2 2 Project collaboration

2 0 4 2 0 0 0 8

































29


Public consciousness

1 0 0 0 0 0 0 1

public engagement

1 0 0 0 0 0 0 1

Sustainability 1 0 0 0 0 0 0 1 Workshop 0 0 1 0 0 0 4 5

7 1 10 3 0 0 13 34

Table 12 – Awareness and Recall of RRI – by technology

Stakeholder Group

CSO funding body

Industry Intermediary political regulatory University Total

Dialogue and communication

0 0 1 0 0 0 0 1

Good practice 0 0 1 0 0 0 1 2 Grant writing 0 1 3 1 0 0 5 10 Long-term impact

2 0 0 0 0 0 1 3

Measure 0 0 0 0 0 0 2 2 Project collaboration

2 0 4 2 0 0 0 8

Public consciousness

1 0 0 0 0 0 0 1

public engagement

1 0 0 0 0 0 0 1

Sustainability 1 0 0 0 0 0 0 1 Workshop 0 0 1 0 0 0 4 5

7 1 10 3 0 0 13 34

































30


Table 13 – Awareness and recall of RRI by stakeholder group.

Stakeholder Group

Total

CSO funding body

industry inter-mediary

political regula-tory

university

14 3 21 9 2 3 25 77

Dialogue creation

0 0 1 0 0 0 0 1

Ethics 0 0 0 0 0 0 2 2 Open access 1 0 5 1 0 0 4 11 Public consciousness

2 0 1 0 0 0 0 3

Public engagement

1 0 0 0 0 0 3 4

Science education

3 0 6 1 0 0 1 11

21 3 34 11 2 3 35 109

Table 14 - Awareness and recall of RRI by Technology.

Technology

Total Medical 3D printing Precision Medicine Synbiotech

0 8 3 11

Clinical trials 0 1 0 1

Ethics 0 0 1 1

Gender activities 0 0 1 1

Open access 12 9 8 29

Public consciousness 0 3 1 4

Public engagement 13 14 21 48

Science education 10 0 4 14 Total

35 35 39 109

































31


Table 15 – Relevance of EC ‘5 keys’ of RRI by Technology

Technology

Total

Medical 3D printing Precision Medicine Synbiotech

3 7 13 23

All relevant 10 17 9 36

Ethics 2 1 1 4

Gender 7 4 2 13

Open access 6 3 2 11


Science education 3 1 0 4

Sustainability 0 1 0 1

Total

35 35 39 109

COMMENTARY

Responsible Research and Innovation (RRI) – from the European Commission’s perspective

i) Only 34 (out of 109) = 31% of respondents were aware of RRI . Of those who were, according to the Baseline ‘Before’ survey, the likely locus of encountering the term was through grant writing and project collaboration. The UK/Synbio respondents had the highest level of recall of RRI (17 out of 34 interviewees). This is likely due to Responsible Innovation being incorporated as a mandatory question and criteria in the UK Research Council calls on Synbio, eg through the UK Centres of Excellence on synthetic biology. This hunch is bourne-out across both the technology-level analysis (where awareness and recall are highest in synbio) whilst different actors attribute their awareness of RRI to mechanisms involving project collaboration.

ii) Whilst three-quarters of interviewees claimed to be aware of RRI, their knowledge of what factors this constitutes was very weak, with 77/109 respondents claiming awareness or knowledge of RRI but not able to expand on this, eg by recollecting the five keys.

iii) Once prompted on the 5 keys, the most likely dimensions which resonated with the interviewees was a) public engagement and b) Open Access

iv) Interestingly, the most frequent response to the prompted question on what RRI means, according to the Commission, and therefore what the sub-theme ‘keys’ that comprise it, the most common response was that all the ‘5 keys’ dimensions are relevant .





































32


Benefits and challenges of rri/RRI

Table 16 – Benefits for individuals

Stakeholder Group

CSO funding


mediary political regulatory university Total

Benefits for Individuals

Collaboration tool

2 0 6 3 1 1 3 16

Cross-divisional

1 0 1 0 0 0 1 3

Data Security 0 0 1 0 0 0 0 1 Formation of codes of practice

7 2 5 2 0 1 15 32

Market acceptance

0 0 10 2 1 0 1 14

Questions

Section 9 – Perceived benefits in deeper reflection and activity around responsibility in research and innovation contexts and situations (however understood, ir both de-facto and EC interpretations

9.1 What are the perceived benefits to yourself/your professional field in developing any of the above ideas or activities?

i) Which activit(ies) does the respndent focus on?

ii) What benefits are aniticipated for the individual/professional development?

9.2 Can you see any benefit for your organisation?

9.3 Can you see any benefits to particular groups in society or at society at large?

Section 10 – Perceived Challenges

10.1 Can you see any challenges to yourself/your professional field?

10.2 Can you see any challenges to your organisation?

10.3 Can you see any challenges to particular groups in society or society at large?





































33


Measure social responsibility

3 0 3 3 0 0 2 11

Personal satisfaction

1 1 4 0 0 0 1 7

Public engagement

6 0 3 2 0 0 12 23

20 3 33 12 2 2 35 107

Table 17 – Benefits for Society – by technology area

Medical 3D printing

Precision Medicine

Synbiotech Total

Benefits for Society

Collaboration tool 1 3 0 4

Formation of codes of practice

3 5 5 13

Grant writing 2 0 1 3

Market acceptance 2 0 2 4

Measure social responsibility 4 0 4 8

Personal satisfaction 0 1 0 1


35 35 39 109

Table 18 – Challenges for individuals – by Technology

Technology

Medical 3D printing

Precision Medicine Synbiotech Total

Challenges for Individuals

Acting agent 2 2 1 5

Fit into practice 4 1 4 9

Lack of enforcement 2 7 3 12

Lack of knowledge 14 6 15 35

Lack of motivation 6 4 1 11

































34


Lack of resources 7 7 11 25

Lack of shared understanding 0 2 2 4

35 29 37 101

Table 19 – Challenges for Society – by Technology

Technology

Medical 3D printing

Precision Medicine

Synbiotech Total

Challenges for Society

Acting agent 2 0 0 2

Fit into practice 3 1 0 4

Lack of enforcement 2 1 1 4

Lack of knowledge 4 1 6 11

Lack of motivation 1 3 0 4

Lack of resources 1 4 2 7

Lack of shared understanding 12 10 16 38

25 20 25 70

Table 20 – Challenges for Society – By Actor Group

Stakeholder Group

CSO funding



tory University Total

Challenges for Society

Acting agent 0 0 1 1 0 0 0 2

Fit into practice 3 0 0 0 0 0 1 4

Lack of enforcement

1 0 2 0 0 0 1 4

Lack of knowledge

1 0 2 0 0 0 8 11

Lack of motivation

1 0 2 0 0 0 1 4

































35


Lack of resources

1 1 3 0 0 0 2 7

Lack of shared understanding

5 1 13 5 0 1 13 38

12 2 23 6 0 1 26 70

COMMENTARY

Benefits and Challenges

i) Once again, the most significant finding is the level of convergence in the results, across all actor groups and 3 technologies. Some interesting divergence ‘at the edges’ would require larger samples in order to tease out the significance of these differences. ii) Once again, differences between individual/professional perspective of respndents; and respondents’ perception of the view of their own organisation, were sufficiently synonomous to leave the ‘organisational’ level out of the presentation of findings. iii) In terms of perceived benefits to the individual respondent and their professional development , the top four dimensions, ranked by number of mentions of the dimension were: a. Formation of Codes of Practice b. Public engagement c. As the basis for collaboration tools d. Market acceptance iv) Similar to the findings on Regulations and Standards, the formation of Codes of Practice provide for the hirerchical specification (vis a vis ‘competing’ professions) of high standards of ethical practice. Codes of practice are typically used by professional bodies to denote their superior status and reputation as an institutional device, particularly when a group of expert labour is under ethical pressure or crisis, or real or perceive contrversy . There is therefore some synergy in the idea of marking professional practice through thedesign and adoption of Codes of Practice, and the desire to develop Standards regimes at the level of industry sectors found elsehwere in the survey v) The signaling of rri/RRI as providing a mechanism and benefit to the individual/profession through processes of engaging publics, echoes findings elsewhere in the survey. vi) The value of using rri/RRI as a mechanism for gather and enrol different actors in order to create multi-actor collaborative groups or ‘coalitions’ echoes findings elsewhere in the survey. vii) It is particularly interesting that the dimension ‘market acceptance’ is dominated by the voice of industry, and echoes findings elswhere in the evaluation that the market logic plays an important role in the motivation of industry to engage with rri/RRI on reputation –enhancing and other crisis avoidance and mitigation grounds, combined with ethical grounds of ‘responsible captilalim and the re-framing of the ‘citizen firm’. viii) There is interesting convergence in perceived ‘benefits’ for individual respondents across all actor groups; and perceived benefits to society, both converfing on Public Engagement and Formation of Codes of Practice. This convergence suggests that structurally, in terms of translating rri/RRI into articulated practice with the greatest level of actor buy-in, the central and pivotal idea of public enagement (embracing ideas of inclusivity and opennes) best connects all actors , motivations and perceived benefits of rri/RRI. ix) Perceived Challenges. Again a high level of convergence characterises the survey findings. The primary challenges highlighted are:

For individual respondents: a. 1) Lack of Knowledge (as bourne out by the low awarness of RRI finding) and b. 2) Lack of resources

For society, depicted through the eyes of multiple actor groups:

c. 3) Lack of shared understanding (tops the view on challenges for society by both analysisby actor groups where both Industry and University respondents highlight this challenge, but, interestingly it has less salience as a challenge according to Civil Society Organisations).





































36


Stage 3 – The Baseline ‘After’ Survey

The questions used for the call-back interviews form the final questions of the Baseline ‘Before’ questionnaire. This enabled the researchers to distinguish and attach the ‘call-back’ interviews to the whether the respondent had also been part of the ‘Basline ‘before’’ survey and attach a common Identification Number, to those who participated in both surveys.

A much smaller group – 40 respondents - participated in the Baseline ‘After’ survey.

The responses to these questions, analysed through Likert-scale resonses to a series of statements, are depicted in a series of pie-charts below:

64%

27%

5%2% 2%

4.1 Compared to a few weeks ago, I have found myself thinking more about

‘responsibility’ and how it applies to the technological development

Strongly Agree

Agree

Neither

Disagree

Strongly Disagree

































37


43%

49%

4% 2% 2%

4.2 Compared to a few weeks ago, I have found myself thinking more about

‘responsibility’ and how it applies to research and innovation related to my own practice

and work

Strongly Agree

Agree

Neither

Disagree

Strongly Disagree

61%

28%

9%

2% 0%

4.3 Compared to a few weeks ago, I have found myself thinking more about ‘responsibility’ and how it applies to the development of the (precision medicine) industry and

the importance of mobilising and including wider constituencies of societal actors i

Strongly Agree

Agree

Neither

Disagree

Strongly Disagree

































38


40%

45%

11%

2% 2%

4.4 ‘Compared to a few weeks ago, I feel I have learnt a lot about The European Commission’s concept RRI ‘

Strongly Agree

Agree

Neither

Disagree

Strongly Disagree

43%

40%

13%

2% 2%

4.5 ‘Compared to a few weeks ago, I feel positive about the European Commission’s

concept about RRI’

Strongly Agree

Agree

Neither

Disagree

Strongly Disagree

































39


53%38%

7%

0% 2%

4.6 Compared to a few weeks ago, I feel I am more likely to continue thinking about

how responsibility in research and innovation translates to my own work and

practice.

Strongly Agree

Agree

Neither

Disagree

Strongly Disagree

51%45%

2% 2% 0%

4.7 Compared to a few weeks ago, I feel I am more likely to continue to think about,

and potentially apply/introduce the aspects that we discussed at the workshop

in my own organisation.

Strongly Agree

Agree

Neither

Disagree

Strongly Disagree

































40


Stage 4 – Qualitative interviews with ‘pilot’ participants before/after the ‘Tool’ piloting phase of SMART-Maps.

This section presents the before/after evaluation of the piloting experience; using open-ended qualitative telephone interviews. The three ‘RRI Tools’ pilots were the result of a selection process which filtered the six ‘prototype’ tools that had been produced by the two IDs per technology field. Or, more accurately ‘tool-boxes’, since each ‘tool ’ contained multiple tool ‘elements’. The three selected tools, one per technology domain, were each piloted over a parallel five-month period (October 2017—April 2018). The tool selection process comprised a set of standardised filtering criteria/questions which guided the tool selection decision, which was undertaken and endorsed by those participating in each pilot. (See Milestone reports on the pilot experiences).

The pilots were:

Technology domain Location of the pilot The piloted ‘tool’

Precision medicine

Valencia/Spain

End-user Advisory Panel.

A series of collaborative group workshops were held with patient groups and physicians, led by the precision medicine company, IMEGEN. The workshops had the effect of improving IMEGEN’s knowledge and insights on the needs and perspectives of users and physicians in order to assist the company to produce products and services better sensitised to patients and physician’s needs.

Synthetic Biology

Manchester/UK

Repository of Learning Cases,

Three illustrative reflexive self-authored Learning Cases were produced, guided by a common Template. They were written by senior industry representatives, focussed on their respective (3) diverse industrial organisations: Cambridge Consultants; Dstl; and Innovation and Sparkling

































41


Science Ltd. The pilot was informed and guided by the deliberations of multi-actor, policy-oriented Task Group in the UK, convened for the purpose of connecting the ‘micro’ level organisations, with the meso-level, policy-bridging Task Group. The cases were written to illustrate the bottom-up reflections and experiences of the 3 companies, embedded in their own particular industry and organisational context. The aim would be to lodge the short cases in a Repository of Learning Cases so that others may learn from them, what constitutes rri/RRI in practice. They were also intended to help policy/industry pioneers learn from these cases what qualities constitutes rri/RRI, as a first ‘learning’ step towards instituting an RRI ‘Standard’, ‘Mark’ of Excellence or RRI Award, or Accreditation scheme.

3D printing in Biomed

Milan/Italy

RRI Training and Multi-Stakeholder Workshop. This pilot was implemented through the participative engagement of the EU H2020 network, SYMBIONICA. The purpose was to raise awareness and knowledge about RRI on the part of the SYMBIONICA network, as a first step towards instituting an RRI ‘Standard’, ‘Mark’ of Excellence or RRI Award, or Accreditation.

The methodology comprised open-ended qualitative telephone interviews, conducted by administering the two interview templates (one before/one after) shown at Appendix II. None of the interviews was conducted by a researcher immediately attached to a particular pilot, so as to try not to introduce biases arising from proximity of the interviewers, to a particular pilot or personnel involved in the pilot. Thus, the UK interviews were conducted by a Hungarian consortium member; whereas the Italian and Spanish pilots were conducted by a native-Spanish speaker attached to the Manchester team. Because of the intimate ‘insider’ facilitating role of the SMART-Map consortium members in each pilot, the leading consortium members’ views were canvassed as part of the ‘Before- pilots’ interviews.

































42


31 interviews were conducted in total, distributed as follows:

Total Precision Medicine Synthetic Biology 3D printing in the

Biomed field

Before-pilots

5 (of which three from the SMART-map

Consortium)

7 (of which three from the SMART-Map

Consortium)

8 (of which three from the SMART-Map

Consortium)

After-pilots

3 (non-consortium members who were all directly involved in the

pilot)

5 (3 UK non-consortium members who were all directly involved in the pilot, plus two Hungarian

interviewees who had attended one or both

of the Synbio IDs)

3 (non-consortium members who were all directly involved in the

pilot)

Number of Interviews

31 8 12 11

































43


3. Findings Synthetic Biology/Manchester,UK Four points can be highlighted from the ‘Before-pilot’ interviews of the Synbio/UK pilot:

1) Intermediation and Multiple Roles. Interviewees who were particularly committed to the experimental RRI tool-piloting process were also articulate in expressing what, for them, were the values, motivations and ‘worth’ of responsibility in research and innovation. They also entered the pilot phase with an optimistic and hopeful view of the anticipated value of the outcomes. These optimistic, values-driven individuals also held multiple intermediating or ‘bridging’ (boundary-spanning) roles in their everyday work and practice. They were all able to speak from the perspective of these different roles, and hence from the ‘two-way relational perspective of their two or more ‘hats’’. For example, the first author of a ‘learning case study’ spoke from the perspective of a company whose role was to span the boundaries of different technology fields or sectors hence operated by ‘translating’ and applying responsible research and innovation principles and perspectives from one sector to another. The second spoke from the perspective of a micro-business but also as a significant policy-influencer in UK Synbio governance in the UK . The third from the perspective of a large anchor Research and Technology Organisation (RTO) , who spoe through experience of working within a large (3,000 employees) organisation, but also from experience of working closely with the research and technology supply base, particularly small and medium enterprises (SMEs) suppliers and the creation of open tendering terms that enabled SMEs to have equal and fair access to the RTO.

2) Practical and pragmatic hopes for the pilots, not abstract or theoretical (academic) exercise. Common across the UK/Synbio ( and in common with the other pilots) was the hope that the outcomes of the pilot would be practically implementable. There was a common resistance from the piloting companies to abstract, theoretical or overly ‘academic’ exercises and a preference to think about RRI through the distinction of theoretical versus practical way of approaching the concept. There was also, from the

































44


commercial companies, a common tendency to view the motivation for RRI to be about engaging societal actors for moral and values-related reasons, but these goals were not the final teleo-affective (means-ends) motivation. The ‘end goal’ was to open and assist the process of market opening, market stabilisation, and commercial and economic resilience:

“bringing products to the market, that should be regarded mainly as a practical approach that affects cultural change, not as a conceptual tool for the academic playground.”

3) Prior-experience with different Responsible Innovation/RRI policy frameworks. The UK stood-out against the other pilots for the level of maturity of RRI. This can be understood as having prior-experience of engaging and reflecting on the topic of what it means to practice or ‘do’ research and innovation ‘responsibly’ (ie de-facto rri); and equally what it means to engage with different Responsible Innovation/RRI ‘family’ of policy frameworks and concepts. It relates to a context of prior experience and awareness of RRI from working on projects and collaborations in response to research calls issued by the UK Research Councils (EPSRC, BBSRC) where evidence of incorporating RI/RRI as a stated funding condition and criterion, in a number of emergent technology areas (in particular synbio) exists. It is also motivated by a curiosity to contrast the UK variant of RRI with approaches adopted by the European Commission,. An important pre-pilots motivation and hope, therefore was to have the opportunity to compare and contrast the UK experience with that of the pilots experiences of the other SMART-Maps pilots/technology fields. Learning, in the sense of sharing experiences between and across different UK stakeholders and technology fields; and learning and exchanging experiences with different EU counterparts, was a notable motivation for the UK/Synbio active pilot participants. Albeit the UK actors (shared with the leading actors in the other two pilots) did not see RRI as ‘new’ but rather an extension or systematic consolidation, of what they were already doing under the governance frameworks which guided daily practice.

































45


“the need for responsibility, and the need for doing things in a well-judged, safely considered way is not new. So I see it just as an extension of what we might do anyway”

4) Longitudinal analysis. Nearly all of the UK/synbio pre-pilot interviewees mentioned “learning / learning as a process” as a significant element, when talking about motivations, hopes, and expectations. Interviewees have a high motivation for learning but also they are quite realistic and pragmatic regarding their expectations on learning outcomes of the project.

“I think from a personal perspective I’m always keen to make sure that I’m personally aware and informed about progress (on RRI) […] being in a position to take back any learnings to the leadership council. […] I’m hoping that we will see positive feedback from the pilots, but at very least we will learn what worked well and what hasn’t, so we can build on that.” .

Member of the policy-influencing Synthetic Biology Leadership Council (SBLC) of the UK

“Certainly to learn, find out what other people think, and struggle with, you know some of the ideas that people have to do better, my honest expectation is what comes out is managing the process and expectations of the process, ... maybe something comes out that needs more work, but certainly a step forward, and an outcome that I start to use, but there will be something that needs to be added, revised, or improved, to a point of being really useful, to take it to that point to become really useful and valuable.”

Piloting company

































46


Analysis of the after-pilot interviews in UK/Synbio generates the following four key findings:

1) Were expectations met? : Coming up with tangible and useful outputs. A common expectation and aspiration across the UK pilot was that concrete usable and pragmatic outputs and outcomes would result from the pilots. There was a high degree of consensus that this hope and expectation was successfully achieved.

“We did come up with a couple of good ideas of which one was about standards for responsible innovation that you can measure against. And the other one was the case studies repository. Information that people can use to get inspiration, ideas, that show how other people handled responsible innovation . Yes, i think we came up with some tangible and useful things”

Piloting company

2) Enjoyable Challenges. Common across the UK pilot was an attitude which embraced challenges and re-framed them as enjoyable opportunities. This is true of objectives like collective learning, or developing social networks by engaging with other participants along the length of the 30month SMART-map project .

“It is always fun to meet people, talk with them having new ideas, have you thought about it like that or like this.I would not say these were challenges. I enjoyed it.”

Piloting company

































47


3) Benefits: Institutional change and transformation, brings along with it wider positive benefits for society. A shared perspective across the UK piloting companies, was an ability to look beyond immediate and personal needs, to wider and longer term benefits to wider constituencies of ‘society’ as this comment below from a piloting organisation shows. The conduit or instrument for achieving this wider institutional change was again seen to be the development of an RRI ‘standard’.

“I think there are benefits there because the ideas around institutional change I think some of the tools are a mechansim to help drive those changes the recognition that things need to be done in this way and here is some examples of what you can do and standrds you might work towards. the core of some ideas that could definitly drive RRI to its wider societal context”

Piloting company

4) Scope, scale-up and possibilities for longer term impact and legacy? Too soon to tell, but a step in the right direction along a longer time-frame and journey.

“We pulled some interesting information together, but I’m not sure that the value of it has been assessed yet. In the case studies what value they bring in terms of other people learning and understanding, is still to be seen. But it certainly relates to the first point of anticipate and assesses potential implications and societal expectations with regard to research and innovation.With our case studies we aim to make them think what do we do with these new technologies, how do we approach them how do we gain a broad understanding of the impact, on how they understand it, whteher they will accept it, or not, so I think these were helpful in assessing expectations and I think the case studies would give people food for thought and help them do change, but the proof of the pudding is in the eating as they say, I think may be we’ll get something out from that, and have an impact, but it is truly

































48


later when we can evaluate whetehr it has really contributed to those things”

Piloting Company

“One of the things we have done in Manchester, I think the benchmarking, setting of standards could be, used by companies to evalute their activities with regard to RRI and put a gold star against them, and that i think could be interesting idea . One that could be aken by on a policy level, a thing that we can do to show people RRI is happening here, this organization is doing it and it doing it well, so it is much more open, so I think the idea of standards could be such. And also long term benefits could be, as a gold star is strong one, demonstarting responsibility.”

Piloting Company

“I strongly recommend that this work is followed up and continued rather than being a tickboxes excercise, as it really really should be used as the foundation for future develoments and activities,”

Piloting Company

Precision Medicine /Valencia, Spain

Before- Pilots interview findings

1) Relational Positionality of the Interviewee :Differences in relation to role & function perspectives Responses from interviewees seem to differ according to their role in the project - eg the company representatives who were also members of the SMART-Map consortium developed a deeper knowledge and appreciation of what RRI is, as a

































49


result of being within the consortium and engaged with the project for the entire 30month project length. From this ‘championing’ perspective, the interviewee appreciates more than others who are at greater ‘distance’, both the objectives and aims of the Commission; and the motivations, enthusiasm and concrete actions of the SMART-Map consortium, and was able to bridge the two. In particular the positive view and disposition was enhanced and illuminated as a result of attending a number of the IDs, in different countries. By contrast, company-based respondents who were less directly engaged with the consortium were vaguer on concrete knowledge and aims of the (EC) description of RRI. But they do say the experience of working with SMART-Map made responsibilities that were more sub-conscious before were more reflexively conscious and front-of mind by the end of the project, representing a shift in mind-set, and perspective, about RRI, which through the pilot was translated into concrete practical action.

2) Relational Positionality of the Interviewee : Differences in relation to distance from the EC. A general finding across the before/after pilots surveys, is that those with greater distance from the Commission have lower-level depth, learning, knowledge and positive disposition towards RRI, as it is understood by the Commission, from those who are closer to the Commission . Eg the consortium members showed a particularly high commitment and buy-in to the interpretation of RRI, as defined by the Commission.

3) Before-pilot expectations : to learn; and to translate that learning into better ways of understanding market and user needs, in the design of new products and services.

“To learn something and to show what is in the assumptions, to make difference on the company and the market.”

Consortium member + company, so ‘two hats’ perspective

“Basically that the patients understand better what we do and the physicians feel part of our daily work”

































50


“Having a large concept map from different users and see how they understand precision medicine”

Piloting Company

4) Benefits of RRI : to gain knowledge and insight, better to understand different actor groups needs (physicians and patients), ultimately to form a stable, resilient and trust-based relationship with ‘users’ within the market context.

“for myself it is getting more knowledge about the society needs - The opportunity to get more information to improve our products”

Piloting company

“Obtain information of what really matters to patients and to get to know what do the physicians expect too”

Piloting company

“learning and getting real information from the end users of our product”

Piloting company

‘After’ pilots interviews

The following two points stand out in the ‘after-pilots’ precision medicine /Valencia, Spain experience

1) Much more positive about RRI at the end of the SMART-Map process and pilots than at the outset. The step-change in positivity towards RRI in Spain (and Italy) compared to the more measured response to this question in UK, can be attributed to a bigger

































51


gap (in Spain and Italy) where, at the start, awareness and concrete examples of working with RRI principles were very low to non-existent before engaging with the SMART-Map process. Thence the higher positivity towards RRI reported at the end of the pilot reflects this rapid and deep immersive experience, derived from working collegiatly with the SMART-map consortium and piloting initiatives - over an extended period of time.

2) Benefits and longer term legacy: As a result of engagement with the SMART-Map project, a new relationship and approach to Precision Medicine has been forged and articulated by the pilot company (IMEGEN). The company representatives report that by the end of the pilot, they now sit far from the company’s earlier singular focus on commercial. We can hypothesise that this values-shift expresses itself in taking an RRI-sensitised step towards developing a new business model which places greater emphasis on ascertaining and meeting users (patients and physicians) collectively articulated futures needs and perspectives on societal problems. It produces a change in mind-set and capabilities and capacities, in order to anticipate emerging cares and concerns around precision medicine. Through the pilot exercise, trust, transparency and openness were built into the process, through the participatory dialogic process, and the company representatives envisage this being a permament shift in the way they do business.

3D printing in the Biomed field , Milan, Italy Findings and Analyis of the Pre-pilot interviews: some repeating and therefore stable insights.

1) A clear sense that the pilots are a first stage towards a longer-term objective and process to create a certification scheme for RRI, relevant to 3D printing in Biomed

































52


applications. This is a common aspiration connecting the 3D printing and Synthetic Biology pilots.

The idea is to build up a discussion a path towards certification with RRI elements within it.

I the link with SYMBIONICA team and I also coordinate the implementation of the activities.

Consortium Member

The process will help to give a clear overview of all aspects that need to be taken in account in order to have a responsible certification.

SYMBIONICA member

2) The benefits of longevity and continuity.. The SYMBIONICA network spanned different countries beyond Italy. This point differentiated the 3D technology/pilot because it meant the allagiance to the SMART-Map pilot was less stable than in Spain and UK. Individuals moved in and out of the pilot process, and benefitting less from the continuity, deep learning, and sharing collectively through the people and processes that comprised the SMART-Map project.

As I don't really know what the objective of the SMART-Maps is, I wouldn't be able to tell you.; If I can look at SYMBIONICA project, it made more sense if we could have done it at the beginning, as we could have applied that from the beginning. Now maybe for future creations/ case studies

































53


SYMBIONICA member

The main challenge, I will start with SYMBIONICA, we have different people and backgrounds, the most difficult thing is make them to communicate about something they probably don't know because is not their field. On the previous meetings they didn't understand what it was that we were doing. Therefore, we need awareness and make them communicate.From my side, my big challenge is not to lose the path, when you start thinking all the aspects of the issue you get lost.

SYMBIONICA member

3) Less reference to market relations. The Italy pre-pilot appears to have paid more attention to the cares and concerns of societal groups, and less attention to RRI as a means of opening and protecting new markets and building maket niches . However, although the Italian pilot was more diverse in its reference to markets development as a motive, this market discourse and justification remains present.

“to get the public to know the 3d printing biomedicine. Combining this with an ethical and social goal. That will reflect what is working well in this certification and what we can improve with this.”

SYMBIONICA member

One way would be to discuss with end users, with different society groups when we are going to develop a new product. We could take into account what are they thoughts towards these products, being user friendly, in order to give personalisation

SYMBIONICA member

































54


4) The benefit of learning from good practice examples was again highlighted in the 3D printing pilot

“As far as I see, there will be some guidance and results from this project, best practices you can learn from them. Implement best practices examples”

SYMBIONICA member

The ‘After-pilots interviews revealed the following key points 1) ‘intrinsic’ values and optimism about RRI. As with the other pilots, a small number

of ‘Champian’ participants appeared ‘instrinsically’ positive about RRI values , prinicles and outcomes. One person stood out in the SYMBIONICA network. Even though (she) had not attended any of the SMART-map training workshops, she had a positive and enthusiastic attitude towards RRI. ..She was also, importantly eloquent in claiming that the expectations of the project had been met..

‘her motivation is instrinsic – she is ‘as if’ been in all the meetings. Has participated in other EU projects. Believes in this one – this one will make a change.

SYMBIONICA member

Main motivation – to improve the Symbionica approach – growing as people. And the consortium. (Inner development)….Get rid of the technological-industry problematic

SYMBIONICA member

































55


4. Discussion and Conclusions

The SMART-Map evaluation has provided ample evidence (both summative and formative) to demonstrate that the project has succeeded in achieving its primary objective: to successfully co-design and prototype (6) RRI ‘tools’ from which (3) tools, (one per technology area) was selected and ‘piloted’ over a five month period . The successful execution of the protyping/tool selection process which contributed to the next level of mutual support : being the active piloting and writing-up of the and practical immersion in the processes of learning and engagement within the dialogic process which characterised the good-will evident within which SMART-Map dreams and piloting phase. Thereafter the write-up of each of the 3 pilot processes , culminated in the production of 3 ‘SMART-Map Roadmaps’. Furthermore, SMART-Maps can be differentiated by the desire to achieve impact and genuine practitioner-led scale-up, legacy, and spill-overs into other sectors and , enabling the benefits and investments of SMART-Map not to be lost, but rather to be sustained, producing systematic multiplier approaches to rri/RRI impact eg through the development of an RRI ‘standard’ ‘certification’ or quality ‘Mark’ . So, The SMART-Map project sees itself as, and can be characterised by, the extent to which the primary objective – to create Roadmaps for the Responsible Development of Industrial Technologies in the Health sector has been met. But it goes further. Through the enthusiasm of a small number of SMART-Map ‘champions’ who volunteered and emerged through the ID process, and the length of the hands-on SMART-Map process, participants reported a high level of positive attachment to the project and its outputs which had a lasting impact and extended beyond the immediate life-span of SMART-Map, providing a more lasting affect on the SMART-map particiants, indicating an impactful step-change at the level of individuals. It highlights the role of organising a structured elangated process and journey through which the particiants pass, rather than one-off or ad-hoc events. The pilots also highlighted how the naturally optimistic, futures oriented pilot volunteers appreciated the culture of the project: as giving a space to engage in co-constructive dialogue

































56


involving actors with very different roles and perspectives. The process also provided space and opportunity to give vent to creative flair in the production of visual tools and creative, experimental, processes. The project thus becomes associated with a higher level of creativity, optimism and enjoyment, albeit with very serious concerns at stake. Here, the SMART-Map project shows signs of achieving mutiplied scale-up, impact and legacy (albeit it was widely acknowledge it is too soon to claim this concretely). In a normative sense, the project achieved a step-change which was acknowledged by all the pilot participants,: it enabled a deeper and reflexive learning process for the participants. In the longer term, this adjustment in learning, coupled with a higher level of care and consideration paid to wider societal concerns by encouraging and providing space for critical reflection on the ‘Business as Usual’ model of undertaking research and innovation in a bubble or removed from societal reflection, the shoots of organised action nder rri/RRI begin to look very very different to the Business as Usual model. The project thus shows signs of having a positive impact on the much bigger and longer term objective of transformational institutional and cultural change, through the conduit of volunteer participants consciously engaging and experiencing hands-on , a process of working within rri/RRI frameworks and objectives. However, an interesting finding from the SMART-Map evaluation appears to challege the central proposition we introduced in the first evaluation report of the project (D-WP7.1). In that report we posed the question, ‘Do (Responsible) Innovation (eco)sytems exist?’ We anticipated the answer to be affirmative. But when we look at the data (D-WP7 of SMART-Map) we need to remain sceptical about applying the notion of innovation ecosystems to the phenomena we observed through the SMART-Map evaluation lens. As we have discussed, the number of individuals in the IDs, who were motivated to volunteer, and invest a considerable amount of time, in the SMART-Map IDs and pilot process, singles them out from the mainstream, and signals that they are a minority. There is no evidence that RRI has yet reached a collective level of shared acceptance and systemic interdependency that would warrent the description ‘Responsible Innovation (Eco)systems . Rather, we witnessed a very rapid rate of

































57


attrittion in our attempts to recruit participants to the Industrial Dialogues, and survey respondents. The evaluation studies showed how the number of participants/respondents quickly narrows, to a small base of champions, committed to the idea of changing the mind-set and practice of research and innovation by enlarging a form of values-driven developmental process taking a more open, participatory, multi-actor, and futures oriented form. Only a handful of the original contacts stayed with the process to the end. We can see this ‘chanelling’ of engaged/enthusiastic people, through the IDs and pilot tools development, presents a phenomenon which does not feature the overflowing, inter-connectedness and inter-dependency characteristics that qualifies systems theory. The absence of system inter-dependences, and system spillovers, suggests a phenomenon which can be more acurately described as ‘Coalitions of the Willing’ (see Figure below).

Figure 5: Coalitions of the Willing

































58


Interpreting the SMART-Map evaluation stages ,we come to the conclusion that the conceptualisation of ‘Coalitions of the Willing’ better represents the empirical phenomenon we witness in the SMART-map project today. As the SMART-Map process has experimentally unfolded, the structure we see does not seem to align with the concept of ‘(Responsible) Innovation Ecosystems’. However, we would need to further monitor the SMART-Map pilots in order to dynamically follow the project and progress of the pilots, in order to ascertain whether the ‘Coalitions of the Willing’ is an interim stage or more permanent state. For example, if introducing an (RRI) standards regime, leads to increased take-up of rri/RRI sensitised practices, thus potentially ‘responsible innovation’ (eco)systems’ may also emerge . If accompanied by the deeper institutionalisation of rri/RRI through the design and deployment of rri/RRI common guidelines in order to embed a particular set of rri/RRI norms into organisations and practices, and if these norms are show-cased to the outside world through (shared learning including the device of Learning Case Studies) contributing to an accreditation process, the effect may possibly trigger processes of enrolment and overflowing, thus the further stabilisation and visible adoption by others, of a variety of interpretations of societal value, aligned to principles and practices demonstrated as convergent with rri/RRI across different countries, actor groups and emergent technologies .

However, regardless of these questions referring to the social organisation of rri/RRI collectives today, what has been interesting (indeed surprising, considering the heterogeneity of the SMART-Map participants’ roles, organisational context, technological focus and geography, )is evidence within he SMART-Map interviews of consolidation and stabilisation of the rri/RRI discourse . Evidence from the SMART-Map project points to the existence of a stable short-list of shared ‘top-line’ descriptive elements, together producing a surprisingly common and shared view of what constitutes rri/RRI.

Within the evaluation results, a remarkably stable interpretation of what rr/RRI ‘is about’, within the European context at least, inductively shines through (highlighted in ‘yellow’ ) in the tables of this report in Section 4. The tables show that, in general terms and regardless of technology, country, or actor-group, respondents opinion of what rri/RRI means to them, can be inductively discerned, in remarkably settled and un-contentious terms.

































59


However, within these common threads, there are differences ‘at the edges’, revealed in the second-order sub-themes. It is in these sub-themes where we find actor-group, and technology differences. One differentiating sub-theme in particular, which appears in both the baseline surveys and the qualitative before/after pilots qualitative interviews is the driving motivation of the market-logic amongst the industry particiants, contrasted with wider ‘societal-good’ logic of the civil society organisations. Within the industry discourse, combining the ‘market’ logic as a guiding framing, with the values-based ‘good citizen corporation’ logic where firms wish to differentiate themselves from the crowd by appealing to their own standing and reputation as ‘morally, ethically and responsible citizen-firm’ , this conjunction of qualities provides a converging rationale for developing a market niche premised on higher standards, of responsibility, ethics, integrity and authenticity. This market niche does not overturn the market logic, but rather qualifies it and assists the formation and stabilisation of the niche market through mechanisms of market qualification, including the drive to develop an RRI standard which we have witnessed throughout the SMART-Map project.

































60


5. References

Bloom, B.S., Hastings, J.T., & Madaus, G.F,; Ed (1971) Handbook of Formative and Summative Evaluation of Student Learning, McGraw-Hill Book Co, New York, USA

Brophy, T. Ed (2019) The Oxford Handbook of Assessment, Policy and Practice in Music Education, Oxford University Press, Oxford, UK.

Edler, J., Cunningham, P., Gok, A., Shapira, P., Eds (2016) Handbook of Innovation Policy Impact, Edward Elgar, Cheltenham UK.

Greve, B. Ed (2017) Handbook of Social Policy Evaluation Edward Elgar, Cheltenham, UK

Guyot, W., (1978) ‘Summative and Formative Evaluation’ The Journal of Business Education, Vol 54, No3

Harlen, W., and James, M., (1997) ‘Assessment and Learning : Differences and Relationships between Formative and Summative Assessments ., Assessment in Education : Principles, Policy and Practice Vol4, No 3 pp365-379

Looney, J.W. (2011) ‘Integrating Formative and Summative Assessment : Progress Toward a Seamless System? OECD Education Working Paper No 58, OECD Publishing

Murray. H.G. ( 1980) ‘ A Comprehensive Plan for the Evaluation of Teaching at the University of Queensland ’ Reflections, 4, pp3-11

Randles, S., Hajhashem, M., Lescai, F., Himmelsbach, R., Lindner, R., Demeny, E., Kakuk, P., Simone, A, Alvarez, MC., (2017) ‘ Transnational Co-ordination, Formative

































61


Evaluation, and Participant Tracking Across Six Countries and Three Emergent Technologies: Design and Methodological Underpinnings of the Formative and Summative Evaluation of the EU H2020 SMART-Map project’ Presentation to the 5th Annual Conference of the Governance of Emerging Technologies (GET), May 17-19, Phoenix USA.

Scriven, M., (1967) ‘The Methodology of Evaluation’ AERA Monograph Series on Evaluation , Vol 1, pp39-83.

Stetler, CB., Legro, MW., Wallace, CM., Bowman, C., Guihan, M., Hagedorn, H., Kimmel ,B., Sharp, N., Smith, JL (2006) The Role of Formative Evaluation in the Implementation Research and QUERI experience’ Journal of General Internal Medicine, 21:S1 DOI https://doi.org/10.1007/s11606-006-0267-9

Stilgoe, J., Owen, R., & Macnaghton, P. (2013) Developing a Framework of Responsible Innovation , Research Policy Vo1 42,No9, pp1568-1580

































https://doi.org/10.1007/s11606-006-0267-9

62


6. Appendices 1. Baseline questionnaire pre and post survey

Inte

rvie

w de

tails

Interview No Name of interviewer Institute Main language

Inte

rvie

wee

deta

ils Name of Interviewee

Organization Position Role in innovation ecosystem

1.1 To help us validate the information we have from the web, could you tell us what are your organisation/company’s activities and role with respect to synthetic biology? (For companies, probe position/role in supply chain) (open question but pre-coded)

i) If industry : o Basic science/knowledge/ o Applied technology (ie with product/technology area in mind) o New product development o Sales/commercialisation?

































63


ii) if finance/insurance/industry support services/professional bodies/consultancies o Providing finance (ethical finance…)

iii) Scientists (social and natural) o Natural In which area/discipline o Social In which area/discipline iv) If CSO o Research (selves or commissioning others, knowledge base) o Campaigning/media/political influence/policy analysis/linked to govt or industry In which area? o Education/policy or delivery? In which area? o Public participation/CSO support/access/inclusion In which area?

v) If policy/govt/regulatory/research councils/standards (strategic shapers and influencers of Responsible Innovation System) o Policy making o Funding research and innovation activities o Creating and influencing standards

































64


1.2 Geography/command & control o Multi-national (HQ in one country – branches/offices in other countries) o National (all activities within-nation) o Local (within region) 1.3 Size (employees – total) o Large – 100+ o Medium 20 - 99 o Small 1- 19

2.1 For the individual : What is your position/function/experience in your organization? (open question but pre-coded).

i) position : o Senior Board/CEO o Senior manager/PI scientist/ o Junior

ii) Function o Research (Active – social or natural scientists) o Commercial o Strategy/policy o Directly on ethics/sustaiabiity/responsibility?

iii) Experience o >20years o 5-20 o <5years.

































65


3.1 For the rest of this short interview we would like to talk about different interpretations and practices of ‘responsibility’ in research in different situations and contexts.

i) Thinking about your own experience, views and daily work practices, What does ‘responsibility’ in ‘research’ mean to you? Can you give examples to help us understand how you ‘apply’ or ‘do’ responsible research in your daily work/role? ii) Thinking about your organisation, what does responsibility in ‘research’ mean to the organisation? Can you give examples to help us understand how this is interpreted and acted upon in your organisation? (probe reports, policy, who is responsible for what eg R&D, CSR, Sustainability reporting etc.) iii) And now thinking about your own experience, views and daily work practices, and the organisations that you work with, What do you think ‘responsibility’ in ‘research’ should be interpreted and acted upon by individuals in their professional capacity and/or organisations? Can you give examples? o According to Individuals/professionals o According to organisations

































66


4.1 For the rest of this short interview we would like to talk about different interpretations and practices of ‘responsibility’ in innovation in different situations and contexts.

i) Thinking about your own experience, views and daily work practices, What does ‘responsibility’ in ‘innovation’ mean to you? ? Can you give examples to help us understand how you ‘apply’ or ‘do’ responsible innovation in your daily work/role? ii) Thinking about your organisation, what does responsibility in ‘innovation ’ mean to the organisation. Can you give examples to help us understand how ‘responsible innovation’ is interpreted and acted upon in your organisation (probe reports, policy, who is responsible for what eg R&D, CSR, Sustainability reporting etc.) iii) And now thinking about your own experience, views and daily work practices, and the organisations that you work with, What do you think ‘responsibility’ in ‘innovation’ should be interpreted and acted upon by individuals in their professional capacity and/or organisations? Can you give examples? o According to Individuals/professionals o According to organisations

































67


5.1 Are you aware of/ Have you heard of the term Responsible Research and Innovation (RRI)? o Yes o No If yes …. From where and what do you understand by the term o Where. o What. 6.1 The European Commission has a particular interpretation of RRI, are you aware of what it is (Note: Reassure this is not a ‘test’ Unprompted awareness) o Yes o Unprompted awareness (write down) o No

































68


7.1 The European Commission’s definition of RRI is … (read out). (Prompted awareness) Responsible research and innovation is an approach that anticipates and assesses potential implications and societal expectations with regard to research and innovation, with the aim to foster the design of inclusive and sustainable research and innovation. https://ec.europa.eu/programmes/horizon2020/en/h2020-section/responsible-research-innovation Does it ring any bells for you? Scale o Completely o Partially o A little o No, not at all 8.1. 5 keys. (unprompted) The European Commission has a particular interpretation of the Content of RRI as comprising 5 Dimensions. Are you aware or can you recall what they are: (record which ones the respondent mentions) i) Public engagement ii) Gender iii) Ethics iv) Science Education v) Open Access

































69


8.2. 5 keys prompted. The European Commission’s has a particular interpretation of the Content of RRI as comprising 5 Dimensions … (read out). (Prompted awareness) (record which ones the respondent has heard about or mentions) i) Public engagement ii) Gender iii) Ethics iv) Science Education v) Open Access

8.3. Existing activity under each of the 5 dimensions Can you tell me what if anything you i) personally ii) your organization, already do under each of these dimensions? i) Public engagement ii) Gender iii) Ethics iv) Science Education v) Open Access

































70


8.4. your organization, already does under each of these dimensions?/which part of your organization is responsible for policy and implementation of activity in each area? i) Public engagement ii) Gender iii) Ethics vi) Science Education v) Open Access

9. Perceived benefits to engaging in deeper reflection and activity around responsibility in research and innovation contexts and situations (however understood from above, ie both de-facto and EC interpretations)

9.1. Can you see any benefits to yourself/your professional field in developing any of the above ideas or activities of responsibility? i) Which interpretation does the respondent focus on? ii) What benefits are anticipated for the individual/professional development (eg their values or their career development)?

































71


9.2. Can you see any benefits to your organisation in developing any of the above ideas or activities of responsibility in R or I? i) Which interpretation ii) What benefits are anticipated iii) Who should progress this within the organisation iv) Who should be involved (external to the organisation) eg civil society, schools, standards agencies etc etc etc.. (to see how responsible innovation systems are constructed by our respondents)

9.3. Can you see any benefits to particular groups in society or society at large in developing any of the above ideas or activities of responsibility in R or I (involve users from the outset? Incluiveness/diversity in R and I outputs and outcomes? Address societal problems /challenges with those affected by the problem? Etc)

10. Perceived challenges to engaging in deeper reflection and activity around responsibility in research and innovation contexts and situations (however understood from above)

































72


10.1. Can you see any challenges to yourself/your professional field in developing any of the above ideas or activities of responsibility? i) Which interpretation do they focus on? ii) What challenges are anticipated for the individual/professional development (eg their values or their career development)?

10.2. Can you see any challenges to your organisation in developing any of the above ideas or activities of responsibility in R or I? i) Which interpretation ii) What challenges are anticipated iii) Who should take responsibility/deal this within the organisation vi) Who should be involved in engaging with these challenges (external to the organisation) eg civil society, govts, schools, standards agencies etc etc etc.. (to see how responsible innovation systems are constructed by our respondents)

































73


10.3. Can you see any challenges to particular groups in society or society at large in developing any of the above ideas or activities of responsibility in R or I i) Lack of knowledge about the issue/capabilities ii) Lack resources/cost/capacity to address the issues iii) Lack of transparency (challenges of open innovation, versus IPR) iv) Lack of shared interests/understanding/values/deep contestation 11.1. As part of the SMART-Map project we are organising a series of multi-stakeholder Dialogue workshops. The one in Manchester about synthetic biology will be on XXX and XXX…. At the workshop we will further discuss the above issues with the aim of creating ‘SMART-Map’ protocols/blue prints adapted to the 3 technology areas, countries and to assist volunteer/pilot companies to develop their own ‘SMART-Maps’ implementation plans in Europe. Already for the (Manchester) workshop we have received acceptance from … (name key people/snow balling technique).. .. may we invite you to join us at the Manchester SMART-map Dialogue? Attend/ Y/n If no, reasons…..

































74


11.2. As part of the SMART-Map project we are creating a ‘communication community’ to stay in touch with those who have participated in the Baseline survey, across 6 countries (Denmark, Germany, Hungary, UK, Italy, Spain; and 3 technologies : synthetic biology; 3 D printing in medical fields, and precision medicine; and across multiple kinds of stakeholders from government, industry and science and particularly including civil society organisations). We will be emailing this community occasionally on our findings, through occasional reports and news items. Would you like us to add you to the e-list for this community? Y N If no why not?

11.3. We will be repeating a version of this survey with those who have been involved in the Baseline survey, in our Dialogue workshops and in our SMART-Maps development and piloting works as part of our project evaluation, may we contact you again in about a years time to repeat this quick survey? Y N

































75


1. Ascertain whether the respondent: i)Attended the Dialogue (two days) (Code 1) ii) Attended the workshop (first day) (Code 2) iii)Attended the workshop (second day) (Code 3) v)Didn’t attend (due to sickness etc) (Code 4)

2. Completed the post-workshop evaluation (use our own forms to try and identify in advance: Y (->Go to Q 4) (Code 1) N (-> Go to Q3) (Code 2)

3. Now that you have had a few days to reflect , would you be happy to answer for us briefly four questions that the participants who attended Day 2 completed by way of a feedback sheet at the end of the workshop?

3.1 Thinking of the workshop, for you ‘What worked’? (open response)

3.2 What could be improved? (open response)

3.3 Do you have any Further Ideas for us to consider regarding the workshop? (open response)

































76


3.4 Do you have any further Questions you would like to put to the consortium? (open response)

4. We would now like to read out 8 statements, for each we would like you to respond that you either: i) Agree strongly (code 1) ii) Agree a little (code 2) iii) Neither particularly agree nor disagree (3) iv) Disagree a little (4) v) Disagree strongly (5)

4.1 Compared to a few weeks ago, I have found myself thinking more about ‘responsibility’ and how it applies to the technological development of Precision Medicine.

4.2 Compared to a few weeks ago, I have found myself thinking more about ‘responsibility’ and how it applies to research and innovation related to my own practice and work related Precision Medicine.

4.3 Compared to a few weeks ago, I have found myself thinking more about ‘responsibility’ and how it applies to the development of the precision medicine industry and the importance of mobilising and including wider constituencies of societal actors in its development.

































77


4.4 ‘Compared to a few weeks ago, I feel I have learnt a lot about The European Commission’s concept RRI ‘

4.5 ‘Compared to a few weeks ago, I feel positive about the European Commission’s concept about RRI’ 4.6 Compared to a few weeks ago, I feel I am more likely to continue thinking about how responsibility in research and innovation as it applies to my own work and practice.

4.7 Compared to a few weeks ago, I feel I am more likely to continue to think about, and potentially apply/introduce the aspects that we discussed at the workshop in my own organisation. 4.8 Compared to a few weeks ago, I would be interested to continue the collective/group work we began at the workshop as I think it would contribute to the current and future responsible development of the wider precision medicine system and industry

4.9 Could you qualify, expand or explain any of your responses to the statements above, so that we can better understand your responses?. (open responses)

































78


5. Would you like the SMART-Maps project to keep in touch with you? (Code all that apply): Yes – I would like to receive the follow-up report from the workshop (code 1) Yes – I would like to receive the newsletter and updates about the SMART-Maps project and e-community. (Code 2) Yes – I would like to become actively involved in the next stages of the project, by collaborating with colleagues from the workshop and other societal actors in the co-construction process; and/or developing the ideas we discussed for potential application within my own organisation (Code 3) No thanks (Code 4)

6. Do you have any final comments or reflections you would like to share with the consortium or feed-back to the European Commission? (Open responses)

2. WP7: SMART-Map Formative Evaluation: Stage 2, At the beginning the Pilots ie ‘Before’

Introduction & Consent:

Please explain the purpose of the Evaluation and ask interviewees to sign and return the SMART-Map Consent form:

































79


“The purpose of collecting and collating the SMART-Maps Pilots evaluation data is to formatively evaluate the SMART-Map project (ie in terms of lessons we can all learn from participating in the pilots and evaluating the experience) by capturing the views of participants, before, during, and after the experience. This will be very helpful and valuable to enable the SMART-Maps consortium (facilitated by SMART-Map WP7 Evaluation team) to draw lessons which will be reported to the European Commission and made freely available to the public via the SMART-Map WP7 Final Deliverable Report and in academic articles”

However:

“All responses will be anonomised and individuals will never be identified or identifiable in the final report or follow-up artcles without the explicit permission of those participating in the interviews and evaluation. A draft of the final report will be circulated to all participants for their comment and scrutiny before it is made publically available”.

The ‘pilots’ Formative Evaluation will comprise 3 stages:

i) (This) telecon/skype ‘Before’ ie at the beginning of the pilots ( a 15-20 minute interview)

ii) Voluntary ‘diary sheets’ to be completed during the pilots as/when the interviewee feels appropriate to capture their views, comments, ideas or experiences during the course of the project rather than await memory at the end

iii) A second telecon/skype interview ‘After’ ie at the end of the pilots, approx March 2018 (a 15-20 minute interview)

Interviewer:

Name ------------------------------------------------------------------------

































80


Date of interview--------------------------------------------------------------------------

Interviewee Contact Details:

Name----------------------------------------------------------------------------

Organisation--------------------------------------------------------------------

Email/phone contact for follow-up re ‘after’ pilot interview

---------------------------------------------------------------------------------

(this is for our administrative record keeping and will not be reported in any way that identifies the respondent. A unique ID will be allocated to each respondent for reporting and analysis purposes).

Has the SMART-Map interview consent form been read out to the interviewee, agreed and signed? (mark with X)

Yes -----------------

Q1 Which pilot does this interview refer to? (mark with X)

































81


Synthetic Biology (UK)

Precision Medicine (Sp)

3D Printing (It)

Q2 What is the role of the interviewee within the pilots?

Piloting Company/organisation

SMART-Map Consortium member

Task Group member (UK only)

Advisory Board or other support or facilitation organisation

Please describe :

-----------------------------------------------------------------------

Q3 Since the commencement of your engagement with the SMART-Map project, would you say you are more (or less) positive/negative about t the idea of Responsible Research and Innovation than before you became involved with SMART-Map?

Much more positve

A little more positive

Neither particularly more nor less positive

A little more negative

Much more negative

































82


Q4 Please explain your answer and describe your experience/learning to date engaging with the SMART-Map project

The SMART-Map pilots – Stage 1 – ‘before’

Q5 : Please describe your SMART-Map pilot as you understand it and your/your organisations role within it:

a) The Smart Map Pilot:

b) The interviewees role within the pilot

c) The interviewees organisation’s role within the pilot

Q6 : What was your motivation for participating in the SMART-Maps pilots? (open-ended, please write verbatim what the interviewee says).

Q7 : What are your hopes and aspirations from participating in the SMART-Map pilots stage? (open-ended, please write verbatim what the interviewee says).

Q8 : And what are your expectations from participating?

































83


Q9 : What challenges from participation do you envisage? a) for you personnally? b) for your organisation?

a) Personnally

b) For the organisation

Q10 : What benefits from participation do you envisage? A) for you personally? b)for your organisation?

a) Personally

b) For the organisation

Q11 : Considering the EU H2020 RRI Objectives and the SMART-Map Advisory Boards additional questions to the pilots , which of the following EU H2020 RRI objectives do you anticipate your pilot will contribute to and how?

Aid Memoir for the interviewee, please read out:

➢ To anticipate and assesses potential implications and societal expectations with regard to research and innovation, with the aim to foster the design of inclusive and sustainable research and innovation

































84


➢ So that societal actors (researchers, citizens, policy makers, business, third sector

organisations, etc.) work together during the whole research and innovation process in order to better align both the process and its outcomes with the values, needs and expectations of society.

By way of:

➢ actions on thematic elements of RRI (such as public engagement, open access, gender, ethics, science education), and

➢ via integrated actions that for example promote institutional change, to foster the uptake of the RRI approach by stakeholders and institutions.

Source: https://ec.europa.eu/programmes/horizon2020/en/h2020-section/responsible-research-innovation

In addition the SMART-Map Advisory Board has asked all pilots to consider the following criteria in all pilots and discuss during the SMART-Map Evaluation your responses and experiences vis-à-vis the following questions:

Companies/organisations, in the SMART-Map pilot process, are asked to consider as part of their pilots they will :

(a) Envisage the role / relevance of society in this experience?

(b) Define society and how do you select / approachsociety representatives?

































https://ec.europa.eu/programmes/horizon2020/node/766





85


(c) What impact on society would you expect in this experience?

(d) How would you be willing to include inputs from society in the operationalization of the pilots?

(e) Do you envisage any risks and how would you handle them?

(f) Would you consider implementing incentives?

Source: SMART-Map D2.3 List of Recommendations from the Joint Meeting of the Advisory Committee, July 2017

Q12 Do you have any further comments, ideas or suggestions at this stage?

































86


Finally : Interviewer please explain to the interviewee:

i) The Voluntary ‘respondent diaries’ stage of the Formative Evaluation and email the diary-sheet. The purpose of the diary sheets is to collect any thoughts, reflections, ideas or comments the interviewee may wish to capture as feed-back on the pilots experience during the course of the 5 month pilots, rather than relying on memory at the end.

ii) That we will be conducting a similar telecon interview near the end of the pilots (March 2018), does the interviewee give permission for a 2nd ‘After the pilots’ interview to be conducted, in order for us to capture the experiences and learning from the pilots.

Yes

No

THANK THE INTERVIEWEE AND CLOSE

3. WP7: SMART-Map Formative Evaluation: Stage 2.2, At the end of the Pilots ie ‘After’

Introduction & Consent:

Please re-explain the purpose of the Evaluation and ask interviewees to agree (and interviewer to mark with X) the SMART-Map Consent statement below:

































87


Please read out:

“As you may recall from my interview with you in January.....the purpose of undertaking the SMART-Maps Pilots evaluation is to learn from participants views and experiences during their journey with the SMART-Map project and consortium .. This will be very helpful and valuable to enable the SMART-Maps consortium (facilitated by SMART-Map WP7 Evaluation team) to draw lessons which will be reported to the European Commission and made freely available to the public via the SMART-Map WP7 Final Deliverable Report and in academic articles......... We are now nearing the end of the SMART-Map project and we finally wish to conduct a small number of interviews with those peole who have been most involved with the project and the 3 pilots in UK (Synthetic Biology), Spain (Precision Medicine) and Italy (3D Bioprinting)

The ‘pilots’ Formative Evaluation comprises two stages:

iv) The telecon/skype ‘Before’ ie at the beginning of the pilots in January

v) This second telecon/skype interview ‘After’ ie at the end of the pilots, ie now, in March 2018 (a 15-20 minute interview)

“As before, all responses will be anonomised and individuals will never be identified or identifiable in the final report or follow-up artcles without the explicit permission of those participating in the interviews and evaluation. A draft of the final report will be circulated to all participants for their comment and scrutiny before it is made publically available”.

Are you happy for me to proceed?’

Y- Proceed [ ]

N – Please note if respondent declines with reasons: [ ]

N__________________________________________________________________---

































88


Interviewer:

Name ------------------------------------------------------------------------

Date of interview--------------------------------------------------------------------------

Interviewee Contact Details:

Name----------------------------------------------------------------------------

Organisation--------------------------------------------------------------------

Email/phone contact for follow-up re ‘after’ pilot interview

---------------------------------------------------------------------------------

(this is for our administrative record keeping and will not be reported in any way that identifies the respondent. A unique ID will be allocated to each respondent for reporting and analysis purposes).

Has the SMART-Map interview consent paragraphs been read out to the interviewee and has the interviewees consented? (mark with X above)

Yes -----------------

Q1 Which pilot does this interview refer to? (mark with X)

































89


Synthetic Biology (UK)

Precision Medicine (Sp)

3D BioPrinting (It)

Q2 Please can you clarify for us which parts of the SMART-Maps pilots you participated in:

Italy (3D Bioprinting)

i) The original Industrial Dialogue workshop in Milan? (Y/N) ________

ii) The Munich Industrial Dialogue workshop? (Y/N) ________

iii) The SYNBIONICA network training event in Greece? (Y/N) ________

iv) The final workshop in Milan (March 2018) (Y/N)________

Please describe your understanding of the 3D Bioprinting SMART-Map objectives, process steps and journey, from the initial Industrial Dialogue workshop to the end of the pilots:

And what was your understanding of the role of your organisation within it? (Differentiate between SYMBIONICA role and thepersons own company/organisation)

And what was your understanding of your individual role within it?

UK

































90


i) The original Industrial Dialogue workshop in Manchester? (Y/N)__________

ii) The Budapest Industrial Dialogue workshop? (Y/N)__________

iii) UK SMART-Map Task Group Member? (Y/N)_________

iv) Learning Case Study author? (Y/N)__________

Please describe your understanding of the SynBio SMART-Map objectives, process steps and journey, from the initial Industrial Dialogue workshop to the end of the pilots:

And what was your understanding of the role of your organisation within it?


Spain

i) The original Industrial Dialogue workshop in Valencia? (Y/N) ________

ii) The Aarhus Industrial Dialogue workshop? (Y/N) ______

iii) The IMOGEN workshop which brought together Doctors/Patient Groups? (Y/N)______

iv) The final multi-stakeholder workshop (y/N)_________

































91


Please describe your understanding of the Precision Medicine SMART-Map objectives, process steps and journey, from the initial Industrial Dialogue workshop to the end of the pilots:

And what was your understanding of the role of your organisation within it?


Consortium/Advisory Board or other support or facilitation organisation

Please describe your role:

-----------------------------------------------------------------------

Q3 Now that we are nearing the end of the SMART-Map project, would you say you are more (or less) positive/negative about t the idea of Responsible Research and Innovation than before you became involved with SMART-Map?

Much more positve

































92


A little more positive

Neither particularly more nor less positive

A little more negative

Much more negative

Q4 Please explain your answer, and describe your experience/learning to date engaging with the SMART-Map project

Q5 : As before, can you tell me what was your motivation for participating in the SMART-Maps pilots? (open-ended, please write verbatim what the interviewee says).

Q7 : What are your hopes and aspirations from participating in the SMART-Map pilots stage? (open-ended, please write verbatim what the interviewee says).

Q8a) : And what were your expectations from participating?

Q8b) Do you think your expectations were met? (Y/N) ____________

Please explain your answer:

































93


Q9 : What challenges from participation did you encounter? a) for you personnally? b) for your organisation c) For the wider societal remit of Responsible Research and Innovation in terms of institutional change and changes to practices?

c) Personnally

d) For the organisation

e) For the wider societal remit of Responsible Research and Innovation in terms of institutional change and changes to practices?

Q10 : What benefits from participation did you gain? A) for you personally? b)for your organisation? C) For the wider societal remit of Responsible Research and Innovation in terms of institutional change and changes to practices?

c) Personally

d) For the organisation

































94


e) For the wider societal remit of Responsible Research and Innovation in terms of institutional change and changes to practices?

Q11 : Considering the EU H2020 RRI Objectives, which of the following EU H2020 RRI objectives did you feel your pilot contributed to and how?

Aid Memoir for the interviewee, please read out:

➢ To anticipate and assesses potential implications and societal expectations with regard to research and innovation, with the aim to foster the design of inclusive and sustainable research and innovation

➢ So that societal actors (researchers, citizens, policy makers, business, third sector organisations, etc.) work together during the whole research and innovation process in order to better align both the process and its outcomes with the values, needs and expectations of society.

By way of:

➢ actions on thematic elements of RRI (such as public engagement, open access, gender, ethics, science education), and

➢ via integrated actions that for example promote institutional change, to foster the uptake of the RRI approach by stakeholders and institutions.

Source: https://ec.europa.eu/programmes/horizon2020/en/h2020-section/responsible-research-innovation






































95


Q12 As we come to the end of the SMART-Map project, do you think the project has been able to create any longer term benefits that can be carried forward into the future?

Y/N______________________

If Y – What do you think these longer term benefits are?

What do you think they COULD be?

What would be needed, by whom, for this longer term potential to be reached?

IF N – why do you think the project has not been able to create longer term benefits that can/could have been carried forward into the future? What could have been done differently to create longer term benefits and legacy from the project?

Q13 Do you have any final comments, suggestions or recommendations that we can pass directly to the SMART-Map Consortium, and/or that you would wish to see in our final report to the European Commission?

































96


ON BEHALF OF THE SMART_MAP CONSORTIUM AND THE FORMATIVE EVALUATION TEAM, MAY WE TAKE THIS OPPORTUNITY TO THANK YOUR FOR YOUR PARTICIAPTION IN THE SERIES OF FORMATIVE EVALUATION SURVEYS THAT WE HAVE BEEN CONDICTING THROUGH THE LENGTH OF THE PROJECT.

ACKNOWLEDGEMENTS

SMART-Map consortium: Ilse Marscalek, Maria Scrammel, Nicola Nosengo, Ralf Lindner, Melek Akca Prill, Marie Carmen Alvarez, Angela Simone, Anna Pellizzone, Raffael Himmelsbach, Francesco Lescai (Co-ordinator)

































Documents

FORMATIVE AND SUMMATIVE EVALUATIONprojectsmartmap.eu/wp-content/uploads/2019/04/D_7_2_final.pdf · understood rri/RRI to be, and the challenges, opportunities benefits and costs that