Embed Size (px)
Citation preview
Involving end users in research proposal evaluation: A case study with the Dutch Heart Foundation
Ingeborg MeijerCentre for Science and Technology Studies, Leiden University
QMM Workshop, Brunel University, London1-2 October 2015
2
Outline
• Societal quality: Ensuring use of of research • DHF and the end user panel• Evaluation in practice - triangulation• Methods
– Observation of peer review– Questionnaire– Interview
• Results• Conclusion
Scientific & societal quality
• Economic returns
• Cultural returns
• Social returns
• Scientific returns
Scientific interaction
sProfessional interactions
Private interaction
s
Public interaction
s
3
Science and society
Evaluation scienceMonitoringInvolving End
users
ProgrammingSocietal Demand driven
• The process of value creation is by transferring knowledge from a research institute into society (private or public parties, or general public), and includes (demand-driven, user-inspired) research programming, and interaction with potential users during the research.
5
Dutch Heart Foundation objectives
Dutch Heart Foundation (DHF)• Charity Fund > 50% of budget to scientific research • Mission: 3 times less….
– Less people get a cardiovascular disease;– Less people suffer from cardiovascular disease;– Less people die from cardiovascular disease.
• Change in objective:– Closer connected to donators,– Faster translation from results science to patients– More participation from stakeholders
• Goal: to experiment with processes in the core of the DHF > evaluation of research proposals.• End User panel (EP) & criteria to evaluate
societal aspects
6
7
Societal quality operationalised5 Criteria• Relevance of the health
problem• Contribution of the research to
solving the health problem• To the next step in the
research (or development) process.
• Focus on activities towards (eventual) application of results in healthcare– CV of applicant– Objectives, strategy and
actions• Participation of stakeholders
• Relation between criteria
Relevance
Activity towards actors
Societal quality
Participation
8
Phase 2 Practicing in reality• Separate calls
– Young investigator personal grant– Focused topic: women and cardiovascular disease– Focused topic: congenital heart disease in children
• End user panel evaluation in parallel to scientific evaluation panel
• Different types of information sharing• No formal input of evaluation of societal aspects
on decision by the board of DHF (despite formal advice)
• Evaluation of the whole process by CWTS
9
Overview of calls and EP input
Societal aspects in evaluation process
Personal Women Congenital
CTMM
Information in call text + + + -Intermediate advice to scientists - + - +Final advice to scientists + + + -Advice to ISC as referent - + - -Advice to ISC in person - - + -Interaction with EP on same day as ISC + + NAInteraction with EP on another day as ISC - - + NA
10
Research questions• Is the EP able to play a role in evaluation of
research proposals, and what kind?• Is there a difference in focus between ISC and EP?• How do scientists value the feedback from
external stakeholders?• Are scientists aware of societal relevance or
quality aspects of their work?• What do they think about the aim and set up of
the DHF?
Evaluation in practice: triangulation
11
Questionnaire Interview
EP Societal
ISC
EPISC
Interaction
Observation
12
Observations
Observations of peer review
• Aim: comparing distribution of arguments between ISC and EP
• 2x ISC and 1x EP • Session was chaired• Only arguments
were scored, not final judgement
13
Observations – Scores• Use of observation score form• Scoring arguments
– Positive– Negative– Person– Before or after presentation– Content of remarks
• Counting occurrence of arguments• Limitation: Validation of scores by independent
other; field knowledge
14
15
Result observation Peer review – congenital EP vs ISC
EP most used“Next step” MA3 > positive “Activity” MA4 > negativeOften in combination with: “Feasibility” WA4 positive“Solution” WA2 negativeNegative goes downAP and CL arguments used
ISC most used“Feasibility” WA4 both in positive and negative contextAll others: very limited
16
Result observation Peer review –ISC congenital & women
Comparison two calls:Feasibility is dominant argument in discussionSecondly, the chosen approach (WA2) and workplan (WA5) are discussed most and it relates to how realistic plan isNote: ISC has not been asked to discuss societal aspects, and contribution of EP in person (congenital) has little effect.
17
Preliminary conclusion
• EP and ISC use different arguments, where the EP uses more scientific argumemts than the ISC societal arguments.
• Suggests that EP has a broader vision compared to ISC which has little attention for applicant or context related aspects.
• EP peer review could have added value
18
Questionnaire
Questionnaire• Aimed at scientists involved in session with End
user panel (n=20)• Goal: Establishing perceptions of “societal” and
opinion on capabilities of End User Panel• Three main questions, worked out through
statements to agree upon (4-point Likert scale)– Vision of scientists on the concept of societal quality– Vision of scientists on the role of the End Users and their
feedback– Weighing the contribution of the EP and societal criteria on the
outcome of the evaluation
19
20
21
Q: What is societal quality? Yes, agree No, don’t agree
Process of knowledge valorisation leads to societal benefit.
I think that societal benefit is mainly related to communication with users.
Societal benefit has social, economic and cultural aspects
I think that valorisation has mainly to do with economic innovation (some doubt)
Societal quality of research relates to value creation through the interaction of the research group with the societal environment
I think that societal quality of research is independent of the scientific quality and interaction with fellow scientists.
Societal quality is related to focused activities towards diverse users.
Paying attention to societal quality in an early stage of a research project increases the chance that results are indeed transferred to the right actor much quicker.
It is important that scientists take into account the context of users, and adjust the way they transfer knowledge to that user.
22
Q: Contribution of EP to assessmentStatement All
(median 2)
personal EP
The judgement of my scientific peers on my proposal is sufficient for me.
2,5 3 2 (2)
I understand that when a proposal is scientifically ‘state of the art’, it may be rejected based on insufficient societal quality.
2,32 1,75 2,63 (3)
I understand that only proposal that have sufficient societal quality proceed to scientific evaluation process.
2,21 1,75 2,5 (2)
I think that scientific and societal quality should count equally
2,18 1,5 2,38 (2)
I find all this attention for societal quality or benefit unnecessary and too much
1,95 2,75 2,25 (2)
Preliminary conclusion• Scientists and EP largely agree on what societal quality is. • Young researchers have less interest in application of
knowledge, don’t want to pay attention to it in an early stage, and think it has mainly to do with communication.
• Societal quality cannot be leading in deciding what proposals to fund (scientists and EP)
• Method wise, also look at distribution of answers since n=low
23
24
Interviews
Interviews – semi-structured• N=20• 3 topics:
– Preparing for the interaction with EP– The actual interaction with the EP– Feedback from EP
• Interview usually 30 minutes, after questionnaire• All comments collected, not counted, broad
picture
25
26
Interviews – snapshot of results
PresentationProfessionals in EPRole in decision
PositiveNext step
CommunicationCareer/grant
Feedback vagueNot transparentPersonal grants
27
Mixed method message
28
Preliminary conclusion• The contribution, assessment, and feedback of the
EP has added value, but in general scientists do not hear anything new. It is a different emphasis.
• Scientsist think that End users have a different view, which is true in some aspects but not in others.
• Interaction EP with ISC could be valuable• Young scientists: different approach necessary
because of funding pressure and scientific careers (MM)
• Procedure needs to be transparent• Feedback from EP more explicit
