8/7/2019 Eval Research

1/30

Robert Tijssen

Centre for Science and Technology Studies (CWTS)Leiden University

Netherlands

tijssen@cwts.leidenuniv.nl

What isWhat is evaluationevaluation ofof

research achievements and impactsresearch achievements and impacts

taking stock of current general issuestaking stock of current general issues

Presentation at the brainstorming meeting on Evaluating Research Outcomes and Impacts

EFC - European Forum on Philanthropy and Research Funding, Brussels 17 April 2008

CWTS

8/7/2019 Eval Research

2/30

For Eur opean foun da t ions , eva lua t ion i s an ins t r um ent

[ an d a p r o ce ss ] w h i ch t e ach e s i m p o r t a n t l esso n s t o b e u se d b y

f oun da t i ons , t he i r p a r t n e r o r gan i sa t i ons , o th e r c i v i l societ y

o rgan i sat i ons , and gove rn m en ts .

I n t h i s ro l e , evalua t i on has inc reasing l y been recogn i sed as hav ing

t h e ca p aci t y t o st r e n g t h e n f o u n d a t i o n s a cco u n t a b i l i t y ,

m anagem en t , unde rs tand ing o f t h e i r w o rk s resu l t s, and t h e ir

cred ib i l i t y , by d i ssem ina t i ng pow er fu l l essons abou t t hose resu l t s.

Source: The role of evaluation in the 21th century foundation / Edward Pauly. - Gtersloh : Bertelsmann Foundation, 2005

8/7/2019 Eval Research

3/30

Research evaluation

[ Research ] ev al ua t i on i s t h e sy st em a t i c det e rm i na t i on

o f m e r i t , w o r t h , and s ign i f i cance o f som e th i ng o r som eone

[ w he re researc h ( - r e la ted ) act i v i t i es m ade a con t r i bu t i on ]

W ha t k i n d o f resear ch ?

How sys tem at i c ?

W h a t k i n d o f co n t r i b u t i o n ?

Ho w si g n if i ca n t w a s t h e co n t r i b u t i o n ( i f an y ) ?

Ho w d i d t h e g r an t i m p a ct o n t h e co n t r i b u t i o n ?

8/7/2019 Eval Research

4/30

Impacts of research grants

Adapted from Lewison (2000)

8/7/2019 Eval Research

5/30

Pressure from (government) accounting agencies

towards more detailed financial and legal record

keeping and towards stricter and more detailed

auditing of financial and legal procedures

Similar pressure from in-house administrators,

financial departments, controllers and control units toprovide more (quantitative) information on (interim)

results of research funding

Expected outcomes may have to be specified before

funding is released, monitored and evaluated

Result: bureaucracy, paperwork and the need for(more) performance indicators and evaluation reports

Financial-administrative environment

http://images.google.com/imgres?imgurl=http://bp3.blogger.com/_mJmwQtPmusk/Rqd0f6Tv1vI/AAAAAAAAAus/6nFtJ6-kNr4/s400/red%2Btape.jpg&imgrefurl=http://burningourmoney.blogspot.com/2007/07/red-tape-industry.html&h=290&w=300&sz=17&hl=en&start=8&tbnid=sZes79erVZ2eTM:&tbnh=112&tbnw=116&prev=/images%3Fq%3Dred%2Btape%26gbv%3D2%26hl%3Den%26sa%3DG

8/7/2019 Eval Research

6/30

Objectives-based/evidence-based

approachOr g a n i si n g p r i n ci p les

To meet a priori objectives (goals, targets, milestones, impacts)

Improve effectiveness and efficiency of funding

Purpose

To relate inputs to the attainment of objectives

Methodo log ies

Qualitative (detecting and interpreting value judgments)

Quantitative (detecting and interpreting quantities)

8/7/2019 Eval Research

7/30

Objectives-based/evidence-based

approachKey st reng t hs

Common sense appeal

Widely used approach

Focus on explicit objectives

Uses established methods and (measurement) techniques

Key w eakn esses

Objectives are not clearly specifiedNarrow basis for judging the level of attainment of a project or program

Focuses primarily on (short term) results

Elements are (not sufficiently) amenable to quantitative measures

8/7/2019 Eval Research

8/30

8/7/2019 Eval Research

9/30

Improving and adaptingevaluation methodologies

Introducing pillars of practical wisdom

http://images.google.com/imgres?imgurl=http://www.cortelimprovement.com/Pillars-of-Success-Org-Exc-.jpg&imgrefurl=http://www.cortelimprovement.com/body_organizational_excellence.html&h=353&w=527&sz=44&hl=en&start=1&tbnid=RJ3tmF1xmmVvEM:&tbnh=88&tbnw=132&prev=/images%3Fq%3Dpillars%26gbv%3D2%26hl%3Den%26sa%3DG

8/7/2019 Eval Research

10/30

8/7/2019 Eval Research

11/30

Evaluation objectives should focus on

the production, dissemination and

utilization of research-basedknowledge, skills and capabilities

http://images.google.com/imgres?imgurl=http://senatesite.com/blog/uploaded_images/dc1-796917.JPG&imgrefurl=http://senatesite.com/blog/&h=768&w=1024&sz=160&hl=en&start=34&tbnid=MjZwFMiKSgvDoM:&tbnh=113&tbnw=150&prev=/images%3Fq%3D%2522policy%2Bissues%2522%26start%3D20%26ndsp%3D20%26svnum%3D10%26hl%3Den%26lr%3D%26sa%3DNhttp://images.google.com/imgres?imgurl=http://www.mcbassi.com/uploaded_graphics/sub.pic.4.250.300.jpg&imgrefurl=http://www.mcbassi.com/freeresources_guides.html&h=300&w=250&sz=60&hl=en&start=13&tbnid=lcreIjWD8uyPhM:&tbnh=116&tbnw=97&prev=/images%3Fq%3D%2522Measurement%2Bmethodologies%2522%26svnum%3D10%26hl%3Den%26lr%3D%26sa%3DG

8/7/2019 Eval Research

12/30

Focus on the defining characteristicsof scientific research

People

Creativity and ideas

Risk taking

Long term agendas

International competition and/or cooperation

8/7/2019 Eval Research

13/30

Trust, but verify

http://images.google.com/imgres?imgurl=http://www.dba-oracle.com/t_oracle_myths_performance_testing_files/image016.jpg&imgrefurl=http://www.dba-oracle.com/t_oracle_myths_performance_testing.htm&h=183&w=251&sz=9&hl=en&start=18&tbnid=jOhB5G1-KVCIIM:&tbnh=81&tbnw=111&prev=/images%3Fq%3DTrust,%2Bbut%2Bverify%26gbv%3D2%26hl%3Den%26sa%3DG

8/7/2019 Eval Research

14/30

Include measurable objectives

http://images.google.com/imgres?imgurl=http://jan.ucc.nau.edu/~d-ctel/OLG/images/GS-Objectives-page_01.jpg&imgrefurl=http://afroditiapostolidou.blogspot.com/2005_12_01_archive.html&h=321&w=258&sz=12&hl=en&start=1&tbnid=1gCZYJfXMMbpcM:&tbnh=118&tbnw=95&prev=/images%3Fq%3Dmeasurable%2Bobjectives%2B%26gbv%3D2%26hl%3Den%26sa%3DG

8/7/2019 Eval Research

15/30

Numbers matter

Measurable criteria and numerical performanceindicators drives behaviors and results

Goodharts Law: when a measure becomes a target, it ceases to be agood measure

http://images.google.com/imgres?imgurl=http://i3.photobucket.com/albums/y99/ElaineSupkis/clnfeb/trade-deficit-disaster-smal.gif&imgrefurl=http://culturelifenewsii.blogspot.com/2006_02_01_archive.html&h=147&w=146&sz=3&hl=en&start=35&tbnid=yJYqOsN0ZNZEDM:&tbnh=95&tbnw=94&prev=/images%3Fq%3DNumbers%2Bmatter%2B%26start%3D20%26gbv%3D2%26ndsp%3D20%26hl%3Den%26sa%3DNhttp://images.google.com/imgres?imgurl=http://i3.photobucket.com/albums/y99/ElaineSupkis/clnfeb/trade-deficit-disaster-smal.gif&imgrefurl=http://culturelifenewsii.blogspot.com/2006_02_01_archive.html&h=147&w=146&sz=3&hl=en&start=35&tbnid=yJYqOsN0ZNZEDM:&tbnh=95&tbnw=94&prev=/images%3Fq%3DNumbers%2Bmatter%2B%26start%3D20%26gbv%3D2%26ndsp%3D20%26hl%3Den%26sa%3DNhttp://images.google.com/imgres?imgurl=http://i3.photobucket.com/albums/y99/ElaineSupkis/clnfeb/trade-deficit-disaster-smal.gif&imgrefurl=http://culturelifenewsii.blogspot.com/2006_02_01_archive.html&h=147&w=146&sz=3&hl=en&start=35&tbnid=yJYqOsN0ZNZEDM:&tbnh=95&tbnw=94&prev=/images%3Fq%3DNumbers%2Bmatter%2B%26start%3D20%26gbv%3D2%26ndsp%3D20%26hl%3Den%26sa%3DNhttp://images.google.com/imgres?imgurl=http://i3.photobucket.com/albums/y99/ElaineSupkis/clnfeb/trade-deficit-disaster-smal.gif&imgrefurl=http://culturelifenewsii.blogspot.com/2006_02_01_archive.html&h=147&w=146&sz=3&hl=en&start=35&tbnid=yJYqOsN0ZNZEDM:&tbnh=95&tbnw=94&prev=/images%3Fq%3DNumbers%2Bmatter%2B%26start%3D20%26gbv%3D2%26ndsp%3D20%26hl%3Den%26sa%3DNhttp://images.google.com/imgres?imgurl=http://i3.photobucket.com/albums/y99/ElaineSupkis/clnfeb/trade-deficit-disaster-smal.gif&imgrefurl=http://culturelifenewsii.blogspot.com/2006_02_01_archive.html&h=147&w=146&sz=3&hl=en&start=35&tbnid=yJYqOsN0ZNZEDM:&tbnh=95&tbnw=94&prev=/images%3Fq%3DNumbers%2Bmatter%2B%26start%3D20%26gbv%3D2%26ndsp%3D20%26hl%3Den%26sa%3DNhttp://images.google.com/imgres?imgurl=http://i3.photobucket.com/albums/y99/ElaineSupkis/clnfeb/trade-deficit-disaster-smal.gif&imgrefurl=http://culturelifenewsii.blogspot.com/2006_02_01_archive.html&h=147&w=146&sz=3&hl=en&start=35&tbnid=yJYqOsN0ZNZEDM:&tbnh=95&tbnw=94&prev=/images%3Fq%3DNumbers%2Bmatter%2B%26start%3D20%26gbv%3D2%26ndsp%3D20%26hl%3Den%26sa%3DN

8/7/2019 Eval Research

16/30

Evaluation toolbox

Use whatever tools and methods

most appropriate to reach all stakeholders

http://images.google.com/imgres?imgurl=http://www.arb.ca.gov/as/laborrelations/supkit/Images/toolbox.gif&imgrefurl=http://wearethebusinessmen.com/%3Fp%3D45&h=480&w=504&sz=13&hl=en&start=3&tbnid=8NOFyaFqzf4YjM:&tbnh=124&tbnw=130&prev=/images%3Fq%3Dtoolbox%26gbv%3D2%26hl%3Den%26sa%3DG

8/7/2019 Eval Research

17/30

Evidence-based evaluations

require well-acceptable metrics

and performance indicators

http://images.google.com/imgres?imgurl=http://www.peter.com.au/photos/photos/2003-03-30-Workshop_14-Measurements.jpg&imgrefurl=http://www.peter.com.au/photos/sungroper.html&h=480&w=640&sz=57&hl=en&start=26&tbnid=qNfMFhvWAeZ98M:&tbnh=103&tbnw=137&prev=/images%3Fq%3D%2522measurements%2522%26start%3D20%26ndsp%3D20%26svnum%3D10%26hl%3Den%26lr%3D%26sa%3DN

8/7/2019 Eval Research

18/30

Past performance

is an acceptable predictor of

current capabilities

and future success

http://images.google.com/imgres?imgurl=http://wcm.pavliks.com/WCMAdmin/Images/wwwleanvelocityca/images/bigstockphoto_Past_Performance_766574.jpg&imgrefurl=http://www.leanvelocity.ca/Content.cfm%3FC%3D6973%26SC%3D1%26SCM%3D0%26MI%3D4618%26L1M%3D4368&h=800&w=1200&sz=220&hl=en&start=14&tbnid=niRk8I7P-dCMyM:&tbnh=100&tbnw=150&prev=/images%3Fq%3Dpast%2Bperformance%26gbv%3D2%26ndsp%3D20%26hl%3Den%26sa%3DN

8/7/2019 Eval Research

19/30

Degrees of success and failure

Specify key objectives and performance criteria as clearly as

possible in advance

Apply uniform definitions, measurements, metrics and (inter)national

scales

http://images.google.com/imgres?imgurl=http://wcm.pavliks.com/WCMAdmin/Images/wwwleanvelocityca/images/bigstockphoto_Past_Performance_766574.jpg&imgrefurl=http://www.leanvelocity.ca/Content.cfm%3FC%3D6973%26SC%3D1%26SCM%3D0%26MI%3D4618%26L1M%3D4368&h=800&w=1200&sz=220&hl=en&start=14&tbnid=niRk8I7P-dCMyM:&tbnh=100&tbnw=150&prev=/images%3Fq%3Dpast%2Bperformance%26gbv%3D2%26ndsp%3D20%26hl%3Den%26sa%3DNhttp://images.google.com/imgres?imgurl=http://wcm.pavliks.com/WCMAdmin/Images/wwwleanvelocityca/images/bigstockphoto_Past_Performance_766574.jpg&imgrefurl=http://www.leanvelocity.ca/Content.cfm%3FC%3D6973%26SC%3D1%26SCM%3D0%26MI%3D4618%26L1M%3D4368&h=800&w=1200&sz=220&hl=en&start=14&tbnid=niRk8I7P-dCMyM:&tbnh=100&tbnw=150&prev=/images%3Fq%3Dpast%2Bperformance%26gbv%3D2%26ndsp%3D20%26hl%3Den%26sa%3DNhttp://images.google.com/imgres?imgurl=http://wcm.pavliks.com/WCMAdmin/Images/wwwleanvelocityca/images/bigstockphoto_Past_Performance_766574.jpg&imgrefurl=http://www.leanvelocity.ca/Content.cfm%3FC%3D6973%26SC%3D1%26SCM%3D0%26MI%3D4618%26L1M%3D4368&h=800&w=1200&sz=220&hl=en&start=14&tbnid=niRk8I7P-dCMyM:&tbnh=100&tbnw=150&prev=/images%3Fq%3Dpast%2Bperformance%26gbv%3D2%26ndsp%3D20%26hl%3Den%26sa%3DNhttp://images.google.com/imgres?imgurl=http://wcm.pavliks.com/WCMAdmin/Images/wwwleanvelocityca/images/bigstockphoto_Past_Performance_766574.jpg&imgrefurl=http://www.leanvelocity.ca/Content.cfm%3FC%3D6973%26SC%3D1%26SCM%3D0%26MI%3D4618%26L1M%3D4368&h=800&w=1200&sz=220&hl=en&start=14&tbnid=niRk8I7P-dCMyM:&tbnh=100&tbnw=150&prev=/images%3Fq%3Dpast%2Bperformance%26gbv%3D2%26ndsp%3D20%26hl%3Den%26sa%3DN

8/7/2019 Eval Research

20/30

Keep it simple

Grant Output ImpactActivities

but dont ignore realities !

Other inputs Other outputs Other impacts

8/7/2019 Eval Research

21/30

Basket of methods and measures

There is no single best method for researchperformance evaluation

http://images.google.com/imgres?imgurl=http://www.networkworld.com/best/2006/newhedBEST06.gif&imgrefurl=http://www.thetotalhealthshop.com/&h=307&w=436&sz=20&hl=en&start=17&tbnid=Opx2WrVNfQGFiM:&tbnh=89&tbnw=126&prev=/images%3Fq%3Dsingle%2Bbest%26gbv%3D2%26hl%3Den%26sa%3DG

8/7/2019 Eval Research

22/30

Garbage in, garbage out

Check the information value and qualityof the required data for monitoring and evaluation

http://images.google.com/imgres?imgurl=http://www.gocarlo.com/lagalerie/images/alleylamp-512.jpg&imgrefurl=http://mastmalang.wordpress.com/2007/04/05/i-feel-like-a-lamp-post/&h=383&w=512&sz=68&hl=en&start=8&tbnid=-8Be0bUUCX4YeM:&tbnh=98&tbnw=131&prev=/images%3Fq%3Dlamppost%2Band%2Blost%2Bkeys%26gbv%3D2%26hl%3Den%26sa%3DG

8/7/2019 Eval Research

23/30

Include external sources

of independent information

Opinions of subject experts and expert panels

Statistical information (bibliometric data)

Peer review based

http://images.google.com/imgres?imgurl=http://senatesite.com/blog/uploaded_images/dc1-796917.JPG&imgrefurl=http://senatesite.com/blog/&h=768&w=1024&sz=160&hl=en&start=34&tbnid=MjZwFMiKSgvDoM:&tbnh=113&tbnw=150&prev=/images%3Fq%3D%2522policy%2Bissues%2522%26start%3D20%26ndsp%3D20%26svnum%3D10%26hl%3Den%26lr%3D%26sa%3DN

8/7/2019 Eval Research

24/30

Peer review basedquality control systems are not infallible

Results of peer review are highly dependent on theselection of experts and expert panels/committees

Knowledgeable and impartial evaluators are difficult to

find (in small countries/scientific communities)

Critical aspects of research, research environment,technical infrastructure, or details of researchfindings, are difficult to grasp by external experts

Experts lack expertise across all relevant and relatedknowledge domains (interdisciplinary andmultidisciplinary areas)

Experts lacking in newly emerging fields (with manyyoung researchers and newcomers)

Lack of consensus, lack of common frame of reference

Bibli t i i di t

http://images.google.com/imgres?imgurl=http://senatesite.com/blog/uploaded_images/dc1-796917.JPG&imgrefurl=http://senatesite.com/blog/&h=768&w=1024&sz=160&hl=en&start=34&tbnid=MjZwFMiKSgvDoM:&tbnh=113&tbnw=150&prev=/images%3Fq%3D%2522policy%2Bissues%2522%26start%3D20%26ndsp%3D20%26svnum%3D10%26hl%3Den%26lr%3D%26sa%3DNhttp://images.google.com/imgres?imgurl=http://www.mcbassi.com/uploaded_graphics/sub.pic.4.250.300.jpg&imgrefurl=http://www.mcbassi.com/freeresources_guides.html&h=300&w=250&sz=60&hl=en&start=13&tbnid=lcreIjWD8uyPhM:&tbnh=116&tbnw=97&prev=/images%3Fq%3D%2522Measurement%2Bmethodologies%2522%26svnum%3D10%26hl%3Den%26lr%3D%26sa%3DG

8/7/2019 Eval Research

25/30

Bibliometric indicators

Bibliometric data are based on a large number ofexpert opinions and peer judgements

(peer-review accepted publications and citations

from peers)

Bibliometrics is non-reactive measurement

methodology

(decisions to accept or cite papers are not or

much less - affected by the possibility of future

evaluation)

Due to the large number of peer votes, and their

international scope, the information base is

expanded

(biases and misconceptions of local experts are also

countered)

8/7/2019 Eval Research

26/30

Partnerships

Researchers and research organizations

should submit high-quality

(cleaned and harmonized) performance data

Evaluation fatigue and

http://images.google.com/imgres?imgurl=http://www.digilore.com/images/partnersImage.jpg&imgrefurl=http://www.digilore.com/partners.shtml&h=300&w=369&sz=16&hl=en&start=2&tbnid=SDSeV5glWo4VyM:&tbnh=99&tbnw=122&prev=/images%3Fq%3Dpartners%26gbv%3D2%26ndsp%3D20%26hl%3Den%26sa%3DN

8/7/2019 Eval Research

27/30

Evaluation fatigue andadministrative burdens

Collect essential information only

Tailor the information need to the relevant issues and contexts

(e.g. fields of science, types of activity, timescales)

http://images.google.com/imgres?imgurl=http://rockycha.files.wordpress.com/2007/09/fatigue.jpg&imgrefurl=http://rockycha.wordpress.com/2007/09/&h=392&w=281&sz=23&hl=en&start=2&tbnid=B6ZIuOwU2cF7yM:&tbnh=123&tbnw=88&prev=/images%3Fq%3Dfatigue%26gbv%3D2%26hl%3Den%26sa%3DGhttp://images.google.com/imgres?imgurl=http://www.antique-hardware.com/key.jpg&imgrefurl=http://www.polyvore.com/cgi/set%3Fid%3D323488&h=315&w=306&sz=13&hl=en&start=2&tbnid=uMmfl9_TVoavCM:&tbnh=117&tbnw=114&prev=/images%3Fq%3Dkey%26gbv%3D2%26hl%3Den%26sa%3DG

8/7/2019 Eval Research

28/30

Learning is the key

Accept trade-offs between accountability andeffectiveness/efficiency of administrative

structures and procedures

Mismatches between information needs and supply

Shift focus from one-off ex-post evaluations ofreturns to ongoing monitoring of capability

development, scientific progress and(sustainable) added values

http://images.google.com/imgres?imgurl=http://www.antique-hardware.com/key.jpg&imgrefurl=http://www.polyvore.com/cgi/set%3Fid%3D323488&h=315&w=306&sz=13&hl=en&start=2&tbnid=uMmfl9_TVoavCM:&tbnh=117&tbnw=114&prev=/images%3Fq%3Dkey%26gbv%3D2%26hl%3Den%26sa%3DG

8/7/2019 Eval Research

29/30

It takes two to tango

Involve stakeholders and users and in all stagesof the evaluation process

Share tasks and responsibilities

From jury judgment to coaching?

L i

8/7/2019 Eval Research

30/30

Learning curves

Focus evaluation on determining the key successfactors (and causes of failures)

Responsiveness to science-related developments

Improving evaluation methods and tools

http://images.google.com/imgres?imgurl=http://www.pms.ac.uk/pphi/institute/masters/images/learningJourneyBig.gif&imgrefurl=http://www.pms.ac.uk/pphi/institute/masters/index.html&h=384&w=581&sz=31&hl=en&start=23&tbnid=NNe7loVVviK1NM:&tbnh=89&tbnw=134&prev=/images%3Fq%3Dlearning%26start%3D20%26gbv%3D2%26ndsp%3D20%26hl%3Den%26sa%3DN