Author  Level  Bibliometrics  

Henk  F.  Moed      

A  talk  at  a  Special  Session  on  Author  Level  Bibliometrics  at  ISSI2013  Vienna,  17  July  2013  

 

 The  MulG-­‐Dimensional  Research  

Assessment  Matrix  

Expert Group on the Assessment of University-Based Research (AUBR, 2010)

Multi‐dimensional Research Assessment Matrix (Part) Unit  of  assessment  

Purpose   Output  dimensions  

Bibliometric  indicators  

Other  indicators  

Individual   Allocate  resources  

Research  producGvity  

PublicaGons   Peer  review  

Research  group  

Improve    performance  

Quality,  scholarly  impact  

Journal  citaGon  impact  

Patents,  licences,  spin  offs  

Department   Increase  mulG-­‐discipl.  research  

InnovaGon  and  social  benefit  

Actual  citaGon  impact  

InvitaGons  for  conferences  

InsGtuGon   Increase  regional  engagement  

Sustainabi-­‐lity  &  Scale  

Internat.  co-­‐authorship  

External  research  income  

Research  field    

PromoGon,  hiring  

Research  infrastruct.  

 citaGon  ‘presGge’    

PhD  com-­‐pleGon  rates  

Multi‐dimensional Research Assessment Matrix (Part) Unit  of  assessment  

Purpose   Output  dimensions  

Bibliometric  indicators  

Other  indicators  

Individual   Allocate  resources  

Research  producGvity  

PublicaGons   Peer  review  

Research  group  

Improve    performance  

Quality,  scholarly  impact  

Journal  citaGon  impact  

Patents,  licences,  spin  offs  

Department   Increase  mulG-­‐discipl.  research  

InnovaGon  and  social  benefit  

Actual  citaGon  impact  

InvitaGons  for  conferences  

InsGtuGon   Increase  regional  engagement  

Sustainabi-­‐lity  &  Scale  

Internat.  co-­‐authorship  

External  research  income  

Research  field    

PromoGon,  hiring  

Research  infrastruct.  

 citaGon  ‘presGge’    

PhD  com-­‐pleGon  rates  

Read    column-­‐  wise  

Multi‐dimensional Research Assessment Matrix (Part) Unit  of  assessment  

Purpose   Output  dimensions  

Bibliometric  indicators  

Other  indicators  

Individual   Allocate  resources  

Research  producGvity  

PublicaGons   Peer  review  

Research  group  

Improve    performance  

Quality,  scholarly  impact  

Journal  citaGon  impact  

Patents,  licences,  spin  offs  

Department   Increase  mulG-­‐discipl.  research  

InnovaGon  and  social  benefit  

Actual  citaGon  impact  

InvitaGons  for  conferences  

InsGtuGon   Increase  regional  engagement  

Sustainabi-­‐lity  &  Scale  

Internat.  co-­‐authorship  

External  research  income  

Research  field    

PromoGon,  hiring  

Research  infrastruct.  

 citaGon  ‘presGge’    

PhD  com-­‐pleGon  rates  

Base  noGon  

The  choice  of  indicators  depends  upon:  •  What  unit  is  to  be  assessed  •  Why  is  the  assessment  done?  •  Which  aspect  is  being  assessed?      Indicators  that  are  appropriate  in  one  type  of  assessment  may  be  inappropriate  in  another  

Valuable  noGons  

•  Data  accuracy  is  crucial    •  Use  data  verified  by  authors  themselves    •  Combine  metrics  and  expert  knowledge  •  Impact  factors  no  subsGtutes  of  actual  impact  •  Use  mulGple  indicators  •  Take  into  account  phase  of  career  •  Take  into  account  unintended  effects  

•  Present  “interpretaGon  manual”,  pros  &  cons  

•  Focus  on  top  vs.  bo_om  of  quality  distribuGon  •  RetrospecGve  vs.  prospecGve  use  of  metrics  

A  base  problem  

•  How  to  assess  the  performance  of  an  individual  by  analyzing  products  that  are  the  result  of  teamwork?    

•  Most  research  arGcles  are  the  result  of  collaboraGon  within  or  between  teams  

More  and  more  individual  researchers  have  to  be  objecGvely  evaluated  for  promoGon  and  hiring  

Use  easily  available  metrics      

Count  total  publicaGons  or  #  arGcles  in  journals  with  the  highest  impact  factors  in  the  field        

Meta  level  :  Policy  issue  

Policy  measure    

Bibliometric  operaGonali-­‐  zaGon  

c  

c  

c  

c  

CASE  1    

Research  community    is  not  sufficiently  oriented  toward  internaGonal    networks  

SGmulate  publicaGon  in  good  internaGonal  journals  

Count  #  arGcles  in  first  impact  quarGle  of  journals  in  subject  field    

Meta  level  :  Policy  issue  

Policy  measure    

Bibliometric  operaGonali-­‐  zaGon  

c  

c  

c  

c  

CASE  2    

Professors  are  legally  bound  to  do  research  but  many  of  them  don’t    

Allow  only  research  acGve  professors  to  decide  on  recruitment  of  new  research  staff  

Select  only  professors  with  >=  3  publicaGons  in  7  years  in  recruitment  commi_ees  

Meta  level  :  Policy  issue  

Policy  measure    

Bibliometric  operaGonali-­‐  zaGon  

c  

c  

c  

c  

CASE  3    

Wider  issues  

•  What  is  an  acceptable  “error  rate”  in  the  assesment  process?  

•  Wrong  in  individual  cases    benificiary  for  the  system  as  a  whole  

•  What  is  a  fair  assessment  process?