Phoenix – Fostering the rebirth of social sciences and humanities in Central Asia

2www.esf.org

Phoenix – Fostering the rebirth of social sciences and humanities in Central Asia

2nd PHOENIX Workshop“Road to Excellence: Research Evaluation

in the Social Sciences and the Humanities”Issyk-Kul (Kyrgystan), 20-22 May 2007

”Supranational Peer Reviewin the Social Sciences and the Humanities:

some European Experiences”

Dr Rüdiger Klein European Science Foundation

Dep Head HumanitiesSSO Research and Foresight

3www.esf.org

2nd PHOENIX Workshop: “Research Evaluation”

”Supranational Peer Review in the Social Sciences and the Humanities: some European Experiences”

1) Presentation of institutional background (ESF)2) Discussion of lead questions on peer review:

- notion of peer review;- examples of peer review;- characterstics of peer review;- steps towards international peer review;- challenges of supranational peer review;

3) European Reference Index for the Humanities (ERIH)

4www.esf.org

ESF Membership

75 Member Organisations in 30 countries, also beyond the European Union

• Research funding organisations (e.g. research councils)

• Research performing organisations (e.g. national research institutes)

• Academies

Combat fragmentation; create critical mass; advance science

• Partnerships with agencies outside Europe

5www.esf.org

ESF History and Role

• Established in Strasbourg in 1974• An independent, non-governmental

organisation• Offices in Strasbourg and Brussels• Budget (2006): € 41 Mio.• Science budgets networked : € 1-3 Mrd.• Workshops and small research programmes

(15.000 – ca. 500.000 €)• 30 large EUROCORES programmes

(ca. 6-12 Mio. €)• EURYI young researchers award (1.25Mio.€)• Research foresight (incl. research

infrastructures)• ESF Member Organisation Fora

6www.esf.org

ESF Mission

ESF provides a common platform for its Member Organisations in order to:

• advance European research• explore new directions for research at supranational

level

Cooperation between ESF Member Organisations, e.g.:- coordinated research programmes;- joint peer review processes and benchmarking; - research infrastructures

ESF serves the needs of the European research community in a global context through collaboration with

• Non-European, national research funding agencies (NSF, JSPS, CASS; ministries; academies: RAN)

• UN (UNESCO etc.); ICSU; UAI; OECD; NATO; etc.

7www.esf.org

European COCOoperation in the field of SScientific and TTechnical research

• Origin: – Started 1971: Ministerial Conference,

– 19 Member States

• Current Participation:– 2006: 34 COST Member States +

1 cooperating state (Israel)

• COST Actions:– Concerted Actions (Networks) of

nationally funded R&D projects[all fields of research]

8www.esf.org

ESF Scientific Standing Committees & Expert Boards

Standing Committees• Humanities• Life, Earth & Environmental Sciences• Medical Sciences• Physical & Engineering Sciences• Social SciencesExpert Boards• Committee on Radio AstronomyFrequencies• European Polar Board• European Space Science Committee • Marine Board • Nuclear Physics European Collaboration

Committee

9www.esf.org

Social Sciences at ESF

Fields:• Business &

Administrative Sciences

• Communication Sciences

• Demography• Economics• Environmental

Sciences• Geography• International

Relations

• Law• Pedagogy & Edu-

cational research• Political Sciences• Psychology &

Cognitive Science• Social

Anthropology• Social Statistics &

Informatics• Sociology• Women’s Studies

10www.esf.org

Humanities at ESF

Disciplines:• Anthropology• Archaeology• Art & art history• History• History &

philosophy of science

• Languages & philologies

• Linguistics• Literary studies• Musicology• Pedagogy & Edu-

cational research• Philosophy• Psychology• Religious studies

& Theology

11www.esf.org

Humanities at ESF

Fields such as:• Area studies• Classical studies• Cognitive science• Communication &

media studies• Culture, develop-

ment, environment• Gender studies• Heritage studies• Urban studies etc.

Research questions such as:• Consciousness• Evolution of cooperation• Global change• Health & welfare• Human dignity• Landscape research• Migration• Security• Technology, culture and society etc.

Research Infrastructures

12www.esf.org

Peer Review: lead questions

Discussion of lead questions on peer review:- notion of peer review;- examples of peer review;- characterstics of peer review;- steps towards international peer review;- challenges of supranational peer review;

13www.esf.org


Examples of Peer review

Assess track-record and potential:• Application for a research grant (individual;

institutional)• Application for a position (individual)

Assess product:• Submission of a manuscript (article, book etc.):

assess “quality” (methods, results, presentation etc.)

• Evaluation of research programmes, research institutes (universities, academies), research funding agencies: assess performance over time [benchmarking]

Sub-category:• Examination of a degree candidate

14www.esf.org


Peer review as a social process

Formal tradition from 17th century (at least)

“market in scientific goods” (knowledge): control over access to resources for research

“competitive struggle for scientific authority” (epistemological conflicts)

Risk: denial and elite privilegeYet: notion of expertise often conveys idea of objective

knowledge (and possibility of progress)

Distortions rarely accounted for or made explicit: economic bias (“mafia”); professional, academic bias (research interests); personal bias

15www.esf.org


Peer review as a social process (cont.’d)

Peers agree / disagree: which is best?

Scientific merit: BUT- Conventional research (interdisciplinarity);- Institutional conservatism (status of researcher or

department);

Ethical Issues- Personal bias / research profile;- Sexism;- “Old Boys Networks”

Institutional traditions of peer review

16www.esf.org


Suggestions for criteria for “successful” peer review

“Success”: likely selection of best proposals according to scientific merit

- Openness: transparency of the process (procedures, criteria, [sometimes names] known to applicants and public)

- Multi-level selection (external mail review; panel)

- Clear conflict of interest guidelines (published): training of panel members; written commitment of external reviewers

- Knowledgeable secretariat (suggestion of peer group: grasp of science; procedures; IT)

- Use of support tools where and as appropriate (e.g.: scientometrics; recognition of outcome/outreach)

17www.esf.org


Suggestions for criteria for “successful” peer review

“Success”: likely selection of best proposals according to scientific merit

IMPORTANT:- recognise fallibility (“struggle for capital”);- maintain flexibility (interdisciplinarity; internationality)- monitor group dynamics;- improve accountability

For international peer review:- acknowledge cross-cultural differences (description of

science fields; interdisciplinarity)

18www.esf.org


Forms of international peer review

• international composition of juries for selection in national competitions;

• international composition of research groups;• competitions of international grant-making agencies;• international coordination of competitive funding from

different national grant-making agencies to support international research groups:“juste retour”:- agencies pays only successful researchers from own country;- funding partly according to ability of agency“common pot”:- all agencies contribute to joint funds;- funding exclusively according to excellence of researchers

19www.esf.org


Steps towards international peer review

Specific challenges in the Humanities:- International research perceived as new

phenomenon (but: academies; learned societies);- Role of national research traditions (incl.

language);- Contribution of Humanities research to national

identity debates (e.g. history, literature, anthropology);

- Support tools lacking (indicators)Overall: Languages considered not as a barrier, but

language diversity considered a cultural and intellectual asset

20www.esf.org

European Reference Index for the Humanities (ERIH)

Starting point:- assessment: unsuitability of existing databases (e.g. ISI) for journal output of European and other non-Anglo-Saxon Humanities research- access: poor visibility of much of journal-based European scholarship in Humanities

N.B.: Internationalisation of research (careers; collaborations; ERC etc.) requires comparability of quality across borders and languages

Objective 1: identify and categorise good quality research journals in the Humanities- New: mapping of Humanities research in all languages located in an international context;- Encourage best practice (peer review): dialogue with publishers and editors;

Objective 2: improve access to and visibility of Humanities research

21www.esf.org


Process:involve research community (different levels of peer review)

- 30 national funding agencies (Europe) collect input through consultation of research community:14.000 suggestions

- 15 international expert panels [unpaid] analyse input and categorise journals (18 months);

- international subject associations and specialist libraries widely consulted: 6.000 comments;

- international journals from the entire world included [A&B] ; language-specific journals so far only from Europe [C]

- open feedback form and continuous updates (April 2007 onward); structured interaction with publishers and editors (and other stakeholders) through workshops etc.

22www.esf.org

European Reference Index for the Humanities (ERIH): disciplinary coverage 2006

Initial lists are highlighted in yellow; draft lists are not highlighted

DisciplineInitial nationalsubmissions

1st draft lists

Consolidated comments Consultation1a

Cnsolidated comments Consultation 1b

Initial lists/ draft lists

Mar-Apr 2006 Mar-Apr 2006 Jul-2006 Oct-2006

Anthropology (social and evolutionary) 752 154 133 17 242

Archaeology 1310 524 290 25 420-440

Art and Art History 938 445 292 16 550-600

Classical Studies 619 256 321 11 260-280

Gender Studies 155 181 37 6 119

History 1419 874 508 71 900-1000

History and Philosophy of Science 806 145 64 4 166

Linguistics 1093 680 391 34 586

Literature 1453 1481 706 10 900-1000

Music and Musicology 204 n/a 187 4 166

Oriental and African Studies 196 588 386 14 600-650

Pedagogical and Educational Research 666 404 271 92 450-470

Philosophy 658 320 153 22 305

Psychology 1198 1201 159 4 600-650

Religious Studies and Theology 745 n/a 580 10 371

23www.esf.org


Next steps: - include new areas: applied ethics; communication & media;

archives, museum, library studies- consider subdivision of large fields;

- testing of robustness of the lists requested

- ERIH National contact points: improve national feedback

- consider models to include monographs, conference proceedings etc.

- widen geographical consultation through partnerships with non-European agencies and initiatives(East Asia; Russia/NIS/Caucasus; Latin America; Africa)

- widen discussion on use to include other perspectives: Research Infrastructure; Virtual Learning Environment(“Permanent Access to the records of science”)

24www.esf.org

2nd PHOENIX Workshop: “Research Evaluation”

...thank youfor your attention

For further information , please visit www.esf.org,or write to: [email protected]

www.esf.org

Documents

Phoenix – Fostering the rebirth of social sciences and humanities in Central Asia