ARRAShort story of

Slovak ranking

Juraj BartaCo-founder and chairman

of executive board

What is ARRA

• What is ARRA

• History

• Methodology

• Achievements

• Information assymetry

• Open questions

What is ARRA

• Independent NGO

• Main purpose: collect information, process it and publish annual ranking of Slovak universities

• Founded in 2004

• First ranking: 2005

• Other projects: • Top Slovak scientists• Students, alumni, teachers survey• Employers demand analysis

Why did we do it?

• Quality of higher education was no topic of disputes

• Lack of data-based opinions on universities

• Growth of number

of schools, students, professors…

• We believed there was a need and demand for an independent view

• We had necessary resources:• Skills & Expertise• Reputation & Contacts • Willingness

Public debate (illustrations)

• Debate on quality lacks quality (Trend)• Knowing the truth about their level shall help the

shools (head of Acreditation committee)

• Quality assessment should be driven by the government (head of Rectors’ conference)

• General feeling: Someone should do it, but not us, not them, not now…

Who founded ARRA?

• Ferdinand Devinsky, former rector of Commenius university, MP

• Jan Pisut, former minister of education

• Renata Kralikova

• Juraj Barta

• Michal Fedak

• Ivan Ostrovsky

Other people

• Board of experts

• Ivan Stich

• Ivan Wilhelm

• Pavel Brunovsky

• Julius Horvath

• Jaromir Pastorek

• Dusan Kovac

(…)

• Board of Trustees

• Jozef Kollar

• Pavol Lancaric

• Imrich Beres

• Rado Bato

• Jan Toth

• Jaroslav Pilat

• Martin Fronc

• Ivan Miklos

International cooperation

• Founding member of International Ranking Experts Group

• Together with other renowned agencies from all over the world

• Prof. Devinsky is a member of the organisations executive body

Principles & Methodology

• World Bank experts to help us on methodology• Don Thornhill• Lewis Purser

• Principles: indepenece, transparency, expertise, data-based statements, no representation of schools nor other bodies

Res ipsa loquitur

Methodology (1)Category 2005 2010 Label Title Institution's 1 RP1 Graduates' quality (employer's view)Reputation 2 RP2 Peer reviewScience 3 1 VV1 Number of publications in WoK (last 10 years) and per number of creative workers (CW)Research 2 VV1a Number of journal articles outside WoK (last year)

(valid only for HUM & SPOL)3 VV1b Number of books outside WoK (last year)

(valid only for HUM & SPOL)4 4 VV2 Number of citations per CW in WoK 5 VV2a Number of citations per one article in WoK (last 10 years)5 VV3 Number of publications from VV1 with 5+ citations per CW 6 VV4 Proportion of full-time PhD students (last year)

per professor/associate professor6 VV4a PhD studies' success rate (last three/six years)

7 7 VV5 Average annual number of PhD graduates (last 3 years) per professor/associate professor

8 8 VV6 Number of full-time PhD students in proportion to the total number of full-time students (last year)

9 VV6a PhD studies' success rate10 9 VV7 Combined grant funding (VEGA, KEGA agencies) per CW (last year)11 10 VV8 Funding from AR (applied research), ISTC, ASST grants per CW12 11 VV9 Overall grant funding per one CW (last year)13 12 VV10 Grant funding from ESF (Europen Social Fund)/other foreign grants

Methodology (2)Category 2005 2010 Label Title Students' 14 SK1 Students with internet access at the facultyComfort 15 SK2 Students with internet access at the dormitory

16 SK3 Success rate of dormitory applications17 SK4 Number of students with scholarships granted by a university18 SK5 Sport facilities and cultural life19 SK6 Study materials accesible via internet20 SK7 Number of students attending survey on quality of education21 SK8 Impact of student surveys on quality of education

Study 22 13 SV1 Student/teacher ratioand 23 14 SV2 Student/(associate) professor ratioEducation 24 15 SV3 Teachers with/without PhD ratio

25 16 SV4 Proportion of professors/associate professors on the total number of teachers

26 SV5 Average age of active professors27 17 SV6 Admissions: applications/places offered28 18 SV7 Admissions: proportion of actually matriculated students 29 19 SV8 Proportion of foreign students30 SV9 Number of graduates unemployed after 6 months31 SV10 Number of core students studying abroad

Financing 32 F1 Expenditure per student33 F2 Success of business activities34 F3 Proportion of grant funding to the overall budget

Methodology (3 – changes)

2005-2010 major changes & developments:- reputation is not considered

- student’s comfort is hard/ineffective to measure and is omitted

- No. of publications w/ 5+ citations is redundant (few satisfy this criteria and these institutions have several good results in other research indices)

- finances and SV9-10 are available only for universities as a whole, not for particular faculties

+ articles and books outside WoK are considered for HUM & SPOL

Schools grouping (compare the comparable)

2005• Natural sciences• Medicine&Pharmacy• Technology• Agriculture• Social sciences • Arts and Humanities

2010• Natural sciences• Medicine&Pharmacy• Technology• Agriculture

• Philosophy

• Theology

• Law

• Teaching

• Economics & Management

• Arts

• Other social sciences

Information assymetry 1 (theory)

• Akerlof’s ‘lemons’

• under prolonged IA quality

standards drop significantly

• situation in HE: schools have more info about quality than students

• cheaper to teach more students (economies of scale)

• easier to have non-individual approach & teach less info

Information assymetry 2 (SK situation)

• only 28% of students on Slovak HEIs chose the school primarily based on quality

• in less than 10 years number of Slovak students in CZE quadrupled (5k->22k)

• there was no information about the quality/ranking of Slovak HEIs

• most of Slovak HEIs are perceived alike

• students, who require some quality assurance, tend to go abroad

Information assymetry 3 (SK solution)

• general ways how to fight IA: warranty, brand, state intervention, independent QA

• warranty is useless in education

• brand consciousness is underdeveloped

• state processes are slow (accreditation took 6 years) and also prone to lobbying (regional politicians, etc.)

• best solution in SK = independent quality assessment agency

Information assymetry 4 (implications)

• debate about the quality of research & edu commenced

• best faculties attract more and better students (STU Chem: from 350->600)

• worst schools in stagnation or even decline (TUAD: from 8100->5700)

Reactions and Achievements

• Provoking discussion

• Both refusals (method, people…) and supports

• Schools refering to the ranking

• Used as a source of data for EUA

• Even schools who neglected it later reffered to it internally

• Debates no longer based on feelings & debating skills

• Consultancy for Ministry of Education, Slovak Academy of Sciences, Universities, Media…

• From ranking agency to a think tank?

But...

• Are rankings objective?

• Do not we make the mess worse?

• What has research to do with education?

• Shall the schools be all the same?

Thank you!

Regional peers

2009 2010 University Location

201-302 201-300 Charles University Prague

303-401 301-400 Eotvos Lorand University Budapest

303-401 301-400 Jagiellonian Univeristy Krakow

303-401 301-400 University of Szeged Szeged

303-401 301-400 University of Warsaw Warsaw

402-501 401-500 University of Ljubljana Ljubljana

2009 2010 University Location

229 267 Charles University Prague

302 304 Jagiellonian University Krakov

349 364 University of Warsaw Warsaw

394 - Czech University of Technology Prague

• US News and World Report

• Shanghai ranking

Differences

Publications of University staff per 1 mil. inhabitants

0

200

400

600

800

1000

1200

1400

1600

19

94

19

95

19

96

19

97

19

98

19

99

20

00

20

01

20

02

20

03

20

04

20

05

20

06

20

07

20

08

20

09

SK

SLO

PL

A

CZ

HU