11

University Rankings as Component of Quality Challenge in University Rankings as Component of Quality Challenge in a Changing Landscape of Higher Educationa Changing Landscape of Higher Education

Jan SadlakPresident, IREG Observatory on Academic Ranking and Excellence

Symposium on Identifying Excellence and Diversity in International Higher Education: Rankings and Beyond

Ecole Normale Superieure, 18 May 2011, Paris, France

2 2

““Changing landscape” of HEChanging landscape” of HE

University rankings need to be seen in the context of “changing University rankings need to be seen in the context of “changing landscape” of higher education:landscape” of higher education:

- - MassificationMassification of student enrollment in higher education; of student enrollment in higher education;- Number of HE institutions and their “diversification”Number of HE institutions and their “diversification” (type (type

of institutions, study programs, modes of learning and teaching);of institutions, study programs, modes of learning and teaching);- Changes in relation between the State and higher education Changes in relation between the State and higher education

[acting on behalf of variety of stakeholders] - Shift from an [acting on behalf of variety of stakeholders] - Shift from an academic-self-governance to “academic-self-governance to “accountable autonomyaccountable autonomy”;”;

- Passing from manufacture to industrial mode of production and Passing from manufacture to industrial mode of production and dissemination of dissemination of research research [“knowledge based economy” and [“knowledge based economy” and creativity];creativity];

- Impact of NIT;Impact of NIT;- From “Internationalization” to “From “Internationalization” to “GlobalizationGlobalization”;”;- ““Quality challengeQuality challenge”: shift from self-declaration to external ”: shift from self-declaration to external

verification of quality.verification of quality.

3

Quality Challenge Quality Challenge

New landscape of higher education New landscape of higher education creates demand for greater greater accountabilityaccountability and and transparencytransparency based on based on shared basis for evaluation as well as shared basis for evaluation as well as informationinformation which is which is readable by a large and diversified stakeholders about readable by a large and diversified stakeholders about quality and performance of HE institutions and their quality and performance of HE institutions and their activities. activities.

Main instruments used for this purpose are:Main instruments used for this purpose are: AccreditationAccreditation;; BenchmarkingBenchmarking;; Rankings/LeagueRankings/League tables. tables.

4

AccreditationAccreditation a process during which HE institution or study program is a subject a process during which HE institution or study program is a subject

evaluation by competent body in order to establish if the given institution evaluation by competent body in order to establish if the given institution or programme meet or programme meet a set of standardsa set of standards [minimal] with regard to [minimal] with regard to teaching, research and services. This is why it is sometime interpreted as a teaching, research and services. This is why it is sometime interpreted as a kind of ‘social contract’, particularly in case of institutional accreditation. kind of ‘social contract’, particularly in case of institutional accreditation.

three types of accreditation: three types of accreditation: institutional, study programsinstitutional, study programs and andprofession-relatedprofession-related..

Multiple and professional Multiple and professional accreditations as quality labelaccreditations as quality label::in engineeringin engineering: : EUR-ACE/European Accreditation of Engineering Programs, EUR-ACE/European Accreditation of Engineering Programs, U.S. ABET Inc./Accreditation Board for Engineering and TechnologyU.S. ABET Inc./Accreditation Board for Engineering and Technologyin business educationin business education: : EQUIS/European Quality Improvement System EQUIS/European Quality Improvement System ofofthe European Foundation for Management Development (EFMD), the European Foundation for Management Development (EFMD), AMBAAMBAof the Association of MBAs; of the Association of MBAs; AACSB International Accreditation AACSB International Accreditation of theof theAssociation to Advance Collegiate Schools of Business [so-called “tripleAssociation to Advance Collegiate Schools of Business [so-called “triplecrown ”accreditation]. crown ”accreditation].

In US: non-governmental, peer-review and usually a volunteer process.In US: non-governmental, peer-review and usually a volunteer process.In Europe: since adoption of the Bologna Process regime it is mostlyIn Europe: since adoption of the Bologna Process regime it is mostlycompulsorycompulsory and growingly and growingly internationalizedinternationalized – a creation of the European – a creation of the EuropeanQuality Assurance Register (EQAR).Quality Assurance Register (EQAR).

5

BenchmarkingBenchmarking

[[internalinternal organizational process which aims to improve the organizational process which aims to improve the organization’s performance by learning about possible organization’s performance by learning about possible improvements of its primary and/or support processes by looking improvements of its primary and/or support processes by looking at these processes in other, better-performing organizations] at these processes in other, better-performing organizations] It is It is a standard/reference point reflecting the best practice ina standard/reference point reflecting the best practice in

a given domain (ESMU). a given domain (ESMU).

It can be established externally or internally in order to:It can be established externally or internally in order to:- serve as a diagnostic instrument to understand the process;serve as a diagnostic instrument to understand the process;- provide comparison with competition in order improve position provide comparison with competition in order improve position

your institution;your institution;- facilitate learning from other members of the “benchmarking club” facilitate learning from other members of the “benchmarking club”

[learning from peers];[learning from peers];- influence setting up of the system-wide standards. influence setting up of the system-wide standards.

It is heavily relying on indicators. It does not produce ranking. It is heavily relying on indicators. It does not produce ranking.

6

RankingsRankings

“Ranking” is an evaluation approach which purpose is to assess and display a comparative standing of whole institutions or certain domains of its performance on the basis of relevant data and information collected according to a pre-established methodology and procedures.

There are two types of ranking:- One-dimensional, which goal is assessing performance of all institutions included in the ranking according to one set of indicators and identical weight attached to a given indicator [expressed as a percent of a total]. A consolidated result of such exercise is presented in ordinal form.- Multi-dimensional, which also use one set of indicators to construct an overall list of performing institutions but its methodology enable users to weight indicators by using own criteria and preferences [it could be called la carte ranking].

7

Rankings: Why it is done [and why they are popular]

to provide basis “to make informed choices” on the standing of higher to provide basis “to make informed choices” on the standing of higher education institutions for individual or group decision-making, i.e. students, education institutions for individual or group decision-making, i.e. students, parents, politicians, foundations, funding agencies, research councils, parents, politicians, foundations, funding agencies, research councils, employers, international organizations; employers, international organizations;

to foster healthy competition among HE institutions [can be a useful tool to foster healthy competition among HE institutions [can be a useful tool against low quality HE institutions and “degree mills”];against low quality HE institutions and “degree mills”];

to stimulate an emergence of centres of excellence; to stimulate an emergence of centres of excellence; to provide additional rationale for allocation of funds;to provide additional rationale for allocation of funds; rankings are convenient tool for marketing and public relations.ankings are convenient tool for marketing and public relations.

Rankings in a way are speaking directly to wider audience over the head of Rankings in a way are speaking directly to wider audience over the head of corporative-oriented academe [faculty and students alike]. corporative-oriented academe [faculty and students alike].

““Rankings have become an inevitable part of public life because universitiesRankings have become an inevitable part of public life because universitieshave moved to centre stage in all modern societies.” have moved to centre stage in all modern societies.” Simon Marginson, Professor of Higher Education, University of Melbourne Simon Marginson, Professor of Higher Education, University of Melbourne

8

Who does rankings:Who does rankings:

The number and type of providers of ranking is quite diversified:

governmental agencies independent professional organizations accrediting bodies funding organizations individual/group initiatives academics themselves international organizations media themselves.

9

Criticisms of rankingsCriticisms of rankings

Questionable methodologies [with regard to weighting scales] and Questionable methodologies [with regard to weighting scales] and validity of data [especially when collected from surveys];validity of data [especially when collected from surveys];

Biased towards the research productivity and encourages Biased towards the research productivity and encourages convergence towards a research-dominated model of HE convergence towards a research-dominated model of HE institutions and reduces the system diversity [academic drift];institutions and reduces the system diversity [academic drift];

Introduces hierarchies among HE institutions and members of the Introduces hierarchies among HE institutions and members of the academic community [not in line with an ethos of “academic academic community [not in line with an ethos of “academic corporatism”];corporatism”];

““Come off as well as possible in a ranking” can become an Come off as well as possible in a ranking” can become an obsession adversely affecting governance and administration; obsession adversely affecting governance and administration;

Hinders open-to-everyone academic cooperation and encourages Hinders open-to-everyone academic cooperation and encourages cluster mentality;cluster mentality;

Tend to focus on certain academic fields and not comprehensive Tend to focus on certain academic fields and not comprehensive performance of entire institution.performance of entire institution.

10

““University rankings smarten up”University rankings smarten up”

Recognize the diversity of institutions and take the different Recognize the diversity of institutions and take the different missions and goals of institutions into account;missions and goals of institutions into account;

Be transparent regarding the methodology used for Be transparent regarding the methodology used for creating the rankings;creating the rankings;

Measure outcomes in preference to inputs whenever Measure outcomes in preference to inputs whenever possible;possible;

Use audited and verifiable data whenever possible;Use audited and verifiable data whenever possible; Provide consumers with a clear understanding of all of the Provide consumers with a clear understanding of all of the

factors used to develop a ranking, and offer them a choice factors used to develop a ranking, and offer them a choice in how rankings are displayed.in how rankings are displayed.

11

IREG Observatory IREG Observatory The adoption by IREG in May 2006 of the Berlin Principles - 16 principles articulate

several standards of good practice.(www.ireg-observatory.org).).

It organizes the bi-annual conferences: IREG-5 – The Academic Rankings: From Popularity to Reliability and Relevance, 6-8

October 2010, Berlin IREG-6 - Academic Rankings and Advancement of Higher Education: Lessons from

Asia and other Regions, 18 - 22 April 2012, Taipei.

There are specific initatives:- IREG Ranking Audit which purpose is to evaluate the coherence of university rankings

to basic standards and quality criteria [based on the Berlin Principles on Ranking of HE Institutions] in order to:

- enhance the transparency about rankings;- give users of rankings a tool to identify trustworthy rankings;- improve the quality of rankings.

Rankings which obtain positive evaluation will be entitled to use the quality label “IREG approved”.

Mapping of Academic Excellence – International Scientific Awards [project to be completed in 2012]. (www.ireg-observatory.org).).

12

Few advicesFew advices

In a field as error-strewn as statistical evidence of academicIn a field as error-strewn as statistical evidence of academicquality caution is always wise. It needs to be repeatedly saidquality caution is always wise. It needs to be repeatedly saidthat ranking can only be a that ranking can only be a proxyproxy reflection of the complex reflection of the complexwork of higher education enterprise. work of higher education enterprise. Therefore:Therefore:

- do not subject the mission of the university or even more so- do not subject the mission of the university or even more sothat of higher education system to “tyranny of seeking status”;that of higher education system to “tyranny of seeking status”;- there is also a possibility of “great scholars and not so great- there is also a possibility of “great scholars and not so greatuniversities” try to find them;universities” try to find them;- it is irresponsible to take any personal or institutional solely - it is irresponsible to take any personal or institutional solely based on rankings;based on rankings;

Institutions and their leaders need to come to terms with newInstitutions and their leaders need to come to terms with newlandscape of HE in which competition is more evident than before butlandscape of HE in which competition is more evident than before butif they are to compete [nationally or internationally] be if they are to compete [nationally or internationally] be inspired butinspired butdo not be captivateddo not be captivated by their position in rankings. by their position in rankings.