Why web-basedrankings ofuniversities shouldnot be neglected

Isidro F. Aguillo

World University Rankings SymposiumIdentifying Excellence and Diversity in International Education: Rankings and BeyondÉcole Normale Supérieure (ENS), Paris. May 18th 2011

2

A rigorous scientific approach to the ranking of universities

Scholars making scientific research– Researchers belonging to the National Research Council (CSIC)

• The largest Spanish research public organization

– Recognised by our peers• 15 years experience in quantitative analysis and evaluation of scholar

communication and academic institutions

• Papers in referred scientific journals, contributions to internationalconferences, reports to governmental bodies

– Funded by public resources• International cooperation projects funded by European Commission

Research Agenda– Promote Open Access initiatives– Global coverage, including universities from developing countries– Building Cybermetrics/Webometrics as an emerging discipline

The Cybermetrics Lab

3

A new powerful tool with a significant impact on Higher Education

The problems of the Higher Education Science in the ‘90s– Limited impact, not suitable for students/scholars

• “Experts” driven

• Complex reports, difficult to understand

– Lack of global scenarios• Focus on “US research university” model

International Rankings (ARWU, 2003)– Huge impact, including governments

• Data driven

• Easy to understand, ONE composite indicator (rank)

– World and continental scenarios• Excellence clearly defined

• Useful for both individual universities and national systems

Why Global Rankings of Universities?

ACTIVITY IMPACT

How? A couple of models

4

ACTIVITY IMPACT

ARWU (Shanghai) 40% 60%

RANKING

QS 30% 70%

TIMES HE (2010) 35% 65%

HEEACT (Taiwan) 20% 80%

WR (Webometrics) 50% 50%

Customizing the modelMost of the Global Rankings use an arbitrary weighting system, withoutan underlying model

5

Feasibility

6

It is not so easy, sometimes there is no data, or they are not trusted orthey are not useful

7

Quality can be measured by consensus or majority of experts evaluatingthe targeted topic

Surveyo Topic: Perceived (subjective) academic reputationo Expertise: Peers with global knowledge (not available)o Size: Medium (several thousands)

Citation analysiso Topic: Empirical (objective) scientific impact (formal communication)o Expertise: Peerso Size: Small (a few hundreds)

Link analysiso Topic: Quality of academic and other web contentso Expertise: Web-editors (peers and non-peers)o Size: Huge (millions)

Quality

8

Surprisingly most of the Rankings are being accepted without a criticalreview of the authorship, the methodology and the results

A critical review

Ranking Missions Indicators Model Results Comments

ARWU Research Scientificexcellence

No model (Top 200)Correlated

World class universities(incomplete, mistakes, old data)

WEBOMETRICS All Activity & Impact(Web)

Impact factorbased

Correlated(relevantexceptions)

World universities(extreme penalties for bad webpolicies, search engines biases)

HEEACT Research Activity & Impact(Papers, Citations)

Striking notexplainedmodel

Correlated World research universities(variables not independent)

LEIDEN Research Impact(Citations/Papers)

Crownindicator

Several“rankings”

World research universities(questioned methodology)

SCIMAGO Research Activity OR Impact(Papers, Citations)

Ranked bysize (!) asdefault

Several“rankings”

World research institutions (nocomposite ranking)

THE All Activities, Prestige(Surveys) &Impact (Citations)

No model Not valid Methodology applied incorrectly,unethical (non-cooperatinguniversities are not identified)

CHEPS All Multi-Rank Customizableby end-users!

No Ranking A collection of colorful smileys

Power-law Distribution

9

SCORE

RANK

WR

QS

CWTS

ARWUHEEACT

log-norm

z-score

• Published since 2004, two editions per year(January & July)• The largest and most updated Directorywith more than 20,000 Higher EducationInstitutions from all over the World• The Ranking provides the Top 12,000universities according to web indicators• The hypothesis is that in the 21st centurythe web reflects the organization, activities,research results, knowledge transfer,prestige, and international visibility of theuniversities• If the web performance is below theexpected position could be due to lack ofcommitment to the electronic publication, orbad web practices

Webometrics RankingThe Ranking Web is one of the first World Rankings to be published andcurrently it is the one with the largest coverage

10

3484336

15435

5943

6176148

339259

3975

5069414

19266(Top 1000)

WRJan 2011

Geographical coverage

11

12

Digital Divide

67115

5069(Top 500)

WRJan 2011

Europe

13

58148

43319411

66330

0

23648

23378

1116

16114

16213

20348

11114

10710

10010

958

757

779

676

6410

601

5712

5016

508

519

480

483

632

352

351

31 33

8

334

322

151

91

370

260

14

Universities in France

15

Different views are correlated but the position of French universities, alsoin the Web ranking, is below expectations

Coverageo Rankings based on research performance where the English is the “lingua

franca” of scholar communication are penalizing French languageo Webometrics takes into consideration also other university missions

including traditional and off campus (distance) teaching in French or transferof knowledge and technology, also in local language

University roleso Research-based rankings describe “World-class” universitieso Web ranking also identify “nation-building” institutions, probably more

important to the local communities (social commitment) and governments(economic and political involvement)

Bad practiceso Surprisingly for the 21st century, there are academic institutions (its leaders or

scholars) that are ignoring the Web: No enough contentso Web policies are inadequate, incorrect or insufficient. Too many institutions,

frequent web domain changes, too few repositories

Comparative Analysis

16

French and Norwegian (and others) Higher Education Ministries visitedShanghai for advice!!. Suggestions provided here are tentative:

Rankings should be useful

ARWUoriented

Choose 2-3 universities as World-classcandidates

Increase their budget at the expense of theother universities

Give legal capability for offering bettersalaries and grants

Attract international & national bestcandidates to these universities

Reinforce research capabilities, transferringelite institutes and hospitals

Fire low performers & close not veryproductive or international disciplines(Humanities!)

Webometricsoriented

All the universities should reinforce theirweb presence

Budget should be available to webpublication for …

* Reflecting the institution’s performance* Attracting international students &scholars

* Disseminating the research results* Giving more visibility to “neglected”disciplines (Humanities!)

* Improve the transfer of technology toindustry and other companies

* Increase the knowledge transfer and thecommunity engagement

17

Web should be the showcase of the University, the place for publishingits basic info & facts, organisation, activities, policies, results …

Contentso First mission: Traditional and off campus (distance) teaching supporting

pageso Second mission: Research groups and (scientists) personal pages, Research

projects websiteso Third mission: Hosting third parties sites

Quality Contentso Portal of journals / Repository of paperso Media (videos, webinars, objects) and software portalso Data repository / (Web) Archiveo Social commitments

International Contentso Multilingual (also English) versionso Guides to foreign students/scholarso Super-sites

Web strategies

Learning management systems (LMS)

MIT Open CourseWare

19

Personal Pages

ENS (?) Repository

21

http://hal-ens.archives-ouvertes.fr/

Why not?http://archiveouverte.ens.fr/

Thesis at USP

22

Web 2.0: Priorities

Rank Facebook YouTube SocialBookmarking

External Blogs InstitutionalBlogs

1 Campus Life Events Courses Faculty Campus Life

2 Sports Campus Life Projects, Non-Research

Research, PhysicalSciences Events

3 Technology Faculty Research, PhysicalSciences Institution Overall Institution Overall

4 Product Services Courses Events Expert Commentary Institution Sub-Groups

5 Events InstitutionOverall Faculty Events Admissions

24

Why the Web?o Web presence is an indicator. Web is reflecting overall performance and not

only online activitieso Web is already the most important scholarly communication toolo Web increases significantly the visibility and impact of the activities and the

results of scholars and researcherso Web is the best channel for online and distance learning, the best showcase

for attracting international talent (foreign students and professors), themeeting point with the society, the economic and industrial stakeholders

Why the Ranking Web?o Webometrics is offering the larger coverage, ranking World-class universities

but also institutions from emerging and developing countrieso Discrepancies on the web ranking informs about problems with the

governance and long-term strategies of the universities, obsolete orinadequate policies and bad web practices

o US universities leadership in Webometrics is showing a concerning and along term threat to the scientific and cultural presence of the Europeans (andtheir languages) in the Web

Summarizing

25

Questions? …Thank you!

Contact:Isidro F. Aguillo, HcPhDThe Cybermetrics Lab. IPP-CCHS-CSICisidro.aguillo@cchs.csic.es

Open Forum