Google Scholar as a cybermetric tool Alastair G SmithVictoria University of Wellington New Zealand

alastair.smith@vuw.ac.nz

Overview

Google Scholar as a cybermetric toolUsed to compare web citation rates with results of a research assessment exercise: New Zealand’s Performance Based Research Fund (PBRF).

Cybermetrics

Quantitative analysis of published research on the InternetEmulates techniques used by bibliometrics in the conventional publishing environment

Tools for cybermetric work

General purpose search engines, e.g. Alta Vista Google

Specialised cybermetric crawler, e.g. Wolverhampton Academic Web Link

Database (http://cybermetrics.wlv.ac.uk/database)

General purpose search engines

Search algorithms not transparentPriority is to provide satisfactory search, not exact resultCover all of web, not just research material

Specialised cybermetric crawler

Search algorithm publicly documentedOriented to cybermetric purposesFocussed on specific research institutions

Google Scholar

Indexes research oriented material on the Internet (though selection criteria not published)Includes citation informationCitations include print material referenced in Internet documentsPotential as cybermetric tool?

PBRF research assessment

Research assessment exercise: NZ Tertiary institutions 2003/4Assessed research outputs from staff Institution ratings include: Total output Quality score: average output per staff member

Ratings determine grants from Performance Based Research FundBenchmark to compare with cybermetric study?

Specialised crawler study

Previous study (Smith & Thelwall 2005) Compared links to NZ university web sites with PBRF rankings.Moderate correlation: Link count/staff cf PBRF quality score

Issues: NZ-NZ links, ignores international linkage Includes non-research material

Google Scholar study

Searched Google Scholar for research originating from the 8 NZ universitiesCitation counts for this material extractedCompared with PBRF ratings

Methodology issues with Google Scholar

Problems identifying output of institutionNo total citation count provided

Identifying institution

No “institution” field Domain name search produces false drops (e.g. mirrored research ), misses research hosted at other institutions (e.g. paper presented at external conference)Searched on words in name of institution, and location e.g. "canterbury university" OR "university of canterbury" zealand OR christchurch OR ilam

Google Scholar result screen

Obtaining citation count

Total citation count not displayedOnly first 1000 hits obtainable (but citation counts appear to be used in ranking)Extracted individual citation counts with macroAssumed citation counts for first 1000 hits a good approximation for total citation count.

Results: Citations vs PBRF output

0

1000

2000

3000

4000

5000

6000

7000

0 10000 20000 30000 40000 50000

Google scholar citations

To

tal P

BR

F o

utp

ut

Results: Hits vs PBRF output

0

1000

2000

3000

4000

5000

6000

0 10,000 20,000 30,000 40,000 50,000 60,000

Google Scholar hits

To

tal

PB

RF

ou

tpu

t

Comparison of cybermetric tools

  Coverage Identification of institutions

Citation count

Transparency

Google Scholar Research on Web

By keyword Individual, cannot display all hits

Little documentation of algorithm, selection of sources

Web of Knowledge

Core journals (some digital)

Specific field For individual items

Sources documented

Scopus Core journals + Web sites (from Scirus)

Specific field “Citation tracker” only for authors

Sources documented

Wolverhampton Crawler

Specific university web sites

By domain Link counts

Sources documented

Pros and Cons of using Google Scholar for cybermetrics

Pros: Good coverage of research on Web Accessible and simple

Cons Not transparent Need to use ad hoc search for

institution

Institutional repositories

Could provide good target for specialised cybermetric crawler, if: Standardised formats for citations etc Standard selection criteria Accessible to crawlers

Summary

Google Scholar provides a useful tool for “quick and dirty” cybermetric workResults have some correlation with a general research assessmentInstitutional repositories may provide a useful way of evaluating institutions