This article was downloaded by: [University of Huddersfield]On: 29 March 2015, At: 02:43Publisher: RoutledgeInforma Ltd Registered in England and Wales Registered Number: 1072954 Registeredoffice: Mortimer House, 37-41 Mortimer Street, London W1T 3JH, UK

Social Work Education: TheInternational JournalPublication details, including instructions for authors andsubscription information:http://www.tandfonline.com/loi/cswe20

Measuring the Quality of Peer-ReviewedPublications in Social Work: ImpactFactors—Liberation or Liability?Eric Blyth a , Steven M. Shardlow b , Helen Masson a , Karen Lyons c

, Ian Shaw d & Sue White ea University of Huddersfield , UKb University of Salford , UKc London Metropolitan University , UKd University of York , UKe University of Lancaster , UKPublished online: 02 Jun 2009.

To cite this article: Eric Blyth , Steven M. Shardlow , Helen Masson , Karen Lyons , Ian Shaw &Sue White (2010) Measuring the Quality of Peer-Reviewed Publications in Social Work: ImpactFactors—Liberation or Liability?, Social Work Education: The International Journal, 29:2, 120-136,DOI: 10.1080/02615470902856705

To link to this article: http://dx.doi.org/10.1080/02615470902856705

PLEASE SCROLL DOWN FOR ARTICLE

Taylor & Francis makes every effort to ensure the accuracy of all the information (the“Content”) contained in the publications on our platform. However, Taylor & Francis,our agents, and our licensors make no representations or warranties whatsoever as tothe accuracy, completeness, or suitability for any purpose of the Content. Any opinionsand views expressed in this publication are the opinions and views of the authors,and are not the views of or endorsed by Taylor & Francis. The accuracy of the Contentshould not be relied upon and should be independently verified with primary sourcesof information. Taylor and Francis shall not be liable for any losses, actions, claims,proceedings, demands, costs, expenses, damages, and other liabilities whatsoever orhowsoever caused arising directly or indirectly in connection with, in relation to or arisingout of the use of the Content.

This article may be used for research, teaching, and private study purposes. Anysubstantial or systematic reproduction, redistribution, reselling, loan, sub-licensing,systematic supply, or distribution in any form to anyone is expressly forbidden. Terms &

Conditions of access and use can be found at http://www.tandfonline.com/page/terms-and-conditions

Dow

nloa

ded

by [

Uni

vers

ity o

f H

udde

rsfi

eld]

at 0

2:43

29

Mar

ch 2

015

MeasuringtheQualityofPeer-ReviewedPublications in Social Work: ImpactFactors—Liberation or Liability?Eric Blyth, Steven M. Shardlow, Helen Masson,Karen Lyons, Ian Shaw & Sue White

Systems for measuring the quality of publications in peer-reviewed academic journals have

achieved importance in the ‘audit culture’ to which academia worldwide has becomeincreasingly subjected. In the United Kingdom this debate has focused on government

proposals to give greater emphasis to bibliometrics (counts of journal articles and theircitations) as a measurement of research quality, in respect of publications in the emergentResearch Excellence Framework (REF) which is set to replace the Research Assessment

Exercise (RAE). This approach impacts on social work educators who are the mainproducers of papers published in peer-reviewed academic journals. It affects their publishing

behaviour by pressurising them to publish their work in journals that are regarded as beingprestigious, for which ‘high impact factor’ journals as determined by Thomson Reuters—a

private commercial information management enterprise with headquarters in the UnitedStates—has become a proxy for quality. In this paper the authors describe and critique the

Thomson Reuters system as it applies to social work and propose an alternate fair, inclusiveand transparent system for assessing the quality of publications based on peer evaluationand incorporating an ethical approach consistent with the discipline’s professional values.

Keywords: Knowledge; Knowledge Transfer; Profession; International; Research

Using the impact factor alone to judge a journal is like using weight alone to judge a

person’s health. (Joint Committee on Quantitative Assessment of Research, 2008, p. 2)

ISSN 0261-5479 print/1470-1227 online q 2010 Taylor & FrancisDOI: 10.1080/02615470902856705

Eric Blyth & Helen Masson, University of Huddersfield, UK; Steven M. Shardlow, University of Salford, UK; Karen Lyons,

London Metropolitan University, UK; Ian Shaw, University of York, UK & Sue White, University of Lancaster, UK.

Correspondence to: Professor Eric Blyth, School of Human and Health Sciences, University of Huddersfield, HHS

Research Building, Huddersfield HD1 3DH, UK. Email: e.d.blyth@hud.ac.uk

Social Work EducationVol. 29, No. 2, March 2010, pp. 120–136

Dow

nloa

ded

by [

Uni

vers

ity o

f H

udde

rsfi

eld]

at 0

2:43

29

Mar

ch 2

015

Introduction

Measurement by agencies and councils, on behalf of government, of the performance of

universities and their staff has become an increasingly common feature of academic lifein the United Kingdom since the 1980s, both in relation to teaching and to research.Teaching quality has been measured by different quality regimes, including: Subject

Review; Teaching Quality Assessment and most recently Institutional Audit, which is apart of the Quality Assurance Framework that was established in 2002. Increased

emphasis by universities has been placed on the results of the annual National StudentSurvey (introduced in 2005) which has become a proxy for teaching quality. Increasingly,

social work educators are expected to generate, through research and other scholarlywork, the knowledge on which the discipline is grounded and teaching depends, and the

quality of this too, is subject to formal assessment. The volume and quality of researchoutput has been measured by the several Research Assessment Exercises (RAE) conductedin 1986, 1989, 1992, 1996, 2001 and, most recently, 2008. Various authors have analysed

these exercises from a social work perspective (see, for example, Lyons and Orme, 1998;Fisher and Marsh, 2003; McLaughlin et al., 2007). Equivalent forms of performance

measurement are evident in other countries (for example, Hong Kong employs an auditsimilar to the UK RAE). In the UK, these performance measurement indicators have

been used to drive the allocation of research income to universities; those that performwell in the RAE being allocated significantly more than those that do not.

While it is desirable, in a broad sense, that universities are accountable to the statefor the ways in which public funds are spent, it is a matter of growing concern that the

development and transfer of knowledge is becoming subject to levels of control thatpotentially reward existing and conventional research but that may inhibit innovationor the development of new research areas. There is some modest anecdotal evidence,

for example in Hong Kong, of an increasing tendency in universities to prescribe wherestaff should publish the results of their research, while in some institutions in Norway

a financial reward is given for ‘appropriate’ publication.This paper provides an analysis and critique of the hitherto dominant model of

assessing the quality of published academic research—and an exploration of theimplications for social work research and research publication. The article then

considers an alternative approach to assessment of the quality of research publicationsthat is fair, inclusive and transparent, drawing on the value base of social work.

How Research Quality (Including Social Work Research) Has Been Assessed

The current structure for determining which journals, in social work, as in otherdisciplines, comprise the ‘gold standard’ of quality assessment, relies upon the Journal

Impact Factor. (See below for details of how the impact factor is derived.) This wasinitially devised by Eugene Garfield (1955), to replace the existing subject-specificindexing system of data management that was perceived as failing to meet the needs of

the scientific research community. Garfield utilised emerging automation andinformation technology to provide what he claimed to be a cost-effective, timely and

Social Work Education 121

Dow

nloa

ded

by [

Uni

vers

ity o

f H

udde

rsfi

eld]

at 0

2:43

29

Mar

ch 2

015

efficient means of indexing and retrieving an ever-increasing volume of research-based

information published in peer-reviewed scientific and medical journals. Twoimportant, if overly simplistic, assumptions behind Garfield’s approach are, first,

that the principal (or sole) ‘users’ of scientific information are those who cite it inother academic journals, and second, that when author B cites the work of author A,

this is a valid/useful proxy measure of the ‘impact’ of author A’s work on authorB. Consequently, the higher the frequency of citations for any particular work, the

greater is its presumed impact and value.Garfield established the Institute for Scientific Information (ISI) which began to

publish the Science Citation Index (SCI) in the 1960s to provide his indexing services

on a commercial basis. ISI subsequently became Thomson Scientific and is nowThomson Reuters (an international US-based publisher) that describes itself as ‘the

world’s leading source of intelligent information for businesses and professionals’.1

Given the sheer volume of academic peer-reviewed publications that has developed

since Garfield’s early work, the task was made (more?) manageable by establishing adatabase of a sample of (‘index’) journals only. Currently, the Thomson Reuters’

database comprises around 9,000 ‘index’ journals across more than 150 scientific,social science, arts and humanities disciplines, representing between 10% and 12% of

all published journals—a selection that is based on the so-called ‘Bradford’s Law’.2

Journal evaluation and selection for inclusion in—or removal from—the index is anongoing process, based on Thomson Reuters’ in-house editorial staff3 assessment of

the basic publishing standards of a journal. These include: timeliness of publication;adherence to international editorial conventions; inclusion of bibliographic

information (i.e. title, abstract, keywords—in English, and application of a peerreview process); editorial content; the international diversity of its authorship,

editorship and editorial board membership; and citation data relating both to thejournal itself and the citation record of contributing authors. As regards this last

criterion, the system is self-reinforcing (Testa, n.d.).A journal’s impact factor measures the average number of citations to published

papers in the two years post-publication, by other subsequent papers in indexed journals.

Citations from other sources are not included. The impact factor is calculated by dividingthe number of citations made in the current year by the total number of articles published

in the journal during the two previous years. An impact factor of 1 means that, on average,the articles published one or two years previously had been cited once. The resultant

statistic then provides the basis for the annual ranking of journals and is published in theJournal Citation Reports (JCR) databases.4 Table 1 lists the 29 journals indexed by

Thomson Reuters in the ‘Social Work’ category5 and their impact factor for 2007.Thomson Reuters’ claim for use of the impact factor is that it:

Enables a variety of information professionals to access and assess key journal data:

. Librarians—can manage and maintain journal collections and budget for subscriptions

. Journal Use Reports, the new analytical tool used in conjunction with JCR, provideslibrarians and administrators with a better way to measure journal usage—with acomplete picture of journal performance, use and research activity at their institution

122 E. Blyth et al.

Dow

nloa

ded

by [

Uni

vers

ity o

f H

udde

rsfi

eld]

at 0

2:43

29

Mar

ch 2

015

. Publishers—can monitor their competitors, identify new publishing opportunities,and make decisions regarding current publications

. Editors—can assess the effectiveness of editorial policies and objectives and trackingthe standing of their journals

. Authors—can identify journals in which to publish, confirm the status of journals inwhich they have published, and identify journals relevant to their research

. Information Analysts—can track bibliometric trends, study the sociology of scholarlyand technical publications, and study citation patterns within and between disciplines.(See http://www.thomsonreuters.com/products_services/scientific/Journal_Citation_Reports)

Problems with the Thomson Reuters System and the Implications for Social Work

The discipline of social work is by no means unique in the nature of difficulty

experienced with the Thomson Reuters system as a proxy for measuring the qualityof research output for the discipline using the impact factor of selected journals.

Table 1 Social Work Index Journals 2007

Title of journal Impact factor

Child Maltreatment 2.352Trauma Violence & Abuse 1.806Child Abuse & Neglect 1.506Social Work Research 1.2Journal of Social Policy 1.177Social Work 1.086American Journal of Community Psychology 1Research on Social Work Practice 0.957Health & Social Care in the Community 0.945Journal of Community Psychology 0.935Social Service Review 0.917Children and Youth Services Review 0.908Family Relations 0.871British Journal of Social Work 0.718Health & Social Work 0.694International Social Work 0.407Journal of Social Work Education 0.403Social Work in Health Care 0.392Child Welfare 0.379Journal of Social Work Practice 0.349International Journal of Social Welfare 0.324Families in Society—The Journal of Contemporary Social Services 0.273Affilia—Journal of Women and Social Work 0.241Administration in Social Work 0.225Journal of Social Service Research 0.164Clinical Social Work Journal 0.158Smith College Studies in Social Work 0.068Asia Pacific Journal of Social Work and Development 0Indian Journal of Social Work 0

Source: Thomson Reuters.

Social Work Education 123

Dow

nloa

ded

by [

Uni

vers

ity o

f H

udde

rsfi

eld]

at 0

2:43

29

Mar

ch 2

015

There are a series of problematic assumptions that underlie the Thomson Reuters

system, which are examined in turn below. In fairness, a number of these difficultieshave been previously identified by Garfield (1994); however, while Thomson Reuters

acknowledge that in some subjects ‘it may take a relatively long time for an article toattract a meaningful number of citations’ (Testa, n.d.), this is totally bypassed by its use

of the two year period post-publication as the ‘one size fits all’ timeframe.

Assumption 1: It is Possible to Identify ‘Core’ Journals for the Discipline of Social Work:

the Index Journals Used by Thomson Reuters Comprise the ‘Core’ Journals in which theMost Significant Social Work Research is Published

As an applied discipline that draws upon a wide range of theoretical material originallyderived from other disciplines (particularly psychology and sociology), which

integrates these theories into applied professional practice, material about social workmay be published in a wider and more expansive range of journals. At a conceptual

level, therefore, it is difficult to identify the core journals in social work. The 29-longjournal list currently identified by Thomson Reuters (Table 1) is insufficiently

extensive, given the breadth of social work as a discipline. Perhaps a more realisticlisting is that produced by Leung and Cheung (2008) in Table 2 which identifies 183

journals—and even this is not claimed to be exhaustive by the authors.To determine which of these journals is ‘core’ and to obtain agreement across the

social work academe would be highly problematic. Such a judgement is unlikely to

turn upon the impact factors of these journals, were such figures available, as thejournals in the Thomson Reuters listing (Table 1) have low impact factors by

comparison with journals in other disciplines. Only five of these journals have animpact factor greater than one; while the last two listed have an impact factor of zero.

There is, in all probability, a financial issue here as each journal included in theThomson Reuters list must increase the costs of producing and maintaining the list of

articles included in the journals and the respective journal rankings for any subject.Hence, there may be little incentive for a commercial organisation, such as Thomson

Reuters, to be responsive to academic demands to widen the listings—assuming thatthis would give a more representative and valid mechanism to capture the nature andextent of publication.

It can be shown that the Thomson Reuters listing (Table 1) ignores much goodresearch because of the 10–12% sample bias mentioned earlier. According to

Universities UK (2007, p. 20) ‘a significant number of citations to published materialare not indexed’ by Thomson Reuters. UUK cites selected data showing levels of non-

indexed publications ranging from 20% in biological sciences to 41% in mechanicalengineering (p. 21).

In the report regarding social work and the 2001 RAE (Evidence/ESRC, 2004),evidence was produced to demonstrate that for social work units graded 5/5* (thehighest ranking) less than half of all journal articles submitted for assessment (43%)

were in Thomson Reuters indexed journals (Table 3). When all social work journalarticles submitted for assessment were analysed, the figure fell to 36%, indicating that

124 E. Blyth et al.

Dow

nloa

ded

by [

Uni

vers

ity o

f H

udde

rsfi

eld]

at 0

2:43

29

Mar

ch 2

015

Table 2 Journals in Social Work and Related Disciplines

Child WelfareAdoption QuarterlyAPSAC Advisor (American Professional Society on the Abuse of Children)Child & Adolescent Social Work JournalChild & Youth ServicesChild Abuse & Neglect: The International JournalChild Abuse ReviewChild MaltreatmentChild WelfareChildren & Schools (previously titled Social Work in Education)Children and Youth Services ReviewContemporary Issues in Early Childhood (E-Journal)Family Preservation JournalJournal of AdolescenceJournal of Adolescent ResearchJournal of Aggression, Maltreatment & TraumaJournal of Applied School PsychologyJournal of Child & Adolescent Substance AbuseJournal of Child & Adolescent TraumaJournal of Child CustodyJournal of Child PsychotherapyJournal of Child Sexual AbuseJournal of Children & PovertyJournal of Early AdolescenceJournal of Emotional AbuseJournal of HIV/AIDS Prevention in Children & YouthJournal of Public Child WelfareJournal of School ViolenceJournal of Sudden Infant Death Syndrome and Infant MortalityJournal of the American Academy of Child and Adolescent PsychiatryJournal of Youth and AdolescenceJournal of Youth StudiesRelational Child & Youth Care Practice (previously titled Journal of Child and Youth Care)Residential Treatment for Children and YouthSchool Social Work JournalTrauma, Violence, & AbuseViolence & Victims

Community PracticeAmerican Journal of Community PsychologyHealth and Social Care in the CommunityJournal of Community PracticeJournal of Community PsychologyJournal of Developing SocietiesJournal of Prevention & Intervention in the Community (previously titled Prevention in HumanServices)Journal of Progressive Human ServicesJournal of Workplace Behavioral Health

FamiliesChild and Family Behavior TherapyChild and Family Social WorkFamilies in SocietyFamily Court Review

(continued)

Social Work Education 125

Dow

nloa

ded

by [

Uni

vers

ity o

f H

udde

rsfi

eld]

at 0

2:43

29

Mar

ch 2

015

Table 2 – Continued

Family Journal: Counseling and Therapy for Couples and FamiliesFamily RelationsJournal of Child and Family StudiesJournal of Family IssuesJournal of Family Social Work (previously titled Journal of Social Work & Human Sexuality)Journal of Family ViolenceJournal of Feminist Family TherapyMarriage & Family Review

GerontologyAgeing and SocietyClinical GerontologistEducational Gerontology: An International JournalGerontologistGerontology and Geriatrics EducationInternational Journal of Aging and Human DevelopmentJournal of Aging & Social PolicyJournal of Aging and HealthJournal of Aging StudiesJournal of Applied GerontologyJournal of Elder Abuse and NeglectJournal of Gerontological Social WorkJournal of Gerontology: Social SciencesJournal of Intergenerational RelationshipsJournal of Social Work in End-of-Life & Palliative CareJournal of Women & AgingJournals of Gerontology: Social SciencesResearch on Aging

Health Care in Social WorkHealth & Social WorkHealth AffairsJournal of Aging and HealthJournal of Health & Social PolicySocial Work in Health Care

International and Multicultural FocusAsia Pacific Journal of Social Work and Development (previously titled Asia Pacific Journal of SocialWork)Asian Journal of Social PolicyAsian Journal of Social PsychologyAustralian Social WorkBritish Journal of Social WorkCanadian Journal of Behavioral ScienceChina Journal of Social WorkEuropean Journal of Social WorkGlobal Social PolicyHong Kong Journal of Social WorkIndian Journal of Social Work, TheInternational Journal of Aging and Human DevelopmentInternational Journal of Social WelfareInternational Social WorkJournal of Asian StudiesJournal of Black StudiesJournal of Blacks in Higher Education

(continued)

126 E. Blyth et al.

Dow

nloa

ded

by [

Uni

vers

ity o

f H

udde

rsfi

eld]

at 0

2:43

29

Mar

ch 2

015

Table 2 – Continued

Journal of Comparative Social Welfare (previously titled New Global Development)Journal of Ethnic & Cultural Diversity in Social Work (previously titled Multicultural Social Work)Journal of European Social PolicyJournal of Immigrant and Refugee Studies (previously titled Journal of Immigrant and RefugeeServices)Journal of Social Development in AfricaSocial Development Issues

Mental HealthAdministration and Policy in Mental Health and Mental Health Services ResearchAmerican Journal of OrthopsychiatryBest Practices in Mental Health: An International JournalClinical Social Work JournalJournal of Applied Behavioral ScienceJournal of Brief TherapyJournal of Consulting and Clinical Psychology (JCCP)Journal of Counseling PsychologyJournal of Emotional and Behavioral Disorders (JEBD)Social Work in Mental Health

PolicyJournal of Health & Social PolicyJournal of Policy Practice (previously titled Social Policy Journal)Journal of Poverty: Innovations on Social, Political & Economic InequalitiesJournal of Social PolicyJournal of Societal & Social Policy

ResearchJournal of Applied Social PsychologyJournal of Applied Social Science StudiesJournal of Social Service ResearchJournal of Social Work Research & Evaluation (currently not active)Psychiatric Rehabilitation Journal (previously titled Psychosocial Rehabilitation Journal andInnovations & Research)Research on Social Work PracticeSocial Science InformationSocial Science ResearchSocial Work ResearchSociological Methods & Research

Social Work Practice (General)Administration in Social WorkArete, 7International Journal of Social WelfareJournal for Specialists in Group WorkJournal of Community PracticeJournal of Evidence-Based Social WorkJournal of Human Behavior in the Social EnvironmentJournal of Religion & Spirituality in Social WorkSocial ThoughtJournal of Social WorkJournal of Social Work PracticeJournal of Social Work Values and EthicsJournal of Sociology and Social WelfareReflections: Narratives of Professional Helping

(continued)

Social Work Education 127

Dow

nloa

ded

by [

Uni

vers

ity o

f H

udde

rsfi

eld]

at 0

2:43

29

Mar

ch 2

015

only slightly more than one third of the work that was selected for the 2001 RAE by the

social work academe in the UK as representative of the very best research is publishedin Thomson Reuters’ index journals.

Evidence from the United States indicates that the Thomson Reuters index journalsare not the most highly ranked by academics. A national survey by Sellers et al. (2006)of 556 social work faculty in the United States, among other things, sought

Table 2 – Continued

Smith College Studies in Social WorkSocial Service ReviewSocial WorkSocial Work with Groups

Social Work Teaching and Field EducationClinical SupervisorJournal of Baccalaureate Social WorkJournal of Social Work EducationJournal of Social Work Values and EthicsJournal of Teaching in Social WorkSocial Work Education

Students JournalsPerspectives on Social Work (a journal for doctoral students)Social Work Perspectives (students’ submissions)

Substance AbuseAddictionAddictive BehaviorsAlcohol & AlcoholismAmerican Journal of Drug and Alcohol AbuseAmerican Journal on Addictions, TheJournal of Child & Adolescent Substance AbuseJournal of Groups in Addiction & RecoveryJournal of Social Work Practice in the Addictions

Technology and Social WorkComputers in Human BehaviorJournal of Technology in Human Services (previously titled Computers in Human Services)

Women and Men’s IssuesAffilia: Journal of Women and Social WorkJournal of Couple & Relationship TherapyJournal of Divorce & RemarriageJournal of Emotional AbuseJournal of Gay & Lesbian Issues in EducationJournal of Gay & Lesbian Politics (has been discontinued)Journal of Gay & Lesbian Social ServicesJournal of GLBT Family StudiesJournal of HIV/AIDS & Social ServicesJournal of HomosexualityJournal of Interpersonal ViolenceJournal of Marriage and FamilySigns: Journal of Women in Culture and SocietyViolence Against WomenWomen and HealthWomen in Sport and Physical Activity Journal

Source: Leung and Cheung (2008).

128 E. Blyth et al.

Dow

nloa

ded

by [

Uni

vers

ity o

f H

udde

rsfi

eld]

at 0

2:43

29

Mar

ch 2

015

respondents’ perceptions of the overall quality of social work journals. Participantswere given a list of 38 journals devised by the authors and were:

(1) asked if they were unfamiliar with any of journals; and(2) given an opportunity to identify up to five other journals not included in the

list.

According to Sellers et al. (2006), 35% of respondents cited at least one additional

journal, although these varied considerably and none received many mentions.The two most frequently mentioned were: (1) American Journal of Orthopsychiatry

(n ¼ 14; 2.5%) and (2) Reflections (n ¼ 8; 1.4%).Given that this is a study of American social work faculty and the inherent US bias

of the Thomson Reuters index, it is interesting to note from Table 4, firstly, the relatively

infrequent citations to any of the journals and, secondly, the fact that the two top-ratedjournals in the Thomson Reuters social work category 2007 (Child Maltreatment and

Trauma Violence and Abuse—39 and 40 in Table 4) were not cited at all by respondents.Sellers et al. (2006) also cite three other US studies that highlight the percentage

of social work academics who publish their research in what are perceived to benon-social work journals—ranging from nearly 30% (Meinert, 1993) through 40%

(Green and Secret, 1996), to 50% or more (Schiele, 1991; Green et al., 2002).Sellers et al. (2006) speculate that these findings may reflect either the

multi-disciplinary nature of social work and social work research, and/or authors’dissatisfaction with social work journals. However, none of these studies—includingthat of Sellers et al. (2006)—has explored the reasons for this particular phenomenon.

Clearly, however, it has potentially significant implications for achieving a meaningfulassessment of social work research.

Assumption 2: Articles Published in the ‘Core’ Journals have the Most Impact

The impact of social work research cannot be measured solely in terms of papers in

refereed academic journals. Given the nature of social work and social work research,this is not only a partial measure of research impact but fails to capture more

Table 3 Nature of Materials Submitted to RAE 2001

Numberof items

Numberof

journalarticles

Numberof ISI

articlesmapped

Articlesas proportion

of articlessubmitted

ISI mappedarticles

as proportionof items

submitted

ISI mappedarticles as

proportion ofarticles

submitted

Social workunits graded4/5/5

853 462 200 0.54 0.23 0.43

Social workall units

1,650 887 323 0.54 0.20 0.36

Source: Evidence/ESRC (2004).

Social Work Education 129

Dow

nloa

ded

by [

Uni

vers

ity o

f H

udde

rsfi

eld]

at 0

2:43

29

Mar

ch 2

015

Table

4So

cial

Wo

rkF

acu

lty

inth

eU

SP

erce

pti

on

so

fJo

urn

alR

anki

ngs

[Ove

rall

jou

rnal

qu

alit

yan

dn

um

ber

of

resp

on

den

tsra

tin

gth

ejo

urn

al(N

B:

tota

ln

um

ber

of

resp

on

den

ts¼

556)

]

Tit

leN

MSD

TR

ran

kin

gT

Rim

pac

tfa

cto

r

1SocialService

Review

360

5.99

0.93

110.

917

2SocialWorkResearch

286

5.52

1.08

41.

23

Journalof

SocialServicesResearch

272

5.37

1.08

250.

164

4ChildAbuse

andNeglect

181

5.35

1.09

31.

506

5Researchon

SocialWorkPractice

258

5.35

1.09

80.

957

6Journalof

SocialPolicy

845.33

1.04

51.

177

7AmericanJournalof

Com

munityPsychology

725.22

1.22

71

8ChildrenandYouth

ServicesReview

145

5.18

1.13

120.

908

9Familiesin

Society

300

5.07

1.23

220.

273

10Journalof

Com

munityPsychology

815.06

1.12

100.

935

11FamilyRelation

s96

5.04

1.13

130.

871

12So

cial

Wor

kan

dSo

cial

Scie

nce

Rev

iew

415.

031.

3413

Internation

alJournalof

SocialWelfare

585.02

1.46

210.

324

14ChildWelfare

235

4.95

1.21

190.

379

15British

Journalof

SocialWork

894.92

1.15

140.

718

16SocialWork

464

4.9

1.34

61.

086

17So

cial

Wor

kw

ith

Gro

ups

127

4.9

1.34

18Administration

inSocialWork

154

4.87

1.11

240.

225

19HealthandSocialWork

226

4.85

1.18

150.

694

20Jo

urn

alof

Soci

olog

yan

dSo

cia

lW

elfa

re19

04.

821.

321

SocialWorkin

HealthCare

139

4.79

1.2

180.

392

22HealthandSocialCare

intheCom

munity

254.77

1.27

90.

945

23Internation

alSocialWork

694.75

1.42

160.

407

24Jo

urn

alof

Ger

onto

logi

cal

Soci

al

Wor

k12

44.

731.

2725

Soci

alW

ork

inE

du

cati

on14

24.

711.

3226

Journalof

SocialWorkEducation

398

4.69

1.32

170.

403

27Jo

urn

alof

Hea

lth

Car

efo

rth

eP

oor

and

Un

der

serv

ed19

4.68

1.78

28Journalof

SocialWorkPractice

624.65

1.29

200.

349

29Affilia—

Journalof

Wom

enin

SocialWork

226

4.61

1.43

230.

241

130 E. Blyth et al.

Dow

nloa

ded

by [

Uni

vers

ity o

f H

udde

rsfi

eld]

at 0

2:43

29

Mar

ch 2

015

30C

hil

dan

dA

dol

esce

nt

Soci

alW

ork

Jou

rnal

120

4.56

1.36

31Sm

ithCollege

Studiesin

SocialWork

142

4.56

1.62

270.

068

32ClinicalSocialWorkJournal

149

4.51

1.56

260.

158

33Jo

urn

alof

Tea

chin

gin

Soci

al

Wor

k18

94.

341.

4134

Cri

sis

Inte

rven

tion

and

Tim

e-L

imit

edT

reat

men

t46

4.33

1.51

35IndianJournalof

SocialWork

264.19

1.52

285

036

Jou

rnal

ofSo

cia

lD

istr

ess

and

the

Hom

eles

s12

41.

9537

AsiaPacificJournalof

SocialWork

163.79

1.97

285

038

Jou

rnal

ofIn

dep

end

ent

Soci

alW

ork

303.

331.

6739

Ch

ild

Mal

trea

tmen

t *–

––

12.

352

40T

rau

ma

Vio

len

cean

dA

buse

*–

––

21.

806

Not

es:

Bo

ld¼

cite

din

Th

om

son

Reu

ters

soci

alw

ork

cate

gory

2007

.*¼

jou

rnal

sin

Th

om

son

Reu

ters

soci

alw

ork

cate

gory

2007

no

tci

ted

by

Sell

ers

etal

.(2

006)

.So

urc

e:Se

ller

set

al.

(200

6).

Social Work Education 131

Dow

nloa

ded

by [

Uni

vers

ity o

f H

udde

rsfi

eld]

at 0

2:43

29

Mar

ch 2

015

important aspects of the impact of social work research, such as its impact on service

users, communities, social work practice, agency and government policy, andlegislation. Journals with a high impact factor may actually have limited impact within

the social work community as a whole. This may be true of many applied andprofessional disciplines where impact upon the professional community and the service

user community may be very different to the impact in academe. As a paper publishedby JUC/SWEC comments:

citation metrics . . . place a premium on impact within the academy when mostsocial work researchers are (and arguably should be) at least as concerned withimpact beyond the academy. (JUC/SWEC, 2008)

The importance of publications, other than those in academic journals, was acceptedby the specific panel that reviewed the quality of social work research for the 2008 RAE

in the UK: a decision was taken by the panel not to privilege publication in anyparticular type of output (Higher Education Funding Council for England et al.,

2006). Thus, in theory at any rate, all outputs were taken to be of equal value(a position that merits argument elsewhere): the quality of a particular piece was ratedand not the output in which it appeared. Such a position was summed up in the

following statement by Universities UK:

key outputs [particularly in applied and policy-related areas] were less likely to be inthe form of journal articles, bibliometric indicators were of less significance,technically difficult to produce and less likely to be acceptable to researchers as ameasure of quality. (Universities UK, 2007 p. 15)

Assumption 3: High Citation Counts are Equivalent to Impact

The Thomson Reuters system is based on an explicit assumption that citation

represents ‘the influence or impact of the idea and its originator on our body ofknowledge’ (Thomson Reuters, n.d.). While a high citation count may have a closer

correlation with impact in certain disciplines than in others, the Joint Committee onQuantitative Assessment of Research (2008) commented that this assumption is

founded on a very limited perception both of ‘user’ of research and of ‘impact’—especially for applied disciplines such as social work. As Garfield (1994) commented:‘Informed and careful use of these impact data is essential. Users may be tempted to

jump to ill-formed conclusions based on impact factor statistics unless several caveatsare considered’.

There can be no automatic assumption that citation is equivalent to either approvalor value. Citation per se may, for example, indicate that a paper (a) is

controversial/bad; or (b) that other authors refute its contents: ‘citation indices donot recognise that a source may be cited either in acclaim or in criticism’ (JUC/SWEC,

2008). Similarly there can be no automatic assumption that citation means theoriginal paper has even been read by the person citing it! Simkin and Roychowdhury(2003) report a method for estimating what percentage of those who cited a paper had

actually read it and conclude that about 20% only of citers read the original. Lawrence(2007) reviewed the 48 citations of a paper of which he was co-author (Casal et al., 2002)

132 E. Blyth et al.

Dow

nloa

ded

by [

Uni

vers

ity o

f H

udde

rsfi

eld]

at 0

2:43

29

Mar

ch 2

015

and claims that of these, only eight were appropriate, three were ‘plain wrong’ and 37

‘incidental’ (in the sense that a more appropriate alternative citation could or shouldhave been used).

The time lag between manuscript submission and, if accepted, journal publicationcan also adversely affect impact factor. Among social work journals, up to two years

from first submission is not unusual. If the time between submission and publicationexceeds two years, less than a year is available for the paper to be cited by anyone else

for this citation to ‘count’ in compiling statistics for the index. So the model isill-suited to disciplines with a relatively slow publication pattern.

Assessing the Quality of Social Work Research—AWay Forward

We do not oppose the appropriate use of bibliometric systems, as proposed by

government for the next round of research quality assessment, but argue that theymust be fit for purpose and should not be dominant. Instead, peer review should be

the major component of determining the quality of scholarly publication. A similarposition was articulated by Gill (2009, p. 5) who reported that a substantial number of

official RAE 2008 Panel Reports have called for the retention of a significantcomponent of peer review as part of future assessments of research quality. As Taylor-

Gooby (2008) (chair of the social work and social policy and administration panel forRAE 2008) commented:

Peer review should predominate in research assessment because it is best fitted tocommand the range of sources, methods, activities and demands in the field, andidentify significance and originality in a rapidly changing context. The most usefulsupportive metrics concern research student completions and research income,taking into account the wide range of relevant sources of funding. Citation countsare difficult to apply, since it is hard to define a unitary established researchcommunity against which to normalise them. Attempts to do so may weaken thecapacity to recognise innovation and damage the UK’s leading international statusin work in this area.

Bibliometrics about scholarly publication could be added to his list, given some keycaveats. Firstly, any system for the determination of quality in the academic worldshould a priori have the attributes of being fair (be held to be fair and reasonable by the

academic community that is in part or fully governed by any such system), inclusive(the system should enable academics to be part of the process that makes judgements

about their work; such process should not apply only to a small elite) and transparent(mechanisms for the determination of priorities and the processes for such

determination should be available for scrutiny by the academic community—and awider public). In particular, for an applied discipline such as social work, user

engagement is an essential element of research assessment. However, there seems littleevidence in what has emerged to date regarding the bibliometrics piloting exercisebeing conducted by HEFCE that the increasing ‘academisation’ inherent in REF will

facilitate user engagement as understood within the social work context (HEFCE,2008a). In a further twist, the company originally commissioned by HEFCE

Social Work Education 133

Dow

nloa

ded

by [

Uni

vers

ity o

f H

udde

rsfi

eld]

at 0

2:43

29

Mar

ch 2

015

to undertake the pilot exercise, Evidence Ltd, has subsequently been acquired by

Thomson Reuters (HEFCE, 2008b)—from which we assume that there will be evenless incentive to challenge the status quo as the only conceivable model for assessment.

In the case of social work (and no doubt other disciplines) a panel could beconstructed that would determine the range of recognised discipline-specific journals:

that is, those journals that relate particularly to a specific discipline and where thescholarly community might expect to find the majority of debate and discussion

within the discipline to be conducted. This presents social work with particularproblems as the discipline is broadly grounded and relates to other academicdisciplines, whilst practice is inter-professional. Hence, material relevant to social

work is likely to be found in a range of journals, certainly within a wider range than the29 publications that comprise the current Thomson Reuters social work index.

We would expect that if a list of discipline-specific journals were made by such a panelit would be more extensive then this listing and it might also question whether

journals such as the American Journal of Community Psychology should be termedsocial work journals and whether the predominance of US based journals is desirable

for either the social work communities in countries other than the US or the globalsocial work community more widely.

Criteria that might be used in developing the list include:

(1) journal scope that relates directly to social work in all its forms;(2) journals having an ISSN number;(3) journals that employ a double blind peer review system, with at least two

independent reviews per paper;(4) clear and unambiguous journal guidance about the nature of judgements to be

made by peer reviewers;(5) the publication of a list of those who review for the journal;(6) journals at least into the third year of publication, as a demonstration of viability

(in the early years of publication new journals are often extensively reliant oncommissioned papers);

(7) periodic sampling to agree that the quality of papers published is of anappropriate scholarly standard, judged by the quality of the argument ratherthan the nature of the content.

Using such criteria it would be possible to judge that articles published within

the body of recognised discipline-specific journals met a core rigorous standard forthe discipline. With such an agreed corpus of work, other bibliometrics might bedeveloped, including the volume of publications for the discipline and for the

academic unit—citation indices whose function would be illustrative and informative,but not normative.

Conclusion

Proposals to employ bibliometrics in the assessment of research quality, including social

work research, appear to be based on unquestioned and naıve perceptions of the rigourand objectivity of the Thomson Reuters system. This paper has sought to draw social work

134 E. Blyth et al.

Dow

nloa

ded

by [

Uni

vers

ity o

f H

udde

rsfi

eld]

at 0

2:43

29

Mar

ch 2

015

educators’ attention to some of the most important flaws in this system and to argue that,

anyway, a range of assessment criteria should be used in future assessment of social workand other research quality. Bibliometrics may have a place in these but we would also

argue that work should be conducted to produce an alternative system which, in line withsocial work values, is fair, inclusive and transparent.

Notes

[1] For details see: http://www.thomsonreuters.com/about/.

[2] In the 1930s, S. C. Bradford claimed that, for any given discipline, the bulk of significant papers

were published in a very small number of ‘core’ journals (Bradford, 1934).

[3] The Thomson Reuters website claims that ‘editors performing journal evaluations have

educational backgrounds relevant to their areas of responsibility as well as experience andeducation in information science’. It is not clear if the social work credentials of staff dealingwith social work include professional qualification or education within the discipline (Testa, n.d.).

[4] The system is still variously described as the Institute of Scientific Information Citation Indices,

ISI Citation Indices or the Web of Science database.

[5] According to the scope notes for the social work category: Social Work covers resources

concerned with homelessness, social casework, social services, social work education, publicwelfare, family counselling, child welfare and abuse, social work administration, social workwith groups, and gerontological social work (http://scientific.thomsonreuters.com/mjl/scope/scope_ssci/). This is self-evidently a restricted definition of the discipline that will, of necessity,narrow the range of journals considered eligible for designation as ‘social work’ journals.

References

Bradford, S. C. (1934) ‘Sources of information on specific subjects’, Engineering: An Illustrated Weekly

Journal, vol. 137, pp. 85–86 [reprinted as Bradford, S.C. (1985) ‘Sources of information on

specific subjects’, Journal of Information Science, vol. 10, no. 4, pp. 176–180.

Casal, J., Struhl, G. & Lawrence, P. (2002) ‘Developmental compartments and planar polarity’,

Drosphila. Current Biology, vol. 12, pp. 1189–1198.

Evidence/ESRC (2004) Bibliometric Profiles for Selected Units of Assessment [online]. Available at:

http://www.esrcsocietytoday.ac.uk/ESRCInfoCentre/Images/Bibliometric%20Profiles%

20for%20RAE%20Outputs%20in%20the%20Social%20Sciences_tcm6-18357.pdf.

Fisher, M. & Marsh, P. (2003) ‘Social work research and the 2001 Research Assessment Exercise: an

initial overview’, Social Work Education, vol. 22, no. 1, pp. 71–80.

Garfield, E. (1955) ‘Citation indexes to science: a new dimension in documentation through

association of ideas’, Science, vol. 122, no. 3159, pp. 108–111 [online]. Available at: http://

garfield.library.upenn.edu/essays/v6p468y1983.pdf.

Garfield, E. (1994) ‘The Thomson scientific impact factor’, Current Contents, 20 June [online].

Available at: http://scientific.thomsonreuters.com/free/essays/journalcitationreports/impact-

factor/.

Gill, J. (2009) ‘Keep peer input in REF, urge panels’, The Times Higher Education, no. 1878, p. 5.

Green, R., Baskind, F. & Bellin, M. (2002) ‘Results of the doctoral faculty publication project: journal

article productivity and its correlates in 1990s’, Journal of Social Work Education, vol. 38, no. 1,

pp. 135–152.

Green, R. & Secret, M. (1996) ‘Publishing by social work scholars in social work and non-social work

journals’, Social Work Research, vol. 20, no. 1, pp. 31–41.

Social Work Education 135

Dow

nloa

ded

by [

Uni

vers

ity o

f H

udde

rsfi

eld]

at 0

2:43

29

Mar

ch 2

015

Higher Education Funding Council for England (2008a) Bibliometrics Pilot Exercise [online].Available at: http://www.hefce.ac.uk/research/ref/pilot/.

Higher Education Funding Council for England (2008b) Evidence Ltd and the Bibliometrics Pilot[online]. Available at: http://www.hefce.ac.uk/research/ref/pilot/evidence/.

Higher Education Funding Council for England, Scottish Higher Education Funding Council,Higher Education Funding Council for Wales & Department for Employment and LearningNorthern Ireland (2006) RAE 2008: Panel Criteria and Working Methods: Panel J, Ref RAE01/2006 (J).

Joint Committee on Quantitative Assessment of Research (2008) Citation Statistics: A Report from theInternational Mathematical Union (IMU) in Cooperation with the International Council ofIndustrial and Applied Mathematics (ICIAM) and the Institute of Mathematical Statistics(IMS), 12 June [online]. Available at: http://www.mathunion.org/fileadmin/IMU/Report/CitationStatistics.pdf.

Joint Universities Council/Social Work Education Committee (2008) Response to HEFCE 2007/34,the Research Excellence Framework Consultation [online]. Available at: http://jucuk.files.wordpress.com/2008/05/jucswec-refresponse0208.pdf.

Lawrence, P. (2007) ‘The mismeasurement of science’, Current Biology, vol. 17, no. 15,pp. R583–R585.

Leung, P. & Cheung, M. (2008) ‘Journals in social work and related disciplines: manuscriptsubmission information’, 5 May 2008 revision [online]. Available at: http://www.sw.uh.edu/documents/cwep/ManuscriptSubmissionInformation.pdf, accessed 5 January 2009.

Lyons, K. & Orme, J. (1998) ‘The 1996 Research Assessment Exercise and the response of social workacademics’, British Journal of Social Work, vol. 28, no. 5, pp. 783–792.

McLaughlin, H., Lawson, J. & Shardlow, S. M. (2007) ‘Comparing the 1996 and 2001 researchselectivity exercises in respect of social work’, Social Work Education, vol. 26, no. 2, pp. 1–12.

Meinert, R. (1993) ‘Should there be a moratorium on articles that rank schools of social work basedon faculty publications?’, Journal of Social Work Education, vol. 29, no. 3, pp. 247–251.

Schiele, J. (1991) ‘Publication productivity of African-American social work faculty’, Journal of SocialWork Education, vol. 40, no. 1, pp. 143–160.

Sellers, S., Mathieson, S., Smith, T. & Perry, R. (2006) ‘Perceptions of professional social workjournals; findings from a national survey’, Journal of Social Work Education, vol. 42, no. 1,pp. 139–160.

Simkin, M. & Roychowdhury, V. (2003) ‘Read before you cite!’, Complex Systems, vol. 14, no. 3,pp. 269–274.

Taylor-Gooby, P. (2008) In Z. Corbyn, ‘Keep peer review at REF core, chairs warn’, Times HigherEducation, 28 August.

Testa, J. (n.d.) The Thomson Scientific Journal Selection Process [online]. Available at: http://www.thomsonreuters.com/business_units/scientific/free/essays/journalselection/.

Thomson, Reuters (n.d.) The History of Citation Indexing [online]. Available at: http://www.thomsonreuters.com/business_units/scientific/free/essays/history/.

Universities UK (2007) The Use of Bibliometrics to Measure Research Quality in UK Higher EducationInstitutions, Universities UK, London [online]. Available at: http://www.universitiesuk.ac.uk/Publications/Pages/Publication-275.aspx.

136 E. Blyth et al.

Dow

nloa

ded

by [

Uni

vers

ity o

f H

udde

rsfi

eld]

at 0

2:43

29

Mar

ch 2

015