View
0
Download
0
Category
Preview:
Citation preview
This article was downloaded by: [University of Huddersfield]On: 29 March 2015, At: 02:43Publisher: RoutledgeInforma Ltd Registered in England and Wales Registered Number: 1072954 Registeredoffice: Mortimer House, 37-41 Mortimer Street, London W1T 3JH, UK
Social Work Education: TheInternational JournalPublication details, including instructions for authors andsubscription information:http://www.tandfonline.com/loi/cswe20
Measuring the Quality of Peer-ReviewedPublications in Social Work: ImpactFactors—Liberation or Liability?Eric Blyth a , Steven M. Shardlow b , Helen Masson a , Karen Lyons c
, Ian Shaw d & Sue White ea University of Huddersfield , UKb University of Salford , UKc London Metropolitan University , UKd University of York , UKe University of Lancaster , UKPublished online: 02 Jun 2009.
To cite this article: Eric Blyth , Steven M. Shardlow , Helen Masson , Karen Lyons , Ian Shaw &Sue White (2010) Measuring the Quality of Peer-Reviewed Publications in Social Work: ImpactFactors—Liberation or Liability?, Social Work Education: The International Journal, 29:2, 120-136,DOI: 10.1080/02615470902856705
To link to this article: http://dx.doi.org/10.1080/02615470902856705
PLEASE SCROLL DOWN FOR ARTICLE
Taylor & Francis makes every effort to ensure the accuracy of all the information (the“Content”) contained in the publications on our platform. However, Taylor & Francis,our agents, and our licensors make no representations or warranties whatsoever as tothe accuracy, completeness, or suitability for any purpose of the Content. Any opinionsand views expressed in this publication are the opinions and views of the authors,and are not the views of or endorsed by Taylor & Francis. The accuracy of the Contentshould not be relied upon and should be independently verified with primary sourcesof information. Taylor and Francis shall not be liable for any losses, actions, claims,proceedings, demands, costs, expenses, damages, and other liabilities whatsoever orhowsoever caused arising directly or indirectly in connection with, in relation to or arisingout of the use of the Content.
This article may be used for research, teaching, and private study purposes. Anysubstantial or systematic reproduction, redistribution, reselling, loan, sub-licensing,systematic supply, or distribution in any form to anyone is expressly forbidden. Terms &
Conditions of access and use can be found at http://www.tandfonline.com/page/terms-and-conditions
Dow
nloa
ded
by [
Uni
vers
ity o
f H
udde
rsfi
eld]
at 0
2:43
29
Mar
ch 2
015
MeasuringtheQualityofPeer-ReviewedPublications in Social Work: ImpactFactors—Liberation or Liability?Eric Blyth, Steven M. Shardlow, Helen Masson,Karen Lyons, Ian Shaw & Sue White
Systems for measuring the quality of publications in peer-reviewed academic journals have
achieved importance in the ‘audit culture’ to which academia worldwide has becomeincreasingly subjected. In the United Kingdom this debate has focused on government
proposals to give greater emphasis to bibliometrics (counts of journal articles and theircitations) as a measurement of research quality, in respect of publications in the emergentResearch Excellence Framework (REF) which is set to replace the Research Assessment
Exercise (RAE). This approach impacts on social work educators who are the mainproducers of papers published in peer-reviewed academic journals. It affects their publishing
behaviour by pressurising them to publish their work in journals that are regarded as beingprestigious, for which ‘high impact factor’ journals as determined by Thomson Reuters—a
private commercial information management enterprise with headquarters in the UnitedStates—has become a proxy for quality. In this paper the authors describe and critique the
Thomson Reuters system as it applies to social work and propose an alternate fair, inclusiveand transparent system for assessing the quality of publications based on peer evaluationand incorporating an ethical approach consistent with the discipline’s professional values.
Keywords: Knowledge; Knowledge Transfer; Profession; International; Research
Using the impact factor alone to judge a journal is like using weight alone to judge a
person’s health. (Joint Committee on Quantitative Assessment of Research, 2008, p. 2)
ISSN 0261-5479 print/1470-1227 online q 2010 Taylor & FrancisDOI: 10.1080/02615470902856705
Eric Blyth & Helen Masson, University of Huddersfield, UK; Steven M. Shardlow, University of Salford, UK; Karen Lyons,
London Metropolitan University, UK; Ian Shaw, University of York, UK & Sue White, University of Lancaster, UK.
Correspondence to: Professor Eric Blyth, School of Human and Health Sciences, University of Huddersfield, HHS
Research Building, Huddersfield HD1 3DH, UK. Email: e.d.blyth@hud.ac.uk
Social Work EducationVol. 29, No. 2, March 2010, pp. 120–136
Dow
nloa
ded
by [
Uni
vers
ity o
f H
udde
rsfi
eld]
at 0
2:43
29
Mar
ch 2
015
Introduction
Measurement by agencies and councils, on behalf of government, of the performance of
universities and their staff has become an increasingly common feature of academic lifein the United Kingdom since the 1980s, both in relation to teaching and to research.Teaching quality has been measured by different quality regimes, including: Subject
Review; Teaching Quality Assessment and most recently Institutional Audit, which is apart of the Quality Assurance Framework that was established in 2002. Increased
emphasis by universities has been placed on the results of the annual National StudentSurvey (introduced in 2005) which has become a proxy for teaching quality. Increasingly,
social work educators are expected to generate, through research and other scholarlywork, the knowledge on which the discipline is grounded and teaching depends, and the
quality of this too, is subject to formal assessment. The volume and quality of researchoutput has been measured by the several Research Assessment Exercises (RAE) conductedin 1986, 1989, 1992, 1996, 2001 and, most recently, 2008. Various authors have analysed
these exercises from a social work perspective (see, for example, Lyons and Orme, 1998;Fisher and Marsh, 2003; McLaughlin et al., 2007). Equivalent forms of performance
measurement are evident in other countries (for example, Hong Kong employs an auditsimilar to the UK RAE). In the UK, these performance measurement indicators have
been used to drive the allocation of research income to universities; those that performwell in the RAE being allocated significantly more than those that do not.
While it is desirable, in a broad sense, that universities are accountable to the statefor the ways in which public funds are spent, it is a matter of growing concern that the
development and transfer of knowledge is becoming subject to levels of control thatpotentially reward existing and conventional research but that may inhibit innovationor the development of new research areas. There is some modest anecdotal evidence,
for example in Hong Kong, of an increasing tendency in universities to prescribe wherestaff should publish the results of their research, while in some institutions in Norway
a financial reward is given for ‘appropriate’ publication.This paper provides an analysis and critique of the hitherto dominant model of
assessing the quality of published academic research—and an exploration of theimplications for social work research and research publication. The article then
considers an alternative approach to assessment of the quality of research publicationsthat is fair, inclusive and transparent, drawing on the value base of social work.
How Research Quality (Including Social Work Research) Has Been Assessed
The current structure for determining which journals, in social work, as in otherdisciplines, comprise the ‘gold standard’ of quality assessment, relies upon the Journal
Impact Factor. (See below for details of how the impact factor is derived.) This wasinitially devised by Eugene Garfield (1955), to replace the existing subject-specificindexing system of data management that was perceived as failing to meet the needs of
the scientific research community. Garfield utilised emerging automation andinformation technology to provide what he claimed to be a cost-effective, timely and
Social Work Education 121
Dow
nloa
ded
by [
Uni
vers
ity o
f H
udde
rsfi
eld]
at 0
2:43
29
Mar
ch 2
015
efficient means of indexing and retrieving an ever-increasing volume of research-based
information published in peer-reviewed scientific and medical journals. Twoimportant, if overly simplistic, assumptions behind Garfield’s approach are, first,
that the principal (or sole) ‘users’ of scientific information are those who cite it inother academic journals, and second, that when author B cites the work of author A,
this is a valid/useful proxy measure of the ‘impact’ of author A’s work on authorB. Consequently, the higher the frequency of citations for any particular work, the
greater is its presumed impact and value.Garfield established the Institute for Scientific Information (ISI) which began to
publish the Science Citation Index (SCI) in the 1960s to provide his indexing services
on a commercial basis. ISI subsequently became Thomson Scientific and is nowThomson Reuters (an international US-based publisher) that describes itself as ‘the
world’s leading source of intelligent information for businesses and professionals’.1
Given the sheer volume of academic peer-reviewed publications that has developed
since Garfield’s early work, the task was made (more?) manageable by establishing adatabase of a sample of (‘index’) journals only. Currently, the Thomson Reuters’
database comprises around 9,000 ‘index’ journals across more than 150 scientific,social science, arts and humanities disciplines, representing between 10% and 12% of
all published journals—a selection that is based on the so-called ‘Bradford’s Law’.2
Journal evaluation and selection for inclusion in—or removal from—the index is anongoing process, based on Thomson Reuters’ in-house editorial staff3 assessment of
the basic publishing standards of a journal. These include: timeliness of publication;adherence to international editorial conventions; inclusion of bibliographic
information (i.e. title, abstract, keywords—in English, and application of a peerreview process); editorial content; the international diversity of its authorship,
editorship and editorial board membership; and citation data relating both to thejournal itself and the citation record of contributing authors. As regards this last
criterion, the system is self-reinforcing (Testa, n.d.).A journal’s impact factor measures the average number of citations to published
papers in the two years post-publication, by other subsequent papers in indexed journals.
Citations from other sources are not included. The impact factor is calculated by dividingthe number of citations made in the current year by the total number of articles published
in the journal during the two previous years. An impact factor of 1 means that, on average,the articles published one or two years previously had been cited once. The resultant
statistic then provides the basis for the annual ranking of journals and is published in theJournal Citation Reports (JCR) databases.4 Table 1 lists the 29 journals indexed by
Thomson Reuters in the ‘Social Work’ category5 and their impact factor for 2007.Thomson Reuters’ claim for use of the impact factor is that it:
Enables a variety of information professionals to access and assess key journal data:
. Librarians—can manage and maintain journal collections and budget for subscriptions
. Journal Use Reports, the new analytical tool used in conjunction with JCR, provideslibrarians and administrators with a better way to measure journal usage—with acomplete picture of journal performance, use and research activity at their institution
122 E. Blyth et al.
Dow
nloa
ded
by [
Uni
vers
ity o
f H
udde
rsfi
eld]
at 0
2:43
29
Mar
ch 2
015
. Publishers—can monitor their competitors, identify new publishing opportunities,and make decisions regarding current publications
. Editors—can assess the effectiveness of editorial policies and objectives and trackingthe standing of their journals
. Authors—can identify journals in which to publish, confirm the status of journals inwhich they have published, and identify journals relevant to their research
. Information Analysts—can track bibliometric trends, study the sociology of scholarlyand technical publications, and study citation patterns within and between disciplines.(See http://www.thomsonreuters.com/products_services/scientific/Journal_Citation_Reports)
Problems with the Thomson Reuters System and the Implications for Social Work
The discipline of social work is by no means unique in the nature of difficulty
experienced with the Thomson Reuters system as a proxy for measuring the qualityof research output for the discipline using the impact factor of selected journals.
Table 1 Social Work Index Journals 2007
Title of journal Impact factor
Child Maltreatment 2.352Trauma Violence & Abuse 1.806Child Abuse & Neglect 1.506Social Work Research 1.2Journal of Social Policy 1.177Social Work 1.086American Journal of Community Psychology 1Research on Social Work Practice 0.957Health & Social Care in the Community 0.945Journal of Community Psychology 0.935Social Service Review 0.917Children and Youth Services Review 0.908Family Relations 0.871British Journal of Social Work 0.718Health & Social Work 0.694International Social Work 0.407Journal of Social Work Education 0.403Social Work in Health Care 0.392Child Welfare 0.379Journal of Social Work Practice 0.349International Journal of Social Welfare 0.324Families in Society—The Journal of Contemporary Social Services 0.273Affilia—Journal of Women and Social Work 0.241Administration in Social Work 0.225Journal of Social Service Research 0.164Clinical Social Work Journal 0.158Smith College Studies in Social Work 0.068Asia Pacific Journal of Social Work and Development 0Indian Journal of Social Work 0
Source: Thomson Reuters.
Social Work Education 123
Dow
nloa
ded
by [
Uni
vers
ity o
f H
udde
rsfi
eld]
at 0
2:43
29
Mar
ch 2
015
There are a series of problematic assumptions that underlie the Thomson Reuters
system, which are examined in turn below. In fairness, a number of these difficultieshave been previously identified by Garfield (1994); however, while Thomson Reuters
acknowledge that in some subjects ‘it may take a relatively long time for an article toattract a meaningful number of citations’ (Testa, n.d.), this is totally bypassed by its use
of the two year period post-publication as the ‘one size fits all’ timeframe.
Assumption 1: It is Possible to Identify ‘Core’ Journals for the Discipline of Social Work:
the Index Journals Used by Thomson Reuters Comprise the ‘Core’ Journals in which theMost Significant Social Work Research is Published
As an applied discipline that draws upon a wide range of theoretical material originallyderived from other disciplines (particularly psychology and sociology), which
integrates these theories into applied professional practice, material about social workmay be published in a wider and more expansive range of journals. At a conceptual
level, therefore, it is difficult to identify the core journals in social work. The 29-longjournal list currently identified by Thomson Reuters (Table 1) is insufficiently
extensive, given the breadth of social work as a discipline. Perhaps a more realisticlisting is that produced by Leung and Cheung (2008) in Table 2 which identifies 183
journals—and even this is not claimed to be exhaustive by the authors.To determine which of these journals is ‘core’ and to obtain agreement across the
social work academe would be highly problematic. Such a judgement is unlikely to
turn upon the impact factors of these journals, were such figures available, as thejournals in the Thomson Reuters listing (Table 1) have low impact factors by
comparison with journals in other disciplines. Only five of these journals have animpact factor greater than one; while the last two listed have an impact factor of zero.
There is, in all probability, a financial issue here as each journal included in theThomson Reuters list must increase the costs of producing and maintaining the list of
articles included in the journals and the respective journal rankings for any subject.Hence, there may be little incentive for a commercial organisation, such as Thomson
Reuters, to be responsive to academic demands to widen the listings—assuming thatthis would give a more representative and valid mechanism to capture the nature andextent of publication.
It can be shown that the Thomson Reuters listing (Table 1) ignores much goodresearch because of the 10–12% sample bias mentioned earlier. According to
Universities UK (2007, p. 20) ‘a significant number of citations to published materialare not indexed’ by Thomson Reuters. UUK cites selected data showing levels of non-
indexed publications ranging from 20% in biological sciences to 41% in mechanicalengineering (p. 21).
In the report regarding social work and the 2001 RAE (Evidence/ESRC, 2004),evidence was produced to demonstrate that for social work units graded 5/5* (thehighest ranking) less than half of all journal articles submitted for assessment (43%)
were in Thomson Reuters indexed journals (Table 3). When all social work journalarticles submitted for assessment were analysed, the figure fell to 36%, indicating that
124 E. Blyth et al.
Dow
nloa
ded
by [
Uni
vers
ity o
f H
udde
rsfi
eld]
at 0
2:43
29
Mar
ch 2
015
Table 2 Journals in Social Work and Related Disciplines
Child WelfareAdoption QuarterlyAPSAC Advisor (American Professional Society on the Abuse of Children)Child & Adolescent Social Work JournalChild & Youth ServicesChild Abuse & Neglect: The International JournalChild Abuse ReviewChild MaltreatmentChild WelfareChildren & Schools (previously titled Social Work in Education)Children and Youth Services ReviewContemporary Issues in Early Childhood (E-Journal)Family Preservation JournalJournal of AdolescenceJournal of Adolescent ResearchJournal of Aggression, Maltreatment & TraumaJournal of Applied School PsychologyJournal of Child & Adolescent Substance AbuseJournal of Child & Adolescent TraumaJournal of Child CustodyJournal of Child PsychotherapyJournal of Child Sexual AbuseJournal of Children & PovertyJournal of Early AdolescenceJournal of Emotional AbuseJournal of HIV/AIDS Prevention in Children & YouthJournal of Public Child WelfareJournal of School ViolenceJournal of Sudden Infant Death Syndrome and Infant MortalityJournal of the American Academy of Child and Adolescent PsychiatryJournal of Youth and AdolescenceJournal of Youth StudiesRelational Child & Youth Care Practice (previously titled Journal of Child and Youth Care)Residential Treatment for Children and YouthSchool Social Work JournalTrauma, Violence, & AbuseViolence & Victims
Community PracticeAmerican Journal of Community PsychologyHealth and Social Care in the CommunityJournal of Community PracticeJournal of Community PsychologyJournal of Developing SocietiesJournal of Prevention & Intervention in the Community (previously titled Prevention in HumanServices)Journal of Progressive Human ServicesJournal of Workplace Behavioral Health
FamiliesChild and Family Behavior TherapyChild and Family Social WorkFamilies in SocietyFamily Court Review
(continued)
Social Work Education 125
Dow
nloa
ded
by [
Uni
vers
ity o
f H
udde
rsfi
eld]
at 0
2:43
29
Mar
ch 2
015
Table 2 – Continued
Family Journal: Counseling and Therapy for Couples and FamiliesFamily RelationsJournal of Child and Family StudiesJournal of Family IssuesJournal of Family Social Work (previously titled Journal of Social Work & Human Sexuality)Journal of Family ViolenceJournal of Feminist Family TherapyMarriage & Family Review
GerontologyAgeing and SocietyClinical GerontologistEducational Gerontology: An International JournalGerontologistGerontology and Geriatrics EducationInternational Journal of Aging and Human DevelopmentJournal of Aging & Social PolicyJournal of Aging and HealthJournal of Aging StudiesJournal of Applied GerontologyJournal of Elder Abuse and NeglectJournal of Gerontological Social WorkJournal of Gerontology: Social SciencesJournal of Intergenerational RelationshipsJournal of Social Work in End-of-Life & Palliative CareJournal of Women & AgingJournals of Gerontology: Social SciencesResearch on Aging
Health Care in Social WorkHealth & Social WorkHealth AffairsJournal of Aging and HealthJournal of Health & Social PolicySocial Work in Health Care
International and Multicultural FocusAsia Pacific Journal of Social Work and Development (previously titled Asia Pacific Journal of SocialWork)Asian Journal of Social PolicyAsian Journal of Social PsychologyAustralian Social WorkBritish Journal of Social WorkCanadian Journal of Behavioral ScienceChina Journal of Social WorkEuropean Journal of Social WorkGlobal Social PolicyHong Kong Journal of Social WorkIndian Journal of Social Work, TheInternational Journal of Aging and Human DevelopmentInternational Journal of Social WelfareInternational Social WorkJournal of Asian StudiesJournal of Black StudiesJournal of Blacks in Higher Education
(continued)
126 E. Blyth et al.
Dow
nloa
ded
by [
Uni
vers
ity o
f H
udde
rsfi
eld]
at 0
2:43
29
Mar
ch 2
015
Table 2 – Continued
Journal of Comparative Social Welfare (previously titled New Global Development)Journal of Ethnic & Cultural Diversity in Social Work (previously titled Multicultural Social Work)Journal of European Social PolicyJournal of Immigrant and Refugee Studies (previously titled Journal of Immigrant and RefugeeServices)Journal of Social Development in AfricaSocial Development Issues
Mental HealthAdministration and Policy in Mental Health and Mental Health Services ResearchAmerican Journal of OrthopsychiatryBest Practices in Mental Health: An International JournalClinical Social Work JournalJournal of Applied Behavioral ScienceJournal of Brief TherapyJournal of Consulting and Clinical Psychology (JCCP)Journal of Counseling PsychologyJournal of Emotional and Behavioral Disorders (JEBD)Social Work in Mental Health
PolicyJournal of Health & Social PolicyJournal of Policy Practice (previously titled Social Policy Journal)Journal of Poverty: Innovations on Social, Political & Economic InequalitiesJournal of Social PolicyJournal of Societal & Social Policy
ResearchJournal of Applied Social PsychologyJournal of Applied Social Science StudiesJournal of Social Service ResearchJournal of Social Work Research & Evaluation (currently not active)Psychiatric Rehabilitation Journal (previously titled Psychosocial Rehabilitation Journal andInnovations & Research)Research on Social Work PracticeSocial Science InformationSocial Science ResearchSocial Work ResearchSociological Methods & Research
Social Work Practice (General)Administration in Social WorkArete, 7International Journal of Social WelfareJournal for Specialists in Group WorkJournal of Community PracticeJournal of Evidence-Based Social WorkJournal of Human Behavior in the Social EnvironmentJournal of Religion & Spirituality in Social WorkSocial ThoughtJournal of Social WorkJournal of Social Work PracticeJournal of Social Work Values and EthicsJournal of Sociology and Social WelfareReflections: Narratives of Professional Helping
(continued)
Social Work Education 127
Dow
nloa
ded
by [
Uni
vers
ity o
f H
udde
rsfi
eld]
at 0
2:43
29
Mar
ch 2
015
only slightly more than one third of the work that was selected for the 2001 RAE by the
social work academe in the UK as representative of the very best research is publishedin Thomson Reuters’ index journals.
Evidence from the United States indicates that the Thomson Reuters index journalsare not the most highly ranked by academics. A national survey by Sellers et al. (2006)of 556 social work faculty in the United States, among other things, sought
Table 2 – Continued
Smith College Studies in Social WorkSocial Service ReviewSocial WorkSocial Work with Groups
Social Work Teaching and Field EducationClinical SupervisorJournal of Baccalaureate Social WorkJournal of Social Work EducationJournal of Social Work Values and EthicsJournal of Teaching in Social WorkSocial Work Education
Students JournalsPerspectives on Social Work (a journal for doctoral students)Social Work Perspectives (students’ submissions)
Substance AbuseAddictionAddictive BehaviorsAlcohol & AlcoholismAmerican Journal of Drug and Alcohol AbuseAmerican Journal on Addictions, TheJournal of Child & Adolescent Substance AbuseJournal of Groups in Addiction & RecoveryJournal of Social Work Practice in the Addictions
Technology and Social WorkComputers in Human BehaviorJournal of Technology in Human Services (previously titled Computers in Human Services)
Women and Men’s IssuesAffilia: Journal of Women and Social WorkJournal of Couple & Relationship TherapyJournal of Divorce & RemarriageJournal of Emotional AbuseJournal of Gay & Lesbian Issues in EducationJournal of Gay & Lesbian Politics (has been discontinued)Journal of Gay & Lesbian Social ServicesJournal of GLBT Family StudiesJournal of HIV/AIDS & Social ServicesJournal of HomosexualityJournal of Interpersonal ViolenceJournal of Marriage and FamilySigns: Journal of Women in Culture and SocietyViolence Against WomenWomen and HealthWomen in Sport and Physical Activity Journal
Source: Leung and Cheung (2008).
128 E. Blyth et al.
Dow
nloa
ded
by [
Uni
vers
ity o
f H
udde
rsfi
eld]
at 0
2:43
29
Mar
ch 2
015
respondents’ perceptions of the overall quality of social work journals. Participantswere given a list of 38 journals devised by the authors and were:
(1) asked if they were unfamiliar with any of journals; and(2) given an opportunity to identify up to five other journals not included in the
list.
According to Sellers et al. (2006), 35% of respondents cited at least one additional
journal, although these varied considerably and none received many mentions.The two most frequently mentioned were: (1) American Journal of Orthopsychiatry
(n ¼ 14; 2.5%) and (2) Reflections (n ¼ 8; 1.4%).Given that this is a study of American social work faculty and the inherent US bias
of the Thomson Reuters index, it is interesting to note from Table 4, firstly, the relatively
infrequent citations to any of the journals and, secondly, the fact that the two top-ratedjournals in the Thomson Reuters social work category 2007 (Child Maltreatment and
Trauma Violence and Abuse—39 and 40 in Table 4) were not cited at all by respondents.Sellers et al. (2006) also cite three other US studies that highlight the percentage
of social work academics who publish their research in what are perceived to benon-social work journals—ranging from nearly 30% (Meinert, 1993) through 40%
(Green and Secret, 1996), to 50% or more (Schiele, 1991; Green et al., 2002).Sellers et al. (2006) speculate that these findings may reflect either the
multi-disciplinary nature of social work and social work research, and/or authors’dissatisfaction with social work journals. However, none of these studies—includingthat of Sellers et al. (2006)—has explored the reasons for this particular phenomenon.
Clearly, however, it has potentially significant implications for achieving a meaningfulassessment of social work research.
Assumption 2: Articles Published in the ‘Core’ Journals have the Most Impact
The impact of social work research cannot be measured solely in terms of papers in
refereed academic journals. Given the nature of social work and social work research,this is not only a partial measure of research impact but fails to capture more
Table 3 Nature of Materials Submitted to RAE 2001
Numberof items
Numberof
journalarticles
Numberof ISI
articlesmapped
Articlesas proportion
of articlessubmitted
ISI mappedarticles
as proportionof items
submitted
ISI mappedarticles as
proportion ofarticles
submitted
Social workunits graded4/5/5
853 462 200 0.54 0.23 0.43
Social workall units
1,650 887 323 0.54 0.20 0.36
Source: Evidence/ESRC (2004).
Social Work Education 129
Dow
nloa
ded
by [
Uni
vers
ity o
f H
udde
rsfi
eld]
at 0
2:43
29
Mar
ch 2
015
Table
4So
cial
Wo
rkF
acu
lty
inth
eU
SP
erce
pti
on
so
fJo
urn
alR
anki
ngs
[Ove
rall
jou
rnal
qu
alit
yan
dn
um
ber
of
resp
on
den
tsra
tin
gth
ejo
urn
al(N
B:
tota
ln
um
ber
of
resp
on
den
ts¼
556)
]
Tit
leN
MSD
TR
ran
kin
gT
Rim
pac
tfa
cto
r
1SocialService
Review
360
5.99
0.93
110.
917
2SocialWorkResearch
286
5.52
1.08
41.
23
Journalof
SocialServicesResearch
272
5.37
1.08
250.
164
4ChildAbuse
andNeglect
181
5.35
1.09
31.
506
5Researchon
SocialWorkPractice
258
5.35
1.09
80.
957
6Journalof
SocialPolicy
845.33
1.04
51.
177
7AmericanJournalof
Com
munityPsychology
725.22
1.22
71
8ChildrenandYouth
ServicesReview
145
5.18
1.13
120.
908
9Familiesin
Society
300
5.07
1.23
220.
273
10Journalof
Com
munityPsychology
815.06
1.12
100.
935
11FamilyRelation
s96
5.04
1.13
130.
871
12So
cial
Wor
kan
dSo
cial
Scie
nce
Rev
iew
415.
031.
3413
Internation
alJournalof
SocialWelfare
585.02
1.46
210.
324
14ChildWelfare
235
4.95
1.21
190.
379
15British
Journalof
SocialWork
894.92
1.15
140.
718
16SocialWork
464
4.9
1.34
61.
086
17So
cial
Wor
kw
ith
Gro
ups
127
4.9
1.34
18Administration
inSocialWork
154
4.87
1.11
240.
225
19HealthandSocialWork
226
4.85
1.18
150.
694
20Jo
urn
alof
Soci
olog
yan
dSo
cia
lW
elfa
re19
04.
821.
321
SocialWorkin
HealthCare
139
4.79
1.2
180.
392
22HealthandSocialCare
intheCom
munity
254.77
1.27
90.
945
23Internation
alSocialWork
694.75
1.42
160.
407
24Jo
urn
alof
Ger
onto
logi
cal
Soci
al
Wor
k12
44.
731.
2725
Soci
alW
ork
inE
du
cati
on14
24.
711.
3226
Journalof
SocialWorkEducation
398
4.69
1.32
170.
403
27Jo
urn
alof
Hea
lth
Car
efo
rth
eP
oor
and
Un
der
serv
ed19
4.68
1.78
28Journalof
SocialWorkPractice
624.65
1.29
200.
349
29Affilia—
Journalof
Wom
enin
SocialWork
226
4.61
1.43
230.
241
130 E. Blyth et al.
Dow
nloa
ded
by [
Uni
vers
ity o
f H
udde
rsfi
eld]
at 0
2:43
29
Mar
ch 2
015
30C
hil
dan
dA
dol
esce
nt
Soci
alW
ork
Jou
rnal
120
4.56
1.36
31Sm
ithCollege
Studiesin
SocialWork
142
4.56
1.62
270.
068
32ClinicalSocialWorkJournal
149
4.51
1.56
260.
158
33Jo
urn
alof
Tea
chin
gin
Soci
al
Wor
k18
94.
341.
4134
Cri
sis
Inte
rven
tion
and
Tim
e-L
imit
edT
reat
men
t46
4.33
1.51
35IndianJournalof
SocialWork
264.19
1.52
285
036
Jou
rnal
ofSo
cia
lD
istr
ess
and
the
Hom
eles
s12
41.
9537
AsiaPacificJournalof
SocialWork
163.79
1.97
285
038
Jou
rnal
ofIn
dep
end
ent
Soci
alW
ork
303.
331.
6739
Ch
ild
Mal
trea
tmen
t *–
––
12.
352
40T
rau
ma
Vio
len
cean
dA
buse
*–
––
21.
806
Not
es:
Bo
ld¼
cite
din
Th
om
son
Reu
ters
soci
alw
ork
cate
gory
2007
.*¼
jou
rnal
sin
Th
om
son
Reu
ters
soci
alw
ork
cate
gory
2007
no
tci
ted
by
Sell
ers
etal
.(2
006)
.So
urc
e:Se
ller
set
al.
(200
6).
Social Work Education 131
Dow
nloa
ded
by [
Uni
vers
ity o
f H
udde
rsfi
eld]
at 0
2:43
29
Mar
ch 2
015
important aspects of the impact of social work research, such as its impact on service
users, communities, social work practice, agency and government policy, andlegislation. Journals with a high impact factor may actually have limited impact within
the social work community as a whole. This may be true of many applied andprofessional disciplines where impact upon the professional community and the service
user community may be very different to the impact in academe. As a paper publishedby JUC/SWEC comments:
citation metrics . . . place a premium on impact within the academy when mostsocial work researchers are (and arguably should be) at least as concerned withimpact beyond the academy. (JUC/SWEC, 2008)
The importance of publications, other than those in academic journals, was acceptedby the specific panel that reviewed the quality of social work research for the 2008 RAE
in the UK: a decision was taken by the panel not to privilege publication in anyparticular type of output (Higher Education Funding Council for England et al.,
2006). Thus, in theory at any rate, all outputs were taken to be of equal value(a position that merits argument elsewhere): the quality of a particular piece was ratedand not the output in which it appeared. Such a position was summed up in the
following statement by Universities UK:
key outputs [particularly in applied and policy-related areas] were less likely to be inthe form of journal articles, bibliometric indicators were of less significance,technically difficult to produce and less likely to be acceptable to researchers as ameasure of quality. (Universities UK, 2007 p. 15)
Assumption 3: High Citation Counts are Equivalent to Impact
The Thomson Reuters system is based on an explicit assumption that citation
represents ‘the influence or impact of the idea and its originator on our body ofknowledge’ (Thomson Reuters, n.d.). While a high citation count may have a closer
correlation with impact in certain disciplines than in others, the Joint Committee onQuantitative Assessment of Research (2008) commented that this assumption is
founded on a very limited perception both of ‘user’ of research and of ‘impact’—especially for applied disciplines such as social work. As Garfield (1994) commented:‘Informed and careful use of these impact data is essential. Users may be tempted to
jump to ill-formed conclusions based on impact factor statistics unless several caveatsare considered’.
There can be no automatic assumption that citation is equivalent to either approvalor value. Citation per se may, for example, indicate that a paper (a) is
controversial/bad; or (b) that other authors refute its contents: ‘citation indices donot recognise that a source may be cited either in acclaim or in criticism’ (JUC/SWEC,
2008). Similarly there can be no automatic assumption that citation means theoriginal paper has even been read by the person citing it! Simkin and Roychowdhury(2003) report a method for estimating what percentage of those who cited a paper had
actually read it and conclude that about 20% only of citers read the original. Lawrence(2007) reviewed the 48 citations of a paper of which he was co-author (Casal et al., 2002)
132 E. Blyth et al.
Dow
nloa
ded
by [
Uni
vers
ity o
f H
udde
rsfi
eld]
at 0
2:43
29
Mar
ch 2
015
and claims that of these, only eight were appropriate, three were ‘plain wrong’ and 37
‘incidental’ (in the sense that a more appropriate alternative citation could or shouldhave been used).
The time lag between manuscript submission and, if accepted, journal publicationcan also adversely affect impact factor. Among social work journals, up to two years
from first submission is not unusual. If the time between submission and publicationexceeds two years, less than a year is available for the paper to be cited by anyone else
for this citation to ‘count’ in compiling statistics for the index. So the model isill-suited to disciplines with a relatively slow publication pattern.
Assessing the Quality of Social Work Research—AWay Forward
We do not oppose the appropriate use of bibliometric systems, as proposed by
government for the next round of research quality assessment, but argue that theymust be fit for purpose and should not be dominant. Instead, peer review should be
the major component of determining the quality of scholarly publication. A similarposition was articulated by Gill (2009, p. 5) who reported that a substantial number of
official RAE 2008 Panel Reports have called for the retention of a significantcomponent of peer review as part of future assessments of research quality. As Taylor-
Gooby (2008) (chair of the social work and social policy and administration panel forRAE 2008) commented:
Peer review should predominate in research assessment because it is best fitted tocommand the range of sources, methods, activities and demands in the field, andidentify significance and originality in a rapidly changing context. The most usefulsupportive metrics concern research student completions and research income,taking into account the wide range of relevant sources of funding. Citation countsare difficult to apply, since it is hard to define a unitary established researchcommunity against which to normalise them. Attempts to do so may weaken thecapacity to recognise innovation and damage the UK’s leading international statusin work in this area.
Bibliometrics about scholarly publication could be added to his list, given some keycaveats. Firstly, any system for the determination of quality in the academic worldshould a priori have the attributes of being fair (be held to be fair and reasonable by the
academic community that is in part or fully governed by any such system), inclusive(the system should enable academics to be part of the process that makes judgements
about their work; such process should not apply only to a small elite) and transparent(mechanisms for the determination of priorities and the processes for such
determination should be available for scrutiny by the academic community—and awider public). In particular, for an applied discipline such as social work, user
engagement is an essential element of research assessment. However, there seems littleevidence in what has emerged to date regarding the bibliometrics piloting exercisebeing conducted by HEFCE that the increasing ‘academisation’ inherent in REF will
facilitate user engagement as understood within the social work context (HEFCE,2008a). In a further twist, the company originally commissioned by HEFCE
Social Work Education 133
Dow
nloa
ded
by [
Uni
vers
ity o
f H
udde
rsfi
eld]
at 0
2:43
29
Mar
ch 2
015
to undertake the pilot exercise, Evidence Ltd, has subsequently been acquired by
Thomson Reuters (HEFCE, 2008b)—from which we assume that there will be evenless incentive to challenge the status quo as the only conceivable model for assessment.
In the case of social work (and no doubt other disciplines) a panel could beconstructed that would determine the range of recognised discipline-specific journals:
that is, those journals that relate particularly to a specific discipline and where thescholarly community might expect to find the majority of debate and discussion
within the discipline to be conducted. This presents social work with particularproblems as the discipline is broadly grounded and relates to other academicdisciplines, whilst practice is inter-professional. Hence, material relevant to social
work is likely to be found in a range of journals, certainly within a wider range than the29 publications that comprise the current Thomson Reuters social work index.
We would expect that if a list of discipline-specific journals were made by such a panelit would be more extensive then this listing and it might also question whether
journals such as the American Journal of Community Psychology should be termedsocial work journals and whether the predominance of US based journals is desirable
for either the social work communities in countries other than the US or the globalsocial work community more widely.
Criteria that might be used in developing the list include:
(1) journal scope that relates directly to social work in all its forms;(2) journals having an ISSN number;(3) journals that employ a double blind peer review system, with at least two
independent reviews per paper;(4) clear and unambiguous journal guidance about the nature of judgements to be
made by peer reviewers;(5) the publication of a list of those who review for the journal;(6) journals at least into the third year of publication, as a demonstration of viability
(in the early years of publication new journals are often extensively reliant oncommissioned papers);
(7) periodic sampling to agree that the quality of papers published is of anappropriate scholarly standard, judged by the quality of the argument ratherthan the nature of the content.
Using such criteria it would be possible to judge that articles published within
the body of recognised discipline-specific journals met a core rigorous standard forthe discipline. With such an agreed corpus of work, other bibliometrics might bedeveloped, including the volume of publications for the discipline and for the
academic unit—citation indices whose function would be illustrative and informative,but not normative.
Conclusion
Proposals to employ bibliometrics in the assessment of research quality, including social
work research, appear to be based on unquestioned and naıve perceptions of the rigourand objectivity of the Thomson Reuters system. This paper has sought to draw social work
134 E. Blyth et al.
Dow
nloa
ded
by [
Uni
vers
ity o
f H
udde
rsfi
eld]
at 0
2:43
29
Mar
ch 2
015
educators’ attention to some of the most important flaws in this system and to argue that,
anyway, a range of assessment criteria should be used in future assessment of social workand other research quality. Bibliometrics may have a place in these but we would also
argue that work should be conducted to produce an alternative system which, in line withsocial work values, is fair, inclusive and transparent.
Notes
[1] For details see: http://www.thomsonreuters.com/about/.
[2] In the 1930s, S. C. Bradford claimed that, for any given discipline, the bulk of significant papers
were published in a very small number of ‘core’ journals (Bradford, 1934).
[3] The Thomson Reuters website claims that ‘editors performing journal evaluations have
educational backgrounds relevant to their areas of responsibility as well as experience andeducation in information science’. It is not clear if the social work credentials of staff dealingwith social work include professional qualification or education within the discipline (Testa, n.d.).
[4] The system is still variously described as the Institute of Scientific Information Citation Indices,
ISI Citation Indices or the Web of Science database.
[5] According to the scope notes for the social work category: Social Work covers resources
concerned with homelessness, social casework, social services, social work education, publicwelfare, family counselling, child welfare and abuse, social work administration, social workwith groups, and gerontological social work (http://scientific.thomsonreuters.com/mjl/scope/scope_ssci/). This is self-evidently a restricted definition of the discipline that will, of necessity,narrow the range of journals considered eligible for designation as ‘social work’ journals.
References
Bradford, S. C. (1934) ‘Sources of information on specific subjects’, Engineering: An Illustrated Weekly
Journal, vol. 137, pp. 85–86 [reprinted as Bradford, S.C. (1985) ‘Sources of information on
specific subjects’, Journal of Information Science, vol. 10, no. 4, pp. 176–180.
Casal, J., Struhl, G. & Lawrence, P. (2002) ‘Developmental compartments and planar polarity’,
Drosphila. Current Biology, vol. 12, pp. 1189–1198.
Evidence/ESRC (2004) Bibliometric Profiles for Selected Units of Assessment [online]. Available at:
http://www.esrcsocietytoday.ac.uk/ESRCInfoCentre/Images/Bibliometric%20Profiles%
20for%20RAE%20Outputs%20in%20the%20Social%20Sciences_tcm6-18357.pdf.
Fisher, M. & Marsh, P. (2003) ‘Social work research and the 2001 Research Assessment Exercise: an
initial overview’, Social Work Education, vol. 22, no. 1, pp. 71–80.
Garfield, E. (1955) ‘Citation indexes to science: a new dimension in documentation through
association of ideas’, Science, vol. 122, no. 3159, pp. 108–111 [online]. Available at: http://
garfield.library.upenn.edu/essays/v6p468y1983.pdf.
Garfield, E. (1994) ‘The Thomson scientific impact factor’, Current Contents, 20 June [online].
Available at: http://scientific.thomsonreuters.com/free/essays/journalcitationreports/impact-
factor/.
Gill, J. (2009) ‘Keep peer input in REF, urge panels’, The Times Higher Education, no. 1878, p. 5.
Green, R., Baskind, F. & Bellin, M. (2002) ‘Results of the doctoral faculty publication project: journal
article productivity and its correlates in 1990s’, Journal of Social Work Education, vol. 38, no. 1,
pp. 135–152.
Green, R. & Secret, M. (1996) ‘Publishing by social work scholars in social work and non-social work
journals’, Social Work Research, vol. 20, no. 1, pp. 31–41.
Social Work Education 135
Dow
nloa
ded
by [
Uni
vers
ity o
f H
udde
rsfi
eld]
at 0
2:43
29
Mar
ch 2
015
Higher Education Funding Council for England (2008a) Bibliometrics Pilot Exercise [online].Available at: http://www.hefce.ac.uk/research/ref/pilot/.
Higher Education Funding Council for England (2008b) Evidence Ltd and the Bibliometrics Pilot[online]. Available at: http://www.hefce.ac.uk/research/ref/pilot/evidence/.
Higher Education Funding Council for England, Scottish Higher Education Funding Council,Higher Education Funding Council for Wales & Department for Employment and LearningNorthern Ireland (2006) RAE 2008: Panel Criteria and Working Methods: Panel J, Ref RAE01/2006 (J).
Joint Committee on Quantitative Assessment of Research (2008) Citation Statistics: A Report from theInternational Mathematical Union (IMU) in Cooperation with the International Council ofIndustrial and Applied Mathematics (ICIAM) and the Institute of Mathematical Statistics(IMS), 12 June [online]. Available at: http://www.mathunion.org/fileadmin/IMU/Report/CitationStatistics.pdf.
Joint Universities Council/Social Work Education Committee (2008) Response to HEFCE 2007/34,the Research Excellence Framework Consultation [online]. Available at: http://jucuk.files.wordpress.com/2008/05/jucswec-refresponse0208.pdf.
Lawrence, P. (2007) ‘The mismeasurement of science’, Current Biology, vol. 17, no. 15,pp. R583–R585.
Leung, P. & Cheung, M. (2008) ‘Journals in social work and related disciplines: manuscriptsubmission information’, 5 May 2008 revision [online]. Available at: http://www.sw.uh.edu/documents/cwep/ManuscriptSubmissionInformation.pdf, accessed 5 January 2009.
Lyons, K. & Orme, J. (1998) ‘The 1996 Research Assessment Exercise and the response of social workacademics’, British Journal of Social Work, vol. 28, no. 5, pp. 783–792.
McLaughlin, H., Lawson, J. & Shardlow, S. M. (2007) ‘Comparing the 1996 and 2001 researchselectivity exercises in respect of social work’, Social Work Education, vol. 26, no. 2, pp. 1–12.
Meinert, R. (1993) ‘Should there be a moratorium on articles that rank schools of social work basedon faculty publications?’, Journal of Social Work Education, vol. 29, no. 3, pp. 247–251.
Schiele, J. (1991) ‘Publication productivity of African-American social work faculty’, Journal of SocialWork Education, vol. 40, no. 1, pp. 143–160.
Sellers, S., Mathieson, S., Smith, T. & Perry, R. (2006) ‘Perceptions of professional social workjournals; findings from a national survey’, Journal of Social Work Education, vol. 42, no. 1,pp. 139–160.
Simkin, M. & Roychowdhury, V. (2003) ‘Read before you cite!’, Complex Systems, vol. 14, no. 3,pp. 269–274.
Taylor-Gooby, P. (2008) In Z. Corbyn, ‘Keep peer review at REF core, chairs warn’, Times HigherEducation, 28 August.
Testa, J. (n.d.) The Thomson Scientific Journal Selection Process [online]. Available at: http://www.thomsonreuters.com/business_units/scientific/free/essays/journalselection/.
Thomson, Reuters (n.d.) The History of Citation Indexing [online]. Available at: http://www.thomsonreuters.com/business_units/scientific/free/essays/history/.
Universities UK (2007) The Use of Bibliometrics to Measure Research Quality in UK Higher EducationInstitutions, Universities UK, London [online]. Available at: http://www.universitiesuk.ac.uk/Publications/Pages/Publication-275.aspx.
136 E. Blyth et al.
Dow
nloa
ded
by [
Uni
vers
ity o
f H
udde
rsfi
eld]
at 0
2:43
29
Mar
ch 2
015
Recommended