Embed Size (px)
Citation preview
dato og ”Enhedens
Alesia Zuccala, Janne Pölönen, Raf Guns, Vidar Røeggen, Emanuel Kulczycki, Kasper Bruun, Eeva Savolainen
Nordic Bibliometrics WorkshopFriday October 16th, 2020
Performance-based publisher ratings and the visibility/impact of individual books:
Small fish in a big pond, or big fish in a small pond?
Tekst starter uden
dato og ”Enhedens
Books as Individual Entities in a PRFSThis study compares publisher ratings to the visibility and impact of individual books and utilizes a 2017 dataset from three Nordic Performance-based Research Systems (PRFSs) of Denmark, Norway, and Finland.
Tekst starter uden
dato og ”Enhedens
Nordic Countries PRFS’s: Journal ratings
In the Performance-based Research Funding Systems (PRFSs) of the Nordic countries, specifically Norway, Denmark and Finland, attention has been given to the development of journal lists with level assignments as opposed to Journal Impact Factors or other journal-level indicators.
Tekst starter uden
dato og ”Enhedens
Nordic Countries PRFS’s: Journal RatingsIn the Performance-based Research Funding Systems (PRFSs) of the Nordic countries, specifically Norway, Denmark and Finland, attention has been given to the development of journal lists with level assignments as opposed to Journal Impact Factors or other journal-level indicators.
Yet, similar to studies comparing the JIF to individual article citation rates (e.g., Waltman & Traag; Pudovkin, 2018), if a correlation is found at the macro-level between journal level ratings and citation scores, there are still many highly cited individual articles in lower rated journals, and articles without citations in higher level journals (Auranen & Pölönen, 2012; Aksnes, 2017).
Tekst starter uden
dato og ”Enhedens
Nordic Countries PRFS’s: Publisher RatingsIn the case of book publishers, there is no established ‘publisher impact factor’ because commercial citation databases (i.e., Web of Science/Scopus) do not accurately index publishers, international monographs, and book chapters.
In order to account for book publications, the Nordic countries and other European countries, have, like journal ratings, developed publisher level ratings
Tekst starter uden
dato og ”Enhedens
Nordic Countries PRFS’s: Publisher RatingsIn the case of book publishers, there is no established ‘publisher impact factor’because commercial citation databases (i.e., Web of Science/Scopus) do not accurately index publishers, international monographs, and book chapters. I
In order to account for book publications, the Nordic countries and other European countries, have, like journal ratings, developed publisher level ratings
As indicators, these ratings are complementary to journal ratings, and also comparable to the JIF in the sense that they attempt to account for differences in the average quality of outputs at the level of the publication channel
Tekst starter uden
dato og ”Enhedens
Nordic Countries PRFS’sLittle is known; however, about the extent to which publisher levels relate to the visibility or impact of individual books.
Accordingly, we introduce a small fish in a big pond versus big fish in a small pond metaphor, where a ‘fish’ is a book and ‘the pond’ represents its publishing house
Tekst starter uden
dato og ”Enhedens
Small Fish-Big Pond Metaphor
Tekst starter uden
dato og ”Enhedens
Research ImplicationsThis research has implications for the use of book publisher ratings in research evaluation, where the book as output is valued according to its publisher.
It is also of interest to individual book scholars, because it is not clear if choice of publisher (at a specific level) is always related to its future visibility or impact
Tekst starter uden
dato og ”Enhedens
Background Literature• We review a significant amount of research
literature related to publisher prestige, quality and specialization …
• Note that the prestige, specialization or quality of a scholarly book publisher is not easily recognized, and has been less transparent to evaluation communities than the accounting and ranking systems currently in place for journals
Tekst starter uden
dato og ”Enhedens
Current Evaluation Systems for BooksVisibility and Impact Indicators:
Although article-level and individual level performance indicators have been both studied and examined frequently (Wildegaard, 2019), there is less certainty about the reliability of indicators at the level of an individual book (see Zuccala and Robinson-Garcia, 2019, pp 720-723).
Tekst starter uden
dato og ”Enhedens
Current Evaluation Systems for BooksPublisher Level Ratings in Denmark, Norway, and Finland
Tekst starter uden
dato og ”Enhedens
Our Dataset - 2017N=743 book titles registered in the Danish, Norwegian, and Finnish National repositories
Tekst starter uden
dato og ”Enhedens
Descriptive Statistics – 2017-2019
Tekst starter uden
dato og ”Enhedens
Descriptive Statistics – 2017-2019
Tekst starter uden
dato og ”Enhedens
Spearman’s correlations – 2017-2019
Tekst starter uden
dato og ”Enhedens
Further Analysis and ResultsPublisher levels and WorldCat holding counts
Tekst starter uden
dato og ”Enhedens
Further Analysis and ResultsPublisher levels and WorldCat holding counts
Tekst starter uden
dato og ”Enhedens
Further Analysis and ResultsPublisher levels and WorldCat holding counts
Tekst starter uden
dato og ”Enhedens
Further Analysis and ResultsPublisher levels and Google Scholar citation counts
Tekst starter uden
dato og ”Enhedens
Analysis and Results
Tekst starter uden
dato og ”Enhedens
Further Analysis and ResultsPublisher level ratings as ‘predictors’ of WorldCat ‘visibility’ and Google Scholar citation ‘impact’
• We use a Negative Binomial Regression (due to overdispersion in our data)
• A positive regression value conveys a positive predictor relationship between the predictor variable and the expected count outcome.
• A negative value reflects a negative relationship between the predictor and the expected count.
Tekst starter uden
dato og ”Enhedens
Further Analysis and ResultsPublisher level ratings as ‘predictors’ of WorldCat ‘visibility’ and Google Scholar citation ‘impact’
Tekst starter uden
dato og ”Enhedens
Further Analysis and ResultsPublisher distribution ‘potential’
Since WorldCat holdings and country distribution counts are strongly and significantly correlated (rho =.959, p < 0.01) we have selected a set of level 1, 2 and 3 publishing houses that have produced 3 or more (≥3) books in our 2017 dataset, to assess their distribution potential.
Tekst starter uden
dato og ”Enhedens
Publisher distribution ‘potential’ – Level 2&3
Tekst starter uden
dato og ”Enhedens
Publisher distribution ‘potential’ – Level 1
Tekst starter uden
dato og ”Enhedens
Conclusions• Indicators that tend to yield relatively high, and
reliable counts: WorldCat holding/country distribution counts; PlumX usage counts; Google Scholar citations
• Our boxplot distributions illustrate to some extent what the middle or central tendency of ‘quality’ means. A lower publisher level (level 0-1_ generally corresponds with lower visibility (WorldCat counts) and lower impact (Google Scholar citations), and a higher publisher level (level 2 or 3) generally corresponds with higher visibility and higher impact…. With some exceptions!
Tekst starter uden
dato og ”Enhedens
Conclusions
• Thus, we show from this 2017 dataset that, despite publisher level, some individual book titles have achieved higher degrees of visibility and citation rates as bigger fish in smaller ponds. Other titles present a lower degree of visibility as smaller fish in bigger ponds.
Tekst starter uden
dato og ”Enhedens
Conclusions• There is no guarantee that the choice of
publishing outlet, as rated by a PRFS, will lead to academic and/or cultural impact… and
• In fact publisher levels are not useful as ’predictors’
• Only publication language – in English – was found to be a significant predictor. But we do not know for certain if PRFS systems, with their encouragement of ‘level 2’ publishers is increasing the number of English-language publications
Tekst starter uden
dato og ”Enhedens
Conclusions
• There is no guarantee that the choice of publishing outlet, as high quality as it is, will lead to academic and/or cultural impact… and
• In fact publisher levels are not useful as ’predictors’
• Only publication language – in English – was found to be a significant predictor. But we do not know for certain if PRFS systems, with their encouragement of ‘level 2’ publishers is increasing English-language publications
Tekst starter uden
dato og ”Enhedens
Conclusions• If authors are too concerned about the
‘internationalization’ of their research and continue to emphasize the acquisition of level 2 points, regional publishers (including research institutions, learned societies, and commercial publishers) may suffer.
• a transparent publisher registry has potential to help scholars with the selection of a publisher, especially in light of regional and multilingual concerns (Kulczycki et al., 2020). It would be valuable to authors in terms of making more ‘organic’ choices, relative to regional/multilingual issues
Tekst starter uden
dato og ”Enhedens
Thank you kindly for your attention!
References:
Aksnes, D. (2017). Artikler i nivå 2-tidsskrifter blir mest sitter. Forskerforum. Available at: https://www.forskerforum.no/artikler-i-niva-2-tidsskrifter-blir-mest-sitert/
Auranen, O., & Pölönen, J. (2012). Classification of scientific publication channels: Final report of the Publication Forum project (2010–2012). Helsinki: Federation of Finnish Learned Societies. URL: http://www.julkaisufoorumi.fi/sites/julkaisufoorumi.fi/files/publication_forum_project_final_report_0.pdf
Pudovkin, A.I. (2018). Comments on the Use of the Journal Impact Factor for Assessing the Research Contributions of Individual Authors. Frontiers in Research Metrics and Analytics, 3(2). DOI: 10.3389/frma.2018.00002
Waltman, L., & Traag, V.A. (2017). Use of the Journal Impact Factor for assessing individual articles need not be wrong. Available at: https://arxiv.org/abs/1703.02334v2
Zuccala, A. & Robinson-Garcia, N. (2019). Reviewing, indicating and counting books for modern research evaluation systems. In Glanzel, W., Moed, H., Schmoch, U. & Thelwall, M. (Eds.) Handbook of Science and Technology Indicators (pp. 715-728). Springer.
Please direct further questions to: [email protected]
