View
810
Download
3
Embed Size (px)
DESCRIPTION
June 18, 2014 NISO Virtual Conference: Transforming Assessment: Alternative Metrics and Other Trends Assessing and Reporting Research Impact – A Role for the Library - Kristi L. Holmes, Ph.D., Director, Galter Health Sciences Library, Northwestern University, Feinberg School of Medicine
Citation preview
Assessing and Reporting Research Impact – A Role for the LibraryKristi L. Holmes, PhDDirector, Galter Health Sciences Library
Associate Professor, Preventative Medicine-Health and Biomedical Informatics
Northwestern University, Feinberg School of Medicine
http://orcid.org/0000-0001-8420-5254
NISO Virtual Conference:
Transforming Assessment: Alternative Metrics and Other Trends
June 18, 2014
overview
• Impact metrics - the ususal suspects and more • A view from the library
• Identifying needs• Roles played by libraries• Developing services
• Moving this space forward on the local and larger level
impact metrics – why do we care?
• Quantify and document research impact
• Justify future requests for funding
• Quantify return on research investment
• Discover how research findings are being used
• Identify similar research projects
• Identify possible collaborators• Determine if research findings
are duplicated, confirmed, corrected, improved or repudiated
• Determine if research findings were extended (different human populations, different animal models/species, etc.)
• Confirm that research findings were properly attributed/credited
• Demonstrate that research findings are resulting in meaningful health outcomes
• Discover community benefit as a result of research findings
• Progress reports• Tenure• Promotion dossiers
https://becker.wustl.edu/impact-assessment/model
alternative metrics• Twitter• Facebook• Blogs• Research Highlights• Google+• Main Stream Media• Reddit• Forums• Q&A – StackExchange• Pintrest• LinkedIn• FigShare • F1000 Reviews• GitHub
Thelwall M, Haustein S, Larivière V, Sugimoto CR (2013) Do Altmetrics Work? Twitter and Ten Other Social Web Services. PLoS ONE 8(5): e64841. doi:10.1371/journal.pone.0064841
Konkiel S (2013) Altmetrics: A 21st-Century Solution to Determining Research Quality. Information Today. Available at http://www.infotoday.com/OnlineSearcher/Articles/Features/Altmetrics-A-stCentury-Solution-to-Determining-Research-Quality-90551.shtml
ArticlesDatasets
SlidesSoftware
Webpages
Understanding the impact of research is a massive task.
How do we understand the impact of a published work?
A person? A research center?
A university?
What IS impact?
Ioannidis JA, Khoury MJ. Assessing Value in Biomedical Research: The PQRST of Appraisal and Reward. JAMA. Published online June 09, 2014. doi:10.1001/jama.2014.6932.
• We need to move assessment toward “desired outcomes: research that is productive, high-quality, reproducible, shareable, and translatable (PQRST)”
• Productivity metrics should reward high-influence science rather than least publishable units and decrease publication bias against negative results.
Research impact and assessment
go beyond counts!
https://becker.wustl.edu/impact-assessment
Investigate and adapt frameworks to help put
things into context
More and more organizations and efforts are considering the impact of their work through frameworks.
The Becker Model (Cathy Sarli)Frameworks from the IOM, CDC, and NIEHSSnowball MetricsDiscipline-specific approachesand more…
go beyond counts!
https://becker.wustl.edu/impact-assessment
Investigate and adapt frameworks to help put
things into context
The Becker Model
• Provides a supplement to publication analysis to provide a more robust and comprehensive perspective of biomedical research impact.– reporting templates, glossary of resources and terms, examples of
relevant indicators of impact across the research process, and readings
• Straightforward framework for tracking diffusion of research outputs and activities to locate indicators that demonstrate evidence of biomedical research impact– individual, core, and institutional-level; modify for different disciplines
• Guidance for quantifying and documenting research impact as well as resources for locating evidence of impact.
• Strategies for enhancing the impact of research
strategies for enhancing the impact of research
https://becker.wustl.edu/impact-assessment/strategies
Repetition, consistency, and an awareness of the intended
audience form the basis of most the strategies.
The strategies focus upon Preparing for Publication,
Dissemination, and Keeping Track of Your Research.
Optimizing discoverability and access of your research is the
surest way to enhance its visibility and impact.
Suggestions for researchers and recommendations to reach out to
their library for assistance.
Understanding the impact of research is a massive task.
How do we understand the impact of a published work?
A person? A research center?
A university?
What IS impact?
How do we scale up and operationalize this process?
How do we support it locally from the library?
leverage ongoing efforts on the local level and beyond
• ORCID• CASRAI• CERIF• VIVO-ISF ontology• NISO Altmetric work
• CV/Biosketch • Large-scale semantic search
• Institutional repositories
• Research networking systems
• Institutional research information and management systems
• Analytical and benchmarking efforts
• Tracking and evaluation
VIVO lends itself to research assessment and impact
• Facilitates a more robust research ecosystem
– Diffusion of research products
– Discovery of shared products
– Reproducibility
– Translation
– Enables credit and visibility by showcasing individual achievements and expertise
• Supports team-based science and collaboration
• Allows better large-scale understanding of the research enterprise
– Temporal relationships (career development, time from publication to research synthesis)
– Peer comparisons
– Strategic planning and visualization
– Identify emerging trends
• Enables more efficient means of collecting & representing meaningful outputs en masse
• New experimental protocols• Datasets• Instrumentation• Software code• New diagnostic criteria• New standard of care• Curriculum guidelines• Measurement instruments• Continuing education
materials• Clinical/practice guidelines
VIVO lends itself to research assessment and impact
• Facilitates a more robust research ecosystem
– Diffusion of research products
– Discovery of shared products
– Reproducibility
– Translation
– Enables credit and visibility by showcasing individual achievements and expertise
• Supports team-based science and collaboration
• Allows better large-scale understanding of the research enterprise
– Temporal relationships (career development, time from publication to research synthesis)
– Peer comparisons
– Strategic planning and visualization
– Identify emerging trends
• Enables more efficient means of collecting & representing meaningful outputs en masse
• New experimental protocols• Datasets• Instrumentation• Software code• New diagnostic criteria• New standard of care• Curriculum guidelines• Measurement instruments• Continuing education
materials• Clinical/practice guidelines
VIVO can help operationalize this process:
Enhance profiles by incorporating meaningful outputs to enhance content and value; can facilitate dissemination and discovery of scholarship
The ontology is the key…
• Plum Analytics gathers metrics across five categories—usage, mentions, captures, social media and citations.
• Metrics are gathered around artifacts
• Collected information is displayed in visualizations, dashboards, and widgets.
• Customized for the institution or organization.
ARTIFACTSarticles
blog postsbook chapters
bookscases
clinical trialsconference papers
datasetsfiguresgrants
interviewslettersmedia
patentsposters
presentationssource code
theses / dissertationsvideos
webpages
http://www.ebscohost.com/newsroom/stories/plum-analytics-becomes-part-of-ebsco-information-services
IR integration
IR integration
Courtesy of Plum
IR integration
Benefits?1. Create a feedback loop for researchers that
gives them timely insight into the impact of their research right from the start
2. Easy access to advanced metrics from PlumX can help build buy-in from researchers, increasing repository support by your contributors
3. Aggregated metrics and reports from the PlumX dashboard puts the repository at the center of communicating about the impacts of your organization’s research
Courtesy of Plum
a spectrum of possibilities…
http://commons.wikimedia.org/wiki/File:Linear_visible_spectrum.svg
Integrate altmetrics in your IR
Lead or support
RNS project (VIVO, etc)
Run citation reports
ORCID services and
support
Encourage prudent use of
online toolsWelcome deposit of alternative outputs
in your IR
Keep current
Collaborate with TT office to
track outputs
Develop standard
reports/visualizations/analyses Hire an impact
and evaluation librarian
Workshop on strategies for
enhancing research impact Run
publication data reports
Establish a consultation
service
Help researchers
obtain DOIs and understand data
options
Understand motivations of funding agencies
local success depends on stakeholder engagement
• Stay up to date on scholarly issues and the scholarly workflow
• Stay up to date on funding/reporting requirements• Brainstorm to identify and understand motivations of
stakeholders• Anticipate need and present solutions• Call on advocates• Think beyond ‘business as usual’• Be persistent!
resources• BECKER MODEL, MOTIVATIONS: https://becker.wustl.edu/impact-assessment/model • Thelwall M, Haustein S, Larivière V, Sugimoto CR (2013) Do Altmetrics Work? Twitter and Ten Other Social Web
Services. PLoS ONE 8(5): e64841. doi:10.1371/journal.pone.0064841• Konkiel S (2013) Altmetrics: A 21st-Century Solution to Determining Research Quality. Information Today. Available at
http://www.infotoday.com/OnlineSearcher/Articles/Features/Altmetrics-A-stCentury-Solution-to-Determining-Research-Quality-90551.shtml
• Ioannidis JA, Khoury MJ. Assessing Value in Biomedical Research: The PQRST of Appraisal and Reward. JAMA. Published online June 09, 2014. doi:10.1001/jama.2014.6932. Available at http://jama.jamanetwork.com/article.aspx?articleid=1881107.
• BECKER MODEL: https://becker.wustl.edu/impact-assessment• BECKER MODEL - STRATEGIES FOR ENHANCING RESEARCH IMPACT:
https://becker.wustl.edu/impact-assessment/strategies • ORCID: http://orcid.org/• ORCID OUTREACH MEETING (with some great ideas and approaches for libraries):
https://orcid.org/content/orcid-outreach-meeting-and-codefest-may-2014 • CASRAI: http://casrai.org/• VIVO: http://vivoweb.org/• VIVO-ISF ONTOLOGY (info and how to participate/learn more):
https://wiki.duraspace.org/display/VIVO/VIVO-ISF+Ontology • CERIF: http://www.eurocris.org/Index.php?page=CERIFreleases&t=1 • CONVERIS RESEARCH INFORMATION SYSTEM: http://www.converis5.com/• VIVO DATA DASHBOARDS AND VISUALIZATIONS: contact [email protected] for more information• PLUM ANALYTICS: http://www.plumanalytics.com/• PLUM METRICS: http://www.plumanalytics.com/metrics.html and the PLUM PRINT:
http://blog.plumanalytics.com/post/87893660825/visualizing-impact-the-plum-print • ALTMETRIC: http://www.altmetric.com/• ALTMETRIC FOR INSTITUTIONS: http://www.altmetric.com/institutional-edition.php
AcknowledgementsSupport:• Northwestern University
Clinical and Translational Sciences Institute, NIH award 8UL1TR000150-05
• VIVO - DuraSpace
Thanks:• Cathy Sarli, MLS, AHIP
• Karen Gutzman – NLM Fellow
• Andrea Michalek at Plum @amichalek
• VIVO Community @VIVOcollab
• Galter Health Sciences Library
Questions:@kristiholmes
Thank you!