DOWNLOADS OR OUTCOMES? : MEASURING AND COMMUNICATING THE CONTRIBUTION OF LIBRARY RESOURCES TO FACULTY AND STUDENT SUCCESS

Rachel A. Fleming-May, Ph.D., M.L.I.S.Assistant Professor, School of Information SciencesThe University of Tennessee-Knoxvillerf-m@utk.edu

AGENDA:• Overview of current concepts and practice related to e-Resource usage measurement Discussion• Alternative and

emerging techniques being investigated by the IMLS Lib-Value Project

MY INTEREST IN THIS ISSUE:• Former Practitioner at an ARL Library• Dissertation: “Use” in the Literature of LIS—

Concept Analysis “What is Library Use? Facets of Concept and

a Typology of its Application in the Literature of Library and Information Science” (in an upcoming, but still unidentified, issue of The Library Quarterly)

• Use…USAGE—measurement of e-Resource usage With Jill Grogg:

The Concept of Electronic Resource Usage and Libraries (Library Technology Reports, Aug./Sept. 2010)1

• IMLS-funded Lib-Value Grant Management Team Member

What is “use”, really?AN EVENT?SOMETHING THAT CAN BE MEASURED?

…WITH NUMBERS?Why does it matter?

“The principle of usefulness says simply that libraries should collect what patrons use.”2

Use is often treated as a PRIMITIVE CONCEPT in Library and Information Science: an idea so fundamental to the theoretical framework as to be indefinable, even when presented as a phenomenon to be measured and quantified.

“An obvious problem is that there is no clear definition of what comprises ‘use’ nor is it likely that library science will soon develop one, for it is as elusive as the concept of information, with which it is confounded.”3

USE IS FREQUENTLY ASSESSED IN ORDER TO GENERATE “OBJECTIVE” DATA FOR DECISION MAKING.

TO MEASURE USEWe focus on Inputs

…and Outputs

Number of patrons who enter the building …such as the number of book circulations.

WHAT ABOUT ELECTRONIC RESOURCES?• Many instances of use are removed

from the library, thus unobservable• Multiple points of access (such as

Google Scholar) further confuse the issue: patrons are less aware that they’re using library resources

“as the pendulum swings from physical library use to online use of libraries, we need to develop measurement and assessment methods to accurately portray how users are using the library”

“some of the basic ‘natural laws of library and information science’ may not apply as well or as consistently in the realm of electronic information discovery and use”4

“Among other changes, the Complete College Tennessee Act:• Funds higher education

based in part on success and outcomes, including higher rates of degree completion.”

Understanding USE Matters.

“Questions such as, ‘Who uses these resources?’ or ‘Are these huge outlays of funds justified in terms of use, or value derived from use?’ or ‘What difference do all of these resources make to students and faculty in universities?’ must be answered if university administrators, trustees, students, and faculty are expected to support ever-increasing levels of funding for the acquisition and development of these resources and services.”5

SO, IS USE A PRIMITIVE CONCEPT?

No. Use does not, in fact, have a singular conceptual meaning in the LIS domain and can signify many actions, processes, and events.

THE USE TYPOLOGY: DIMENSIONS OF USE

I. Use as an Abstraction Ia. Use as a FacilitatorII. Use as an ImplementIII. Use as a ProcessIV. Use as a Transaction IVa. Use as a Connector

“Of the 57,148 households [surveyed], 27,511 (48.1%) had a household member who used the public library in the past year. ”6

Use as an Abstraction • A GENERAL TERM FOR ALL

TYPES OF LIBRARY/INFORMATION USE

• DISASSOCIATED FROM ANY SPECIFIC INSTANCE OF THE PHENOMENON

USE AS A PROCESSApplication of library/information resources, materials, and/or services…

“This study reveals that undergraduate students experience information use in a complex, multi-tiered way that needs to be addressed by higher educators when creating information literacy pedagogy.”7

· To complete a complex or multi-stage task

· To the solution of a problem

USE AS A TRANSACTION

•Isolated instances of library or information use

•Can be recorded and quantified

•Removed from the user

“statistics provided by electronic book vendors…show that [our] community uses e-books quite heavily. The data do not show, however, how books are used. For instance, the available statistics show that a book has been accessed but do not differentiate between a one-second click on a title and a five-hour immersion in a book…the data also do not tell us why an electronic version of a book was used instead of the paper version”8

•Vendor-supplied data (COUNTER compliant or otherwise)

•Transaction log analysis Including page view time measurement (are

they really reading?) Log-ons—what about database timeouts?

TRANSACTIONAL MODEL OF USE=OVER-RELIANCE ON STATIC ASSESSMENTS OF ELECTRONIC RESOURCE USAGE, SUCH AS…

UNDERSTANDING OF USE AS

PROCESS

Article Download

Visit to the

Reference Desk

Db A: Log on

Not exclusively statistical

Requires multiple data collection methods

Requires “bipartisan” support, i.e., working with public services to gain a fuller understanding of how and why patrons use the resources they do.

HOW, SPECIFICALLY?•Observation•Focus groups• Interviews•Surveys• Inter-institution

information sharing

•Usability testing•Triangulation.

THE LIB-VALUE PROJECT:• Grant funded by IMLS, December 2009-2012• Principal Investigators:

Carol Tenopir, University of Tennessee Martha Kyrillidou, Association of Research Libraries Paula Kaufman, University of Illinois …other participants include librarians, academic faculty, and other

researchers from multiple institutions and disciplinary backgrounds

• Purpose: “…to study the value of academic libraries to students, faculty, policymakers, funders…” and Return on Investment (ROI) in academic libraries

• Follows two previous projects: Phase I: Return on Investment in Academic Library (UTK and U of

IL) as measured by successfully funded grants9

Phase II: Expanded to 8 international universities10

HOW IS LIB-VALUE DIFFERENT?• It’s a

Comprehensive Effort to analyze Value and ROI using models that incorporate all

inputs in the library system (faculty, staff, students, library resources) and determine how each influences the system

articulate all values of the library and areas of investment and return

Teaching/Learning

Research Social/Professional

e-Science

Collaborative Scholarship

Institutional Repositories

Functional Areas

Scho

larly

End

eavo

rs

Slide adapted from Carol Tenopir’s presentation, “ForumValue, Outcomes, and Return on Investment of Academic Libraries (Lib-Value) (funded by IMLS)” at the January, 2010 ARL Assessment Forum, Boston, MA.

KEY QUESTIONS:• Does the reputation of a

university’s library influence Enrollment? Recruitment of faculty

and students? Material or financial

donations?• Do library resources and/or

services play a role in Student success? Retention?

WHAT CAN PRACTITIONERS EXPECT FROM THE PROJECT?

•A fact-based articulation of the value and ROI of the university library resources and services within the wider mission of university administration.

•Develop a model for ROI and tools that implement this model which can be used by other academic libraries. Modular: libraries will be able to choose from a

“menu” of approaches depending on the particular services and/or resources they wish to assess

Slide adapted from Carol Tenopir’s presentation, “ForumValue, Outcomes, and Return on Investment of Academic Libraries (Lib-Value) (funded by IMLS)” at the January, 2010 ARL Assessment Forum, Boston, MA.

QUESTIONS?

Thank you for your time!

1. Fleming-May, Rachel A., and Jill E. Grogg. 2010. The concept of electronic resource usage and libraries. Vol. 46, Library Technology Reports.

2. Swigger, Keith, and Adeline Wilkes. 1991. The use of citation data to evaluate serials subscriptions in an academic library. Serials Review 17 (2):41-46; 52.

3. Miller, Rush, and Sherrie Schmidt. 2002. E-Metrics: Measures for Electronic Resources. Serials: The Journal for the Serials Community 15 (1):19-25.

4. Peters, Thomas A. 2002. What's the use? the value of e-resource usage statistics. New Library World 103 (1172/3):39-47.

5. Ibid.6. Sin, Sei-Ching Joanna, and Kyung-Sun Kim. 2008. Use and non-use of public libraries

in the information age: A logistic regression analysis of household characteristics and library services variables. Library & Information Science Research 30 (3):207-215.

7. Maybee, C. (2006). Undergraduate Perceptions of Information Use: The Basis for Creating User-Centered Student Information Literacy Instruction. The Journal of Academic Librarianship, 32(1), 79-85.

8. Levine-Clark, Michael. 2006. Electronic Book Usage: A Survey at the University of Denver. portal: Libraries and the Academy 6 (3):285-299.

9. Luther, Judy. 2008. University investment in the library: What's the return? In Library Connect White Papers.

10. Tenopir, Carol. 2010. University Investment in the Library, Phase II: An International Study of the Library's Value to the Grants Process. In Library Connect White Papers.