Discovery or Displacement?A Major Longitudinal Study of the

Effect of Web-Scale Discovery Services on Online (Journal) Usage

SCELC ColloquiumMarch 5, 2014

Michael Levine-Clark, University of DenverJohn McDonald, University of Southern California

Jason Price, SCELC Consortium

“…a steep increase in full text downloads and link resolver click‐throughs suggests Summon had a dramatic impact on user behavior and the use of library collections during this time period.”

The Impact of Web-scale Discovery on the Use of a Library CollectionDoug Way (2010) http://scholarworks.gvsu.edu/library_sp/9/

http://www.oclc.org/partnerships/econtent/solutions.en.html

Vendor marketing

Does implementation of a discovery service impact usage of publisher-

hosted journal content?

What did we measure?

• Whether there is an effect

• NOT why that effect exists (that’s a future study!)

• “Society will need to shed some of its obsession for causality in exchange for simple correlations: not knowing why, but only what”

• (Cukier & Mayer-Schonberger. 2013. Big data: A revolution that will

transform how we live, work, and think.)

Data collection• List of libraries with discovery services

>Searched on lib-web-cats

• Surveyed Libraries>Discovery service Implemented>Implementation Date (month/year)>Search box location>Marketing effort

• 149 Libraries Gave Approval>33 libraries selected for this phase>6 for each of the 4 major discovery services and a

group of 9 libraries with no service

Dataset• 33 Libraries

– 28 US, 2 CA, 1 each from UK, AUS, NZ

– WorldCat book holdings>Average: 1,114,193 ; Range: ~300k to ~2.6mil

• Implementation dates (Discovery Libraries): >2010 (3), 2011 (19), 2012 (2)

• 6 Publishers

• 9,206 Journals

• 163,545 Usable Observations

Methodology

Compared COUNTER JR1 total full text article views for the

12 months before vs 12 months after implementation date

June

201

0St

art

Impl

emen

tatio

nM

ay 2

011

May

201

2En

d

Year 1 Year 2

Included implementation month in Year 1 to ensure that both periods included an entire academic year

Examine Data for Outliers

Observations by Library & Service

Observations by Publisher

Average Usage Change By Discovery & Publisher

Analyzing Usage Change: % vs Total

Use 12 months before

Use 12 months

after% Change Total

Change

Journal A 500 600 20% 100

Journal B 5 15 200% 10

Which is the better measure?

Is it the same for publisher- & journal-level data?

Reducing variation due to institution sizeCurrently converting to change per FTE

Values are shown as x 1,000 to bring the change metric back per journal-library combination to a minimum of 0.1

2013 JISC Discovery study took a similar approach

Average Usage Change By Discovery & Publisher

Per Journal & Per 10,000 FTE

Full Model

Including Discovery Service, Publisher, and Library

Including Discovery Service, Publisher, and Library

Does the effect of discovery service differ across libraries?

Library 10-15 Library 16-21 Library 22-27 Library 28-33Library 1-9

Nested ANOVA Model

[all three factors – preliminary results]

Does usage change vary across libraries?

Institution (sorted by Mean Change)

Does usage change vary across publishers?

Publisher (sorted by Mean Change)

Does usage change vary across discovery services?

Publisher

Does the effect of discovery service differ across publishers?

ResultsCan we detect differences between Discovery Services, Publishers, and/or Libraries and/or their interactions? • Library – Yes• Publisher – No• Discovery Service – Yes

• Differential discovery service effect by publisher – Yes

Next Steps• Design & test for effects of:

–Aggregator full text availability–Publisher Size–Journal Subject–Overall usage trends (Requires Disc Srvc ‘control’)–Configuration options in Discovery services

• Expand pool of libraries• Perhaps explore WHY

Sharing Data• With participating libraries

–Customized reports for each library• With participating publishers

–Customized reports for each publisher–Presentations as requested

• With discovery vendors–Presentations as requested

• In publications and presentations–Maintaining anonymity of data

Doing “Resarch”, SCELC Style!

• Why SCELC?• SCELC Discovery Project

–Partial funding provided by SCELC• SCELC Participation

–Survey: http://bit.ly/DSparticipation

Past/Future Presentations• Ithaka Sustainable Scholarship Conference (October 2013)• Charleston Conference (November 2013)• ER&L/Library Journal Webinar (December 2013)• Shangai Jiao Tong Univ / Beijing Univ Forum (Jan 2014)• SCELC Colloquium (March 2014)• ER&L (March 2014)• UKSG (April 2014)• Presentations posted on slideshare; linked from:

–http://visualcv.com/lpq4t1s

michael.levine-clark@du.edu | johndmcd@usc.edu | jason@scelc.org

Sharing SCELC library experiences with Discovery Systems

Lala Badal, California Lutheran University (WorldCat Local)Linda Wobbe, Saint Mary's College (Ebsco Discovery Service)

Beth Namei, University of Southern California (Summon)Panel Moderator: Jason Price (SCELC)

SCELC Colloquium March 5, 2014

Ten Minute Assessments

1) Two institutional goals eacha) All the same - Percent of content discoverable

b) All different - Choose one thats interesting and assessable

2) Impact of your discovery system on instruction and reference

3) What is the single biggest factor that would increase use of your institution’s discovery system?

WorldCat Local Discovery Service at CLU

Lala Badal,California Lutheran UniversityMarch 2014

CALIFORNIA LUTHERAN UNIVERSITY

• 35 undergraduate majors and 34 minors. • Graduate programs – doctorates in educational leadership, higher education leadership and clinical psychology.• Master’s degrees in education, psychology, computer science, business administration, public policy and administration, information systems and technology, and economics. • Total enrollment – 4,282 students.

PEARSON LIBRARY

Collection features about 132,000 physical volumes, 130 active print periodicals, 158 databases, 219,000 eBooks, 65,000 fulltext journals, and 10,500 streaming online e-videos.WorldShare Management Services & WorldCat Discovery System was implemented in September 2011.

Discovery System that

o Provides single point access to many library research resources - In addition to our print collection, 80% of the library online resources overall are discoverable and searchable in our WorldCat Local.

o Enables positive end user experience o Increases usage of the subscribed electronic resources

Percentage of discoverable and searchable resources by material type

Full-text downloads: pre and post discovery system assessment

Online Catalog Usage

Selected Year (January 2013 – December 2013) Total: 472,907 Selected Period (January 2014 - February 2014)52 Weeks Prior Total: 391,151 Total: 64,159

Use at Reference

Using WorldCat Local for reference assistance is the “default action” for our Reference/Information Commons “staff”.

If the discovery service search does not yield relevant results, then they start using subject related databases and/or research guides.

Impact on Instruction

The WorldCat Local introduced to all lower–division classes.All 4 librarians at CLU teach the discovery layer and if needed additional resources (specific databases, e-journals, websites, and etc.) introduced as well.History Subject GuideBased on the CLU librarians impute and available research on impact of discovery systems on information literacy sessions, our Information Literacy Manager outlined why, when and how we use DS in instruction

Based on the CLU librarians impute and available research on impact of discovery systems on information literacy sessions, our Information Literacy Manager outlined why, when and how we use DS in instruction:

Impact on Instruction

Pros ConsMultidisciplinary searching Large result setsIntuitive information retrieval; less of a need to teach searching Some lack of relevant results

Helps identify useful databases Not knowing how results are indexedTeach at the reference/IC desk Some resistance from instruction librarians. Obviates/renders

unnecessary teaching students search process

Good for articles when few are needed Not precise enough for graduate-level researchGeared more to undergraduates Items/articles may not be covered in databasesEase of use Masks the complexities of the search process; makes it appear easier

than it really is

One-stop-shopping. Easier to sift through all the database offerings Less focus on differences in resources found

Audio/video content more visible  Good for wide-range of topics/interdisciplinary  Good for searching a range of formats and materials  

Shifts student thinking to critically evaluating what they found rather than how to search

 

Shift away from “brand names” to streamlined searching  

Instruction approachesIntroduce WCL as a place to launch your research, as one-stop shopping.Metaphor: A discovery tool is similar to a large department store such as Wal-Mart where everything is available and the quality of the merchandise can vary, whereas a subject database is more comparable to a boutique store, smaller selection but higher quality. Buck & Steffy (2013)Class Activities Have students explore the discovery tool and a more conventional database to discover similarities and differences. Then have a class discussion about the value of each tool.Have students work in teams of two to work on different options for refining a search and share the experience with the class.Have students list the ways they might use a discovery tool in their future work.Quick Writes (See lesson plan on the Discovery Tool vs. the Web)WCL Lesson Plan - IL Session

What is the single biggest factor that would increase use of WCL discovery system at CLU?

AWARENESS

Unique Features:

- Allows users to discover relevant content beyond the library specific holdings

- Provides librarians with collection development tool and generates more PDA through ILL

- Provides patrons with opportunity to save searches and results, create and share reading lists and bibliography.

Unique Features:

- Provides patrons with opportunity to save searches and results, create and share reading lists and bibliography.

Unique Features:

DISCOVERY SERVICE ASSESSMENTLinda Wobbe Saint Mary’s College of CA 2014

SMC Goals

◻ Google-like searching◻ Access the entire Library collection◻  Discovery of overlooked resources◻  Improve cross-disciplinary access

Discovery: search everything?

What is missing?

Reference Desk

◻Do you use Discovery at the Reference Desk?

YES: 100%

Reference

◻YES: as a way to retrieve known articles. ◻YES: when I'm not sure what discipline a topic will

fall into.◻YES: when no results are found in the subject

specific database, then I'll show Multisearch to "discover" things that might be hiding somewhere else.

◻YES: students often aren't sure whether they want a book or article

◻YES: students at the desk need something NOW and Multisearch is great for that.

Instruction

Do you teach Multisearch to undergraduates? ◻YES: 67%

Instruction for undergraduates

◻YES: one search box is easy for students◻YES: perfect for lower division courses◻YES: "Google-like" experience that makes

research easy◻YES: limit results by full text

Instruction for Undergraduates

◻NO: lower division classes focus on learning the database for that discipline.

◻NO: brings up too much, which is overwhelming for the lower division students' needs.

Graduate Instruction

◻Do you teach Discovery to graduate students

NO: 67%

Instruction for Graduate & Upper

◻YES: as a supplement to the disciplinary database particularly with interdisciplinary topics

◻NO: they need more options for limiting◻NO: their more sophisticated needs are met in

a database that has subject nuanced search features

◻NO: The search feature in subject specific databases is detailed to the particular subjects

Increase use?

◻Number one:

Present Discovery Search as the default

How to increase use?

◻ Improve precision⬜Material type limiters⬜Discipline-specific limiters⬜Take advantage of catalog scopes

◻Sign truce with ProQuest⬜Would percentage included to 60% from 50%

◻Enhance reference handbook

and statistical resources content

Multisearch Results

Impact & Assessment of Summon @ USC

Beth Namei, University of Southern California

3/5/2014

USC’s Two Major Discovery Service Implementation Goals:

#1:Provide better discoverability of our subscription and purchased content (via a unified access point)

#2: Provide (more) relevant results

Goal #1: Provide better discoverability of our

subscription and purchased content (via a unified access point)

757

85% (647) of our database content is indexed by Summon

62,797

91% (56,980) of our current journals (with active ISSNs) are indexed in Summon

OpenURL ClickThroughs to Full-Text

Summon added as default search option on the libraries’ homepage(July 2010)

Summon rolled out Direct Linking to Full-Text, bypassing the OpenURL link resolver (Nov. 2011)

Top Referring Sources to Full-text Content (via our OpenURL link-resolver)

In March 2013, 32 A&I databases were added to USC’s Summon instance.

This led us to revise goal #2: Provide better discoverability and access to information, via a single search box, regardless of whether we own or subscribe to the content.

March 2013: 32 A&I databases were turned on in Summon

(over 134 million records)

New ILL Accounts CreatedThere was a 25% increase in new ILL accounts after March 2013, when USC added an ILL button to Summon

Goal #2: Provide (more) relevant results

Typical complaints about our catalog:

Even if you know the title of the book you're looking for, HOMER searches often result in random/not relevant results. I find myself going to WorldCat just to locate books in our library! We need a better search engine.

...say I was trying to find out if the library has "The Name of War" by Jill Lepore. If I search "Name of War" or "The Name of War" as the title, HOMER gives me a list of results that are totally wrong. In fact, the first result is "Domesticating Vigilantism in Africa." What?? On the other hand, if I go to Google Books or WorldCat and type in "The Name of War," Lepore's book is the very first result.

April 2010 - Pre-Summon homepage

July 2010 - Summon is launched as the default tab

July 2012 - Summon-centric homepage (Homer tab is removed)

The catalog is still an option, just a less prominent one:

via a drop-down menu

an icon below the main Summon search box

Summon added as default search tab (July 2010)

Catalog search tab removed from homepage (July 2012)

Over 3 Million Summon Searches in 2013

How Users are Getting to the Catalog (2013)

Success!

We have had no complaints about our catalog’s (bad) relevancy since July 2012.

But…users are still not completely satisfied with the results they’re getting from Summon

Usability Study of the Homepage, 2013

The USC’s version of Quick Search just doesn’t generally turn up reliable results for me. I’ll search a very obvious keyword or a very specific keyword and it won’t turn up the most relevant results first even though I know the highly relevant results are in there it won’t bring them up so it’s probably a backend USC libraries problem with the Quick Search function. But because it’s difficult to narrow down by content and by type that I want and because the results just aren’t that relevant or they don’t turn up the best results first, I tend not to use Quick Search. Unless I’m feeling really, really lazy and I have to turn something in in 20 minutes and I’m like ‘whatever’s on top, I’ll take that!

What effect has implementation of Summon had on library instruction and reference service at USC?

Results from instruction surveySurveyed 47 instruction/reference librarians; 25 responded

(53%)

• 14 out of 15 librarians who teach introductory library sessions for freshman writing courses teach Summon:

o 1 does not teach Summon at all

o 3 spend 1/4 of the class session demonstrating it (10-20 minutes), but

also demonstrate 1-2 (or more) other databases/tools.

o 6 briefly demonstrate it (no more than 5 minutes) before moving on to

demonstrating other databases/tools (most of these respondents said

they felt obliged to address it since it was so prominent on our

website)

o 5 spend the majority of their teaching time demonstrating it

Do you teach Summon in upper level/subject specific courses?

Yes: 12

No: 9

n/a: 4

Do you use Summon at the Reference Desk?

Yes 11

No: 12

n/a: 2

In what ways (if any) has incorporating Summon into your instruction sessions changed your teaching?

• hasn’t changed my instruction at all

• allows for more time on other resources

• I spend less time with other [resources] to incorporate this one

• allowed for less lecture time spent on our webpage and what a

database is and more focus on content type, evaluations, building a

search. Also: I have significantly cut down my lecture time to allow

them to figure out these things through hands on activities.

• I spend more time talking about the information cycle and what the

different source types mean. I think it's allowed me to focus a little

more on the higher-order skills of research because they don't have

to be bogged down in the tool.

What is the single biggest factor that would increase use of USC’s discovery system?

better relevancy

&

better linking