Discovery and Delivery: Innovations and Challenges

September 26, 2012

Speakers: Lucy Harrison, Timothy Babbitt, David Bietila

[insert web address for NISO webinar page]

2

Introducing theOpen Discovery

InitiativeNISO Webinar: Discovery and Delivery:

Innovations and Challenges

Lucy Harrison, Florida Virtual Campus, D2D Liaison

September 26, 2012

Topics

• What are centralized indexes?• What are their strengths and

weaknesses?  • What is the NISO ODI initiative?• How will it help improve the

discovery landscape?

3

Evolution of Library Search• Card Catalogs• Online Catalogs• Federated search tools• Next-generation library catalogs

(discovery interfaces)• Index-based discovery services

4

Discovery Interfaces

Search: Digital Collections

ProQuest

EBSCOhost

…MLA

Bibliography

ABC-CLIO

Search Results

Real-time query and responses

ILS Data

Local Index

Federated S

earch E

ngine

Index-based Discovery

Search: Digital

Collections

ProQuest

EBSCOhost

…MLA

Bibliography

ABC-CLIO

Search Results

Harvesting and indexing performed in advance

Consolidated Index

ILS Data

Strengths of Index-based Discovery• Fast response time (vs. federated

search)• Structured metadata:

– Improves search & retrieval – Faceted navigation– Improves integration of search results

• Indexing full-text of content amplifies access

7

Issues with Index-based Discovery• Important to understand depth of

indexing– Currency, dates covered, full-text or

citation, quality of metadata• Uneven participation diminishes

impact• Ecosystem dominated by private

agreements • Complexity and uncertainty poses

barriers for participation

Need to Bring Order to Chaos• Libraries need the ability to

understand and evaluate tools, content, providers

• Information providers need the confidence that their content is being treated fairly

• Service providers need the ability to more efficiently integrate content

9

Key Areas for Libraries

• Strategic investments (in subscriptions and discovery solutions)

• Expect comprehensive representation of resources in discovery indexes

• Need to be able to evaluate the depth and quality of these index-based discovery products

• Usage reporting

Collection Coverage Questions

• How well does the index cover the body of scholarly content?

• Why do some publishers not participate?• How can libraries understand the differences in

coverage among competing services?• How are your library’s content packages

represented by the discovery service?• Which resources are not represented in index?• Is content indexed at the citation or full-text level?• What is the quality of the metadata?• What are the restrictions for non-authenticated

users?

Key Areas for Service Providers

• Encourage information providers to participate

• Lower thresholds of technical involvement

• Clarify the business rules associated with involvement

• Common industry standards and definitions

• Usage reporting

Key Areas for Information Providers• Discovery brings uncertainty• Want to expose content widely

(increase usage), but• There are trust issues

– With Access / Authentication– With “Fair” Linking

• Private agreements• Usage reporting

Need healthy ecosystem among discovery service providers, libraries and information providers

OPEN DISCOVERY INITIATIVEPromoting Transparency in Discovery

15

ODI Pre-History

• June 26, 2011: Exploratory meeting @ ALA Annual

• July 2011: NISO expresses interest• Aug 7, 2011: Proposal drafted by

participants submitted to NISO

16

ODI Proposal

Define standards and/or best practices for index-based discovery services

– Evaluate the breadth and depth of content

– Evaluate availability of content to different institutions and to different users

– Streamline workflows– Define models for fair linking – Determine what usage statistics should

be collected and disseminated 17

ODI Pre-History

• June 26, 2011: Exploratory meeting @ ALA Annual

• July 2011: NISO expresses interest• Aug 7, 2011: Proposal drafted by

participants submitted to NISO• Aug 2011: Proposal accepted by D2D• Vote of approval by NISO

membership• Oct 2011: ODI launched• Feb 2012: ODI Workgroup Formed

18

ODI Charge and Objectives• Improve information services to end

users as mediated through index-based discovery services

• Create an environment that broadens stakeholder participation and ensures confidence

• Foster development of best practices and effective means of assessment

19

Specific Benefits

Librarians – Can offer their users as wide a

range of content as possible via their discovery service of choice

– Can better evaluate discovery services to address their needs

20

Specific Benefits

Information providers – Have the confidence that the

discovery service providers are handling their content in an appropriate manner

– Are therefore encouraged to make available the widest range of content—in terms of breadth and depth – for indexing by the discovery service providers

21

Specific Benefits

Discovery service providers – Receive more standardized and

efficient integration with the information providers through common industry definitions and communications

22

Balance of Constituents

Libraries

Information Providers

Service Providers

23

Marshall Breeding, Co-ChairJamene Brooks-Kieffer, Kansas State University Laura Morse, Harvard UniversityKen Varnum, University of Michigan

Michele Newberry, Florida Virtual Campus

Sara Brownmiller, University of OregonLucy Harrison, Florida Virtual Campus (D2D liaison/observer)

Lettie Conrad, SAGE PublicationsRoger Schonfeld, ITHAKA/JSTOR/PorticoJeff Lang, Thomson Reuters

Linda Beebe, American Psychological Assoc

Aaron Wood, Alexander Street PressPeter Noerr, MuseGlobal

Jenny Walker, Ex Libris Group (Co-Chair)John Law, Serials SolutionsMichael Gorrell, EBSCO Information ServicesDavid Lindahl, University of Rochester (XC)

Jeff Penka, OCLC (D2D liaison/observer)

Organization

• Reports in NISO through Document to Delivery topic committee (D2D)

• Staff support from NISO (Nettie Lagace)• Co-Chairs

– Jenny Walker (Ex Libris)– Marshall Breeding (Library Consultant)

• D2D Observers: – Jeff Penka (OCLC), Lucy Harrison (FLVC)

24

ODI Project Goals

1. Identify, possibly through surveys or other questionnaires, the needs and requirements of the three stakeholder groups in this area of work

• Created subgroups for information gathering:– Level of Indexing– Library Rights– Technical formats– Usage Statistics– Fair Linking

25

ODI Project Goals

1. Identify, possibly through surveys or other questionnaires, the needs and requirements of the three stakeholder groups in this area of work

• Created subgroups for information gathering

• Conducted interviews with stakeholders• Created survey with input from all sub

groups• Survey is currently live (closes October 4)

26

27

https://www.surveymonkey.com/s/QBXZXSB

ODI Project Goals

1. Identify, possibly through surveys or other questionnaires, the needs and requirements of the three stakeholder groups in this area of work

• Created subgroups for information gathering

• Conducted interviews with stakeholders• Created survey with input from all sub

groups• Survey is currently live (closes October 4)• Analyze results as input to Goal 2 28

ODI Project Goals:

2. Create recommendations and tools to streamline the process by which information providers, discovery service providers, and librarians work together to better serve libraries and their users

ODI Project Goals:

3. Provide effective means for librarians to assess the level of participation by information providers in discovery services, to evaluate the breadth and depth of content indexed and the degree to which this content is made available to the user

Specific deliverables

• Standard Vocabulary• NISO Recommended Practice:

– Data format & transfer– Communicating content rights– Levels of indexing, content availability– Linking to content– Usage statistics– Evaluate compliance

• Inform and Promote Adoption31

Timeline

Milestone Target Date Status

Appointment of working group December 2011

Approval of charge and initial work plan March 2012

Agreement on process and tools June 2012

Survey completed Oct 4, 2012

Completion of information gathering October 2012

Completion of initial draft January 2013

Completion of final draft May 2013

32

Connect with ODI

• ODI Project website:http://www.niso.org/workrooms/odi/

• Interest group mailing list:http://www.niso.org/lists/opendiscovery/

• Email ODI:odi@niso.org

33

Seeing Discovery Through User Colored GlassesNISO Webinar: Discovery and Delivery: Innovations and Challenges

Timothy Babbitt, Senior Vice President, Platform Management, ProQuest

Understanding What Is Valuable to UsersFoundation of Librarianship

S.R. Ranganathan’s The Five Laws of Library Science (Madras India: Madras Library Association, 1931)1. Books are for use.

2. Every reader his [or her] book.

3. Every book its reader.

4. Save the time of the reader.

5. The library is a growing organism.

Even at a time when the emphasis was entirely on physical media the focus was on the individual goals and

needs each type of user (the reader)

The Five Laws Updated

The proliferation of web based information tools allows us to track users and their behavior with increasing precision but understanding the unique needs and

behaviors of different researchers requires deep analysis and interpretation of a variety of data – traditional usage data does not tell the whole story.

Ranganathan1. Books are for use.

2. Every reader his [or her] book.

3. Every book its reader.

4. Save the time of the reader.

5. The library is a growing organism.

Updated Laws1. Information in all of its forms is for

use. 2. Every researcher their information.3. Every medium and delivery

platform its user.4. Enable efficient discovery by the

user.5. The library is part of an evolving

research ecosystem.

How do we measure value in the evolving research landscape?

Traditional Model Usage data to measure value (searches and retrievals) –

more usage = more value

Growing Trend Web based analytics – behavioral and attitudinal dimensions

segmented by types of users

Key Question How can we combine both approaches and what can they provide

that traditional usage data cannot? Usage Data = what they did Other Analytics = what they were trying to do; did they succeed; what

was the context; and who was doing it?

Value Differs by UserWho is doing the research matters

Librarians Making their patrons successful Promoting services of library to students and faculty with

confidence Delivering services that are convenient for their users

Faculty Extending their influence and reach in their discipline

through published research Efficiently directing students to materials that meet

learning objectives Obtaining research grants

Students Completing coursework in accordance with faculty

directives Remaining in compliance with source attribution policies Accessing information conveniently

Conventional Wisdom Might Tell Us

High use = high value High satisfaction = high value Use (search and retrievals) is homogenous

Including Attitudinal and Behavioral dimensions to traditional usage data allows us to challenge many long held assumptions

Three Dimensions of Analysis

Usage

BehavioralAttitudinal

Focus of Analysis:The discovery and consumption of documents (COUNTER Reporting e.g. Searches and Retrieves) etc.

Focus of Analysis:What is the user’s context?(Page Views per visit, Time on Site, and Clickstream etc.)

Focus of Analysis:

What is the user

satisfaction?

Three Dimensions of Analysis

Usage

BehavioralAttitudinal

Focus of Analysis:The discovery and consumption of documents (COUNTER Reporting e.g. Searches and Retrieves) etc.

Usage in the new age of discovery

A case study

Earlier this year, a library contacted us about a large increase in search usage in one of their databases:

Month 11-Jan 11-Feb 11-Mar 11-Apr…

Searches Run 50 250 43 199

Month 12-Jan 12-Feb 12-Mar 12-Apr…

Searches Run 265 616 1176 847

By default, the user runs an all database search(thus showing as a “hit” against all DB’s in the library’s subscription.)

In this scenario, the user starts their search for “space tourism” via a discovery service such as Summon. Finding a document they like, they click through to the ProQuest platform and land at the document level. Then decide to run another search, which “hits” all databases.

Web scale discovery and workflow

Usage in the new age of discovery

The cause? Heavily utilization of a new web scale discovery service.

Users now used our website differently than users who started at the library “Databases A-Z” page. The change in how their users were coming to our site led to a change in usage.

Usage data showed the effect of the change of their patrons’ research environment. Determining the cause required looking beyond the usage data.

For more details see the white paper “Usage in the New Age of Discovery”

Three Dimensions of Analysis

Usage

BehavioralAttitudinal

Focus of Analysis:What is the user’s context?(Page Views per visit, Time on Site, and Clickstream etc.)

Behavioral Analysis:Precise Measure of Visits

Behavioral AnalyticsPages Viewed/ Visits

Three Dimensions of Analysis

Usage

BehavioralAttitudinal

Focus of Analysis:

What is the user

satisfaction?

Attitudinal – Satisfaction Comparison

Survey Methodology Over 19,500 Surveys Data from November 2011 through September 2012 Continuous monitoring Predictive modeling

Librarians Faculty Students

Satisfaction Comparison: Role

Satisfaction Comparison: RoleNumber of Respondents

Attitudinal Analysis:Student Satisfaction Trend-line

Student Satisfaction: By Week

Satisfaction Comparison: Primary Purpose

Satisfaction Comparison: Primary PurposeNumber of Respondents

Attitudinal AnalysisSatisfaction Comparison: School Type

Attitudinal AnalysisSatisfaction Comparison: School Type

Cross Tab: Primary Purpose and School TypeData from November 8, 2011 through August 12, 2012

“Deeper Dive” with Segment Analysis:Data: Nov. 8, 2011 through Aug 7, 2012

Seeing Value from a User Perspective

We need the triangulation of Usage data Behavioral data Attitudinal data

Drive understanding of What did the users do? What was the context of use? What were they trying to do and were they successful?

Important because the research ecosystem is changing i.e. Web scale discovery

Toward understanding what is valuable to users

Updated Laws1. Information in all of its forms is for use. 2. Every researcher their information.3. Every medium and delivery platform its

user.4. Enable efficient discovery by the user.5. The library is part of an evolving

research ecosystem.

Next steps

Complete integration and analysis of Usage data Behavioral data Attitudinal data

Give libraries a deep understanding of value from user segments

Next stop…content and search analysis!

Collecting Patron Perspectives on Discovery Tools

NISO Webinar

Discovery and Delivery: Innovations and Challenges

September 26, 2012

David BietilaWeb Program DirectorThe University of Chicago Library

Discovery ToolCatalog

Databases & Indexes

Finding Aids

Digital Collections

Web Pages

Data Sources

Assessment Questions

Product Characteristics Technical Functional Interface

Relevance to Users Define use case

Novice users of databases and electronic resources who need to find articles on a topic.

Assess product’s applicability to this use case

Topic User attitudes toward the product, and specific aspects valued by users.

Method Posted link to a comment form in the header.Set up an info table for three days.Required minimal effort to collect data.

Sample Questions What type of resource were you looking for with the Articles Plus?Please share any positive or negative comments about your experience.

Results Unprecedented proportion of positive comments.Users valued the speed of this search, and the ability to search both books and articles in one place.

1. User Comments

Functional & Interface Assessment

Resource sought Journal article

Successful? Yes

Affiliation Graduate or Professional School Student

Comment This was a quick search and gave me exactly what I needed.

Resource sought Academic journal articles

Successful? No

Affiliation Graduate or Professional School Student

Comment Record included books, needed a way to filter this outresults from Time magazine, not what I was looking for at all

Resource sought Both books and articles on certain topics (ones related to philosophy, psychology and literature).

Successful? Partial

Affiliation Graduate or Professional School Student

Comment

It's wonderful to have ONE place where you can search for both articles and books! However, it seems like more books should show up because some books relevant to my search showed up in lens but not in Articles Plus. If you don't choose this search tool, please do adopt some search tool that allows comprehensive searching of both books and articles!

Topic Usage relative to major databases

Method Examined data presented in the discovery tool’s admin interface.Compared usage with that of high traffic databases (Web of Science, JSTOR, Academic Search Premier).

Results Observed increasing usage, reaching parity in the initial year with Academic Search Premier.Verified that users were finding and using the tool.

2. Usage Statistics

Interface Assessment

March April

May

JuneJuly

August

September

October

November

0

5000

10000

15000

20000

25000

30000

35000

40000

45000

50000

Searches per Month - Articles Plus and Major Databases

EDS Foundation Index (ArticlesPlus)

Web of Science

JSTOR

Academic Search Premier

Topic Clarity of search interface, including functional elements and labels.User ability to complete article discovery tasks.

Method Assigned representative article searching tasks.Recording screen activity and spoken comments for analysis.

Sample Questions Can you locate full text of the following article? Mary L. Dudziak, “Desegregation as a Cold War Imperative,” 41 Stanford Law Review 61 (1988)

You are working to prepare a summary of developments in the field of astronomy. Can you locate five articles on astronomy that were published in Nature in 2010?

3. Usability Testing

Functional & Interface Assessment

Session Participant Status

Participant Department

Task

1 2 3 4 5 6 7 8 9 10

1 Graduate Public Policy X

2 Graduate CMES ~ X X X X

3 Undergrad Undeclared ~ X X

4 Undergrad Anthro ~

5 Graduate MAPH

6 Undergrad Physics ~ ~

7 Undergrad English ~ X ~

8 Graduate MAPSS ~ ~ ~

3. Usability Testing

Functional & Interface Assessment

Results• Cut or relabeled certain facets• Made heavily used features more prominent.• Removed unused or confusing features• Determined collections to be retained or removed from coverage.

Iterative Evaluation

Refine Use Case

Evaluate Tool

Experienced researchers Searching interdisciplinary topics Searching outside their primary area Searching within their primary area

Other Use Cases

Topic Applicability of the discovery tool for use by advanced researchers in disciplines with a variety of requirements.

Method Enlisted bibliographers and subject specialists as proxies for different user constituencies.

Created an evaluation rubric to ensure comparable results across over 40 disciplines assessed.

Results Determined which disciplines would be best served by the tool.Identified strength of coverage in numerous disciplines.

4. Subject Assessments

Functional Assessment

Subject Evaluation Form

Subjects – Social Sciences

First 25 ResultsCompared to Core DatabaseResults

Compared to JSTOR/ Google Scholar Results

Anthropology/Geography/Maps ~ ~ ~

Gender Studies ~ ~ -

History + +  + 

International Political Economy ~ - ~

Psychology + - ~

Sexuality Studies + + + 

Recommendation to purchase Improvements

Local configuration Enhancement requests

Marketing & Communication Benefit of Iterative Assessment Future assessment

Applicability of new coverage Revisiting usability Overlap with other search tools

Outcomes & Future Assessment

Thank you for joining us today. Please take a moment to fill out the brief online survey.

We look forward to hearing from you!

THANK YOU