Embed Size (px)
Citation preview
Montreal Canada | | 29 October 2019
From Discovery to Fulfilment: Improving the User Experience at Every Stage
Christopher Cyr, Ph.D.Associate Research Scientist, OCLC
Two Related Projects
Discovery and Access 2018-2019
• How users navigate path from discovery to access of library
resources
Fulfillment 2019
• Experience when library requests are fulfilled from the user
and the librarians point of view
https://www.flickr.com/photos/lucasartoni/14941319645
Lynn Silipigni ConnawayDirector of Library Trends
and User Research
@LynnConnaway
Peggy GallagherMarket Analysis
Manager
@PeggyGal1
Christopher CyrAssociate Research
Scientist
@ChrisCyr19
Erin HoodResearch Support
Specialist
@ErinMHood1
Brittany BrannonResearch
Support Specialist
Research team
DISCOVERY AND ACCESS
Discovery and Access Project
• How do academic library users navigate the path from discovery to access?
• What do academic users do when searches don't result in
fulfillment?
• What differentiates searches that lead to access from
searches that don’t?
• What demographic characteristics influence the access of
users?
• How does access correlate with success?
https://www.flickr.com/photos/75487768@N04/6786332997
Theoretical Background• Log analysis to collect large amounts of unbiased user data
(Jansen 2006, Connaway and Radford 2017)
• Logs used to study how people use online systems
• Catalog Search failure rates
• Behavior of digital library users
• Use of e-journals
• User experience with video and music streaming services
(Hunter 1991; Jamali, Nicholas, and Huntington 2005; Lamkhede and
Das 2019; Nouvellet, et al. 2019)
https://www.flickr.com/photos/75487768@N04/6786397239
Theoretical Background• Problems with log analysis
• Ambiguity of log events
• Actions not captured in logs
• Combining log analysis with user interviews
• Users matched up with their search session
• Asked questions about search and analyzed transaction logs
(Connaway, Budd, and Kochtanek 1995)
• No indication that combining search logs with individual interviews
has been used since
https://www.flickr.com/photos/75487768@N04/7598096750
WORLDCAT DISCOVERY SEARCH LOG ANALYSIS“Log analysis is everything that a lab study is not.”
(Jansen 2017, 349)
1. Did a keyword search but mistyped
it
- Had 0 results
2. Redid keyword search with correct
spelling
- Had 759,902 results
3. Began typing in additional keyword
4. Selected one of the autosuggested
keyword phrases
- Had 1,761 results
What do the raw logs tell us?
USER INTERVIEWS“User interviews can help capture search and discovery behavior as the user understands it, rather than as a computer system understands it.”
(Connaway, Cyr, Brannon, Gallagher, and Hood 2019)
• Search session reconstructed with search log• Interview protocol developed using critical incident
technique• Users asked to reflect on specific searches and
requests during the search sessions
Interview Protocol Development
https://www.flickr.com/photos/gpoo/293086443
Example Questions• “Please tell us what you were looking for and why you
decided to do an online search.”
• “Did the item you were searching for come up in your
search results? In other words, did you find it?”
• “I’d like to understand how you felt about your search
experience overall. Would you say you were delighted with
your search experience?”
https://www.flickr.com/photos/enekobidegain/11327202173
What the logs told us:• Began keyword search but mistyped it
o Had 0 results
• Redid keyword search with correct
spelling
o Had 759,902 results
• Began typing in additional keyword
• Selected one of the autosuggested
phrases
o Had 1,761 results
• Just starting work on a paper on a broad topic;
didn’t yet have a direction for the paper
• Was overwhelmed with number of search
results
• Abandoned “library search” to do “Google
searching” to better determine a direction for
the paper
• Later came back to the library search and
found it useful
• Also received help from student workers in the
library
• Felt “prepared” to use the library search due to
1st-year library instruction
What Do The Interviews Tell Us?
https://www.flickr.com/photos/eric_duquenoy/7244639090
Interview coding themes
Item formatsSearch strategies
Decision-making factors
Liked ordesired features
Evaluation of resources
Feelings of frustration and
delight
Influence of librarian
BULK LOG ANALYSIS
Summary of Results
• Average of 5 minutes per session
• Average of 2.2 searches per session
• Average of 5.1 words per search
• 12% of sessions had search refinements
• 33% of sessions had multiple searches
n=282,307 sessions
https://www.flickr.com/photos/eric_duquenoy/7244639090
Types of RequestsSearch results
Physical access options
Online access attempt
Attempt to save
Physical access attempt
The user made a request for search results. This could include a new
search, refinement of an existing search, or the addition of limiters.
The user clicked an item or made a request to digitally access the full text of the item.
The user attempted to export or otherwise save the citation.
The user clicked an item or made a request to place a holdon a physical copy of the item.
Some users left the system after looking at holding, where they were
able to identify the physical item call number and/or location. These users
were categorized as having the option to physically access the item.
39%
54%
20%
19%
30%
16%
5%
6%
2%
2%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Last requests
(n=274,346
requests)
All click events
(n=1,961,168
events)
All click events vs. Last requests by type of request
Search results Physical access option Online access attempt Attempt to save Physical access attempt
While search results account for over half (54%) of all click events, they account for just over a third (39%) of last requests
Probability of fulfillmentNumber of searches 2
Number of search refinements 0
Words per search 2
Results per search 1000
Keyword limiter (1 if yes, 0 if no) 1
Author limiter (1 if yes, 0 if no) 0
Title limiter (1 if yes, 0 if no) 0
Chance of Fulfillment 69.09%
Number of searches 2
Number of search refinements 0
Words per search 7Results per search 1000
Keyword limiter (1 if yes, 0 if no) 1
Author limiter (1 if yes, 0 if no) 0
Title limiter (1 if yes, 0 if no) 0
Chance of Fulfillment 70.32%
Number of searches 2
Number of search refinements 0
Words per search 2
Results per search 1000
Keyword limiter (1 if yes, 0 if no) 1
Author limiter (1 if yes, 0 if no) 1Title limiter (1 if yes, 0 if no) 0
Chance of Fulfillment 84.76%
Impact of Study• Identify why and what users did during the search and
when acquiring resources
• Develop a new methodology for studying user behaviors
– Winner of ALISE methodology award
• Influence product and system development
https://www.flickr.com/photos/boklm/37726606441
FULFILLMENT PROJECT
Rationale• Purpose of the research: Influence and guide strategic
direction and product development of OCLC delivery services.
• Methodology: The project will entail focus group interviews with librarians and semi-structured interviews with faculty members and graduate/postgraduate students at select OCLC Research Library Partnership institutions
Image:https://www.flickr.com/photos/satanoid/5851823198/by satanoid/CC BY 2.0
FOCUS GROUP INTERVIEWS
Focus Group Research Questions
Image: https://www.flickr.com/photos/rexness/4513981469/ by Rexness/CC BY-SA 2.0
• RQ1: How do resource sharing/ILL librarians currently spend their time?
• RQ2: How would they like to spend their time?• RQ3: What aspects of lending and fulfilment do they believe could be
unmediated?• RQ4: Where are the inefficiencies in their workflow and what would
help them overcome these and streamline their processes?• RQ5: What are their expectations, if any, for standardized, shared ILL
and resource sharing policies?
General Information
ACRLSix participants (n=6)Cleveland, OhioApril 2019
ALIANine participants (n=9)Sydney, AustraliaFebruary 2019
Colorado ILLFifteen participants (n=15)Westminster, ColoradoApril 2019
Resource Sharing ConferenceTen participants (n=10)Jacksonville, FloridaMarch 2019
Total n=40Image:https://www.flickr.com/photos/thetejon/4051985062/ by Jon/(CC BY-SA 2.0)
Key Librarian Issues
Standardization
Image: https://www.flickr.com/photos/56001877@N04/8126838599/by runmonty/CC BY-NC-ND 2.0
FundingRequest Volume
Standardization
• High number of consortia• Variation between library systems• Cross consortia sharing• Makes logistics difficult to manage
Image: https://www.flickr.com/photos/strausser/10926019275/ by free range jace/CC BY-NC 2.0
Request Volume
• High number of requests• Many last minute or short notice requests• Often held responsible for system
problems and user error
Image: https://www.flickr.com/photos/loloieg/3459011184/ by Loloieg/CC BY-ND 2.0
Funding
• Keeping ILL within budget• Increasing number of ILL holdings• Cost of licensing
Image: https://www.flickr.com/photos/hsin-yen/35173724402/by Hsin-Yen Lin/CC BY-NC-ND 2.0
SEMI-STRUCTURED INTERVIEWS
GenderFemale: 18 Male:11
Age Ranges19-25: 525-34: 535-44: 945-54: 655-64: 4
Academic LevelFaculty: 15Graduate/Postgraduate students: 14
CountriesAustralia: 21USA: 8
InstitutionsUniversity of Adelaide: 9Monash University: 4Australian National University: 2University of Melbourne: 6Montana State University: 5Swarthmore College: 3
N = 29
Semi-Structured Interview Demographics
Image: https://www.flickr.com/photos/thaths/4314445838/ by thaths/(CC BY-NC 2.0)
Semi-Structured Interview Research Questions
Image: https://www.flickr.com/photos/skellner/4203154536/ by Stefan Kellner/CC BY-NC 2.0
• RQ1: How do graduate/postgraduate students and faculty get access to resources?
• RQ2: How do they prefer to get access to resources?• RQ3: Under what circumstances and for what purposes do they
choose ways of accessing resources from the library, e.g., borrowing, downloading/accessing online., ILL or requests-to-purchase?
• RQ4: Under what circumstances and for what purposes do they choose ways of accessing resources beyond the library, e.g., purchasing from Alibris; purchasing from Amazon, Barnes & Noble, etc.; borrowing from a friend or a colleague?
Key User Considerations
Image: https://www.flickr.com/photos/duncan/15262071481/by Duncan c/CC BY-NC 2.0
Speed Cost Format
Accessibility
Communication
Speed• Heavy focus on fast fulfilment• Speed overshadows quality• Slow items are considered unavailable/not worth the time• ILL perceived as a “slower option”• Digital access regarded as quickest• “High speed” mentioned in 21 out of 29 total interviews
Image:https://www.flickr.com/photos/fransdewit/4052612802/by Frans de Wit/CC BY-NC-ND 2.0
Cost
• Little mention of specific price points• Free vs paid rather than cheap vs expensive• Users wish items were free even when institutions
cover the cost• Users will ignore paid options if free ones are
available
Image: https://www.flickr.com/photos/davem/14803922/ by Dave & Lorelle/CC BY-NC 2.0
Format
• Speed and cost important to users• Dependent on context• Some users have strong preference for one format
Image:https://www.flickr.com/photos/laurie_brooker/354514974/by Laurie Brooker/CC BY-NC-ND 2.0
Digital
• Journals, eBooks and other academic materials• Flexibility of access and use• Users access digital materials from multiple sources
• Google Scholar• Databases• Amazon
Image: https://www.flickr.com/photos/130540439@N05/16544443591/by LT Photography/CC BY-NC-ND 2.0
Physical
• Almost always prefer physical books• For both personal and professional use
• For immersion and accessibility• Frequent reference to tactility and “feel”
Image: https://www.flickr.com/photos/ncole458/5251071720/ by Nicholas Cole/CC BY 2.0
ILL and Communication Clarity• Users tend to avoid ILL
• Concerns about clarity of communication between library and user
• Prefer correspondence via email or automated messages• Puts user’s mind at ease for time sensitive
materials• Users don’t want unnecessary updates, but want
to be informed of status of requestsImage: https://www.flickr.com/photos/huangce/8390725539/ by TripNotice.com/CC BY 2.0
Faculty and Graduate/Postgraduate Student Responses
• Graduate/postgraduate students appear to be more sensitive to speed than faculty
• Graduate/postgraduate students have greater preference for PDFs than faculty
• Open content mentioned more by Australians than Americans• Faculty more interested in open content than
graduate/postgraduate students• Americans want more clarity in communication• No difference in faculty and graduate/postgraduate student need
for clarity in communication• Most users search Google to find information and resources
Recommendations Based on User Interviews
Image:https://www.flickr.com/photos/majiksbox/24923950387/by Dave – resting/CC BY-SA 2.0
• Clarity of communication (Tracking)• Convenience/Ease of use• Consolidation/One stop shopping• Citation tools• Speed/Timeliness
#ALAAC19
• Path from discovery to access to fulfilment a complex,
interlinked process
• While there are many points of frustration along the
way…
• …Users are generally understanding and appreciative
of library services
Overall Conclusions
https://www.flickr.com/photos/axelrd/4019174359
ACKNOWLEDGEMENTS
We would like to thank Nick Spence for his assistance in preparing this presentation.
Questions &
Discussion
Christopher Cyr, Ph.D.
Associate Research Scientist, [email protected]@ChrisCyr19
