Academic Libraries and Virtual ReferenceUser Preferences & Usability of Reference Services
July 12, 2011
Rebecca Croxton, MLISAnthony S. Chow, Ph.D.
Purpose of Study
• Better understand information seeking preferences of library users in an academic setting
• Develop an understanding of user perceptions of “usability” of different virtual reference mediums offered by an academic library
Research Questions
1. Which virtual reference mediums do library reference users prefer? (Email, Telephone, Online Chat, Video Conferencing, Text Messaging)
2. Are there differences in user preferences for virtual reference mediums based on type of question?
3. Are there differences in user preferences for virtual reference mediums based on type of user and type of question?
4. Which virtual reference mediums do library users find most usable?
Findings in the Literature
Findings vary across studiesWashington State University
(2 campuses) (N=197)South Atlantic (2 Universities)
(N=276)1st Choice When Seeking Library
Assistance1st Choice When Seeking Reference
Help with ResearchLibrary Web Site (42.4%) Face-to-Face (66.4%)Reference Librarian (14.5%) Email (20.2%)Friend (10.9%) Telephone (9.2%)
Online Chat (10.3%) Online Chat (4.2%)Telephone (10.3%)Email (7.2%)Other (3.6%)Joel Cummings, Lara Cummings, and Linda Frederiksen, "User Preferences in Reference Services: Virtual Reference and Academic Libraries," Portal 7, no. 1 (January 2007).
Corey M. Johnson, "Online Chat Reference," Reference & User Services Quarterly 43, no. 3 (Spring 2004): 241.
Study Methodology
• Campus-wide Survey (936 participants)
• Usability Study (31 university student participants)
• Focus Group of Reference Library Staff (13 participants)
• Interview with Head of Reference at Academic Library
Participants – Campus Wide Survey
Campus-Wide Survey - 936 respondents• Sent to a randomly selected sample of university students
(N=2,552 / 14.2% response rate)• All permanent faculty and staff (N=3,840 / 14.3% response rate)
31%
30%
22%
17%
Survey Participants
Staff
Faculty
Undergraduates
Graduate Students
Instrumentation – Campus Wide Survey
• 11-item online survey• Questions:
– Prior experience with and knowledge of reference services
– Rank order virtual reference seeking preferences for answering a procedural based, research question and a quick, factual question
– Choose one reference service for procedural, research question and quick, factual question – this time including face-to-face reference as option
Key Findings – Campus-Wide Survey
Fifty-four percent of respondents had prior experience with at least one reference medium.
Key Findings - Campus-Wide SurveyResearch Questions
Email is most popular choice for research questions.
Key Findings – Campus-Wide SurveyResearch Questions
If face-to-face is an option, survey respondents prefer face-to-face for research questions.
• Faculty prefer email over face-to-face reference.• After face-to-face, undergraduates nearly equally prefer email and online chat.
Key Findings – Campus-Wide SurveyQuick, Factual Questions
Telephone is most popular for quick, factual questions.
Key Findings – Campus-Wide SurveyQuick, Factual Questions
With face-to-face as an option, telephone is most preferred for factual questions.
• Online chat and telephone were nearly equally preferred for undergraduates.• Faculty preferred email over all other choices.
Participants – Usability Study
Usability Study - 31 participants• 12 Randomly Selected Undergraduate Students • 19 Randomly Selected Graduate Students
Of these participants:• 15 Students Tested Reference Mediums at University A• 16 Students Tested Reference Mediums at University B
Universities Utilized for Study:• Two mid-sized, doctoral granting state universities in the
Southeast
Instrumentation – Usability Study
• Participants given written instructions via email to contact a designated university library and ask a pre-assigned procedural, research based question and a quick factual question via:– Email
– Telephone
– Online Chat
– Skype Video*
– Text Messaging
*University B did not offer Skype Video Reference
Instrumentation – Usability Study (Cont.)
• Participants completed online surveys after testing each medium and at completion of study
– Rank order virtual reference seeking preferences for answering a procedural based, research question and a quick, factual question
– Choose one reference service for procedural, research question and quick, factual question – this time including face-to-face reference as option
– Rate usability of virtual reference mediums using measures defined by ISO 9241-11:1998 of Effectiveness, Efficiency, and Satisfaction
Key Findings – Usability StudyResearch Questions
Online chat is most preferred for research questions.
Key Findings – Usability StudyResearch Questions
With face-to-face as an option, online chat is still most preferred for research questions
Key Findings – Usability StudyQuick, Factual Questions
Online chat is most preferred for quick, factual questions.
Key Findings – Usability StudyQuick, Factual Questions
With face-to-face as an option, online chat is still most preferred for quick, factual questions.
Key Findings – Research Questions
• Campus-Wide Survey– Email is most preferred virtual reference medium
– When face-to-face reference is an option, users prefer face-to-face over all other choices• Exception – Faculty preferred email over F2F
• Usability Study (students only)
– Online chat is preferred reference medium over all other choices (including face-to-face)
Key Findings – Quick, Factual Questions
• Campus-Wide Survey– Telephone is most preferred virtual reference medium
– When face-to-face reference is an option, users prefer telephone over all other choices
– Exceptions:– Faculty prefer email over face-to-face reference– Students equally prefer telephone and online chat
• Usability Study (students only)
– Online chat is preferred reference medium over all other choices (including face-to-face)
Summary of PreferencesResearch Question
1st ChoiceQuick, Factual Question
1st Choice
Virtual Reference
Overall(w/ Face-to-
Face)
Virtual Reference
Overall(w/ Face-to-
Face)
All Users Combined
Email Face-to-Face Telephone Telephone
Undergraduates Online Chat Face-to-Face* Telephone* (followed closely by Online Chat)
Online Chat
Graduate Students
Email* Face-to-Face* Telephone* (followed closely by Online Chat)
Telephone* (followed closely by Online Chat)
Faculty Email Email (followed closely by F2F)
Telephone (followed closely
by Email)
Staff Email Face-to-Face Telephone Telephone*In usability study, both graduate and undergraduate students rated Online Chat as their first choice, regardless of the availability of Face-to-Face Reference.
Usability - Defined
“Usability is a quality attribute relating to how easy something is to use. … it refers to how quickly people can learn to use something, how efficient they are while using it, how memorable it is, how error-prone it is, and how much users like using it. If people can’t or won’t use a feature, it might as well not exist.”
J. Nielsen & H. Loranger, Prioritizing Web Usability. (Berkeley, CA: New Riders, 2006).
Usability - Operationalized
Per ISO 9241-11:1998, usability is measured using three factors:
• Efficiency• Effectiveness• Satisfaction
Usability - Effectiveness
• Effectiveness (Jordan, 2001)– Task Completion: “Were you able to successfully use [reference
medium] to ask your questions and have them answered?” – Quality of Output: “How satisfied were you with the overall quality of
the [reference medium] responses and the extent to which they answered your questions?”
P.W. Jordan, An Introduction to Usability. (Philadelphia, PA: Taylor & Francis, 2001).
Effectiveness – Usability Ratings(1=lowest satisfaction / 10=highest satisfaction)
Task Completion Quality of Output
Overall Effectiveness
Email (n=31) 8.29 6.03 7.16Telephone (n=30) 8.77 8.53 8.65Online Chat (n=30) 9.50 9.70 9.60Skype Video (n=15)* 6.33 6.07 6.20Text Messaging (n=29) 6.21 6.00 6.10*Skype reference was only available at “University A”
Usability - Efficiency
• Efficiency (Jordan, 2001)– Deviations from Critical Path: “How satisfied are you that you were able to
use [reference medium] without any problems?”– Error Rate: “How satisfied were you that you accomplished your goal with very
few or no errors?”– Time-on-Task: “How satisfied are you that you accomplished your goal with
minimal time wasted?”– Mental Effort: “How satisfied are you that you accomplished your goal with
minimal mental effort?”Efficiency – Usability Ratings
(1=lowest satisfaction / 10=highest satisfaction)Deviations from
Critical PathError Rate
Time-on-Task
Mental Effort
Overall Efficiency
Email (n=31) 8.65 8.77 7.26 8.45 8.28Telephone (n=30) 9.07 9.07 8.03 8.53 8.68Online Chat (n=30) 9.4 9.47 9.33 9.27 9.37Skype Video (n=15)* 5.07 5.47 5.27 5.20 5.25Text Msg (n=29) 6.45 6.52 6.00 6.48 6.36*Skype reference was only available at “University A”
Usability - Satisfaction
• Satisfaction (Jordan, 2001)– “Overall, how satisfied were you with the
general usability of using [reference medium] to ask reference questions?”
Satisfaction – Usability Ratings (1=lowest satisfaction / 10=highest satisfaction)
Email (n=30) 8.03 Telephone (n=30) 8.70 Online Chat (n=30) 9.60 Skype Video (n=15)* 5.67 Text Messaging (n=29) 6.38 *Skype reference was only available at “University A”
Usability – Overall Results
• Overall, research subjects found “online chat” to the be “most usable” virtual reference medium.
Overall Usability Ratings(1=lowest satisfaction / 10=highest satisfaction)
Overall Effectiveness
Overall Efficiency
Overall Satisfaction
Overall Usability (mean)
Email (n=31) 7.16 8.28 8.03 7.82Telephone (n=30) 8.65 8.68 8.70 8.68Online Chat (n=30) 9.60 9.37 9.60 9.52Skype Video (n=15)* 6.20 5.25 5.67 5.71Text Messaging (n=29) 6.10 6.36 6.38 6.28*Skype reference was only available at “University A”
Instrumentation – Focus Group & Interview
Focus Group• Reference Library Staff (n=13) at University A• 7 Questions re: thoughts, preferences, and future
trends surrounding virtual reference• Sample Question:
– “Based on your experience, which of the reference mediums (email, telephone, online chat, Skype video, or text messaging) is most popular with library patrons?
Interview with Reference Dept Manager• 18 open-ended questions re: thoughts & opinions re:
virtual reference services
Key Findings – Focus Group
• Online chat is “without a doubt” the most popular reference medium choice for library patrons both for quick factual questions and research questions.
• If individuals already have established relationships with particular librarians (such as a subject librarian or embedded in a course), email seems to be more popular.
• Usage of online chat and email fluctuates depending on the time of day and time frame in which a library patron needs a response.
• The needs of the users drive the transaction.
Key Findings – Interview with Reference Manager
In the past 3 ½ years…• Telephone inquiries have gone down
dramatically.• Online Chat inquiries have increased
substantially (330% increase since 2005).• Email inquiries at reference desk have
decreased, but emails with departmental liaisons have increased.
Answering the Research Questions
1. Which virtual reference mediums do library reference users prefer? – Results suggest library user preferences are dependent upon type of question
and type of user. Overall for virtual reference, findings suggest users prefer Email for research questions and Telephone for quick, factual questions.
2. Are there differences in user preferences for virtual reference mediums based on type of question?– Yes! Preferences differ significantly depending upon type of question.
3. Are there differences in user preferences for virtual reference mediums based on type of user and type of question?– Yes! Preferences differ significantly depending upon type of question and type
of user.
4. Which reference mediums do library users find most usable?– Study findings suggest users find “Online Chat” to be the most usable virtual
reference medium.
Explaining the Results
• Poole’s Principle of Least Effort.– Information seekers will attempt to minimize the overall work
associated with seeking information and will seek the most convenient source to meet their information needs at any given time.
• A person’s role may determine the depth of an information need, thereby influencing their choice of reference medium.
• Familiarity, convenience, and ease-of-use appear to be primary factors behind information seeking preferences.
Herbert Poole, Theories of the Middle Range (Norwood, NJ: Ablex, 1985).
Using Factor Analyses to Help Interpret the Results
• Factor Analysis of Campus Wide Survey participants’ qualitative comments suggest:– Email – primary factors behind preferences focused on:
• Convenience• Familiarity • Ease of Use• Written Record that was precise and in-depth
– Online Chat, Telephone, and Text Messaging – primary factors behind preferences focused on
• Speed and Quickness of Response• Convenience• Familiarity• Ease of Use
What Next?
• Replicate study with other populations to see if findings remain consistent.
• Watch the trends for video conferencing and text messaging reference, particularly among younger populations.
• Millennial generation will start becoming faculty and staff at universities.
References
Joel Cummings, Lara Cummings, and Linda Frederiksen, "User Preferences in Reference Services: Virtual Reference and Academic Libraries," Portal 7, no. 1 (January 2007).
Corey M. Johnson, "Online Chat Reference," Reference & User Services Quarterly 43, no. 3 (Spring 2004): 241.
J. Nielsen & H. Loranger, Prioritizing Web Usability. (Berkeley, CA: New Riders, 2006).
P.W. Jordan, An Introduction to Usability. (Philadelphia, PA: Taylor & Francis, 2001).
Herbert Poole, Theories of the Middle Range (Norwood, NJ: Ablex, 1985).
Recommended
