Upload
naomi-norris
View
219
Download
1
Tags:
Embed Size (px)
Citation preview
Demonstrating Library Value: PracticalApproaches to Effective and Sustainable
Library Assessment
Steve Hiller Director, Assessment and Planning
University of Washington Libraries, Seattle USA
Martha KyrillidouDirector, Statistics and Service Quality Programs
Association of Research LibrariesWashington D.C.
Haifa University 2 April 2008
Free speech wall, Charlottesville, Sept 2006
Clear and present danger• Bangor University considers removing librarians posted by Blake
on Thursday January 27, 2005 @07:30AM -753 hits Ms Information writes "News from the University of Wales Bangor in the UK. senior management no longer feel that subject librarians / academic liaison librarians are needed in the modern academic library. They have made restructuring proposals which include removing all but one of the subject librarians and a tier of the library management, including the Head of Bibliographic Services. The university management thinks that technology has 'deskilled' literature searching. As far as I know, this proposal is unprecedented in the United Kingdom. In essence, there will remain 4 professional librarians serving a 'research-led' university of 8,000 plus FTEs and with 8 library sites. These will be the university librarian, cataloguing librarian, acquisitions librarian and Law librarian.
• Has anything like this happened anywhere that you know of? If so, what have been the effects?
Assessment Librarian
What’s Driving the Assessment Agenda
• Environmental Changes– Exploding growth in use and applications of technology
– Increased customer expectations for services, including quality and responsiveness
– “Competition” from other sources
• Budgetary Constraints– Justification for spending $$$ on libraries
– Increasing competition for resources
– Cost savings and/or reallocation
• Justifying our Existence– Fighting to remain relevant
The Challenge for Libraries
• Traditional statistics/measures don’t demonstrate the Library’s value– Emphasize inputs, expenditures, acquisitions, holdings – how big
and how much– Can no longer tell the library’s story– May not be aligned with library/organizational goals and plans– Don’t define and measure service quality
• Need metrics describing outcomes: success and value from the user’s perspective
• Provide the organizational culture, structure, tools, & skill sets needed for libraries to acquire & use this information
What difference do we make to our communities?
The Role of Library Assessment
Library assessment provides a structured process to learn about our communities, their work and the libraries connection to what they do
The information acquired through library assessment is used in an iterative manner to improve library programs and services and make our libraries responsive to the needs of our communities.
Libraries do not exist in a vacuum but are part of a
larger organization and community. Assessment may take place at the library level as well as at the broader community level.
Library Assessment Provides Information That Can Be Used for:
• Accountability and justification• Measure progress/achievement • Improvement of services • Comparison with others• Identify changing use and needs patterns
– Remote/In-person– Relevancy of current services and resources– New services
• Marketing and promotion• Telling “our” story• Decisions based on data, not assumptions
– Assumicide!
Customer-Centered Library and the Culture of Assessment
Customer-Centered Library • All services and activities are
viewed through the eyes of the customers
• Customers determine quality
• Library services and resources add value to the customer
Culture of Assessment • Organizational environment in
which decisions are based on facts, research and analysis,
• Services are planned and delivered to maximize positive customer outcomes
It’s about making our academic community successful
What Do We Need to Know About Our Customers?
• Who are our customers (and potential customers)?• What do they do? How is information used in their work?• How do they work? What’s important to them?• How do libraries support work, learning, leisure and life?• How do they use library services? What would they
change?• How do they differ from each other in library use/needs?
How does the library add value to their work or needs?How does the library contribute to their success?
Understanding Our Communities is Critical
• Your communities need library services which are embedded in their:– Workflows– Learnflows– Leisureflows– Lifeflows
• Engage with their languages and their processes of learning, research, leisure and … life
• Learn what’s important to them• Because they may not come to youSupport learning, research and life where they occur –
and surface that support within their lifeflows
How Do We Get Customer Information?
• Surveys• Usage statistics• Focus groups• Observation• Usability• Interviews• Embedding• Data mining (local, institutional)• Logged activities• Comments, suggestions, “over the counter”
Which Services Do We Assess
• Services that are strategically and operationally important• Services that are critical to the success of the library AND
the community• Services that are mandated for review• High performing, heavily used services
– Recognize their success; ensure sufficient resources• High performing, but underused services
– Is marketing needed?– Is the service outmoded?
• Poorly performing services– Can they be fixed? Should they be discontinued?
• New services
Choosing the Assessment Method
• Define the question and need before you start?• Is there existing information you can use?• Is the method appropriate? • Is it timely?• Is it cost effective?• What expertise is needed to conduct and analyze?• How will the results be used?
Are you assessing what’s important?Are you assessing what’s important to your community?
Using Existing Information
• Community/institutional data sources • Previous assessment efforts (library/organization)• Library use data (including e-metrics)• Acquisition requests and interlibrary loan data• Computer/Web log data• Comparative or trend data from other sources
User Surveys
• Is what I think what is really widely believe? Generalizability• Reaching large numbers of people• Comparing groups and subgroups• Measuring underlying concepts in a reliable and valid way• Recognized as a legitimate way of collecting information• Survey population factors
– Sample size, representativeness, response rate, survey fatigue– Types: transaction-based, print-mailed, web, phone
• Expertise needed for design, analysis and interpretation• Questions often asked from “our” perspective & language• Recognize the value of your respondent’s time
Why would I (or you) respond to a survey?
Reasons for Survey Response • Civic duty• Personal connection• Authority• Public/social good• Self-interest• Reciprocation• Incentives• Feel strongly about . . .
Qualitative Approaches
• Increasing use of such qualitative methods as comments interviews, focus groups, usability, observation
• Statistics often can’t tell us – Who, how, why– Value, impact, outcomes
• Qualitative provides information directly from users– Their language– Their issues– Their work
• Qualitative provides context and understanding
Observational Studies
• Describe user activities in terms of:what they do
how they do it
how much time they take
problems they encounter
• Can be obtrusive or unobtrusive• Can be tied in with interviews or usability• Well-developed data collection method/protocol essential• Room counts/facilities use most common• Quick and inexpensive; can use sampling
Interviews and Focus Groups
• Increasingly used for understanding user needs, work, behavior and outcomes
• Purpose defined; questions should be well-thought out• High degree of customer involvement• Clarify and add context to previously identified issues• Customer defined language and issues• Objective and effective interviewer/facilitator needed • Can identify broader patterns, themes, consistency but
not generalizeable to broader population• Interview/focus group themes can be followed up with
other methods or they can add context to other data
Focus Groups
• Structured discussion to obtain user perceptions and observations on a topic
• Usually composed of 6-10 participants and may be repeated several times with different groups
• Participants within group should be similar• Facilitator or moderator guides discussion • Participants encouraged to share perspectives• Participants learn from each other
Usability
• User-centered design• Apply both to virtual (Web) and physical space
(wayfinding)• Asks users to demonstrate:
– How easy is it to navigate– How easy is it to understand– How easy is it to find– How easy is it to perform– Overall how easy is it to use
• Iterative process
Comments
• Unsolicited• Unstructured (open suggestion “boxes”)• Prompted on specific issues (tell us what you think
about . . . )• Part of another assessment tool (e.g. open-ended
questions/comments on survey)
Comments are valuable for identifying issues of concern, and can add richness and texture. They generally need to be corroborated by other information
Once You Have Data . . . .
• “…but to suppose that the facts, once established in all their fullness, will ‘speak for themselves’ is an illusion.”
Carl BeckerAnnual Address of the President of the American Historical Association, 1931
. . . Comes the difficult parts:• Understanding your data• Communicating results• Using the results for improvement
Use Results Wisely
• Understand your results• Know the limitations of your data• Use appropriate analysis methods and tools• Comparative data provide context and understanding• Seek internal or external validation• Identify what is important and why
Mission:Shaping the future of research libraries in the changing
environment of public policy and scholarly communication.
Members: 123 major research libraries in North America.
Ratios: 4 percent of the higher education institutions providing 40
percent of the information resources.
Users: Three million students and faculty served.
Expenditures: 40 percent is invested in access to electronic resources.
Assessment at ARL
A gateway to assessment tools: StatsQUAL®:
– ARL Statistics -- E-Metrics– LibQUAL+®
– DigiQUAL® – MINES for Libraries®
– ClimateQUAL™
Building the library assessment community– Library Assessment Conferences– Service Quality Evaluation Academy– Library Assessment Blog– Library Assessment Consulting Service
• Effective, Sustainable, Practical Library Assessment
LibQUAL+®:
• Multidimensional user satisfaction survey• Focuses on the users’ point of view (outcomes)• Highlights difference between user’s perceived level
of service and their minimum/desirable levels• Requires limited local survey expertise and resources• Tested and validated at more than 1000 institutions
on 5 continents and in 19 languages• Analysis available at local, national and inter-
institutional levels• Offers opportunities for highlighting and improving
your status within the institution
Survey Composition
• 22 Core Questions– Affect of Service– Information Control– Library as Place
• 5 Local Questions (optional)• 5 Information Literacy Questions• 3 General Satisfaction Questions• Library Usage Patterns• Demographics• Free Text Comments Box
Survey Structure
Zone of Tolerance
Visual Representation of Data9
5
7
8
4
6
Building Assessment Capability in Libraries through Consultation Services
• ARL project “Making Library Assessment Work” approved in 2004
• Funded by participating libraries• Conducted by Steve Hiller and Jim Self – 1.5 day site visit
– Presentation and best practices– Interviews and meetings – Report to the library with recommendations
• 24 libraries in U.S. and Canada visited in 2005-06• Succeeded by Effective, Sustainable and Practical Library
Assessment in 2007 (open to all libraries)– 15 libraries participating in 2007-08– Including 4 outside North America
What We Found: Organizational Development & Structure Issues are Critical
• Strong interest in using assessment to improve customer service and demonstrate value of library
• Many libraries uncertain on how to establish, maintain, and sustain effective assessment; need assessment skills
• Effectiveness of assessment program not dependent on library size or budget
• Each library has a unique culture and mission. No “one size fits all” approach works.
• Strong customer-focus and leadership support are keys to developing effective and sustainable assessment
Effective Assessment
• Is customer-focused• Is aligned with library and organizational goals• Assesses what is important• Develops criteria for success• Uses appropriate and multiple assessment methods• Uses corroboration from other sources• Provides results that can be used
Sustainable Assessment needs . . • Organizational leadership• Sufficient resources• Supportive organizational culture• Identifiable organizational responsibility • Connection to strategic planning and library priorities• Iterative process of data collection, analysis, and use • Involvement of customers, staff and stakeholders
Practical Assessment • Keep it simple and focused – “less is more”• Know when enough is enough• Use assessment that adds value for customers• Present results that are understandable• Organization is prepared to act on results
A Strategic Framework for Assessment The Balanced Scorecard at the University
of Virginia Library
• Provides performance measurement indicators (measures), based on library objectives
• Reflects the organization's mission and strategies• Evaluates current performance and potential for the
future
Core Questions
• User Perspective – How well is the library meeting user needs?
• Internal Processes– Do the library’s processes function efficiently?
• Finance– How well are the library’s finances managed?
• Learning and Growth– Is the library well positioned for the future?
Choosing the Metrics
• Reflecting Values– What is important?– What are we trying to accomplish?
• Diversity and Balance– Innovations and operations– Variety of measurements
• Ensuring validity– Does the measurement accurately reflect the reality?
• Being Practical– Use existing measures when possible– Use sampling– Collect data centrally– Minimize work by front line
Balanced Scorecard Examples
User Perspective• Overall rating in student and faculty surveys
– Target1: An average score of at least 4.25 (out of 5.00) from each of the major constituencies
– Target2: A score of at least 4.00
Internal Processes• Processing time for routine acquisitions
– Target1: Process 90% of in-print books from North America within one month.
– Target2: Process 80% of in-print books from North America within one month
Balanced Scorecard Examples
Finance• Unit Cost (cost per download) of Electronic Serial
Use– Target1: There should be no increase in unit cost each year.– Target2: Less than 5% annual increase in unit cost.
Learning• Comparing librarian salaries to peer groups
– Target1: Average librarian salaries should rank in the top 40% of average salaries at ARL libraries.
– Target2: Rank in top 50%.
University of Washington(Site of the 2008 Library Assessment Conference!)
• Located in beautiful Seattle metro population 3.2 million
• Comprehensive public research university – 27,000 undergraduate students– 12,000 graduate and professional
students (80 doctoral programs)– 4,000 research and teaching
faculty
• $800 million annually in federal research funds (2nd in U.S.)
• Large research library system– $40 million annual budget– 150 librarians on 3 campuses
UW Libraries Assessment Priorities:Customer Needs, Use Patterns, and Success
• Information seeking behavior and use• Patterns of library use• Value of library• User needs• Library contribution to customer success• User satisfaction with services, collections, overall• Data to make informed and wise decisions that lead to
resources and services that contribute to user success
UW Libraries: Assessment Methods Used
• Large scale user surveys every 3 years (“triennial survey”): 1992, 1995, 1998, 2001, 2004, 2007 – All faculty – Samples of undergraduate and graduate students– Research scientists, Health Sciences fellow/residents
2004-
• In-library use surveys every 3 years beginning 1993• Focus groups/Interviews • Observation (guided and non-obtrusive)• Usability• Use statistics/data miningInformation about assessment program available at:
http://www.lib.washington.edu/assessment/
Case Study: UW Libraries Biosciences Review
Reasons for review• Better understand how bioscientists work• Growing inter/multi/trans disciplinary work• Significant change in use patterns• Libraries responsiveness to these changes• Value of research enterprise to the University• Strengthening library connection to research• Making good decisions on services and resources
Ensuring our services and resources support the work of the UW biosciences community
Biosciences Review Process (2006) • Define scope (e.g. what is “bioscience”?)• Identify and mine existing data sources
– Extensive library assessment data
– Institutional and external data
• Acquire new information through a customer-centered
qualitative approach – Environmental scan
– Interviews (faculty)
– Focus groups (faculty, grad students, undergrads)
– Peer library surveys
Biosciences Interview & Focus Group Questions
Interviews• Describe your work• What type of information do
you need and how do you find it?
• What barriers do you encounter?
• What role does the library play in connecting you to information
• What else could the library do to support your work
Focus Groups• How do you find the
information you need?• What barriers do you
encounter?• What other services would
you like to have available?• What services in the library
do you like or appreciate?
Biosciences Interview/Focus Group Themes• Content is primary link to the library
– Identify library with ejournals; want more titles & backfiles
• Provide library-related services and resources in our space not yours– Discovery begins primarily outside of library space with
Google and Pub Med; Web of Science also important– Library services/tools seen as overly complex and
fragmented
• Print is dead, really dead– If not online want digital delivery/too many libraries– Go to physical library only as last resort
• Difficulty coming up with “new” library services– Graduate students interested in data management services
Biosciences Task Force Recommendations
• Integrate search/discovery tools into users workflow• Expand/improve information/service delivery options• Make physical libraries more inviting/easier to use
– Consolidate libraries, collections and service points– Reduce print holdings; focus on services
• Use an integrated approach to collection allocations• Get librarians to work outside library space• Lead/partner in scholarly communications & E-science • Provide more targeted communication and marketing
Biosciences Review Follow-up : 2007 Actions
• Appointed a Director, Cyberinfrastructure Initiatives & Special Asst to the Univ Libr for Biosciences & E-Science– Meet with interested parties on campus
• Helped shape Libraries Strategic Plan Priority Initiatives– Improve discovery to delivery (WorldCat Local etc.)– Reshape our physical facilities as discovery and learning centers – Strengthen existing delivery services, both physical and digital,
while developing new, more rapid delivery services– Enhance and strengthen the Libraries support for UW’s scientific
research infrastructure – Do market research before developing & promoting services
• Informed development of Libraries 2007 Triennial Survey
Confirming Findings
Did themes raised in the interviews/focus groups reflect the bioscience population? The campus community? The 2007 Triennial Survey as corroborating source
Related Questions
Mode of access (in-person, remote)
Resource type importance
Sources consulted for research
Primary reasons for using Libraries Web sites
Libraries contribution to work and academic success
Useful library services (new and/or expanded)
UW Triennial Library Survey Number of Respondents and Response Rate 1992-2007
Large number of respondents allows for analysis within groups
2007 2004 2001 1998 1995 1992
Faculty 1455
36%
1560
40%
1345
36%
1503
40%
1359
31%
1108
28%
Grad/Prof
Students
580
33%
627
40%
597
40%
457
46%
409
41%
560
56%
Undergrads 467
20%
502
25%
497
25%
787
39%
463
23%
407
41%
Mode of Library Use by Group 2007 (weekly or more often)
Visit Only 2%
Visit Only 1%
Visit Only 27%
Remote & Visit 19%
Remote & Visit 39%
Remote & Visit45%
Remote Only72%
Remote Only47%
Remote Only14%
Non- Weekly 8%
Non- Weekly6%
Non- Weekly 20%
Faculty
Grad
Undergrad
Frequency of In-Person Visits (at least weekly
Faculty
Faculty
Grad
Grad
Undergrad
Undergrad
20%
30%
40%
50%
60%
70%
80%
1998 2001 2004 2007
Frequency of Off-Campus Remote Use (% connecting at least 2x week)
Faculty
Undergrad
Grad
0%
10%
20%
30%
40%
50%
60%
70%
1998 2001 2004 2007
0%
10%
20%
30%
40%
50%
60%
70%
Primary Reasons for Faculty Use of Libraries Web Sites (at least 2x per week)
15%
25%
35%
45%
55%
65%
75%
Health Sci Science-Engin Hum-Soc Sci
Library Catalog Bib Database Online journal articlesThe ability to access full-text or pdf research articles online through the library subscriptions is my primary use of the library and is central to my research. Neurobiology Grad Student
Sources Consulted for Information on Research Topics (Scale of 1 “Not at All” to 5 “Usually”)
Faculty
Grad
Undergrad
2.5 2.75 3 3.25 3.5 3.75 4 4.25 4.5
BibliographicDatabases
Library Catalog
Open Internet RefSource
Open Internet Search
Importance of Books, Journals, Databasesby Faculty Academic Area
Books
Journals<1985
Bib
Databases
Journals>1985
3
3.25
3.5
3.75
4
4.25
4.5
4.75
5
Health Sciences Science-Engineering Hum-Soc Science
Books Journals<1985 Bib Databases Journals>1985
Libraries Contribution to: (Scale of 1 “Minor” to 5 “Major”)
3 3.25 3.5 3.75 4 4.25 4.5 4.75
Academic Success
Efficient Use of Time
Finding info in new orrelated areas
Keeping current inyour field
Being a moreproductiveresearcher
Grad Faculty
Usefulness of New/Expanded Services Faculty and Grad
0% 10% 20% 30% 40% 50% 60% 70% 80%
Manage your info anddata
Integrate services intocampus Web sites
Office delivery ofbooks
Digitize collections
Scan and send
Grad Faculty
Survey Follow-Up Actions
• Probe deeper on specific library contributions to research and student academic success using qualitative methods– Nursing students focus groups beginning Spring 2008– Review scope and effectiveness of information literacy programs
• Develop plan to deliver “print” content to faculty & grad students in their format of choice and in their space– Pilot test “scan on demand” began January 2008
• Strengthen our subject librarian liaison efforts to better understand and support research in their areas– Develop standardized toolkit for assessing library connection to
research enterprise. Revisit scholarly communications policy
• Integrate library services & resources into user workflows
How UW Libraries Has Used Assessment
• Extend hours in Undergraduate Library (24/5.5)• Create more diversified student learning spaces• Eliminate print copies of journals • Enhance usability of discovery tools and website• Provide standardized service training for all staff• Stop activities that do not add value • Consolidate and merge branch libraries• Change/reallocate collections budget• Change/reallocate staffing• Support budget requests to University
Overall Satisfaction by Group 1995-2007
Faculty 4.25
4.44
4.33 4.33
Faculty 4.56
Undergrad 3.97 3.99
4.22
4.32
UW Seattle UG 4.36
4.34
4.26
4.11Grad 4.18
Grad 4.36
3.8
3.9
4
4.1
4.2
4.3
4.4
4.5
4.6
1995 1998 2001 2004 2007
3.8
3.9
4
4.1
4.2
4.3
4.4
4.5
4.6
Assessment can…
• Support decision making• Establish accountability• Strengthen organizational mission, goals, and
objectives• Realign library goals and mission statements with the
needs of users• Identify trends in user behaviors and technology• Measure learning outcomes and impact• Improve programs and services• Help us know our customers better
Thank youThank you
• Steve Hiller
• Martha Kyrillidou