View
218
Download
0
Category
Preview:
Citation preview
http://www.minesforlibraries.orghttp://www.arl.org/stats
Demonstrating Value and Creating Value:Evidence-Based Library Management
through MINES for Libraries™
Martha KyrillidouDirector of the ARL Statistics and Measurement Program
Association of Research Libraries
Brinley FranklinVice Provost for University Libraries
University of Connecticut
Scholars Portal Forum, Ontario Council of University LibrariesFebruary 1 2006
Toronto
www.arl.org/stats/
http://www.minesforlibraries.orghttp://www.arl.org/stats
Today is
Tomorrow
- Peter Melinchok expressing dismay at his parents’ distorted sense of time
http://www.minesforlibraries.orghttp://www.arl.org/stats
• Bangor University considers removing librarians posted by Blake on Thursday January 27, @07:30AM -753 hits Ms Information writes "News from the University of Wales Bangor in the UK. senior management no longer feel that subject librarians / academic liaison librarians are needed in the modern academic library. They have made restructuring proposals which include removing all bar one of the subject librarians and a tier of the library management, including the Head of Bibliographic Services. The university management thinks that technology has 'deskilled' literature searching. As far as I know, this proposal is unprecedented in the United Kingdom.In essence, there will remain 4 professional librarians serving a 'research-led' university of 8,000 plus FTEs and with 8 library sites. These will be the university librarian, cataloguing librarian, acquisitions librarian and Law librarian.Has anything like this happened anywhere that you know of? If so, what have been the effects?
http://www.minesforlibraries.orghttp://www.arl.org/stats
Total Circulation
Note. M. Kyrillidou and M. Young. (2003).ARL Statistics 2002-03. Washington, D.C.: ARL, p.8.
http://www.minesforlibraries.orghttp://www.arl.org/stats
Reference Transactions
Note. M. Kyrillidou and M. Young. (2003).ARL Statistics 2002-03. Washington, D.C.: ARL, p.8.
http://www.minesforlibraries.orghttp://www.arl.org/stats
Libraries Remain a Credible Resource in Libraries Remain a Credible Resource in 2121stst Century Century
98% agree with statement98% agree with statement, “My … library , “My … library contains information from credible and contains information from credible and known sources.”known sources.”
Note. Digital Library Federation and Council on Library and InformationResources. (2002). Dimensions and Use of the Scholarly Information Environment.
http://www.minesforlibraries.orghttp://www.arl.org/stats
Changing BehaviorsChanging Behaviors
Recent Survey:Recent Survey:Only Only 15.7% agreed with the statement15.7% agreed with the statement “The “The Internet has not changed the way I use the Internet has not changed the way I use the library.”library.”
Note. Digital Library Federation and Council on Library and InformationResources. (2002). Dimensions and Use of the Scholarly Information Environment.
http://www.minesforlibraries.orghttp://www.arl.org/stats
Early data from ethnographic interviews– “I use Google because I heard it searches for more
things” (than other sources).– “I believe I can find anything on the Internet. There hasn’t
been anything I haven’t been able to find.” – “Because I’m lazy.”– Books have “so much information that no one can go
through it all.”– I use “the Internet first is because it is more convenient.”– I go to the library “because that’s what teachers like.” – “Google has gotten me through college.”
Source: Steve Jones, The Internet Goes to College, ARL Talk
The Internet Goes to College
http://www.minesforlibraries.orghttp://www.arl.org/stats
… a revolution in making
Il est plus nécessaire d'étudier les hommes que les livres
—FRANÇOIS DE LA ROCHEFOUCAULD (1613–1680)
http://www.minesforlibraries.orghttp://www.arl.org/stats
Into the future …
• StatsQUAL+™– ARL Statistics – E-Metrics– LibQUAL+™– DigiQUAL+™
– MINES for Libraries™
The MINES Survey Methodology
brinley.franklin@uconn.eduhttp://www.minesforlibraries.org
Brinley FranklinVice Provost for University Libraries
University of Connecticut
www.arl.org/stats/
MINES for Libraries
http://www.minesforlibraries.orghttp://www.arl.org/stats
What is MINES?
• Action research– Set of recommendations for research design– Set of recommendations for web survey
presentation– Set of recommendations for information
architecture in libraries– Plan for continual assessment of networked
electronic resources
http://www.minesforlibraries.orghttp://www.arl.org/stats
MINES for LibrariesTM
• MINES is a transaction-based research methodology consisting of a web-based survey form and a random moments sampling plan.
• MINES typically measures who is using electronic resources, where users are located at the time of use, and their purpose of use.
• MINES was adopted by the Association of Research Libraries (ARL) as part of the “New Measures” toolkit in May, 2003.
• MINES is different from other electronic resource usage measures that quantify total usage (e.g., Project COUNTER, E-Metrics) or measure how well a library makes electronic resources accessible (LibQual+TM).
http://www.minesforlibraries.orghttp://www.arl.org/stats
MINES for LibrariesTM Survey Form Five Questions and a Comment Box
http://www.minesforlibraries.orghttp://www.arl.org/stats
Questions Addressed By MINES for Libraries™ for the OCUL Scholars Portal
• How extensively do sponsored researchers use OCUL’s Scholars Portal? How much usage is for non-funded research, instruction/education, student research papers, and course work?
• Are researchers more likely to use the Scholars Portal from inside or outside the library? What about other classifications of users?
• Are there differences in Scholars Portal based on the user’s location (e.g., in the library; on-campus, but not in the library; or off-campus)?
• Could MINES, combined with usage counts, provide an infrastructure to make Scholars Portal usage studies routine, robust, and easily integrated into OCUL’s administrative decision-making process for assessing networked electronic resources?
http://www.minesforlibraries.orghttp://www.arl.org/stats
OCUL/MINES Methodological Considerations
• The sampling plan was determined at the outset. Surveys were conducted
once a month for two hours a month between May, 2004 and April, 2005
• The selection of the monthly survey periods were weighted based on usage
counts by time of day and were chosen randomly.
• Participation was mandatory, negating on-respondent bias, was based on
actual use in real-time, and was brief (to minimize user inconvenience).
• OCUL designed the local questions, mounted the survey, collected data and
sent it to ARL for tabulation in aggregate and by individual institution.
• If more than one search was conducted by a user, the survey form was
auto-populated with initial responses as the default.
http://www.minesforlibraries.orghttp://www.arl.org/stats
OCUL/MINES Methodological Considerations (continued)
• Each participating library explained the survey and its confidentiality provisions to their local constituency.
• Research ethics officers and/or Ethics Review Boards, where necessary, reviewed and approved the survey instrument and methodology.
• OCUL determined that individual institutions and their institution-specific data collected during the survey periods would not be disclosed. Individual data was anonymous.
• The mandatory nature of the survey required discussion on some campuses and caused one OCUL member library to withdraw from the study.
• Two institutions pre-tested the survey in January, 2004. Data collection programming and configurations/links had to be revised in February and March, 2004.
• After completing the survey, users were connected to their desired Scholars Portal networked electronic resource.
http://www.minesforlibraries.orghttp://www.arl.org/stats
Issues with web surveys
• Research design– Coverage error
• Unequal access to the Internet• Internet users are different than non-users
– Response rate • Response representativeness
– Random sampling and inference– Non-respondents
• Data security
http://www.minesforlibraries.orghttp://www.arl.org/stats
MINES strategy• A representative sampling plan, including sample size, is
determined at the outset. Typically, there are 48 hours of surveying over 12 months at a medical library and 24 hours a year at a main library.
• Random moment/web-based surveys are employed at each site.
• Participation is usually mandatory, negating non-respondent bias, and is based on actual use in real-time.
• Libraries with database-to-web gateways or proxy re-writers offer the most comprehensive networking solution for surveying all networked services users during survey periods.
http://www.minesforlibraries.orghttp://www.arl.org/stats
MINES strategy (continued)• Placement
– Point of use– Not remembered, predicted or critical incident
• Usage rather than user– What about multiple usages– Time out ?– Cookie or other mechanism with auto-population
• Distinguish patron association with libraries. – For example, medical library v. main library. – But what if the resources are purchased across
campus for all. Then how to get patron affiliation?
http://www.minesforlibraries.orghttp://www.arl.org/stats
Web Survey Design Guidelines
• Web survey design guidelines that MINES followed: – Presentation
• Simple text for different browsers – no graphics– Different browsers render web pages differently
• Few questions per screen or simply few questions• Easy to navigate• Short and plain• No scrolling• Clear and encouraging error or warning messages• Every question answered in a similar way - consistent
– Radio buttons, drop downs• ADA compliant• Introduction page or paragraph• Easy to read
– Must see definitions of sponsored research. • Can present questions in response to answers – for example if
sponsored research was chosen, could present another survey
http://www.minesforlibraries.orghttp://www.arl.org/stats
Quality Checks• Target population is the population frame – surveyed the patrons
who were supposed to be surveyed - except in libraries with outstanding open digital collections.
• Check usage against IP. In this case, big numbers may not be good. May be seeing the survey too often.
• Alter order of questions and answers, particularly sponsored and instruction.
• Spot check IP against self-identified location• Spot check undergraduates choosing sponsored research –
measurement error• Check self-identified grant information against actual grants • Content validity – discussed with librarians and pre-tested. • Turn-aways – number who elected not to fill out the survey• Library information architecture -- Gateway v. HTML pages – there
is a substantial difference in results.
http://www.minesforlibraries.orghttp://www.arl.org/stats
Mandatory – UConn Libraries (3 months)
Mandatory Sponsored
All Networked Electronic Services Use
Research Instruction Other Total
In-Library 21 186 90 297 19.04%Off-Campus 49 138 146 333 21.35%On-Regional Campus 35 77 22 134 8.59%On-Main Campus 174 498 124 796 51.03%
Total 279 899 382 1560 100.00%Total as a Percentage 17.88% 57.63% 24.49% 100.00%
http://www.minesforlibraries.orghttp://www.arl.org/stats
Optional – UConn Libraries (3 months)
Optional Sponsored
All Networked Electronic Services Use
Research Instruction Other Total
In-Library 19 106 37 162 19.13%Off-Campus 12 148 66 226 26.68%On-Regional Campus 8 41 29 78 9.21%On-Main Campus 66 262 53 381 44.98%
Total 105 557 185 847 100.00%Total as a Percentage 12.40% 65.76% 21.84% 100.00%
http://www.minesforlibraries.orghttp://www.arl.org/stats
Issues with web surveys:brief bibliography
• Cook, Colleen; Heath, Fred; and Russell L. Thompson. 2000 (December). “A Meta-Analysis of Response Rates in Web- or Internet-Based Surveys.” Educational and Psychological Measurement 60(6): 821-836.
• Couper, Mick P.; Traugott, Michael W.; and Lamias, Mark J. 2001. "Web Survey Design and Administration," Public Opinion Quarterly, 65 (2): 230-253.
• Covey, Denise Troll. . 2002. Usage and Usability Assessment: Library Practices and Concerns. CLIR Publication 105. Washington DC: Council on Library and Information Resources.
– http://www.clir.org/pubs/reports/pub105/contents.html• Dillman, D.A. 2000 (December). Mail and Internet Surveys, The Tailored Design Method. 2nd
Ed. New York: John Wiley & Sons. • Gunn, Holly. 2002. “Web-based Surveys: Changing the Survey Process.” FirstMonday 7(12).
– http://firstmonday.org/issues/issue7_12/gunn/index.html• LIBQUAL+ ™ Spring 2004 Survey. 2004. Cook, Colleen, and others.
– http://www.libqual.org/documents/admin/ARL_Notebook2004.pdf• Schonlau, Matthias; Fricker Jr., Ronald D.; and Elliott, Marc N. 2002. Conducting Research
Surveys via E-Mail and the Web. Santa Monica, CA: RAND.• Tenopir, Carol, with the assistance of Brenda Hitchcock and Ashley Pillow. 2003 (August). Use
and Users of Electronic Library Resources: An Overview and Analysis of Recent Research Studies. Washington DC: Council on Library and Information Resources.
– http://www.clir.org/pubs/reports/pub120/contents.htmls• Thomas, Susan J. 2004. Using Web and Paper Questionnaires for Data-Based Decision
Making: From Design to Interpretation of the Results. Thousand Oaks, Corwin Press. • Thompson, Bruce.; Cook, Colleen.; Thompson, Russell L. 2002. Reliability and Structure of
LibQUAL+™ scores: Measuring Perceived Library Service Quality. portal: Libraries and the Academy.3-12.
http://www.minesforlibraries.orghttp://www.arl.org/stats
OCUL MINES What do the data tells us?
http://www.minesforlibraries.org
Martha KyrillidouDirector of the ARL Statistics and Measurement Program
Association of Research Libraries
www.arl.org/stats/
MINES for Libraries
http://www.minesforlibraries.orghttp://www.arl.org/stats
• How extensively do sponsored researchers use OCUL’s Scholars Portal? How much usage is for non-funded research, instruction/education, student research papers, and course work?
• Are researchers more likely to use the Scholars Portal from inside or outside the library? What about other classifications of users?
• Are there differences in Scholars Portal based on the user’s location (e.g., in the library; on-campus, but not in the library; or off-campus)?
• Could MINES, combined with usage counts, provide an infrastructure to make Scholars Portal usage studies routine, robust, and easily integrated into OCUL’s administrative decision-making process for assessing networked electronic resources?
Questions Addressed By MINES for Libraries™ for the
OCUL Scholars Portal
http://www.minesforlibraries.orghttp://www.arl.org/stats
MINES for LibrariesTM Survey Form Five Questions and a Comment Box
http://www.minesforlibraries.orghttp://www.arl.org/stats
Analysis
• Web deliverables:– Crosstabulations in html for all OCUL data– Interactive crosstabs for all OCUL and
institutions
• Print deliverables:– summary tables for OCUL – summary tables for each institution– Final report
http://www.minesforlibraries.orghttp://www.arl.org/stats
Affiliation by Purpose of Use
Purpose of Use
Affiliation CourseworkOther
ActivitiesOther
ResearchPatient
Care Sponsored Teaching Total
Applied Sciences 24.0% 7.6% 17.7% 0.6% 46.3% 3.7% 100.0%
Business 34.8% 7.6% 30.0% 0.9% 10.8% 16.0% 100.0%
Education 40.9% 5.4% 17.1% 0.8% 11.8% 24.0% 100.0%
Environmental Studies 43.5% 2.5% 24.0% 0.3% 23.3% 6.3% 100.0%
Fine Arts 56.3% 6.9% 20.6% 1.3% 5.6% 9.4% 100.0%
Humanities 51.5% 10.8% 21.0% 0.5% 9.5% 6.7% 100.0%
Law 67.5% 6.8% 12.8% 0.9% 2.6% 9.4% 100.0%
Medical Health 29.7% 5.5% 18.4% 8.6% 32.0% 5.7% 100.0%
Other 51.9% 22.8% 10.9% 2.1% 7.4% 5.0% 100.0%
Sciences 44.6% 9.7% 11.1% 0.4% 31.8% 2.4% 100.0%
Social Sciences 62.6% 4.5% 14.4% 0.7% 13.6% 4.2% 100.0%
Total 42.0% 7.5% 16.2% 2.4% 26.2% 5.6% 100.0%
http://www.minesforlibraries.orghttp://www.arl.org/stats
User Status by Purpose of Use
Purpose of Use
User Status CourseworkOther
ActivitiesOther
ResearchPatient
Care Sponsored Teaching Total
Faculty 1.5% 4.7% 21.2% 4.4% 42.6% 25.6% 100.0%
Graduate Professional 19.5% 3.9% 25.5% 2.5% 45.4% 3.2% 100.0%
Library Staff 23.5% 24.1% 13.1% 16.5% 17.7% 5.2% 100.0%
Other 6.0% 35.2% 20.8% 8.7% 26.8% 2.5% 100.0%
Staff 3.5% 9.5% 20.6% 2.1% 51.6% 12.7% 100.0%
Undergraduate 75.8% 7.8% 7.7% 0.9% 5.9% 1.9% 100.0%
Total 42.0% 7.5% 16.2% 2.4% 26.2% 5.6% 100.0%
http://www.minesforlibraries.orghttp://www.arl.org/stats
Location by Purpose of Use
Purpose of Use
Location CourseworkOther
ActivitiesOther
ResearchPatient
Care Sponsored Teaching Total
Library 52.8% 14.9% 10.8% 1.2% 12.3% 7.9% 100.0%
Off-campus 47.2% 7.0% 17.3% 4.1% 19.9% 4.6% 100.0%
On-campus 29.2% 4.0% 17.9% 0.9% 42.2% 5.7% 100.0%
Total 42.0% 7.5% 16.2% 2.4% 26.2% 5.6% 100.0%
http://www.minesforlibraries.orghttp://www.arl.org/stats
Reason for Use
Reason for Use (n=20293) Frequency Reason for Use (n=20293) Percent
Important Journal 10219 Important Journal 50.4%
Recommended Colleague 2436 Recommended Colleague 12.0%
Reference/Citation 6090 Reference/Citation 30.0%
Recommended Librarian 620 Recommended Librarian 3.1%
Course Reading 925 Course Reading 4.6%
Other 4388 Other 21.6%
http://www.minesforlibraries.orghttp://www.arl.org/stats
How extensively do sponsored researchers use OCUL’s Scholars Portal? How much usage is for non-funded research, instruction/education, student research papers, and course work?
MINES for Libraries™shows that the Scholars Portal resources are heavily used by faculty and students in all OCUL. The majority of the use is from the sciences and the medical field and particularly in those fields the majority of the use is for sponsored research purposes.
http://www.minesforlibraries.orghttp://www.arl.org/stats
Are researchers more likely to use the Scholars Portal from inside or outside the library? What about other classifications of users?
Most faculty, graduate professionals and undergraduates uses of the Scholars Portal are from outside the library building. Undergraduates though do show many uses of the Scholars Portal from within the library as they are probably becoming more exposed to these resources by having more physical contact with the library.
http://www.minesforlibraries.orghttp://www.arl.org/stats
Are there differences in Scholars Portal based on the user’s location (e.g., in the library; on-campus, but not in the library; or off-campus)?
Most of the faculty and graduate professionals use Scholars Portal either from on-campus locations outside the library or from other off-campus locations. Most of the uses from these locations outside the library are for sponsored research purposes.
http://www.minesforlibraries.orghttp://www.arl.org/stats
Could MINES, combined with usage counts, provide an infrastructure to make Scholars Portal usage studies routine, robust, and easily integrated into OCUL’s administrative decision-making process for assessing networked electronic resources?
Yes
http://www.minesforlibraries.orghttp://www.arl.org/stats
Discussion
• How do we allocate expenditures for electronic resources?
• How do we allocate indirect costs for electronic resources?
• What is the appropriate balance between electronic and print?
• What is the appropriate balance between centralized and distributed purchasing?
• How are electronic resources affecting learning and research outcomes?
http://www.minesforlibraries.orghttp://www.arl.org/stats
OCUL MINES Possible Future Directions
http://www.minesforlibraries.org
Brinley FranklinVice Provost for University Libraries
University of Connecticut
www.arl.org/stats/
MINES for Libraries
http://www.minesforlibraries.orghttp://www.arl.org/stats
OCUL MINES Possible Future Directions
• Individual Institutional Analysis
• Longitudinal Data Collection
• Beyond the Scholar’s Portal
http://www.minesforlibraries.orghttp://www.arl.org/stats
OCUL Scholars Portal Users by Purpose of Use
(n=20,300)
8%
6%
2%
16%
26%
42%
Sponsored Research
Coursework
Other Research
Other Activities
Teaching
Patient Care
http://www.minesforlibraries.orghttp://www.arl.org/stats
64%
36%
Inside the Library
Outside the Library
21%
79%80%
20%
U.S. Main LibrariesTotal Users
n = 25,698
U.S. Medical Libraries
Total Usersn = 31,883
OCUL (Canada)
LibrariesTotal Users
n = 20,300
MINES for LibrariesTM
Location of Electronic Resources Users
http://www.minesforlibraries.orghttp://www.arl.org/stats
MINES for LibrariesTM
Demographics by Location of UserOntario Council of University Libraries
14%
14%
69%
3%
Graduate Students
Faculty, Staff,Research Fellows
UndergraduateStudents
All Other Users
45%
24%
29%
2%
31%
16%
48%
5%
Off-Campusn = 9,163
Inside the Libraryn = 4,047
On Campus, Not in the Libraryn = 7,090
http://www.minesforlibraries.orghttp://www.arl.org/stats
Beyond the Scholar’s Portal
• DD/ILL– ILLiad – enable at the ILLiad logon screen
• Ask Reference– Enable at the Ask Reference page or icon
• Digital libraries– Represent an enormous investment– Primary clientele is outside the library. – Introduces non-authenticated group
http://www.minesforlibraries.orghttp://www.arl.org/stats
Beyond the Scholar’s Portal
• Online catalog– 856 field
• Serials solutions– List of ejournals
• Referrer server– Create a passthrough gateway
• Mirrored web server– Drop in mirrored HTML page with survey links
at survey period• Mirrored HTML pages enabled by scripts
http://www.minesforlibraries.orghttp://www.arl.org/stats
Beyond the Scholar’s Portal
• Because of the point of use requirement, libraries that have either a virtual gateway in library web architecture (or mount their own files like OCUL or OhioLink) succeed the best.
• Rewriting proxy server
• Database-to-web solutions
• Serials Solutions
• OpenURL solutions are a gateway.
http://www.minesforlibraries.orghttp://www.arl.org/stats
Beyond the Scholar’s Portal
• Networked electronic resources are accessible from many different web pages and web servers
• Patrons bookmark networked electronic resources locally on their own workstations.
• Academic departments, librarian liaisons, anyone with a web page copies and pastes library links into their own web sites
• The survey data must be collected and commensurable for all networked electronic resources, including e-journals, e-books, online databases or traditional library request services offered in the online environment, such as Interlibrary Loan.
• The results of the survey have to be uninfluenced by caching issues, both local, web browser caching and proxy server or Internet Service Provider caching.
• The survey has to be meaningful for networked electronic resources, no matter how they were implemented.
• Different authentication methods have to be accommodated, whether the institution used IP, password, referring URL, or an authentication and access gateway.
• Remote usage has to be measured, regardless of the channel of communication, whether locally implemented proxy server, modem pool, or other institutional service.
http://www.minesforlibraries.orghttp://www.arl.org/stats
September 25-27, 2006
Charlottesville, VA, USA
Recommended