Http:// MINES for Libraries™ Presented by Martha Kyrillidou Director of the ARL Statistics and...

http://www.minesforlibraries.orghttp://www.arl.org/stats

MINES for Libraries™

Presented by

Martha KyrillidouDirector of the ARL Statistics and Service Quality Programs

Association of Research Libraries

Rutgers University LibraryJune 1 2007

New Brunswick, NJ

www.arl.org/stats/

ARL Overall

Libraries Remain a Credible Resource in Libraries Remain a Credible Resource in 2121stst Century Century

98% agree with statement98% agree with statement, “My … library , “My … library contains information from credible and contains information from credible and known sources.”known sources.”

Note. Digital Library Federation and Council on Library and InformationResources. (2002). Dimensions and Use of the Scholarly Information Environment.

Changing BehaviorsChanging Behaviors

Only Only 15.7% agreed with the statement15.7% agreed with the statement “The “The Internet has not changed the way I use the Internet has not changed the way I use the library.”library.”

Note. Digital Library Federation and Council on Library and InformationResources. (2002). Dimensions and Use of the Scholarly Information Environment.

ARL Toolkit…

• StatsQUAL+™– ARL Statistics – LibQUAL+®– E-Metrics– DigiQUAL+™

– MINES for Libraries™

What is MINES?

• Action research– Historically rooted in indirect cost studies– Set of recommendations for research design– Set of recommendations for web survey

presentation– Set of recommendations for information

architecture in libraries– Plan for continual assessment of networked

electronic resources– An opportunity to benchmark across libraries

MINES for LibrariesTM

• MINES is a transaction-based research methodology consisting of a web-based survey form and a random moments sampling plan

• MINES typically measures who is using electronic resources, where users are located at the time of use, and their purpose of use in the least obtrusive way

• MINES was adopted by the Association of Research Libraries (ARL) as part of the “New Measures” toolkit in May, 2003.

• MINES is different from other electronic resource usage measures that quantify total usage (e.g., Project COUNTER, E-Metrics) or measure how well a library makes electronic resources accessible (LibQUAL+™).

Questions Addressed By MINES for Libraries™ for the OCUL Scholars Portal

• How extensively do sponsored researchers use OCUL’s Scholars Portal? How much usage is for non-funded research, instruction/education, student research papers, and course work?

• Are researchers more likely to use the Scholars Portal from inside or outside the library? What about other classifications of users?

• Are there differences in Scholars Portal based on the user’s location (e.g., in the library; on-campus, but not in the library; or off-campus)?

• Could MINES, combined with usage counts, provide an infrastructure to make Scholars Portal usage studies routine, robust, and easily integrated into OCUL’s administrative decision-making process for assessing networked electronic resources?

MINES for LibrariesTM Survey Form Five Questions and a Comment Box

Methodological considerationsExperience with the MINES Survey

Terry PlumAssistant Dean

Simmons GSLIS

Rutgers UniversityJune 1 , 2007

Issues with web surveys

• Research design– Coverage error

• Unequal access to the Internet• Internet users are different than non-users

– Response rate • Response representativeness

– Random sampling and inference– Non-respondents

• Data security

MINES strategy• A representative sampling plan, including sample size, is

determined at the outset. Typically, there are 48 hours of surveying over 12 months at a medical library and 24 hours a year at a main library.

• Random moment/web-based surveys are employed at each site.

• Participation is usually mandatory, negating non-respondent bias, and is based on actual use in real-time.– IRB waiver or approval

• Libraries with database-to-web gateways or proxy re-writers offer a comprehensive networking solution for surveying all networked services users during survey periods.

MINES strategy• Placement

– Point of use– Not remembered, predicted or critical incident

• Usage rather than user– What about multiple usages– Time out ?– Cookie or other mechanism with auto-population or

more recently counting invisibly with a time out. • Distinguish patron association with libraries.

– For example, medical library v. main library. – But what if the resources are purchased across

campus for all. Then how to get patron affiliation?

Web Survey Design Guidelines

• Web survey design guidelines that MINES followed: – Presentation

• Simple text for different browsers – no graphics– Different browsers render web pages differently

• Few questions per screen or simply few questions• Easy to navigate• Short and plain• No scrolling• Clear and encouraging error or warning messages• Every question answered in a similar way - consistent

– Radio buttons, drop downs• ADA compliant• Introduction page or paragraph• Easy to read

– Must see definitions of sponsored research. • Can present questions in response to answers – for example if

sponsored research was chosen, could present another survey

Quality Checks• Target population is the population frame – surveyed the patrons

who were supposed to be surveyed - except in libraries with outstanding open digital collections.

• Check usage against IP. In this case, big numbers may not be good. May be seeing the survey too often.

• Alter order of questions and answers, particularly sponsored and instruction.

• Spot check IP against self-identified location• Spot check undergraduates choosing sponsored research –

measurement error• Check self-identified grant information against actual grants • Content validity – discussed with librarians and pre-tested. • Turn-aways – number who elected not to fill out the survey• Library information architecture -- Gateway v. HTML pages – there

is a substantial difference in results.

Documenting the Purpose and Use of Electronic Resources:Experience with the MINES Survey

Brinley FranklinVice Provost for University Libraries

University of Connecticut

Rutgers UniversityJune 1 , 2007

“It is useless to tell the acquisitions librarian that half the monographs ordered will never be used,unless we can specify which 50% to avoid buying.” (Galvin and Kent, 1977)

Reliance on Vendor Statistics

Vendor statistics, while more reliable than in the past, are still maturing.

Measuring Digital Content Use

The most popular current method of measuring usage of electronic resources by libraries is not through web-based usage surveys, but through vendor supplied data of library patron usage or transaction usage.

Web-based usage surveys are increasingly relevant in the collection of usage data to make collection development and service decisions, to document evidence of usage by certain patron populations, and to collect and analyze performance outputs.

Brinley Franklin and Terry Plum, “Successful Web Survey Methodologies for Measuring the Impact of Networked Electronic Services (MINES for LibrariesTM)” IFLA Journal 32 (1) March, 2006

A web-based transactional survey that collects data on users’ demographics and their purpose of use. It is administered in real time over the course of at least a year using a random moments sampling plan.

MINES for LibrariesTM has been administered at 40 North American universities in the last four years. More than 100,000 North American networked services users have been surveyed using a standard protocol.

Library User SurveyPatron Status

Library User SurveyAffiliation

Library User SurveyLocation

Library User SurveyPurpose

Sample Survey Data File Generated

Other UConnInstruction/Education/Departmental (Non-Funded) Researchhttp://newfirstsearch.oclc.org/done=referer;dbname=WorldCat;autho=100122319;FSIP12:36:5012/3/2004Off CampusUConn Faculty12.101.104.84

Family StudiesInstruction/Education/Departmental (Non-Funded) Researchhttp://www.jstor.org/cgi-bin/jstor/gensearch12:37:4312/3/2004Off CampusUConn Undergraduate Student12.101.104.84Non-UConnOther Activitieshttp://homerweb.lib.uconn.edu/cgi-bin/Pwebrecon.cgi?DB=local&PAGE=First13:08:4112/3/2004Off CampusNon-UConn12.18.36.40

Non-UConnInstruction/Education/Departmental (Non-Funded) Researchhttp://magic.lib.uconn.edu/index_real.html13:31:2912/3/2004Off CampusNon-UConn12.76.131.246

Non-UConnOther Activitieshttp://magic.lib.uconn.edu/index_real.html12:11:0612/3/2004Off CampusNon-UConn130.132.86.156Agriculture & Natural ResourcesInstruction/Education/Departmental (Non-Funded) Researchhttp://magic.lib.uconn.edu/index_real.html12:33:5712/3/2004

Off CampusNon-UConn131.128.89.54EducationInstruction/Education/Departmental (Non-Funded) Researchhttp://www.euromonitor.com/womdas/12:57:4412/3/2004

Off CampusNon-UConn134.241.135.70Non-UConnInstruction/Education/Departmental (Non-Funded) Researchhttp://homerweb.lib.uconn.edu/cgi-bin/Pwebrecon.cgi?DB=local&PAGE=First13:28:5212/3/2004

Off CampusNon-UConn134.74.42.82Business AdministrationOther Activitieshttp://homerweb.lib.uconn.edu/cgi-bin/Pwebrecon.cgi?DB=local&PAGE=First12:56:4612/3/2004In the LibraryUConn Faculty137.99.1.122

Liberal Arts & SciencesOther Activitieshttp://www.siam.org/journals/simax/simax.htm12:52:1712/3/2004On Campus - StorrsUConn Graduate Student137.99.1.122

EngineeringInstruction/Education/Departmental (Non-Funded) Researchhttp://homerweb.lib.uconn.edu/cgi-bin/Pwebrecon.cgi?DB=local&PAGE=First12:04:3112/3/2004On Campus - StorrsUConn Graduate Student137.99.1.219

Business AdministrationInstruction/Education/Departmental (Non-Funded) Researchhttp://proquest.umi.com/pqdweb?RQT=31812:16:3312/3/2004On Campus - StorrsUConn Graduate Student137.99.100.105

Business AdministrationInstruction/Education/Departmental (Non-Funded) Researchhttp://www.jstor.org/journals/07322399.html12:16:5212/3/2004On Campus - StorrsUConn Graduate Student137.99.100.105

Business AdministrationInstruction/Education/Departmental (Non-Funded) Researchhttp://homerweb.lib.uconn.edu/cgi-bin/Pwebrecon.cgi?DB=local&PAGE=First12:29:5312/3/2004On Campus - StorrsUConn Graduate Student137.99.100.175

Business AdministrationInstruction/Education/Departmental (Non-Funded) Researchhttp://homerweb.lib.uconn.edu/cgi-bin/Pwebrecon.cgi?DB=local&PAGE=First12:48:4112/3/2004On Campus - StorrsUConn Graduate Student137.99.100.175

Business AdministrationInstruction/Education/Departmental (Non-Funded) Researchhttp://proquest.umi.com/login?COPT=SU5UPTAmVkVSPTImREJTPTE3MjErMysxNkJD&clientId=4899612:04:2312/3/2004On Campus - StorrsUConn Graduate Student137.99.100.175

Demographics by Location of UserU.S. Main Libraries

GraduateStudents

Faculty, Staff,Research Fellows

UndergraduateStudents

All Other Users

On Campus, Not in the Libraryn = 6,391

Inside the Libraryn = 9,172

Off-Campusn = 4,953

MINES for Libraries™

Graduate Students

Faculty, Staff,Research Fellows

UndergraduateStudents

All Other Users

Off-Campusn = 9,163

Demographics by Location of UserOntario Council of University Libraries

Inside the Libraryn = 4,047

On Campus, Not in the Libraryn = 7,090

Purpose of Use

Are users engaged in coursework, funded (or unfunded) research, public service, patient care, or other activities?

SponsoredResearch

Instruction

OtherSponsoredActivities

Purpose of Use By LocationU.S. Main Campus Libraries

2003 – 2005

*72% of sponsored research usage of electronic resources occurred outside the library; 83% took place on campus.

On-Campus, Not in the Libraryn = 9,460

In the Libraryn = 9,733

Off-Campusn = 7,790

Overall Usen = 26,983

OCUL Scholars Portal Users Purpose of Use

In a sample of 20,300 electronic resources uses at OCUL libraries,

there were four uses outside the library for each use in the library.

8%6%2%

26% 42% Sponsored Research

Coursework

Other Research

Other Activities

Teaching

Patient Care

Questions?

Learn more about LibQUAL+TM, DigiQUALTM, & MINES for LibrariesTM at: www.statsqual.org

Analysis

• Web deliverables:– Crosstabulations in html for all OCUL data– Interactive crosstabs for all OCUL and

institutions

• Print deliverables:– summary tables for OCUL – summary tables for each institution– Final report

OCUL Scholars Portal UsageAffiliation

Affiliation by Purpose of Use

Purpose of Use

Affiliation CourseworkOther

ActivitiesOther

ResearchPatient

Care Sponsored Teaching Total

Applied Sciences 24.0% 7.6% 17.7% 0.6% 46.3% 3.7% 100.0%

Business 34.8% 7.6% 30.0% 0.9% 10.8% 16.0% 100.0%

Education 40.9% 5.4% 17.1% 0.8% 11.8% 24.0% 100.0%

Environmental Studies 43.5% 2.5% 24.0% 0.3% 23.3% 6.3% 100.0%

Fine Arts 56.3% 6.9% 20.6% 1.3% 5.6% 9.4% 100.0%

Humanities 51.5% 10.8% 21.0% 0.5% 9.5% 6.7% 100.0%

Law 67.5% 6.8% 12.8% 0.9% 2.6% 9.4% 100.0%

Medical Health 29.7% 5.5% 18.4% 8.6% 32.0% 5.7% 100.0%

Other 51.9% 22.8% 10.9% 2.1% 7.4% 5.0% 100.0%

Sciences 44.6% 9.7% 11.1% 0.4% 31.8% 2.4% 100.0%

Social Sciences 62.6% 4.5% 14.4% 0.7% 13.6% 4.2% 100.0%

Total 42.0% 7.5% 16.2% 2.4% 26.2% 5.6% 100.0%

User Status by Purpose of Use

Purpose of Use

User Status CourseworkOther

ActivitiesOther

ResearchPatient

Faculty 1.5% 4.7% 21.2% 4.4% 42.6% 25.6% 100.0%

Graduate Professional 19.5% 3.9% 25.5% 2.5% 45.4% 3.2% 100.0%

Library Staff 23.5% 24.1% 13.1% 16.5% 17.7% 5.2% 100.0%

Other 6.0% 35.2% 20.8% 8.7% 26.8% 2.5% 100.0%

Staff 3.5% 9.5% 20.6% 2.1% 51.6% 12.7% 100.0%

Undergraduate 75.8% 7.8% 7.7% 0.9% 5.9% 1.9% 100.0%

Total 42.0% 7.5% 16.2% 2.4% 26.2% 5.6% 100.0%

Location by Purpose of Use

Purpose of Use

Location CourseworkOther

ActivitiesOther

ResearchPatient

Library 52.8% 14.9% 10.8% 1.2% 12.3% 7.9% 100.0%

Off-campus 47.2% 7.0% 17.3% 4.1% 19.9% 4.6% 100.0%

On-campus 29.2% 4.0% 17.9% 0.9% 42.2% 5.7% 100.0%

Total 42.0% 7.5% 16.2% 2.4% 26.2% 5.6% 100.0%

Reason for Use

Reason for Use (n=20293) Frequency Reason for Use (n=20293) Percent

Important Journal 10219 Important Journal 50.4%

Recommended Colleague 2436 Recommended Colleague 12.0%

Reference/Citation 6090 Reference/Citation 30.0%

Recommended Librarian 620 Recommended Librarian 3.1%

Course Reading 925 Course Reading 4.6%

Other 4388 Other 21.6%

Library Assessment Conference

August 4-6, 2008

Seattle, WA, USA

Http:// MINES for Libraries™ Presented by Martha Kyrillidou Director of the ARL Statistics and...

Documents

TEXAS A&M UNIVERSITY LIBRARIES Colleen Cook, Texas A&M University Martha Kyrillidou, Association of Research Libraries

SPACE: DESCRIBING & ASSESSING LIBRARY & OTHER LEARNING … · 2016-11-17 · space: describing & assessing library & other learning spaces bob fox steve hiller martha kyrillidou joan

ARL Statistics and Measurement Program Effective, Sustainable and Practical Assessment Martha Kyrillidou Director, ARL Statistics and Service Quality Programs

LibQUAL+ ® Survey Introduction Presented by: Martha Kyrillidou Association of Research Libraries LibQUAL+® and Beyond The University of

Kyrillidou final

Www.arl.org Outcome Assessment Tools for the Library of the Future ACRL Conference 2005 April 7, 2005 Minneapolis, MN Martha Kyrillidou Director, ARL Statistics

LibQUAL+ ® Survey Administration LibQUAL+® Exchange Northumbria Florence, Italy August 17, 2009 Presented by: Martha Kyrillidou Senior Director, Statistics

LibQUAL+™ Process Overview Using the Web as a Management Tool Martha Kyrillidou LibQUAL+™ Canada Meeting Ottawa, Canada June 14, 2006 old.libqual.org

Preparing to Work Effectively with LibQUAL+ ® Survey Results ALA Annual Conference Washington, DC June 26, 2007 Raynna Bowlby Martha Kyrillidou Association

Library Quality Assessment through LibQUAL+ ® IATUL 2009 Leuven, Belgium June 2, 2009 Presented by Martha Kyrillidou Director, Statistics and Service Quality

LibQUAL+ ® 2007: An Introduction American Library Association (ALA) Midwinter Meeting Seattle, WA January 22, 2007 MaShana Davis and Martha Kyrillidou

LibQUAL+ ® in South Africa Sherrie Schmidt, Arizona State and ARL President Martha Kyrillidou, ARL Stellenbosch, South Africa August 13, 2007 old.libqual.org

Performance Measurement in Academic Libraries Martha Kyrillidou, ARL Steve Hiller, University of Washington Jim Self, University of Virginia EBLIP 4 Workshop

LibQUAL+® Survey Introduction · LibQUAL+® Survey Introduction Presented by: Martha Kyrillidou Association of Research Libraries Michael Maciel, Texas A&M University Libraries American

Crystal Reports - Ind - 2015 · University of Massachusetts Amherst Association of Research Libraries / Texas A&M University 2015 Survey 12,536 Colleen Cook Martha Kyrillidou

LibQUAL+® Survey Results · LibQUAL+® Survey Results Presented by: Martha Kyrillidou Amy Yeager Association of Research Libraries American Library Association Midwinter Meeting

An Introduction to LibQUAL+™ New Ways of Listening to Users Washington, DC November 5, 2005 Martha Kyrillidou Amy Hoseth Jonathan Sousa old.libqual.org

Capturing Digital Developments through Qualitative Inquiry...Capturing Digital Developments through Qualitative Inquiry Martha Kyrillidou, ARL Colleen Cook, Texas A&M University Libraries

Utah State University · Utah State University Association of Research Libraries / Texas A&M University Contributors Colleen Cook Martha Kyrillidou Texas A&M University Association

Making Library Assessment Work Progress Report of an ARL Project Steve Hiller University of Washington Jim Self University of Virginia & Martha Kyrillidou