Data Warehousing and Mining Data from Library

and University Systems for Assessment of Library

OperationsUnderstanding Library Systems and Software Applications

School of Communication and Information, Rutgers University, New Brunswick, New Jersey, Thursday, October 25, 2012

Ray Schwartz, Systems Specialist Librarian

Cheng Library, William Paterson University, Wayne, New Jersey, USAschwartzr2 @ wpunj.edu

2

Outline• What is Data Mining and Data

Warehousing and Why Do We Do It?• Our Library and University• Patron Statistical Categories• Application Server• Reporting

3

What is Data Mining and Data Warehousing

• Extracting data from legacy systems and other resources;

• cleaning, scrubbing and preparing data for decision support;

• maintaining data in appropriate data stores; • accessing and analysing data using a variety

of end user tools; • and mining data for significant relationships.

• Chaffey, D., Mayer, R., Johnston, K., & Ellis-Chadwick, F. (2002). Internet Marketing: Strategy, Implementation and Practice (2nd ed.). Financial Times/ Prentice Hall.

4

• The primary purpose of these efforts is to provide easy access to specifically prepared data that can be used with decision support applications such as management reports, queries, decision support systems, executive information systems and data mining.

• Chaffey, D., Mayer, R., Johnston, K., & Ellis-Chadwick, F. (2002). Internet Marketing: Strategy, Implementation and Practice (2nd ed.). Financial Times/ Prentice Hall.

5

Our University• 9000 undergraduates• 1000 graduates (mostly education

majors)• 400 faculty• 800 adjuncts• 1000 staff

6

Our Library• 19 librarians and 26 library staff• 350,000 volumes• 18,000 audiovisual items• 47,000 print and electronic periodicals • 124 general and subject specific

databases• $1,100,000 Non-Salary Allocations

7

Our Transactions• 600,000 Database Searches• 413,000 Gate Counts• 40,000 Library Materials Circulation• 34,000 Equipment Circulation• 19,000 Reference Queries• 3,000 Interlibrary Loans • 5,000 Documents Delivered

8

Our Systems• Voyager ILS • Clio ILL Software• EZProxy Server• Banner – University ERP• University Networked Drive K:• University Email Server• University Web Server

9

Vendor Services• Serials Solutions

• A to Z list• MARC Record Service• Link Resolver

• OCLC – Bibliographic Utility• Worldcat Collection Analysis

• Coutts (was Blackwell) – Book Jobber

• Ebsco – Subscription Agent• Marcive – Authority Control• Database Vendors

10

Voyager Overdue and Fine Notices - Daily

11

Quarterly Extract for Serials Solutions AtoZ

Service

12

What would we like to see?• Breakdowns by department and

majors.

• Combined usage by department/majors of more than one library service.

• Which categories of patrons are accessing which services?

13

• Voyager Patron Database allows a maximum of 10 statistical categories per patron record.

• Worked with our University Information Systems Department to extract the relevant data from the relevant sources.

• Weekly extract from SIS and HRS to load into

Voyager

Patron Statistical Categories

14

Groups and Services• Major• Status

– Undergrad or Grad– Faculty, Adjunct Faculty

or Staff• Department• College• Degree• No. of Credits• Year of Study• Campus Location

• Circulation– Books– Media– Reserve– By Fund Code– Location

• ILL / Document Delivery• Databases• Library Web Pages

– Subject Area Resource Guides

– Reference Requests• Catalog• Other Vendor Services

– Serials Solutions

• College and Mercer Identifier• Class Level (Freshman, Sophomore, Junior, Senior, Graduate)• Total Hours Registered for Current Semester• Major• 2nd Major• Degree• CA-Collection Agency• SOILS• Student Entrance Level (New Non-Traditional Freshman, New

First Time Transfer, etc.)• Department

From Students

From Faculty / Staff / Adjuncts

• College• Full or Part-Time• Status (Faculty, Adjunct, Staff, Professional Staff, Tenured,

Tenure-Track)• Division• Departments

17

History Department - 12 months -Feb. 2008

Library Total = declared undergrad & grad majors, adjuncts & full time faculty borrowers

BORROWER = any member who borrowed materials

MEMBER = declared major or department member

EQUIPMENT CIRCULATION = camcorders, overhead & data projectors, laptops, easels, DVD players, etc.

MEDIA CIRCULATION = audio & video materials, including media reserves

BOOK CIRCULATION = books, book disks, maps, oversize, Curriculum materials, reserve books, NJ History, Leisure Lounge

DEFINITIONS:

10.597.1167% 4,981 7,418 52,756 20,703 8,713 23,370 LIBRARY TOTALS

19.9315.6679% 242 308 4,824 988 443 3,393 HISTORY TOTALS

20.3519.5096% 23 24 468 194 115 159 FULL-TIME FACULTY

9.255.7863% 20 32 185 20 65 100 ADJUNCT FACULTY

39.0836.2993% 13 14 508 76 13 419 GRADUATE STUDENTS

19.6915.3978% 186 238 3,663 698 250 2,715 UNDERGRADUATE STUDENTS

CIRC/ BORROWER

CIRC/ MEMBER

% BORROW

INGBORROWERSMEMBERSTOTAL CIRCEQUIP CIRCMEDIA CIRCBOOK CIRCPATRON STATUS

18

Communications Majors FY08/09Statistical Categories // Item Type / Location / Call No Type / Call No

Communications Majors Freshman Sophomore Junior Senior

M- DVD / Media Services / Other / DVD 194 17 31 52 94M- VideoCass / Media Services / Other / VC 228 11 40 67 110T- Book / 2nd Floor - Circulating / Library of Congress / B 34 9 8 11 6T- Book / 2nd Floor - Circulating / Library of Congress / BD 3 1 2T- Book / 2nd Floor - Circulating / Library of Congress / BF 30 5 5 12 8... 2nd Floor Circulating 1531 222 310 403 596T- Juvenile / CMC / 125 14 26 20 35T- NJDoc / Askew Documents Room / Other / 1 1New Jersey History 10 0 2 7 1T- ReserveBk / Reserves Desk / 189 13 46 68 62T- SpecColl / Special Collection / Library of Congress / LC 3 3 T- Book-McNaughton / Leisure Lounge / Library of Congress / F 2 1 1T- Book-McNaughton / Leisure Lounge / Library of Congress / HF 1 1 T- Book-McNaughton / Leisure Lounge / Library of Congress / HS 2 2 T- Book-McNaughton / Leisure Lounge / Library of Congress / HV 5 1 2 2T- Book-McNaughton / Leisure Lounge / Library of Congress / ML 1 1 T- Book-McNaughton / Leisure Lounge / Library of Congress / PN 3 3 T- Book-McNaughton / Leisure Lounge / Library of Congress / PS 29 4 10 15T- Book-McNaughton / Leisure Lounge / Library of Congress / RC 2 1 1T- Book-McNaughton / Leisure Lounge / Library of Congress / TL 1 1Leisure Lounge 49 9 1 19 20

19

Challenges with combining data from various services• Little to no linkage of data

• Multiple user IDs for authentication

20

Application Server• A machine or its software that

works in conjunction with a web server to deliver application services such as the dynamic creation of a webpage from content stored in a database. From http://www.webtools.ca.gov/help/Glossary.asp• Web Server Software (Apache or IIS)

• Database Management System – DBMS (MySQL, Oracle, MS SQL Server)

• Scripting Language (Perl, PHP, ColdFusion, ASP)

21

Why an Application Server?• Relevant data in logfiles need to

be in a database to be analyze.

• Need your own DBMS to create new tables and queries.

22

Authentication of ILL and other forms are routed through the EZProxy server

23

Daily and Weekly Email Reports from the

Application ServerCirc Fines Audit Daily Report - Daily at 6:05 AM. Dupe Patron Record Report - Daily at 5:56 AM. Hobart Media Services Equipment Pickup Summary - Daily at 6:58 AM. Media Service Scheduling Rooms Report - Daily at 6:02 AM. Media Services Equipment Pickup Summary - Daily at 7:00 AM. Received Title Alert - Daily at 6:59 AM. Reserves Overdues - Daily at 5:59 AM. Scheduled LIS Tasks - Daily at 6:00 AM.

ILL Borrowing Overdues Report - Weekly at 5:59 AM. ILL Lending Reports - Weekly at 6:15 AM.

24

Monthly Email Reports from the Application ServerCirc Fines Audit - Monthly at 6:10 AM. Circulation by Location and Item Type - Monthly at 6:21 AM. Circulation Lost and Paid - Monthly at 6:25 AM. Circulation Online Renewal Count - Monthly at 6:30 AM. Media Circulation - Monthly at 6:35 AM. Reserve Circulation - Monthly at 6:40 AM.

25

26

On Demand Reports

27

Lists of patrons with fines between $10 and $19.99 • Student and Alumni fines list - Sorted by either Name, Amount or Notice

Date.• PALS and Courtesy Patron fines list - Sorted by Name.• All other Patron fines list - Sorted by Name.    Lists of patrons with fines over $19.99 • Student and Alumni fines list - Sorted by either Name, IID, Amount, Notice

Date or Notes.• PALS and Courtesy Patron fines list - Sorted by Name.• VALE Patron fines list - Sorted by Name.• All other Patron fines list - Sorted by Name.    Lists of patrons with overdues older than 30 days • Student and Alumni overdues list - Sorted by either Name, IID or Notes.• PALS and Courtesy Patron overdues list - Sorted by Name. • All other Patron overdues list except VALE - Sorted by Name.

Lending Services Reports

28

Lists of VALE patrons with overdues older than 6 months • VALE patron overdues list - Sorted by Name. Miscellaneous Reports • Patrons with the word "Collection Agency" or "CA" in their notes.• Patrons with the word "FINE" in one of their notes. • Patrons with the word "SOILS" in their notes. • Patrons with the word "FALL07 SOILS" in their notes. • Patrons with the word "HOLD" in their notes. • Combined list of HOLD, FINE, and CA. Circulation Reports by Item Type from 2003 to the present• All Staff.• All Colleges • Undergraduates by Major. • Graduates by Major • Patrons that have reached a total fine balance of $10 or more after 31-

Dec-2009 and 30-Nov-2009 

Lending Services Reports, cont.

29

One of Our Projects• Mining EZProxy logfiles and linking to

patron statistical categories from the Voyager Patron Database

– What majors and departments are accessing which database services?

– What majors and departments are accessing the ILL services?

EZProxy via LDAP authenticates

access to:DatabasesElectronic journalsILL/Doc Delivery forms

31

Example of EZProxy log entrynj.dhcp.embarqhsd.net

-

theuser

1/1/2008 4:25:15 AM

GET

http://ezproxy.wpunj.edu:2048/connect?session=sGHMbeSss121YxZa&url=http://www.wpunj.edu/scripts/webscript.exe?fs.scr

HTTP/1.1

302

537

http://ezproxy.wpunj.edu:2048/login?url=http://www.wpunj.edu/scripts/webscript.exe?fs.scr

Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; .NET CLR 1.1.4322)

• Ip address

• (Not used)

• user id

• date/time

• Method

• page retrieved

• Version

• response code

• no. of bytes

• Referring URL

• User agent

32

Patron Privacy and Standards

33

Using Voyager as the model for Patron Privacy

34

• Active Circ transactions are stored in a table with patron ID and linked to statistical categories.

• Completed Circ transactions are moved to another table without the patron ID, but still linked with the patron statistical categories.

• The Patron Table contains the total counts of transactions for each patron, but no link to which transactions they are.

• Extract patron statistical categories out of Voyager and build them into the MySQL database.

• EZProxy transactions would be stored in one table and linked to patron statistical categories via the user ID.

• Once completed, user ids are deleted.• Logs are collected monthly and loaded and

deleted monthly.35

MySQL operations

Slide removed for Privacy Reasons

Slide removed for Privacy Reasons

38

48

ILL request form authentications by majorMajor

90M- History28M- Non-Degree25M- Pub Pol & Intl Affairs20M- Spanish18M- English16M- Undecided14M- Art14M- Education11M- Sociology10M- Biology

9M- Music9M- Special Programs8M- Psychology7M- Biotechnology7M- Political Science6M- Anthropology6M- Music - Jazz Studies4M- Business4M- Communication4M- Nursing

Book CountMajor

62M- Psychology60M- Sociology42M- Applied Clinical Psych35M- Education31M- History30M- Spanish29M- Nursing

1919M- Communication14M- Biotechnology14M- Counseling14M- English12M- Non-Degree10M- Community/Sch Health

7M- Biology7M- Political Science6M- Undecided5M- Comm Media Studies5M- Reading4M- Business

Article Count

M- Communication Disorders

49

50

Reporting and Standards• Reporting

– Emailed periodically - e.g., daily dossiers, and other event triggered reports.

– On demand, via email, web pages or a printer.

• Standards– Share data for comparative research. – Groups of libraries and consortia

51

52

53

Further Reading• Coombs, Karen A. (2005). Lessons learned from analyzing

library database usage data. Library Hi Tech, 23:4, 598.

• Diana, Birkin James. dashboard_beta. http://library.brown.edu/dashboard/info/

• Metridoc. http://code.google.com/p/metridoc/

• Morton-Owens, Emily (2011) Trends at a glance. LITA 2011. http://connect.ala.org/files/79651/trends_at_a_glance_dashboards_pdf_12068.pdf

55

Questions?

Ray Schwartz, Systems Specialist Librarian

Cheng Library, William Paterson University,

Wayne, New Jersey, USAschwartzr2 @ wpunj.edu