Perl DBI Scripting with the ILS

Roy Zimmer

Western Michigan University

What do I need for database interactions with Perl?

What do I need for database interactions with Perl?

DBI – DataBase Interface (always)

What do I need for database interactions with Perl?

DBI – DataBase Interface (always)

DBD – DataBase Driver (one for each type of database you must access)

What do I need for database interactions with Perl?

DBI – DataBase Interface (always)

DBD – DataBase Driver (one for each type of database you must access)

a database? (some DBDs let you access non-database data, such as CSV files)

What do I need for database interactions with Perl?

DBI – DataBase Interface (always)

DBD – DataBase Driver (one for each type of database you must access)

a database? (some DBDs let you access non-database data, such as CSV files)

some Perl proficiency

What do I need for database interactions with Perl?

DBI – DataBase Interface (always)

DBD – DataBase Driver (one for each type of database you must access)

a database? (some DBDs let you access non-database data, such as CSV files)

some Perl proficiency

But we only care about using our Voyager database, so we’ll stick with that.

What does this mean to you?

What does this mean to you?

DBI – DataBase Interface (already on your Voyager box)

What does this mean to you?

DBI – DataBase Interface (already on your Voyager box)

DBD – DataBase Driver (for , already on your Voyager box)

What does this mean to you?

DBI – DataBase Interface (already on your Voyager box)

DBD – DataBase Driver (for , already on your Voyager box)

a database? (already on your Voyager box)

What does this mean to you?

DBI – DataBase Interface (already on your Voyager box)

DBD – DataBase Driver (for , already on your Voyager box)

a database? (already on your Voyager box)

some Perl proficiency (the only thing you supply!)

A simple program

Always needed for database access

nameletters.pl

A simple program

Get access information from a setup file and connect to a database.

File format (the only record):

library.box.university.edu/username/password/VGER

nameletters.pl

A simple program

Connect to the database and get a handle.

nameletters.pl

A simple program…for each letter, how many patrons’ last names start with that letter

Create the query

sprintf is your friend

A sample query here is:

select count (*) from wmichdb.patron where last_name like ‘S%’

nameletters.pl

A simple program…for each letter, how many patrons’ last names start with that letter

Prepare the query, associating it with a database, giving it a handle.

nameletters.pl

A simple program…for each letter, how many patrons’ last names start with that letter

Run the query, get a return code.

nameletters.pl

A simple program…for each letter, how many patrons’ last names start with that letter

Get the query result…

nameletters.pl

A simple program…for each letter, how many patrons’ last names start with that letter

…and print it.

nameletters.pl

A simple program…for each letter, how many patrons’ last names start with that letter

A: 003016B: 007113C: 005041D: 003792E: 001322F: 002605G: 003603H: 005388I: 000368J: 001970K: 004371L: 003763M: 007039N: 001622O: 001299P: 003792Q: 000121R: 003770S: 008217T: 002528U: 000248V: 001791W: 004487X: 000018Y: 000562Z: 000709

Output

nameletters.pl

Nested query example…getting barcodes of retirees who’ve borrowed something within the past year

retirees.pl

Nested query example…getting barcodes of retirees who’ve borrowed something within the past year

Outer query

Get patron information that meets the criteria, and connecting information for barcodes.

(sprintf actually not needed here)

retirees.pl

Nested query example…getting barcodes of retirees who’ve borrowed something within the past year

Inner query

Remember the outer query?

Note the positional correspondence of the query parameters with the while statement receiving a query row.

retirees.pl

Nested query example…getting barcodes of retirees who’ve borrowed something within the past year

Inner query

The inner query has a different query string name…

retirees.pl

Nested query example…getting barcodes of retirees who’ve borrowed something within the past year

Inner query

…as do the other variables for this query.

retirees.pl

Nested query example…getting barcodes of retirees who’ve borrowed something within the past year

Inner query

Doe, John R doe1 pgroup: EMERIT/RET

21141002502810 08/16/1999 Other

99952803601000 11/25/2001 Expired

Doe, Josephine doe2 pgroup: RETIREDSTF

21141001297289 08/16/1999 Other

21141002300660 08/16/1999 Other

11130755000000 09/04/2002 Expired

retirees.pl

Sample output:

Termination Notes

$sth->finish finish a query

$dbh->disconnect disconnect from a database

finish seems like a good practice, but it’s not really needed. Yet…

disconnect isn’t really needed (for our purposes, read-only). However, without using the finish statement, you could run into error situations.

leading us to…

CPAN

the Comprehensive Perl Archive Network

Great resource for *all* kinds of Perl modules

Documentation for modules

- comprehensively so for DBI and DBD

- about 100 pages for DBI and 50 pages for DBD

A bit more documentation…

$sth->fetchrow_array

returns a single value, array, or list

There are a number of other ways to get query data, but this seems to work best.

You can also get lots of details about your database environment via DBI calls. See the DBI documentation for more information.

A quick little DBI utility

dbddrivers.pl

A quick little DBI utility

Available Perl DBD drivers on this systemDBMExamplePFileGoferOracleProxySponge

Output

dbddrivers.pl

What if you need to look at every bib record you’ve got?

What if you need to look at every bib record you’ve got?

And you need to access the marc record (synonymous with )?

Here’s one approach…

THE BLOB

connect to the database

query bib_data table to get the maximum bib ID

set increment to 50,000

set ending bib ID to increment

set beginning bib ID to 0

Traversing every record and dealing with THE BLOB along the way

connect to the database

query bib_data table to get the maximum bib ID

set increment to 50,000

set ending bib ID to increment

set beginning bib ID to 0

while beginning bid ID < maximum bib ID

chunkthroughdb()

provide feedback of progress

beginning bib ID = ending bib ID + 1

add increment to ending bib ID

end

provide final feedback

Traversing every record and dealing with THE BLOB along the way

sub chunkthrudb

query bib_data for blob data based on bib IDs >= beginning bib ID and

bib IDs < ending bib ID

for each record from the query

assemble each bib ID's data into a marc record

do the required processing for this record

end

end

Traversing every record and dealing with THE BLOB along the way

Traversing every record and dealing with THE BLOB along the way

Discussion of this approach

Alternative method: read the whole database at once!

- might be impossible

- might not be feasible

- probably not efficient

The program presented here…seems to be the better method.

Traversing every record and dealing with THE BLOB along the way

Discussion of this approach

If nothing else, “chunking” your way through your database is more efficient

We have close to 1.6 million bib records. Based on the program we’re about to see, traversing our database…

using 50,000 record chunks takes about 50 minutes

without “chunking” it takes about 76 minutes

Traversing every record and dealing with THE BLOB along the way

findbadleader.pl

Traversing every record and dealing with THE BLOB along the way

Get our boundary condition

findbadleader.pl

Traversing every record and dealing with THE BLOB along the way

0-50,000 – our first chunk of records

findbadleader.pl

Traversing every record and dealing with THE BLOB along the way

the loop

findbadleader.pl

Traversing every record and dealing with THE BLOB along the way

set up the next chunk

the loop

findbadleader.pl

Traversing every record and dealing with THE BLOB along the way

- chunking subroutine (chunkthrudb)

“seqnum desc” is key to getting the blob data for each record

This gets one chunk’s worth of MARC records (blob data)

findbadleader.pl

Traversing every record and dealing with THE BLOB along the way

- chunking subroutine (chunkthrudb)

Creates the MARC record for each bib retrieved.

For larger records that don’t fit in one table row, assemble the record in reverse order (from the query).

findbadleader.pl

Traversing every record and dealing with THE BLOB along the way

- chunking subroutine (chunkthrudb)

Process each record.

In this case, we’re looking for records with bad leaders (not ending in “4500”).

Sample output:

findbadleader.pl

Being in a is a good thing…bind

sprintf is a friend, but bind is a better friend

Being in a is a good thing…bind

sprintf is a friend, but bind is a better friend

usual query method

Being in a is a good thing…bind

sprintf is a friend, but bind is a better friend

usual query method

query with bind values method

testbind.pl

Illustrating and contrasting the query with bind method

create, prepare, and execute a query

testbind.pl

Illustrating and contrasting the query with bind method

prepare and execute a query

testbind.pl

Illustrating and contrasting the query with bind method

prepare and execute a query, improved

prepare moved outside of the loop

testbind.pl

Illustrating and contrasting the query with bind method

Using testbind.pl for about 80,000 patron records, these are the results:

Create, prepare and execute a query

20

Prepare and execute a query (with bind)

11

Prepare and execute a query, improved (with bind)

10

Method Runtime (seconds)

Running reports, queries via Perl on your PC

ActiveState sells products and also has free versions. I recommend their free Perl.

Get the MSI version.

It comes with DBI and DBD for Oracle, among many other modules (and lots of documentation).

Still (!) requires external Oracle client software (at least as of version 5.10.0, build 1003)

The files listed below are available at

http://homepages.wmich.edu/~zimmer/files/eugm2008

dbi.ppt this presentation

nameletters.pl for every letter, find number of patrons whose lastnames begin with that letter

retirees.pl get barcodes of retirees who’ve borrowed somethingin the last year

dbddrivers.pl get list of DBD drivers on your system

findbadleader.pl find bib records with bad leaders (not ending in 4500)

testbind.pl demonstrates varying query method efficiencies

Resources

CPAN http://cpan.org

DBI http://search.cpan.org/~timb/DBI-1.605/DBI.pm

DBD Oracle http://search.cpan.org/~pythian/DBD-Oracle-1.21/Oracle.pm

Active State Perl

http://activestate.com/downloads/index.mhtml

There is also a book on Perl and DBI. I’d recommend

using it along with the most current documentation

from the CPAN sites above.

Resources

Thank you for listening.

Questions?

roy.zimmer@wmich.edu

Picture © 2008 by Roy Zimmer