Perl/DBI - accessing databases from Perl

Dr. Andrew C.R. Martinmartin@biochem.ucl.ac.ukhttp://www.bioinf.org.uk/

Aims and objectivesUnderstand the need to access

databases from PerlKnow why DBI?Understand the structure of DBIBe able to write a Perl/DBI script

to read from or write to a databasePRACTICAL: write a script to read

from a database

Why access a database from Perl?

CGIScript

Why access a database from Perl?

Send request for page to web server

Pages

ExternalPrograms

RDBMS

Web browserWeb server

CGI can extract parameters sent with the page request

Why access a database from Perl?

Populating databasesNeed to pre-process and re-format

data into SQLIntermediate storage during data

processing

Reading databasesGood database design can lead to

complex queries (wrappers)Need to extract data to process it

Why use DBI?

Why Perl/DBI?

Many relational databases available.Commercial examples:

OracleDB/2SQLServerSybaseInformixInterbase

Open source examples:PostgreSQLmySQL

Why Perl/DBI?Databases use a common query

language: ‘structured query language’ (SQL)

Queries can easily be ported between different database softwareMinor variations in more advanced

features

Proprietary extensions

Why Perl/DBI?Can call command-line interface

from within your program.

$result = `psql -tqc “SELECT * FROM table” `;

@tuples = split(/\n/, $result);foreach $tuple (@tuples){ @fields = split(/\|/, $tuple);}

Inefficient: new process for each database access

Why Perl/DBI?Databases generally provide own

APIs to allow access from programming languagese.g. C, Java, Perl

Proprietary APIs all differ

Very difficult to port software between databases

Standardized APIs have thus become available

Why Perl/DBI?Perl/DBI is the standardized API for

Perl

Easy to port Perl scripts from one database to another

DBI and ODBCODBC (Open DataBase Connectivity)Consortium of vendors in early 1990s

SQL Access GroupOctober 1992 & 1993, draft

standard:‘Call Level Interface’ (‘CLI’ - an API)Never really adopted

Microsoft ‘embraced and extended’ it to create ODBC

DBI and ODBC‘DBPerl’ designed as database

interface specifically for Perl4September 1992 (i.e. pre-ODBC)

Just before release, Perl5 announced with OO facilities

DBPerl modified to support OO and loosely modelled on CLI standard

This became DBI

DBI and ODBC

Standard SQL syntax Dodged this issue!

Standard error codes Check $DBI::err or $DBI::errstr (DBI provides methods for standard errors, but drivers don’t use them)

Meta-data on database structure

Tables and types only

Many attributes and options to tweak underlying driver

Very limited control

ODBC DBI

DBI and ODBCThere is an ODBC driver for DBI DBI can be used to access any

ODBC database

The structure of DBI

DBI Architecture

Oracledriver

mySQLdriver

PostgreSQLdriver

Sybasedriver

DBI

DBI ArchitectureMulti-layer design

Perl script

DBI

DBD

Database API

RDBMS

DatabaseIndependent

DatabaseDependent

DBD::Oracle

DBD::Oracle

DBD::Oracle

PerlScript

PerlScript

DBI Architecture

PerlScript DBI DBD::mysql

DBD::Oracle

DBD::Pg

Oracle

mySQL

PostgreSQL

DBI Architecture

Returns a list of installed (DBD) drivers

@drivers = DBI->available_drivers();

DBI Architecture

DBD::OracleDriver Handle

DBD::pgDriver Handle

DatabaseHandle

DatabaseHandle

DatabaseHandle

StatementHandle

StatementHandle

StatementHandle

StatementHandle

StatementHandle

StatementHandle

StatementHandle

DBI Architecture

Driver Handles

References loaded driver(s)One per driver

Not normally referenced in programs

Standard variable name $drh

DBI ArchitectureDatabase HandlesCreated by connecting to a database

References a database via a driverCan have many per database (e.g.

accessing different user accounts)Can access multiple databasesStandard variable name $dbh

$dbh = DBI->connect($datasource, ... );

DBI Architecture

Statement HandlesCreated by ‘preparing’ some SQL

Can have many per database handle (e.g. multiple queries)

Standard variable name $sth

$sth = $dbh->prepare($sql);

DBI Architecture

DBD::pgDriver Handle

DatabaseHandle

StatementHandle

$dbh = DBI->connect($datasource, ... );

$sth = $dbh->prepare($sql);

SQL Preparation

prepare()

Perl script DBI and Driver Database

Pass statementto database engine

$sth

Encapsulate as DBI

statement handle

ParseStatement

EncapsulateStatement

if valid

Pass statementto database engine

execute()

fetchrow_array()

Maintain cursorinto results

ExecuteStatement

Writing Perl/DBI scripts

Accessing DBI from PerlMust have the DBI package and

appropriate DBD package installedDBD::Oracle, DBD::Pg, DBD::mysql, etc.

use DBI;

$dbh = DBI->connect($datasource, $username, $password);

Create a ‘handle’ to access the database:

The DBD moduleand d/b to be used

Data sources

$dbname = “mydatabase”;

$dbserver = “dbserver.cryst.bbk.ac.uk”;$dbport = 5432;

$datasource = “dbi:Oracle:$dbname”;$datasource = “dbi:mysql:database=$dbname;host=$dbserver”;$datasource = “dbi:Pg:dbname=$dbname;host=$dbserver;port=$dbport”;

Format varies withdatabase module

$dbh = DBI->connect($datasource, $username, $password);

Optional; supported by some databases.

Default: local machine and default port.

Username and password

$username and $password also optionalOnly needed if you normally need a

username/password to connect to the database.

Remember CGI scripts run as a special web-server user. Generally, ‘nobody’ or ‘apache’.

Database must allow access by this useror specify a different username/password

$dbh = DBI->connect($datasource, $username, $password);

SQL commands with no return valueSQL commands other that SELECT don’t

return values may return success/failure flagnumber of entries in the database affected

For example: creating a table inserting a row modifying a row

SQL commands with no return value

From Perl/DBI:

INSERT INTO idac VALUES (‘LYC_CHICK’, ‘P00698’)

e.g. insert a row into a table:

$sql = “INSERT INTO idac VALUES (‘LYC_CHICK’, ‘P00698’)”;$dbh->do($sql);

SQL commands that return a single rowSometimes, can guarantee that a

database query will return only one rowor you are only interested in the first row

$sql = “SELECT * FROM idac WHERE ac = ‘P00698’”;@values = $dbh->selectrow_array($sql);

Columns placed in an array

Could also have been placed in a list:

($id, $ac) = $dbh->selectrow_array($sql);

SQL commands that return a multiple rows

Most SELECT statements will return many rows

Three stages must be performed:

preparing the SQL

executing it

extracting the results

SQL commands that return a multiple rows

(Can also obtain array or hash reference) NB: statement handle / fetchrow_array rather than db handle / selectrow_array

$sql = “SELECT * FROM idac”;

$sth = $dbh->prepare($sql);if($sth->execute){ while(($id, $ac) = $sth->fetchrow_array) { print “ID: $id AC: $ac\n”; }}

SQL commands that return a multiple rows If you need to stop early you can do:

$sql = “SELECT * FROM idac”;

$sth = $dbh->prepare($sql);if($sth->execute){ for($i=0; $i<10; $i++) { if(($id, $ac) = $sth->fetchrow_array) { print “ID: $id AC: $ac\n”; } } $sth->finish;}

SQL commands that return a multiple rowsA utility method is also available to print

a complete result set:

$sql = “SELECT * FROM idac”;$sth = $dbh->prepare($sql);if($sth->execute){ $nrows = $sth->dump_results;}

(Mostly useful for debugging)

Repeated SQL callsOften want to repeat essentially the

same query, but with some different value being checked.

For example:

foreach $ac (‘P00698’, ‘P00703’){ $sql = “SELECT * FROM idac WHERE ac = ‘$ac’”; @values = $dbh->selectrow_array($sql); print “@values\n”;}

(using special option for 1-row returns)

Repeated SQL callsCould also be do:

foreach $ac (‘P00698’, ‘P00703’){ $sql = “SELECT * FROM idac WHERE ac = ‘$ac’”; $sth = $dbh->prepare($sql); $sth->execute; while(@values = $sth->fetchrow_array) { print “@values\n”; }}

i.e. don’t use special option for 1-row returns $dbh->selectrow_array($sql)

Repeated SQL calls Increase in performance by ‘binding’ a

variable:$sql = “SELECT * FROM idac WHERE ac = ?”;$sth = $dbh->prepare($sql);

foreach $ac (‘P00698’, ‘P00703’){ $sth->bind_param(1, $ac); $sth->execute; while(@values = $sth->fetchrow_array) { print “@values\n”; }}

Parameter Number

Variableto bind

Repeated SQL callsNOTE:Performance increase depends on

database and driverAlthough strings normally enclosed in

single inverted commas, the bound variable is not quoted.

If you have a number which you need to be treated as a string, then you do:

$sth->bind_param(1, 42, SQL_VARCHAR);

SummaryDBI provides a standard API It does not standardize the SQLDBI is an older standard than ODBC

They can be used together and they are both evolving

Basic 3-step process:prepare / execute / fetch

Shortcut calls for no return or 1-row return

Many other functions available