Topic 8: Databases CSE2395/CSE3395 Perl Programming Llama3 chapter 16, pages 221-224 Camel3 pages 363-398 Programming the Perl DBI chapters 2-5, pages

Topic 8: DatabasesTopic 8: Databases

CSE2395/CSE3395Perl Programming

CSE2395/CSE3395Perl Programming

Llama3 chapter 16, pages 221-224

Camel3 pages 363-398

Programming the Perl DBI chapters 2-5, pages 7-135

perlfunc, perlmod, perlmodlib, Data::Dumper, perltie,

DB_File, DBI manpages

2Original Slides by Debbie Pickett, Modified by David Abramson, 2006, Copyright Monash University

In this topicIn this topic

Modules► use Module;► Perl standard library► CPAN► writing modules

Databases► persistent data► Data::Dumper► DBM

– databases in files► pack and unpack

– flat-file databases► DBI

– interfacing with SQL databases







ModulesModules

Much of Perl’s functionality is contained in optional modules

A module is a file which contains re-usable Perl code for a particular task

Much of Perl’s functionality is contained in optional modules

A module is a file which contains re-usable Perl code for a particular task

Your Perl code

Module

Your code is importing; module is exporting

imported module


ModulesModules

Modules can be► found in the Perl Standard Library

– supplied with Perl and already installed► downloaded from CPAN

– tar xzf Module.tar.gz– cd Module– perl Makefile.PL– make– make install

► written by you– using package keyword and saving file as Module.pm

Modules can be► found in the Perl Standard Library

– supplied with Perl and already installed► downloaded from CPAN

– tar xzf Module.tar.gz– cd Module– perl Makefile.PL– make– make install

► written by you– using package keyword and saving file as Module.pm

Camel3 pages 299-307, 831-915perlmod, perlmodlib, perlfunc manpages


Using a moduleUsing a module

Import a module into your code with use keyword at top of your program

use Data::Dumper; use CGI ":all"; use Dog qw(bark bite); Read module documentation to see how to

access module’s functionality► perldoc Data::Dumper► perldoc CGI

Import a module into your code with use keyword at top of your program

use Data::Dumper; use CGI ":all"; use Dog qw(bark bite); Read module documentation to see how to

access module’s functionality► perldoc Data::Dumper► perldoc CGI

Llama3 pages 282-283Camel3 pages 299-300, 822-823perlfunc manpage


Writing a moduleWriting a module

To create a module called Dog:► start file with package Dog;

– package keyword ensures that Dog’s variables and functions don’t interfere with anyone else’s

► use Exporter;► @ISA = qw(Exporter);► set @EXPORT_OK to a list of function names (bark, bite)

– these two lines allow Perl to put Dog’s functions into code that says use Dog ("bark", "bite");

► write the functions for Dog (bark, bite)► end file with a nonzero expression

– needed or else use Dog will fail► save file as Dog.pm in current directory

– or any directory named in special Perl @INC array

To create a module called Dog:► start file with package Dog;

– package keyword ensures that Dog’s variables and functions don’t interfere with anyone else’s

► use Exporter;► @ISA = qw(Exporter);► set @EXPORT_OK to a list of function names (bark, bite)

– these two lines allow Perl to put Dog’s functions into code that says use Dog ("bark", "bite");

► write the functions for Dog (bark, bite)► end file with a nonzero expression

– needed or else use Dog will fail► save file as Dog.pm in current directory

– or any directory named in special Perl @INC array

Camel3 pages 301-305perlmod, perlfunc manpages


TimeoutTimeout

package DayOfWeek;use Exporter; @ISA = qw(Exporter);@EXPORT_OK = "weekday"; # Allow import of &weekday.

# weekday(y,m,d): Return weekday of date (0 = Sun).sub weekday{ my ($y, $m, $d) = @_; # Uses array references (topic 10). ( [[6,2,2,5,0,3,5,1,4,6,2,4], [6,2,3,6,1,4,6,2,5,0,3,5]] -> [ ((! ($y % 400) || ($y % 100)) && (! ($y % 4))) || 0 ][ $m-1 ] + $d + $y + int(($y-1)/4) - int(($y-1)/100) + int(($y-1)/400) ) % 7;}1; # Indicate successful definition of module.

package DayOfWeek;use Exporter; @ISA = qw(Exporter);@EXPORT_OK = "weekday"; # Allow import of &weekday.

# weekday(y,m,d): Return weekday of date (0 = Sun).sub weekday{ my ($y, $m, $d) = @_; # Uses array references (topic 10). ( [[6,2,2,5,0,3,5,1,4,6,2,4], [6,2,3,6,1,4,6,2,5,0,3,5]] -> [ ((! ($y % 400) || ($y % 100)) && (! ($y % 4))) || 0 ][ $m-1 ] + $d + $y + int(($y-1)/4) - int(($y-1)/100) + int(($y-1)/400) ) % 7;}1; # Indicate successful definition of module.


TimeoutTimeout

# Import the weekday function from DayOfWeek.pmuse DayOfWeek qw(weekday);

# Names of days of week.@days = qw(Sunday Monday Tuesday Wednesday Thursday Friday Saturday);

# Check usage.die "Usage: $0 year month day\n" if @ARGV != 3;

$daynum = weekday(@ARGV[0..2]);

print "That day is $days[$daynum]\n";

# Import the weekday function from DayOfWeek.pmuse DayOfWeek qw(weekday);

# Names of days of week.@days = qw(Sunday Monday Tuesday Wednesday Thursday Friday Saturday);

# Check usage.die "Usage: $0 year month day\n" if @ARGV != 3;

$daynum = weekday(@ARGV[0..2]);

print "That day is $days[$daynum]\n";


DatabasesDatabases

A database► contains organized data► contains relations between data► is persistent

A hash► contains organized data► has a key/value relationship► ceases to exist at end of scope

– never beyond end of program

A persistent hash would be great► but how?

A database► contains organized data► contains relations between data► is persistent

A hash► contains organized data► has a key/value relationship► ceases to exist at end of scope

– never beyond end of program

A persistent hash would be great► but how?


Persistent dataPersistent data

To make a variable persistent, need a way to make it last between invocations of program

► by saving its state in a file

Write-back (“freeze/thaw”, serialization) strategy► dump the contents of variable to file at program exit (“freeze”)► initialize variable with dumped contents at program entry (“thaw”)► file is up-to-date only when program is not running► for example: Data::Dumper, Storable

Write-through (“tie”) strategy► update file every time the variable is changed► requires help from Perl to tie the variable to the file► file is always up-to-date► for example: DBM files

To make a variable persistent, need a way to make it last between invocations of program

► by saving its state in a file

Write-back (“freeze/thaw”, serialization) strategy► dump the contents of variable to file at program exit (“freeze”)► initialize variable with dumped contents at program entry (“thaw”)► file is up-to-date only when program is not running► for example: Data::Dumper, Storable

Write-through (“tie”) strategy► update file every time the variable is changed► requires help from Perl to tie the variable to the file► file is always up-to-date► for example: DBM files


Data::DumperData::Dumper

Perl Standard Library module► use Data::Dumper;

Example of write-back (freeze/thaw) strategy Writes Perl code to file

► when later evaluated by Perl, data is reconstructed To dump (freeze) a variable %hash:

► open the persistent file for writing► Data::Dumper->Dump([\%hash], [qw(*hash)]);

– [...] is array reference, see topic 10.

To restore (thaw) from a file file:► do "file";► Perl evaluates the statements in file, reconstructing the

variable %hash

Perl Standard Library module► use Data::Dumper;

Example of write-back (freeze/thaw) strategy Writes Perl code to file

► when later evaluated by Perl, data is reconstructed To dump (freeze) a variable %hash:

► open the persistent file for writing► Data::Dumper->Dump([\%hash], [qw(*hash)]);

– [...] is array reference, see topic 10.

To restore (thaw) from a file file:► do "file";► Perl evaluates the statements in file, reconstructing the

variable %hash

Camel3 pages 702, 882Data::Dumper, perlfunc manpages


TimeoutTimeout

# Freezing and thawing some variables.use Data::Dumper;

# Thaw saved values, if they exist.if (-f "storage") { do "storage"; # Restore previous values.} else { $important = 0; %valuable = (); # First run.}

# ... do stuff with $important and %valuable ...

# Freeze values for next time before program exit.open STORAGE, ">storage";print STORAGE Data::Dumper->Dump( [$important, \%valuable], [qw(important *valuable)]);

# Freezing and thawing some variables.use Data::Dumper;

# Thaw saved values, if they exist.if (-f "storage") { do "storage"; # Restore previous values.} else { $important = 0; %valuable = (); # First run.}

# ... do stuff with $important and %valuable ...

# Freeze values for next time before program exit.open STORAGE, ">storage";print STORAGE Data::Dumper->Dump( [$important, \%valuable], [qw(important *valuable)]);


DBM filesDBM files

DBM files are a simple database format common to Unix► also available on other platforms

Data stored in a binary file► usually with a sophisticated data structure like a B-

tree

Roughly equivalent to a hash► can look up value given key► keys are strings, values are strings► may be arbitrary limits on data size

DBM files are a simple database format common to Unix► also available on other platforms

Data stored in a binary file► usually with a sophisticated data structure like a B-

tree

Roughly equivalent to a hash► can look up value given key► keys are strings, values are strings► may be arbitrary limits on data size

Llama3 pages 221-222Camel3 pages 696-697, 883-884, 985perlmod manpage


DBM filesDBM files

Perl provides a simple interface to DBM files► using dbmopen or tie functions► connects a hash to the DBM file► accessing the hash transparently accesses the file

instead– all hash operations supported

► example of write-through (tie) strategy To tie a hash %hash to a file file:

► either dbmopen %hash, "file", 0600– mode (0600) only used if file needs to be created

► or tie %hash, "DB_file", "file"

Perl provides a simple interface to DBM files► using dbmopen or tie functions► connects a hash to the DBM file► accessing the hash transparently accesses the file

instead– all hash operations supported

► example of write-through (tie) strategy To tie a hash %hash to a file file:

► either dbmopen %hash, "file", 0600– mode (0600) only used if file needs to be created

► or tie %hash, "DB_file", "file"

Llama3 pages 222-224Camel3 pages 363-398, 697-698perltie, perlfunc manpages


TimeoutTimeout

# Use DBM to track mail aliases.

# Use Berkeley-DB format for DBM file.# This is the most flexible one.use DB_File;dbmopen %alias, "aliases", 0600 or die "Can't tie to DBM file: $!\n";

# Define functions that modify file# through %alias hash.sub get_alias { return $alias{$_[0]}; }sub set_alias { $alias{$_[0]} = $_[1]; }sub delete_alias { delete $alias{$_[0]}; }sub exists_alias { return exists $alias{$_[0]}; }

# Use DBM to track mail aliases.

# Use Berkeley-DB format for DBM file.# This is the most flexible one.use DB_File;dbmopen %alias, "aliases", 0600 or die "Can't tie to DBM file: $!\n";

# Define functions that modify file# through %alias hash.sub get_alias { return $alias{$_[0]}; }sub set_alias { $alias{$_[0]} = $_[1]; }sub delete_alias { delete $alias{$_[0]}; }sub exists_alias { return exists $alias{$_[0]}; }


pack and unpackpack and unpack

Files store only strings If data other than strings are to be stored, need to

encode into strings before storing► and decode from strings when fetching

To encode data into a string, use pack► $string = pack "a10 l", $word, $int;

To decode data from a string, use unpack► ($word, $int) = unpack "a10 l", $string;

Format string ("a10 l") specifies how to pack or unpack

► dozens of different specifiers

Files store only strings If data other than strings are to be stored, need to

encode into strings before storing► and decode from strings when fetching

To encode data into a string, use pack► $string = pack "a10 l", $word, $int;

To decode data from a string, use unpack► ($word, $int) = unpack "a10 l", $string;

Format string ("a10 l") specifies how to pack or unpack

► dozens of different specifiers

Llama3 page 224Camel3 pages 757-762, 819-821perlfunc manpage


Random-access filesRandom-access files

Can store fixed-length records in a file► e.g., 30 bytes per record

– 20 for name, 8 for ID number, 1 for age, 1 for mark► bytes 0 to 29 for first record► bytes 30 to 59 for second record► etc.

Can store fixed-length records in a file► e.g., 30 bytes per record

– 20 for name, 8 for ID number, 1 for age, 1 for mark► bytes 0 to 29 for first record► bytes 30 to 59 for second record► etc.

record 1 record 2 record 3


Random-access filesRandom-access files

Open file for both reading and writing To modify or create record

► create record with pack► seek to position in file with seek► write packed record to file with print

To read record► seek to position in file with seek► read appropriate number of bytes with read► extract fields from string with unpack

Open file for both reading and writing To modify or create record

► create record with pack► seek to position in file with seek► write packed record to file with print

To read record► seek to position in file with seek► read appropriate number of bytes with read► extract fields from string with unpack

Llama3 pages 225-227Camel3 pages 769, 780perlfunc manpage


TimeoutTimeout

open FILE, "+< database"; # Open for read/write.

# write_record(recno, name, id, age, mark)sub write_record { my $recno = shift; my $string = pack "a20 a8 c c", @_; seek FILE, ($recno-1)*30, 0; print FILE $string;}

# (name, id, age, mark) = read_record(recno)sub read_record { my $recno = shift; seek FILE, ($recno-1)*30, 0; read FILE, $string, 30; return unpack "a20 a8 c c", $string;}

open FILE, "+< database"; # Open for read/write.

# write_record(recno, name, id, age, mark)sub write_record { my $recno = shift; my $string = pack "a20 a8 c c", @_; seek FILE, ($recno-1)*30, 0; print FILE $string;}

# (name, id, age, mark) = read_record(recno)sub read_record { my $recno = shift; seek FILE, ($recno-1)*30, 0; read FILE, $string, 30; return unpack "a20 a8 c c", $string;}


Client/server databasesClient/server databases

Sometimes necessary to interact with an existing database► database is kept on a server

– e.g., Oracle, MySQL, Microsoft Access (ODBC), etc.– server handles mundane aspects like storage, access control,

file/record locking, etc.► your program is client

– makes requests to server on database’s behalf

Need to interact with server using its interface► usually requires making a network connection

Commonly done with SQL► Structured Query Language, a standard way of phrasing queries

and database transformations► SQL is not examinable in this subject!

But► SQL dialects vary slightly► network protocols and interfaces vary

Sometimes necessary to interact with an existing database► database is kept on a server

– e.g., Oracle, MySQL, Microsoft Access (ODBC), etc.– server handles mundane aspects like storage, access control,

file/record locking, etc.► your program is client

– makes requests to server on database’s behalf

Need to interact with server using its interface► usually requires making a network connection

Commonly done with SQL► Structured Query Language, a standard way of phrasing queries

and database transformations► SQL is not examinable in this subject!

But► SQL dialects vary slightly► network protocols and interfaces vary


DBIDBI

Perl provides a unified interface for all database servers► called Database Interface, or DBI► use DBI;

Client uses DBI to access database► DBI speaks to database through a system-specific

database driver (DBD)► DBD translates client’s SQL request for specific

server dialects► DBD massages server responses into standard form

for client

Perl provides a unified interface for all database servers► called Database Interface, or DBI► use DBI;

Client uses DBI to access database► DBI speaks to database through a system-specific

database driver (DBD)► DBD translates client’s SQL request for specific

server dialects► DBD massages server responses into standard form

for client


DBIDBI

Your Perl program

DBI (interface)

DBD::Oracle (driver)

DBD::CSV (driver)

DBI system

Oracle database

Oracle database

CSV database

Always one instance of DBI

One driver per type of database


DBI handlesDBI handles

DBI system uses handles to manage connections to database system► database handle

– one per database– used to perform instantaneous actions and to initiate longer

statements► statement handle

– one per SQL statement– usually used to perform lengthy SELECT queries

DBI system uses handles to manage connections to database system► database handle

– one per database– used to perform instantaneous actions and to initiate longer

statements► statement handle

– one per SQL statement– usually used to perform lengthy SELECT queries


Connecting to a databaseConnecting to a database

Need to know database’s data source name Need to know database’s data source name

DBI:mSQL:frotz.org:base:4910

Always DBI

Database driver

Database identifier; format differs for each driver (here: hostname,

database name, port number


TimeoutTimeout

use DBI;

# Connect to the database.# OO: class->method() notation is how to call# a class method (in this case, a constructor).my $db_handle = DBI->connect( "DBI:Ingres:data", $username, $passwd, { PrintError => 1 } # Hash ref: topic 10 ) or die "Can't connect: $DBI::errstr\n";

# ... do stuff ...

# Now disconnect.# OO: $object->method() notation is how to call# an object method (here, a destructor).$db_handle->disconnect() or warn "Disconnection error: $DBI::errstr\n";

use DBI;

# Connect to the database.# OO: class->method() notation is how to call# a class method (in this case, a constructor).my $db_handle = DBI->connect( "DBI:Ingres:data", $username, $passwd, { PrintError => 1 } # Hash ref: topic 10 ) or die "Can't connect: $DBI::errstr\n";

# ... do stuff ...

# Now disconnect.# OO: $object->method() notation is how to call# an object method (here, a destructor).$db_handle->disconnect() or warn "Disconnection error: $DBI::errstr\n";


Executing DBI statementsExecuting DBI statements

Actions on database are performed through database handle

Simple (atomic) statements can be performed using do method► $db_handle->do("DELETE FROM ...");► return value indicates success or failure

Complex statements, which may produce several results in sequence, require a statement handle► $st_handle = $db_handle->prepare("SELECT...");► statement handle used to iterate over results

– @fields = $st_handle->fetchrow_array()

Actions on database are performed through database handle

Simple (atomic) statements can be performed using do method► $db_handle->do("DELETE FROM ...");► return value indicates success or failure

Complex statements, which may produce several results in sequence, require a statement handle► $st_handle = $db_handle->prepare("SELECT...");► statement handle used to iterate over results

– @fields = $st_handle->fetchrow_array()


TimeoutTimeout

# Insert a name into a phone database# usage: thisprogram name number

use DBI;

# ... connect to the database ...

# quote() method correctly escapes strings# with metacharacters in them.$db_handle->do(" INSERT INTO phonedir (name, phone_number) VALUES ( " . $db_handle->quote($ARGV[0]) . ", " . $db_handle->quote($ARGV[1]) . ")") or warn "Insert error: $DBI::errstr\n";

# Insert a name into a phone database# usage: thisprogram name number

use DBI;


# quote() method correctly escapes strings# with metacharacters in them.$db_handle->do(" INSERT INTO phonedir (name, phone_number) VALUES ( " . $db_handle->quote($ARGV[0]) . ", " . $db_handle->quote($ARGV[1]) . ")") or warn "Insert error: $DBI::errstr\n";


TimeoutTimeout

# Extract entries from the phone database# usage: thisprogram name


$st_handle = $db_handle->prepare(" SELECT name, phone_number FROM phonedir WHERE name LIKE " . $db_handle->quote($ARGV[0]));$st_handle->execute();

# Iterate over the data returned by the query.while (($name, $num) = $st_handle->fetchrow_array()){ print "$name: $num\n";}

# Extract entries from the phone database# usage: thisprogram name


$st_handle = $db_handle->prepare(" SELECT name, phone_number FROM phonedir WHERE name LIKE " . $db_handle->quote($ARGV[0]));$st_handle->execute();

# Iterate over the data returned by the query.while (($name, $num) = $st_handle->fetchrow_array()){ print "$name: $num\n";}


Covered in this topicCovered in this topic












Going furtherGoing further

Tying► adding new behaviour to built-in-types► perldoc -f tie

Transactions► protecting databases from corruption► supported by some DBD modules

Multi-level DBM files► storing nested data structures in a DBM file► MLDBM module

ODBC► Microsoft’s database standard► DBI::ODBC and Win32::ODBC modules

Tying► adding new behaviour to built-in-types► perldoc -f tie

Transactions► protecting databases from corruption► supported by some DBD modules

Multi-level DBM files► storing nested data structures in a DBM file► MLDBM module

ODBC► Microsoft’s database standard► DBI::ODBC and Win32::ODBC modules


Next topicNext topic

The World Wide Web Writing a Perl web client

► LWP module

Dynamic web pages► Common Gateway Interface (CGI)

The World Wide Web Writing a Perl web client

► LWP module

Dynamic web pages► Common Gateway Interface (CGI)

LWP, CGI manpages

Documents

Topic 8: Databases CSE2395/CSE3395 Perl Programming Llama3 chapter 16, pages 221-224 Camel3 pages 363-398 Programming the Perl DBI chapters 2-5, pages