1

541: Database Systems

S. Muthu Muthukrishnan

2

Course Project Pick a dataset.

Stock market data, US patent data, web data, internet traffic data. UC Irvine data repository. http://odwin.ucsd.edu/idata/ Set of conf papers: http://www.acm.org/sigs/sigmod/record/xml/ Medical, ecological, biological, text, movie database. Rutgers labs..

How to collect it? How to make it up? HW 0: Decide by 02/26. Submit a writeup of what data, how

you will collect it, how much, what application you will build—what queries are important, what challenges you foresee, schedule+timeline and how you are going to divide work, etc.

Midterm project review 03/25. Experiment with different indices, join methods, different ways of posing queries, schemas, etc.

Project demo and project writeup due: 04/22. Check out http://paul.rutgers.edu/~eiman/cs541_fall03.html for details.

3

Homeworks

HW 1:1 Write a short note about yourself: your educational background, interest in d/b, career goals, anything you’d like to bring to my attention.

EX 1-2: Data procuring. Can you build a web crawler to pull data into a flat file. (Extra Credit).

We looked at an overview of DBMS. Question:

ER diagrams.

4

Exercise Consider Rutgers Univ data comprising information about

various departments, courses and students, and determine the number of bytes needed to store this data for one year. Write a report discussing what is the data you considered, how you estimated the size of various data sets, how you estimated the number of bytes needed to store them.

Solve problem 2.5 in the book. Describe major design decisions. Puzzles.

Size? Problems? How does it grow?

5

Data(base) Compressions

Homework: Study data compression, write about issues in database compression versus data compression and table compression in data warehouses.

Use www.cs.wisc.edu/~joldst/ compressing relations and indexes.

Use Table compression paper.

How to compress the web data?

6

Homework II

Solve problem 2.5 in the book. Describe major design decisions.

7

The Relational Model

Chapter 3

Most widely used in commerical databases.

8

Relational Database: Definitions Relational database: a set of relations Relation: made up of 2 parts:

Instance : a table, with rows and columns. #Rows = cardinality, #fields = degree / arity.

Schema : specifies name of relation, plus name and type of each column. E.G. Students(sid: string, name: string, login: string,

age: integer, gpa: real).

Can think of a relation as a set of rows or tuples (i.e., all rows are distinct).

9

Example Instance of Students Relation

sid name login age gpa

53666 Jones jones@cs 18 3.4

53688 Smith smith@eecs 18 3.2

53650 Smith smith@math 19 3.8

Cardinality = 3, degree = 5, all rows distinct Do all columns in a relation instance have to

be distinct?

Field names

Fields/attributes/columns

Records/Tuples/Rows

10

Relational Query Languages

A major strength of the relational model: supports simple, powerful querying of data.

Queries can be written intuitively, and the DBMS is responsible for efficient evaluation. The key: precise semantics for relational queries. Allows the optimizer to extensively re-order operations,

and still ensure that the answer does not change.

11

The SQL Query Language

Developed by IBM (system R) in the 1970s Need for a standard since it is used by many

vendors Standards:

SQL-86 SQL-89 (minor revision) SQL-92 (major revision, current standard) SQL-99 (major extensions)

12

The SQL Query Language

To find all 18 year old students, we can write:

SELECT *FROM Students SWHERE S.age=18

•To find just names and logins, replace the first line:

SELECT S.name, S.login

sid name login age gpa

53666 Jones jones@cs 18 3.4

53688 Smith smith@ee 18 3.2

*: All fields. S: Variable over tuples.

13

Querying Multiple Relations

What does the following query compute?

SELECT S.name, E.cidFROM Students S, Enrolled EWHERE S.sid=E.sid AND E.grade=“A”

S.name E.cid

Smith Topology112

sid cid grade53831 Carnatic101 C53831 Reggae203 B53650 Topology112 A53666 History105 B

Given the following instance of Enrolled:

we get:

14

Creating Relations in SQL

Creates the Students relation. Observe that the type (domain) of each field is specified, and enforced by the DBMS whenever tuples are added or modified.

As another example, the Enrolled table holds information about courses that students take.

CREATE TABLE Students(sid: CHAR(20), name: CHAR(20), login: CHAR(10), age: INTEGER, gpa: REAL)

CREATE TABLE Enrolled(sid: CHAR(20), cid: CHAR(20), grade:

CHAR(2))

15

Destroying and Altering Relations

Destroys the relation Students. The schema information and the tuples are deleted.

DROP TABLE Students

The schema of Students is altered by adding a new field; every tuple in the current instance is extended with a null value in the new field.

ALTER TABLE Students ADD COLUMN firstYear: integer

16

Adding and Deleting Tuples

Can insert a single tuple using:

INSERT INTO Students (sid, name, login, age, gpa)VALUES (53688, ‘Smith’, ‘smith@ee’, 18, 3.2)

Can delete all tuples satisfying some condition (e.g., name = Smith):

DELETE FROM Students SWHERE S.name = ‘Smith’

What happens when the WHERE clause checks a condition that involves modified attribute?

17

Integrity Constraints (ICs)

IC: condition that must be true for any instance of the database; e.g., domain constraints. ICs are specified when schema is defined. ICs are checked when relations are modified.

A legal instance of a relation is one that satisfies all specified ICs. DBMS should not allow illegal instances.

If the DBMS checks ICs, stored data is more faithful to real-world meaning. Avoids data entry errors, too!

18

Primary Key Constraints

A set of fields is a key for a relation if :1. No two distinct tuples can have same values in all key

fields, and

2. This is not true for any subset of the key. Part 2 false? A superkey. If there’s >1 key for a relation, one of the keys is

chosen (by DBA) to be the primary key.

E.g., sid is a key for Students. (What about name?) The set {sid, gpa} is a superkey.

Primary key can not have null value

19

Primary and Candidate Keys in SQL

Possibly many candidate keys (specified using UNIQUE), one of which is chosen as the primary key.

CREATE TABLE Enrolled (sid CHAR(20) cid CHAR(20), grade CHAR(2), PRIMARY KEY (sid,cid) )

“For a given student and course, there is a single grade.” vs. “Students can take only one course, and receive a single grade for that course; further, no two students in a course receive the same grade.”

Used carelessly, an IC can prevent the storage of database instances that arise in practice!

CREATE TABLE Enrolled (sid CHAR(20) cid CHAR(20), grade CHAR(2), PRIMARY KEY (sid), UNIQUE (cid, grade) )

20

Foreign Keys, Referential Integrity

Foreign key : Set of fields in one relation that is used to `refer’ to a tuple in another relation. (Must correspond to primary key of the second relation.) Like a `logical pointer’.

E.g. sid is a foreign key referring to Students: Enrolled(sid: string, cid: string, grade: string) If all foreign key constraints are enforced, referential

integrity is achieved, i.e., no dangling references. Can you name a data model w/o referential integrity?

Links in HTML!

21

Foreign Keys in SQL Only students listed in the Students relation should be

allowed to enroll for courses.

CREATE TABLE Enrolled (sid CHAR(20), cid CHAR(20), grade CHAR(2), PRIMARY KEY (sid,cid), FOREIGN KEY (sid) REFERENCES Students )

sid name login age gpa

53666 Jones jones@cs 18 3.453688 Smith smith@eecs 18 3.253650 Smith smith@math 19 3.8

sid cid grade53666 Carnatic101 C53666 Reggae203 B53650 Topology112 A53666 History105 B

EnrolledStudents

22

Enforcing Referential Integrity Consider Students and Enrolled; sid in Enrolled is a

foreign key that references Students. What should be done if an Enrolled tuple with a non-

existent student id is inserted? (Reject it!) What should be done if a Students tuple is deleted?

Also delete all Enrolled tuples that refer to it. Disallow deletion of a Students tuple that is referred to. Set sid in Enrolled tuples that refer to it to a default sid. (In SQL, also: Set sid in Enrolled tuples that refer to it to a special

value null, denoting `unknown’ or `inapplicable’.) Similar if primary key of Students tuple is updated.

23

Referential Integrity in SQL/92

SQL/92 supports all 4 options on deletes and updates. Default is NO ACTION

(delete/update is rejected) CASCADE (also delete all

tuples that refer to deleted tuple)

SET NULL / SET DEFAULT

(sets foreign key value of referencing tuple)

CREATE TABLE Enrolled (sid CHAR(20), cid CHAR(20), grade CHAR(2), PRIMARY KEY (sid,cid), FOREIGN KEY (sid) REFERENCES Students

ON DELETE CASCADEON UPDATE SET DEFAULT )

24

Where do ICs Come From? ICs are based upon the semantics of the real-world

enterprise that is being described in the database relations. We can check a database instance to see if an IC is

violated, but we can NEVER infer that an IC is true by looking at an instance. An IC is a statement about all possible instances! From example, we know name is not a key, but the assertion that

sid is a key is given to us.

Key and foreign key ICs are the most common; more general ICs supported too.

25

Views

A view is just a relation, but we store a definition, rather than a set of tuples.

CREATE VIEW YoungActiveStudents (name, grade)AS SELECT S.name, E.gradeFROM Students S, Enrolled EWHERE S.sid = E.sid and S.age<21

Views can be dropped using the DROP VIEW

command. How to handle DROP TABLE if there’s a view on the

table? DROP TABLE command has options to let the user

specify this.

26

Views and Security Views can be used to present necessary

information (or a summary), while hiding details in underlying relation(s). Given YoungStudents, but not Students or Enrolled, we

can find students s who have are enrolled, but not the cid’s of the courses they are enrolled in.

27

How to Update Views?

Users need to know the difference between views and the base tables.

Problem: Modifying the view leads to modifying the underlying base tables which needs lot of care: delete a row (what happens when the key is not part of

the view?) insert a row.

28

Homework III.2

Pick a database of your interest. Define 2 or 3 useful views. Discuss difficulties with updating the views,

giving examples.

Is it worth allowing users to update views?What do commerical systems do?

29

Logical DB Design: ER to Relational

Entity sets to tables.

CREATE TABLE Employees (ssn CHAR(11), name CHAR(20), lot INTEGER, PRIMARY KEY (ssn))

Employees

ssnname

lot

30

Relationship Sets to Tables

In translating a relationship set to a relation, attributes of the relation must include: Keys for each participating

entity set (as foreign keys). This set of attributes

forms a superkey for the relation.

All descriptive attributes.

CREATE TABLE Works_In( ssn CHAR(1), did INTEGER, since DATE, PRIMARY KEY (ssn, did), FOREIGN KEY (ssn) REFERENCES Employees, FOREIGN KEY (did) REFERENCES Departments)

31

Review: Key Constraints

Each dept has at most one manager, according to the key constraint on Manages.

Translation to relational model?

Many-to-Many1-to-1 1-to Many Many-to-1

dname

budgetdid

since

lot

name

ssn

ManagesEmployees Departments

32

Translating ER Diagrams with Key Constraints

Map relationship to a table: Note that did is the

key now! Separate tables for

Employees and Departments.

Since each department has a unique manager, we could instead combine Manages and Departments.

CREATE TABLE Manages( ssn CHAR(11), did INTEGER, since DATE, PRIMARY KEY (did), FOREIGN KEY (ssn) REFERENCES Employees, FOREIGN KEY (did) REFERENCES Departments)

CREATE TABLE Dept_Mgr( did INTEGER, dname CHAR(20), budget REAL, ssn CHAR(11), since DATE, PRIMARY KEY (did), FOREIGN KEY (ssn) REFERENCES Employees)

33

Review: Participation Constraints

Does every department have a manager? If so, this is a participation constraint: the participation of

Departments in Manages is said to be total (vs. partial). Every did value in Departments table must appear in a row of

the Manages table (with a non-null ssn value!)

lot

name dnamebudgetdid

sincename dname

budgetdid

since

Manages

since

DepartmentsEmployees

ssn

Works_In

34

Participation Constraints in SQL

We can capture participation constraints involving one entity set in a binary relationship, but little else (without resorting to CHECK constraints).

CREATE TABLE Dept_Mgr( did INTEGER, dname CHAR(20), budget REAL, ssn CHAR(11) NOT NULL, since DATE, PRIMARY KEY (did), FOREIGN KEY (ssn) REFERENCES Employees, ON DELETE NO ACTION)

35

Review: Weak Entities A weak entity can be identified uniquely only by

considering the primary key of another (owner) entity. Owner entity set and weak entity set must participate in a one-to-

many relationship set (1 owner, many weak entities). Weak entity set must have total participation in this identifying

relationship set.

lot

name

agepname

DependentsEmployees

ssn

Policy

cost

36

Translating Weak Entity Sets Weak entity set and identifying relationship set are

translated into a single table. When the owner entity is deleted, all owned weak

entities must also be deleted.

CREATE TABLE Dep_Policy ( pname CHAR(20), age INTEGER, cost REAL, ssn CHAR(11) NOT NULL, PRIMARY KEY (pname, ssn), FOREIGN KEY (ssn) REFERENCES Employees, ON DELETE CASCADE)

37

Review: Binary vs. Ternary Relationships

If each policy is owned by just 1 employee: Key constraint on

Policies would mean policy can only cover 1 dependent!

What are the additional constraints in the 2nd diagram?

agepname

DependentsCovers

name

Employees

ssn lot

Policies

policyid cost

Beneficiary

agepname

Dependents

policyid cost

Policies

Purchaser

name

Employees

ssn lot

Bad design

Better design

38

Binary vs. Ternary Relationships (Contd.)

The key constraints allow us to combine Purchaser with Policies and Beneficiary with Dependents.

Participation constraints lead to NOT NULL

constraints. What if Policies is a

weak entity set?

CREATE TABLE Policies ( policyid INTEGER, cost REAL, ssn CHAR(11) NOT NULL, PRIMARY KEY (policyid). FOREIGN KEY (ssn) REFERENCES Employees, ON DELETE CASCADE)

CREATE TABLE Dependents ( pname CHAR(20), age INTEGER, policyid INTEGER, PRIMARY KEY (pname, policyid). FOREIGN KEY (policyid) REFERENCES Policies, ON DELETE CASCADE)

39

Relational Model: Summary

A tabular representation of data. Simple and intuitive, currently the most widely used. Integrity constraints can be specified by the DBA, based

on application semantics. DBMS checks for violations. Two important ICs: primary and foreign keys In addition, we always have domain constraints.

Powerful and natural query languages exist. Rules to translate ER to relational model

40

Homework III.3

Homework III.3: Solve problem 3.15.

41

Relational Algebra

Chapter 4, Part A

42

Relational Query Languages

Query languages: Allow manipulation and retrieval of data from a database.

Relational model supports simple, powerful QLs: Strong formal foundation based on logic. Allows for much optimization.

Query Languages != programming languages! QLs not expected to be “Turing complete”. QLs not intended to be used for complex calculations. QLs support easy, efficient access to large data sets.

43

Formal Relational Query LanguagesTwo mathematical Query Languages form the basis

for “real” languages (e.g. SQL), and for implementation:

Relational Algebra: More operational, very useful for representing execution plans.

Relational Calculus: Lets users describe what they want, rather than how to compute it. (Non-operational, declarative.)

Understanding Algebra & Calculus is key to understanding SQL, query processing!

44

Preliminaries

A query is applied to relation instances, and the result of a query is also a relation instance. Schemas of input relations for a query are fixed (but query

will run regardless of instance!) The schema for the result of a given query is also fixed!

Determined by definition of query language constructs.

Positional vs. named-field notation: Positional notation easier for formal definitions, named-

field notation more readable. Both used in SQL

45

Example Instances

sid sname rating age

22 dustin 7 45.0

31 lubber 8 55.558 rusty 10 35.0

sid sname rating age28 yuppy 9 35.031 lubber 8 55.544 guppy 5 35.058 rusty 10 35.0

sid bid day

22 101 10/10/9658 103 11/12/96

R1

S1

S2

“Sailors” and “Reserves” relations for our examples.

We’ll use positional or named field notation, assume that names of fields in query results are `inherited’ from names of fields in query input relations.

46

Relational Algebra Basic operations:

Selection ( ) Selects a subset of rows from relation. Projection ( ) Deletes unwanted columns from relation. Cross-product ( ) Allows us to combine two relations. Set-difference ( ) Tuples in reln. 1, but not in reln. 2. Union ( ) Tuples in reln. 1 and in reln. 2.

Additional operations: Intersection, join, division, renaming: Not essential, but (very!) useful.

Since each operation returns a relation, operations can be composed! (Algebra is “closed”.)

47

Projectionsname rating

yuppy 9lubber 8guppy 5rusty 10

sname rating

S,

( )2

age

35.055.5

age S( )2

Deletes attributes that are not in projection list.

Schema of result contains exactly the fields in the projection list, with the same names that they had in the (only) input relation.

Projection operator has to eliminate duplicates! (Why??) Note: real systems typically don’t

do duplicate elimination unless the user explicitly asks for it. (Why not?)

48

Selection

rating

S8

2( )

sid sname rating age28 yuppy 9 35.058 rusty 10 35.0

sname ratingyuppy 9rusty 10

sname rating rating

S,

( ( ))8

2

Selects rows that satisfy selection condition.

No duplicates in result! (Why?)

Schema of result identical to schema of (only) input relation.

Result relation can be the input for another relational algebra operation! (Operator composition.)

49

Union, Intersection, Set-Difference

All of these operations take two input relations, which must be union-compatible: Same number of fields. `Corresponding’ fields

have the same type. What is the schema of result?

sid sname rating age

22 dustin 7 45.031 lubber 8 55.558 rusty 10 35.044 guppy 5 35.028 yuppy 9 35.0

sid sname rating age31 lubber 8 55.558 rusty 10 35.0

S S1 2

S S1 2

sid sname rating age

22 dustin 7 45.0

S S1 2

50

Cross-Product Each row of S1 is paired with each row of R1. Result schema has one field per field of S1 and R1, with

field names `inherited’ if possible. Conflict: Both S1 and R1 have a field called sid.

( ( , ), )C sid sid S R1 1 5 2 1 1

(sid) sname rating age (sid) bid day

22 dustin 7 45.0 22 101 10/ 10/ 96

22 dustin 7 45.0 58 103 11/ 12/ 96

31 lubber 8 55.5 22 101 10/ 10/ 96

31 lubber 8 55.5 58 103 11/ 12/ 96

58 rusty 10 35.0 22 101 10/ 10/ 96

58 rusty 10 35.0 58 103 11/ 12/ 96

Renaming operator:

51

Joins Condition Join:

Result schema same as that of cross-product. Fewer tuples than cross-product, might be able to compute

more efficiently Sometimes called a theta-join.

R c S c R S ( )

(sid) sname rating age (sid) bid day

22 dustin 7 45.0 58 103 11/ 12/ 9631 lubber 8 55.5 58 103 11/ 12/ 96

S RS sid R sid

1 11 1

. .

52

Joins

Equi-Join: A special case of condition join where the condition c contains only equalities.

Result schema similar to cross-product, but only one copy of fields for which equality is specified.

Natural Join: Equijoin on all common fields.

sid sname rating age bid day

22 dustin 7 45.0 101 10/ 10/ 9658 rusty 10 35.0 103 11/ 12/ 96

S Rsid

1 1

53

Division Not supported as a primitive operator, but useful for

expressing queries like: Find sailors who have reserved all boats.

Let A have 2 fields, x and y; B have only field y: A/B = i.e., A/B contains all x tuples (sailors) such that for every y tuple

(boat) in B, there is an xy tuple in A. Or: If the set of y values (boats) associated with an x value (sailor)

in A contains all y values in B, the x value is in A/B.

In general, x and y can be any lists of fields; y is the list of fields in B, and x y is the list of fields of A.

x x y A y B| ,

54

Examples of Division A/B

sno pnos1 p1s1 p2s1 p3s1 p4s2 p1s2 p2s3 p2s4 p2s4 p4

pnop2

pnop2p4

pnop1p2p4

snos1s2s3s4

snos1s4

snos1

A

B1B2

B3

A/B1 A/B2 A/B3

55

Expressing A/B Using Basic Operators

Division is not essential op; just a useful shorthand. (Also true of joins, but joins are so common that systems

implement joins specially.)

Idea: For A/B, compute all x values that are not `disqualified’ by some y value in B. x value is disqualified if by attaching y value from B, we obtain

an xy tuple that is not in A.

Disqualified x values:

A/B:

x x A B A(( ( ) ) )

x A( ) all disqualified tuples

56

Find names of sailors who’ve reserved boat #103

Solution 1: sname bidserves Sailors(( Re ) )

103

Solution 2: ( , Re )Temp servesbid

1103

( , )Temp Temp Sailors2 1

sname Temp( )2

Solution 3: sname bidserves Sailors( (Re ))

103

57

Find names of sailors who’ve reserved a red boat

Information about boat color only available in Boats; so need an extra join:

sname color redBoats serves Sailors((

' ') Re )

A more efficient solution:

sname sid bid color redBoats s Sailors( ((

' ') Re ) )

A query optimizer can find this given the first solution!

58

Find sailors who’ve reserved a red or a green boat

Can identify all red or green boats, then find sailors who’ve reserved one of these boats:

( , (' ' ' '

))Tempboatscolor red color green

Boats

sname Tempboats serves Sailors( Re )

Can also define Tempboats using union! (How?)

What happens if is replaced by in this query?

59

Find sailors who’ve reserved a red and a green boat

Previous approach won’t work! Must identify sailors who’ve reserved red boats, sailors who’ve reserved green boats, then find the intersection (note that sid is a key for Sailors):

( , ((' '

) Re ))Tempredsid color red

Boats serves

sname Tempred Tempgreen Sailors(( ) )

( , ((' '

) Re ))Tempgreensid color green

Boats serves

60

Find the names of sailors who’ve reserved all boats

Uses division; schemas of the input relations to / must be carefully chosen:

( , (,

Re ) / ( ))Tempsidssid bid

servesbid

Boats

sname Tempsids Sailors( )

To find sailors who’ve reserved all ‘Interlake’ boats:

/ (' '

) bid bname Interlake

Boats.....

61

Summary

The relational model has rigorously defined query languages that are simple and powerful.

Relational algebra is more operational; useful as internal representation for query evaluation plans.

Several ways of expressing a given query; a query optimizer should choose the most efficient version.