Ladan Tahvildari, PEng, SMIEEE Associate Professor

Software Technologies Applied Research (STAR) Group

Dept. of Elect. & Comp. Eng.

University of Waterloo

ECE250: Algorithms and Data Structures

Dynamic Programming – Part B

Materials from CLRS: Chapter 15.4

Acknowledgements

v The following resources have been used to prepare materials for this course: Ø  MIT OpenCourseWare Ø  Introduction To Algorithms (CLRS Book) Ø  Data Structures and Algorithm Analysis in C++ (M. Wiess) Ø  Data Structures and Algorithms in C++ (M. Goodrich)

v Thanks to many people for pointing out mistakes, providing suggestions, or helping to improve the quality of this course over the last ten years: Ø  http://www.stargroup.uwaterloo.ca/~ece250/acknowledgment/

Lecture 26 ECE250 2

Lecture 25-26 ECE250 3

v Two matrices, A – n×m matrix and B – m×k matrix, can be multiplied to get C with dimensions n×k, using nmk scalar multiplications

v Problem: Compute a product of many matrices efficiently

v Matrix multiplication is associative Ø  (AB)C = A(BC)

Multiplying Matrices

, , ,1

m

i j i l l jl

c a b=

= ⋅∑11 12

1311 1221 22 22

2321 2231 32

... ... ...

... ...

... ... ...

a abb b

a a cbb b

a a

⎛ ⎞ ⎛ ⎞⎛ ⎞⎜ ⎟ ⎜ ⎟=⎜ ⎟⎜ ⎟ ⎜ ⎟⎝ ⎠⎜ ⎟ ⎜ ⎟

⎝ ⎠ ⎝ ⎠

ECE250 4

Multiplying Matrices (cont’)

v Let M(i,j) be the minimum number of multiplications necessary to compute

v Key Observations Ø The outermost parenthesis partition the chain of

matrices (i,j) at some k, (i≤k<j): (Ai… Ak)(Ak+1… Aj)

Ø The optimal parenthesization of matrices (i,j) has optimal parenthesizations on either side of k: for matrices (i,k) and (k+1,j)

j

kk i

A=∏

Lecture 25-26

ECE250 5

Multiplying Matrices (cont’) v Recurrence:

v  It requires only Θ(n2) space to store the optimal cost M(i,j) for each of the sub-problems: half of a 2d array M[1..n,1..n] Ø  Thus, we have to solve sub-problems in the increasing

length of sub-problems: first sub-problems of length 2, then of length 3 and so on.

v To reconstruct an optimal parenthesization for each pair (i,j) we record in c[i, j]=k the optimal split into two sub-problems (i, k) and (k+1, j)

{ }1

( , ) 0

( , ) min ( , ) ( 1, )i k j i k j

M i i

M i j M i k M k j d d d≤ < −

=

= + + +

Lecture 25-26

ECE250 6

Multiplying Matrices (cont’)

v After the execution: M [1,n] contains the value of an opt imal solut ion and c contains opt imal subdivisions (choices of k) of any sub-problem into two sub-sub-problems

v We ran the algorithm on the six matrices: A1 is a 30x35 matrix, A2 is a 35x15 matrix, A3 is a 15x5 matrix, A4 is a 5x10 matrix, A5 is a 10x20 matrix, and A6 is a 20x25 matrix.

Lecture 25-26

ECE250 7

Multiplying Matrices (cont’) Matrix-Chain-Order(d0…dn) 01 for i←1 to n do 02 M[i,i] ← 0 03 for L←2 to n do 04 for i←1 to n-L+1 do 05 j ← i+L-1 06 M[i,j] ← ∞07 for k←i to j-1 do 08 q ← M[i,k]+M[k+1,j]+di-1dkdj 09 if q < M[i,j] then 10  M[i,j] ← q 11  c[i,j] ← k 12 return M, c

Running time: O(n3); is also Ω(n3) From exponential time to polynomial

Lecture 25-26

ECE250 8

Dynamic Programming v  A sequence of four steps:

1.  Show optimal substructure – an optimal solution to the problem contains within it optimal solutions to sub-problems Ø  Solution to a problem:

ü  Making a choice out of a number of possibilities (look what possible choices there can be)

ü  Solving one or more sub-problems that are the result of a choice (characterize the space of sub-problems)

Ø  Show that solutions to sub-problems must themselves be optimal for the whole solution to be optimal (use “cut-and-paste” argument)

Lecture 25-26

ECE250 9

Dynamic Programming (cont’)

2.  Write a recurrence for the value of an optimal solution

Ø  Mopt = Minover all choices k {(Combination (e.g., sum) of Mopt of all sub-problems, resulting from choice k) + (the cost associated with making the choice k)}

Ø  Show that the number of different instances of sub-problems is bounded by a polynomial

Lecture 25-26

ECE250 10

Dynamic Programming (cont’)

3.  Compute the value of an optimal solution in a bottom-up fashion, so that you always have the necessary sub-results pre-computed

4.  Construct an optimal solution from computed

information (which records a sequence of choices made that lead to an optimal solution)

Lecture 25-26

ECE250 11

Longest Common Subsequence

v Two text strings are given: X and Y

v There is a need to quantify how similar they are

v Applications: Ø Comparing DNA sequences in studies of evolution

of different species Ø Spell checkers

v One of the measures of similarity is the length of a Longest Common Subsequence (LCS)

Lecture 25-26

ECE250 12

LCS: Definition

v Z is a subsequence of X, if it is possible to generate Z by skipping some (possibly none) characters from X

v For example: X =“ACGGTTA”, Y=“CGTAT”, LCS(X,Y) = “CGTA” or “CGTT”

v To solve LCS problem we have to find “skips” that generate LCS(X,Y) from X, and “skips” that generate LCS(X,Y) from Y

v The longest-common-subsequence problem: find a maximum-length common subsequence

Lecture 25-26

ECE250 13

Step1 - LCS: Optimal Substructure

v We make Z to be empty and proceed from the ends of Xm=“x1 x2 …xm” and Yn=“y1 y2 …yn”

Ø  If xm=yn, append this symbol to the beginning of Z, and find optimally LCS(Xm-1, Yn-1)

Ø  If xm≠yn, §  Skip either a letter from X §  or a letter from Y §  Decide which decision to do by comparing LCS(Xm, Yn-1) and

LCS(Xm-1, Yn)

Ø  “Cut-and-paste” argument

Lecture 25-26

ECE250 14

Step 2 - LCS: Recurrence

v The algorithm could be easily extended by allowing more “editing” operations in addition to copying and skipping (e.g., changing a letter)

v Let c[i,j] = LCS(Xi, Yj)

0 if 0 or 0[ , ] [ 1, 1] 1 if , 0 and

max{ [ , 1], [ 1, ]} if , 0 and i j

i j

i jc i j c i j i j x y

c i j c i j i j x y

⎧ = =⎪

= − − + > =⎨⎪ − − > ≠⎩

Lecture 25-26

ECE250 15

Step 3 - LCS: Compute the Optimum

Lecture 25-26

LCS-Length(X, Y, m, n) 1 for i←1 to m do 2 c[i,0] ← 0 3 for j←0 to n do 4 c[0,j] ← 0 5 for i←1 to m do 6 for j←1 to n do 7 if xi = yj then 8  c[i,j] ← c[i-1,j-1]+1 9 b[i,j] ← ”copy” 10 else if c[i-1,j] ≥ c[i,j-1] then 11  c[i,j] ← c[i-1,j] 12  b[i,j] ← ”skipX” 13 else 14  c[i,j] ← c[i,j-1] 15  b[i,j] ← ”skipY” 16 return c, b

Step 4: Constructing an LCS

ECE250 16

PRINT-LCS(b, X, i, j) 1   if i==0 and j==0 2   return 3   if b[i,j] == ”copy” 5   PRINT-LCS(b, X, i-1, j-1) 6   print xi 7   elseif b[i,j] == ”skipX” 8   PRINT-LCS(b, X, i-1, j) 9   else PRINT-LCS(b, X, i, j-1)

Lecture 25-26

ECE250 17

LCS: Examples

v Let’s run: X=“ABCBDAB”, Y=“BDCABA”

v Another one: X =“GGTTCAT”, Y=“GTATCT”

Lecture 25-26