A Note on Useful Algorithmic Strategies

Kun-Mao Chao (趙坤茂 )Department of Computer Science an

d Information EngineeringNational Taiwan University, Taiwan

WWW: http://www.csie.ntu.edu.tw/~kmchao

2

Greedy Algorithm

• A greedy method always makes a locally optimal (greedy) choice.– the greedy-choice property: a globally optimal

solution can be reached by a greedy choice.– optimal substructures

3

Huffman Codes (1952)

A

0.1

0.2

C

0.1

G

0.3

T

0.5

0.5

1.0David Huffman

(August 9, 1925 –

October 7, 1999)

4

A

0.1

0.2

C

0.1

G

0.3

T

0.5

0.5

1.00

0

0 1

1

1

000 001 01 1

Huffman CodesExpected number of bits per character

= 3x0.1+3x0.1+2x0.3+1x0.5

= 1.7

(vs. 2 bits by a simple scheme)

5

An example

Sequence: GTTGTTATCGTTTATGTGGC

By Huffman Coding:

0111011100010010111100010110101001

20 characters; 34 bits in total

6

Divide-and-Conquer

1. Divide the problem into smaller subproblems.

2. Conquer each subproblem recursively.

3. Combine the solutions to the child subproblems into the solution for the parent problem.

7

Merge Sort(Invented in 1938; Coded in 1945)

65 16 77368128525

65 16 77368128525

65 16 8525 812 7736

65 16 77368128525

John von Neumann(December 28, 1903 – February 8, 1957 )

8

8 12 857765362516

16 25

7736128

8565

Merge Sort(Merge two solutions into one.)

9

8 12 857765362516

16 25 77361288565

16 65 8525 128 7736

65 16 77368128525

Merge Sort

10

Dynamic Programming• Dynamic programming is a class of

solution methods for solving sequential decision problems with a compositional cost structure.

• Richard Bellman was one of the principal founders of this approach.

Richard Ernest Bellman (1920–1984)

11

Two key ingredients

• Two key ingredients for an optimization problem to be suitable for a dynamic-programming solution:

Each substructure is optimal.

(Principle of optimality)

1. optimal substructures

2. overlapping subproblems

Subproblems are dependent.

(otherwise, a divide-and-conquer approach is the choice.)

12

Three basic components

• The development of a dynamic-programming algorithm has three basic components:– The recurrence relation (for defining the val

ue of an optimal solution);– The tabular computation (for computing th

e value of an optimal solution);– The traceback (for delivering an optimal sol

ution).

13

Fibonacci numbers

.for 21

11

00

i>1i

Fi

FiF

F

F

The Fibonacci numbers are defined by the following recurrence:

Leonardo of Pisa (c. 1170 – c. 1250)

0, 1, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89, 144, 233, 377, 610, 987, 1597, 2584, 4181, 6765, 10946, 17711, 28657, 46368, 75025, 121393, ...

14

How to compute F10？

F10

F9

F8

F8

F7

F7

F6

……

15

Tabular computation

• The tabular computation can avoid recompuation.

F0 F1 F2 F3 F4 F5 F6 F7 F8 F9 F10

0 1 1 2 3 5 8 13 21 34 55

16

Longest increasing subsequence(LIS)

• The longest increasing subsequence is to find a longest increasing subsequence of a given sequence of distinct integers a1a2…an .

e.g. 9 2 5 3 7 11 8 10 13 6

2 3 7

5 7 10 13

9 7 11

3 5 11 13

are increasing subsequences.

are not increasing subsequences.

We want to find a longest one.

17

A naive approach for LIS• Let L[i] be the length of a longest increasing

subsequence ending at position i.

L[i] = 1 + max j = 0..i-1{L[j] | aj < ai}(use a dummy a0 = minimum, and L[0]=0)

9 2 5 3 7 11 8 10 13 6L[i] 1 1 2 2 3 4 ?

18

A naive approach for LIS

9 2 5 3 7 11 8 10 13 6L[i] 1 1 2 2 3 4 4 5 6 3

L[i] = 1 + max j = 0..i-1 {L[j] | aj < ai}

The maximum length

The subsequence 2, 3, 7, 8, 10, 13 is a longest increasing subsequence.

This method runs in O(n2) time.

19

An O(n log n) method for LIS

• Define BestEnd[k] to be the smallest number of an increasing subsequence of length k.

9 2 5 3 7 11 8 10 13 69 2 2

5

2

3

2

3

7

2

3

7

11

2

3

7

8

2

3

7

8

10

2

3

7

8

10

13

BestEnd[1]

BestEnd[2]

BestEnd[3]

BestEnd[4]

BestEnd[5]

BestEnd[6]

20

An O(n log n) method for LIS

• Define BestEnd[k] to be the smallest number of an increasing subsequence of length k.

9 2 5 3 7 11 8 10 13 69 2 2

5

2

3

2

3

7

2

3

7

11

2

3

7

8

2

3

7

8

10

2

3

7

8

10

13

2

3

6

8

10

13

BestEnd[1]

BestEnd[2]

BestEnd[3]

BestEnd[4]

BestEnd[5]

BestEnd[6]

For each position, we perform a binary search to update BestEnd. Therefore, the running time is O(n log n).

23

Longest Common Subsequence (LCS)

• A subsequence of a sequence S is obtained by deleting zero or more symbols from S. For example, the following are all subsequences of “president”: pred, sdn, predent.

• The longest common subsequence problem is to find a maximum-length common subsequence between two sequences.

24

LCS

For instance,

Sequence 1: president

Sequence 2: providence

Its LCS is priden.

president

providence

25

LCS

Another example:

Sequence 1: algorithm

Sequence 2: alignment

One of its LCS is algm.

a l g o r i t h m

a l i g n m e n t

26

How to compute LCS?

• Let A=a1a2…am and B=b1b2…bn .

• len(i, j): the length of an LCS between a1a2…ai and b1b2…bj

• With proper initializations, len(i, j)can be computed as follows.

,

. and 0, if)),1(),1,(max(

and 0, if1)1,1(

,0or 0 if0

),(

ji

ji

bajijilenjilen

bajijilen

ji

jilen

27

procedure LCS-Length(A, B)

1. for i ← 0 to m do len(i,0) = 0

2. for j ← 1 to n do len(0,j) = 0

3. for i ← 1 to m do

4. for j ← 1 to n do

5. if ji ba then

" "),(

1)1,1(),(

jiprev

jilenjilen

6. else if )1,(),1( jilenjilen

7. then

" "),(

),1(),(

jiprev

jilenjilen

8. else

" "),(

)1,(),(

jiprev

jilenjilen

9. return len and prev

28

i j 0 1 p

2 r

3 o

4 v

5 i

6 d

7 e

8 n

9 c

10 e

0 0 0 0 0 0 0 0 0 0 0 0

1 p 2

0 1 1 1 1 1 1 1 1 1 1

2 r 0 1 2 2 2 2 2 2 2 2 2

3 e 0 1 2 2 2 2 2 3 3 3 3

4 s 0 1 2 2 2 2 2 3 3 3 3

5 i 0 1 2 2 2 3 3 3 3 3 3

6 d 0 1 2 2 2 3 4 4 4 4 4

7 e 0 1 2 2 2 3 4 5 5 5 5

8 n 0 1 2 2 2 3 4 5 6 6 6

9 t 0 1 2 2 2 3 4 5 6 6 6

29

procedure Output-LCS(A, prev, i, j)

1 if i = 0 or j = 0 then return

2 if prev(i, j)=” “ then

ia

jiprevALCSOutput

print

)1,1,,(

3 else if prev(i, j)=” “ then Output-LCS(A, prev, i-1, j)

4 else Output-LCS(A, prev, i, j-1)

30

i j 0 1 p

2 r

3 o

4 v

5 i

6 d

7 e

8 n

9 c

10 e

0 0 0 0 0 0 0 0 0 0 0 0

1 p 2

0 1 1 1 1 1 1 1 1 1 1

2 r 0 1 2 2 2 2 2 2 2 2 2

3 e 0 1 2 2 2 2 2 3 3 3 3

4 s 0 1 2 2 2 2 2 3 3 3 3

5 i 0 1 2 2 2 3 3 3 3 3 3

6 d 0 1 2 2 2 3 4 4 4 4 4

7 e 0 1 2 2 2 3 4 5 5 5 5

8 n 0 1 2 2 2 3 4 5 6 6 6

9 t 0 1 2 2 2 3 4 5 6 6 6

Output: priden

31

Longest Common Increasing Subsequence

• Proposed by Yang, Huang and Chao– IPL 2005

9 2 5 3 7 11 8 10 13 6

6 5 2 8 3 7 4 10 1 13