Chapter 4: Divide and Conquer - Intro

• A recurrence equation (recurrence) is an equation or inequality that describesa function in terms of its values on smaller inputs

• There are various forms:

1. Recurse on subproblems of equal size (typical of divide and conquer)

T (n) = cT

(n

c

)+ f(n)

2. Recurse on subproblems of unequal size (typical of divide and conquer)

T (n) = T

(c1n

c3

)+ T

(c3 − c1)nc3

+ f(n)

3. Recurse on a single smaller problem size (typical of decrease and conquer)

T (n) = T (n− c) + f(n)

• Issues:

1. Sometimes recurrences are expressed as inequalities

– If T (n) ≤ some recurrence function, solution will be in terms of big ohnotation

– If T (n) ≥ some recurrence function, solution will be in terms of bigomega notation

2. For exact solutions, must usually consider floor and ceiling functions

– For example, when n is not even

T (n) = T

(⌈n

2

⌉)+ T

(⌊n

2

⌋)+ f(n)

3. Boundary conditions are generalized

– Rather than specify an exact function, usually represent the aspects ofminor importance using asymptotic notation

– Unless an exact solution is required, the base case need not be repre-sented by an exact function

1

Chapter 4: Divide and Conquer - The Max Subarray Problem

• Consider a vector A[n] of numeric values

• The maximum subarray problem is to find i0, j0 such that i0 ≤ j0 ≤ n and∑j0k=i0

A[k] is greater or equal to the sum over any other assignments of i and j

• The brute force approach would be to id all pairs < i, j > that meet the abovecriteria and find the largest sum over those pairs

– The number of pairs for (i0, j0) is

n2

– Value of each subarray can be computed in constant time using values of

previously computed subarrays, so overall ∈ Θ(n2)

• Divide and conquer approach

– Strategy:

∗ Divide array in half

∗ The max subarray is either

1. Entirely in the left half

2. Entirely in the right half

3. Spanning the middle

∗ The alg will

1. Recurse on the left and right halves

2. Find the largest subarray that spans the middle

3. Return the largest of the three

2

Chapter 4: Divide and Conquer - The Max Subarray Problem (2)

– The crossing alg:

FindMaxCrossingSubarray (A, low, mid, high)

1 left_sum = -infinity

2 sum = 0

3 for (i = mid down to low)

4 sum = sum + A[i]

5 if (sum > left_sum)

6 left_sum = sum

7 max_left = i

8 right_sum = -infinity

9 sum = 0

10 for (j = mid + 1 to high)

11 sum = sum + A[j]

12 if (sum > right_sum)

13 right_sum = sum

14 max_right = j

15 return(max_left, max_right, left_sum + right_sum)

∗ Analysis:

· Body of each for loop ∈ Θ(1)

· Total number of iterations of the two loops is (mid−low+1)+(high−mid) = high− low + 1 = n

· Total effort is n ∗Θ(1) = Θ(n)

3

Chapter 4: Divide and Conquer - The Max Subarray Problem (3)

– The main alg:

FindMaxSubarray (A, low, high)

1 if (high == low)

2 return(low, high, A[low]) //base case

else

3 mid = floor((low + high)/2)

4 left_low, left_high, left_sum = FindMaxSubarray (A, low, mid)

5 right_low, right_high, right_sum = FindMaxSubarray (A, mid + 1, high)

6 cross_low, cross_high, cross_sum = FindMaxCrossingSubarray (A, low, mid, high)

7 if ((left_sum >= right_sum) && (left_sum >= cross_sum)

8 return (left_low, left_high, left_sum)

9 else if ((right_sum >= left_sum) && (right_sum >= cross_sum)

10 return (right_low, right_high, right_sum)

else

11 return (cross_low, cross_high, cross_sum)

∗ Analysis:

· Let n = 2i, i ≥ 0

· T (1) ∈ Θ(1)

· Each reursion is on n/2 elements

· Return statements, conditionals ∈ Θ(1)

· Recursion:

T (n) =

Θ(1) n = 12T

(n2

)+ Θ(n) + Θ(1) n > 1

where First term represents recursive calls Second term finding the spanning subarray Third the remaining instructions

∗ This is the same result obtained for merge sort, so T (n) ∈ Θ(n lg n)

4

Chapter 4: Divide and Conquer - Matrix Multiplication Intro

• Consider 2 n× n matrices A and B

• Let C = A ∗B

• Then

Cij =n−1∑j=0

aij ∗ bji

• For a 2× 2 matrix, this requires 8 multiplications and 4 additions

• A brute force algorithm that implements the above:

Square-Matrix-Multiply (A, B)

1 n = A.rows

2 C[n][n] = new array

3 for (i = 1 to n)

4 for (j = 1 to n)

5 C[i][j] = 0

6 for (k = 1 to n)

7 C[i][j] = C[i][j] + A[i][k] * B[k][j]

– By virtue of the triply-nested loops, each of which executes n times, Square-Matrix-Multiply ∈ Θ(n3)

5

Chapter 4: Divide and Conquer - Matrix Multiplication Recursive Approach

• Again consider square matrices A and B of size n >= 2

• Each matrix can be considered to be a matrix of square sub matrices; e.g.,

A =

A11 A12

A21 A22

• We can then represent matrix multiplication in terms of these sub matrices: C11 C12

C21 C22

=

A11 A12

A21 A22

∗ B11 B12

B21 B22

• The multiplication is achieved recursively, where C11 = A11 ∗ B11 + A12 ∗ B21,

etc.

• The algorithm:

Square-Matrix-Multiply-Recursive (A, B)

1 n = A.rows

2 C[n][n] = new array

3 if (n == 1)

4 C[1][1] = A[1][1] * B[1][1]

else

5 partition A, B, C into 4 sub matrices each

6 for (k = 1 to n)

7 C11 = Square-Matrix-Multiply-Recursive(A11, B11) +

Square-Matrix-Multiply-Recursive(A12, B21)

8 C12 = Square-Matrix-Multiply-Recursive(A11, B12) +

Square-Matrix-Multiply-Recursive(A12, B22)

9 C21 = Square-Matrix-Multiply-Recursive(A21, B11) +

Square-Matrix-Multiply-Recursive(A22, B21)

10 C22 = Square-Matrix-Multiply-Recursive(A21, B12) +

Square-Matrix-Multiply-Recursive(A22, B22)

6

Chapter 4: Divide and Conquer - Matrix Multiplication Recursive Approach(2)

• Analysis

– The partitioning (line 5) can be done in Θ(1) time if index calculationsare used instead of copying the elements into new matrices (which wouldrequire Θ(n2) time)

– Base case when n = 1 ∈ Θ(1) time

– Recurse eight times on matrices of size n/2× n/2 (lines 7 - 10)

– Four additions performed on matrices that hold n2/4 elements (Θ(n2) lines7 - 10)

– The recursion is then

T (n) =

Θ(1) n = 18T (n2 ) + Θ(n2) n > 1

7

Chapter 4: Divide and Conquer - Strassen’s Algorithm

• Strassen’s algorithm improves on this by making the recursion tree slightly lessbushy at the expense of increased additions and subtractions

– But these are negligible wrt asymptotic growth, being subsumed by themultiplications

• As in the recursive algorithm, n × n matrices A, B, C are partitioned inton/2× n/2 sub matrices: C11 C12

C21 C22

=

A11 A12

A21 A22

∗ B11 B12

B21 B22

• The four subarrays of C are computed in terms of n/2× n/2 matrices mi:

C =

P5 + P4 − P2 + P6 P1 + P2

P3 + P4 P5 + P1 − P3 − P7

• The Pi are defined as

1. P1 = A11 ∗ S1 = A11 ∗B12 − A11 ∗B22

2. P2 = S2 ∗B22 = A11 ∗B22 + A12 ∗B22

3. P3 = S3 ∗B11 = A21 ∗B11 + A22 ∗B11

4. P4 = A22 ∗ S4 = A22 ∗B21 − A22 ∗B11

5. P5 = S5 ∗ S6 = A11 ∗B11 + A11 ∗B22 + A22 ∗B11 + A22 ∗B22

6. P6 = S7 ∗ S8 = A12 ∗B21 + A12 ∗B22 − A22 ∗B21 − A22 ∗B22

7. P7 = S9 ∗ S10 = A11 ∗B11 + A11 ∗B12 − A21 ∗B11 − A21 ∗B12

8

Chapter 4: Divide and Conquer - Strassen’s Algorithm (2)

• The Si are defined as

1. S1 = B12 −B22

2. S2 = A11 + A12

3. S3 = A21 + A22

4. S4 = B21 −B11

5. S5 = A11 + A12

6. S6 = B11 +B22

7. S7 = A12 − A22

8. S8 = B21 +B22

9. S9 = A11 − A21

10. S10 = B11 +B12

• This requires only 7 multiplications, but 18 additions/subtractions

• The multiplications will be performed recursively

• Analysis:

– The steps involved:

1. Divide A, B, C into n/2× n/2 subarrays:Θ(1) as discussed in the straight recursive approach

2. Create ten n/2× n/2 Si arrays:Θ(n2)

3. Recursively create seven n/2× n/2 Pi arrays:7T (n2 )

4. Combine the Pi into C11, C12, C21, C22 n/2× n/2 arrays:Θ(n2)

– Run time is then

T (n) =

Θ(1) n = 17T (n2 ) + Θ(n2) n > 1

9

Chapter 4: Divide and Conquer - Strassen’s Algorithm (3)

• Correctness

1. C11 = P5 + P4 − P2 + P6

A11 ∗B11 +A11 ∗B22 +A22 ∗B11 +A22 ∗B22

−A22 ∗B11 +A22 ∗B21

−A11 ∗B22 −A12 ∗B22

−A22 ∗B22 −A22 ∗B21 +A12 ∗B22 +A12 ∗B21

A11 ∗B11 +A12 ∗B21

2. C12 = P1 + P2

A11 ∗B12 −A11 ∗B22

−A11 ∗B22 +A12 ∗B22

A11 ∗B12 +A12 ∗B22

3. C21 = P3 + P4

A21 ∗B11 +A22 ∗B11

−A22 ∗B11 +A22 ∗B21

A21 ∗B11 +A22 ∗B21

4. C22 = P5 + P1 − P3 − P7

A11 ∗B11 +A11 ∗B22 +A22 ∗B11 +A22 ∗B22

−A11 ∗B22 +A11 ∗B12

−A22 ∗B11 −A21 ∗B11

−A11 ∗B11 −A11 ∗B12 +A21 ∗B11 +A21 ∗B12

+A22 ∗B22 +A21 ∗B12

10

Chapter 4: Divide and Conquer - Analysis Techniques

• Comparisons of OoG

limn→∞t(n)g(n) =

0⇒ t(n)′s OoG < g(n)′s OoGc⇒ t(n)′s OoG = g(n)′s OoG∞⇒ t(n)′s OoG > g(n)′s OoG

• L ’Hopital’s Rule

limn→∞t(n)g(n) = limn→∞

t′(n)g′(n)

– Useful in regards to the above

• Forward substitution

– Given a recurrence of the form

T (n) = cT

(n

d

)+ f(n)

– Let n = di, and iteratively compute T (i+ 1) in terms of T (i):

1. Start with base condition/time

2. Determine the value of the caller by substituting the value of the basecondition into the recursion equation

3. Continue in this manner for a few iteration in the hope that a formulawill emerge

– For example, consider

T (n) = cT

(n

d

)+ f(n)

Let n = di with base case T (1) = d0 = b (step i = 0)

Then,

i = 1: T (d1) = cT ((d1)/d) + f(d1) = cT (d0) + f(d1) = cb+ f(d1)i = 2: T (d2) = cT ((d2)/d) + f(d2) = cT (d1) + f(d2) = c[cb+ f(d1)] +f(d2) = c2b+ cf(d1) + f(d2))etc.

– Will ultimately express i in terms of n

11

Chapter 4: Divide and Conquer - Analysis Techniques (2)

• Backward substitution

– Start at the top level (n elements) and work towards the base case

– Substitute the value from the next recursion into the recursion equation;i.e.,Express T (di) in terms of T (di−1) and substitute in original recurrencerelation

– Repeat with successively smaller terms until a formula that expresses thesequence can be determined

– For example, consider

T (n) = cT

(n

d

)+ f(n)

Let n = dk with base case T (1) = d0 = b (step i = 0)

Then,

i = 1: T (dk) = cT ((dk)/d) + f(dk) = cT (dk−i) + f(dk)Since T (dk−1) = cT ((dk−1)/d) + f(dk−1) = cT (dk−2) + f(dk−1) we canrewrite T (dk) asi = 2: T (dk) = c[cT (dk−i)+f(dk−i+1)]+f(dk) = c2T (dk−i)+cf(dk−i+1)+f(dk)etc.

– Will ultimately express i in terms of n

12

Chapter 4: Divide and Conquer - Analysis Techniques, The Substitution Method

• This technique involves

1. Guess a form for the equation (i.e., g(n))

2. Use induction to find asymptotic constant(s) and show that the guess works

• The guess might be based on

1. A previous analysis that had a similar equation

2. Result of another proof technique

• The proof itself consists of

1. Substituting the guess for the recurrence in the equation

2. Find a constant (could be two) that satisfies the relation in the asymptoticdefinition

3. Demonstrate that the guess holds for the base case too

• Example: Consider (pp 83 - 84)

T (n) = 2T

(⌊n

2

⌋)+ n

– The guess is T (n) = O(nlg n)

– To prove guess correct, must show that T (n) ≤ cnlg n for some c > 0

– Assume that this holds for all m < n (i.e.,⌊n2

⌋)

– Substitute cnlg n in the right-hand side of the inequality, using⌊n2

⌋for n

– Then,T (n) ≤ 2

(c⌊n2

⌋lg

(⌊n2

⌋))+ n

≤ cnlg(n2

)+ n

= cnlg n− cnlg 2 + n= cnlg n− cn+ n

≤ cnlg n

which holds for c ≥ 1

13

Chapter 4: Divide and Conquer - Analysis Techniques, The SubstitutionMethod (2)

– Next, need to demonstrate that the guess holds for the base case

∗ This step can sometimes require some ingenuity

∗ Assume that T (1) = 1 for the example above; then

T (1) = 1 ≤ c ∗ 1 ∗ lg 1 = 0

∗ This contradicts the hypothesis

∗ What we must do is distinguish between the base case of the equation,and the base case of the proof

∗ While the base case of the recursion is n = 1, for the proof we can uselarger values (because we only have to show that it holds for all n ≥some n0, which we essentially get to choose)

∗ Since n = 1 is problemmatic, judicious choices for base cases are n = 2and n = 3

· These are the two values that result in recursion with n = 1

∗ Using these values as base cases,

· T (2) = 4 (i.e., 2T (1) + 2 = 2 + 2 ≤ 2lg 2)

· T (3) = 5 (i.e., 2T (1) + 3 = 2 + 3 ≤ 3lg 3)

• Heuristics for making a good guess:

1. If know bounds for a similar recursion, try it

2. Prove weak upper and lower bounds, then iteratively try to improve themby tightening them

• Issues

1. May have correct bound but induction doesn’t work

– Reason might be because induction isn’t strong enough to prove thedetailed (exact) bound

– Can sometimes subtract a lower-order term to make it work

14

Chapter 4: Divide and Conquer - Analysis Techniques, TheSubstitution Method (3)

– For example (p 85)

T (n) = T

(⌊n

2

⌋)+ T

(⌈n

2

⌉)+ 1

Guess T (n) = O(n)Need to show that T (n) ≤ cn

On substitution:

T (n) = c

⌊n

2

⌋+ c

⌈n

2

⌉+ 1 = cn+ 1

Clearly this isn’t less than cn for any value of n

– The problem is the constant term 1

– To remedy the situation, subtract a constant d ≥ 0

– The revised guess is T (n) ≤ cn− d

T (n) ≤(c

⌊n

2

⌋− d

)+

(c

⌈n

2

⌉− d

)+ 1 = cn− 2d+ 1 ≤ cn− d

15

Chapter 4: Divide and Conquer - Analysis Techniques, The SubstitutionMethod (4)

2. You must prove the exact form of the inductive hypothesis

– For example, if you want to prove T (n) ∈ O(n), you must showT (n) ≤ cn

– Consider the following common example of incorrect logic

∗ Want to show

T (n) = 2T

(⌊n

2

⌋)+ n ∈ O(n)

∗ ThenT (n) ≤ 2

(c⌊n2

⌋)+ n

≤ cn+ n ∈ O(n)

∗ But this reasoning is fallacious because have not shown thatT (n) ≤ cn

3. Sometimes changing variables makes proofs easier

– ConsiderT (n) = 2T (b

√nc) + lg n

– Let m = lg n⇒ n = 2m

– ThenT (2m) = 2T (2

m2 ) +m

– Then, let S(m) = T (2m), which lets us rewrite the above as

S(m) = 2S

(m

2

)+m

– This has the same form as

T (n) = 2T

(⌊n

2

⌋)+ n ∈ O(n)

– So we can guess a solution of

S(m) = O(mlg m) = O((lg n)(lg lg n))

16

Chapter 4: Divide and Conquer - Analysis Techniques, Recursion Trees

• This approach can be used to find a good guess to OoG to be used in thesubstitution method, or - if done in an exacting manor - it can be a proof initself

• The approach:

1. Create a recursion tree

2. Calculate the cost of each level

3. Calculate the cost over all levels

• Consider

T (n) = 3T

(⌊n

4

⌋)+O(n2)

– Consider the following examples of ’sloppiness’ (i.e., laxness) that is allow-able since we are looking for a guesstimate

∗ Let n = 4i, eliminating the need for the floor function in the above

∗ Replace O(n2) by cn2

– This generates the following recursion tree:

17

Chapter 4: Divide and Conquer - Analysis Techniques, Recursion Trees(2)

– Let i represent the level of the tree; then

Level Cost

i = 0 c30(n40

)2= cn2

i = 1 c31(n41

)2=(316

)cn2

i = 2 c32(n42

)2=(316

)2cn2

. . . . . . . . .

– The leaf level corresponds to 4i = n⇒ log4(4i) = log4 n

Thus i = log4n, which corresponds to the height of the tree

– Cost of the leaf level is 3log4nT (1) = cnlog43

– The total cost of the tree is

T (n) =∑log4n−1i=0

(316

)icn2 + cnlog43

<∑∞i=0

(316

)icn2 + cnlog43

= 11− 3

16

cn2 + cnlog43

= 1613cn

2 + cnlog43

∈ O(n2)

– The root of the tree contributes the greatest amount to the total cost (cout of 16

13c, with 313c contributed from the lower levels of the tree)

18

Chapter 4: Divide and Conquer - Analysis Techniques, Recursion Trees (3)

• To prove that T (n) ∈ O(n2) (via substitution), must show that T (n) ≤ dn2 forsome d > 0

T (n) ≤ 3T(⌊n4

⌋)+ cn2

≤ 3d⌊n4

⌋2+ cn2

≤ 3d(n4

)2+ cn2

= 316dn

2 + cn2

≤ dn2

when d ≥ 1613c

19

Chapter 4: Divide and Conquer - Analysis Techniques, The Master Theorem

• The Master Theorem allows easy solutions to recursions of the form

T (n) = aT

(n

b

)+ f(n)

– f(n) represents the cost of dividing and combining

– Omitting floors and ceilings has no effect on the results

• The Master Theorem:

Let

1. a ≥ 1, b > 1 be constants

2. f(n) be a function that is asymptotically positive

3. T (n) is defined for integers n ≥ 0 as

T (n) = aT

(n

b

)+ f(n)

where nb is either dn/be or bn/bc

Then T (n) has the following asymptotic bounds:

T (n) =

Θ(nlogba) when f(n) ∈ O(nlogba−ε) for constant ε > 0

Θ(nlogbalg n) when f(n) ∈ Θ(nlogba)

Θ(f(n)) when f(n) ∈ Ω(nlogba+ε) for constant ε > 0,af(nb ) ≤ cf(n) for some constant c < 1, and

for all sufficiently large n

• Points re the theorem:

1. Form of solution based on comparison of f(n) with nlogba, the larger beingthe determining factor

– In the first case, f(n) can’t be just smaller, but must be so by a factorof nε

– In the third case, f(n) can’t be just larger, but must be so by a factorof nε AND must satisfy af(nb ) ≤ cf(n)

2. Not all situations are covered

– For example, when f(n) > nlogba but is not polynomially smaller (andsimilarly for < case)

20

Chapter 4: Divide and Conquer - Analysis Techniques, The Master Theorem(2)

• Examples:

1. T (n) = 9T (n/3) + na = 9, b = 3, f(n) = nnlogba = nlog39 = n2

Since f(n) ∈ O(n2−ε), where ε = 1, the Master Theorem appliesT (n) ∈ Θ(n2)

2. T (n) = T (2n/3) + 1a = 1, b = 3/2, f(n) = 1nlogba = nlog3/21 = n0 = 1Since f(n) ∈ Θ(1),T (n) ∈ Θ(lg n)

3. T (n) = 3T (n/4) + nlg n

a = 3, b = 4, f(n) = nlg n

nlogba = nlog43 = n0.793

Since f(n) ∈ Ω(nlog43+ε), where ε ∼ 0.2, andcf(n/b) = 3(n/4)lg(n/4) ≤ 3

4nlg n = cf(n) for c = 34 , the Master Theorem

appliesT (n) ∈ Θ(nlg n)

4. T (n) = 2T (n/2) + nlg n

a = 2, b = 2, f(n) = nlg n

nlogba = nlog22 = nnlg n is asymptotically greater than n, but not polynomially larger:

f(n)

nlogba=nlg n

n= lg n

which is asymptotically smaller then n2 for any ε > 0The Master Theorem does not apply

21

Chapter 4: Divide and Conquer - Analysis Techniques, Proof of the MasterTheorem

22