Download ppt - 1 Analysis of Algorithms Chapter - 02 Recurrences

1

Analysis of Algorithms

Chapter - 02Recurrences

2

Definition: A recurrence is an equation or inequality that describes a function in terms of its value on smaller inputs. The worst-case running time T(n) of an algorithm could be described as by the recurrence

T(n) = aT(n/b) + f(n) where a≥1, b>1, and f(n) is a given function. Three methods will be discussed to solve the recurrences obtaining asymptotic bounds on the solution. Substitution method:

We guess a bound and then use mathematical induction to prove that our guess is correct.

Recursion-tree method: It converts the recurrence into a tree whose nodes represent the cost incurred at various levels of the recursion We use techniques for bounding summations to solve the recurrence.

Master method: It provides bounds for recurrences of the above form. It requires memorization of three cases. Once you memorize the cases, determining asymptotic bounds for many recurrences is easy.

Methods

3

Definition of Proof: The process or an instance of establishing the validity of a statement especially by derivation from other statements in accordance with principles of reasoning.

Proof by Induction: Let Pn be a statement for all the positive integers (n = 1, 2, 3, . . .). If the following two properties hold:

1. P1 is true.2. Pk+1 is true if Pk is true for each positive integer k.

Then Pn is true for all n.

Why Induction Works? Let S be the set of all numbers n for which Pn is false. Let k be the minimum number in S. k > 1 since by the first property of the induction definition, P1 is

true. By the minimality of k, Pk−1 is true and Pk is false. A contradiction to the second property of the induction definition.

Induction

4

Prove that for any integer n ≥ 1:1 + 2 + 3 + · · · + n = n(n+1)/2 .

Define:

− L(n) = 1 + 2 + 3 + · · · + n.− R(n) = n(n+1)/2 .

Prove that L(n) = R(n) for n ≥ 1.

Verifying the Claim

First check for small values of n:

A Summation Problem

nn L(n)L(n) R(n)R(n)

11 1=11=1 1.2/2=11.2/2=1

22 1+2=31+2=3 2.3/2=32.3/2=3

33 1+2+3=61+2+3=6 3.4/2=63.4/2=6

:: :: ::

1010 1+2+3+…+10=551+2+3+…+10=55 10.11/2=5510.11/2=55

5

Idea: Compute the value of 2L(n).

Example: 2(1 + 2 + 3) = (1 + 2 + 3) + (3 + 2 + 1)

= (1 + 3) + (2 + 2) + (3 + 1)= 3 · 4= 12= 2(3·4/2)

Hence2·L(n) = (1 + · · · + n) + (n + · · · + 1)

= (1 + n) + (2 + (n − 1)) + · · · + (n + 1) = (n + 1) + (n + 1) + · · · + (n + 1) = n(n + 1)

This implies that: L(n) = n(n+1)/2 = R(n).

A Direct Proof

6

For n=1, L(1) = 1 and R(1) = (1.2)/2 = 1 So, for n=1, it is true.

Let us assume that it is true for n=k, that is, L(k) = R(k) =>1+2+3+…..+k = k(k+1)/2

Now, if we can prove that it is true for n=k+1 also, then it can be said that it is true for all n. For n=k+1,

L(k + 1) = 1 + 2 + · · · + k + (k + 1)= L(k) + (k + 1)= R(k) + (k + 1)= k(k + 1)/2 + (k + 1)= (k + 1)(k/2 + 1)= (k + 1)(k + 2)/2= R(k + 1)

Hence L(n) = R(n), for all integer n.

A Proof by Induction

7

Substitution Method

8

The method for solving recurrences entails two steps:− Guess the form of the solution− Use mathematical induction to find the constants and show that the solution works.

This method is powerful, but it obviously can be applied only in cases when it is easy to guess the form of the solution. This method can be used to establish either upper or lower bounds on a recurrence.

Problem: Let us determine an upper bound on the following recurrence:

T(1) = 0.T(n) = 2T(n/2) + n, for n>1.

Solution: Compute the solution for small powers of 2:T(2) = 2T(1) + 2 = 2.T(4) = 2T(2) + 4 = 8.T(8) = 2T(4) + 8 = 24.T(16) = 2T(8) + 16 = 64.T(32) = 2T(16) + 32 = 160.

Recurrence -1

9

Guessing the Solution - Guess T(n) = n log2 n for n a power of 2.

A Proof by InductionFor n=1, T(1)=0, that is T(n)=nlog2 nSo, for n=1, it is true.

Let us assume that it is true for n=k/2, that is, T(k/2)=(k/2) log2 (k/2)

Now, if we can prove that it is true for n=k also, then it can be said that it is true for all n.

For n=k,T(k) = 2T(k/2) + k = 2(k/2) log2(k/2) + k = k(log2 k − 1) + k = k log2 k

Hence T(n) = n log2 n, for all integer n.

Recurrence -1 (Contd.)

10

Recurrence -2


T(1) = a.T(n) = 2T(n/2) + bn.For some constants a, b (independent of n).

Solution: Compute the solution for small powers of 2: T(2) = 2T(1) + 2b = 2b + 2a. T(4) = 2T(2) + 4b = 8b + 4a. T(8) = 2T(4) + 8b = 24b + 8a. T(16) = 2T(8) + 16b = 64b + 16a. T(32) = 2T(16) + 32b = 160b + 32a.

Guessing the Solution- Guess T(n) = bn log2 n + an, for n a power of 2

11

Verify the guess for small numbers: b · 1 log2 1 + a · 1 = a. b · 2 log2 2 + a · 2 = 2b + 2a. b · 4 log2 4 + a · 4 = 8b + 4a. b · 8 log2 8 + a · 8 = 24b + 8a. b · 16 log2 16 + a · 16 = 64b + 16a. b · 32 log2 32 + a · 32 = 160b + 32a.

A Proof by Induction For n=1, T(1)=a, that is T(n)=b.nlog2 n + a.n So, for n=1, it is true. Let us assume that it is true for n=k/2, that is, T(k/2)=b.(k/2) log2 (k/2) + a.(k/2) Now, if we can prove that it is true for n=k also, then it can be said that it is true for all n.

For n=k, T(k) = 2T(k/2) + bk = 2(b(k/2) log2(k/2) + a(k/2)) + bk = bk(log2 k − 1) + ak + bk = bk log2 k + ak

Hence T(n) = b.n log2 n + a.n, for all integer n

12

Recurrence -3


T(1) = a.T(n) = T(n/2) + b.For some constants a, b (independent of n).

Solution: Compute the solution for small powers of 2: T(2) = T(1) + b = b + a. T(4) = T(2) + b = 2b + a. T(8) = T(4) + b = 3b + a. T(16) = T(8) + b = 4b + a. T(32) = T(16) + b = 5b + a.

Guessing the Solution- Guess T(n) = b log2 n + a, for n a power of 2

13

Verify the guess for small numbers: b · log2 1 + a = a. b · log2 2 + a = b + a. b · log2 4 + a = 2b + a. b · log2 8 + a = 3b + a. b · log2 16 + a = 4b + a. b · log2 32 + a = 5b + a.

A Proof by Induction For n=1, T(1)=a, that is T(n)=b log2 n + a

So, for n=1, it is true.Let us assume that it is true for n=k/2, that is, T(k/2)=b log2 (k/2) + aNow, if we can prove that it is true for n=k also, then it can be said that it is true for all n. For n=k,T(k) = T(k/2) + b = (b log2(k/2) + a) + b = b(log2 k − 1) + a + b = b log2 k + a

Hence T(n) = b log2 n + a, for all integer n

TT((nn) = 2) = 2TT((nn/2) + /2) + nn = O( = O(nn lg lg nn)) Thus, we need to show that Thus, we need to show that TT((nn) ) c nc n lg lg nn with an with an

appropriate choice of appropriate choice of cc Inductive hypothesis: assume

T(n/2) c (n/2) lg (n/2) Substitute back into recurrence to show that

T(n) c n lg n follows, when c 1 T(n) = 2 T(n/2) + n

2 (c (n/2) lg (n/2)) + n = cn lg(n/2) + n= cn lg n – cn lg 2 + n= cn lg n – cn + n cn lg n

for c 1= O(n lg n)

for c 1

Recurrence -4

Iteration MethodIteration Method

Iteration method:Iteration method:

Expand the recurrence k times

Work some algebra to express as a summation

Evaluate the summation

Iteration Method – ExampleIteration Method – Example

T(n) = n + 2T(n/2)T(n) = n + 2T(n/2)T(n) = n + 2T(n/2) T(n) = n + 2T(n/2)

= n + 2(n/2 + 2T(n/4)) = n + 2(n/2 + 2T(n/4)) = n + n + 4T(n/4)= n + n + 4T(n/4) = n + n + 4(n/4 + 2T(n/8))= n + n + 4(n/4 + 2T(n/8)) = n + n + n + 8T(n/8)= n + n + n + 8T(n/8)… … = in + 2= in + 2iiT(n/2T(n/2ii)) = kn + 2= kn + 2kkT(1) T(1) = nlgn + nT(1) = = nlgn + nT(1) = ΘΘ(nlgn)(nlgn)

Assume: n = 2k

n = 2k

Taking lg on both sideslg n = lg (2k )lg n = klg2lg n = k x 1lg n = k

TT((nn)) = = cc + + TT((nn-1)-1) == cc + + cc + + TT((nn-2)-2)== 22cc + + TT((nn-2)-2) == 22cc + + cc + + TT((nn--

3)3)== 33cc + + TT((nn-3)-3) …… kckc + + TT((nn--kk) ) = = ckck + + TT((nn--kk))

So far for So far for nn kk we have we have T(n) = ck + T(n-k)

To stop the recursion, we should have To stop the recursion, we should have n - k = 0 k = n T(n) = cn + T(0) = cn

Thus in general Thus in general TT((nn) = ) = OO((nn))

0)1(

00)(

nnTc

nnT

TT((nn) ) = = nn + + TT((nn-1)-1) = = nn + + nn-1 + -1 + TT((nn-2)-2)

= = nn + + nn-1 + -1 + nn-2 + -2 + TT((nn-3)-3)= = nn + + nn-1 + -1 + nn-2 + -2 + nn-3 + -3 + TT((nn-4)-4)= …= …= = nn + + nn-1 + -1 + nn-2 + -2 + nn-3 + … + (-3 + … + (nn--kk+1) + +1) + TT((nn--kk))

== for for nn kk

To stop the recursion, we should have To stop the recursion, we should have nn - - kk = 0 = 0 kk = = nn

0)1(

00)(

nnTn

nnT

2

10)0(

11

nniTi

n

i

n

i

)(2

1)( 2nO

nnnT

)(1

knTin

kni

The Master Method Based on the Master theorem. T(n) = aT(n/b) + f(n)

• a 1, b > 1 are constants.

• f(n) is asymptotically positive.

Requires memorization of three cases.

The Master Theorem

Theorem:Let a 1 and b > 1 be constants, let f(n) be a function, and

Let T(n) be defined on nonnegative integers by the recurrence T(n) = aT(n/b) + f(n), where we can replace n/b by n/b or n/b. T(n) can be bounded asymptotically in three cases:

1. If f(n) = O(nlogba–) for some constant > 0, then T(n) = (nlogba).

2. If f(n) = (nlogba), then T(n) = (nlogbalg n).3. If f(n) = (nlogba+) for some constant > 0,

and if, for some constant c < 1 and all sufficiently large n, we have a·f(n/b) c f(n), then T(n) = (f(n)).

Theorem:Let a 1 and b > 1 be constants, let f(n) be a function, and

Let T(n) be defined on nonnegative integers by the recurrence T(n) = aT(n/b) + f(n), where we can replace n/b by n/b or n/b. T(n) can be bounded asymptotically in three cases:

1. If f(n) = O(nlogba–) for some constant > 0, then T(n) = (nlogba).

2. If f(n) = (nlogba), then T(n) = (nlogbalg n).3. If f(n) = (nlogba+) for some constant > 0,

and if, for some constant c < 1 and all sufficiently large n, we have a·f(n/b) c f(n), then T(n) = (f(n)).

The Master TheoremThe Master Theorem

Given: a Given: a divide and conquerdivide and conquer algorithm algorithm

An algorithm that divides the problem of size n into a subproblems, each of size n/b

Let the cost of each stage (i.e., the work to divide the problem + combine solved subproblems) be described by the function f(n)

The Master TheoremThe Master Theorem

if T(n) = aT(n/b) + f(n) if T(n) = aT(n/b) + f(n) where a where a ≥ 1 & b > 1≥ 1 & b > 1 thenthen

1

0

largefor )()/(

& )(

)(

)(

)(

lg)(

log

log

log

log

log

c

nncfbnaf

nnf

nnf

nOnf

nf

nn

n

nT

a

a

a

a

a

b

b

b

b

b

Understanding Master TheoremUnderstanding Master Theorem

In each of the three cases, we are comparing f(n) In each of the three cases, we are comparing f(n) with with nnloglog

bba a , the solution to the recurrence is , the solution to the recurrence is

determined by the larger of the two functions.determined by the larger of the two functions.

In case 1, if the function In case 1, if the function nnloglogbb

a a is the larger, then the is the larger, then the solution T(n) = solution T(n) = ΘΘ((nnloglog

bbaa).).

In case 3, if the function f(n) is the larger, then the In case 3, if the function f(n) is the larger, then the solution is T(n) = solution is T(n) = (f(n)).(f(n)).

In case 2, if the two functions are the same size, then In case 2, if the two functions are the same size, then the solution is T(n) = the solution is T(n) = ΘΘ((nnloglog

bba a lg nlg n) = ) = ΘΘ(f(n) lg n).(f(n) lg n).


In case 1, not only must f(n) be smaller than In case 1, not only must f(n) be smaller than nnloglogbb

aa, , it must be polynomially smaller. That is f(n) must it must be polynomially smaller. That is f(n) must be asymptotically smaller than be asymptotically smaller than nnloglog

bbaa by a factor of by a factor of

nnεε for some constant for some constant εε > 0. > 0.

In case 3, not only must f(n) be larger than In case 3, not only must f(n) be larger than nnloglogbb

aa , it , it must be polynomially larger and in addition satisfy must be polynomially larger and in addition satisfy the “regularity” condition that:the “regularity” condition that:

a f(n/b) a f(n/b) ≤ c f(n).≤ c f(n).


It is important to realize that the three cases do not It is important to realize that the three cases do not cover all the possibilities for f(n).cover all the possibilities for f(n).

There is a gap between cases 1 and 2 when f(n) is There is a gap between cases 1 and 2 when f(n) is smaller than smaller than nnloglog

bbaa but not polynomially smaller. but not polynomially smaller.

There is a gap between cases 2 and 3 when f(n) is There is a gap between cases 2 and 3 when f(n) is larger than larger than nnloglog

bbaa but not polynomially larger. but not polynomially larger.

If f(n) falls into one of these gaps, or if the If f(n) falls into one of these gaps, or if the regularity condition in case 3 fails to hold, the regularity condition in case 3 fails to hold, the master method cannot be used to solve the master method cannot be used to solve the recurrence.recurrence.

Using The Master MethodUsing The Master MethodCase 1Case 1

T(n) = 9T(n/3) + nT(n) = 9T(n/3) + n a=9, b=3, f(n) = n nlogb a = nlog3 9 = (n2) >> from log332

Since f(n) = O(nlog3 9 - ) = O(n2-0.5) = O(n1.5) where =0.5 case 1 applies:

Thus the solution is T(n) = (n2)

aa bb nOnfnnT loglog )( when )(


T(n) = T(2n/3) + 1T(n) = T(2n/3) + 1 a=1, b=3/2, f(n) = 1 nlogba = nlog3/21 = n0 = 1 Since f(n) = (nlogba) = (1) case 2 applies:

Thus the solution is T(n) = (lg n)

aa bb nnfnnnT loglog )( when lg)(


T(n) = 3T(n/4) + n lg nT(n) = 3T(n/4) + n lg n a=3, b=4, f(n) = n lg n nlogba = nlog43 = n0.793 = n0.8

Since f(n) = (nlog43+) = (n0.8+0.2)= (n) where 0.2, and for sufficiently large n,

a . f(n/b) = 3(n/4) lg(n/4) < (3/4) n lg n for c = 3/4 case 3 applies:

Thus the solution is T(n) = (n lg n)

abnnfnfnT log)( when )()(

29

Recursion-tree Method

30

Although the substitution method can provide a concise proof that a solution to a recurrence is correct, it is sometimes difficult to come up with a good guess.

Drawing out a recursion-tree is a straight forward way to devise a good guess.

In a recursion-tree, − each node represents the cost of a single sub-problem somewhere in the set of recursive function invocations.− We sum the costs within each level of the tree to obtain a set of per-level costs.− Then we sum all the per-level cost to determine the total cost of all levels of the recursion.

Recursion trees are particularly useful when the recurrence describes the running time of a divide-and-conquer algorithm.

A recursion tree is best used to generate a good guess, which is then verified by the substitution method.

In this section, we will use recursion-trees to generate good guess.

The method

31

Let us see a recursion-tree would provide a good guess for the recurrence:

We start by focusing on finding an upper bound for the solution. Following figure shows the derivation of the recursion-tree, assuming n as exact power of 4.

We create a recursion tree for T(n) = 3T(n/4) + cn2, c>0.

Example-1

)()4/(3)( 2nnTnT

32

Part (a) of the figure shows T(n), which is expanded in Part (b) into an equivalent tree representing the recurrences.

The cn2 term at the root represents the cost at the top level of recursion, and the three subtrees of the root represent the cost incurred by the subproblems of size n/4.

Part (c) shows this process carried one step further by expanding each node with cost T(n/4) from part (b). The cost for each of the three children of the root is c(n/4)2.

We continue expanding each node in the tree by breaking it into its constituent parts as determined by the recurrence.

Since the subproblem sizes decrease as we get further from the root, we eventually must reach a boundary condition.

How far from the root do we reach one? The subproblem size for a node at depth i is n/4i. Thus the

subproblem size hits n=1, when n/4i=1, that is, i=log4 n. Thus the tree has levels (0, 1, 2, ……, log4 n).

Explanation

33

Now, we determine the cost at each level of the tree. Each level has three times more nodes than the level above, So the

number of nodes at depth i is 3i. Each node at depth i has a cost of c(n/4i), for i=0, 1, 2, ….., log4n -1. Multiplying, we get that total cost over all nodes at depth i is 3ic(n/4i)2 =

(3/16)i cn2. The last level at, depth log4n, has 3log

4n = nlog

43 nodes, each contributing

cost T(1), for a total cost of nlog43 .T(1) , which is Θ(nlog

43).

Now add up the costs over all levels to determine the cost for the entire tree.

Explanation (Contd.)

.2

34

log2)16/3(1

1

0

34

log216

3

14

log

0

34

log216

3

)3

4log

(21

4log

16

3....2

2

16

3216

32)(

n

ncn

incn

i

n

incn

i

ncnn

cncncnnT

34

Now, we can use the substitution method to verify that our guess was correct, that is T(n)=O(n2) is an upper bound for the recurrence

We want to show that T(n) ≤ dn2, for some d>0. Using the same constant c>0 as before, we have


)()4/(3)( 2nnTnT

).()(,

.)3/16(,

16

3

)4/(3

4/3

)4/(3)(

2

2

22

22

22

2

nnTHence

cdwheredn

cndn

cnnd

cnnd

cnnTnT

35

Let us see a recursion-tree would provide a good guess for the recurrence:

T(n) = T(n/3) + T(2n/3) + O(n).

Let c>0 be the constant factor for the term O(n).

Example-2

36

Now, when we add the values across the levels of the recursion-tree, we get a value of cn for every level.

The longest path from root to a leaf isn→(2/3)n →(2/3)2n →….. →1.

Since (2/3)kn =1, when k = log3/2n, The height of the tree is log3/2n. Intuitively, we expect the solution to the recurrence be O(cn.log3/2n) = O(n.lg

n) We can use the substitution method to verify that our guess was correct,

that is T(n)=O(n.lg n) is an upper bound for the given recurrence . We want to show that T(n) ≤ dn.lg n, for some d>0. Using the same constant c>0 as before, we have


).lg()(,

.lg

)3/23(lglg

)2lg)3/2(3lg)3/2(3lg)3/((lg

))2/3lg()3/2(3lg)3/((lg

))2/3lg()3/2(lg)3/2((

)3lg)3/(lg)3/((

)3/2lg()3/2()3/lg()3/(

)3/2()3/()(

nnnTHence

ndn

cndnndn

cnnnndndn

cnnndndn

cnndnnd

ndnnd

cnnndnnd

cnnTnTnT