Recurrence Tutorial

8/13/2019 Recurrence Tutorial

1/14

Recurrence Relation Tutorial

from The Design and Analysis of Algorithms by Levitan

Document last modified: 01/30/2014 09:55:54

RESOURCES

C251 Text: Discrete Mathematics and Its Applications by Kenneth Rosen

The Design and Analysis of Algorithms by Anany Levitan

http://www.research.att.com/~njas/sequences- Type in a few sequence terms and finds the closed form, if oneexists. Try: 1, 3, 6, 10

OVERVIEW

We are interested indetermining the

running timebehavior ofalgorithms

expressed asbounds using or

O representation.The diagram at

right reflects theoverall approach for

analyzing analgorithm.

Recurrencerelations result

naturally from theanalysis of

recursivealgorithms,

specifying thealgorithm's runtime; solving recurrence relations yields a closed-end formula for calculation of run time.

For example, the recursive n! algorithm of:

int factorial (int n) {

if (n == 1) return 1;

return factorial (n-1) * n;

}

The running time, T(n), can be defined as recurrence:

T(1) = 1

T(n) = T(n-1) + 1 for all n>1

where 1 is the running time for each execution of the factorial function.

Solving the recurrence by recognizing the summation leads to closed-endformula:

T(1) = 1

T(2) = T(1) + 1 = 1 + 1 = 2

T(3) = T(2) + 1 = 2 + 1 = 3

:

T(n) = T(n-1) + 1 = n - 1 + 1 = n

or by substitution backward to some base

T(n) = T(n-1)+1 = T(n-2)+1+1 = ... = T(1) + (n-1) = 1 + (n-1) = n

rence Tutorial http://homepages.ius.edu/RWISMAN/C455/html/notes/chapter4/R

4 1/30/2014


2/14

The run time for execution of factorialcan now be directly computed for a given n.

T(10) = 10

More importantly, when expressing the worst case run time bounds we can then write:

| O(1) if n=1T(n) = | | O(n) if n>1

SEQUENCES AND RECURRENCE RELATIONS

Sequence - a numerical, ordered list of numbers

In the following

T - a recurrence function defining a sequence

n - a parameter to function T

T(n) - the sequence term generated by functionTfor parameter n

Example

T(n) = T(n-1) + n for n > 0 B1

T(0) = 0 B2

Generate sequence values

T(0) = 0

T(1) = T(1-1) + 1 = 0 + 1 = 1

T(2) = T(2-1) + 2 = T(1) + 2 = 1 + 2 = 3

T(3) = T(3-1) + 3 = T(2) + 3 = 3 + 3 = 6

T(4) = T(4-1) + 4 = T(3) + 4 = 6 + 4 = 10

Examples:

T(0) = 0

T(n) = T(n-1) + 1

T(n) 1 2 3 4

n 1 2 3 4

n-1 0 1 2 3

T(0) = 0

T(n) = T(n-1) + 2

T(n) 2 4 6 8

n 1 2 3 4

n-1 0 1 2 3

T(0) = 0 T(1) = 1

T(n) = T(n/2) + n

T(n) 0 1 3 4 7 8 10 11

n 0 1 2 3 4 5 6 7

n/2 0 0 1 1 2 2 3 3

Question 4.1 - What is T(13)?

T(0) = 0 T(1) = 1

T(n) = T(n/2) + n

T(n) 0 1 3 6 7 11 12 14

n 0 1 2 3 4 5 6 7

n/2 0 1 1 2 2 3 3 4

T(0) = 0

T(n) = T(n/4) + n

T(n) 0 1 2 3 5 6 7 8 10 11 12 13 15 16 17 18 21 22 23 24 26 27 28 29 31

n 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24

n/4 0 0 0 0 1 1 1 1 2 2 2 2 3 3 3 3 4 4 4 4 5 5 5 5 6

n/4 0 1 1 1 1 2 2 2 2 3 3 3 3 4 4 4 4 5 5 5 5 6 6 6 6

Question 4.2 - What is T(97)?

Inductive proofs on the recurrence: T(n) = T(n-1) + 1


4 1/30/2014


3/14

Typically assume the case for parameter nthat generates a sequence term for recurrence T(n) and prove

T(n+1)

T(n) = T(n-1) + 1

T(n+1)=T((n+1)-1)+1 = T(n) + 1

Assuming the problem size is a multiple of 2, n = 2k,

T(1) = 1

T(n) = T(n/2) + n

T(2k) = T(2k/2) + 2k = T(2k-1) + 2k

T(n) 1 3 7 15 31 63 127

n 1 2 4 8 16 32 64

k 0 1 2 3 4 5 6

Question 4.3 For above recurrence:

What is the next value of n after 64?

What is the next term after T(2k)?

Notation

Either Tnor T(n) which stresses that a sequence is a function where T(n) is the generic nth term.

Defining a sequence (two ways)

Closed-endformula of generic term as a function of n

T(n) = 2n for n 0

Recurrenceequation relating generic term to one or more other sequence terms combined with one or more

explicit values of first term(s).

T(n) = T(n-1) + n for n > 0 B1

T(0) = 0 B2

B1above is a recurrence relation, where the generic term, T(n), is defined by other sequence

terms, T(n-1)+n.

B2above is an initial condition

Recurrence Relation

Example:Fibonacci sequence definition:

int fibonacci (int n) {

if( n==0 || n==1 ) return n;

return fibonacci(n-1) + fibonacci(n-2);

}

has the corresponding recurrence:

F(0) = 0 Initial

F(1) = 1 InitialF(n) = F(n-1) + F(n-2) for n>1 Recurrence


4 1/30/2014


4/14

Solve Recurrence Relation - find closed-end formula for generic nthterm of sequence that satisfies both the recurrence term

and initial condition or prove that the sequence does not exist.

Example

T(n) = T(n-1) + n for n>0 Recurrence

T(0) = 0 Initial condition

Solution of subject to initial condition T(0) = 0 is:

T(n) = n(n+1)for n 0 2

Induction Hypothesis

Verify closed-end solution - Substitute solution into recurrence formula to check that equality holds or use induction to prove.

B1 must hold for every n > 0:

Substitution Induction

T(n) = n(n+1) 2

= (n2+n)/2

= (n2-n+2n)/2

= (n2-n)/2+2n/2

= (n-1)n + n 2

= T(n-1) + n

IH: T(n) = n(n+1) 2

Base: Initial condition must hold for n=0:

T(0) = 0(0+1) = 2

0

T(1) = 1(1+1) = 2

1

Show: T(n+1) = (n+1)(n+1+1) = (n+1)(n+2)

2 2

T(n+1) = T(n+1-1) + (n + 1)

= T(n) + n+1

Recurrence definitionT(n) = T(n-1) + n

= n(n+1)+ n+1 2 IH: T(n) = n(n+1) 2

= (n2+n)/2 + 2(n+1)/2

= (n2+3n+2)/2

= (n+1)(n+2) 2

Particular solution to recurrence - a specific sequence that satisfies recurrence;

T(n) = n(n+1)for n 0 2

is a particular solution.

General solution to recurrence - formula that specifies all sequences, typically includes arbitrary constants. A general solutionfor recurrence is:

T(n) = c + n(n+1)for n 0

2

choosing different values for cgives all solutions.

METHODS FOR SOLVING RECURRENCE RELATIONS

No universal method but common techniques for a variety of recurrences.


4 1/30/2014


5/14

Forward substitution to finde x a c t closed-end equation for T(n) - limited to simple recurrences

Start with initial term(s) given by initial conditions, generate first few terms in hope of seeing a pattern that can beexpressed by a closed-end formula

Example

T(n) = 2T(n-1)+1 for n > 1

T(1) = 1

Generate

T(1) = 1= 21-1

T(2) = 2T(2-1)+1 = 2T(1)+1 = 2*1+1 = 3= 22-1

T(3) = 2T(3-1)+1 = 2T(2)+1 = 2*3+1 = 7= 23-1

T(4) = 2T(4-1)+1 = 2T(3)+1 = 2*7+1 = 15= 24-1

Hope

Observe pattern that suggests from n=1, 2, 3, 4

T(n) = 2T(n-1)+1 = 2

n

-1

Prove T(n) = 2n-1 correct by substitution or induction

Substitution

Inductive Hypothesis

T(n) = 2n-1

or

T(n-1) = 2n-1-1

Basis

T(1) = T(2-1) = 22-1-1 = 21-1 = 1

T(n) = 2n-1 IH

= 2*2n-1 - 2 + 1

= 2*(2n-1-1) + 1

= 2T(n-1) + 1 Substitute T(n-1)

Induction

Basis step

T(1)=21-1=1


T(n) = 2n-1

Inductive step

Show T(n+1) = 2n+1-1

T(n+1) = 2T(n+1-1)+1 Recurrence: T(n) = 2T(n-1)+1

= 2*T(n) + 1

= 2*(2n-1)+1 IH: T(n) = 2n-1

= 2n+1-2+1

= 2n+1-1

Question 4.4

T(n) = T(n-1)+1 for n > 0

T(0) = 1

Generate

T(1)=

T(2)=

T(3)=

Closed form

Observe pattern T(n) =


4 1/30/2014


6/14

Question 4.5 - Use substitutionto prove T(n) = closed form

Question 4.6 - Useinductionto prove T(n) = closed form

Substitution


T(n) = __________________

or

T(n-1) = _________________

Basis

T(0) = 0 + 1 = 1

T(n) = ____________

= ___________ Substitute T(n-1)

Induction

Basis step

T(0) = 0 + 1 = 1


T(n) = ______________

Inductive step

Show T(n+1) = ______________

T(n+1) = ___________________ Recurrence

= ___________________ IH: T(n) =

= ___________________

Backward substitution to finde x a c t closed-end equation for T(n)

express T(n-1) as a function of T(n-2)substitute result to give T(n) as a function of T(n-2)

repeating for T(n-2) gives T(n) as function of T(n-3)hope to see a pattern to express T(n) as function of T(n-i) for i=1, 2, ...

select ito make n-ireach the initial conditionclosed-end formula

use standard summation formula from Appendix A will often lead to closed-end formula1.when lucky, get form similar to: T(n) = Base case + f(n), and can solve by substitution2.

Example - Backward substitution to findexactclosed-end equation for T(n)

T(n) = T(n-1)+1 for n > 0

T(1) = 1

T(n) = T(n-1) + 1 = [T(n-2)+1] + 1 sub. T(n-1) = T((n-1)-1)+1 = T(n-2)+1

= T(n-2) + 2 = [T(n-3)+1] + 2 sub. T(n-2) = T(n-3)+1

= T(n-3) + 3 = [T(n-4)+1] + 3 sub. T(n-3) = T(n-4)+1

= T(n-4) + 4

After 4substitutions, observe that:

T(n) = T(n-4) + 4

T(5) = T(5-4) + 4

= T(1) + 4

= 1 + 4

= 5

After i substitutions, observe that:

T(n) = T(n-i) + i

T(n) = T(n-4) + 4 after 4 substitutions

Initial condition must hold: T(1) = T(n-i)

n-i = 1 i = n-1


4 1/30/2014


7/14

n=1: T(n-i) = T(n-(n-1))

= T(1-(1-1))

= T(1)

Closed-end equation

T(n) = T(n-i) + i

= T(n -(n-1)) + n - 1

= T(1) + n - 1

= 1 + n - 1

= n

Question 4.7

T(1) = 1

T(n) = T(n-1) + 5 for n>1

a. What is T(2), T(3), T(4)?

T(n-1)=T(n-2)+5

T(n) = T(n-1) + 5 = [T(n-2)+5]+5

b. Use backward substitution to find the closed-end equation.

c. Verify the closed-end equation is correct for T(4).

Example - Backward substitution to findexactclosed-end equation for T(n)

T(1) = 1

T(n) = 2T(n-1)+1 for n > 1

T(n) = 2T(n-1)+1 = 2[2T(n-2) + 1] + 1 sub. T(n-1) = 2T((n-1)-1)+1 = 2T(n-2)+1

= 22T(n-2)+2+1 = 22[2T(n-3)+1]+2+1 sub. T(n-2) = 2T(n-3)+1

= 23T(n-3)+22+2+1 = 23[2T(n-4)+1]+4+2+1 sub. T(n-3) = 2T(n-4)+1

= 24T(n-4)+23+22+2+1

After 4substitutions:

T(n) = 24T(n-4)+23+22+2+1

T(5) = 24T(5-4)+23+22+2+1

= 24T(1)+23+22+2+1

= 24+23+22+2+1

After i substitutions, observe that:

T(n) = 2iT(n-i) + 2i-1+... +23+22+21+20

Initial condition must hold: T(1) = 2iT(n-i)

n-i = 1 i=n-1

n=1: 2iT(n-i) = 2n-1T(n-(n-1)) = 21-1T(1-(1-1)) = 20T(1) = T(1)

Closed-end formula by exponential series:

From Appendix A.5, p. 1147


4 1/30/2014


8/14

T(n) = 2iT(n-i) + 2i-1+ 2i-2+ 2i-3+...+23+22+21+20

= 2iT(n-i) + 2i- 1

= 2n-1T(n-(n-1)) + 2n-1- 1 i = n-1

= 2n-1T(1) + 2n-1- 1 T(1)=1

= 2n-1+ 2n-1- 1

= 2*2n-1- 1

= 2n- 1

Proof -same as earlier forward substitution or induction

Example - Backward Substitution

T(0) = 0

T(n)=T(n-1) + n

T(n) =T(n-1) + n sub. T(n-1) = T((n-1)-1) + (n-1) = T(n-2) + n-1

=T(n-2) + n-1+ n sub. T(n-2) = T(n-3) + n-2

=T(n-3) + n-2+ n-1 + n sub. T(n-3) = T(n-4) + n-3

=T(n-4) + n-3+ n-2 + n-1 + n

After 4substitutions:

T(n) = T(n-4) + n-3 + n-2 + n-1 + n

T(4) = T(0) + 1 + 2 + 3 + 4 = 0 + 1 + 2 + 3 + 4

After isubstitutions, observe that:

T(n) = T(n-i) + (n-i+1) + (n-i+2) + ... + (n-2) + (n-1) + n

Initial condition must hold: T(0) = T(n-i)

n-i = 0 n = i

n=i=0: T(n-i) = T(n-n) = T(0)

Closed-end formula by standard summationfor n=i

T(n) = T(n-n) + (n-n+1) + (n-n+2) + ... + (n-2) + (n-1) + n

T(n) = T(0) + 1 + 2 + ... + n = 0 + 1 + 2 + ... + n = n(n+1)/2

Proof:

T(n) = n(n+1)/2 correct by substitution or induction

Substitution

Show: T(n)=T(n-1) + n

T(n) =n(n+1)/2

= (n2+ n)/2

= (n2- n +2n)/2

Induction

Basis step

T(0) = 0(0+1)/2 = 0

T(1) = 1(1+1)/2 = 1


T(k) =k(k+1)/2


4 1/30/2014


9/14

= (n2- n)/2 + 2n/2

= n(n - 1)/2 + n

= T(n-1) + n

Inductive step

Show T(k+1) = (k+1)(k+2)/2

T(k+1) = T(k+1-1)+(k+1) Recurrence

= T(k) + (k+1)

= k(k+1)/2+(k+1) IH substitution

= (k2+k)/2+(k+1)

= (k2+k)/2 + 2(k+1)/2

= (k2+3k+2)/2

= (k+1)(k+2)/2

Question 4.7.1 - Prove by induction the recurrence:

T(1) = 1

T(n) = T(n-1) + 5 for n > 1

has the closed form:

T(n) = 5(n-1) + 1

IH:

Basis:

Show:

Proof:

COMMON RECURRENCE TYPES IN ALGORITHM ANALYSIS

Decrease-by-onealgorithms exploits relationship between instance of size nand smaller instance of size n-1.Decrease-by-a-constant factor algorithms reduce instance of size nto an instance of size n/b(b=2 for many such

algorithms), solving the smaller instance recursively. If necessary, the solution of the smaller instance is combined to asolution of the given instance.

Divide-and-conquer divides a given instance into smaller instances, solving each recursively. If necessary solutions tosmaller instances are combined to give a solution to a given instance.

Decrease-by-onealgorithms exploits relationship between instance of size nand smaller instance of size n-1.

For the recursive n! algorithm:

int factorial (int n) {

if (n == 1) return 1; // Cost = 1

return factorial (n-1) * n; // Cost = T(n-1) + 1}

The running time, T(n), can be defined as:

T(1) = 1

T(n) = T(n-1) + 1 for all n > 1

where 1 is the running time for each execution of the factorial function.

Solving the recurrence by recognizing the summationleads to closed-end formula:

T(n) = T(1) + 1 + ... + 1 = 1 + 1 + ... + 1 = n


4 1/30/2014


10/14

Typical form of recurrence equation, wheref(n)is the time to reduce an instance to a smaller one is:

T(n) = T(n-1) + f(n)

Applying backward substitutions yields:

T(n) = T(n-1) + f(n)

= T(n-2) + f(n-1) + f(n)

= ...

= T(0)+

For a specific function f(n), can usually be computed exactly or order of growth determined.

As examples, for:

f(n)=1, = n

f(n)=lg n, (n lg n)

f(n)=nk, (nk+1) Consider: 4k + 4k + 4k + 4k = 4*4k=

4k+1

The sum, , can also be approximated by formulas involving integrals.

Decrease-by-a-constant factor algorithms reduce instance of size nto an instance of size n/b(b=2 for many suchalgorithms), solving the smaller instance recursively. If necessary, the solution of the smaller instance is extended to a

solution of the given instance.

For binary search:

int binarySearch(int A[], int key, int low, int high) {1.

if (low > high) return -1;2.

int mid=(low+high)/2;3.

if (key == A[mid]) return mid;4.

if(key > A[mid]) return binarySearch(A, key, mid+1, high);5.

else return binarySearch(A, key, low, mid-1);6.

}7.

Recurrence

T(1) = 1 Lines 1-4

T(n) = T(n/2) + 1 for n>1 Lines 5-6, problem size now n/2

Let n=2k(power of 2) and solve using backwards substitutions (see below) or other method.

T(2k) = T(2k/2) + 1 = k+1 = lg 2k+ 1 = lg n + 1

Typical form of recurrence equation, wheref(n)is the time to reduce an instance to a smaller one is:


14 1/30/2014


11/14

T(n) = T(n/b) + f(n)

where b>1 and f(n) is the time to reduce an instance to a smaller and to extend the solution to a solution of

a larger instance.

Valid only for n=bkfor k=0, 1, 2, .. ., that is nis a power of b.

Where nis not a power of bthere is round-off, usually involving floor or ceiling functions.

Standard approach is solve for n=bkfirst, then tweak to make valid for all n's.

For n=bk

, applying backward substitutions:

T(bk) = T(bk-1) + f(bk)

= T(bk-2) + f(bk-1) + f(bk)

= ...

= T(1) +

For a specific function f(x), can usually be computed exactly or order of growth

determined.

Example of binary search, f(x) = 1:

= k = logbn

Then for the binary search example:

T(1) = 1

T(n) = T(n/2) + 1 for n>1

Let n=2k

Note that k=lg n

T(n) = T(2k) = T(1) + = 1 + k = log2n + 1 or lg n + 1

Example - Binary Search

Recurrence

T(1) = 1

T(n) = T(n/2) + 1 for n>1

Let n=2k(power of 2) and solve using backwards substitutions.

T(2k) = T(2k/2)+1 = T(2k-1)+1 sub. T(2k-1) = T(2k-1/2)+1

= T(2k-1/2)+1+1 = T(2k-2)+1+1 sub. T(2k-2) = T(2k-2/2)+1

= T(2k-2/2)+1+1+1 = T(2k-3)+1+1+1 sub. T(2k-3) = T(2k-3/2)+1

= T(2k-3/2)+1+1+1+1 = T(2k-4)+1+1+1+1

After 4 substitutions:

T(2k) = T(2k-4)+1+1+1+1

T(24) = T(24-4)+1+1+1+1 = T(1)+4 = 1 + 4 = 5


14 1/30/2014


12/14

After isubstitutions:

T(2k) = T(2k-i) + i

Need iso that T(2k-i) = T(20) = T(1):

i=k

Solution

Base case: k=0, n = 2k= 20 = 1

T(20) = T(2k-i) + i = T(20-0) +0 = T(1) = 1

Closed-end equation by substitution

In general, for i=k

T(2k) = T(2k-i) + i

= T(20) + k

= T(1) + k = k + 1

k + 1 = lg 2k+ 1 since log22k=k

= lg n + 1 n=2k

Show: T(n) = lg n + 1 correct by substitution or induction.

Let n=2k (power of 2)

Substitution

Show: T(2k)=T(2k/2) + 1

T(2k) =lg 2k + 1

= k + 1

=(k-1+1)+ 1

= (lg 2k-1 + 1) + 1

= T(2k-1) + 1

= T(2k/2) + 1

Induction

Basis step

T(20)=T(1) = 1


T(2

k

) = lg 2

k

+ 1 = k + 1

Inductive step

Show T(2k+1) = k + 2

T(2k+1) = T(2k+1)/2+1 Recurrence

= T(2k+1-1) + 1

= T(2k) + 1

= (k+1) +1 IH, T(2k) = k + 1

= k + 2

= lg 2k + 2

Divide-and-conquer divides a given instance into smaller instances, solving each recursively. If necessary solutions

to smaller instances are combined to give a solution to a given instance.

Mergesort example

void mergeSort( double A[] ) { // A.length is power of 21.

int n = A.length;2.


14 1/30/2014


13/14

if (n > 1) {3.

double B[]=new double[n/2], C[]=new double[n/2];4. copy( A[0..n/2-1], B);5.

copy( A[n/2, n-1, C);6. mergeSort( B );7.

mergeSort( C );8. merge( B, C, A );9. }10.

}11.

Recurrence

T(1) = b

T(n) = T(n/2) + T(n/2) + cn for n>1

where:

b is the time to solve a problem of size 1

n the problem size, in this case a power of 2.

T(n/2) is the time to solve (mergeSort) half of the given problem (lines 7 and 8).

cnis the time to divide the problem in half (lines 4-6) and combine solutions to each half (line 9).

By backward substitution the recurrence yields:

T(n) = 21T(n/21) + cn = 21[2T(n/22)+cn/2] + cn sub. T(n/21) = 2T(n/22)+cn/2

= 22T(n/22) + 2cn = 22[2T(n/23)+cn/22] + 2cn sub. T(n/22) = 2T(n/23)+cn/22

= 23T(n/23) + 3cn = 23[2T(n/24)+cn/23] + 3cn sub. T(n/23) = 2T(n/24)+cn/23

= 24T(n/24) + 4cn

Substituting itimes yields (we're lucky, no summation!):

T(n)= 2iT(n/2i) + icn

Closed-end

Initial condition must hold: T(1) = 2iT(n/2i) = 20T(n/20)

2i=n=1 when i=0

2i=n=2lg nor i=lg n.

Closed-end solution:

T(n) = 2iT(n/2i) + icn replace i=lg n

= 2lg nT(n/2lg n) + (lg n)cn

= nT(n/n) + (lg n)cn

= nT(1) + (lg n)cn

= nb + cn lg n

Proof: T(n) = nb + cn lg n

Recurrence:

T(1) = b

T(n) = 2T(n/2) + cn for n>1

Basis:


14 1/30/2014


14/14

T(1) = 1b + c1 lg 1 = b + c*0 = b

Inductive Hypothesis:

T(n/2) = (n/2)b + c(n/2) lg (n/2)

Show:

T(n) = nb + cn lg n

T(n) = 2T(n/2) + cn Recurrence

= 2[ (n/2)b + c(n/2) lg (n/2)] + cn IH T(n/2) = (n/2)b + c(n/2) lg (n/2)

= nb + cn lg (n/2) + cn

= nb + cn ( lg n - lg 2) + cn

= nb + cn ( lg n - 1) + cn

= nb + cn lg n - cn + cn

= nb + cn lg n

General form of recurrence equation, where:

nis the problem sizeb 2 the number of problem divisions

n/bis the size of the smaller problem after divisiona 1is the number of subproblem solutions

f(n)is the time to divide an instance to a smaller ones and combine the solutions

T(n) = aT(n/b) + f(n)

Which, by backward substitution, we can derive:

where the order of growth solution T(n) depends upon the values of constants aand band the order of growth of thefunctionf(n).

Under certain assumptions about f(n), the above can be simplified to get explicit results about the order of growth of

T(n). That will be examined when considering the Master Theorem.


Documents

Recurrence Tutorial