Compare the growth rate of functionshuding/331material/notes2.pdfOutline 1 Compare the growth rate of functions 2 Limit Test 3 L’Hospital Rule 4 Stirling Formula c Hu Ding (Michigan

Outline

1 Compare the growth rate of functions

2 Limit Test

3 L’Hospital Rule

4 Stirling Formula

c©Hu Ding (Michigan State University) CSE 331 Algorithm and Data Structures 1 / 18

Compare the growth rate of functions

We have two algorithms A1 and A2 for solving the same problem,with runtime functions T1(n) and T2(n), respectively. Whichalgorithm is more efficient?We compare the growth rate of T1(n) and T2(n).If T1(n) = Θ(T2(n)), then the efficiency of the two algorithms areabout the same (when n is large).If T1(n) = o(T2(n)), then the efficiency of the algorithm A1 will bebetter than that of algorithm A2 (when n is large).By using the definitions, we can directly show whetherT1(n) = O(T2(n)), or T1(n) = Ω(T2(n)). However, it is not easy toprove the relationship of two functions in this way.



We have two algorithms A1 and A2 for solving the same problem,with runtime functions T1(n) and T2(n), respectively. Whichalgorithm is more efficient?

We compare the growth rate of T1(n) and T2(n).If T1(n) = Θ(T2(n)), then the efficiency of the two algorithms areabout the same (when n is large).If T1(n) = o(T2(n)), then the efficiency of the algorithm A1 will bebetter than that of algorithm A2 (when n is large).By using the definitions, we can directly show whetherT1(n) = O(T2(n)), or T1(n) = Ω(T2(n)). However, it is not easy toprove the relationship of two functions in this way.



We have two algorithms A1 and A2 for solving the same problem,with runtime functions T1(n) and T2(n), respectively. Whichalgorithm is more efficient?We compare the growth rate of T1(n) and T2(n).

If T1(n) = Θ(T2(n)), then the efficiency of the two algorithms areabout the same (when n is large).If T1(n) = o(T2(n)), then the efficiency of the algorithm A1 will bebetter than that of algorithm A2 (when n is large).By using the definitions, we can directly show whetherT1(n) = O(T2(n)), or T1(n) = Ω(T2(n)). However, it is not easy toprove the relationship of two functions in this way.



We have two algorithms A1 and A2 for solving the same problem,with runtime functions T1(n) and T2(n), respectively. Whichalgorithm is more efficient?We compare the growth rate of T1(n) and T2(n).If T1(n) = Θ(T2(n)), then the efficiency of the two algorithms areabout the same (when n is large).

If T1(n) = o(T2(n)), then the efficiency of the algorithm A1 will bebetter than that of algorithm A2 (when n is large).By using the definitions, we can directly show whetherT1(n) = O(T2(n)), or T1(n) = Ω(T2(n)). However, it is not easy toprove the relationship of two functions in this way.



We have two algorithms A1 and A2 for solving the same problem,with runtime functions T1(n) and T2(n), respectively. Whichalgorithm is more efficient?We compare the growth rate of T1(n) and T2(n).If T1(n) = Θ(T2(n)), then the efficiency of the two algorithms areabout the same (when n is large).If T1(n) = o(T2(n)), then the efficiency of the algorithm A1 will bebetter than that of algorithm A2 (when n is large).

By using the definitions, we can directly show whetherT1(n) = O(T2(n)), or T1(n) = Ω(T2(n)). However, it is not easy toprove the relationship of two functions in this way.



We have two algorithms A1 and A2 for solving the same problem,with runtime functions T1(n) and T2(n), respectively. Whichalgorithm is more efficient?We compare the growth rate of T1(n) and T2(n).If T1(n) = Θ(T2(n)), then the efficiency of the two algorithms areabout the same (when n is large).If T1(n) = o(T2(n)), then the efficiency of the algorithm A1 will bebetter than that of algorithm A2 (when n is large).By using the definitions, we can directly show whetherT1(n) = O(T2(n)), or T1(n) = Ω(T2(n)). However, it is not easy toprove the relationship of two functions in this way.


Outline


2 Limit Test

3 L’Hospital Rule

4 Stirling Formula


Limit TestLimit Test is a powerful method for comparing functions.

Limit TestLet T1(n) and T2(n) be two functions. Let c = limn→∞

T1(n)T2(n) .

1 If c is a constant > 0, then T1(n) = Θ(T2(n)).

2 If c = 0, then T1(n) = o(T2(n)).

3 If c =∞, then T1(n) = ω(T2(n)).

4 If c does not exists (or if we do not know how to compute c), the limit testfails.

Proof of (1): c = limn→∞T1(n)T2(n) means: ∀ε > 0, there exists n0 ≥ 0 such that for

any n ≥ n0:∣∣∣ T1(n)

T2(n) − c∣∣∣ ≤ ε; or equivalently: c− ε ≤ T1(n)

T2(n) ≤ c + ε. Let ε = c/2and let c1 = c− ε = c/2 and c2 = c + ε = 3c/2, we have

c1T2(n) ≤ T1(n) ≤ c2T2(n)

for all n ≥ n0. Thus T1(n) = Θ(T2(n)) by definition.


Limit TestLimit Test is a powerful method for comparing functions.

Limit TestLet T1(n) and T2(n) be two functions. Let c = limn→∞

T1(n)T2(n) .

1 If c is a constant > 0, then T1(n) = Θ(T2(n)).

2 If c = 0, then T1(n) = o(T2(n)).

3 If c =∞, then T1(n) = ω(T2(n)).

4 If c does not exists (or if we do not know how to compute c), the limit testfails.

Proof of (1): c = limn→∞T1(n)T2(n) means: ∀ε > 0, there exists n0 ≥ 0 such that for

any n ≥ n0:∣∣∣ T1(n)

T2(n) − c∣∣∣ ≤ ε; or equivalently: c− ε ≤ T1(n)

T2(n) ≤ c + ε. Let ε = c/2and let c1 = c− ε = c/2 and c2 = c + ε = 3c/2, we have

c1T2(n) ≤ T1(n) ≤ c2T2(n)

for all n ≥ n0. Thus T1(n) = Θ(T2(n)) by definition.c©Hu Ding (Michigan State University) CSE 331 Algorithm and Data Structures 4 / 18

Example

Example 1

T1(n) = 10n2 + 15n− 60, T2(n) = n2

limn→∞

T1(n)

T2(n)= lim

n→∞

10n2 + 15n− 60n2 = lim

n→∞(10+

15n−60

n2 ) = 10+0−0 = 10

Since 10 is a constant > 0, we have T1(n) = Θ(T2(n)) = Θ(n2) by thestatement 1 of Limit Test (as expected).


Log function

The log functions are very useful in algorithm analysis.

lg = log2 n

log n = log10 n

ln n = loge n

(ln n is the log function with the natural base e = 2.71828 . . .).


Log function

The log functions are very useful in algorithm analysis.

lg = log2 n

log n = log10 n

ln n = loge n

(ln n is the log function with the natural base e = 2.71828 . . .).


Log base change formula


For any 1 < a, b, logb n =loga nloga b = logb a · loga n.

Proof: Let k = logb n. By definition: n = bk.

Take loga on both sides: loga n = loga(bk) = k · loga bThis implies: logb n = k =

loga nloga b .

Let n = a in this formula and note 1 = loga a:

logb a =loga aloga b

=1

loga b

This proves the second part of the formula.





Proof: Let k = logb n. By definition: n = bk.Take loga on both sides: loga n = loga(bk) = k · loga b

This implies: logb n = k =loga nloga b .



=1

loga b






Proof: Let k = logb n. By definition: n = bk.Take loga on both sides: loga n = loga(bk) = k · loga bThis implies: logb n = k =

loga nloga b .



=1

loga b







loga nloga b .



=1

loga b







loga nloga b .



=1

loga b



Outline


2 Limit Test

3 L’Hospital Rule

4 Stirling Formula


L’Hospital Rule

L’Hospital RuleIf limn→∞ f (n) = 0 and limn→∞ g(n) = 0, then

limn→∞

f (n)

g(n)= lim

n→∞

f ′(n)

g′(n)

If limn→∞ f (n) =∞ and limn→∞ g(n) =∞, then

limn→∞

f (n)

g(n)= lim

n→∞

f ′(n)

g′(n)


L’Hospital Rule

L’Hospital RuleIf limn→∞ f (n) = 0 and limn→∞ g(n) = 0, then

limn→∞

f (n)

g(n)= lim

n→∞

f ′(n)

g′(n)

If limn→∞ f (n) =∞ and limn→∞ g(n) =∞, then

limn→∞

f (n)

g(n)= lim

n→∞

f ′(n)

g′(n)


Example

Example 2T1(n) = n2 + 6, T2(n) = n lg n. (Recall: lg n = log2 n.)

limn→∞T1(n)T2(n)

= limn→∞n2+6n lg n = limn→∞

n+ 6n

lg n

= limn→∞1− 6

n21

ln 2·n(by L’Hospital Rule)

= ln 2 limn→∞(n− 6n) = ln 2(∞− 0) =∞

By Limit Test, we have n2 + 6 = ω(n lg n).


Example

Example 3T1(n) = (ln n)k, T2(n) = nε, where k > 0 is any (large) constant andε > 0 is any (small) constant. (Recall: ln n = loge n.)


= limn→∞(ln n)k

nε (use L’Hospital Rule)

= limn→∞k(ln n)k−1×(1/n)

εn(ε−1)

= kε limn→∞

(ln n)k−1

nε (use L’Hospital Rule again and simplify)= k(k−1)

ε2 limn→∞(ln n)k−2

nε (use L’Hospital Rule k times).....

= k(k−1)···2·1εk limn→∞

1nε = 0

So by Limit Test, (ln n)k = o(nε) for any k and ε. For example, takek = 100 and ε = 0.01, we have (ln n)100 = o(n0.01).


Example

Example 3T1(n) = (ln n)k, T2(n) = nε, where k > 0 is any (large) constant andε > 0 is any (small) constant. (Recall: ln n = loge n.)


= limn→∞(ln n)k

nε (use L’Hospital Rule)

= limn→∞k(ln n)k−1×(1/n)

εn(ε−1)

= kε limn→∞

(ln n)k−1

nε (use L’Hospital Rule again and simplify)= k(k−1)

ε2 limn→∞(ln n)k−2

nε (use L’Hospital Rule k times).....

= k(k−1)···2·1εk limn→∞

1nε = 0

So by Limit Test, (ln n)k = o(nε) for any k and ε. For example, takek = 100 and ε = 0.01, we have (ln n)100 = o(n0.01).


Example

Example 4T1(n) = nk, T2(n) = an, where k > 0 is any (large) constant and a > 1 isany constant bigger than 1.


= limn→∞nk

an (using L’Hospital Rule)

= limn→∞k·nk−1

ln a·an = kln a limn→∞

nk−1

an ( using L’Hospital Rule k times)= k(k−1)···2·1

(ln a)k limn→∞n0

an

= k(k−1)···2·1(ln a)k limn→∞

1an = 0

So by Limit Test, nk = o(an) for any k > 0 and a > 1. For example, takek = 1000 and a = 1.001, we have n1000 = o((1.001)n).


Example

Example 4T1(n) = nk, T2(n) = an, where k > 0 is any (large) constant and a > 1 isany constant bigger than 1.


= limn→∞nk

an (using L’Hospital Rule)

= limn→∞k·nk−1

ln a·an = kln a limn→∞

nk−1

an ( using L’Hospital Rule k times)= k(k−1)···2·1

(ln a)k limn→∞n0

an

= k(k−1)···2·1(ln a)k limn→∞

1an = 0

So by Limit Test, nk = o(an) for any k > 0 and a > 1. For example, takek = 1000 and a = 1.001, we have n1000 = o((1.001)n).


Example

Example 5T1(n) = loga n, T2(n) = logb n, where a > 1 and b > 1 are any twoconstants bigger than 1.

By the Log Base Change Formula: logb n = logb a · loga n

Thus: limn→∞T1(n)T2(n)

= limn→∞loga nlogb n = limn→∞

loga nlogb a·loga n = 1

logb a

Since 1logb a > 0 is a constant, we have loga n = Θ(logb n) by Limit Test.

So: the growth rates of the log functions are the same for anybase > 1.


Example


By the Log Base Change Formula: logb n = logb a · loga nThus: limn→∞

T1(n)T2(n)



logb a




Example



T1(n)T2(n)



logb a




Example



T1(n)T2(n)



logb a




Example

Example 6T1(n) = an, T2(n) = bn, where 1 < a < b are any two constants.

We have: limn→∞T1(n)T2(n)

= limn→∞an

bn = limn→∞(ab)n = 0.

Thus: an = o(bn) (for any 1 < a < b) by Limit Test.

The list of common functions:The following list shows the functions commonly used in algorithmanalysis, in the order of increasing growth rate (a, b, c, d, k, ε arepositive constants, ε < 1, k > 1, d > 1 and a < b):

c, logd n, (logd n)k, nε, n, nk, an, bn, n!, nn

in the sense that if f (n) and g(n) are any two consecutive functions inthe list, we have f (n) = o(g(n))


Example



= limn→∞an







Example



= limn→∞an







Example



= limn→∞an







Examples

Example 7T1(n) = n! and T2(n) = an (a > 1)

limn→∞an

n! =?

L’Hospital Rule doesn’t help: We don’t know how to take derivativeof n!

an

n!=

a1· a

2· · · a

2dae︸︷︷︸2dae terms

· a2dae+ 1

· · · an︸︷︷︸

(n−2dae) termsThe first part is a constant c > 0. In the second part, each term < 1/2.So:

0 ≤ limn→∞

an

n!≤ c · lim

n→∞(12

)(n−2dae) = 0

By Limit Test: an = o(n!).


Examples


limn→∞an

n! =?


an

n!=

a1· a

2· · · a


· a2dae+ 1

· · · an︸︷︷︸


0 ≤ limn→∞

an

n!≤ c · lim

n→∞(12

)(n−2dae) = 0



Examples


limn→∞an

n! =?


an

n!=

a1· a

2· · · a


· a2dae+ 1

· · · an︸︷︷︸


0 ≤ limn→∞

an

n!≤ c · lim

n→∞(12

)(n−2dae) = 0



Examples


limn→∞an

n! =?


an

n!=

a1· a

2· · · a


· a2dae+ 1

· · · an︸︷︷︸

(n−2dae) terms

The first part is a constant c > 0. In the second part, each term < 1/2.So:

0 ≤ limn→∞

an

n!≤ c · lim

n→∞(12

)(n−2dae) = 0



Examples


limn→∞an

n! =?


an

n!=

a1· a

2· · · a


· a2dae+ 1

· · · an︸︷︷︸


0 ≤ limn→∞

an

n!≤ c · lim

n→∞(12

)(n−2dae) = 0



Examples


limn→∞an

n! =?


an

n!=

a1· a

2· · · a


· a2dae+ 1

· · · an︸︷︷︸


0 ≤ limn→∞

an

n!≤ c · lim

n→∞(12

)(n−2dae) = 0



Examples


limn→∞an

n! =?


an

n!=

a1· a

2· · · a


· a2dae+ 1

· · · an︸︷︷︸


0 ≤ limn→∞

an

n!≤ c · lim

n→∞(12

)(n−2dae) = 0

By Limit Test: an = o(n!).c©Hu Ding (Michigan State University) CSE 331 Algorithm and Data Structures 15 / 18

Outline


2 Limit Test

3 L’Hospital Rule

4 Stirling Formula


Stirling Formula

Stirling Formula√

2πn(n

e

)ne

112n+1 ≤ n! ≤

√2πn

(ne

)ne

112n

or: n! =√

2πn(n

e

)n·(

1 + Θ

(1n

))When n = 10;

n! = 3628800.√

2πn( n

e

)n= 3598696, 99% accurate.


Examples

Example 7 (another solution)T1(n) = n! and T2(n) = an (a > 1)

limn→∞

n!

an ≥ limn→∞

√2πn

( nae

)n= lim

n→∞

√2πn · lim

n→∞

( nae

)n

The first limit is∞. The second limit goes to∞∞. So it’s also∞. Thuslimn→∞

n!an =∞ and n! = ω(an) by limit test.

Example 8T1(n) = nn and T2(n) = n!

By using similar method as in Example 7, we can show: n! = o(nn)


Examples


limn→∞

n!

an ≥ limn→∞

√2πn

( nae

)n= lim

n→∞

√2πn · lim

n→∞

( nae

)n






Examples


limn→∞

n!

an ≥ limn→∞

√2πn

( nae

)n= lim

n→∞

√2πn · lim

n→∞

( nae

)n






Documents

Compare the growth rate of functionshuding/331material/notes2.pdfOutline 1 Compare the growth rate of functions 2 Limit Test 3 L’Hospital Rule 4 Stirling Formula c Hu Ding (Michigan