Growth of functions: asymptotic notation To characterize the

Growth of functions: asymptotic notation

To characterize the time cost of algorithms, we focus on functions that mapinput size to (typically, worst-case) running time. (Similarly for space costs.)

We are interested in precise notation for characterizing running-timedifferences that are likely to be significant across different platforms anddifferent implementations of the algorithms.

This naturally leads to an interest in the “asymptotic growth” of functions.We focus on how the function behaves as its input grows large.

Asymptotic notation is a standard means for describing families of functionsthat share similar asymptotic behavior.

Asymptotic notation allows us to ignore small input sizes, constantfactors, lower-order terms in polynomials, and so forth.

Big O notation (asymptotic upper bound)Definition f (n) ∈ O(g(n)) if there are positive c, n0 s.t.

|f (n)| ≤ c|g(n)| for all n ≥ n0 .

We assume that these functions map reals to reals.They may be partial functions, but, as we’ll explain shortly, we will assume thatthey are “asymptotically-defined on N”.

It is common to write f (n) = O(g(n)) instead of f (n) ∈ O(g(n)).

Beware: this is not the same as writing O(g(n)) = f (n).

What about the awkwardness of writing f (n) instead of f , and g(n) instead ofg? (That is, writing an expression for the value of function f at n, when wewish to name the function f itself.)

It corresponds to the standard (and convenient) practice of writing, say, O(n2)to stand for the set of functions f : R→ R s.t. for some positive c, n0,

|f (n)| ≤ c|n2| for all n ≥ n0 .

Is g(n) = O(g(n)) ?

Asymptotically defined on N

We say we’re interested in functions from R to R, but . . .

. . . what about a function like log2 n ?

Problem: not defined for n ≤ 0.

An approach that couldn’t talk about log-space and log-time would not be veryuseful.

And what about a function like n! ? (Commonly defined with domain N .)

It is convenient to consider any (partial) function from R to R that is“asymptotically defined on N” — that is, defined for all natural numbersn ≥ n0, for some constant n0.

So let’s assume that in asymptotic notation we consider (partial) functionsfrom R to R that are asymptotically defined on N .

Ignoring constantsAsymptotic notation is supposed to allow us to ignore constants. Let’s look athow this works. . .

Is n + k = O(n)?

f (n) ∈ O(g(n)) if there are positive c, n0 s.t.

|f (n)| ≤ c|g(n)| for all n ≥ n0 .

If k = 0, the result is trivial. (Take c = 1, n0 = 1.)

If k is positive:We can take c = 2 and n0 = k. Notice that

|n + k| ≤ 2|n| for all n ≥ k .

What if k is negative?

Then we can take c = 1 and n0 = −k, and notice that

|n + k| ≤ |n| for all n ≥ −k .

So, for all k ∈ R,n + k = O(n) .

If f (n) = O(g(n)) , is kf (n) = O(g(n)) ?

f (n) ∈ O(g(n)) if there are positive c, n0 s.t.

|f (n)| ≤ c|g(n)| for all n ≥ n0 .

If k = 0, the result is immediate, so assume otherwise.

Assume f (n) = O(g(n)).

So there are positive constants c ′, n′0 s.t.

|f (n)| ≤ c ′|g(n)| for all n ≥ n′0 .

Take c = c ′|k| and n0 = n′0.

Observe that, for all n ≥ n0 ,

|kf (n)| = |k| · |f (n)|≤ |k| · c ′|g(n)|= c|g(n)|

Sokf (n) = O(g(n))

whenever f (n) = O(g(n)).

If f (n) = O(g(n)) , is f (n) + k = O(g(n)) ?

f (n) ∈ O(g(n)) if there are positive c, n0 s.t.

|f (n)| ≤ c|g(n)| for all n ≥ n0 .

Assume that f (n) = O(g(n)).

We need positive c and n0 s.t.

|f (n) + k| ≤ c|g(n)| for all n ≥ n0 .

In general, such c and n0 may fail to exist. But here’s a simple condition thatwill guarantee their existence. . .

Assume that |g(n)| has an asymptotic nonzero lower bound. That is, for somepositive b and m0,

|g(n)| ≥ b for all n ≥ m0 .

Since f (n) = O(g(n)), there are c ′, n′0 s.t.

f (n) ≤ c ′g(n) for all n ≥ n′0 .

It suffices to take c = |k/b|+ c ′ and n0 = m0 + n′0. Let’s check that. . .

Given: For some positive b and m0,

|g(n)| ≥ b for all n ≥ m0 ,

and, for some positive c ′, n′0,

|f (n)| ≤ c ′|g(n)| for all n ≥ n′0 .

NTS: There are positive c and n0 s.t.

|f (n) + k| ≤ c|g(n)| for all n ≥ n0 .

Take c = |k/b|+ c ′ and n0 = m0 + n′0.

Notice that, for all n ≥ n0,

|f (n) + k| ≤ |f (n)|+ |k|≤ c ′|g(n)|+ |k|≤ c ′|g(n)|+ |k| · |g(n)|

b

= c ′|g(n)|+ |k/b| · |g(n)|= (c ′ + |k/b|) · |g(n)|= c · |g(n)|

So, if |g(n)| has a nonzero asymptotic lower bound, then

f (n) + k = O(g(n))

whenever f (n) = O(g(n)).

Polynomials and big OOur results thus far can explain, for instance, the well-known fact that

ank = O(nk)

and, similarly, that

nk + b = O(nk) (k ≥ 0)

But what about drawing the conclusion that

ank + b = O(nk) (k ≥ 0)

It looks like this should be true, but we’d need another argument.

There is a much more general fact about polynomials and Big O that youshould know:

Any polynomial expression whose highest order term is

ank

belongs toO(nk) .

Let’s prove it. . .

Polynomials of degree at most k belong to O(nk)Consider an arbitrary expression of the form

aknk + ak−1n

k−1 + · · ·+ a2n2 + a1n

1 + a0n0

with k ∈ N and ak , . . . , a0 ∈ R. (Notice that any of ak , . . . , a0 may equal zero.If all do, the result is trivial, so we’ll assume that at least one does not.)

For convenience, we’ll write this expression in the form

kXi=0

aini .

Notice that, for all n ≥ 1,˛Pki=0 ain

i˛≤

Pki=0 |ain

i |=

Pki=0 |ai |ni

≤“Pk

i=0 |ai |”· nk

=“Pk

i=0 |ai |”· |nk |

which shows thatkX

i=0

aini = O(nk) .

(Take n0 = 1 and c =Pk

i=0 |ai |.)

If f (n) = O(g(n)) and g(n) = O(h(n)), then f (n) = O(h(n))

Assume f (n) = O(g(n)) and g(n) = O(h(n)).

It follows that there are positive c ′, n′0, c′′, n′′0 s.t.

|f (n)| ≤ c ′|g(n)| for all n ≥ n′0

and|g(n)| ≤ c ′′|h(n)| for all n ≥ n′′0 .

Take c = c ′c ′′ and n0 = n′0 + n′′0 . Notice that

|f (n)| ≤ c ′|g(n)| ≤ c ′c ′′|h(n)| = c|h(n)| for all n ≥ n0 .

If f (n) = O(h(n)) and g(n) = O(h(n)), then f (n) + g(n) = O(h(n))

Assume f (n) = O(h(n)) and g(n) = O(h(n)).

It follows that there are positive c ′, n′0, c′′, n′′0 s.t.

|f (n)| ≤ c ′|h(n)| for all n ≥ n′0

and|g(n)| ≤ c ′′|h(n)| for all n ≥ n′′0 .

Take c = c ′ + c ′′ and n0 = n′0 + n′′0 . Then

|f (n) + g(n)| ≤ |f (n)|+ |g(n)| ≤ c|h(n)| for all n ≥ n0 .

That is, f (n) + g(n) = O(h(n)).

It follows that if fi (n) = O(g(n)) for all i ∈ N , then

kXi=0

fi (n) = O(g(n))

for any k ∈ N .

If f (n) + g(n) = O(h(n)) and f (n) and g(n) are asymptotically nonnegative,

then f (n) = O(h(n))

Assume that f (n) + g(n) = O(h(n)) and that both f (n) and g(n) areasymptotically nonnegative.

It follows that there are positive c ′, n′0, m0, m′0 s.t.

|f (n) + g(n)| ≤ c ′|h(n)| for all n ≥ n′0

and 0 ≤ f (n) for all n ≥ m0

and 0 ≤ g(n) for all n ≥ m′0

Take c = c ′ and n0 = n′0 + m0 + m′0. Then

|f (n)| ≤ c|h(n)| for all n ≥ n0 .

What if one of f (n),g(n) is not asymptotically nonnegative?

(Consider, for example, f (n) = n, g(n) = −n, h(n) = 0.)

If f (n) · g(n) = O(h(n)) and g(n) has a positive asymptotic lower bound,

then f (n) = O(h(n))Assume that f (n) · g(n) = O(h(n)) and that g(n) has a positive asymptoticlower bound. So there are positive c ′, n′0, b, m0 s.t.

|f (n) · g(n)| ≤ c ′|h(n)| for all n ≥ n′0

andb ≤ g(n) for all n ≥ m0 .

Take c = c ′/b and n0 = n′0 + m0. Notice that

b|f (n)| = |bf (n)| ≤ |f (n) · g(n)| ≤ c ′|h(n)| for all n ≥ n0

Therefore,

|f (n)| ≤ c ′

b|h(n)| = c|h(n)| for all n ≥ n0

Is it true that if f (n) · g(n) = O(h(n)), then at least one of f (n), g(n) belongsto O(h(n))? Suppose

f (n) =

n , if n is even0 , otherwise

g(n) =

n , if n is odd0 , otherwise

h(n) = 0

If f (n) = O(g(n)) and f ′(n) = O(g ′(n)), then f (n) · f ′(n) = O(g(n) · g ′(n))

Assume f (n) = O(g(n)) and f ′(n) = O(g ′(n)).

So there are positive c ′, n′0, c′′, n′′0 s.t.

|f (n)| ≤ c ′|g(n)| for all n ≥ n′0

and|f ′(n)| ≤ c ′′|g ′(n)| for all n ≥ n′′0 .

Take c = c ′c ′′ and n0 = n′0 + n′′0 . Then, for all n ≥ n0,

|f (n) · f ′(n)| = |f (n)| · |f ′(n)|≤ c ′|g(n)| · c ′′|g ′(n)|= (c ′ · c ′′) · |g(n)| · |g ′(n)|= c(|g(n)| · |g ′(n)|)= c|g(n) · g ′(n)|

It follows that if fi (n) = O(gi (n)) for all i ∈ N , then

kYi=0

fi (n) = O

kY

i=0

gi (n)

!

for any k ∈ N .

nj 6= O(nk) if j > k

Assume j > k. Suppose there are positive c, n0 s.t.

|nj | ≤ c|nk | for all n ≥ n0 .

Thennj−k ≤ c for all n ≥ n0 .

But this is not possible, since j − k is positive. Contradiction. We concludethat the supposition was false. Hence nj 6= O(nk) when j > k.

Asymptotic lower bound, asymptotically tight boundDefinition (asymptotic lower bound)f (n) ∈ Ω(g(n)) if there are positive c, n0 s.t.

c|g(n)| ≤ |f (n)| for all n ≥ n0 .

Claim: f (n) = Ω(g(n)) iff g(n) = O(f (n)). Easy to show. (Try it.)

Definition (asymptotically tight bound)f (n) ∈ Θ(g(n)) if there are positive c, d , n0 s.t.

c|g(n)| ≤ |f (n)| ≤ d |g(n)| for all n ≥ n0 .

Theorem f (n) = Θ(g(n)) iff f (n) = Ω(g(n)) and f (n) = O(g(n)).

f (n)?= Ω(f (n))

f (n)?= Θ(f (n))

You might want to consider whether other properties of big O hold for Ω, Θ.

Other asymptotic notationsDefinition (asymptotically strict upper bound)f (n) ∈ o(g(n)) if for every positive c there is a positive n0 s.t.

|f (n)| < c|g(n)| for all n ≥ n0 .

Notice: If f (n) = o(g(n)) then f (n) = O(g(n)).

Claim: If f (n) = o(g(n)) then f (n) 6= Ω(g(n)).

Definition (asymptotically strict lower bound)f (n) ∈ ω(g(n)) if for every positive c there is a positive n0 s.t.

c|g(n)| < |f (n)| for all n ≥ n0 .

Notice: If f (n) = ω(g(n)) then f (n) = Ω(g(n)).

Claim: If f (n) = ω(g(n)) then f (n) 6= O(g(n)).

Claim: f (n) = o(g(n)) iff g(n) = ω(f (n)).

A few more laws

If f (n) = Θ(g(n)) and g(n) =

8>>>><>>>>:o(h(n))O(h(n))Θ(h(n))Ω(h(n))ω(h(n))

9>>>>=>>>>; then f (n) =

8>>>><>>>>:o(h(n))O(h(n))Θ(h(n))Ω(h(n))ω(h(n))

9>>>>=>>>>; .

If f (n) =

o(g(n))ω(g(n))

ffand g(n) =

O(h(n))Ω(h(n))

ffthen f (n) =

o(h(n))ω(h(n))

ff.

If f (n) =

O(g(n))Ω(g(n))

ffand g(n) =

o(h(n))ω(h(n))

ffthen f (n) =

o(h(n))ω(h(n))

ff.

Some facts about hierarchy of asymptotic classes

nn beats factorial:n! = o(nn)

Factorial beats exponential:kn = o(n!)

Exponential beats polynomial:

nk = o(bn) if b > 1

Higher-degree polynomial beats lower-degree polynomial:

nj = o(nk) if 0 ≤ j < k

Polynomial beats (poly)logarithmic:

logab n = o(nk) if b > 1 and k > 0

Note: logab n is an abbreviation for (logb n)a.

The base of a (poly)logarithm doesn’t matter:

logka n = Θ(logk

b n) if a, b > 1

The base of an exponential matters:

an = o(bn) if 1 ≤ a < b

Adding a constant to the exponent (of an exponential) doesn’t matter:

an = Θ(an+k)

A constant factor in the exponent matters:

an = o(abn) if a, b > 1

A hierarchy of asymptotically strict bounds22n

nn

n!

3n

n · 2n

2n

n3

n2 log n

n2

n log n

n√n

log3 n

log2 n

log nplog n

log log n

log log log n

1

1/n

0