Elements of Algorithm Analysis

Algorithms

Elements of Algorithm Analysis

Partha Pratim Das

Department of Computer Science and EngineeringIndian Institute of Technology, Kharagpur

[email protected]

T10KT Main Workshop

May 25, 2015

Partha Pratim Das (IIT, Kharagpur) Elements of Algorithm Analysis May 25, 2015 1 / 44

Getting Started

Why?

Set the motivation for algorithm analysis:Why analyse?

What?

Identify what all need to be analysed:What to analyse?

How?

Learn the techniques for analysis:How to analyse?

Where?

Understand the scenarios for application:Where to analyse?

When?

Realize your position for seeking the analysis:When to analyse?


Why analyse?

Practical reasons:

Resources are scarce

Greed to do more with less

Avoid performance bugs

Core Issues:

Predict performanceHow much time does binary search take?

Compare algorithmsHow quick is Quicksort?

Provide guaranteesSize notwithstanding, Red-Black tree inserts in O(log n)

Understand theoretical basisSorting by comparison cannot do better than (n log n)


What to analyse?

Core Issue: Cannot control what we cannot measure

TimeStory starts here with Analytical EngineMost common analysis factorRepresentative of various related analysis factors like Power,Bandwidth, ProcessorsSupported by Complexity Classes

SpaceWidely exploredImportant for hand-held devicesSupported by Complexity Classes


What to analyse?

Examples:

Sum of Natural Numbers

Minimum of a Sequence of Numbers


What to analyse?


int sum(int n) {

int s = 0;

for(; n > 0; --n)

s = s + n;

return s;

}

Time T (n) = n (additions)

Space S(n) = 2 (n, s)


What to analyse?


int min(int a[], int n) {

for(int i = 0; i < n; ++i)

cin >> a[i];

int t = a[--n];

for(; n > 0; --n)

if (t < a[--n])

t = a[n];

return t;

}

Time T (n) = n 1 (comparison of value)Space S(n) = n + 3 (a[]s, n, i, t)


What to analyse?



for(int i = 0; i < n; ++i)

cin >> a[i];

int t = a[--n];

for(; n > 0; --n)

if (t < a[--n])

t = a[n];

return t;

}

Time T (n) = n 1 (comparison of value)

Space S(n) = n + 3 (a[]s, n, i, t)


What to analyse?



for(int i = 0; i < n; ++i)

cin >> a[i];

int t = a[--n];

for(; n > 0; --n)

if (t < a[--n])

t = a[n];

return t;

}

Time T (n) = n 1 (comparison of value)Space S(n) = n + 3 (a[]s, n, i, t)


What to analyse?


int min(int n) {

int x;

cin >> x;

int t = x;

for(; n > 1; --n) {

cin >> x;

if (t < x)

t = x;

}

return t;

}

Time T (n) = n 1 (comparison of value)Space S(n) = 3 (n, x, t)


What to analyse?


int min(int n) {

int x;

cin >> x;

int t = x;

for(; n > 1; --n) {

cin >> x;

if (t < x)

t = x;

}

return t;

}

Time T (n) = n 1 (comparison of value)

Space S(n) = 3 (n, x, t)


What to analyse?


int min(int n) {

int x;

cin >> x;

int t = x;

for(; n > 1; --n) {

cin >> x;

if (t < x)

t = x;

}

return t;

}

Time T (n) = n 1 (comparison of value)Space S(n) = 3 (n, x, t)


How to analyse?

Mathematical / Counting Models

Asymptotic Analysis

Master Theorem


How to analyse? Counting Models

Mathematical / Counting ModelsCore Idea: Total running time = Sum of cost frequency for alloperations

Need to analyse program to determine set of operations

Cost depends on machine, compiler

Frequency depends on algorithm, input data

Machine Model: Random Access Machine (RAM) Computing Model

Input data & size

Operations

Intermediate Stages

Output data & size



Query:Can we assume that two large numbers of arbitrary size can be added upin constant time? Why?



Examples:

Factorial of a Number



Factorial (Recursive)

int fact(int n) {

if (0 != n) return n*fact(n-1);

return 1;

}

Solution by Counting:

Time T (n) = n 1 (multiplication)Space S(n) = n + 1 (ns in recursive calls)




int fact(int n) {


return 1;

}


Time T (n) = n 1 (multiplication)

Space S(n) = n + 1 (ns in recursive calls)




int fact(int n) {


return 1;

}


Time T (n) = n 1 (multiplication)Space S(n) = n + 1 (ns in recursive calls)



Factorial (Iterative)

int fact(int n) {

int t = 1;

for(; n > 0; --n)

t = t * n;

return t;

}

Time T (n) = n (multiplication)

Space S(n) = 2 (n, t)



Mathematical / Counting ModelsFrequency Analysis Tool: Recurrence of count of operations

Define recurrence

Identify base condition

Solve recurrence in closed form



Examples:


Factorial of a Number

nth Fibonacci Number



Sum of n Natural Numbers

int sum(int n) {

int s = 0;

for(; n > 0; --n)

s = s + n;

return s;

}

Time T (n) (additions)

T (n) = T (n 1) + 1, n > 1= 1, n = 1

Solve Recurrence in Long hand:T (n) = T (n 1) + 1

= T (n 2) + (1 + 1)= = T (1) + (n 1)= 1 + (n 1)= n

Space S(n) = 2




int fact(int n) {


return 1;

}

Time T (n) (multiplication)T (n) = T (n 1) + 1, n > 0

= 0, n = 0

T (n) = n

Space S(n) (ns in recursive calls)S(n) = S(n 1) + 1, n > 0

= 1, n = 0

S(n) = n + 1



Factorial (Iterative)

int fact(int n) {

int t = 1;

for(; n > 0; --n)

t = t * n;

return t;

}

Time T (n) (multiplication)

T (n) = T (n 1) + 1, n > 0= 0, n = 0

T (n) = n

Space S(n) (n, t)

S(n) = S(n 1), n > 0= 2, n = 0

S(n) = 2



Fibonacci (Recursive)

int fibo(int n) {

if (n > 1) return fibo(n-1)+fibo(n-2);

return n;

}

Time T (n) (additions)T (n) = T (n 1) + T (n 2) + 1, n > 1

= 0, n = 0, 1Solution by Generating Function:

T (n) = n (approx) where = 1+5

2

Space S(n) (ns in recursive calls)S(n) = S(n 1) + 1, n > 1

= 1, n = 0, 1

S(n) = n + 1



Fibonacci (Iterative)

int fibo(int n) {

int FIBO[n+1];

if (n > 1) {

FIBO[0] = 0; FIBO[1] = 1;

for(int i = 2; i 1

= 0, n = 0, 1

T (n) = n 1Space S(n) = n + 3 (FIBO[]s, n, i)




int fibo(int n) {

int FIBO[n+1];

if (n > 1) {

FIBO[0] = 0; FIBO[1] = 1;

for(int i = 2; i 1

= 0, n = 0, 1

T (n) = n 1

Space S(n) = n + 3 (FIBO[]s, n, i)




int fibo(int n) {

int FIBO[n+1];

if (n > 1) {

FIBO[0] = 0; FIBO[1] = 1;

for(int i = 2; i 1

= 0, n = 0, 1

T (n) = n 1Space S(n) = n + 3 (FIBO[]s, n, i)



Fibonacci (Iterative)int fibo(int n) {

int x = 0, y = 1, t = n;

if (n > 1) {

for(int i = 2; i 1

= 0, n = 0, 1

T (n) = n 1Space S(n) = 4 (n, x, y, t)




int x = 0, y = 1, t = n;

if (n > 1) {

for(int i = 2; i 1

= 0, n = 0, 1

T (n) = n 1

Space S(n) = 4 (n, x, y, t)




int x = 0, y = 1, t = n;

if (n > 1) {

for(int i = 2; i 1

= 0, n = 0, 1

T (n) = n 1Space S(n) = 4 (n, x, y, t)


How to analyse? Asymptotic Analysis

Asymptotic AnalysisCore Idea: Cannot compare actual times; hence compare Growth or howtime increases with input size

Function Approximation (tilde () notation)

Common Growth Functions

Big-Oh (O(.)), Big-Omega ((.)), and Big-Theta ((.)) Notations

Solve recurrence with Growth Functions



Function Approximation (tilde () notation)Consider:

int count = 0;

for (int i = 0; i < N; i++)

for (int j = i+1; j < N; j++)

if (a[i] + a[j] == 0)

count++;

Operation Frequencyvariable declaration N + 2

assignment statement N + 2

less than compare 12(N + 1)(N + 2)

equal to compare 12N(N 1)array access N(N 1)increment 12N(N 1) to N(N 1)



Estimate running time (or memory) as a function of input size N.Ignore lower order terms.

when N is large, terms are negligiblewhen N is small, we dont care

Operation Frequency Approximationvariable declaration N + 2 Nassignment statement N + 2 Nless than compare 12(N + 1)(N + 2) 12N2equal to compare 12N(N 1) 12N2array access N(N 1) N2increment 12N(N 1) to N(N 1) 12N2 to N2

f (n) g(n) meanslim

Nf (n)

g(n)= 1


Definition. If f (N) ~ c g(N) for some constant c > 0, then the order of growthof f (N) is g(N).Ignores leading coefficient.Ignores lower-order terms.

Ex. The order of growth of the running time of this code is N 3.

Typical usage. With running times.

Common order-of-growth classifications

42

int count = 0;for (int i = 0; i < N; i++) for (int j = i+1; j < N; j++) for (int k = j+1; k < N; k++) if (a[i] + a[j] + a[k] == 0) count++;

where leading coefficientdepends on machine, compiler, JVM, ...

Courtesy: Algorithms by Robert Sedgewick & Kevin Wayne

Good news. The set of functions

1, log N, N, N log N, N 2, N 3, and 2Nsuffices to describe the order of growth of most common algorithms.


43

1K

T

2T

4T

8T

64T

512T

logarithmic

expo

nential

constant

linearithm

ic

linear

quadratic

cubic

2K 4K 8K 512K

100T

200T

500T

logarithmic

exponential

constant

size

size

linearithm

ic

linear

100K 200K 500K

time

time

Typical orders of growth

log-log plot

standard plot

cubicquadratic



44

order of growth

name typical code framework description example T(2N) / T(N)

1 constant a = b + c; statement add two numbers 1

log N logarithmic while (N > 1){ N = N / 2; ... } divide in half binary search ~ 1

N linear for (int i = 0; i < N; i++){ ... } loopfind the

maximum 2

N log N linearithmic [see mergesort lecture] divideand conquer mergesort ~ 2

N 2 quadraticfor (int i = 0; i < N; i++)

for (int j = 0; j < N; j++) { ... }

double loopcheck all

pairs4

N 3 cubicfor (int i = 0; i < N; i++)

for (int j = 0; j < N; j++) for (int k = 0; k < N; k++)

{ ... }

triple loopcheck all triples 8

2N exponential [see combinatorial search lecture] exhaustivesearchcheck all subsets T(N)


53

Commonly-used notations in the theory of algorithms

notation provides example shorthand for used to

Big Thetaasymptotic

order of growth (N2)

N 210 N 2

5 N 2 + 22 N log N + 3N

classifyalgorithms

Big Oh (N2) and smaller O(N2)

10 N 2100 N

22 N log N + 3 N

developupper bounds

Big Omega (N2) and larger (N2)

N 2N 5

N 3 + 22 N log N + 3 N

developlower bounds



Formal Definitions of Asymptotic Notationf (n) ConditionO(g(n)) : c > 0 n0 > 0 n > n0 f (n) c.g(n)(g(n)) : c > 0 n0 > 0 n > n0 f (n) c.g(n)(g(n)) : c1 > 0 c2 > 0 n0 > 0 n > n0 c1.g(n) f (n) c2.g(n)o(g(n)) : c > 0 n0 > 0 n > n0 |f (n)| c. |g(n)|(g(n)) : c > 0 n0 > 0 n > n0 |f (n)| c. |g(n)|


How to analyse? Master Theorem

Example:Find the kth smallest element in A where A.length = n, 1 k n.Algorithm: SELECT(A[1, n], k)

1 Split A into n/5 groups of 5 elements each.2 Let bi be the median of the i

th group.3 Let B = [b1, b2, , bn/5].4 medianB = SELECT(B, B.length/2).5 Rearrange A so that all elements smaller than medianB come before

medianB, all elements larger than medianB come after medianB, andelements equal to medianB are next to medianB.

6 j = position of medianB in rearranged A (if more medianBs, thentake the closest position to n/2).

7 If k < j return SELECT(A[1, j 1], k).8 If k = j return medianB.9 If k > j return SELECT(A[j + 1, n], k j).



Find the kth smallest element in A where A.length = n, 1 k n.Algorithm: SELECT(A[1, n], k) T (n)

1 Split A into n/5 groups of 5 elements each. O(n)2 Let bi be the median of the i

th group. O(1) per group O(n) total3 Let B = [b1, b2, , bn/5]. O(n)4 medianB = SELECT(B, B.length/2). T (n/5)5 Rearrange A so that all elements smaller than medianB come before

medianB, all elements larger than medianB come after medianB, andelements equal to medianB are next to medianB. O(n)

6 j = position of medianB in rearranged A (if more medianBs, thentake the closest position to n/2). O(n)

7 If k < j return SELECT(A[1, j 1], k). T (j)8 If k = j return medianB. O(1)9 If k > j return SELECT(A[j + 1, n], k j). T (n j)

Solve: T (n) = T (n5 ) + T (3n4 ) + cn



Solving Recurrence by Hypothesize and Test

T (n) T (n5 ) + T (3n4 ) + cn, if n > 5 c , if n 5

Hypothesis: There exists a constant d such that T (n) < dn.Base Case: n 5: T (n) c , if n 5. Since we want to show thatT (n) < dn for some d , we need d c .Inductive Case: n > 5:T (n5 ) d n5 By HypothesisT (3n4 ) d 3n4 By HypothesisT (n) T (n5 ) + T (3n4 ) + cn, if n > 5

d n5 + d 3n4 + cn (1920d + c)n dn We need this

(1920d + c) dd 20c

T (n) O(n)Partha Pratim Das (IIT, Kharagpur) Elements of Algorithm Analysis May 25, 2015 33 / 44


Master TheoremCore Idea: Asymptotic recurrence forms tend to recur across problems

Use a parametrized formulation

Solve for the general form for a set of cases

Given a situation

extract parametersidentify the caseadopt solution



Master TheoremLet T (n) be a monotonically increasing function that satisfies

T (n) = aT (n/b) + f (n)

T (1) = c

where a 1, b 2, c > 0. If f (n) (nd) where d 0, then

(nd) if a < bd

T (n) (nd log n) if a = bd(nlogb a) if a > bd

CorollaryIf f (n) (nlogb a logk n) then

T (n) (nlogb a logk+1 n)



Master Theorem cannot be used if

T (n) is not monotone, ex: T (n) = sin n

f (n) is not a polynomial, ex: T (n) = 2T (n/2) + 2n

b cannot be expressed as a constant, ex: T (n) = T (n)



ExamplesT (n) = a b d Condition Case T (n) =

T ( n2

) + 12n2 + n 1 2 2 1 < 22 1 (nd) = (n2)

2T ( n4

) +n + 42 2 4 1

22 = 4

12 2 (nd log n) = (

n log n)

3T ( n2

) + 34n + 1 3 2 1 3 > 21 3 (nlogb a) = (nlog2 3)


Where to analyse?

Algorithmic SituationCore Idea: Identify data configurations or scenarios for analysis

Best Case

Worst Case

Average Case

Probabilistic Case

Amortized Case


8Types of analyses

Worst case. Running time guarantee for any input of size n.Ex. Heapsort requires at most 2 n log2 n compares to sort n elements.

Probabilistic. Expected running time of a randomized algorithm.

Ex. The expected number of compares to quicksort n elements is ~ 2n ln n.

Amortized. Worst-case running time for any sequence of n operations.Ex. Starting from an empty stack, any sequence of n push and pop operations takes O(n) operations using a resizing array.

Average-case. Expected running time for a random input of size n. Ex. The expected number of character compares performed by 3-way

radix quicksort on n uniformly random strings is ~ 2n ln n.

Also. Smoothed analysis, competitive analysis, ...

Courtesy: Algorithm Design by Jon Kleinberg & Eva Tardos

When to analyse?

Determination of Quality of an AlgorithmCore Idea: Are we doing well, and can we do better?

Let A be an algorithm for solving a problem P of size n.

Is A the best algorithm to solve P?

Upper Bound Finding better Algorithm for the Problem

If A is O(f (n)), then A is an O(f (n)) upper bound for P because wealready know an algorithm (that is, A) that can solve P in O(f (n)).

Let A1 be another algorithm that solves P in O(f1(n)). Naturally, A1is a better upper bound for P, if O(f1(n)) O(f (n)).Let A2 be yet another algorithm that solves P in O(f2(n)). Naturally,A2 is a better upper bound for P, if O(f2(n)) O(f1(n)).Let A3 be . When do we stop?


When to analyse?






Let A1 be another algorithm that solves P in O(f1(n)). Naturally, A1is a better upper bound for P, if O(f1(n)) O(f (n)).

Let A2 be yet another algorithm that solves P in O(f2(n)). Naturally,A2 is a better upper bound for P, if O(f2(n)) O(f1(n)).Let A3 be . When do we stop?


When to analyse?






Let A1 be another algorithm that solves P in O(f1(n)). Naturally, A1is a better upper bound for P, if O(f1(n)) O(f (n)).Let A2 be yet another algorithm that solves P in O(f2(n)). Naturally,A2 is a better upper bound for P, if O(f2(n)) O(f1(n)).

Let A3 be . When do we stop?


When to analyse?






Let A1 be another algorithm that solves P in O(f1(n)). Naturally, A1is a better upper bound for P, if O(f1(n)) O(f (n)).Let A2 be yet another algorithm that solves P in O(f2(n)). Naturally,A2 is a better upper bound for P, if O(f2(n)) O(f1(n)).Let A3 be . When do we stop?


When to analyse?

Lower Bound Finding better Analysis of the Problem

We must look at the input for P (really?). If that takes g(n) effort, Phas a lower bound of (g(n)). We cannot do better than (g(n)).

Lower bound is established by Adversary Argument. Let P be theproblem of sorting n numbers. We claim that P is (n), that is, onemust look at all inputs (known as Input Complexity). If not, suppose itis enough to consider only n 1 numbers. Then we can always designthe nth number as larger than the largest and make the sorting fail.Input complexity may be lower than (n). Let P be a problem to finda number that is not the largest in a list of n numbers. P is (2).Similarly Output Complexity can be defined.

Suppose we find another adversary argument to show that P is(g1(n)). It is better for P, if (g(n)) (g1(n)).Suppose we find yet another adversary argument to show that P is(g2(n)). It is better for P, if (g1(n)) (g2(n)).And so on . When do we stop?


When to analyse?



Lower bound is established by Adversary Argument. Let P be theproblem of sorting n numbers. We claim that P is (n), that is, onemust look at all inputs (known as Input Complexity). If not, suppose itis enough to consider only n 1 numbers. Then we can always designthe nth number as larger than the largest and make the sorting fail.

Input complexity may be lower than (n). Let P be a problem to finda number that is not the largest in a list of n numbers. P is (2).Similarly Output Complexity can be defined.



When to analyse?



Lower bound is established by Adversary Argument. Let P be theproblem of sorting n numbers. We claim that P is (n), that is, onemust look at all inputs (known as Input Complexity). If not, suppose itis enough to consider only n 1 numbers. Then we can always designthe nth number as larger than the largest and make the sorting fail.Input complexity may be lower than (n). Let P be a problem to finda number that is not the largest in a list of n numbers. P is (2).

Similarly Output Complexity can be defined.



When to analyse?






When to analyse?




Suppose we find another adversary argument to show that P is(g1(n)). It is better for P, if (g(n)) (g1(n)).

Suppose we find yet another adversary argument to show that P is(g2(n)). It is better for P, if (g1(n)) (g2(n)).And so on . When do we stop?


When to analyse?




Suppose we find another adversary argument to show that P is(g1(n)). It is better for P, if (g(n)) (g1(n)).Suppose we find yet another adversary argument to show that P is(g2(n)). It is better for P, if (g1(n)) (g2(n)).

And so on . When do we stop?


When to analyse?






When to analyse?




Optimal Algorithm

A is Optimal for P, if O(f (n)) = (f (n)), that is, the lower boundmatches the upper bound.

Complexity of A is (f (n)).

Till an optimal algorithm is found we continuously work on thealgorithm to lower the upper bound and devise adversary argumentsto upper the lower bound.


When to analyse?

Analysis of Sorting

Upper Bound

Brute-force: Generate all n! permutations and select the sortedsequence: O(nn), O(nn)Bubble Sort: O(n2), O(n2)Insertion Sort: O(n2), O(n2)Quick Sort: O(n2), O(n log n)Merge Sort: O(n log n), O(n log n)Heap Sort: O(n log n), O(n log n)

Lower Bound

Input Complexity: (n) must read all numbers.Comparison Tree: (n log n) choose from one of n! or O(nn)permutations with binary choice at every stage. Hence, O(log nn) =O(n log n).

Optimal Algorithm

Merge Sort / Heap Sort


When to analyse?

Analysis of Sorting

Upper Bound

Brute-force: Generate all n! permutations and select the sortedsequence: O(nn), O(nn)

Bubble Sort: O(n2), O(n2)Insertion Sort: O(n2), O(n2)Quick Sort: O(n2), O(n log n)Merge Sort: O(n log n), O(n log n)Heap Sort: O(n log n), O(n log n)

Lower Bound


Optimal Algorithm



When to analyse?

Analysis of Sorting

Upper Bound

Brute-force: Generate all n! permutations and select the sortedsequence: O(nn), O(nn)Bubble Sort: O(n2), O(n2)

Insertion Sort: O(n2), O(n2)Quick Sort: O(n2), O(n log n)Merge Sort: O(n log n), O(n log n)Heap Sort: O(n log n), O(n log n)

Lower Bound


Optimal Algorithm



When to analyse?

Analysis of Sorting

Upper Bound

Brute-force: Generate all n! permutations and select the sortedsequence: O(nn), O(nn)Bubble Sort: O(n2), O(n2)Insertion Sort: O(n2), O(n2)

Quick Sort: O(n2), O(n log n)Merge Sort: O(n log n), O(n log n)Heap Sort: O(n log n), O(n log n)

Lower Bound


Optimal Algorithm



When to analyse?

Analysis of Sorting

Upper Bound

Brute-force: Generate all n! permutations and select the sortedsequence: O(nn), O(nn)Bubble Sort: O(n2), O(n2)Insertion Sort: O(n2), O(n2)Quick Sort: O(n2), O(n log n)

Merge Sort: O(n log n), O(n log n)Heap Sort: O(n log n), O(n log n)

Lower Bound


Optimal Algorithm



When to analyse?

Analysis of Sorting

Upper Bound

Brute-force: Generate all n! permutations and select the sortedsequence: O(nn), O(nn)Bubble Sort: O(n2), O(n2)Insertion Sort: O(n2), O(n2)Quick Sort: O(n2), O(n log n)Merge Sort: O(n log n), O(n log n)

Heap Sort: O(n log n), O(n log n)

Lower Bound


Optimal Algorithm



When to analyse?

Analysis of Sorting

Upper Bound


Lower Bound


Optimal Algorithm



When to analyse?

Analysis of Sorting

Upper Bound


Lower Bound

Input Complexity: (n) must read all numbers.

Comparison Tree: (n log n) choose from one of n! or O(nn)permutations with binary choice at every stage. Hence, O(log nn) =O(n log n).

Optimal Algorithm



When to analyse?

Analysis of Sorting

Upper Bound


Lower Bound


Optimal Algorithm



Summary

Need for analysing the running-time and space requirements of aprogram.

Asymptotic growth rate or order of the complexity of differentalgorithms.

Worst-case, average-case and best-case analysis.

Developing recurrence equations.

Solving recurrences using Inductive Derivation Procedure and theMaster Theorem.


Getting StartedWhy analyse?What to analyse?How to analyse?Mathematical / Counting ModelsAsymptotic AnalysisMaster Theorem

Where to analyse?When to analyse?Summary

Documents

Elements of Algorithm Analysis