Analysis of Algorithms

Analysis of AlgorithmsAnalysis of Algorithms

Two Approaches for measuring Two Approaches for measuring running time of a programrunning time of a program BenchmarkingBenchmarking

A small collection of typical inputs that can A small collection of typical inputs that can serve as performance standardsserve as performance standards

AnalysisAnalysis Determining the general time a program Determining the general time a program

takes as a function of input sizetakes as a function of input size

Running TimeRunning Time

T(n)T(n) a function to represent the units of time a function to represent the units of time

taken by a program with input of size ntaken by a program with input of size n

T(n) T(n) # statements executed# statements executed

Running TimeRunning Time

TTww(n)(n) worst case running timeworst case running time maximum running time among all inputs of size maximum running time among all inputs of size

nn

TTavgavg(n)(n) average running timeaverage running time avg running time over all inputs of size navg running time over all inputs of size n more realistic; harder to computemore realistic; harder to compute

Calculate T(n)Calculate T(n)

small = i;small = i;

for (j=i+1; j<=n; j++) {for (j=i+1; j<=n; j++) {

if (A[j] < A[small])if (A[j] < A[small])

small = j;small = j;

Count 1 unit of time for each assign, compareCount 1 unit of time for each assign, compare

Big-O notationBig-O notation

A tool to analyze a program's efficiency.A tool to analyze a program's efficiency.

An approximation ofAn approximation of work an algorithm performs work an algorithm performs as a function of size of inputas a function of size of input

WorkWork

A measure of A measure of effort expended effort expended by computerby computer in performing a computationin performing a computation

Big-Big-OO

f(x)f(x) is O( is O(g(x)g(x))) if there are constantsif there are constants C C and and kk such that such that

| | f(x)f(x) | | C C | | g(x)g(x) | |

whenever whenever xx > > kk

Show f(x) = xShow f(x) = x22+2x+1 is O(x+2x+1 is O(x22))

Prove:Prove: |x|x22+2x+1|+2x+1| C C | | xx22 | |

Choose k > 0, and f(x) is always positiveChoose k > 0, and f(x) is always positive

Let C be the sum of f(x) coefficientsLet C be the sum of f(x) coefficients

xx22+2x+1+2x+1 1x1x22+2x+2x22+1x+1x2 = 2 = 4x4x22

This inequality hold beginning at x=1This inequality hold beginning at x=1

Choosing k=1, C=4, xChoosing k=1, C=4, x22+2x+1 is O(x+2x+1 is O(x22))

Show f(x) = 7xShow f(x) = 7x33 is O(x is O(x22))

Prove:Prove: |7x|7x33|| C C | | xx2 2 ||

Assuming k > 0, divide both sides by xAssuming k > 0, divide both sides by x22

7x7x C C

No such C exists, since x is arbitrarily largeNo such C exists, since x is arbitrarily large (Choose any C, and there is an x > C)(Choose any C, and there is an x > C)

7x7x33 is is notnot O(x O(x22))

Commonly used Big-O values Commonly used Big-O values (orders of magnitude)(orders of magnitude)

Big-O Big-O Name Name O(1)O(1) Constant time Constant time LowerLower

O(log n)O(log n) Logarithmic time Logarithmic time

O(n)O(n) Linear time Linear time

O(n log n) O(n log n)

O(nO(n22)) Quadratic time Quadratic time

O(nO(n33)) Cubic time Cubic time

O(nO(nkk) ) Polynomial time Polynomial time

O(2O(2nn)) Exponential time Exponential time HigherHigher

O(n!) O(n!) Factorial time Factorial time Lower is always O(higher) for nLower is always O(higher) for n

Big-O Big-O Example algorithmExample algorithm

O(1)O(1) Assigning value to ith array element; Assigning value to ith array element;

always the same number of stepsalways the same number of steps

not necessarily shortnot necessarily short

a program which takes 320 steps, a program which takes 320 steps,

reguardless of input valuesreguardless of input values

O(log n)O(log n) Binary search, Binary search,

What power of 2 is greater than a input number? What power of 2 is greater than a input number?

A loop whose terminal value is being successively A loop whose terminal value is being successively halved or doubledhalved or doubled

Big-O Big-O Example algorithmExample algorithm

O(n)O(n) Printing all elements of an array; Printing all elements of an array; searching an unordered array; searching an unordered array; A loop which executes 1 to N times A loop which executes 1 to N times (where N is the input size, (where N is the input size, the number of data values being the number of data values being

processed)processed)

O(n log n) O(n log n) Faster sorts; Faster sorts; A loop whose index is being halved/doubled A loop whose index is being halved/doubled inside a loop executing from 1 to Ninside a loop executing from 1 to N

Big-O Big-O Example algorithmExample algorithm

O(nO(n22) ) Slower sorts; Slower sorts; A loop which executes from 1 to N times A loop which executes from 1 to N times inside a loop executing from 1 to N inside a loop executing from 1 to N

timestimes

O(nO(n33) ) Incrementing all the elements in a NxNxN array; Incrementing all the elements in a NxNxN array; A 1..N loop inside a 1..N loop inside a 1..N loopA 1..N loop inside a 1..N loop inside a 1..N loop

O(nO(nkk) ) kk levels of loops inside loops levels of loops inside loops

Big-O Big-O Example algorithmExample algorithm

O(2O(2nn)) List all the subsets of a set with n elements; List all the subsets of a set with n elements; practical only with small values of n; practical only with small values of n; n=64 takes 5 years on a supercomputern=64 takes 5 years on a supercomputer

O(n!) O(n!) List all the possible arrangements of a set with List all the possible arrangements of a set with n n elementselements

Table of Common Running TimesTable of Common Running Times

N Linear N Log N Quadratic Cubic

10 10 10.0 100 100020 20 26.0 400 800030 30 44.3 900 2700040 40 64.0 1600 6400050 50 84.9 2500 12500060 60 106.7 3600 21600070 70 129.2 4900 34300080 80 152.2 6100 48800090 90 175.9 8100 729000

100 100 200.0 10000 1000000

log n

n

n lo

g n

n2

2n

n!

Growth Rate of Some Growth Rate of Some FunctionsFunctions

0100200300400500600700800900

1000

0 10 20 30 40 50 60 70 80 90 100

Linear

nLogn

Quadratic

Cubic

Computing order of complexity of Computing order of complexity of an algorithman algorithm a.) Multiplicative constants do not matter.a.) Multiplicative constants do not matter.

O(3n) = O(n)O(3n) = O(n)

b.) Addition is done by taking the max. b.) Addition is done by taking the max. O(nO(n22) + O(n) = O(max(n) + O(n) = O(max(n22, n)) = O(n, n)) = O(n22)) O(nO(n33) + O(log n) = O(max(n) + O(log n) = O(max(n33, log n)) = O(n, log n)) = O(n33))

c.) Multiplication remains multiplication.c.) Multiplication remains multiplication. O(n) * O(log n) = O(n log n)O(n) * O(log n) = O(n log n) O(n) * O(nO(n) * O(n22) = O(n) = O(n33))

Example ComplexitiesExample Complexities

4n4n3 3 + 20n + 30 = O(n+ 20n + 30 = O(n33)) n + 10000 = O(n)n + 10000 = O(n) 4n4n4 4 + 20n + 30 = O(n+ 20n + 30 = O(n44)) 22nn + n + n33 = O(2 = O(2nn)) 200 = O(1)200 = O(1)

What is the Big-O ofWhat is the Big-O of

outputoutput inputinput assignmentassignment ifif if-elseif-else

a blocka block while loopwhile loop for loopsfor loops nested for loopsnested for loops

Running Time of StatementsRunning Time of Statements

Simple statements Simple statements (i.e. initialization of variables) are O(1)(i.e. initialization of variables) are O(1)

Loops are O(g(n) f(n))Loops are O(g(n) f(n)) g(n) is upper bound on number of loop g(n) is upper bound on number of loop

iterations iterations f(n) is upper bound on the body of the loop. f(n) is upper bound on the body of the loop.

((If g(n) and f(n) are constant, then this is If g(n) and f(n) are constant, then this is constant timeconstant time).).

Running Time of Statements Running Time of Statements (continued)(continued) Conditional statements are Conditional statements are

O(max(f(n),g(n))) O(max(f(n),g(n))) where f(n) is upper bound on then part and where f(n) is upper bound on then part and g(n) is upper bound on else partg(n) is upper bound on else part

Blocks of statements with complexities Blocks of statements with complexities ff11(n), f(n), f22(n), ..,f(n), ..,fkk(n) (n)

have complexity have complexity O(fO(f11(n) + f(n) + f22(n)+ ...+ f(n)+ ...+ fkk(n))(n))

Simple AnalysisSimple Analysis

cin >> n; // 1if (n > 20) // 1

cout << “Number is > 20” << endl; // 1else

cout << “Number is <= 20” << endl; // 1

T(n) = 2 + max (1,1) = 3 = O(1)

Example AnalysisExample Analysis

cin >> n; // 1factorial = 1; // 1

for (i = 2; i <=n; i++) // 1 init + 1 test + //(1 test + 1 inc) per iteraton

factorial *= i; // 1

cout << factorial; // 1

T(n) = 5 + 3*(n-1) = O(n)

Another exampleAnother example

cin >> n; // 1if (n > 0) { // 1 factorial = 1; // 1 for (i = 2; i <=n; i++) // 1 initialization +1 test +

//(1 test + 1 inc) per iterationfactorial *= i; // 1

cout << factorial; // 1}else cout << “Can’t do factorial of” << n; // 1 T(n) = 2 + max(4+3*(n-1), 1) = O(n)

Analysis of simple function Analysis of simple function callscalls

int factorial(int n){ int fact=1; for (i = 2; i <=n; i++)

fact *= i; return fact;} void main(){ int n; // 1

cin >> n; // 1 cout << factorial(n) << endl; //O(n)}

Main is O(n)

Analysis of Nested LoopsAnalysis of Nested LoopsEXAMPLE 1:for (int i = 0; i < n; i++) // n*O(n) = O(n2) for (int j=0; j<n;j++) // n*O(1) = O(n) x++; // O(1)

EXAMPLE 2:for (int i = 0; i < n; i++) for (int j=i; j<n;j++) x++; // O(1)

T(n) = n+(n-1) + (n-2)+…+ 1 = n(n+1)/2 = O(n2)

Summing blocks of Summing blocks of statementsstatements

for (int i = 0; i < n; i++) // O(n2) for (int j=0; j<n;j++) x++;

// Next block follows in same programfor (int i = 0; i < n; i++) // O(n) x++;

T(n) = O(n2) + O(n) = O(n2 + n) = O(n2)

Loops with different limitsLoops with different limits

for (int i = 0; i < n; i++) // n*O(m) = O(mn) for (int j=0; j<m;j++) // m*O(1) = O(m) x++; // O(1)

Recursive AlgorithmsRecursive Algorithms

Complexity of a recursive algorithm is the Complexity of a recursive algorithm is the product ofproduct of

Amount of work done on any one levelAmount of work done on any one level Number of recursive callsNumber of recursive calls

Recursive FactorialRecursive Factorialint factorial(int n){ if ( n <= 1)

return 1; else return n * factorial(n - 1);} void main(){ int n; // 1

cin >> n; // 1 cout << factorial(n) << endl; //O(n)}

Main is O(n)

More complex loopsMore complex loops

for (i = 1;i < n; i *=2) x++;

for (i=n; i > 0; i/=2) x++;

Repetitive halving or doubling results in logarithmic complexity. Both loops are O(log n)

Worst-Case and Average-Worst-Case and Average-Case AnalysisCase Analysis Worst-case is a guarantee over Worst-case is a guarantee over all all

inputs of a given size inputs of a given size

Average-case is the running time as an Average-case is the running time as an average over all the possible inputs of a average over all the possible inputs of a given sizegiven size

What is the Big-O ofWhat is the Big-O of

outputoutput inputinput assignmentassignment ifif if-elseif-else

a blocka block while loopwhile loop for loopsfor loops nested for loopsnested for loops