Intro Analysis

7/29/2019 Intro Analysis

1/114

1

Introduction to ALgorithmicAnalysis

In this chapter, we will answer thefollowing two questions

What does it mean to be an efficientalgorithm?

How can one tell that it is more efficient thanother algorithms?

based on some easy-to-understandsearching and sorting algorithms that wemay have seen earlier.

Thursday, September 15, 11


2/114

2

Searching Problem

Assume A is an array with n elements A[1],A[2], A[n]. For a given element x, we must

determine whether there is an indexj; 1 j n, such that x=A[j]

Two algorithms, among others, address this

problem Linear Search

Binary Search



3/114

3

Linear Search Algorithm

Algorithm: LINEARSEARCH

Input: arrayA[1..n] ofn elements and an

elementx.Output:jifx=A[j], 1 jn, and 0 otherwise.

1. j 1

2.while (j < n) and(xA[j])

3. j j + 1

4. end while

5. if x= A[j] then return j else return 0



4/114

4

Analyzing Linear Search

One way to measure efficiency is to count howmany statements get executed before the algorithmterminates

One should keep an eye, though, on statements thatare executed repeatedly.

What will be the number of element comparisons if

X first appears in the first element of A

X first appears in the middle element of A

X first appears in the last element of A

X doesnt appear in A.



5/114

5

Binary Search

We can do better than linear search if we knewthat the elements of A are sorted, say in non-decreasing order.

The idea is that you can compare x to themiddle element of A, say A[middle]. If x < A[middle] then you know that x cannot be an

element from A[middle+1], A[middle+2], A[n]. Why?

If x > A[middle] then you know that x cannot be an

element from A[1], A[2], A[middle-1]. Why?



6/114

6

Binary Search Algorithm

Algorithm: BINARYSEARCH

Input:An arrayA[1..n] ofn elements sorted innondecreasing order and an elementx.

Output:jifx=A[j], 1 jn, and 0 otherwise.

1. low 1; high n; j 02.while (lowhigh) and(j = 0)

3. mid (low+ high)/2

4. if x= A[mid] then j mid

5. else if x < A[mid] then high mid - 16. else low mid+ 1

7. end while

8. return j



7/114

7

Worst Case Analysis of Binary

What to do: Find the maximum number of elementcomparisons

How to do:

The number of element comparisons is equal to the

number of iterations of the while loop in steps 2-7. HOW? How many elements of the input do we have in the

First iteration

Second iteration

Thrid iteration

ith iteration

The last iteration occurs when the size of array = 1.



8/114

8

Theorem

The number of comparisons performed byAlgorithm BINARYSEARCH on a sortedarray of size n is at most



9/114

9

Merging Two Sorted Lists

Problem Description: Given two lists (arrays) that aresorted in non-decreasing order, we need to merge theminto one list sorted in non-decreasing order.

Example:

3 7 9 12 1 2 4 13 14

Input

Output



10/114

9

Merging Two Sorted Lists

Problem Description: Given two lists (arrays) that aresorted in non-decreasing order, we need to merge theminto one list sorted in non-decreasing order.

Example:

3 7 9 12 1 2 4 13 14

Input

Output

1 2 3 4 7 9 12 13 14Thursday, September 15, 11


11/114

We will be interested in merging twosubarrays.

Input: A[1..n], p, q, r.

Merge A[p..q] with A[q+1..r].

10



12/114

10

How to merge two subarrays?

3 7 8

A[4..6]A[1..3]B[1..6] (temp)

2 4 5



13/114

10


3 7 8

A[4..6]A[1..3]B[1..6] (temp)

3 7 8

2 4 5

2 4 52



14/114

10


3 7 8

A[4..6]A[1..3]B[1..6] (temp)

3 7 8

3 7 8

2 4 5

2 4 5

2 4 52

2 3



15/114

10


3 7 8

A[4..6]A[1..3]B[1..6] (temp)

3 7 8

3 7 8

3 7 8

2 4 5

2 4 5

2 4 5

2 4 5

2

2 3

2 3 4



16/114

10


3 7 8

A[4..6]A[1..3]B[1..6] (temp)

3 7 8

3 7 8

3 7 8

3 7 8

2 4 5

2 4 5

2 4 5

2 4 5

2 4 5

2

2 3

2 3 4

2 3 4 5



17/114

10


3 7 8

A[4..6]A[1..3]B[1..6] (temp)

3 7 8

3 7 8

3 7 8

3 7 8

3 7 8

2 4 5

2 4 5

2 4 5

2 4 5

2 4 5

2 4 5

2

2 3

2 3 4

2 3 4 5

2 3 4 5 7 8



18/114

11

Algorithm MERGE

Algorithm: MERGE

Input: An arrayA[1..m] of elements and threeindicesp, q and r, with 1 p q


19/114

12

Algorithm MERGE (Cont.)

1. s p; tq + 1; kp

2. while s q and t r

3. ifA[s] A[t] then

4. temp[k] A[s]

5. s s + 1

6. else

7. temp[k] A[t]

8. tt+ 1

9. end if

10. kk+ 1

11. end while

12. if (s = q + 1) then temp[k..r] A[t..r]

13. else temp[k..r] A[s..q]

14. end if

15.A[p..r] temp[p..r]

merge A[p..q] with A[q+1..r]



20/114

13

Analyzing MERGE

Assuming arrays A[p,q] and A[q+1,r],where p =1 and r =n,

n element assignments are needed to

copy A to temp. n element assignments are needed to

copy temp to A.

The total number of element assignmentsis 2n.

Hence, the time complexity is O(n).



21/114

14

Selection Sort

Algorithm: SELECTIONSORTInput:An arrayA[1..n] ofn elements.

Output:A[1..n] sorted in nondecreasing order.

1. for i 1 to n - 1

2. k i3. for j i + 1 to n

{Find the index of the ithsmallest element}

4. if A[j] < A[k] then k j

5.end for

6. if k i then interchange A[i] andA[k]

7. end for



22/114

15

Selection Sort Example

42 85 9i k



23/114

15


42 85 9i k

1



24/114

15


42 85 9i k

1 2



25/114

15


42 85 9

45 82 9

i k

1 2



26/114

15


42 85 9

45 82 9

i k

1 2

2



27/114

15


42 85 9

45 82 9

i k

1 2

2 5



28/114

15


42 85 9

45 82 9

i k

1 2

54 82 92 5



29/114

15


42 85 9

45 82 9

i k

1 2

54 82 92 5

3



30/114

15


42 85 9

45 82 9

i k

1 2

54 82 92 5

3 5



31/114

15


42 85 9

45 82 9

i k

1 2

54 82 92 5

94 82 53 5



32/114

15


42 85 9

45 82 9

i k

1 2

54 82 92 5

94 82 53 5

4



33/114

15


42 85 9

45 82 9

i k

1 2

54 82 92 5

94 82 53 5

4 4



34/114

15


42 85 9

45 82 9

i k

1 2

54 82 92 5

94 82 53 5

94 82 54 4


A l i S l ti S t


35/114

16

Analyzing Selection Sort

Iteration # of comparisons1 n-1

2 n-2

3 n-3.. ..

n-1 1

Total is Hence, time complexity is



36/114

17

Insertion Sort

Algorithm: INSERTIONSORTInput:An arrayA[1..n] ofn elements.

Output:A[1..n] sorted in nondecreasing order.1. for i 2 to n

2. xA[i]3. j i - 14. while (j >0) and(A[j] > x)

5. A[j + 1] A[j]6. j j - 17. end while

8. A[j + 1] x9. end for



37/114

18

Insertion Sort Example

42 85 9x=2


I i S E l


38/114

18


42 85 9x=2

45 89


I ti S t E l


39/114

18


42 85 9x=2

45 82 9


I ti S t E l


40/114

18


42 85 9x=2

45 82 9x=9


I ti S t E l


41/114

18


42 85 9x=2

45 82 9x=9

48


I ti S t E l


42/114

18


42 85 9x=2

45 82 9x=9

45 82 9


I ti S t E l


43/114

18


42 85 9x=2

45 82 9x=9

45 82 9x=8




44/114

18


42 85 9x=2

45 82 9x=9

45 82 9x=8

4




45/114

18


42 85 9x=2

45 82 9x=9

45 82 9x=8

49




46/114

18


42 85 9x=2

45 82 9x=9

45 82 9x=8

45 92




47/114

18


42 85 9x=2

45 82 9x=9

45 82 9x=8

45 92 8




48/114

18


42 85 9x=2

45 82 9x=9

45 82 9x=8

45 92 8x=4




49/114

18


42 85 9x=2

45 82 9x=9

45 82 9x=8

45 92 8x=4




50/114

18


42 85 9x=2

45 82 9x=9

45 82 9x=8

45 92 8x=4

9




51/114

18


42 85 9x=2

45 82 9x=9

45 82 9x=8

45 92 8x=4

98




52/114

18


42 85 9x=2

45 82 9x=9

45 82 9x=8

45 92 8x=4

985




53/114

18


42 85 9x=2

45 82 9x=9

45 82 9x=8

45 92 8x=4

982 5




54/114

18


42 85 9x=2

45 82 9x=9

45 82 9x=8

45 92 8x=4

94 82 5


Analyzing Insertion Sort


55/114

19

Analyzing Insertion Sort

The minimum number of elementcomparisons is n-1which occurs when input is sorted

The maximum number of element

comparisons is

which occurs when input is reverselysorted. Time complexity is


Bottom-Up Merge Sort


56/114

20

Bottom Up Merge Sort

Informally, the algorithm does thefollowing

1. Divide the array into pairs of elements (with

possibly a single elements in case thenumber of elements is odd).

2. Merge each pair in non-decreasing order(with possibly a single pair left)

3. Repeat step 2 until there is only one pairleft.


Bottom-Up Merge Sort Example


57/114

21

Bottom Up Merge Sort Example

42 318 1075 9 12 6




58/114

21


5 2

42 318 1075 9 12 6




59/114

21


5 2 9 8

42 318 1075 9 12 6




60/114

21

o o Up e ge So a p e

5 2 9 8 4 12

42 318 1075 9 12 6




61/114

21

p g p

5 2 9 8 4 12 7 1

42 318 1075 9 12 6




62/114

21

p g p

5 2 9 8 4 12 7 1 3 6

42 318 1075 9 12 6




63/114

21

p g p

5 2 9 8 4 12 7 1 3 6 10

42 318 1075 9 12 6




64/114

21

p g p

5 2 9 8 4 12 7 1 3 6 10

52 98 4 12 71 3 6 10

42 318 1075 9 12 6




65/114

21

p g p

5 2 9 8 4 12 7 1 3 6 10

52 98 4 12 71 3 6 10

52 98

42 318 1075 9 12 6




66/114

21

g

5 2 9 8 4 12 7 1 3 6 10

52 98 4 12 71 3 6 10

52 98 4 1271

42 318 1075 9 12 6




67/114

21

5 2 9 8 4 12 7 1 3 6 10

52 98 4 12 71 3 6 10

52 98 4 1271 3 6 10

42 318 1075 9 12 6




68/114

21

5 2 9 8 4 12 7 1 3 6 10

52 98 4 12 71 3 6 10

52 98 4 1271 3 6 10

52 984 1271

42 318 1075 9 12 6




69/114

21

5 2 9 8 4 12 7 1 3 6 10

52 98 4 12 71 3 6 10

52 98 4 1271 3 6 10

52 984 1271 3 6 10

42 318 1075 9 12 6




70/114

21

5 2 9 8 4 12 7 1 3 6 10

52 98 4 12 71 3 6 10

52 98 4 1271 3 6 10

52 984 1271 3 6 10

52 984 1271 3 6 10

42 318 1075 9 12 6


Algorithm BOTTOMUPSORT


71/114

22

Algorithm: BOTTOMUPSORT

Input:An arrayA[1..n] ofn elements.

Output:A[1..n] sorted in nondecreasing order.1. t 1

2.while t < n

3. s t; t 2s; i 04. while i + t n

5. MERGE(A, i + 1, i + s, i + t)

6. i i + t

7.end while

8. if i + s < n then

9. MERGE(A, i + 1, i+ s, n)

10. end while


Analyzing AlgorithmWith no loss of generality, assume the size of the array is a power of 2.


72/114

23

With no loss of generality, assume the size of the array is a power of 2.

In the first iteration, we have pairs that are merged using

element assignments.

In the first iteration, we have pairs that are merged using

assignments.

.

In the last iteration, we have 1 pair that are merged using element

assignments.

n

2

n

2

*2*2 = 2n

n

4

* 4 * 2 = 2n

n

4

2 * 2 *n

2

= 2n


Analyzing AlgorithmBOTTOMUPSORT


73/114

24

Number of iterations is log n, since the number ofpairs is reduced by half in each iteration.

Hence, run time is 2n x log n = O(n log n).


Time Complexity


74/114

25

One way of measuring the performance of analgorithm is how fast it executes. The question ishow to measure this time? Is having a digital stop watch suitable?

In general, we are not so much interested in thetime and space complexity for small inputs.

For example, while the difference in time

complexity between linear and binary search ismeaningless for a sequence with n = 10, it isgigantic for n = 230.


Complexity

F l l t t l ith A d B


75/114

26

For example, let us assume two algorithms A and B

that solve the same class of problems.

The time complexity of A is 5,000n, the one for B is1.1n for an input with n elements.

For n = 10, A requires 50,000 steps, but B only 3,so B seems to be superior to A.

For n = 1000, however, A requires 5,000,000 steps,while B requires 2.51041 steps.


Complexity

C i i l i i f l i h A d B


76/114

28

Comparison: time complexities of algorithms A and B

Algorithm A Algorithm BInput Size

n10

100

1,0001,000,000

5,000n50,000

500,000

5,000,0005109

1.1

n

3

2.510

41

13,781

4.81041392


Order of Growth

This means that algorithm B cannot be used for


77/114

28

This means that algorithm B cannot be used for

large inputs, while algorithm A is still feasible.

So what is important is the growth of thecomplexity functions.

The growth of time and space complexity withincreasing input size n is a suitable measure forthe comparison of algorithms.

we focus on asymptotic analysis


Example


78/114

29Growth rate for some function


Example


79/114

30

Growth rate for same previous functions

showing larger input sizes


Running Times for Different Sizesof Inputs of Different Functions


80/114

31

p


Running Times for Different Sizesof Inputs of Different Functions


81/114

31

Complexity10 20 30 40 50 60

n 110-5 sec 210-5 sec 310-5 sec 410-5 sec 510-5 sec 610-5 sec

n2 0.0001 sec 0.0004 sec 0.0009 sec 0.016 sec 0.025 sec 0.036 sec

n3 0.001 sec 0.008 sec 0.027 sec 0.064 sec 0.125 sec 0.216 sec

n5 0.1 sec 3.2 sec 24.3 sec 1.7 min 5.2 min 13.0 min2n 0.001sec 1.0 sec 17.9 min 12.7 days 35.7 years 366 cent

3n 0.59sec 58 min 6.5 years 3855 cent 2108cent 1.31013cent

log2

n 310-6 sec 410-6 sec 510-6 sec 510-6 sec 610-6 sec 610-6 sec

n log2n 310-5 sec 910-5 sec 0.0001 sec 0.0002 sec 0.0003 sec 0.0004 sec


Asymptotic Analysis: Big-oh (O())


82/114

32

Definition: ForT(n) a non-negatively valuedfunction, T(n) is in the set O(f(n)) if there existtwo positive constants cand n0 such that T(n)

cf(n) for all n > n0.

Meaning: For all data sets big enough (i.e.,n>n0), the algorithm always executes in lessthan or equal to cf(n) steps.


The Growth of Functions

Th id b hi d th bi O t ti i t t bli h


83/114

33

The idea behind the big-O notation is to establish an

upper boundary for the growth of a function f(n) forlargen.

This boundary is specified by a function g(n) that isusually much simplerthan f(n).

We accept the constant c in the requirement f(n) cg(n) whenever n > k, because c does not growwith n.

We are only interested in large n, so it is OK iff(n) > cg(n) for n k.



Enample:


84/114

34

Enample:

Show that f(n) = n2 + 2n + 1 is O(n2).

For n > 1 we have:

n2

+ 2n + 1 n2

+ 2n2

+ n2

n2 + 2n + 1 4n2

Therefore, for c = 4 and k = 1:

f(n) cn2 whenever n > k. f(n) is O(n2).




85/114

35

Question: If f(n) is O(n

2

), is it also O(n

3

)?

Yes. n3 grows faster than n2, so n3 grows alsofaster than f(n).

Therefore, we always try to find the smallestsimple function g(n) for which f(n) is O(g(n)).



Popular functions g(n) are


86/114

36

n log n, 1, 2n

, n2

, n!, n, n3

, log nListed from fastest to slowest growth:

1log n

nn log nn2

n3

2n

n!


Asymptotic Analysis: Big-Omega (())


87/114

37

Definition: ForT(n) a non-negatively valuedfunction, T(n) is in the set (g(n)) if there existtwo positive constants cand n

0such that T(n)

>= cg(n) for all n > n0.

Meaning: For all data sets big enough (i.e., n> n

0), the algorithm always executes in more

than or equal to cg(n) steps.

() notation indicates a lower bound.


Asymptotic Analysis: Big Theta (())


88/114

38

When O() and () meet, we indicate thisby using () (big-Theta) notation.

Definition: An algorithm is said to be(f(n)) if it is in O(f(n)) and it is in (f(n)).


Using the limit criterion


89/114

41

Let

- If , then

- If , then

- If , then41

L 0

L = c > 0

L = limn

f(n)

g(n)

f(n)=

(g(n))

f(n) = (g(n))

f(n) =O(g(n))L


Summary

limf(n)

f (n) =O(g(n)) if


90/114

42

g(n) =O(f(n))f(n) = (g(n)) if

limn

g(n)

f(n) O(g(n)) if

f(n) = (g(n)) if and

g(n) =O(f(n))f(n) = (g(n)) if f(n) =O(g(n)) and

f(n) =O(g(n))f(n) = (g(n))


Examples 1


91/114

44

Hence,

It follows that


Examples 2


92/114

45

Hence,

It follows that


Examples 3


93/114

46

Since 100 > 0,

This implies that

and


Example 4

Prove that


94/114

47

Prove that

We only need to show that the two functions are not related by O or

Hence, which means

Or

Since the limit is not a constant > 0,


Example 5


95/114

39

Show that log(n!) is in (n log n).


Estimating the Running Time of anAlgorithm

We need to focus on counting those


96/114

50

We need to focus on counting thoseoperations which represent, in general,the behavior of the algorithm

This is achieved byCounting the

frequency ofbasic operations.

A Basic operation is an operation withhighest frequency to within a

constant factor among all otherelementary operations


Computing running time continued..


97/114

54


Number of iterations (additions)


98/114

16



While Iteration # of additions


99/114

16


1 n

2 n/2

3 n/4

.. ..

k n/n





100/114

16


1 n

2 n/2

3 n/4

.. ..

k n/n



101/114

54


Computing running time continued..

Algorithm 1.9 count2


102/114

51

Input: A positive integer n.Output: count = number of times Step 5 is executed.

1. count 02. for i 1 to n3. mn/i4. for j 1 to m5. count count + 16. end for

7. end for8. return count




103/114

16





104/114

16

1 n

2 n/2

3 n/3

.. ..

n n/n





105/114

16

1 n

2 n/2

3 n/3

.. ..

n n/n

ni=0

n

i

ni=0

n

i= n

ni=0

1

i= O(n log n).


Algorithm 1.9 count2

Input: A positive integer n.


106/114

ni=0

n

i

ni=0

n

i= n

ni=0

1

i= O(n log n).

53

Output: count = number of times Step 5 is executed.1. count 02. for i 1 to n3. mn/i

4. for j 1 to m5. count count + 16. end for7. end for8. return count


Useful Summation Formulas


107/114

54




108/114

54

Pnj=1 j =

n(n+1)2 = (n

2)




109/114

54

Pnj=1 j =

n(n+1)2 = (n

2)

Pn

j=0

aj = a

n+11

a1

; a 6= 1




110/114

54

Pnj=1 j =

n(n+1)2 = (n

2)

Pnj=0

1

2j= 2 1

n= (1)

Pn

j=0

aj = a

n+11

a1

; a 6= 1




111/114

54

Pnj=1

1j lnn = (logn)

Pnj=1 j =

n(n+1)2 = (n

2)

Pnj=0

1

2j= 2 1

n= (1)

Pn

j=0

aj = a

n+11

a1

; a 6= 1


Space Complexity


112/114

48

Space complexity refers to the number ofmemory cells needed to carry out thecomputational steps required in an algorithmexcluding memory cells needed to hold theinput.

Compare additional space needed to carry outSELECTIONSORT to that of

BOTTOMUPSORT if we have an array with 2million elements!


Examples

What is the space complexity for


113/114

49

Linear search

Binary search

Selection sort

Insertion sort Merge (that merges two sorted lists)

Bottom up merge sort


Complexity Classes and small-oh(o())

Using ()notation, one can divide the functions


114/114

40

into different equivalence classes, where f(n)and g(n) belong to the same equivalence classiff(n) = (g(n))

To show that two functions belong to different

equivalence classes, the small-oh notation hasbeen introduced Definition: Let f(n) and g(n) be two functions

from the set of natural numbers to the set ofnon-negative real numbers. f(n) is said to be in

o(g(n)) if for every constant c > 0, there is apositive integern0such that f(n) < cg(n) for all nn0.


Documents

Intro Analysis