# Lecture 1: Asymptotics, Recurrences, Elementary Sortingsaravanan- · PDF fileLecture 1: Asymptotics, Recurrences, Elementary Sorting Instructor: Saravanan Thirumuruganathan CSE 5311

• View
216

0

Embed Size (px)

### Text of Lecture 1: Asymptotics, Recurrences, Elementary Sortingsaravanan- · PDF fileLecture 1:...

• Lecture 1: Asymptotics, Recurrences, ElementarySorting

Instructor: Saravanan Thirumuruganathan

CSE 5311 Saravanan Thirumuruganathan

• Outline

1 Introduction to Asymptotic Analysis

Rate of growth of functionsComparing and bounding functions: O,,Specifying running time through recurrencesSolving recurrences

2 Elementary Sorting Algorithms

Bubble, Insertion and Selection sortStability of sorting algorithms

CSE 5311 Saravanan Thirumuruganathan

• In-Class Quizzes

URL: http://m.socrative.com/Room Name: 4f2bb99e

CSE 5311 Saravanan Thirumuruganathan

http://m.socrative.com/
• Analyzing Algorithms

Time Complexity:Quantifies amount of time an algorithm needs to complete asa function of input size

Space Complexity:Quantifies amount of space an algorithm needs to completeas a function of input size

Function: Input size Vs {Time, Space}

CSE 5311 Saravanan Thirumuruganathan

• Analyzing Algorithms

Best Case Complexity:of an algorithm is the function that determines the minimumnumber of steps taken on any problem instance of size n

Worst Case Complexity:. . . maximum . . .

Average Case Complexity:. . . average . . .

Function: Input size Vs {Time, Space}

CSE 5311 Saravanan Thirumuruganathan

• Rate of Growth of Functions

Growth Function T (n)Input is positive integers n = 1, 2, 3, . . .

Asymptotically positive (returns positive numbers for large n)

How does T (n) grow when n grows?

n is size of input

T (n) is the amount of time it takes for an algorithm to solvesome problem

CSE 5311 Saravanan Thirumuruganathan

• Rate of Growth of Functions

CSE 5311 Saravanan Thirumuruganathan

• Quiz!

Question:You have a machine that can do million operations per second.

Suppose size of input is 1 million

How long does the algorithm takes for this input?

CSE 5311 Saravanan Thirumuruganathan

• Quiz!

Machine can do 1M operations per second

Running time = (1M)2

1M = 1M seconds

1M seconds = 1M606024 = Approximately 12 days

CSE 5311 Saravanan Thirumuruganathan

• Why does it matter?

Running time of different algorithms for various input sizes

1

1Table 2.1 from K&T Algorithm Design. Very long means it takes morethan 1025 years.

CSE 5311 Saravanan Thirumuruganathan

• Why does it matter?

The Big Data era

CSE 5311 Saravanan Thirumuruganathan

• Functions in Real World

This is how functions look in the real world!2

2Skiena Lecture notesCSE 5311 Saravanan Thirumuruganathan

• Solution: Analyze Asymptotic Behavior

Analyze the asymptotic behavior of algorithms

What happens to f (n) when n?

T (n) = 1000n T (n) = n2

n = 10 10K 100

n = 100 100K 10K

n = 1000 1M 1M

n = 10K 10M 100M

n = 100K 100M 10B

CSE 5311 Saravanan Thirumuruganathan

• Solution: Bound the functions

Identify known functions (such as n, n2, n3, 2n, . . .) that canbound T (n)How to bound? - asymptotic upper, lower and tight boundsFind a function f (n) such that T (n) is proportional to f (n)Why proportional (as against equal)?Ignore aspects such as programming language, programmercapability, compiler optimization, machine specification etc

CSE 5311 Saravanan Thirumuruganathan

• O,,3

3CLRS bookCSE 5311 Saravanan Thirumuruganathan

• Big-O Notation

Upper bounds: T (n) is O(f (n)) if there exists constantsc > 0 and n0 0 such that T (n) c f (n) for all n n0

4

4From K&T: Algorithm DesignCSE 5311 Saravanan Thirumuruganathan

• Big-O Notation

Upper bounds: T (n) is O(f (n)) if there exists constantsc > 0 and n0 0 such that T (n) c f (n) for all n n0

Example: T (n) = 32n2 + 17n + 1. Is T (n) in O(n2)?

Yes! Use c = 50, n0 = 1

Simple Proof:

T (n) 32n2 + 17n + 1 32n2 + 17n2 + 1n2

50n2

cn2

c = 50 and n0 = 1

Note: Not necessary to find the smallest c or n0

CSE 5311 Saravanan Thirumuruganathan

• Quiz!

Example: T (n) = 32n2 17n + 1. Is T (n) in O(n2)?

CSE 5311 Saravanan Thirumuruganathan

• Big-O Notation

Example: T (n) = 32n2 17n + 1. Is T (n) in O(n2)?Yes! Use c = 50, n0 = 1

Simple Proof:

T (n) 32n2 17n + 1 32n2+17n + 1 32n2 + 17n2 + 1n2

50n2

cn2

c = 50 and n0 = 1

CSE 5311 Saravanan Thirumuruganathan

• Quiz!

Example: T (n) = 32n2 17n + 1. Is T (n) in O(n3)?

CSE 5311 Saravanan Thirumuruganathan

• Big-O Notation

Example: T (n) = 32n2 17n + 1. Is T (n) in O(n3)?Yes! Use c = 50, n0 = 1

Simple Proof:

T (n) 32n2 17n + 1 32n2+17n + 1 32n2 + 17n2 + 1n2

50n2

50n3

cn3

c = 50 and n0 = 1

CSE 5311 Saravanan Thirumuruganathan

• Quiz!

Example: T (n) = 32n2 + 17n + 1. Is T (n) in O(n)?

CSE 5311 Saravanan Thirumuruganathan

• Big-O Notation

Example: T (n) = 32n2 + 17n + 1. Is T (n) in O(n)?

No!

32n2 + 17n + 1 c n

32n + 17 +1

n c

32n c (ignore constants for now)n c (ignore constants for now)

This inequality does not hold for n = c + 1!

CSE 5311 Saravanan Thirumuruganathan

• Set Theoretic Perspective

O(f (n)) is set of all functions T (n) where there exist positiveconstants c , n0 such that 0 T (n) c f (n) for all n n0Example: O(n2) = { n2, . . . , 32n2 + 17n + 1, 32n2 17n + 1,. . . , n, 2n, . . .}Notation: T (n) = O(f (n)) or T (n) O(f (n))

CSE 5311 Saravanan Thirumuruganathan

• Limit based Perspective

T (n) is O(f (n)) if lim supn

T (n)f (n)

• Big-Omega Notation

Lower bounds: T (n) is (f (n)) if there exists constantsc > 0 and n0 0 such that T (n)c f (n) for all n n0

5

5From K&T: Algorithm DesignCSE 5311 Saravanan Thirumuruganathan

• Big-Omega Notation

Lower bounds: T (n) is (f (n)) if there exists constantsc > 0 and n0 0 such that T (n) c f (n) for all n n0

Example: T (n) = 32n2 + 17n + 1. Is T (n) in (n2)?

Yes! Use c = 32, n0 = 1

Simple Proof:

T (n) 32n2 + 17n + 1 32n2

cn2

c = 32 and n0 = 1

CSE 5311 Saravanan Thirumuruganathan

• Big-Theta Notation

Tight bounds: T (n) is (f (n)) if there exists constantsc1 > 0, c2 > 0 and n0 0 such that c1 f (n) T (n) c2 f (n)for all n n0

6

6From K&T: Algorithm DesignCSE 5311 Saravanan Thirumuruganathan

• Big-Theta Notation

Tight bounds: T (n) is (f (n)) if there exists constantsc1 > 0, c2 > 0 and n0 0 such that c1 f (n) T (n) c2 f (n)for all n n0

Example: T (n) = 32n2 + 17n + 1. Is T (n) in (n2)?

Yes! Use c1 = 32, c2 = 50 and n0 = 1

Combine proofs from before

Theorem: For any two functions f (n) and g(n), we havef (n) = (g(n)) if and only if f (n) = O(g(n)) and f (n) = (g(n))

CSE 5311 Saravanan Thirumuruganathan

• Limit based Definitions

Let lim supn

T (n)f (n) = c

If c 0 then T (n) is (f (n)) (also O(f (n)) and (f (n)))

If c = then T (n) is (f (n))

CSE 5311 Saravanan Thirumuruganathan

• Some Big-O tips

Big-O is one of the most useful things you will learn in thisclass!

Big-O ignores constant factors through c

Algorithm implemented in Python might need a larger c thanone implemented in C++

Big-O ignores small inputs through n0Simply set a large value of n0

Suppose T (n) is O(f (n)). Typically, T (n) is messy while f (n)is simple

T (n) = 32n2 + 17n + 1, f (n) = n2

Big-O hides constant factors. Some times using an algorithmwith worser Big-O might still be a good idea (e.g. sorting,finding medians)

CSE 5311 Saravanan Thirumuruganathan

• Survey of Running Times

Complexity Name ExampleO(1) Constant time Function that returns a constant

(say 42)

O(log n) Logarithmic Binary Search

O(n) Linear Finding Max of an array

O(n log n) Linearithmic Sorting (for e.g. Mergesort)

O(n3) Cubic Floyd-Warshall

O(nk) Polynomial Subset-sum with k elements

O(2n) Exponential Subset-sum with no cardinalityconstraints

CSE 5311 Saravanan Thirumuruganathan

• Dominance Rankings7

n! 2n n3 n2 n log n n log n 1Exponential algorithms are useless even at n >= 50

Quadratic algorithms at around n 1MO(n log n) at around n 1B

7Skiena lecture notesCSE 5311 Saravanan Thirumuruganathan

• Closer Look at T (n)

So far we assumed someone gave us T (n)

What is n? (Program Analysis)

How do we get T (n)? (Recurrences)

CSE 5311 Saravanan Thirumuruganathan

• Program Analysis

for i=1 to n

{

constant time

operations

}

for i=1 to n

{

for j=1 to n

{

constant time

operations

}

}

CSE 5311 Saravanan Thirumuruganathan

• Recurrences

Typically programs are lot more complex than that

Recurrences occur in recursion and divide and conquerparadigms

Specify running time as a function of n and running time overinputs of smaller sizes

Examples:

fibonacci(n) = fibonacci(n 1) + fibonacci(n 2)T (n) = 2T ( n2 ) + nT (n) = T (n 1) + n

CSE 5311 Saravanan Thirumuruganathan

• Solving Recurrences

Unrolling

Documents
Documents
Documents
Documents
Documents
Documents
Documents
Documents
Documents
Documents
Documents
Education
Documents
Documents
Documents
Documents
Documents
Documents
Documents
Documents
Documents
Documents
Documents
Documents
Documents
Documents
Documents
Documents
Documents
Documents