# Lecture 1: Asymptotics, Recurrences, Elementary Sortingsaravanan- · PDF fileLecture 1: Asymptotics, Recurrences, Elementary Sorting Instructor: Saravanan Thirumuruganathan CSE 5311

• View
216

0

Embed Size (px)

### Text of Lecture 1: Asymptotics, Recurrences, Elementary Sortingsaravanan- · PDF fileLecture 1:...

• Lecture 1: Asymptotics, Recurrences, ElementarySorting

Instructor: Saravanan Thirumuruganathan

CSE 5311 Saravanan Thirumuruganathan

• Outline

1 Introduction to Asymptotic Analysis

Rate of growth of functionsComparing and bounding functions: O,,Specifying running time through recurrencesSolving recurrences

2 Elementary Sorting Algorithms

Bubble, Insertion and Selection sortStability of sorting algorithms

CSE 5311 Saravanan Thirumuruganathan

• In-Class Quizzes

URL: http://m.socrative.com/Room Name: 4f2bb99e

CSE 5311 Saravanan Thirumuruganathan

http://m.socrative.com/
• Analyzing Algorithms

Time Complexity:Quantifies amount of time an algorithm needs to complete asa function of input size

Space Complexity:Quantifies amount of space an algorithm needs to completeas a function of input size

Function: Input size Vs {Time, Space}

CSE 5311 Saravanan Thirumuruganathan

• Analyzing Algorithms

Best Case Complexity:of an algorithm is the function that determines the minimumnumber of steps taken on any problem instance of size n

Worst Case Complexity:. . . maximum . . .

Average Case Complexity:. . . average . . .

Function: Input size Vs {Time, Space}

CSE 5311 Saravanan Thirumuruganathan

• Rate of Growth of Functions

Growth Function T (n)Input is positive integers n = 1, 2, 3, . . .

Asymptotically positive (returns positive numbers for large n)

How does T (n) grow when n grows?

n is size of input

T (n) is the amount of time it takes for an algorithm to solvesome problem

CSE 5311 Saravanan Thirumuruganathan

• Rate of Growth of Functions

CSE 5311 Saravanan Thirumuruganathan

• Quiz!

Question:You have a machine that can do million operations per second.

Suppose size of input is 1 million

How long does the algorithm takes for this input?

CSE 5311 Saravanan Thirumuruganathan

• Quiz!

Machine can do 1M operations per second

Running time = (1M)2

1M = 1M seconds

1M seconds = 1M606024 = Approximately 12 days

CSE 5311 Saravanan Thirumuruganathan

• Why does it matter?

Running time of different algorithms for various input sizes

1

1Table 2.1 from K&T Algorithm Design. Very long means it takes morethan 1025 years.

CSE 5311 Saravanan Thirumuruganathan

• Why does it matter?

The Big Data era

CSE 5311 Saravanan Thirumuruganathan

• Functions in Real World

This is how functions look in the real world!2

2Skiena Lecture notesCSE 5311 Saravanan Thirumuruganathan

• Solution: Analyze Asymptotic Behavior

Analyze the asymptotic behavior of algorithms

What happens to f (n) when n?

T (n) = 1000n T (n) = n2

n = 10 10K 100

n = 100 100K 10K

n = 1000 1M 1M

n = 10K 10M 100M

n = 100K 100M 10B

CSE 5311 Saravanan Thirumuruganathan

• Solution: Bound the functions

Identify known functions (such as n, n2, n3, 2n, . . .) that canbound T (n)How to bound? - asymptotic upper, lower and tight boundsFind a function f (n) such that T (n) is proportional to f (n)Why proportional (as against equal)?Ignore aspects such as programming language, programmercapability, compiler optimization, machine specification etc

CSE 5311 Saravanan Thirumuruganathan

• O,,3

3CLRS bookCSE 5311 Saravanan Thirumuruganathan

• Big-O Notation

Upper bounds: T (n) is O(f (n)) if there exists constantsc > 0 and n0 0 such that T (n) c f (n) for all n n0

4

4From K&T: Algorithm DesignCSE 5311 Saravanan Thirumuruganathan

• Big-O Notation

Upper bounds: T (n) is O(f (n)) if there exists constantsc > 0 and n0 0 such that T (n) c f (n) for all n n0

Example: T (n) = 32n2 + 17n + 1. Is T (n) in O(n2)?

Yes! Use c = 50, n0 = 1

Simple Proof:

T (n) 32n2 + 17n + 1 32n2 + 17n2 + 1n2

50n2

cn2

c = 50 and n0 = 1

Note: Not necessary to find the smallest c or n0

CSE 5311 Saravanan Thirumuruganathan

• Quiz!

Example: T (n) = 32n2 17n + 1. Is T (n) in O(n2)?

CSE 5311 Saravanan Thirumuruganathan

• Big-O Notation

Example: T (n) = 32n2 17n + 1. Is T (n) in O(n2)?Yes! Use c = 50, n0 = 1

Simple Proof:

T (n) 32n2 17n + 1 32n2+17n + 1 32n2 + 17n2 + 1n2

50n2

cn2

c = 50 and n0 = 1

CSE 5311 Saravanan Thirumuruganathan

• Quiz!

Example: T (n) = 32n2 17n + 1. Is T (n) in O(n3)?

CSE 5311 Saravanan Thirumuruganathan

• Big-O Notation

Example: T (n) = 32n2 17n + 1. Is T (n) in O(n3)?Yes! Use c = 50, n0 = 1

Simple Proof:

T (n) 32n2 17n + 1 32n2+17n + 1 32n2 + 17n2 + 1n2

50n2

50n3

cn3

c = 50 and n0 = 1

CSE 5311 Saravanan Thirumuruganathan

• Quiz!

Example: T (n) = 32n2 + 17n + 1. Is T (n) in O(n)?

CSE 5311 Saravanan Thirumuruganathan

• Big-O Notation

Example: T (n) = 32n2 + 17n + 1. Is T (n) in O(n)?

No!

32n2 + 17n + 1 c n

32n + 17 +1

n c

32n c (ignore constants for now)n c (ignore constants for now)

This inequality does not hold for n = c + 1!

CSE 5311 Saravanan Thirumuruganathan

• Set Theoretic Perspective

O(f (n)) is set of all functions T (n) where there exist positiveconstants c , n0 such that 0 T (n) c f (n) for all n n0Example: O(n2) = { n2, . . . , 32n2 + 17n + 1, 32n2 17n + 1,. . . , n, 2n, . . .}Notation: T (n) = O(f (n)) or T (n) O(f (n))

CSE 5311 Saravanan Thirumuruganathan

• Limit based Perspective

T (n) is O(f (n)) if lim supn

T (n)f (n)

• Big-Omega Notation

Lower bounds: T (n) is (f (n)) if there exists constantsc > 0 and n0 0 such that T (n)c f (n) for all n n0

5

5From K&T: Algorithm DesignCSE 5311 Saravanan Thirumuruganathan

• Big-Omega Notation

Lower bounds: T (n) is (f (n)) if there exists constantsc > 0 and n0 0 such that T (n) c f (n) for all n n0

Example: T (n) = 32n2 + 17n + 1. Is T (n) in (n2)?

Yes! Use c = 32, n0 = 1

Simple Proof:

T (n) 32n2 + 17n + 1 32n2

cn2

c = 32 and n0 = 1

CSE 5311 Saravanan Thirumuruganathan

• Big-Theta Notation

Tight bounds: T (n) is (f (n)) if there exists constantsc1 > 0, c2 > 0 and n0 0 such that c1 f (n) T (n) c2 f (n)for all n n0

6

6From K&T: Algorithm DesignCSE 5311 Saravanan Thirumuruganathan

• Big-Theta Notation

Tight bounds: T (n) is (f (n)) if there exists constantsc1 > 0, c2 > 0 and n0 0 such that c1 f (n) T (n) c2 f (n)for all n n0

Example: T (n) = 32n2 + 17n + 1. Is T (n) in (n2)?

Yes! Use c1 = 32, c2 = 50 and n0 = 1

Combine proofs from before

Theorem: For any two functions f (n) and g(n), we havef (n) = (g(n)) if and only if f (n) = O(g(n)) and f (n) = (g(n))

CSE 5311 Saravanan Thirumuruganathan

• Limit based Definitions

Let lim supn

T (n)f (n) = c

If c 0 then T (n) is (f (n)) (also O(f (n)) and (f (n)))

If c = then T (n) is (f (n))

CSE 5311 Saravanan Thirumuruganathan

• Some Big-O tips

Big-O is one of the most useful things you will learn in thisclass!

Big-O ignores constant factors through c

Algorithm implemented in Python might need a larger c thanone implemented in C++

Big-O ignores small inputs through n0Simply set a large value of n0

Suppose T (n) is O(f (n)). Typically, T (n) is messy while f (n)is simple

T (n) = 32n2 + 17n + 1, f (n) = n2

Big-O hides constant factors. Some times using an algorithmwith worser Big-O might still be a good idea (e.g. sorting,finding medians)

CSE 5311 Saravanan Thirumuruganathan

• Survey of Running Times

Complexity Name ExampleO(1) Constant time Function that returns a constant

(say 42)

O(log n) Logarithmic Binary Search

O(n) Linear Finding Max of an array

O(n log n) Linearithmic Sorting (for e.g. Mergesort)

O(n3) Cubic Floyd-Warshall

O(nk) Polynomial Subset-sum with k elements

O(2n) Exponential Subset-sum with no cardinalityconstraints

CSE 5311 Saravanan Thirumuruganathan

• Dominance Rankings7

n! 2n n3 n2 n log n n log n 1Exponential algorithms are useless even at n >= 50

Quadratic algorithms at around n 1MO(n log n) at around n 1B

7Skiena lecture notesCSE 5311 Saravanan Thirumuruganathan

• Closer Look at T (n)

So far we assumed someone gave us T (n)

What is n? (Program Analysis)

How do we get T (n)? (Recurrences)

CSE 5311 Saravanan Thirumuruganathan

• Program Analysis

for i=1 to n

{

constant time

operations

}

for i=1 to n

{

for j=1 to n

{

constant time

operations

}

}

CSE 5311 Saravanan Thirumuruganathan

• Recurrences

Typically programs are lot more complex than that

Recurrences occur in recursion and divide and conquerparadigms

Specify running time as a function of n and running time overinputs of smaller sizes

Examples:

fibonacci(n) = fibonacci(n 1) + fibonacci(n 2)T (n) = 2T ( n2 ) + nT (n) = T (n 1) + n

CSE 5311 Saravanan Thirumuruganathan

• Solving Recurrences

Unrolling ##### Asymptotic Notation, Recurrences: Substitution, sourav/Lecture-02.pdf Asymptotic Notation, Recurrences: Substitution, Iteration, Master Method Lecture 2 L2.2 Solving recurrences •The
Documents ##### Merge Sort and Recurrences - .Merge Sort and Recurrences . COSC 581, ... using merge sort. â€¢ Combine:
Documents ##### Asymptotic Notation, Recurrences: Substitution, sourav/Lecture-02.pdfAsymptotic Notation, Recurrences: Substitution, Iteration, Master Method Lecture 2 L2.2 Solving recurrences â€¢The
Documents ##### STATISTICAL ASYMPTOTICS anjali/apts/statasymp/statasymp_ STATISTICAL ASYMPTOTICS This is a commentary on the APTS module ‘Statistical Asymptotics’. Please notify the author of
Documents ##### Solving Recurrences - York · PDF file called solving the recurrence. The goal of these notes is to look at ways of solving recurrences using an elementary approach. Elementary does
Documents ##### Solving recurrences - mlavrov/arml/13-14/recurrences-02-02-14.pdf Feb 02, 2014  · Warm-upSimple methodsLinear recurrences Exercises Solutions: # 1 We solve x n by the method of sums
Documents ##### Recurrences and Tridiagonal solvers: From Classical Tricks ... · PDF file Recurrences and Tridiagonal solvers: From Classical Tricks to Parallel Treats ... Recurrences and Tridiagonal
Documents ##### Solving recurrences - · PDF file Solving recurrences DenisTRYSTRAM LecturenotesMathsforComputerScience–MOSIG1–2017 1 Recall of the principle MathematicalinductionisusedforprovingthatastatementP(n
Documents Documents