1
Growth of Functions
CS 202
Epp, section ???
Aaron Bloomfield
2
How does one measure algorithms
• We can time how long it takes a computer– What if the computer is doing other things?– And what happens if you get a faster computer?
• A 3 Ghz Windows machine chip will run an algorithm at a different speed than a 3 Ghz Macintosh
• So that idea didn’t work out well…
3
How does one measure algorithms
• We can measure how many machine instructions an algorithm takes– Different CPUs will require different amount of
machine instructions for the same algorithm
• So that idea didn’t work out well…
4
How does one measure algorithms
• We can loosely define a “step” as a single computer operation– A comparison, an assignment, etc.– Regardless of how many machine instructions it
translates into
• This allows us to put algorithms into broad categories of efficient-ness– An efficient algorithm on a slow computer will always
beat an inefficient algorithm on a fast computer
5
Bubble sort running time
• The bubble step take (n2-n)/2 “steps”
• Let’s say the bubble sort takes the following number of steps on specific CPUs:– Intel Pentium IV CPU: 58*(n2-n)/2– Motorola CPU: 84.4*(n2-2n)/2– Intel Pentium V CPU: 44*(n2-n)/2
• Notice that each has an n2 term– As n increases, the other terms will drop out
6
Bubble sort running time
• This leaves us with:– Intel Pentium IV CPU: 29n2
– Motorola CPU: 42.2n2
– Intel Pentium V CPU: 22n2
• As processors change, the constants will always change– The exponent on n will not
• Thus, we can’t care about the constants
7
An aside: inequalities
• If you have a inequality you need to show:x < y
• You can replace the lesser side with something greater:x+1 < y
• If you can still show this to be true, then the original inequality is true
• Consider showing that 15 < 20– You can replace 15 with 16, and then show that 16 < 20.
Because 15 < 16, and 16 < 20, then 15 < 20
8
An aside: inequalities
• If you have a inequality you need to show:x < y
• You can replace the greater side with something lesser:x < y-1
• If you can still show this to be true, then the original inequality is true
• Consider showing that 15 < 20– You can replace 20 with 19, and then show that 15 < 19.
Because 15 < 19, and 19 < 20, then 15 < 20
9
An aside: inequalities
• What if you do such a replacement and can’t show anything?– Then you can’t say anything about the original
inequality
• Consider showing that 15 < 20– You can replace 20 with 10– But you can’t show that 15 < 10– So you can’t say anything one way or the other about
the original inequality
10
Review of last time
• Searches– Linear: n steps– Binary: log2 n steps– Binary search is about as fast as you can get
• Sorts– Bubble: n2 steps– Insertion: n2 steps– There are other, more efficient, sorting techniques
• In principle, the fastest are heap sort, quick sort, and merge sort• These each take take n * log2 n steps• In practice, quick sort is the fastest (although this is not
guaranteed!), followed by merge sort
11
Big-Oh notation
• Let b(x) be the bubble sort algorithm• We say b(x) is O(n2)
– This is read as “b(x) is big-oh n2”– This means that the input size increases, the running time of the
bubble sort will increase proportional to the square of the input size
• In other words, by some constant times n2
• Let l(x) be the linear (or sequential) search algorithm• We say l(x) is O(n)
– Meaning the running time of the linear search increases directly proportional to the input size
12
Big-Oh notation
• Consider: b(x) is O(n2)– That means that b(x)’s running time is less than (or
equal to) some constant times n2
• Consider: l(x) is O(n)– That means that l(x)’s running time is less than (or
equal to) some constant times n
13
Big-Oh proofs• Show that f(x) = x2 + 2x + 1 is O(x2)
– In other words, show that x2 + 2x + 1 ≤ c*x2
• Where c is some constant• For input size greater than some x
• We know that 2x2 ≥ 2x whenever x ≥ 1• And we know that x2 ≥ 1 whenever x ≥ 1• So we replace 2x+1 with 3x2
– We then end up with x2 + 3x2 = 4x2
– This yields 4x2 ≤ c*x2
• This, for input sizes 1 or greater, when the constant is 4 or greater, f(x) is O(x2)
• We could have chosen values for c and x that were different
14
Big-Oh proofs
15
Sample Big-Oh problems
• Show that f(x) = x2 + 1000 is O(x2)– In other words, show that x2 + 1000 ≤ c*x2
• We know that x2 > 1000 whenever x > 31– Thus, we replace 1000 with x2
– This yields 2x2 ≤ c*x2
• Thus, f(x) is O(x2) for all x > 31 when c ≥ 2
16
Sample Big-Oh problems
• Show that f(x) = 3x+7 is O(x)– In other words, show that 3x+7 ≤ c*x
• We know that x > 7 whenever x > 7– Duh!– So we replace 7 with x– This yields 4x ≤ c*x
• Thus, f(x) is O(x) for all x > 7 when c ≥ 4
17
A variant of the last question
• Show that f(x) = 3x+7 is O(x2)– In other words, show that 3x+7 ≤ c*x2
• We know that x > 7 whenever x > 7– Duh!– So we replace 7 with x– This yields 4x < c*x2
– This will also be true for x > 7 when c ≥ 1
• Thus, f(x) is O(x2) for all x > 7 when c ≥ 1
18
What that means
• If a function is O(x)– Then it is also O(x2)– And it is also O(x3)
• Meaning a O(x) function will grow at a slower or equal to the rate x, x2, x3, etc.
19
Function growth rates• For input size n = 1000
• O(1) 1• O(log n) ≈10• O(n) 103
• O(n log n) ≈104
• O(n2) 106
• O(n3) 109
• O(n4) 1012
• O(nc) 103*c c is a consant• 2n ≈10301
• n! ≈102568
• nn 103000
Many interesting problems fall into these categories
20
Function growth rates
Logarithmic scale!
21
Integer factorization
• Factoring a composite number into it’s component primes is O(2n)– Where n is the number of bits in the number
• This, if we choose 2048 bit numbers (as in RSA keys), it takes 22048 steps– That’s about 10617 steps!
22
Formal Big-Oh definition
• Let f and g be functions. We say that f(x) is O(g(x)) if there are constants c and k such that
|f(x)| ≤ C |g(x)|
whenever x > k
23
Formal Big-Oh definition
24
A note on Big-Oh notation
• Assume that a function f(x) is O(g(x))
• It is sometimes written as f(x) = O(g(x))– However, this is not a proper equality!– It’s really saying that |f(x)| ≤ C |g(x)|
• In this class, we will write it as f(x) is O(g(x))
25
NP Completeness
• A full discussion of NP completeness takes 3 hours for somebody who already has a CS degree– We are going to do the 15 minute version of it today– More will (probably) follow on Friday
• Any term of the form nc, where c is a constant, is a polynomial– Thus, any function that is O(nc) is a polynomial-time function– 2n, n!, nn are not polynomial functions
26
Satisfiability
• Consider a Boolean expression of the form:• (x1 x2 x3) (x2 x3 x4) (x1 x4 x5)
– This is a conjunction of disjunctions
• Is such an equation satisfiable?– In other words, can you assign truth values to all the
xi’s such that the equation is true?– The above problem is easy (only 3 clauses of 3
variables) – set x1, x2, and x4 to true• There are other possibilities: set x1, x2, and x5 to true, etc.
– But consider an expression with 1000 variables and thousands of clauses
27
Satisfiability
• If given a solution, it is easy to check if such a solution works– Plug in the values – this can be done quickly, even by hand
• However, there is no known efficient way to find such a solution– The only definitive way to do so is to try all possible values for
the n Boolean variables– That means this is O(2n)!– Thus it is not a polynomial time function
• NP stands for “Not Polynomial”
• Cook’s theorem (1971) states that SAT is NP-complete– There still may be an efficient way to solve it, though!
28
NP Completeness
• There are hundreds of NP complete problems– It has been shown that if you can solve one of them
efficiently, then you can solve them all– Example: the traveling salesman problem
• Given a number of cities and the costs of traveling from any city to any other city, what is the cheapest round-trip route that visits each city once and then returns to the starting city?
• Not all algorithms that are O(2n) are NP complete– In particular, integer factorization (also O(2n)) is not
thought to be NP complete
29
NP Completeness
• It is “widely believed” that there is no efficient solution to NP complete problems– In other words, everybody has that belief
• If you could solve an NP complete problem in polynomial time, you would be showing that P = NP– And you’d get a million dollar prize (and lots of fame!)
• If this were possible, it would be like proving that Newton’s or Einstein’s laws of physics were wrong
• In summary: – NP complete problems are very difficult to solve, but easy to
check the solutions of– It is believed that there is no efficient way to solve them
Recommended