Upload
chengwei-dai
View
223
Download
0
Embed Size (px)
Citation preview
8/12/2019 02 Complexity Analysis
http://slidepdf.com/reader/full/02-complexity-analysis 1/28
Lecture 2:
Complexity Analysis
EECS 281: Data Structures & Algorithms
8/12/2019 02 Complexity Analysis
http://slidepdf.com/reader/full/02-complexity-analysis 2/28
Assignments
• First reading assignment (now) – CLRS chapter 1 (short)
– Link to textbook available on CTools
8/12/2019 02 Complexity Analysis
http://slidepdf.com/reader/full/02-complexity-analysis 3/28
8/12/2019 02 Complexity Analysis
http://slidepdf.com/reader/full/02-complexity-analysis 4/28
Input Size versus Runtime
• Rate of growth independent of most factors – CPU speed, compiler, etc.
• Does doubling input size mean doubling runtime?
• Will a “fast” algorithm still be “fast” on large inputs?
Input Size (n)
Runtime(s)
How do we measure input size?
8/12/2019 02 Complexity Analysis
http://slidepdf.com/reader/full/02-complexity-analysis 5/28
Measuring & Using Input Size
• Number of bits – In an int, a double? (32? 64?)
• Number of items: what counts as an item? – Array of integers? One integer? One digit? …
– One string? Several strings? A char?
• Notation and terminology – n = input Size
– f(n) = max number of stepstaken by an algorithmwhen input has length n
– O(f(n)) or Big-Oh of f(n)• upper bounds up to constants (a whole topic in itself)
8/12/2019 02 Complexity Analysis
http://slidepdf.com/reader/full/02-complexity-analysis 6/28
Input Size Example
What should we measure?
• Vertices?
• Edges?
• Vertices and Edges?
Graph G = <V, E>:V = 5 Vertices
E = 6 Edges
When in doubt, measureinput size in bits
n = V + E
Using V and E tells which contributes more to the total number of steps
Examples: ElogV, EV, V2logE
8/12/2019 02 Complexity Analysis
http://slidepdf.com/reader/full/02-complexity-analysis 7/28
Examples of f(n) Runtime
0
5
10
15
20
25
0 1 2 3 4 5 6 7 8 9 10
1log n
nn log nn2
2n
8/12/2019 02 Complexity Analysis
http://slidepdf.com/reader/full/02-complexity-analysis 8/28
What Counts as One Stepin a Program ?
Primitive operations
• a) Variable assignment
• b) Arithmetic operation• c) Comparison
• d) Array indexing or pointer reference
• e) Function call (not counting the data)
• f) Function return (not counting the data)
Runtime of 1 step is independent on input
8/12/2019 02 Complexity Analysis
http://slidepdf.com/reader/full/02-complexity-analysis 9/28
int fun(int n)int sum =0;
for (int i=0; i<n; i++)sum += i;return sum;
int fun1(int n)
int sum =0;
for (int i=0; i<n; i++)
for (int j=0; j<n; j++)
sum++;
for (int k=0; k<n; k++)
sum--;
return sum;
1 step
1 + n *(2 steps)
1 step
1 step
Total steps: 4 + 3n
1 step
1 +1+ n *(2 steps)
1 step
1 +1+ n * (2steps)
1 step
Total steps: 3n^2 + 7n + 6
1 +1 + n * (2steps)
1 step
Counting Steps
8/12/2019 02 Complexity Analysis
http://slidepdf.com/reader/full/02-complexity-analysis 10/28
Algorithm Exercise
• How many multiplications will it take if size = n
//REQUIRES: in and out are arrays with size elements
//MODIFIES: out
//EFFECTS: out[i] = in[0] *...* in[i-1] *
// * in[i+1] *...* in[size-1]
void f(int *out, const int *in, int size);
void f(int *out, const int *in, int size)
for (int i=0; i<size; ++i)
out[i] = 1;
for (int j=0; j<size; ++j)
if (i == j) continue;
out[i] *= in[j];
10
8/12/2019 02 Complexity Analysis
http://slidepdf.com/reader/full/02-complexity-analysis 11/28
Algorithm Exercise
• How many multiplications and divisions, if size = n?
void f(int *out, const int *in, int size)
int product = 1;
for (int i=0; i<size; ++i)
product *= in[i];
for(int i=0; i<size; ++i)
out[i] = product / in[i];
11
8/12/2019 02 Complexity Analysis
http://slidepdf.com/reader/full/02-complexity-analysis 12/28
Big-Oh - Definition 1
f(n) = O(g(n)) if and only if there are constants
such that f(n) ≤ c g(n) whenever n ≥ n0
f(n) = n
g(n) = n2
n0 = 1
c > 0n0 ≥ 0
Is n = O(n2)?
0
0.5
1
1.5
2
2.5
0 1
8/12/2019 02 Complexity Analysis
http://slidepdf.com/reader/full/02-complexity-analysis 13/28
Big-Oh: Sufficient (but not necessary) Condition
O(g(n))isf(n)thendg(n)
f(n)
If limn
log2n = O(2n)? lim
limf(n) = log2n
g(n) = 2n
Use L’Hopital’s Rule
log2n = O(2n)
f(n) =
g(n) = 100
lim Condition does not hold butit is true that f(n) = O(g(n))
:
:
:
:
O(100)?100nsin
100
nsin
100
sin100
n
n
n
2n
nlog
2n
1
d0
/
n
8/12/2019 02 Complexity Analysis
http://slidepdf.com/reader/full/02-complexity-analysis 14/28
Big-Oh: Can We Drop Constants?
2n
76nlimn
3n2 + 7n + 42 = O(n2)?
f(n) = 3n2 + 7n + 42g(n) = n2
c = 5
g(n) = 5n2
Definition
n0
Sufficient Condition
2
limn
6
2
2
n n
427n3nlim
dg(n)f(n)lim
n
8/12/2019 02 Complexity Analysis
http://slidepdf.com/reader/full/02-complexity-analysis 15/28
Common Orders of Functions
Notation Name
O(1) Constant
O(log n) LogarithmicO(n) Linear
O(n log n) Loglinear
O(n2) Quadratic
O(cn) Exponential
O(n!) Factorial
Doubly Exponentialn
22
8/12/2019 02 Complexity Analysis
http://slidepdf.com/reader/full/02-complexity-analysis 16/28
Rules of Thumb
1. Lower-order terms can be ignored
– n2 + n + 1 = O(n2)
– n2 + log(n) + 1 = O(n2)
2. Coefficient of the highest-order term canbe ignored
– 3n2 + 7n + 42 = O(n2)
8/12/2019 02 Complexity Analysis
http://slidepdf.com/reader/full/02-complexity-analysis 17/28
Log Identities
Identity Example
loga(xy) = logax + logay log2(12) =
loga(x/y) = logax-logay log2(4/3) =
loga(xr ) = r logax log28 =
loga(1/x) = -logax log21/3 =
log79 =
logaa = ?
loga1 = ?
Identity Example
a(n+m) = anam 25 =
a(n-m) = an/am 23-2 =
(a(n))m = anm (22)3 =
2-4 =
a-1= ?
a0 = ?
a1 = ?
Power Identities
aln
xln
alog
xlogxloga
n
n
a
1a
8/12/2019 02 Complexity Analysis
http://slidepdf.com/reader/full/02-complexity-analysis 18/28
Exercise
True or false?10100 = O(1)
3n4 + 45n3 = O(n4)
3n = O(2n)
2n = O(3n)
45log(n) + 45n = O(log(n))
log(n2) = O(log(n))
[log(n)]2 = O(log(n))
Find f(n) and g(n), suchthat f(n)≠O(g(n)) and
g(n)≠O(f(n))
8/12/2019 02 Complexity Analysis
http://slidepdf.com/reader/full/02-complexity-analysis 19/28
Big-Oh (O) Big-Omega (Ω)
Defines Asymptotic upperbound
Asymptotic tight bound Asymptotic lower bound
Definition f(n) = O(g(n)) if and only
if there exists an integer
n0 and a real number c
such that for all n ≥ n0,f(n) ≤ c·g(n)
f(n) = Θ(g(n) if and only if
there exists an integer n0
and real constants c1 and c2
such that for all n ≥ n0:
c1·g(n) ≤ f(n) ≤ c2·g(n)
f(n) = Ω(g(n)) if and only if
there exists an integer n0 and
a real number c such that for
all n ≥ n0, f(n) ≥ c·g(n)
Mathematical
Definition
f 1(n)=2n +1 O(n) or O(n2) or O(n3)
…
Ω(n) or Ω(1)
f 2(n)=n2 +n +5 O(n2) or O(n3) … Ω(n2) or Ω(n) or Ω(1)
Big-Oh, Big-Theta, and Big-Omega
g(n)cf(n),nn
:RcN,n
0
0
g(n)cf(n),nn
:RcN,n
0
0
θ(n)
)θ(n2
))(( n f
O(f(n))
θ(f(n))
)(θTheta
8/12/2019 02 Complexity Analysis
http://slidepdf.com/reader/full/02-complexity-analysis 20/28
8/12/2019 02 Complexity Analysis
http://slidepdf.com/reader/full/02-complexity-analysis 21/28
Additional Examples of O(log n) Timeunsigned logB(unsigned n)
// find binary log, round upunsigned r =0;while(n>1) n=n/2, r++;return r ;
unsigned ctz(unsigned n)//count trailing zeros in binary
if (n==0) return 0;unsigned r =0;while (n%2==0) n=n/2,r++;return r ;
unsigned onecount (unsigned n)//count nonzero bits
unsigned r=0;for( ; n; n = n & (n-1)) r++;return r ;
int* bsearch (int* lo, int* hi, int val)//find position of val between a,b
while (hi>=lo) int* mid=lo+(hi-lo)/2;if (*mid<val) lo=mid+1;
else if (*mid>val) hi=mid-1;else return mid;
return NULL;
// how can this code be optimized ?
8/12/2019 02 Complexity Analysis
http://slidepdf.com/reader/full/02-complexity-analysis 22/28
Complexity Analysis• What is it?
– Each step should take O(1) time
– Given an algorithm and input size n,how many steps are needed?
– As input size grows, how does number of steps change?• Focus is on TREND
• How do we measure it? – Express the rate of grown as a function f(n)
– Use the big-Oh notation
• Why do we care? – Tells us how well an algorithm scales to larger inputs
– Given two algorithms, we can compare performancebefore implementation
8/12/2019 02 Complexity Analysis
http://slidepdf.com/reader/full/02-complexity-analysis 23/28
Metrics of Algorithm Complexity
• Best-Case – Least number of steps required, given ideal input – Analysis performed over inputs of a given size – Example: Data is found in the first place you look
• Worst-Case
– Most number of steps required, given hard input – Analysis performed over inputs of a given size – Example: Data is found in the last place you could possibly look
• Average-Case – Average number of steps required, given any input – Average performed over all possible inputs of a given size
Array of
n items
Using a linear search over n items,how many steps will it take to find item x?
Best-Case: 1 stepWorst-Case: n steps
Average-Case: n/2 steps
8/12/2019 02 Complexity Analysis
http://slidepdf.com/reader/full/02-complexity-analysis 24/28
Amortized Complexity
• A type of worst-case complexity• Analysis performed over a sequence of inputs of a given size
– The sequence is selected to be a worst case
• Considers the average cost of one step over a sequence of operations – Best/Worst/Average-case only consider a single operation – Different from average-case complexity!
• Key to understanding expandable arrays and STL’s vector class,STL’s implementations of stacks, queues, priority queues, hash tables
• Example: pre-paid telephone cards – Pay $20 upfront and call many times, until $20 is exhausted – Amortizes to, say, 10c per minute (then recharge with another $20) – Better than paying for each call at international rates – Worst-case sequences of calls: any sequence that exhausts $20 – Sequences that do not require recharge are not worst-case sequences
8/12/2019 02 Complexity Analysis
http://slidepdf.com/reader/full/02-complexity-analysis 25/28
From Analysis to Application
• Algorithm comparisons independent of hardware,compilers and implementation tweaks
• Predict which algorithms will eventually be faster – For large enough inputs
– O(n2) time algorithms will take longer than O(n) algorithms
• Constants can often be ignored becausethey do not affect asymptotic comparisons – Algorithm with 20n steps runs faster than
algorithm with 10n steps. Why?
20n10n
8/12/2019 02 Complexity Analysis
http://slidepdf.com/reader/full/02-complexity-analysis 26/28
Exercise
• You have N balls. All have equal weight,except for one which is heavier. Find theheavy ball using only a balance.
• Describe an O(n2) algorithm
• Describe an O(n) algorithm
• Describe an O(logn) algorithm
• Describe another O(logn) algorithm
28
8/12/2019 02 Complexity Analysis
http://slidepdf.com/reader/full/02-complexity-analysis 27/28
Two O(logn) solutions
• True or false? Why?
• log3(n) = O(log2n)
• log2(n) = O(log3n)
8/12/2019 02 Complexity Analysis
http://slidepdf.com/reader/full/02-complexity-analysis 28/28
Job Interview Question
• Implement this function
//returns x^y
int power(int x, unsigned int y);
• The obvious solution uses y multiplications – 28 = 2*2* ... *2
• Less obvious: O(log n) multiplications – Hint: 28 = ((22)2)2
– Does it work for 27 ?
• Write two solutions