Lesson 1 .1 - Introduction

8/11/2019 Lesson 1 .1 - Introduction

1/115

Why Study Algorithms

CS 854 Advance Algorithm Analysis

Anis ur RahmanDepartment of Computing

NUST-SEECSIslamabad

September 10, 2014

1/115


2/115

Course Information

1 Course Information

2 Algorithm analysis


4 Big-O notation

2/115


3/115

Course Information Course Topics

This course is about

1 Vocabulary for design and analysis of algorithmsBig-Oh notationhigh level reasoning about algorithmsgeneral analysis methods (Master Method)

3/115


4/115




2 Divide and conquer algorithm design paradigm

applied to integer multiplication, sorting, matrix multiplication,closest pair, etc.

4/115


5/115






3 Randomization in algorithm design

applied to QuickSort, primality testing, graph partitioning, hashing.

5/115


6/115








4 Primitives for reasoning about graphs

connectivity information, shortest paths, structure of informationand social networks.

6/115

C I f i C T i


7/115








4 Primitives for reasoning about graphs

connectivity information, shortest paths, structure of informationand social networks.

5 Use and implementation of data structures

Heaps, balanced binary search trees, hashing, and many more

7/115

C I f ti C T i


8/115


Skills Youll Learn

1 Become a better programmer2 Sharpen your mathematical and analytical skills

3 Start thinking algorithmically

4 Literacy with computer sciences greatest hits

5 Ace your technical interviews

8/115



9/115


Who Are You?

1 Ideally, you know some programming.2 Doesnt matter which language(s) you know.

But you should be capable of translating high-level algorithmdescriptions into working programs in some programming language.

3 Some mathematical experience.

basic discrete math, proofs by induction, etc.Check out Mathematics for Computer Science, by Eric Lehman andTom Leighton.

9/115



10/115


Logistics

Class meets.

Monday 18:00 20:00

Thursdays 17:00 18:00

Web Site.

lms.nust.edu.pkE-mail.

[email protected]

10/115



11/115


Source Material

No required textbook. A few of the many good ones:

1 Cormen/Leiserson/Rivest/Stein, Introduction to Algorithms, 2009(3rd edition).

2 Kleinberg/Tardos, Algorithm Design, 2005.

3 Dasgupta/Papadimitriou/Vazirani, Algorithms, 2006.

4 Mehlhorn/Sanders, Data Structures and Algorithms: The BasicToolbox, 2008.

5 Web is the best and greenest textbook

No specific development environment required.

But you should be able to write and execute programs.

11/115



12/115


General information

Prerequisites.

CS110, CS250

Assessment.

5-6 Quizzes (15%).

2-3 Assignments (10%).Term paper (5%).

OHTs (30%).

ESE (40%).

12/115



13/115

p

A barrage of show-of-hands questions

13/115



14/115

Tiny assignment for today

Q. A good example problem for showing algorithms with different

orders of magnitude is the classic anagram detection problem forstrings. One string is an anagram of another if the second is simply a

rearrangement of the first. For example, heart and earth areanagrams; python and typhon are anagrams as well. For the sake of

simplicity, assume that the two strings in question are of equal lengthand are made up of the set of 26 lowercase alphabetic characters. Ourgoal is to write a boolean function that will take two strings and return

whether they are anagrams.

14/115

Algorithm analysis


15/115




4 Big-O notation

15/115

Algorithm analysis Algorithms


16/115

Algorithm: background

What is an Algorithm?

In mathematics, computer science, and related subjects, an algorithmis aneffectivemethod for solving a problem expressed as afinite

sequence of instructions.

16/115



17/115

Algorithm: background

What is an Algorithm?

In mathematics, computer science, and related subjects, an algorithmis aneffectivemethod for solving a problem expressed as afinite

sequence of instructions.

Abu Abdullah Muhammad bin Musa al-Khwarizmi

a Muslim mathematician.

well-known work: Al-Jabr.

invented algorithms to solve quadratic equations.

word algorithm derived by his Latin name Algorithmi.

17/115



18/115

Algorithm: question

Which one is an algorithm?

A recipe for making tomato soup?

A procedure to sort 1000 numbers?

A procedure to recognize a particular face in a crowd?

A procedure to judge beauty?

18/115



19/115

Algorithm: designing algorithms

Four steps:

What is a Problem?

Correctness. is whether the proposed solution works.

Efficiency.is the desired running time of the algorithm comparedto some benchmark.

How to express an Algorithm

19/115



20/115

Algorithm: solving a problem

Q. How to solve a new problem?

20/115



21/115

Algorithm: solving a problem

Q. How to solve a new problem?

use basic building blocks

to achieve correctness then to worry about efficiency

so simple only a genius could have thought of it"Albert Einstein

21/115


22/115



23/115

Algorithm: designing algorithms

Start with simple building blocksUse tested tools and techniques

Initial design should be as simple as possible

Worry about correctness

Then worry about efficiency

23/115


24/115


25/115



26/115

Algorithm analysis: problem

Find the eldest person six billion people.

a) We need to perform six billion operations?

b) We can do it by doing less than six billion operations by usingfast computers?

c) We can do it by doing about three billion operations by usingintelligent algorithms?

26/115



27/115

Algorithm: how to express one?

InputOutput

Basic idea in simple language

Complexity calculations

27/115


28/115



29/115

Algorithm: how to express one?

Precondition: An imageXof sizeWH

1 function StaticPathway(X)

2 X RetinalFilter(X)3 Mu,v GaborFilterBank(X

)4 Mu,v Interactions(Mu,v)

5 Mu,v Normalizations(Mu,v)6 Ms Fusion(Mu,v)

7 returnMs

29/115



30/115

Algorithm: problem

Input:An Array A of k numbers. A [1...k]. The firstk1 numbers are

sorted in ascending order, the last number is out of place.

1,3,7,9,13,17,23,4

Output: An arrayA [1...k], which is sorted in ascending order.

1,3,4,7,9,13,17,23

Q. Write the algorithm, verify its correctness.

30/115



31/115

Algorithm: moveMin

f o r i = k t o 1

if A[i] < A[i-1]

SWAP(A[i],A[i-1])

Q. Will it swap till the first element or stop?

Q. Is this enough for moveMin?

31/115


32/115



33/115

Algorithm: moveMin

f o r i = k t o 1

if A[i] < A[i-1]

SWAP(A[i],A[i-1])

Q. Will it swap till the first element or stop?

For correctness, it is important to check the entire arrayto be sure.Also, for simplicity of the algorithm.

Q. Is this enough for moveMin?

This is not a general algorithm to sort any unordered array. It is just abuilding block for the sorting algorithm.

33/115


S


34/115

Summary

Algorithms are a finite set of operations to solve a problem.Initial design of an algorithm should be:

Simple.using known concepts and principles.Clear.described in plain English, or simple notation.Correct. solve the problem.

34/115


35/115

Algorithm analysis Overview

Al ith l i b k d


36/115

Algorithm analysis: background

Why is a algorithm analysis important?

Computer scientists learn by experience.

Learn techniques to solve for one problem, and use it for another.

Algorithm design helps to solve more challenging problems.

36/115


Al ith l i b k d


37/115


Why is a algorithm analysis important?

Computer scientists learn by experience.

Learn techniques to solve for one problem, and use it for another.

Algorithm design helps to solve more challenging problems.

For example,Consider two solutions for one problem.

Both give the correct result.

One might be resource efficient compared to the other.

37/115


38/115


39/115


40/115




41/115


It is convenient to have a measure of the amount of work involved in a

computing process, even though it may be a very crude one. We may

count up the number of times that various elementary operations are

applied in the whole process..."

Alan Turing (1947)

41/115


42/115


43/115


44/115


Algorithm analysis: quantitative analysis


45/115

Algorithm analysis: quantitative analysis

What to use quantitative, or qualitative analysis.

As an engineer, for the proposed algorithm:one must determine the actual costs (memory, time, monetary).

comparison of qualitiesfaster, less memoryare unimportant.

45/115


Algorithm analysis: running time


46/115


Running time of the program is a functionT(N),

represents thenumber of units of timetaken by an algorithm onany input of sizeN, or

the totalnumber of statementsexecuted by the program.

For example, a program may have a running time T(N) =cN, wherecis

some constant.the program is linearly proportional to the size of the input on

which it is run.

running time is linear time, or just linear.

46/115


Algorithm analysis: empirical analysis


47/115

Algorithm analysis: empirical analysis

Q. How to time a program?

manual.using a stopwatch

automatic. using some timer function

Run the program for various input sizes and measure running time.

data size time (ms)

250 5500 8

1000 10

2000 154000 308000 5016000 75

47/115


48/115


Algorithm analysis: question


49/115

g y q

What are the different factor affecting therunning-time of an algorithm?

49/115




50/115

g y g

Different factors influencing the running time of an algorithm are:

processing power of computer

algorithm used

size of the input

structure of the input

amount of available memory

implementation language

50/115




51/115

g y g

We considerthree simplificationsto analyze algorithms:

without any implementation

taking the worst possible inputs

ignoring details

51/115




52/115


without any implementation => RAM model

taking the worst possible inputs => worst-case running time

ignoring details => Big-O or Landau notation

52/115




53/115


without any implementation=> RAM model

taking the worst possible inputs => worst-case running time


53/115

Algorithm analysis RAM model

RAM model: overview


54/115

Random access machine.

Considers a very simple computer.

Motivation is to analyze an algorithm ona minimal computermodel.

Q. What do you consider to be a minimal computer model for analysis?

54/115


RAM model: overview


55/115

A simple model to find how an algorithm will perform on a real

computer.

Mouse

Keyboard

Display

CD-ROM

Graphics processor

Working memory

Processor

Operating system

File system

Input/Output

Programming capability

CD-ROM

55/115


56/115


RAM model: question


57/115

Simple operations (+,-,*,/) take 1 timestep

Loops comprise a condition plus multiple operations in eachiteration

Read/write to memory is free

57/115


RAM model: question


58/115

1 a=12 b=2 *a

1 s=02 while s


59/115

1 a=12 b=2 *a

# of timesteps = 1

1 s=02 while s


60/115

1 a=12 b=2 *a

# of timesteps = 1

1 s=02 while s


61/115

We observe some problem with RAM,

memory access times depends whether from cache or disk.

unlimited memory is unrealistic.

a+ b a b.

However, itssimplicitymakes it an excellent tool to analyze algorithms.

61/115

Algorithm analysis Worst case running time



62/115

We consider three simplifications to analyze algorithms:

without any implementation => RAM model

taking the worst possible inputs=> worst-case running time


62/115


Worst case running time: overview


63/115

RAM simply counts operations for any input.

Worst case running timecombinesnotion of running time withstructure of input.

determinesgood or bad algorithmbased on all instances of inputs.

63/115


Worst case running time: question


64/115

Consider an example,

1 count = 02 f or c ha r in s tr :3 i f c ha r == 'a ':

4 co unt += 1

In the example,

N: length of string

A: number of as in strRunning time = N+ A+

64/115




65/115



4 co unt += 1

In the example,

N: length of string

A: number of as in strRunning time = 2N+ 1A+ 0 or 1

65/115


Worst case running time: the cases


66/115

Consider a sample problem, e.g. sorting (arranging items in order)

66/115


67/115


Worst case running time: the challenge


68/115

Actual data might not match input running time model?

Need to understand input to effectively process it.Two directions to follow:

Approach 1.design for the worst case.Approach 2.randomize, depend on probabilistic guarantee.

68/115


Worst case running time: the outcomes


69/115

Goals.

establishdifficultyof a problem.developoptimalalgorithms.

Approach.

suppress detailsin analysis; analyze to within a constant factor.

eliminate variabilityin input model by focusing on the worst case.Optimal algorithm.

performance guarantee (to within a constant factor) for any input.

no other algorithm can provide a better performance guarantee.

69/115




70/115

Q. Which one do you consider is used? And why?

70/115




71/115

Q. Which one do you consider is used? And why?

Best case is improbable.Average case is difficult to determine.

Worst-case only guarantees running times.

71/115




72/115


1 count = 02 f or c ha r in s tr :3 i f c ha r == 'a ':4 co unt += 1

In the example consider,N: length of string

A: number of as in str

Question?

Worst case running time= N+ A+

Best case running time= N+ A+

Average case running time=

72/115




73/115


1 count = 02 f or c ha r in s tr :3 i f c ha r == 'a ':4 co unt += 1

In the example consider,N: length of string

A: number of as in str

Question?

Worst case running time= 3N+ 0A+ 1

Best case running time= 2N+ 0A+ 1

Average case running time= ?

73/115


Worst case running time: the problem


74/115

Sample code counts ab substring in stringstr.

1 count = 02 fo r i i n r an ge ( N ):3 if s tr [ i] = = 'a ':

4 if

str[i+1] == 'b':5 co unt += 1

As a function grows larger,

counting exact running times becomes tedious.

determining complexities becomes difficult.To further simplify we use Big-O notation.

74/115


Algorithm analysis: running time (optional)


75/115

According to Donald E. Knuth in the book,The Art of Computer

Programming",Total running time = sum of cost frequency for all operations.

The mathematical model for analysis is defined as,

analyze the program to determine set of operations.

determine cost and frequency,frequency. depends on algorithm, input data.cost.depends on machine, compiler.

75/115


Algorithm analysis: the power law


76/115

The power law. a Nb

N is the size of the data.b is the slope, determines the order of growth.

a is a constant.

76/115




77/115

System independent factors, or frequencies.

Algorithm.

Input data.

System dependent factors, or costs.

Hardware: CPU, memory, cache, ...

Software: compiler, interpreter, garbage collector, ...

System: operating system, network, other apps, ...

77/115




78/115


Algorithm.

Input data.

determines exponent b

in power law.


Hardware: CPU, memory, cache, ...Software: compiler, interpreter, garbage collector, ...

System: operating system, network, other apps, ...

78/115



S


79/115


Algorithm.

Input data.


Hardware: CPU, memory, cache, ...Software: compiler, interpreter,

garbage collector, ...

System: operating system,

network, other apps, ...

determines constant ain power law.

79/115


Algorithm analysis: mathematical models

I i i l h i h i l d l f l i h


80/115

In principle, there exist accurate mathematical models for algorithm

analysis.In practice, such models,

use complicate formulas.

require advanced mathematics.

defined and used only for experts.

80/115



An example model could be


81/115

An example model could be,

T(N) =c1

A+ c2

B+ c3

C+ c4

D+ c5

E

A = array accessB= integer addC= integer compareD= incrementE= variable assignment

where,

costs[c1,c2,c3,c4,c5](depend on machine, compiler).

frequencies[A ,B,C,D,E](depend on algorithm, input).

81/115



An example model could be


82/115

An example model could be,

T(N) =c1

A+ c2

B+ c3

C+ c4

D+ c5

E

A = array accessB= integer addC= integer compareD= incrementE= variable assignment

where,

costs[c1,c2,c3,c4,c5](depend on machine, compiler).

frequencies[A ,B,C,D,E](depend on algorithm, input).

Much easier and cheaper to determine approximate running times,

can run huge number of experiments

However, it is difficult to get precise measurements.

In this course, we will use approximate models, i.e. for matrix multiplicationT(N) cN3.

82/115 Big-O notation


83/115




4 Big-O notation

83/115 Big-O notation Running time




84/115


without any implementation => RAM modeltaking the worst possible inputs => worst-case running time

ignoring details=> Big-O or Landau notation


Algorithm analysis: quadratic growth

Consider the two functions


85/115

Consider the two functions,

f(n) =n 2

g(n) =n 2 3n+ 2

Around n = 3, they look very different.



Around n = 10 they look similar


86/115

Around n = 10, they look similar.



Around n = 100 they are almost same


87/115

Around n = 100, they are almost same.

87/115


88/115

Big-O notation Running time


The absolute difference is large, such that,


89/115

The absolute difference is large, such that,

f(1000) = 1000000

g(1000) = 997002

but the relative diff

erence is very small,

f(1000)g(1000)

f(1000)

= 0.002998 < 0.3%

and this difference goes to zero asn .


Algorithm analysis: polynomial growth

To demonstrate with another example,


90/115

o e o s e o e e p e,

f(n) =n 6

g(n) =n 6 76n5 + 343n4 345n3 + 45n2 344n

Around n = 10, they are very different



Around n = 100, they are not similar.


91/115

, y



Around n = 1000, they seem similar.


92/115

, y



Both function pairs of polynomials are similar, they each had the same


93/115

leading term:

n2 in the first case

n6 in the second case

In each case, both functions exhibit the same rate of growth,

one is always proportionally larger than the other.

93/115 Big-O notation Asymptotic analysis

Big-O notation

Consider two algorithms to solve some problem,


94/115

Algorithm 13n2 + 2n+ 33 Algorithm 22n 5n+ 5

Here,n is the problem size.

Question

Which one will you prefer?


Big-O notation



95/115

Algorithm 13n2 + 2n+ 33 Algorithm 22n 5n+ 5


Question

Which one will you prefer?

When answering, you ignore all unnecessary details.

Onlyn 2 < 2n considered.


Big-O notation



96/115

Algorithm 13n2 + 2n+ 33f(n)

Algorithm 22n 5n+ 5g(n)


n0andc, wherec g(n0) f(n0) n >n0, wherec g(n) f(n)


Big-O notation



97/115

Algorithm 13n2 + 2n+ 33f(n)

Algorithm 22n 5n+ 5g(n)


n0andc, wherec g(n0) f(n0) n >n0, wherec g(n) f(n)

f(n) O(g(n)), orf(n) =O(g(n))It means the functionf(n)has growth rate no greater than g(n).


Big-O notation


98/115

Upper and lower bounds valid for n >n0smooth out the behavior of complexfunctions.


Big-O notation

Formally,


99/115

f(

n) =

O(

g(

n))means

c g(

n)is an upper bound on

f(

n). Thus,there exists some constantcsuch thatf(n)is always cg(n), for

large enough n (i.e. ,n n0for some constantn0).

f(n) =(g(n))meansc g(n)is a lower bound onf(n). Thusthere exists some constantcsuch thatf(n)is always cg(n), for

alln n0.f(n) = (g(n))meansc1 g(n)is an upper bound onf(n)andc2 g(n)is a lower bound onf(n), for alln n0. Thus there existconstantsc1andc2such thatf(n) c1 g(n)andf(n) c2 g(n).

This means thatg(n)provides a nice, tight bound onf(n).


Big-O notation


100/115


Big-O notation

f (n) g(n)


101/115

f(n) g(n)

g(n)grows at least as fast asf(n)


Big-O notation

f(n) g(n)


102/115

( ) g( )


5n+ 2 O(n)


Big-O notation

f(n) g(n)


103/115

( ) g( )


5n+ 2 O(n)12n2 10 O(n2)

103/115

Big-O notation Asymptotic analysis

Big-O notation

f(n) g(n)


104/115

( ) g( )


5n+ 2 O(n)12n2 10 O(n2)3n 3n8 + 23 O(3n )

104/115


Big-O notation

f(n) g(n)


105/115

( ) ( )


5n+ 2 O(n)12n2 10 O(n2)3n 3n8 + 23 O(3n )

3n n3 3n8 + 23 O(3n n3)

105/115


Big-O notation

f(n) g(n)


106/115


5n+ 2 O(n)=> 5n+ 2 O(n2)12n2 10 O(n2)=> 12n2 10 O(n3)3n 3n8 + 23 O(3n )

3n n3 3n8 + 23 O(3n n3)

However, it is preferable to keep the bound as tight as possible.

106/115


Big-O notation


107/115

The most common Big-O functions are given names:

O(1)=> constant

O(ln(n))=> logarithmic

O(n)=> linear

O(nln(n))=> log linear

O(n2)=> quadratic

O(n3)=> cubic

2

n ,en ,4

n

=> exponential

107/115


Big-O notation

Plot for common Big-O functions:


108/115

108/115


Big-O notation


109/115

Sample code counts ab substring in stringstr.


4 co unt += 1

Using Big-O simplification,

We require no counting of operations.

It can be simply written as,O(n), a linear algorithm.

109/115


Big-O notation: problem

Given the following code fragment, what is its Big-O running time?


110/115

1 test = 02 fo r i i n r an ge ( n ):3 fo r j i n r an ge ( n) :4 test = test + i * j

a)O

(n

)b)O(n2)c)O(logn)d)O(n3)

110/115



Given the following code fragment what is its Big-O running time?


111/115

1 test = 02 fo r i i n r an ge ( n ):3 test = test + 145 fo r j i n r an ge ( n ):6 test = test - 1

a)O(n)b)O(n2)c)O(logn)d)O(n3)

111/115



Given the following code fragment what is its Big-O running time?


112/115

1 i = n2 while i > 0 :3 k = 2 + 24 i = i / 25 }

a)O(n)b)O(n2)c)O(logn)d)O(n3)

112/115



Running time of a sorting algorithm.

Computer A:


113/115

p

Speed is one billion instructions per second.Uses Insertion Sort and requires 2n2 instructions to sort n numbers.

Computer B:

Speed is 10 million instructions per second.Uses Merge Sort and requires 50nlgninstructions.

113/115



Computer A takes.


114/115

2 (107)2instructions

1010instructions/second= 20,000secs(> 5.5hrs)

Computer B takes.

50 (107)lg107instructions

107instructions/second= 1,163secs(< 20mins)

Computer B runs 17 times faster than computer A,

using an algorithm that grows slowly.

even if uses a slow processor.

114/115


Summary

We will use Landau symbols to describe the complexity of algorithms.


115/115

e.g., adding a list of n numbers in linear time

(n)algorithm.An algorithm is said to have polynomial time complexity if its runtimemay be described byO(nd)for some fixedd 0.

We will consider such algorithms to be efficient.

Problems that have no known polynomial-time algorithms are said tobeintractable.

e.g. Traveling salesman problem. find the shortest path that visits

n cities.

Best run time: O(n2

2n

)In general, you dont want to implement exponential-time or

exponential-memory algorithms.

115/115

Documents

Lesson 1 .1 - Introduction