Theory of Algorithms:Theory of Algorithms:IntroductionIntroduction
Theory of Algorithms:Theory of Algorithms:IntroductionIntroduction
James Gain and Edwin Blake{jgain | edwin} @cs.uct.ac.za
Department of Computer Science
University of Cape Town
August - October 2004
ObjectivesObjectives
To define an algorithm To introduce:
Problem types The Process of Algorithm Design Solution strategies Ways of Analysing Algorithms
To cover the structure of the course, including practicals
DefinitionsDefinitions
An algorithm is a sequence of unambiguous instructions for solving a problem For obtaining a required output for any legitimate input
in a finite amount of time
Does not require implementation in software
Not an answer but a method for deriving an answer
Historical Perspective: Named after Muhammad ibn Musa al-Khwarizmi – 9th century
mathematician
www.lib.virginia.edu/science/parshall/khwariz.html
Notion of algorithmNotion of algorithm
“computer”
problem
algorithm
input output
• Each step of the algorithm must be unambiguous
• The range of inputs must be specified carefully
• The same algorithm can be represented in different ways
• Several algorithms for solving the same problem may exist - with different properties
What is an algorithm?What is an algorithm?
Recipe, process, method, technique, procedure, routine,… with following requirements:
1. Finiteness Terminates after a finite number of steps
2. Definiteness Rigorously and unambiguously specified
3. Input Valid inputs are clearly specified
4. Output Can be proved to produce the correct output given a valid input
5. Effectiveness Steps are sufficiently simple and basic
Example: SortingExample: Sorting
Statement of problem: Input: A sequence of n numbers <a1, a2, …, an>
Output: A reordering of the input sequence <a´1, a´
2, …, a´
n> so that a´i ≤ a´
j whenever i < j
Instance: The sequence <5, 3, 2, 8, 3>
Algorithms:- Selection sort
- Insertion sort
- Merge sort
- (many others)
Selection SortSelection Sort
Input: array a[1],..,a[n] Output: array a sorted in non-decreasing order Algorithm:
for i=1 to n swap a[i] with smallest of a[i],…,a[n]
for i1 to n domin ifor j i+1 to n do
if a[j] < a[min] min jswap a[i] and a[min]
Exercise: Bridge PuzzleExercise: Bridge Puzzle
Problem: 4 People want to cross a bridge. You have 17 minutes to get them
across
Constraints: It is night and you have 1 flashlight. Max of 2 on the bridge at one
time. All start on the same side
Those crossing must have the flashlight with them. The flashlight must be walked back and forth (no throwing)
People walk at different speeds: person A = 1 minute to cross, person B = 2 minutes, person C = 5 minutes, person D = 10 minutes
A pair walks at the speed of the slower person’s pace
Rumour: this problem is given to Microsoft interviewees
Solution: Bridge PuzzleSolution: Bridge Puzzle
Start (0 min): A B C D
AB Across (2 min): A B C D
A Back (1 min): B A C D
CD Across (10 min): B C D A
B Back (2 min): C D A B
AB Across (2 min): A B C D Total Time = 17 minutes
Extension ExerciseExtension Exercise
This is an instance of a problem. How would you generalise it?
Can you derive an algorithm to solve this generalised problem? Must show the sequence of moves
Must output the minimum time required for crossing
Are there any special cases to watch out for?
Are there any constraints on the input?
Extension SolutionExtension Solution
Input: a list a of crossing times for n people, numbered 1, …, n
Output: total time to cross
Strategy: use 1 & 2 as shuttles and send the others across in pairs
for i 2 to n/2 dot a[2] // 1 & 2 acrosst t + a[1] // 1 backt t + a[i*2] // i*2 & (i*2)-1 acrosst t + a[2] // 2 back
t a[2] // 1 & 2 acrossreturn t
Extension ProblemsExtension Problems
This is an inadequate solution It falsely assumes certain inputs List may not be sorted in ascending order
Sort a
n may not be even numbered Alter final iteration of loop
n > 3 not guaranteed Special case for n = 1, 2, 3
Is not optimal for all inputs, e.g. 1, 20, 21, 22 Can you quantify the nature of these inputs? Suggest an
alternative.
Final solution is left as an exercise. Attempt to make your solution elegant
Fundamentals of Algorithmic Problem Fundamentals of Algorithmic Problem SolvingSolving
Understanding the Problem Make sure you are solving the correct problem and for all legitimate
inputs
Ascertaining the Capabilities of a Computational Device Sequential vs. Parallel.
What are the speed and memory limits?
Choosing between exact and approximate Problem Solving Is absolute precision required? Sometimes this may not be possible
Deciding on Appropriate Data Structures Algorithms often rely on carefully structuring the data
Fundamental Data Structures: array, linked list, stacks, queues, heaps, graphs, trees, sets
Fundamentals of Algorithm DesignFundamentals of Algorithm Design
Applying an Algorithm Design Technique Using a general approach to problem solving that is applicable to a
variety of problems
Specifying the Algorithm Pseudocode is a mixture of natural language and programming
constructs that has replaced flowcharts
Proving an Algorithms Correctness Prove that an algorithm yields a required result for legitimate inputs
in finite time
Analyzing an Algorithm Consider time efficiency, space efficiency, simplicity, generality,
optimality Analysis can be empirical or theoretical
Coding an Algorithm
Well known Computational ProblemsWell known Computational Problems
Sorting Searching String Processing
String Matching
Graph Problems Graph Traversal, Shortest Path, Graph Colouring
Combinatorial Problems Find a combinatorial object - permutation, combination, subset - subject to
constraints
Geometric Problems Closest-Pair, Convex-Hull
Numerical Problems Solving systems of equations, computing definite integrals, evaluating
functions, etc.
Algorithm Design StrategiesAlgorithm Design Strategies
Brute force A straightforward approach to solving a problem, usually
directly based on the problem’s statement
Divide and conquer Divide a problem into smaller instances, solve smaller
instances (perhaps recursively), combine
Decrease and conquer Exploit relationship between the problem and a smaller
instance reduced by some factor (often 1)
Transform and conquer Transform the problem to a simpler instance, another
representation or an instance with a known solution
More Algorithm Design StrategiesMore Algorithm Design Strategies
Greedy approach Make locally optimal steps which (hopefully) lead to a
globally optimal solution for an optimization problem
Dynamic programming Technique for solving problems with overlapping sub-
domains
Backtracking and Branch and bound A way of tackling difficult optimization and combinatorial
problems without exploring all state-space
Space and time tradeoffs Preprocess the input and store additional information to
accelerate solving the problem
How to Solve It: How to Solve It: Understanding the ProblemUnderstanding the Problem
Taken from G. Polya, “How to Solve It”, 2nd edition. A classic textbook on problem solving for mathematics
1. You have to understand the problem. What is the unknown? What are the data? Is the
problem statement sufficient, redundant, contradictory
Draw a figure. Introduce suitable notation
Separate the various parts of the problem. Can you write them down?
Devising a PlanDevising a Plan Find the connection between the data and the unknown. You may be
obliged to consider auxiliary problems if an immediate connection cannot be found. You should obtain eventually a plan of the solution.
Have you seen it before? Or have you seen the same problem in a slightly different form?
Do you know a related problem? Do you know a theorem that could be useful?
Look at the unknown! And try to think of a familiar problem having the same or a similar unknown.
Could you restate the problem? Could you restate it still differently? Go back to definitions.
If you cannot solve the proposed problem try to solve first some related problem. Are the unknown and the new data nearer to each other?
Did you use all the data? Did you use the whole condition? Have you taken into account all essential notions involved in the problem?
Carrying it ThroughCarrying it Through
3. Carry out the Plan Carrying out your plan of the solution, check each
step. Can you see clearly that the step is correct? Can you prove that it is correct?
4. Looking Back Can you check the result? Can you check the
argument?
Can you derive the solution differently? Can you see it at a glance?
Can you use the result, or the method, for some other problem?
Analysis of AlgorithmsAnalysis of Algorithms
How good is the algorithm? Correctness
Time efficiency
Space efficiency
Simplicity
Does there exist a better algorithm? Lower bounds
Optimality
Why Study Algorithms?Why Study Algorithms?
Theoretical importance
The core of computer science
Practical importance
A practitioner’s toolkit of known algorithms
Framework for designing and analyzing algorithms for new problems
Useful mindset
Course StructureCourse Structure
Fundamentals of the Analysis of Algorithms (Ch. 2) Asymptotic notations, analysis of recursive and non-recursive
algorithms, empirical analysis
Algorithmic Strategies (Ch. 3-9) Brute force, Divide-and-Conquer, Decrease-and-Conquer,
Transform-and-Conquer, Space and Time Tradeoffs, Greedy Techniques, Biologically-inspired techniques, Dynamic Programming
Limitations of Algorithms (Ch. 10 + handouts) Turing Machines, Computability, Problem Classification
Coping with Limitations on Algorithms (Ch. 11) Backtracking and Branch and Bound
Anany Levitin, “Introduction to the Design and Analysis of Algorithms”, International Edition, Addison-Wesley, 2003
PracticalsPracticals
Weekly mini prac exams
Given a problem specification that is solvable using the algorithm design strategies presented in the course Design Algorithm
Code it in C++
Submit it for automatic marking
After the 3-hour lab session will be asked to do a short analysis of the solution