1

Smoothed Analysis of Algorithms

Shang-Hua Teng

Boston University

Akamai Technologies Inc

Joint work with Daniel Spielman (MIT)

2

Outline

• Part I: Introduction to Algorithms

• Part II: Smoothed Analysis of Algorithms

• Part III: Geometric Perturbation

3

Part I: Introduction to Algorithms

• Type of Problems

• Complexity of Algorithms

• Randomized Algorithms

• Approximation

• Worst-case analysis

• Average-case analysis

4

Algorithmic Problems• Decision Problem:

– Can we 3-color a given graph G?

• Search Problem:– Given a matrix A and a vector b, find an x

– s.t., A x = b

• Optimization Problem: – Given a matrix A and a vector b, and an objective

vector c, find an x that

– maximize cT x s.t. A x b

5

The size and family of a problem

• Instance of a problem– Input:

• for example, a graph, a matrix, a set of points

– desired output• {yes,no}, coloring, solution-vector, convex hull

– Input and output size• Amount of memory needed to store the input and output

• For example: number of vertices in a graph, dimensions of a matrix, the cardinality of a point set

• A problem is a family of instances.

6

An Example

Median of a set of numbers

• Input: a set of numbers {a1,, a2,…, an}

• Output: ai that maximizes

n

aaaaaa ijjijj |)}:{||,}:{min(|

1 2 3 4 5 6 7 8 9 10

7

Quick Selection

Quick Selection ({a1,, a2,…, an}, k )

1. Choose a1 in {a1,, a2,…, an}

2. Divide the set into

3. Cases:

1. If k = |L| + 1, return a1;

2. If k < |L|, recursively apply Quick_Selection(L,k)

3. If k > |L|+1, recursively apply Quick_Selection(L,n-k-1)

}{},{},{ 111 aaRaaaL jj

8

Worse-Case Time ComplexityLet T({a1,, a2,…, an} ) be the number of basic steps needed

in Quick-Selection for input {a1,, a2,…, an}.• We classify inputs by their size• Let An be the set of all input of size n.

• T(n) = n2 - n

}),,({max)( 1},{ 1

nAaa

aaTnTnn

9

Better Algorithms from Worst-case View point

• Divide and Conquer– Linear-time algorithm– Blum-Floyd-Pratt-Rivest-Tarjan

10

Average-Case Time Complexity

• Let An be the set of all input of size n.

• Choose {a1,, a2,…, an} uniformly at random

• E(T(n)) = O(n)

}),,({)( 1},{ 1

nAaa

aaTEnTAvgnn

11

Randomized Algorithms

Quick Selection ({a1,, a2,…, an}, k )

1. Choose a random element s in {a1,, a2,…, an}

2. Divide the set into

3. Cases:1. If k = |L| + 1, return s;2. If k < |L|, recursively apply Quick_Selection(L,k) 3. If k > |L|+1, recursively apply

Quick_Selection(L,n-k-1)

}{},{},{ saRssaL jj

12

Expected Worse-Case Complexityof Randomized Algorithms

}),,({max)( 1},{ 1

nAaa

aaTEnTEnn

E(T(n)) = O(n)

13

Approximation Algorithms

Sampling-Selection ({a1,, a2, …, an})

1. Choose a random element ai from {a1,, a2,…, an}

2. Return ai.

ai is a δ–median if

Prob[ai is a (1/4)–median ] = 0.5

n

aaaaaa ijjijj |)}:{||,}:{min(|

14

Approximation Algorithms

Sampling-Selection ({a1,, a2, …, an})

1. Choose a set S of random k elements from {a1,,

…, an}

2. Return the median ai of S.

Complexity: O(k) 21212/1Pr k

i emedianaisa

15

When k = 3

a b c

M id d le -o f-3

16

Iterative Middle-of-3(Miller-Teng)

m id -o f-3 m id -o f-3 m id -o f-3

m id -o f-3

Randomly assign elements from {a1,, a2, …, an} to the leaves

2/12/1Pr ki emedianaisa

17

Summary

• Algorithms and their complexity

• Worst-case complexity

• Average-cast complexity

• Design better worse-case algorithm

• Design better algorithm with randomization

• Design faster algorithm with approximation

18

Sad and Exciting Reality• Most interesting optimization problems are hard• P vs NP• NP-complete problems

– Coloring, maximum independent set, graph partitioning

– Scheduling, optimal VLSI layout, optimal web-traffic assignment, data mining and clustering, optimal DNS and TCPIP protocols, integer programming…

• Some are unknown:– Graph isomorphism

– factorization

19

Good News

• Some fundamental problems are solvable in polynomial time– Sorting, selection, lower dimensional

computational geometry– Matrix problems

• Eigenvalue problem• Linear systems

– Linear programming (interior point method)– Mathematical programming

20

Better News I

• Randomization helps– Testing of primes (essential to RSA)– VC-dimension and sampling for computational

geometry and machine learning– Random walks various statistical problems– Quicksort– Random routing on parallel network– Hashing

21

Better News II

• Approximation algorithms– On-line scheduling– Lattice basis reduction (e.g. in cryptanalysis)– Approximate Euclidean TSP and Steiner trees– Graph partitioning– Data clustering– Divide-and-conquer method for VLSI layout

22

Real Stories• Practical algorithms and heuristics

– Great-performance empirically – Used daily by millions and millions of people– Worked routinely from chip design to airline

scheduling

• Applications– Internet routing and searching– Scientific simulation– Optimization

23

PART II: Smoothed Analysis of Algorithms

• Introduction of Smoothed Analysis

• Why smoothed analysis?

• Smoothed analysis of the Simplex Method for Linear Programming

24

Smoothed Analysis of Algorithms: Why The Simplex Method Usually Takes

Polynomial Time

Daniel A. Spielman (MIT)

Shang-Hua Teng (Boston University)

Gaussian Perturbationwith variance 2

25

Remarkable Algorithms and Heuristics

Work well in practice, but

Worst case: bad, exponential, contrived.

Average case: good, polynomial, meaningful?

26

Random is not typical

27

Smoothed Analysis of Algorithms:

worst casemaxx T(x)

average caseavgr T(r)

smoothed complexitymaxx avgr T(x+r)

28

Smoothed Analysis of Algorithms

• Interpolate between Worst case and Average Case.

• Consider neighborhood of every input instance

• If low, have to be unlucky to find bad input instance

29

Smoothed ComplexitySmoothed Complexity

30

Classical Example: Simplex Method for Linear Programming

max zT xs.t. A x y

• Worst-Case: exponential

• Average-Case: polynomial

• Widely used in practice

31

Carbs Protein Fat Iron Cost

1 slice bread 30 5 1.5 10 30¢

1 cup yogurt 10 9 2.5 0 80¢

2tsp Peanut Butter 6 8 18 6 20¢

US RDA Minimum 300 50 70 100

Minimize 30 x1 + 80 x2 + 20 x3

s.t. 30x1 + 10 x2 + 6 x3 300 5x1 + 9x2 + 8x3 50 1.5x1 + 2.5 x2 + 18 x3 70 10x1 + 6 x3 100 x1, x2, x3 0

The Diet Problem

32

max zT xs.t. A x y

Max x1 +x2

s.t x1 1 x2 1 -x1 - 2x2 1

Linear Programming

33

Smoothed Analysis of Simplex Method

max zT xs.t. A x y

G is Gaussian

max zT xs.t. (A +G) x y

34

• Worst-Case: exponential

• Average-Case: polynomial

• Smoothed Complexity: polynomial

max zT xs.t. ai

T x ±1, ||ai|| 1

max zT xs.t. (ai+gi )T x ±1

Smoothed Analysis of Simplex Method

35

Perturbation yields Approximation

For polytope of good aspect ratio

36

But, combinatorially

37

The Simplex Method

38

• Simplex Method (Dantzig, ‘47)• Exponential Worst-Case (Klee-Minty ‘72) • Avg-Case Analysis (Borgwardt ‘77, Smale ‘82,

Haimovich, Adler, Megiddo, Shamir, Karp, Todd)• Ellipsoid Method (Khaciyan, ‘79)• Interior-Point Method (Karmarkar, ‘84)• Randomized Simplex Method (mO(d) )

(Kalai ‘92, Matousek-Sharir-Welzl ‘92)

History of Linear Programming

39

Shadow Vertices

40

Another shadow

41

Shadow vertex pivot rule

objective

start z

42

Theorem: For every plane, the expected size of the shadow of the perturbed tope is poly(m,d,1/)

43

Theorem: For every z, two-Phase Algorithm runs in expected time

poly(m,d,1/)

z

44

Vertex on a1,…,ad maximizes z iff

z cone(a1,…,ad )

0

a1

a2

z

z

A Local condition for optimality

45

Polar ConvexHull(a1, a2, … am)

Primala1

T x 1a2

T x 1…am

T x 1

46

47

max z ConvexHull(a1, a2, ..., am)

z

Polar Linear Program

48

Initial Simplex

Opt Simplex

49

Shadow vertex pivot rule

50

51

Count facets by discretizing

to N directions, N

52

Count pairs in different facets

Pr Different Facets[ ]< c/N

So, expect c Facets

53

Expect cone of large angle

54

Angle

Distance

55

Isolate on one Simplex

56

Integral Formulation

57

Example:

For a and b Gaussian distributed points, given that ab intersects x-axis

Prob[ < ] = O(2)

a

b

58

a

b

59

a

b

60

a

b

61

ab

62

Claim: For < P < 2

a

b

ba

dadbbaaxisabc,

10 )()(]0][[

]0Pr[ axisabP

63

Change of variables

a

b

zu

v

da db = |(u+v)sin(du dv dz d

),,()( 00 zua

),,()( 11 zvb

64

ddzdvduzvzuvucPzvu

),,(),,(|)|)( 10

,,

sin(

Analysis: For < P < 2

Slight change in has little effect on i for all but very rare u,v,z

65

Distance: Gaussian distributed corners

a1

a2

a3

p

66

Idea: fix by perturbation

67

Trickier in 3d

68

Future Research – Simplex Method

• Smoothed analysis of other pivot rules

• Analysis under relative perturbations.

• Trace solutions as un-perturb.

• Strongly polynomial algorithm for linear programming?

69

A Theory Closer to Practice• Optimization algorithms and heuristics, such as

Newton’s Method, Conjugate Gradient, Simulated Annealing, Differential Evolution, etc.

• Computational Geometry, Scientific Computing and Numerical Analysis

• Heuristics solving instances of NP-Hard problems.

• Discrete problems?

• Shrink intuition gap between theory and practice.

70

Part III: Geometric Perturbation

Three Dimensional Mesh Generation

Delaunay Triangulations for Well-Shaped 3D Mesh Generation

Shang-Hua Teng

Boston University

Akamai Technologies Inc.

Collaborators:

Siu-Wing Cheng, Tamal Dey, Herbert Edelsbrunner, Micheal FacelloDamrong GuoyGary Miller, Dafna Talmor, Noel WalkingtonXiang-Yang Li and Alper Üngör

73

3D Unstructured Meshes

74

Surface and 2D Unstructured Meshes

courtesy N. Amenta, UT Austin courtesy NASA

courtesy Ghattas, CMU

Numerical Methods

Point Set:Triangulation:

ad hoc

octree Delaunay

Domain, Boundary, and PDEs

element difference

volume

Finite

Ax=b

direct method

Mesh Generationgeometric structures

Linear Systemalgorithm

data structures

ApproximationNumerical Analysis

FormulationMath+Engineering

iterative method

multigrid

Outline• Mesh Generation in 2D

– Mesh Qualities

– Meshing Methods

– Meshes and Circle Packings

• Mesh Generation in 3D– Slivers

– Numerical Solution: Control Volume Method

– Geometric Solution: Sliver Removal by Weighted Delaunay Triangulations

– Smoothed Solution: Sliver Removal by Perturbation

Badly Shaped Triangles

Aspect Ratio (R/r)

Meshing Methods

• Advancing Front• Quadtree and Octree Refinement• Delaunay Based

– Delaunay Refinement– Sphere Packing– Weighted Delaunay Triangulation– Smoothing by Perturbation

The goal of a meshing algorithm is to generate a well-shaped mesh that is as small as possible.

Balanced Quadtree Refinements

(Bern-Eppstein-Gilbert)

Quadtree Mesh

Delaunay Triangulations

Why Delaunay?

• Maximizes the smallest angle in 2D.

• Has efficient algorithms and data structures.

• Delaunay refinement: – In 2D, it generates optimal size, natural

looking meshes with 20.7o (Jim Ruppert)

84

Delaunay Refinement

(Jim Ruppert)

2D insertion 1D insertion

85

Delaunay Mesh

86

Local Feature Spacing (f)

f: R

The radius of the smallest sphere centered at a point that intersects or contains two non-incident input features

87

Well-Shaped Meshes and f

88

f is 1-Lipschitz and Optimal

89

Sphere-Packing

90

p

-Packing a Function f

• No large empty gap: the radius of the largest empty sphere passing q is at most f(q).

f(p)/2

q

91

• The Delaunay triangulation of a -packing is a well-shaped mesh of optimal size.

• Every well-shaped mesh defines a -packing.

The Packing Lemma (2D)

(Miller-Talmor-Teng-Walkington)

92

Part I: Meshes to Packings

93

Part II: Packings to Meshes

94

3D Challenges

• Delaunay failed on aspect ratio• Quadtree becomes octree

(Mitchell-Vavasis)• Meshes become much larger• Research is more Challenging!!!

Badly Shaped Tetrahedra

Slivers

Radius-Edge Ratio

(Miller-Talmor-Teng-Walkington)

R L

R/L

98

The Packing Lemma (3D)

(Miller-Talmor-Teng-Walkington)

• The Delaunay Triangulation of a -packing is a well-shaped mesh (using radius-edge ratio) of optimal size.

• Every well-shaped (aspect-ratio or radius-edge ratio) mesh defines a -packing.

99

Uniform Ball Packing

• In any dimension, if P is a maximal packing of unit balls, then the Delaunay triangulation of P has radius-edge at most 1.

||e|| is at least 2

r is at most 2r

100

Constant Degree Lemma (3D)

(Miller-Talmor-Teng-Walkington)

• The vertex degree of the Delaunay triangulation with a constant radius-edge ratio is bounded by a constant.

101

Delaunay Refinement in 3D Shewchuck

Slivers

Sliver: the geo-roach

Coping with Slivers: Control-Volume-Method(Miller-Talmor-Teng-Walkington)

Sliver Removal by Weighted Delaunay (Cheng-Dey-Edelsbrunner-Facello-Teng)

Weighted Points and Distance

p z

Orthogonal Circles and Spheres

Weighted Bisectors

Weighted Delaunay

Weighted Delaunay and Convex Hull

Parametrizing Slivers

DY

L

Interval Lemma

0 N(p)/3

• Constant Degree: The union of all weighted Delaunay triangulations with Property [] and Property [] has a constant vertex degree

Pumping Lemma

(Cheng-Dey-Edelsbrunner-Facello-Teng)

DY

z

H

r s

pP

q

Sliver Removal by Flipping

• One by one in an arbitrary ordering• fix the weight of each point• Implementation: flip and keep the best

configuration.

115

Experiments

(Damrong Guoy, UIUC)

116

Initial tetrahedral mesh: 12,838 vertices, all vertices are on the boundary surface

117

Dihedral angle < 5 degree: 13,471

118

Slivers after Delaunay refinement: 881

119

Slivers after sliver-exudation: 12

120

15,503 slivers

142 slivers, less elements, better distribution

1183 slivers

121

5636 slivers

18 slivers, less elements, better distribution

563 slivers Triceratops

122

4532 slivers

1 sliver, less elements, better distribution

173 slivers

Heart

123

Smoothing and Perturbation

• Perturb mesh vertices

• Re-compute the Delaunay triangulation

124

Well-shaped Delaunay Refinement(Li and Teng)

• Add a point near the circumcenter of bad element

• Avoids creating small slivers

• Well-shaped meshes