Lecture 8 1

Linear and Combinatorial

Optimization

Lecture 8:

Algorithm Complexity,

Shortest Path

Dynamic Programming

• Algorithm complexity.

– The simplex method not polynomial.

– LP polynomial.

– Interior point methods - Karmarkar

– The ellipsoid method

• Shortest path

• Dynamic programming

Lecture 8 2

Algorithm complexity

A problem is called decidable if there is an

algorithm that solves the problem. (In finite

time).

Example: The traveling salesman’s problem is

decidable, for example, through exhaustive search.

Time complexity: How long time does the

algorithm take?

Memory complexity: How much memory does the

algorithm need?

Since it takes time to use memory, the memory

complexity is not worse than the time complexity.

Lecture 8 3

Time bounds

Little ordo

We write f(x) = o(g(x)) if limx→∞ f/g = 0.

Example: x2 = o(x7).

Big Ordo

We write f(x) = O(g(x)) if there is a constant k

such that f(x) ≤ kg(x) when x goes to ∞ (that is

if f/g is bounded when x → ∞).

Assumption: all kinds of operations require unit

time.

Of all inputs possible, the worst-case behavior

decides the complexity of an algorithm.

Lecture 8 4

Rates of growth for algorithms

• polynomial - if f(n) = O(nk) for some k

• subexponential - if f is not O(nk) for any k

but O((1 + ǫ)n) for every ǫ > 0.

• exponential - if f(n) = O(qn) for some q > 1.

• superexponential - if f grows faster than

every expontentially growing function.

Usually, one is only interested in the two

categories polynomial versus exponential.

Often, in exponential, both sub and super are

included.

Lecture 8 5

Several versions of anoptimization problem

• Optimization: Find an optimal feasible

solution x∗.

• Decision: Is there a feasible solution x with

f(x) ≤ L

Example: LP versus LI.

LI (linear inequalities) is the following problem:

Given an integer m× n matrix A and m-vector b,

is there an n-vector x such that Ax ≤ b?

Lecture 8 6

Problem classification

Most often the decision version is used to

determine a problem’s degree of complexity.

A decision problem is in the class P if there is a

polynomial algorithm for its solution.

A decision problem is in the class NP if: when

the answer is YES for a given solution, then this

solution can be verified in polynomial time.

(For NP-problems, one only needs to verify if a

given solution is feasible and better than L in

polynomial time - certificate.)

Lecture 8 7

Problem reduction and NP-completeness

Certain problems are related so that if there is a

fast algorithm for solving one problem, one can

convert this algorithm to solve the other.

We say a problem (P) reduces to another (P’) if

any algorithm that solves (P’) can be converted

to an algorithm for solving (P).

There is a collection of NP-problems such that if

one of them could be solved in polynomial time,

then all others could also be solved in polynomial

time. These are called NP-complete problems.

Example: Traveling salesman problem, knapsack

problem.

A crude categorisation of problems: easy

problems are those that can be solved in

polynomial time and hard problems are

NP-complete which one suspects require

exponential time.

Lecture 8 8

Simplex method notpolynomialTheorem 8.1. For every d > 1 there is an LP

problem with 2d equations, 3d variables, and

integer coefficients (≤ 4) such that the simplex

method may take 2d − 1 iterations to find the

optimum.

Further, one can show for any pivoting rule in the

simplex, one can always construct a problem

which takes exponential time to solve with

simplex.

Lecture 8 9

LP is still in the class P

There are other algorithms with polynomial time

complexity.

Examples:

• Karmarkar’s algorithm

• Ellipsoid method

Lecture 8 10

Karmarkar’s algorithm

Consider the problem

min z = cTx

Ax = b

x ≥ 0

Suppose we are given a feasible solution x0 > 0,

that is, in the interior.

We want to choose ∆x such that x1 = x0 +∆x

has a lower value of the goal function and still

fulfills the constraints.

b = Ax1 = Ax0 +A∆x = b+A∆x,⇒ A∆x = 0

cTx1 < cTx0,⇒ cT∆x < 0

0 ≤ x1,⇒ ∆x ≥ −x0

Lecture 8 11

First, we choose direction, Pz, according to

Pz = (I −AT (AAT )−1A)z .

The vector Pz lies in the nullspace of A since

A(Pz) = Az −AAT (AAT )−1Az = Az −Az = 0 .

Let z = −c and choose ∆x = −Pz then cT∆x is

as large negative as possible.

Then choose the step length as 0.98 times the

distance from x0 to the border in the direction

−Pc.

One can show that this algorithm has polynomial

time complexity.

Lecture 8 12

Binary search

Relationship between LI (decision problem) and

LP (optimization problem).

An integer between 1 and B can be determined

with log(B) questions of the type: Is x > a?

LP: Given integer matrices A, b and c, solve the

problem max cTx,Ax = b, x ≥ 0.

LI: Given integer matrices A and b, find an x with

Ax = b, x ≥ 0.

Theorem 8.2. There is a polynomial algorithm

for LP if and only if there is a polynomial

algorithm for LI (decision problem).

Lecture 8 13

Proof: (⇒) For the solution of an LP, one also

needs to examine if there are feasible solutions.

(⇐) By adding constraints of the type cTx ≥ ck

one can bound the optimal goal function with

binary search. When enough precision is

obtained, one can stop.

(For general LP problems, show first that LI can

be used to check if the problem has feasible

solutions or if the problem is unbounded, and

then one can extract the optimal solution by

binary search.)

Lecture 8 14

The Ellipsoid algorithm

Idea:

1. Let a large ellipsoid include all feasible

solutions and thereby also the optimal

solution.

2. Pick a point in the ellipsoid (for example, the

centre). If it is feasible, then stop. Otherwise

construct a hyperplane through this point in

such a way that it is easy to determine in

which half of the ellipsoid possible feasible

solutions are contained.

3. Compute the smallest ellipsoid containing the

correct half of the ellisoid.

4. Repeat step 2 and 3 until some stop criteria.

One can show that this algorithm is polynomial.

Lecture 8 15

The Shortest Path Problem

Consider a Graph G = (N,E) with n nodes

s = 1 denotes the source node

t = n denotes the sink node

c(i,j) denotes the length (cost) of the edge (i, j)

Find the shortest path from the s to the t (P):

min z =∑

(i,j)∈E

cijxij

∑

j:(i,j)∈E

xji −∑

j:(i,j)∈E

xij =

−1 i = s = 1

+1 i = t = n

0 o.w.

xij ≥ 0

Minimize the cost.

The flow from the source is one.

The flow to the sink is one.

No accumulation in the nodes.

Lecture 8 16

Properties of the LPTheorem 8.3. If c(i,j) ≥ 0 ∀c(i,j) then

x(i,j) ≤ 1 ∀c(i,j).

Theorem 8.4. If there exists a finite feasible

solution to the LP problem, then x(i,j) ∈ Z.

Consider the dual problem (D):

maxw = yt − ys

yj − yi ≤ cij

Put ys = 0 since one yi is redundant (one

constraint in (P) is redundant).

Lecture 8 17

Properties of the dualLemma 8.1. If y feasible for (D) then

z ≥ yt − ys.

A feasible solution to (D) gives a lower bound on

(P).

Theorem 8.5. If y feasible for (D) and Psk is a

path from s to k then c(Psk) ≥ yk. (ys = 0)

Proof.

c(Psk) =∑

(i,j)∈Psk

cij ≥∑

(i,j)∈Psk

(yj−yi) = yk−ys = yk

Corollary 8.1. Every path Psk with c(Psk) = yk

is a cheapest path.

Lecture 8 18

Bellmans method

Observe that yj − yi ≤ cij implies yj ≤ yi + cij

and that the problem

maxw = yt

yj ≤ yi + c(i,j)

implies yj = mini(cij + yi).

Assume that the G is acyclic:

1. Sort the nodes s.t. (i, j) ∈ E ⇒ i < j.

2. Put y1 = 0.

3. Mark nodes from node 1 using

yj = mini(cij + yi).

Lecture 8 19

Djikstras method

Assume that cij ≥ 0.

Notation:

Nu = set of non-reached nodes

p = predecessor index

q = temporary predecessor index

v = temporary node prices

k = node to be marked

1. Initialize: Nu = V \ {s}, ys = 0, yj = csj ,

qj = s ∀j ∈ Nu : (s, j) ∈ E,

vj = M ∀j ∈ Nu : (s, j) /∈ E

2. Put yk := vk = minj∈Nuvj

3. Put Nu := Nu \ {k} and pk = qk

4. If t /∈ Nu or Nu = ∅ stop

5. ∀j ∈ Nu : (k, j) ∈ E, if ckj + yk < vj put

qj := k and vj = ckj + yk

6. Goto 2

Lecture 8 20

Fords method

No Assumptions!

1. Initialize: Nu = V , ys = 0, yj = M, j 6= s

2. Put yk := minj∈Nuyj

3. Put Nu := Nu \ {k}

4. ∀j; (k, j) ∈ Nu,

if ckj + yk < yj put pj := k and yj = ckj + yk,

if j /∈ Nu put Nu := Nu ∪ {j}

5. If Nu = ∅ stop

6. Goto 2

Lecture 8 21

Dynamic programming

Related to branch and bound - implicit

enumeration of solutions.

Suppose discrete-time sequential decision process,

t = 1, . . . , T and decision variables x1, . . . , xT .

At time t, the process is in state st−1.

Assumptions:

(i) the contribution to the objective function for

xt depends only st−1 and xt.

(ii) st depends only on st−1 and xt.

Lecture 8 22

Dynamic programming

Problem formulation:

maxx1,...,xT

T∑

t=1

gt(st−1, xt) (1)

st = Φ(st−1, xt) for t = 1, . . . , T (2)

s0 is given. (3)

Recursive optimization scheme:

Define

zk(sk−1) = maxxk,...,xT

T∑

t=k

gt(st−1, xt)

for k = T, T − 1, . . . , 1. We want to find z1(s0).

Theorem 8.6.

zk(sk−1) = maxxk

{gk(sk−1, xk) + zk+1(sk)}

Examples: Shortest path problem, knapsack

problem.

Lecture 8 23

Repetition - Lecture 8

• Algorithm Complexity

• Karmarkar’s Algorithm

• The Ellipsoid Method

• Dynamic Programming