Lp Notes 906

8/13/2019 Lp Notes 906

1/101

ISE 702

Mathematical Programming:Linear

Marc Posner

8/13/2019 Lp Notes 906

2/101

MISCELLANEOUS INFORMATION

Computer Rooms: BE 344, 351, 356 (all students should have access tothese rooms)

Practice Problem Answers: On ISE 702 web page

Start at www-iwse.eng.ohio-state.edu

Click on Industrial and Systems Engineering (top center)

Click on Courses (left column)

Click on ISE Course Descriptions

Click on 702 Mathematical Programming

Click on Answers to practice problems

To login to the programs:Login name: [start of osu email address]Password: [last 4 digits of social security number] [rst and last initials]

Ex: name, osu email, ss #John Doe, Doe.1osu.edu, 123-45-6789

Login name: Doe.1Password: 6789jd

1

8/13/2019 Lp Notes 906

3/101

LP computer packages:

Accessibility Ease of Use Power

EXCEL A D D

LINDO A- A B+CPLEX B A- A

MATLAB B+

To run EXCEL:

Build appropriate spreadsheet. Then, go to EXCEL solver

To run LINDO (for MATLAB, use the MATLAB icon):Click on Start Programs Applications Lindo 6.0 forWindowsorDouble click on Applications on the desktop, double click onLindo 6.0 for Windows

To run CPLEX:

Click on Start Programs Command PromptIn command prompt, type in the following:

C:\ > F:F: \ > cd cplexF: \ CPLEX > cplex

Frequently Asked Questions (FAQ):http://www-unix.mcs.anl.gov/otc/Guide/faq/linear-programming-faq.html

2

8/13/2019 Lp Notes 906

4/101

INTRODUCTION

Why Study Linear Programming?

Many practical applications in a variety of elds - both nding optimalsolutions and as approximation

Production, capital budgeting and resource allocation, blending andmix, portfolio analysis, etc,

Used in health care, food packaging, paper companies, air and waterpollution control, utilities, etc.

Standard tool for analysis of problemsIBMs largest selling commercial package (OSL)Tool in Microsoft Excel

Foundation for many theoretical developments in Operations Research

The Simplex Method is considered to be one of the most importantcomputational algorithms in Engineering and the Sciences (Computing

in Science and Engineering, 2000).

Solution Steps

1. Formulate LP

a. Dene decision variablesb. Determine objective function

c. Determine constraints

2. Solve LP - Different methodologies are available

3. Interpretation of solutions (sensitivity analysis)

3

8/13/2019 Lp Notes 906

5/101

Linear Program in Canonical Form:

Maximize z(x) = cx (objective function)

subject to Ax b (structural constraints)x 0 (non-negativity constraints)

Decision variables: x = ( x1 , x2 , . . . , x n ) En IR n column vector

(This is assumed throughout the course unless otherwise noted)

Parameters: c = (1 n) row vector

A = ( m n) matrixb = ( m 1) column vector

Ai (a i 1 , a i 2 , . . . , a in ) (row vector)

Aj (a1j , a 2j , . . . , a mj )t (column vector)

LP Assumptions

Deterministic: Each coefficient known with certainty

Additive: No interaction effects, i.e., contribution from(x i and xj jointly) = x i + x j (linear)

Proportional: Values of costs and contributions increase proportionally tosize (linear - no setup costs, economies of scale or higher orderterms)

Divisible: Any fractional level permissible

4

8/13/2019 Lp Notes 906

6/101

Linear Programming Example

Two machines ( M 1 , M 2) can produce goods. M 1 is newer and faster butmore costly. M 1 costs $3 to produce one unit. M 2 costs $2 to produce one

unit. The units sell for $5 each.Machine constraints: Cant produce more than 8 units on M 1 or morethan 6 units on M 2 . Must produce at least 1 unit on M 1 .

Material restrictions: Need 1,000 gallons of water for 1 unit on M 1 and2000 gallons for one unit on M 2 . There is 16,000 gallons.

Inventory restrictions: Cant produce more than a total of 13 units.

x i = number of units produced on M i , i = 1 , 2

Maximize z(x1 , x2) = (5 3)x1 + (5 2)x2 = 2x1 + 3x2subject to x1 8

x2 6

x1 1

x1 + 2x2 16x1 + x2 13

x i 0, i = 1 , 2

68

13

1 8 13 16

5

8/13/2019 Lp Notes 906

7/101

Other Problems

1. minimize cx = maximize cx.

2. If xi can be negative, then create variables xi and xi .

Replace each occurrence of xi with xi x i . Include xi 0 and xi 0.

If xi > 0, then xi > 0 and xi = 0.

If xi < 0, then xi = 0 and xi > 0.

3. If Ai x = bi , then we can rewrite the two equations as

Ai x bi Ai x bi .

6

8/13/2019 Lp Notes 906

8/101

Other Formulations

1. If bi 0, then

a i 1x1 + + a in xn bi a i 1x1 + a in xn + xs = bi

where xs 0 and cs = 0.

xs is called a slack variable.

2. If bi < 0, then

a i 1x1 + + a in xn bi a i 1x1 a in xn ( bi )

a i 1x1 a in xn x l = ( bi )

where xl 0 and cl = 0.

x l is called a surplus variable.

Together, 1 and 2 give Ax = b and b 0. (Standard Form)

7

8/13/2019 Lp Notes 906

9/101

Applications

Product Mix Problems: Minimize cost given demand and resourcecapacity constraints.

Production mix, media buying selections, nancial portfolios, etc.

Typical constraints are

production of product i demand for product i

amount of resource j used for each process amount of j available.

Diet Problems: Mixing of raw materials to achieve certain minimum levelsof attributes.

Mixing of livestock feed for cattle, mixing of foods in school cafeterias,etc.


ingredients requirement .

Example: Let xi = amount of food i used. Food i has ti calories. Sincethe minimum daily requirement is 2000 calories,

t1x i + t2x2 + + tn xn 2000.

8

8/13/2019 Lp Notes 906

10/101

Blending Problems: Mixing of raw materials to achieve product withcertain minimum qualities.

Blending of crude oil to make various gasoline products, mixing of chemicals, blending of alloys to make steel, mixing of recycled paper tomake various qualities of paper, making of hotdogs, etc.


quantity of attributetotal quantity

% of attribute required.

Then, multiply by denominator to linearize.

Example: Crude oil i has octane rating r i per gallon. The octane rating of gas must be at least d per gallon. Let xi = amount of crude oil i that goesinto gas.

r 1x1 + r 2x2 + + r n xnx i

d

r 1x1 + r 2x2 + + r n xn d x i(r 1 d)x1 + ( r 2 d)x2 + + ( r n d)xn 0

9

8/13/2019 Lp Notes 906

11/101

Multiperiod Planning Problems: Make decisions over several periods.

Inventory problems, nancial problems, work scheduling, etc.

Typical constraints areDemand and capacity constraints for a given period

Constraints that tie periods together

Example: Letdt = demand in period ts t = production capacity in period t

x t = amount produced in period tyt = inventory in period t 1 carried over to t.

x t stx t + yt dt

yt +1 = xt + yt dt

10

8/13/2019 Lp Notes 906

12/101

Goal Programming: Objective is to be as close as possible to a set of desired goals.

Meeting environmental standards, production scheduling (deliver itemson time), product mix, etc.

Objective function is of the form min i wi |G i ci x | whereG i = goal i, wi = weight of goal i.

Nonlinear objective function is transformed into

mini

wi (d+i + di )

sub. to. ci x d+i + di = Gi for all id+i , d

i 0 for all i

where d+i = amount over goal Gi , and di = amount under goal Gi .

Note: Can have different weights for d+i and di .

Network Problems: Shortest path, max ow, min cost ow, etc.

Transportation, communication, shipping, etc.

Typical constraints are ow constraints:

ow into a node = ow out of a node

i

x ik =j

xkj , for each node k

where xij = the ow from node i directly to node j .

11

8/13/2019 Lp Notes 906

13/101

Transportation Problem: Ship items from production facilities tocustomers at minimum cost.

cij = cost to ship from facility i to customer j , i = 1 , . . . , m , j = 1 , . . . , n

s i = supply of facility i, i = 1, . . . , mdj = demand of customer j , j = 1, . . . , nx ij = amount shipped from i to j , i = 1 , . . . , m , j = 1 , . . . , n

Minimizem

i=1

n

j =1

cij x ij

subject ton

j =1

x ij si , i = 1 , . . . , m

m

i=1

x ij dj , j = 1 , . . . , n

x ij 0, i = 1, . . . , m, j = 1, . . . , n ,

where mi=1 s i =

nj =1 dj .

Can assume nj =1 x ij = s i and

mi=1 x ij = dj because it is never optimal

to ship extra.

Specialized solution techniques such as Stepping Stone Method. (Seetextbook for further details.)

As network ow problem

s i dj

12

8/13/2019 Lp Notes 906

14/101

Assignment Problem: Matching of two types of objects

Jobs to machines, crews to routes, etc.

Minimize n

i=1

n

j =1

cij x ij

subject ton

j =1

x ij = 1, i = 1 , . . . , n

n

i=1

x ij = 1, j = 1, . . . , n

x ij {0, 1}, i = 1 , . . . , n, j = 1 , . . . , n

Special type of Transportation Problem

13

8/13/2019 Lp Notes 906

15/101

Denitions

6

8

1 8 16

FS

x

Feasible region (Feasible Set) FS { x | Ax b, x 0}.

Feasible solution x0 FS.

Optimal Set (maximization) OS { x FS | cx cx0 for all x0 FS }.

Optimal Solution (max) Global Optimal Solution x OS z(x) z(x) for all x FS

Extreme point corner point of FS.

Optimal solutions for an LP occur at extreme points of FS.

14

8/13/2019 Lp Notes 906

16/101

Ball B (x0 ,) = {x IR n | || x0 x || }

Local Optimal Solution (max) x0 S where there is an > 0 such thatf (x0) f (x) for all x S B (x0 ,)

for some neighborhood around x0 , x0 is the best solution

Local

OptimumGlobal

Optimum

15

8/13/2019 Lp Notes 906

17/101

Graphical Solution Method

1. Make picture of the constraint region.

2. Move isoquant lines of objective function in direction c until optimumis reached.

3. Determine coordinates of extreme point(s).

Maximize z(x1 , x2) = (5 3)x1 + (5 2)x2 = 2x1 + 3x2subject to x1 8

x2 6

x1 1

x1 + 2x2 16

x1 + x2 13

x i 0, i = 1 , 2

6

8

13

1 8 13 16

x = (8 , 4), z = 28

z = 2

z = 16

16

8/13/2019 Lp Notes 906

18/101

Outcomes of a Linear Program

1. Unique Optimal Solution

2. Multiple Optimal Solutions

3. Unbounded Solution

4. Infeasible Solution

2. Multiple Optimal Solutions

Maximize z(x) = x1

subject to x1 8x2 6

x1 + 2x2 16

x 0

Entire set between optimal extreme points is optimal all points inconvex combination = { i x i | x i is an optimal extreme point,

i = 1, 0 i 1 for all i}.

Example: 14 (8, 4) + 34 (8, 0) = (8 , 1)

(8,0)

(8,4)z

17

8/13/2019 Lp Notes 906

19/101

3. Unbounded Solution

Maximize z(x) = 2 x1 + 3x2subject to x2 6

x1 + x2 1

x 0

6

1

Note: Solution unbounded FS unbounded.

FS unbounded solution unbounded.

z(x) = 2x1 3x2

18

8/13/2019 Lp Notes 906

20/101

4. Infeasible Solution

Maximize z(x) = 2 x1 + 3x2subject to x1 8

x2 6

x1 + 2x2 16

x1 + x2 15

x 0

6

8

15

8 15 16

19

8/13/2019 Lp Notes 906

21/101

CONVEXITY AND POLYHEDRAL SETS

Hyperplane { x IR n | x = k, IR n }.

is normal (perpendicular) to hyperplane.

Halfspace { x IR n | x k, IR n } set of points on one side of hyperplane.

Polyhedral set intersection of a nite number of halfspaces.

FS is a polyhedral set.

Convex combination x is a convex combination of x1 , x2 , . . . x m if thereexists IR m such that x =

mi=1 i x

i for i 0 and mi=1 i = 1.

{x | x = x 1 + (1 )x2 , 0 1} = line segment with endpoints x1 andx2 .

Convex set S is a convex set if for all x1 , x2 S and (0, 1), thenx 1 + (1 )x2 S .

Theorem: FS is a convex set.

Proof: Let x1 , x2 F S . This implies that Ax1 b and Ax2 b.

Thus,

A(x 1 + (1 )x2) = Ax 1 + (1 )Ax2

b + (1 )b = b.

20

8/13/2019 Lp Notes 906

22/101

Constraint Types for Ai x bi , i {1, 2, . . . , m }:

Binding or Tight for x0 Ai x0 = bi .

Non-binding or Loose for x0 Ai x0 < b i .

Redundant {x | Ak x bk , k = 1, . . . , i 1, i + 1 , . . . , m ; x 0} = {x | Ax b, x 0}.

Faces of a Polyhedral Set

Face F of a polyhedral set F {x | Ax b, x 0} {x | AI x = bI , x J = 0}, for some

I {1, 2, . . . , m } and J {1, 2, . . . , n }.

0-dimensional face = extreme point1-dimensional face = edge

(n 1)-dimensional face = facet (hyperplane)

n-dimensional face = FS ( I = J = {})

21

8/13/2019 Lp Notes 906

23/101

Extreme point of a convex set Let S be a convex set. x0 S is anextreme point of S if there does not exist (0, 1) and x1 , x2 S ,where x1 , x2 = x0 , such that x0 = x 1 + (1 )x2

i.e. x0

is not a convex combination of x1

and x2

.

At extreme point x0 , there are at least n tight constraints from the m + nconstraints Ax b, x 0.

degenerate x0 is a degenerate extreme point if more than n constraintsare tight at x0 .

adjacent extreme points x1 and x2 are adjacent if the line segment joining them is an edge of {x | Ax b, x 0}.

Ray emanating at x0 { x0 + d | 0} for some dIR n .

Direction of ray r d such that r = {x0 + d | 0}.

Direction of convex set S direction of recession dIR n such thatx + d S for all x S and 0.

d may be normalized for recession direction, i.e., di = 1.

22

8/13/2019 Lp Notes 906

24/101

8/13/2019 Lp Notes 906

25/101

Characterization of Polyhedral Sets

Caratheodorys Theorem. Let x1 , x2 , . . . , x k be the set of extreme points and let d1 , d2 , . . . , d l be the set of extreme directions of {x | Ax b, x 0}. x0 {x | Ax b, x 0} iff there exist , such that x0 =

ki=1 i x

i + lj =1 j dj ,

ki=1 i = 1 , , 0.

Proof sketch (see book for details).

. Suppose x0 = ki=1 i x

i + lj =1 j dj ,

ki=1 i = 1, , 0. Because

X = {x | Ax b, x 0} is convex, x = ki=1 i x

iX . d =

lj =1 j d

j isa recession direction of X . Thus, x0 = x + d X .

. Suppose x0X . Let M be sufficiently large so thatk

i=0nj =1 x

ij M . Since xi X = X {x |

nj =1 x j M }, let

x1 , . . . , x k , xk +1 , . . . , x k + u be the extreme points of X .

We rst show that we can write x0 as a convex combination of theextreme points of X . Suppose that the constraintsG {1, 2, . . . , m + n + 1 } are tight at x0 . If |G| n, then this part iscomplete. Otherwise, select a direction d such that the constraints of G

remain tight, i.e., Gd = 0. Move in direction d until an additionalconstraint is tight at y1 . Repeat in direction d to nd point y2 . x0 is aconvex combination of y1 and y2 . If y1 and y2 are extreme points of X ,then this part is complete. Otherwise, repeat for each point until nconstraints are tight at all of the points. Using these points we can writex0 as a convex combination of the extreme points of X .

If these points are extreme points of X , then we are done. Otherwise,observe that each of xk+1 , xk +2 , . . . , x k+ u are adjacent to some extremepoint of X . Let xk+ i be adjacent to xv( i ) X . Since the constraint

ki=0

nj =1 x

ij M is articial, xk + i xv( i ) is an extreme direction.

Thus, xk+ i = xv( i ) + t ( i ) dt ( i ) for some t(i). This implies thatx0 =

ki=1 i x

i + lj =1 j dj .

24

8/13/2019 Lp Notes 906

26/101

Concave function f : S IR is concave if f (x 1 + (1 )x2) f (x1) + (1 )f (x2) for all x1 , x2 S and (0, 1).

Theorem. If x0 is a local maximum of a concave function over a convex set S , then x0 is a global maximum.

Proof: If x0 is a local maximum, then there exists an -neighborhood N around x0 such that f (x0) f (x) for all x N .

Let x be a global maximum. Since S is convex, x+ (1 )x0 S forall (0, 1). Select such that x = x + (1 )x0 N .

By the concavity of f , f (x) = f (x + (1 )x0) f (x)+ (1 )f (x0).

Since x N , f (x0) f (x). Thus,

f (x0) f (x) + (1 )f (x0)

f (x0) f (x)

f (x0) f (x)

and x0 OS .

25

8/13/2019 Lp Notes 906

27/101

Linear functional f : IR n IR such that f (x + y) = f (x) + f (y) forall x, y IR n , IR .

The objective function is a linear functional.

Theorem. OS is convex.

Proof: Let x1 , x2 OS, and let x = x 1 + (1 )x2 for [0, 1].Since x1 , x2 FS and FS is convex, x FS.Since z is a linear functional,

z(x) = z(x1

+ (1 )x2

)= z(x1) + (1 )z(x2)

= z(x1).

Convex function f : S IR is convex if f (x 1 + (1 )x2) f (x1) + (1 )f (x2) for all x1 , x2 S and

(0, 1).

Closed set a set that includes its boundary points.

Theorem. There exists a local maximum, x0 , of a convex function over a closed set S such that x0 is on the boundary of S.

LP has an optimal solution on the boundary of {x | Ax b, x 0}.

26

8/13/2019 Lp Notes 906

28/101

Theorem. If OS = , then z(x) achieves a maximum at an extreme point of FS.

Proof: For any x F S ,

x =k

i=1

i x i +l

j =1

j dj

k

i=1

i = 1

, 0,

where {x i } are the extreme points and {dj } are the extreme directions of

FS.The LP is equivalent to

maxk

i=1

(cxi ) i +l

j =1

(cdj )j

sub to.k

i=1

i = 1

, 0.

If cdj > 0 for any j , then z = and unbounded optimum.

If cdj 0 for all j , then we can set j = 0 for all j .

27

8/13/2019 Lp Notes 906

29/101

Therefore, LP is

maxk

i=1

(cxi ) i

sub to.k

i=1

i = 1

0.

The optimum value is max {cxj }, and the optimal solution is at anextreme point.

Examine neighboring extreme points until no further improvement ispossible.

28

8/13/2019 Lp Notes 906

30/101

Characterizing an Extreme Point Solution

Let Ax = b. Add surplus and slack variables if necessary.

To have a solution, rank (A) = rank (A, b).

Assume that the rank (A) = m.

Basis of A B m linearly independent columns of A.

A = ( B | N ) after possible rearrangement of the columns

We use B to represent both the columns of A and the indicescorresponding to the columns. The same is true for N . The meaningfollows from the context.

Ax = ( B | N )xBxN

= b

Bx B + Nx N = b

Bx B = b Nx N

Since rank (B ) = m and B is (m m), B 1 exists.

Therefore, xB = B 1b B 1Nx N .

29

8/13/2019 Lp Notes 906

31/101

Theorem. x0 is an extreme point of FS iff there exists B {1, 2, . . . n }such that B has m elements, rank (B ) = m, x0B = B

1b 0, and x0N = 0.

Basic variables xB .Nonbasic variables Zero variables xN .

Basic feasible solution x Bx N =B 1 b

0 .

Each basic feasible solution corresponds to a unique extreme point.

Degenerate solution xj = 0 for some j B .

A degenerate extreme point corresponds to multiple bases.

Each non-degenerate extreme point corresponds to a unique basis.

Example:x1 8 x1 + x3 = 8

x2 6 = x2 + x4 = 6

x1 + 2x2 12 x1 + 2x2 + x5 = 12

x 0 x 0

Point (0 , 6, 8, 0, 0) is degenerate because

B = {1, 2, 3}, N = {4, 5}

B = {2, 3, 4}, N = {1, 5}

B = {2, 3, 5}, N = {1, 4}

(0,6)x1 = 0 x1 = 8

x2 = 0

x2 = 6

x1 + 2x2 = 12

At most nm bases implies at mostnm extreme points.

Two extreme points are adjacent only if the set of basic variables differsby 1 index (assuming no degeneracy).

30

8/13/2019 Lp Notes 906

32/101

Let ej = (0 , . . . , 0, 1, 0, . . . , 0) (The 1 is the j th entry).

Theorem. d is an extreme direction of {x | Ax = b, x 0} iff for some j N , B 1N j 0 and d = dB

dN = B

1 N j

e j.

Proof sketch: ( B | N ) dBdN = 0 (necessary condition for d to be an extremedirection) implies that ( I | B 1N ) dBdN = 0.

Thus, dB = B 1NdN .

Let dN = ej . Then dBdN = B 1 N j

e j .

Example:

x1 + x2 1 x1 + x2 + x3 = 1

x1 + 2x2 3 = x1 + 2x2 + x4 = 3

x 0 x 0

At (1 , 2, 0, 0), B = {1, 2}, N = {3, 4},

B = 1 1 1 2 , B 1 =

2 1 1 1 , N 3 =

10 , N 4 =

01

B 1N 3 = 2 1 0. Thus, d = (2 , 1, 1, 0)t

increase x3 , keep x4 = 0

B 1N 4 = 11 > 0. Thus, no extreme direction for this j .

31

8/13/2019 Lp Notes 906

33/101

THE SIMPLEX METHOD

We now assume that the objective is to minimize (to follow book)

The Algorithm

1. Convert problem to appropriate form (minimize and equalityconstraints).

2. Find an initial basic feasible solution.

3. Check for optimality.

If optimal, then stop. The current extreme point is optimal.

Otherwise, nd a non-basic variable that will improve (reduce) thesolution if increased.

4. Move to an adjacent extreme point.

a. Increase the non-basic variable as much as feasible.

b. Adjust the set of basic and non-basic variables.

Go to 3.

32

8/13/2019 Lp Notes 906

34/101

Recognizing an Optimal Solution

min z(x) = cB xB + cN xN

At each extreme point, xB = B 1b, xN = 0.

Thus, z(x) = cB B 1b = z0 = current value.

But xB = B 1b B 1Nx N . Therefore,

z(x) = cB (B 1b) cB B 1Nx N + cN xN = z0 (cB B 1N cN )xN = z0

j N

(cB B 1N j cj )x j

= z0 j N

(zj cj )x j

where zj = cB B 1N j = opportunity cost.

If zj cj 0 for all j N , then we are at an optimal extreme point.

Otherwise, we can improve the solution by increasing xj .

33

8/13/2019 Lp Notes 906

35/101

Which Nonbasic Variable to Increase

1. Steepest ascent ( max j N {zj cj }): Dantzigs rule

2. Largest improvement rule - Largest leaving and entering pairExperimental results are discouraging.

3. Partial pricing - Steepest ascent from a small collection of choices

Useful for large scale problems.

4. Lexicographical - Smallest index where zj cj > 0

Useful for avoiding degeneracy.

Amount of Solution Improvement

xB 0

B 1b B 1Nx N 0

B 1b B 1N j x j 0B 1b B 1N j x j , j N

We can increase xj until ( B 1b)i = ( B 1N j ) i x j for some i B .

Minimum ratio test: min iB {(B 1b)i / (B 1N j )i | (B 1N j )i > 0}

book denes ( B 1b)i bi and (B 1N j )i yi

This implies that xi becomes 0.x j is entering variable (the variable that enters the basis).

x i is leaving variable (the variable that leaves the basis).

Since only one variable in basis changes, the extreme points are adjacent.

34

8/13/2019 Lp Notes 906

36/101

Resolving for B 1 each iteration is a lot of work.

We keep the current basis and associated values in a tableau to reduce thework and make current information accessible.

The Simplex Tableau

Initial:decision vars. slack vars. RHS

z c 0 0

xs A I b

Later:xB xN RHS

z 0 cB B 1N cN cB B 1b

xB I B 1N B 1b

35

8/13/2019 Lp Notes 906

37/101

Example:

max z(x) = 2 x1 + 3x2 min z(x) = 2x1 3x2sub. to x1 8 sub. to x1 + x3 = 8

x2 6 = x2 + x4 = 6

x1 + 2x2 16 x1 + 2x2 + x5 = 16

x 0 x 0

B = {3, 4, 5}, N = {1, 2}

x3 = 8 x1

x4 = 6 x2x5 = 16 x1 2x2

z = 0 2x1 3x2

x1 x2 x3 x4 x5 RHS

z 2 3 0 0 0 0

cB B 1N cN 0 cB B 1b

x3 1 0 1 0 0 8x4 0 1 0 1 0 6

x5 1 2 0 0 1 16

B 1N I B 1b

B = B 1 =1 0 0

0 1 0

0 0 1

, N =1 0

0 1

1 2

,

cB = (0 , 0, 0), cN = ( 2, 3)

Increase x2 (steepest ascent rule)

36

8/13/2019 Lp Notes 906

38/101

B 1b =

8

6

16

B 1N 2x2 =1 0 0

0 1 0

0 0 1

0

1

2

x2

= x2 = min {6, 8} = 6, x2 is entering variable, x4 is leaving variable

B = {2, 3, 5}, N = {1, 4}

x3 = 8 x1x2 = 6 x4x5 = 16 x1 2x2 = 16 x1 2(6 x4) = 4 x1 + 2x4

z = 0 2x1 3(6 x4) = 18 2x1 + 3x4

Use elementary row operations to get I for {x3 , x2 , x5}. Note equivalenceto algebra.

x1 x2 x3 x4 x5 RHS

z 2 0 0 3 0 18

x3 1 0 1 0 0 8

x2 0 1 0 1 0 6x5 1 0 0 2 1 4

B =

1 0 0

0 1 0

0 2 1

, B 1 =

1 0 0

0 1 0

0 2 1

, N =

1 0

0 1

1 0

,

cB = (0 , 3, 0), cN = ( 2, 0)B 1 can be found from the columns of the original slack variables because

B 1(A | I ) = ( I | B 1N | B 1).

Increase x1

37

8/13/2019 Lp Notes 906

39/101

8

6

4

B 1N 1x1 =1 0 0

0 1 0

0 2 1

1

0

1

x1


B = {1, 2, 3}, N = {4, 5}

x3 = 8 x1 = 8 (4 + 2 x4 x5) = 4 2x4 + x5x2 = 6 x4x1 = 4 + 2 x4 x5

z = 18 2(4 + 2 x4 x5) + 3 x4 = 26 x4 + 2x5

x1 x2 x3 x4 x5 RHS

z 0 0 0 1 2 26

x3 0 0 1 2 1 4

x2 0 1 0 1 0 6

x1 1 0 0 2 1 4

B =

1 0 1

0 1 0

0 2 1

, B 1 =

1 2 1

0 1 0

0 2 1

, N =

0 0

1 0

0 1

,

cB = (0 , 3, 2), cN = (0 , 0)

Increase x4

38

8/13/2019 Lp Notes 906

40/101

4

6

4

B 1N 4x4 =1 2 1

0 1 0

0 2 1

0

1

0

x4


B = {1, 2, 4}, N = {3, 5}

x4 = 2 0.5x3 + 0 .5x5x2 = 4 + 0 .5x3 0.5x5x1 = 8 x3

z = 28 + 0 .5x3 + 1 .5x5

x1 x2 x3 x4 x5 RHS

z 0 0 1/ 2 0 3/ 2 28

cB B 1b

x4 0 0 1/ 2 1 1/ 2 2

x2 0 1 1/ 2 0 1/ 2 4

x1 1 0 1 0 0 8B 1 B 1b

Solution is optimal.

B =0 0 1

1 1 0

0 2 1

, B 1 =1/ 2 1 1/ 2

1/ 2 0 1/ 2

1 0 0

, N =1 0

0 0

0 1

,

cB = (0 , 3, 2), cN = (0 , 0)

39

8/13/2019 Lp Notes 906

41/101

Multiple Optimal Solutions

At optimal solution, some nonbasic variable has zj cj = 0. Thus, canincrease variable at no additional cost.

Example:

min z(x) = 32 x1 3x2sub. to x1 + x3 = 8

x2 + x4 = 6

x1 + 2x2 + x5 = 16

x 0

At optimal solution (4 , 6, 4, 0, 0) B = {1, 2, 3}, N = {4, 5} the tableau is

x1 x2 x3 x4 x5 RHS

z 0 0 0 0 3/ 2 24

x3 0 0 1 2 1 4

x2 0 1 0 1 0 6x1 1 0 0 2 1 4

Increase x4 by 2 to get optimal solution (8 , 4, 0, 2, 0)

x1 x2 x3 x4 x5 RHS

z 0 0 0 0 3/ 2 24

x4 0 0 1/ 2 1 1/ 2 2

x2 0 1 1/ 2 0 1/ 2 4x1 1 0 1 0 0 8

OS = set of convex combinations of optimal extreme points ={(4, 6, 4, 0, 0) + (1 )(8 , 4, 0, 2, 0) | [0, 1]}.

40

8/13/2019 Lp Notes 906

42/101

Unbounded Solution

At some basic feasible solution, some nonbasic variable has zj cj > 0and has B 1N j 0. Thus, nonbasic variable can go to and continue to

improve solution.

Example:

min z(x) = 2x1 3x2sub. to x2 + x3 = 6

x1 + x2 + x4 = 1

x 0

A last basic feasible solution is when B = {1, 2}, N = {3, 4}

x1 = 1 + x2 + x4 = 1 + (6 x3) + x4 = 5 x3 + x4x2 = 6 x3

z = 2(5 x3 + x4) 3(6 x3) = 28 + 5x3 2x4

x1 x2 x3 x4 RHSz 0 0 5 2 28

x1 1 0 1 1 5

x2 0 1 1 0 6

Increase x4

56

B 1N 4x4 = 1 11 0

01

x4 = 10

x4

= increase x4 indenitely

41

8/13/2019 Lp Notes 906

43/101

STARTING SOLUTION AND CONVERGENCE

Finding an Initial Basic Feasible Solution

May not have a basic feasible solution at start.

Example:


x2 6 = x2 + x4 = 6

x1 1 x1 x5 = 1

x1 + 2x2 = 16 x1 + 2x2 = 16

x 0 x 0

x3 = 8 x1x4 = 6 x2

x5 = 1 + x1 ? = 16 x1 2x2 solving with x1 basic makes x3 < 0

For , subtract surplus variable and add articial variable.

For =, add articial variable.

x1 + x

3 = 8

x2 + x4 = 6

x1 x5 + xa = 1

x1 + 2x2 + xb = 16

where B = {3, 4,a ,b}

42

8/13/2019 Lp Notes 906

44/101

Two Phase Simplex Method

Phase I: Solve

min xartsub. to Ax = b

x 0

If not all xart = 0, then problem is infeasible. Stop.

Otherwise, pivot until all xart are nonbasic (if possible). Drop articial

variables (columns) if desirable, or keep nonbasic.If some articial variables remain in basis, then remove thecorresponding rows (they are redundant), or keep variables at 0.

Phase II: Solve

min cx

sub. to Ax = b

x 0

starting at current basic feasible solution.

43

8/13/2019 Lp Notes 906

45/101

Phase I:

x1 x2 x3 x4 x5 xa xb RHS

z 0 0 0 0 0 1 1 0

z 2 2 0 0 1 0 0 17

x3 1 0 1 0 0 0 0 8

x4 0 1 0 1 0 0 0 6

xa 1 0 0 0 1 1 0 1

xb 1 2 0 0 0 0 1 16

x1 = entering variable, xa = leaving variable

z 0 2 0 0 1 2 0 15

x3 0 0 1 0 1 1 0 7

x4 0 1 0 1 0 0 0 6

x1 1 0 0 0 1 1 0 1

xb 0 2 0 0 1 1 1 15

x2 = entering variable, x4 = leaving variable

z 0 0 0 2 1 2 0 3

x3 0 0 1 0 1 1 0 7

x2 0 1 0 1 0 0 0 6

x1 1 0 0 0 1 1 0 1

xb 0 0 0 2 1 1 1 3

44

8/13/2019 Lp Notes 906

46/101

x5 = entering variable, xb = leaving variable


z 0 0 0 0 0 1 1 0

x3 0 0 1 2 0 0 1 4

x2 0 1 0 1 0 0 0 6

x1 1 0 0 2 0 0 1 4

x5 0 0 0 2 1 1 1 3

A basic feasible solution has been reached. End of Phase I.

Phase II:


z 2 3 0 0 0 0 0 0

z 0 0 0 1 0 0 2 26

x3 0 0 1 2 0 0 1 4

x2 0 1 0 1 0 0 0 6

x1 1 0 0 2 0 0 1 4

x5 0 0 0 2 1 1 1 3


z 0 0 1/ 2 0 0 0 3/ 2 28

x4 0 0 1/ 2 1 0 0 1/ 2 2x2 0 1 1/ 2 0 0 0 1/ 2 4

x1 1 0 1 0 0 0 0 8

x5 0 0 1 0 1 1 0 7

45

8/13/2019 Lp Notes 906

47/101

Infeasible Solution

Example:


x2 6 = x2 + x4 = 6

x1 15 x1 x5 + xa = 15

x1 + 2x2 16 x1 + 2x2 + x6 = 16

x 0 x 0

Phase I:

x1 x2 x3 x4 x5 x6 xa RHS

z 0 0 0 0 0 0 1 0

z 1 0 0 0 1 0 0 15

x3 1 0 1 0 0 0 0 8

x4 0 1 0 1 0 0 0 6

xa 1 0 0 0 1 0 1 15

x6 1 2 0 0 0 1 0 16


z 0 0 1 0 1 0 0 7

x1 1 0 1 0 0 0 0 8

x4 0 1 0 1 0 0 0 6

xa 0 0 1 0 1 0 1 7

x6 0 2 1 0 0 1 0 8

Optimal, but cant get xa out of basis

46

8/13/2019 Lp Notes 906

48/101

Articials Cannot be Removed From Basis

Example:

max z(x) = 2 x1 + 3x2 min z(x) = 2x1 3x2sub. to x1 = 8 sub. to x1 + xa = 8

x2 = 6 = x2 + xb = 6

x1 + x2 15 x1 + x2 + x3 = 15

x1 + 2x2 = 20 x1 + 2x2 + xc = 20

x 0 x 0

End of Phase I:x1 x2 x3 xa xb xc RHS

z 0 0 0 2 3 0 0

x1 1 0 0 1 0 0 8

x2 0 1 0 0 1 0 6

x3 0 0 1 1 1 0 1

xc 0 0 0 1 2 1 0

Phase II:

x1 x2 x3 xa xb xc RHS

z 2 3 0 0 0 0 0

z 0 0 0 2 3 0 34

Notice 0 in z row for xc . If xa and xb dont enter basis, then xc will neverchange

47

8/13/2019 Lp Notes 906

49/101

Big M Method

Add a large value to articial variables to make them leave the basis.

If they cant, then solution infeasible.

Method is similar to Two Phase method.

Advantages:

1. In case of ties, move direction of good LP solution

Disadvantages:

1. Causes rounding errors

2. How big is big enough must take coefficient and RHS intoaccount

.000001x1 1 large value for x1

Methods almost equivalent

Commercial codes use Two Phase method

Improvement is Little , i.e., make original variables small. Then roundingerrors dont affect Phase I.

48

8/13/2019 Lp Notes 906

50/101

Example of move in good direction:

max z(x) = 2 x1 + 3x2 min z(x) = 2x1 3x2 + Mx a + Mx bsub. to x1 8 sub. to x1 + x3 = 8

x2 6 = x2 + x4 = 6

x1 1 x1 x5 + xa = 1

x1 + 2x2 = 16 x1 + 2x2 + xb = 16

x 0 x 0


z 2 3 0 0 0 M M 0

z 2M + 2 2M + 3 0 0 M 0 0 17M

x3 1 0 1 0 0 0 0 8

x4 0 1 0 1 0 0 0 6

xa 1 0 0 0 1 1 0 1

xb 1 2 0 0 0 0 1 16

Final Tableau

z 0 0 1/ 2 0 0 M M 3/ 2 28

x4 0 0 1/ 2 1 0 0 1/ 2 2

x2 0 1 1/ 2 0 0 0 1/ 2 4

x1 1 0 1 0 0 0 0 8

x5 0 0 1 0 1 1 0 7

Same as Two Phase method, except for articial columns in z row

49

8/13/2019 Lp Notes 906

51/101

Degeneracy

Degeneracy too many hyperplanes go through an extreme point (at

least n m + 1)Thus, some basic variable has 0 value

Example:


x2

6 = x2 + x

4 = 6

2x1 + 5x2 38 2x1 + 5x2 + x5 = 38

x1 + 2x2 16 x1 + 2x2 + x6 = 16

x 0 x 0

After rst pivot:

x1 x2 x3 x4 x5 x6 RHS

z 2 0 0 3 0 0 18x3 1 0 1 0 0 0 8

x2 0 1 0 1 0 0 6

x5 2 0 0 5 1 0 8

x6 1 0 0 2 0 1 4

(4,6)

x1 is entering variablex5 or x6 is leaving variable. Before degeneracy, more than one variable

can enter basis.

50

8/13/2019 Lp Notes 906

52/101

Degenerate Tableau:

x1 x2 x3 x4 x5 x6 RHS

z 0 0 0 2 1 0 26

x3 0 0 1 5/ 2 1/ 2 0 4x2 0 1 0 1 0 0 6

x1 1 0 0 5/ 2 1/ 2 0 4

x6 0 0 0 1/ 2 1/ 2 1 0

x4 enters and x6 leaves for no progress

Cycling Simplex method cycles between bases at a degenerate point andnever leaves extreme point

Fake example: alternate between ( x4 , x5), (x4 , x6), and ( x5 , x6)

Real example appears in book p 165 - 166

Stalling Simplex method cycles between bases at a degenerate pointand takes a long time to leave

Extreme point can be degenerate without cycling or stalling occurring

51

8/13/2019 Lp Notes 906

53/101

Methods to Prevent Cycling

Basic idea is to create a rule to break ties in an ordered way so that old

basis is not repeated

1. Lexicographical

2. Bland

Suppose x1 , x2 , . . . , x k have 0 value at a degenerate point. Make it moredesirable to have xi in the basis over any of xi+1 , x i+2 , . . . , x k . Thus,

never return to a previously visited basis because would get worse.

Methods to prevent cycling are not employed in most commercial codes

1. Cost It is too time consuming to check

2. Unlikely It seldom happens in practice because computerrounding errors breaks ties

Maybe in network problems where ow conservationequations have lots of equations at 0 value

52

8/13/2019 Lp Notes 906

54/101

SPECIAL IMPLEMENTATIONS

Revised Simplex Method

Keeps all information in smaller array

All commercial Simplex codes use this method

The Simplex Tableau:

xB xN RHS

z 0 cB B 1N cN cB B 1b

xB I B 1N B 1b

Revised Simplex Tableau:

z cB B 1 cB B 1b

xb B 1 B 1b

Revised Simplex Method

1. For each nonbasic variable calculate zj cj = cB B 1N j cj .Select index k = argmax j N {zj cj }.If zk ck 0, then stop.

2. Entering variable is k.Leaving variable isr = argmin

iB{(B 1b)i / (B 1N k )i | (B 1N k )i > 0}.

3. Update tableau using elementary row operations that turn N r intoer . Go to 1.

Standard Simplex array size is ( m + 1)( n + 1) while Revised Simplex arraysize is (m + 1) 2

53

8/13/2019 Lp Notes 906

55/101

Example:

min z(x) = 2x1 3x2sub. to x1 + x3 = 8

x2 + x4 = 6x1 + 2x2 + x5 = 16

x 0

B = {3, 4, 5}, N = {1, 2}

z 0 0 0 0

cB B

1 cB B

1bx3 1 0 0 8

x4 0 1 0 6

x5 0 0 1 16

B 1 B 1b

N =1 0

0 1

1 2

, cB = (0 , 0, 0), cN = ( 2, 3)

cB B 1N 1 c1 = (0 , 0, 0)1

0

1

( 2) = 2

cB B 1N 2 c2 = (0 , 0, 0)01

2

( 3) = 3

Increase x2 (steepest ascent rule)

54

8/13/2019 Lp Notes 906

56/101

B 1b =

8

6

16

B 1N 2x2 =1 0 0

0 1 0

0 0 1

0

1

2

x2


B = {2, 3, 5}, N = {1, 4}

Since B 1N 2 =

0

1

2

=

0

1

0

, add -2(row 2) to row 3

z 0 3 0 18

x3 1 0 0 8

x2 0 1 0 6

x5 0 2 1 4

N =1 00 1

1 0

, cB = (0 , 3, 0), cN = ( 2, 0)

cB B 1 = (0 , 3, 0)1 0 0

0 1 0

0 2 1

= (0 , 3, 0)

55

8/13/2019 Lp Notes 906

57/101

cB B 1N 1 c1 = (0 , 3, 0)1

0

1

( 2) = 2

cB B 1N 4 c4 = (0 , 3, 0)0

1

0

0 = 3

Increase x1

8

64

B 1

N 1x1 =

1 0 0

0 1 00 2 1

1

01

x1


B = {1, 2, 3}, N = {4, 5}

Since B 1N 1 =

1

01

=

0

01

, add -(row 3) to row 1

z 0 1 2 26

x3 1 2 1 4

x2 0 1 0 6

x1 0 2 1 4

N =0 0

1 0

0 1

, cB = (0 , 3, 2), cN = (0 , 0)

56

8/13/2019 Lp Notes 906

58/101

cB B 1 = (0 , 3, 2)1 2 1

0 1 0

0 2 1

= (0 , 1, 2)

cB B 1N 4 c4 = (0 , 1, 2)0

1

0

0 = 1

cB B 1N 5 c5 = (0 , 1, 2)0

0

1

0 = 2

Increase x4

4

6

4

B 1N 4x4 =1 2 1

0 1 0

0 2 1

0

1

0

x4

= x4 = min {2, 6} = 2, x4 is entering variable, x3 is leaving variableB = {1, 2, 4}, N = {3, 5}

57

8/13/2019 Lp Notes 906

59/101

Since B 1N 4 =

2

1

2

=

1

0

0

, multiply row 1 by 1/2, add 12 (row

1) to row 2, add row 1 to row 3

z 1/ 2 0 3/ 2 28

cB B 1 cB B 1b

x4 1/ 2 1 1/ 2 2

x2 1/ 2 0 1/ 2 4

x1

1 0 0 8

B 1 B 1b

N =1 0

0 0

0 1

, cB = (0 , 3, 2), cN = (0 , 0)

cB B 1 = (0 , 3, 2)1/ 2 1 1/ 2

1/ 2 0 1/ 2

1 0 0

= ( 1/ 2, 0, 3/ 2)

cB B 1N 3 c3 = ( 1/ 2, 0, 3/ 2)1

0

0

0 = 1/ 2

cB B 1N 5 c5 = ( 1/ 2, 0, 3/ 2)0

0

1

0 = 3/ 2

Therefore, optimal solution

58

8/13/2019 Lp Notes 906

60/101

Product Form of the Inverse

Used in some commercial codes uses sparsity, reduces rounding errors,keeps tableau small

To obtain new B 1 (but not needed), for entering variable k at row r,add multiple of row r to other rows to make column k equal to er premultiply B 1 by matrix

E =

1 0 0 0 (B 1N k )1 / (B 1N k )r 0 0

0 1 0 0 (B 1N k )2 / (B 1N k )r 0 0...

0 0 0 1 (B 1N k )r 1 / (B 1N k )r 0 0

0 0 0 0 1/ (B 1N k )r 0 0

0 0 0 0 (B 1N k )r +1 / (B 1N k )r 1 0...

0 0 0 0 (B 1N k )m / (B 1N k )r 0 1

Let B 1i be B 1 at the ith simplex iteration. Then,

B 11 = I,

B 12 = E 1B 13 = E 2E 1

...

B 1t = E t 1E t 2 E 1

Only need to store the column and position for each E

If number of E s become large, reinvert basis and restart at new basis

59

8/13/2019 Lp Notes 906

61/101

8/13/2019 Lp Notes 906

62/101

cB B 1 = cB E t 1E t 2 E 1

Postmultiply by E

If column r of E is g = ( g1 , g2 , . . . , gm )t, then

cE = ( c1 , c2 , . . . , c m )

1 0 0 g1 0 0

0 1 0 g2 0 0...

0 0 1 gr 1 0 0

0 0 0 gr 0 00 0 0 gr +1 1 0

...

0 0 0 gm 0 1

= ( c1 , c2 , . . . , c r 1 ,cg,cr +1 , . . . , cm )

61

8/13/2019 Lp Notes 906

63/101

Example:

min z(x) = 2x1 3x2sub. to x1 + x3 = 8

x2 + x4 = 6x1 + 2x2 + x5 = 16

x 0

xB =x3x4x5

, N = {1, 2}

B 1 =1 0 0

0 1 0

0 0 1

, N =1 0

0 1

1 2

, cB = (0 , 0, 0), cN = ( 2, 3)

cB B 1N 1 c1 = (0 , 0, 0)1

0

1

( 2) = 2

cB B 1N 2 c2 = (0 , 0, 0)0

1

2

( 3) = 3

Increase x2

62

8/13/2019 Lp Notes 906

64/101

B 1b =

8

6

16

B 1N 2x2 =0

1

2

x2


xB =x3x2x5

Store for E 1 ,(column of E ; entering row r) = ( 0/ 1, 1/ 1, 2/ 1;2) = (0 , 1, 2;2)

xN =x1x4

, N =1 0

0 1

1 0

, cB = (0 , 3, 0), cN = ( 2, 0)

cB B 1 = cB E 1 = 0, (0, 3, 0)0

1

2

, 0 = (0 , 3, 0)

(cB E 1)N 1 c1 = (0 , 3, 0)1

0

1

( 2) = 2

(cB E 1)N 4 c4 = (0 , 3, 0)0

1

0

0 = 3

Increase x1

63

8/13/2019 Lp Notes 906

65/101

B 1b = E 1b =8

0

16

+0 6

1 6

2 6

=8

6

4

E 1N 1x1 =1

0

1

+0 0

1 0

2 0

x1 =1

0

1

x1


xB =x3x2x1

Store for E 2 , ( 1/ 1, 0/ 1, 1/ 1;3) = ( 1, 0, 1;3)

xN = x4x5

, N =

0 0

1 0

0 1

, cB = (0 , 3, 2), cN = (0 , 0)

cB B 1 = cB E 2E 1 = 0, 3, (0, 3, 2) 1

0

1

E 1

= 0, (0, 3, 2)01

2

, 2 = (0 , 1, 2)

64

8/13/2019 Lp Notes 906

66/101

(cB E 2E 1)N 4 c4 = (0 , 1, 2)0

1

0

0 = 1

(cB E 2E 1)N 5 c5 = (0 , 1, 2)0

0

1

0 = 2

Increase x4

B 1b = E 2

8

6

4

=8

6

0

+ 1 4

0 4

1 4

=4

6

4

E 2E 1N 4x4 = E 2

0

0

0

+0 1

1 1

2 1

x4 = E 2

0

1

2

x4

=0

1

0

+( 1)( 2)

0( 2)

1( 2)

x4 =2

1

2

x4


xB =x4x2x1

, N = {3, 5}

Store for E 3 , (1/ 2, 1/ 2, 2/ 2;1) = (1 / 2, 1/ 2, 1;1)

65

8/13/2019 Lp Notes 906

67/101

N =

1 0

0 0

0 1

, cB = (0 , 3, 2), cN = (0 , 0)

cB E 3E 2E 1 = (0, 3, 2)1/ 2

1/ 2

1

, 3, 2 E 2E 1

= 1/ 2, 3, ( 1/ 2, 3, 2) 1

0

1

E 1

= 1/ 2, ( 1, 3, 3/ 2)0

1

2

, 3/ 2

= ( 1/ 2, 0, 3/ 2)

(cB E 3E 2E 1)N 3 c3 = ( 1/ 2, 0, 3/ 2)10

0

0 = 1/ 2

(cB E 3E 2E 1)N 5 c5 = ( 1/ 2, 0, 3/ 2)0

0

1

0 = 3/ 2

Therefore, optimal solution

66

8/13/2019 Lp Notes 906

68/101

LU Factorization

Improvement over product form of inverse because dont have to complete

entire Gaussian EliminationUsed in many commercial codes

To use an upper triangular matrix to solve problem

B11 B12 B1m0 B22 B2m

.

..0 0 Bmm

x1

x2...

xm

=

b1b2...

bm

xm = bm /B mm

xm 1 = ( bm 1 Bm 1,m xm )/B m 1,m 1 . Substitute for xm and solve.

Repeat for xm 2 , xm 3 , . . . , x 1

To get B in upper triangular form1. Move rows around to get approximately upper triangular

(equivalent to switching two rows of I )

2. Perform partial Gaussian Elimination below main diagonal(equivalent to premultiplying by a lower triangular matrix)

Thus, RB = U , where R is lower triangular and U is upper triangular

Thus, B = R 1U = LU and R 1 is lower triangular

Store partial permutations of I and columns of R required to get U

Updating is similar to Product Form of Inverse, but only partial columnsare needed

67

8/13/2019 Lp Notes 906

69/101

Bounded Variables

Upper and lower bounds on the xs

To put all bounds in tableau adds 2 n constraints, n slack variables, nsurplus variables and n articial variables

Treat bounds similar to way Simplex Method does for nonnegativityconstraints

Each nonbasic variable k is at lower bound lk or upper bound uk

Partition A = ( B | N l | N u ) where xN l = lN l and xN u = uN u

(Book uses N 1 and N 2)Working basis B

xB = B 1b B 1N l lN l B 1N u uN u

z = cB xB + cN l xN l + cN u xN u

= cB (B 1b B 1N l xN l B 1N u xN u ) + cN l xN l + cN u xN u

= cB B 1b (cB B 1N l cN l )xN l (cB B 1N u cN u )xN u

Bounded Variable Simplex Tableau:

xB xN l xN u RHS

cB B 1b

z 0 cB B 1N l cN l cB B 1N u cN u (cB B 1N l cN l )lN l (cB B 1N u cN u )uN u

xB I B 1N l B 1N u B 1b B 1N l lN l B 1N u uN u

68

8/13/2019 Lp Notes 906

70/101

Improving z:

If there is j N l such that zj cj > 0, then we can improve z

If there is j N u such that zj cj < 0, then we can improve z

Find max max j N l {zj cj }, max j N u { zj + cj } , and let k be theindex associated with the maximum.

If max value is 0, then stop. We have an optimal solution.

Otherwise, change the value xk .

Let b = current RHS and z = current value

Change in Tableau:

If k N l , then xk increases

basic variable reaches lower bound

1 = min iB {(b lB )i / (B 1N k )i | (B 1N k )i > 0}

basic variable reaches upper bound

2 = min iB { (uB b/ (B 1N k )i | (B 1N k )i < 0}

Increase xk by = min { 1 , 2 , uk lk }

Update z = z ( zk ck )

Update RHS for pivot row = lk +

Update other RHS = b N k

If < u k lk , then do usual Simplex update for remaining tableau.

Otherwise, do nothing

69

8/13/2019 Lp Notes 906

71/101

If k N u , then xk decreases

basic variable reaches lower bound

1 = min iB { (b lB )i / (B 1N k )i | (B 1N k )i < 0}

basic variable reaches upper bound

2 = min iB {(uB b)i / (B 1N k )i | (B 1N k )i > 0}

Decrease xk by = min { 1 , 2 , uk lk }

Update z = z + ( zk ck )

Update RHS for pivot row = uk

Update other RHS = b + N k

If < u k lk , then do usual Simplex update for remaining tableau.Otherwise, do nothing

To nd an initial starting solution:

1. Set each nonbasic variable to bound of choice

2. Adjust RHS and z value to account for bounds

3. Multiply by -1 if necessary so that bi 0 for all i

4. If necessary, add articials and use Two Phase Method

70

8/13/2019 Lp Notes 906

72/101

8/13/2019 Lp Notes 906

73/101

Adjust z value and RHS

z = 2 18

x3 = 15 12

x4 = 10 6

l u

x1 x2 x3 x4 RHS

z 2 3 0 0 20

x3 1 2 1 0 3

x4 1 1 0 1 4

Increase x1 (Note that x2 must be negative to be considered)

= min {(3 0)/ 1, (4 0)/ 1, 8 1} = 3

= x3 is leaving variable

Adjust RHS and z value

z = 20 6

x1 = 1 + 3

x4 = 4 3

u l

x1 x2 x3 x4 RHS

z 0 1 2 0 26

x1 1 2 1 0 4

x4 0 1 1 1 1

Decrease x2

72

8/13/2019 Lp Notes 906

74/101

= min {(8 4)/ 2, (1 0)/ 1, 6 0} = 1

= x4 is leaving variable

Adjust RHS and z value

z = 26 1

x1 = 4 + 2 1

x2 = 6 1

l l

x1 x2 x3 x4 RHS

z 0 0 1 1 27x1 1 0 1 2 6

x2 0 1 1 1 5

Solution is optimal

73

8/13/2019 Lp Notes 906

75/101

DUALITY

Associated with each LP is another LP. Solving one is equivalent tosolving the other

Original problem is the Primal Problem. The associated problem is theDual Problem.

min cx max wb

sub. to Ax b sub. to wA c

x 0 w 0

Example:

min 2x1 3x2 max 8w1 6w2 16w3sub. to x1 8 sub. to w1 w3 2

x2 6 = w2 2w3 3

x1 2x2 16 w 0

x 0

Primal problem has n variables and m constraints

Dual problem has m variables and n constraints

Generally better to use one with smaller number of constraints (smallerB ) or one with constraints

74

8/13/2019 Lp Notes 906

76/101

Other Forms

Since Ax b Ax b

min cx max wb

sub. to Ax b sub. to wA c

x 0 w 0

max wb

sub. to wA c

w 0

Since Ax = b Ax b,Ax b,

min cx max w1b + w2b

sub. to Ax = b sub. to w1A + w2A c

x 0 w1 0

w2 0max wb

sub. to wA c

w unrestricted

75

8/13/2019 Lp Notes 906

77/101

MINIMIZATION MAXIMIZATION

Variables Constraints

0 0

unrestricted =

Constraints Variables

0

0

= unrestricted

76

8/13/2019 Lp Notes 906

78/101

max 2x1 + 3x2 min 8w1 + 6w2 + 16w3sub. to x1 8 sub. to w1 + w3 2

x2 6 = w2 + 2w3 3x1 + 2x2 16 w 0

x 0

Final Tableaus:

x1 x2 x3 x4 x5 RHS

z 0 0 1/ 2 0 3/ 2 28

x1 1 0 1 0 0 8

x2 0 1 1/ 2 0 1/ 2 4

x4 0 0 1/ 2 1 1/ 2 2

w1 w2 w3 w4 w5 RHS

z 0 2 0 8 4 28

w1 1 1/ 2 0 1 1/ 2 1/ 2

w3 0 1/ 2 1 0 1/ 2 3/ 2

77

8/13/2019 Lp Notes 906

79/101

Theorem. Dual of the Dual is the Primal.

min cx max wb

sub. to Ax b sub. to AT wT c

x 0 w 0

min cx

sub. to ( AT )T x b

x 0

Weak Duality Theorem. Let x0 be a feasible solution to the primal problem (min) and w0 be a feasible solution to the dual problem (max).Then, cx0 w0b.

Proof: Let x0 {x | Ax b, x 0} and w0 {w | wA c, w 0}. Then,cx0 (w0A)x0 = w0(Ax0) w0b.

78

8/13/2019 Lp Notes 906

80/101

Shadow price of constraint i The marginal value of relaxing constraint i

At optimality z = cB B 1b j N (zj cj )x j = wb j N (zj cj )x j

Suppose that the optimal solution is nondegenerate

Then, z/b i = wi = shadow price of constraint i = rate of change of the optimal value with a unit increase in bi

Alternative optimal degenerate bases in primal Alternative optimalextreme point solutions in dual

* With degeneracy, + z

b i = max {wji | wj is an optimal dual vertex }

z

b i = min {wji | w

jis an optimal dual vertex } *

Complementary Slackness Theorem. Let x be an optimal solution min{x | Ax b, x 0} and w be an optimal solution tomax {w | wA c, w 0}. Then, w(Ax b) = 0 and (c wA)x = 0 .

Proof. Follows from the Karush-Kuhn-Tucker optimality conditions.

Either constraint is tight or the shadow price is 0

79

8/13/2019 Lp Notes 906

81/101

Strong Duality Theorem. If either primal or dual has an optimal solution, then they both do. Further, the objective values are equal.

Proof: From Complementary Slackness Theorem,cx = wAx = wb.

Theorem. With regards to the Primal and Dual problems, exactly one of the following is true

1. Both have optimal solutions x and w respectively, and cx = wb.

2. One problem is unbounded and the other is infeasible.

3. Both problems are infeasible.

Proof. 1. follows from Strong Duality Theorem

2. follows from Weak Duality Theorem

3. follows from the following example

min x1 x2 max w1 + w2

sub. to x1 x2 1 sub. to w1 w2 1 x1 + x2 1 = w1 + w2 1

x 0 w 0

80

8/13/2019 Lp Notes 906

82/101

Economic Interpretation of the Dual

Primal

min total cost = j (unit cost of activity j ) (level of activity j )sub. to j (production of commodity i as result of 1 unit of j ) (levelof activity j ) demand for commodity iDual

max income =

i (unit price of commodity i) (demand for commodity i)sub. to j (production of commodity i as result of 1 unit of j ) (unit priceof commodity i) unit cost of activity jTotal charge for goods produced by 1 unit of j

Constraint implies fairness in pricing

81

8/13/2019 Lp Notes 906

83/101

Dual Simplex Method

Start with dual feasible problem: zj cj 0 for j N , some of the bi for

i M may be negativePivot keeping zj cj 0 for j N until bi 0 for all i

Very useful when new constraints are added to problem. The objectivefunction remains the same, but the current solution probably violatesthe new constraint, i.e., RHS is negative.

82

8/13/2019 Lp Notes 906

84/101

SENSITIVITY ANALYSIS

Interested in the sensitivity of the solution to changes in the parametervalues

Sensitivity Report (EXCEL)

Decision Final Reduced Objective Allowable Allowable

Variables Value Cost Coefficient Increase Decrease

changing cells zj cj c

Final Shadow Constraint Allowable Allowable

Constraints Value Price RHS Increase Decrease

zj cj b

Outputs from most other packages are similar

I. Objective Function: If coefficient changes by , does the primalsolution change?

Nonbasic Variable j N :

Cost change is 0 since the variable is nonbasic

zj cj 0 value of the z row in the nal tableau

Basic Variable t B :

Cost change is et B 1b = z + bt

Since zj cj = cB B 1N j cj for all j N ,

zj cj + et B 1N j 0 for some j N

83

8/13/2019 Lp Notes 906

85/101

Example:

min z(x) = 2x1 3x2sub. to x1 + x3 = 8

x2 + x4 = 6x1 + 2x2 + x5 = 16

x 0

x1 x2 x3 x4 x5 RHS

z 0 0 1/ 2 0 3/ 2 28

x4 0 0 1/ 2 1 1/ 2 2

x2 0 1 1/ 2 0 1/ 2 4x1 1 0 1 0 0 8

Nonbasic variables: New basis if c3 1/ 2 or if c5 3/ 2

Basic variables:

zN cN = ( 1/ 2, 3/ 2) et B 1

N = et

1/ 2 1/ 2

1/ 2 1/ 21 0

For x1 , cost change = 8/unit: 1/ 2 and 3/ 2 0 (e3)

1/ 2 = optimal for c1 2 + 1 / 2

For x2 , cost change = 4/unit: 1/ 2 1/ 2 and 3/ 2 1/ 2 ( e2)

1 3 = 3 1 c2 3 + 3

For x4 , cost change = 2/unit: 1/ 2 1/ 2 and 3/ 2 1/ 2 ( e1)

3 1 = 3 c4 1

84

8/13/2019 Lp Notes 906

86/101

II. RHS: If a RHS coefficient changes by , does the dual solution change(the primal basis change)?

Slack constraints: 0 cost

move until tight (-value of slack or surplus variable)

Tight constraints: Cost is shadow price

move until constraint reaches new extreme point basic variable becomes 0 (B 1b + B 1ej ) i 0 for some i B

Example:

For x2 6, change in cost = 0: x4 = 2 = 6 2 b2

For x1 8, change in cost = -1/2 per unit:

2

4

8

+1/ 2

1/ 2

1

0 4 8 = 4 b1 16

For x1 + 2x2 16, change in cost = -3/2 per unit:

2

4

8

+ 1/ 2

1/ 2

0

0 8 4 = 8 b3 20

85

8/13/2019 Lp Notes 906

87/101

Simultaneous Change of Several Parameters

For objective function coefficients or RHS

Let j = proposed change in coefficient j ,

Let j = maximum change in the same direction of j that does notviolate j s optimality range

If nj =1 j / j 1, then change does not violate optimality range

100% Rule: Basis doesnt change as long as total change is 100%

Objective Coefficient Example:

For N 3 , the rst column gives 1 / 2 1 and 1 2

If we take 0.50 of each change, thenc = ( 2 + 14 , 3

12 , 0, 0, 0) = ( 1

34 , 3

12 , 0, 0, 0)

RHS Example:

For x1 8, the second row gives 1 8For x1 + 2x2 16, the second row gives 8 3

If we take 0.75 of 1 and 0.25 of 3 , then b = (8 + 6 , 6, 16 2) = (14 , 6, 14)

86

8/13/2019 Lp Notes 906

88/101

III. Constraint Matrix A:

Nonbasic column:

Since zj cj = cB B 1N j cj for all j N ,

zj cj + cB B 1( ej ) 0 for j N

Example:

If change x1 + (1 + ) x3 = 8,

z3 c3 = 1/ 2 cB B 1 e3

= (0, 3, 2)

1/ 2 1 1/ 2

1/ 2 0 1/ 21 0 0

00

= (1 / 2, 0, 3/ 2)

0

0

1 If (1 + ) < 0, then increase x3 to make constraint slack

ex: If = 2, then x1 x3 = 8

Thus, (10 , 3, 2, 0, 0) FS and z = 29

For x2 + x3 + x4 = 6, 1/ 2 0 ,any value of is OK because constraint is slack

87

8/13/2019 Lp Notes 906

89/101

For x1 + 2x2 + x3 + x5 = 16

1/ 2 (1/ 2, 0, 3/ 2)0

0

1/ 3

If < 1/ 3, then increase x3 to make constraint slack

ex: If = 1/ 3, then x1 + 2x2 x3 / 3 + x5 = 16

Thus, (5 , 6, 3, 0, 0) FS and z = 28

Basic column:

Possible but complicated, B 1 changes which affects entries in everycolumn

88

8/13/2019 Lp Notes 906

90/101

Parametric Analysis

Find set of optimal solutions and values over a range of a parameters value

Objective function: Set cost coefficients to c + c where 0

RHS: Set RHS to b+ b where 0

Procedure:

1. Find optimal solution for = 0.

2. Use sensitivity results to determine how much must increase to moveto a new extreme point. If there is no such point, then stop.

3. Move to the new extreme point. Go to Step 2.

89

8/13/2019 Lp Notes 906

91/101

DECOMPOSITION

Dantzig-Wolf Decomposition

Large scale problem with special structure

2 types of constraints

general complicating constraints

constraints with special structure

Method: solve 2 separate types of LPsMaster problem (for the general constraints)

Subproblems (one for each special structure)

Master problem passes cost coefficients to Subproblems

Subproblems send trial solutions (columns) to Master problem

Iterate between the problems until Subproblems have no solution to send.At this point, an optimal solution is reached

Method is called a column generation procedure

90

8/13/2019 Lp Notes 906

92/101

min zP (x) = ( c1c2 . . . cT )x(P )

sub. to

A1 A2 . . . AT B1

B2. . .

BT

x1

x2...

xT

=

b0

b1

b2...

bT

For Subproblem i = 1 , 2, . . . , T , P i = {x i | B i x i = bi , x i 0}

Let xij = extreme points of P i for j = 1 , 2, . . . , t i anddik = extreme directions of P i for k = 1 , 2, . . . , l i

From Caratheodorys Theorem, each feasible x satisesx i =

t ij =1 ij x

ij + l ik =1 ik dik

where t ij =1 ij = 1, and , 0

Therefore, the Master Problem can be written as:

min zM (, ) =T

i=1

t i

j =1

(ci x ij ) ij +l i

k =1

(ci dik )ik

(M )

sub toT

i=1

t i

j =1

(Ai x ij ) ij +l i

k =1

(Ai dik )ik = b0

t i

j =1

ij = 1

, 0

91

8/13/2019 Lp Notes 906

93/101

zP = z

M when all extreme points and directions are included

Otherwise, zP zM

Each xij

or dik

represents a column in (M)

Suppose we have an optimal extreme point solution ( , ) to (M) for agiven set of xij and dik

Let the dual solution be(w1 , . . . , w n , 1 , . . . , T ) = ( ci x ij , ci dik )B B 1 = ciB B

1

w are dual multipliers of the constraints, are from convexity

constraints ij = 1Thus, zij cij = wAi x ij + i ci x ij for any new ij N zik cik = wAi dik ci dik for any new ik N

Solve each subproblem max( wAi ci )x i + i , sub to xi P i

If wi Ai x i + i ci x i > 0, then can improve Master Problem by addingeither the optimal extreme point (if one exists) or an extreme direction

(if unbounded). Go back and resolve Master Problem.If no candidate, then optimal

92

8/13/2019 Lp Notes 906

94/101

COMPLEXITY ISSUES ANDPOLYNOMIAL TIME ALGORITHMS

Problem Class Set of all instances of same type of problem.

Example: Linear Programming Problems

Purpose

Classify problem classes according to the computational effort required forsolution

Generally concerned with worst case performance

Size of Problem Amount of information necessary to represent aninstance

number of variables and constraints, and size of data

Computational Effort Steps required to solve problem

Good approximation is number of elementary operations, i.e. +, -, , , comparisons, etc.

93

8/13/2019 Lp Notes 906

95/101

Problem classes are categorized by computational effort in terms of input size using term with largest order, and ignoring constants and lower order terms.

Example: 3 n2 + 1000 n = O(n2) = Order n2 , where n is size of input

Example: Solving 0-1 Integer Program by total enumeration

1. Select value for x {0, 1}n

2. Check if x is feasible and nd value of solution

3. Repeat for all values of x

Step 2 requires O(mn ) effort

Process is repeated 2 n times

Complexity is O(2n mn )

Complexity representation is only concerned with asymptotic behavior asn . (Otherwise, lower order terms may dominate.)

Example: 10 10 n is O(n) while 10 10 n2 is O(n2)

The rst is smaller only when n 1020

94

8/13/2019 Lp Notes 906

96/101

Problem Classes

P Set of problem classes which can be solved in a time that is boundedby a polynomial function of the input size

Decision, Recognition or Threshold problem a problem that can beanswered with yes or no

Example: Does there exist an LP solution where cx K ?

Nondeterministic Procedure for recognition problem

1. Guess a solution

2. Check for feasibility

N P Set of problem classes that can complete NondeterministicProcedure in a time bounded by a polynomial function of the input size

95

8/13/2019 Lp Notes 906

97/101

Proposition. P N P

The MAJOR open question in Complexity Theory: P = N P ?

We assume not!

Reducible Problem class P 1 is reducible to P 2 if P 1 can be solved bysolving P 2

If P 1 is reducible to P 2 in polynomial time, then P 2 is at least as hard asP 1

N P -hard Problem class P 2 N P -hard if for every problem classP 1 N P -hard, P 1 transforms to P 2

Problems in N P -hard are at least as hard as those in N P

Frequently optimization versions of N P problems

N P -complete ( N PC ) Problem class P N PC if P N P andP N P -hard

Some N PC problem classes: General Integer Programming, TravelingSalesman Problem

To show problem class P N PC , must show that some N PC problemclass can be transformed to P in polynomial time

96

8/13/2019 Lp Notes 906

98/101

Simplex Method May Take Exponential Time

Variant of Klee-Minty (1972):

maximizen

j =1

10n j x j

sub. to 2i 1

j =1

10i j x j + x i 100i 1 i = 1 , 2, . . . , n

x 0

x = (0 , . . . , 0, 100n 1) z = 100 n 1

Steepest ascent rule requires 2 n 1 iterations

Largest increase rule requires 1 iteration (There exist problems wherelargest increase rule is exponential)

Example: n = 3

max 100x1 + 10x2 + x3sub. to x1 1

20x1 + x2 100

200x1 + 20x2 + x3 10000

x 0

Add slacks s1 , s2 and s3

97

8/13/2019 Lp Notes 906

99/101

max 100x1 + 10x2 + x3sub. to x1 + s1 = 1

20x1 + x2 + s2 = 100200x1 + 20x2 + x3 + s3 = 10000

x 0

s 0

Basis z

s1 s2 s3 0

x1 s2 s3 100

x1 x2 s3 900

s1 x2 s3 1000

s1 x2 x3 9000

x1 x2 x3 9100

x1 s2 x3 9900

s1 s2 x3 10000

All 23 = 8 extreme points are examined

However, Simplex Method takes O(n3) on average

98

8/13/2019 Lp Notes 906

100/101

Khachians Method (1979)

Finding a feasible solution is as hard as nding an optimal solution (use

binary search)

Ellipsoid Algorithm: nds a feasible solution if one exists by movingaround feasible region

1. Cover feasible region with an ellipsoid (make it sufficient large)

2. If center of ellipsoid is in feasible region, then STOP

Otherwise, nd a hyperplane through center that does notintersect feasible region

3. Construct a new ellipsoid based on where old ellipsoid intersectsthe hyperplane

Go to 2

Let E k = kth ellipsoidV ol(E k +1 )/V ol(E k ) 2 1/ 2( n +1) volume reduces exponentially

converges to a feasible solution in polynomial time

Never worked well in practice - invert large matrices to nd ellipsoid

99

8/13/2019 Lp Notes 906

101/101

Karmarkars Method (1984)

Affine Scaling procedure: Moves through feasible region

1. Start at an interior point

2. Transform space so that current point is approximately at centerof feasible region

3. Move in gradient direction

Go to 2 until near an extreme point

Large improvement using gradient if at centerConverge to an optimal extreme point in a log number of steps

Beats Simplex Method for certain classes of sparse matrices

Documents

Lp Notes 906