L12 LaGrange Multiplier Method

• Homework• Review• Summary• Test

1

Constrained OptimizationLaGrange Multiplier Method

2

nkx x x

mjg

pi= h

f :MINIMIZE

(L )kk

(U )k

j

i

1

...10)(

...10)(: ToSubject

) (

x

x

x

Remember:1.Standard form2.Max problems f(x) = - F(x)

KKT Necessary Conditions for Min

3

))(( )()( 2

11i

m

iii

p

iii sguhυfL

xxxsu,v,x,

nkx

gu

x

hυ

x

f m

j k

jj

p

i k

ii

k

to1for 011

mjsg

ph

jj

i

to1for 0*)(

to1ifor 0*)(2

x

x

Regularity check - gradients of active inequality constraints are linearly independent

mjs j to1for 02 mjsu jj to1for 0*

mju j to1for 0*

Prob 4.122

4

]4[]13[

...)3()3(

)()(),(

222121

22

21

2222

211121

sxxuxx

xxL

sgusguxxfL

4

13subject to

)3()3(),(

21

21

22

2121

xxg

xxh

xxxxfMin

KKT Necessary Conditions

5

int?Regular po0

0,004

013

03)3(2

0)3(2

:ConditionsNecessary KKT

2221

21

22

11

uuss

sxxg

xxh

uxx

L

uxx

L

Case 1

61031

24026

18306

1031

6320

18306

1031

6320

6102

1031

6320602

04

013

003)3(200)3(2

0u Case1

21

11

21

221

21

2

1

xx

xxxx

sxxg

xxh

xx

!6.0

049.07.3

04

7.324)9.0(269.020/18

180200

24026

18306

60186

24026

18306

1031

24026

18306

2

2

221

1

1

2

BADs

s

sxxg

xx

x

Case 2

7

40011

10031

61320

61102

40011

10031

6132061102

004

013

03)3(20)3(2

0 Case2

21

21

11

21

21

21

2

1

uxx

uxx

uxxuxx

xxg

xxh

uxux

s

25.31)75.0(3

75.04/330040

10031

61320

61102

40011

10031

61320

61102

1

1

2

xxx

Case 2 cont’d, find multipliers

8

25.15.1)75.0(33

75.0

34000

5.13300

5.41300

5.13300

5.41300

5.01100

613)75.0(20

6110)25.3(2

u

125.500

075.025.175.025.3

2

1

fsu

xx

Case 2 cont’d, regular pt?

9

Regular Pt? 1. pt feasible, YES2. active constraint gradients independent

11

,3

1gh

Are active constraint gradients independent i.e. parallel?

0

13

11

A

A

Determinant of Constraint gradientsnon-singular?

Case 2 results in a KKT point!

Graphical Solution

10

21

00

75.075.0

75.325.1

5.45.0

00

75.075.0

75.325.1

)375.0(2)325.3(2

00

11

75.031

25.1)3(2)3(2

0

2

1

xx

guhf

Constraint Sensitivity

11

125.500

075.025.175.025.3

2

1

fsu

xx

Note how relaxing h increases the feasible region but is in the wrong “direction.” Recall ν can be ±!Multiply h by -1, ah ha!

Sufficient Condition

12

Is this a convex programming problem?Check f(x) and constraints:

)3(2

)3(2

22

11

xx

f

xx

f

4,220

02

21

MM

fH

From convexity theorems:1. Hf is PD2. All constraints are linear

Therefore KKT Pt is global Min!

True/False

13

LaGrange Multiplier Method• May produce a KKT point• A KKT point is a CANDIDATE minimum

It may not be a local Min

• If a point fails KKT conditions, we cannot guarantee anything….The point may still be a minimum.

• We need a SUFFICIENT condition14

15

Convex set:All pts in feasible region on a straight line(s).

Convex sets

Non-convex setPts on line are not in feasible region

16

10);(10;)1(

)1()2()2(

)2()2(

ααααα

xxxxxxx

Multiple variables Fig 4.21

0122

21 xx

What if it were an equality constraint?

misprint

17

.

Figure 4.22 Convex function f(x)=x2

Bowl that holds water.

10)]()([)())1((10))()(()()(

10;)()1()()(

12112

121

12

αxfxfαxfxααxfαxfxfαxfxf

αxfαxαfxf

1810)]()([)())1((

10))()(()()(10;)()1()()(

12112

121

12

αxfxfαxfxααxfαxfxfαxfxf

αxfαxαfxf

Fig 4.23 Convex function.

Test for Convex Function

19

10))()(()()(

10;)()1()()()1()2()1(

)1()2(

αffαff

αfααff

xxxx

xxx

Difficult to use above definition!

However, Thm 4.8 pg 163:If the Hessian matrix of the function is PD ro PSD at all points in the set S, then it is convex.

PD… “strictly” convex, otherwisePSD… “convex”

Theorem 4.9

20

}to1,0)(g; to1for,0)(|{

SetConstraint

j mjpihS i

xxx

Given:

S is convex if:1. hi are linear2. gj are convex i.e. Hg PD or PSD

When f(x) and S are convex= “convex programming problem”

“Sufficient” Theorem 4.10, pg 165

21

The first-order KKT conditions are Necessary and Sufficient for a GLOBAL minimum….if:

1. f(x) is convexHf(x) Positive definite

2. x is defined as a convex feasible set SEquality constraints must be linearInequality constraints must be convex

HINT: linear functions are convex!

Summary• LaGrange multipliers are the

instantaneous rate of change in f(x) w.r.t. relaxing a constraint.

• Equality constraints may need tightening rather than loosening

• Convex sets assure contiguity and or the smoothness of f(x)

• KKT pt of a convex programming problem is a GLOBAL MINIMUM!

22