Convex

Optim

ization

in

Engineering

A

nalysisand

D

esign

Stephen

B

oyd

E

lectricalE

ngineering

D

epartm

ent,Stanford

M

IT

E

E

C

olloquium

Lecture

3/14/94

1

Basic

idea

�M

any

problem

s

arising

in

engineering

analysis

and

design

can

be

cast

as

convex

optim

ization

problem

s

�H

ence,are

fundam

entally

tractable

�R

ecent

interior-point

m

ethods

can

exploit

problem

structure

to

solve

such

problem

s

very

e�

ciently

2

Exam

ple

1

�linear

elastic

structure;

forces

f1;:::;f100induce

de ections

d1;:::;d300

�0

�fi�Fmax

i

,several

hundred

other

constraints:

m

ax

load

per

oor,m

ax

w

ind

load

per

side,etc.

f1

f2

f3

f4

Problem

1a:�nd

w

orst-case

de ection,i.e.,m

axijdij

Problem

1b:�nd

w

orst-case

de ection,w

ith

each

force

\on"

or

\o�",i.e.,fi=

0

or

Fmax

i

3

Exam

ple

1

P

roblem

1a

is

very

easy

�readily

solved

in

a

few

m

inutes

on

sm

allw

orkstation

�generalproblem

has

polynom

ialcom

plexity

P

roblem

1b

is

very

di�

cult

�could

take

w

eeks

to

solve

:::

�generalproblem

N

P

-com

plete 4

Exam

ple

2

Polytope

described

by

linear

inequalities,aT ix�b i,i=

1;:::;L

a1

0

Problem

2a:�nd

point

closestto

origin,i.e.,m

in

kxk

Problem

2b:�nd

point

farthestfrom

origin,i.e.,m

ax

kxk

5

Exam

ple

2

P

roblem

2a

is

very

easy

�readily

solved

on

sm

allw

orkstation

�polynom

ialcom

plexity

P

roblem

2b

is

very

di�

cult

�di�

cult

even

w

ith

supercom

puter

�N

P

-com

plete

6

m

oral:

very

di�

cultand

very

easy

problem

s

can

look

quite

sim

ilar

7

Outline

�C

onvex

optim

ization

�Som

e

exam

ples

�Interior-point

m

ethods

8

Convex

optim

ization

m

inim

ize

f0(x)

subject

to

f1(x)

�0;:::fL(x)

�0

fi:R

n

!

R

are

convex,i.e.,for

allx,y,0

���1,

fi(�x+

(1

��)y)

��fi(x)+

(1

��)fi(y)

�can

have

linear

equality

constraints

�di�erentiability

not

needed

�exam

ples:

Linear

P

rogram

s

(LP

s),problem

s

1a,2a

�other

form

ulations

possible

(feasibility,m

ulticriterion)

9

(R

oughly

speaking,)

Convex

optim

ization

problem

sare

fundam

entally

tractable

�com

putation

tim

e

is

sm

all,grow

s

gracefully

w

ith

problem

size

and

required

accuracy

�large

problem

s

solved

quickly

in

practice

�supported

by

strong

theoreticalresults

(N

em

irovsky

and

Y

udin)

�not

w

idely

enough

appreciated

10

W

hat

\solve"

m

eans:

��nd

globaloptim

um

w

ithin

a

given

tolerance,or,

��nd

proof(certi�cate)

ofinfeasibility

A

lgorithm

s:

�C

lassicaloptim

ization

algorithm

s

do

notw

ork

�E

llipsoid

algorithm

{

very

sim

ple,universally

applicable

{

e�

cient

in

term

s

ofw

orst-case

com

plexity

theory

{

slow

but

robust

in

practice

�(G

eneral)

interior-point

m

ethods

(m

ore

later

:::)

{

e�

cient

in

theory

and

practice

11

Outline

�C

onvex

optim

ization

�Som

e

exam

ples

�Interior-point

m

ethods

12

W

ell-known

exam

ple:FIR

�lterdesign

transfer

function:

H(z)

� =

n X i=0

hiz�

i

design

variables:

x� =

[h0h1:::hn]T

sam

ple

convex

constraints:

�H(ej

0)=

1

(unity

D

C

gain)

�H(ej!0)=

0

(notch

at

!0)

�jH(ej

!)j�0:01

for

!s

�!��

(m

in.40dB

atten.in

stop

band)

�jH(ej!)j�1:12

for

0

�!�!b

(m

ax.1dB

upper

ripple

in

pass

band)

�hi=

hn�

i

(linear

phase

constraint)

�s(t)

� =

t X i=0

hi�1:1H(ej

0)

(m

ax.10%

step

response

overshoot)

13

FIR

�lter

design

exam

ple

(M

.G

rant)

�sam

ple

rate

2=nsec�

1

�linear

phase

�m

ax

�1dB

ripple

up

to

0:4H

z

�m

in

40dB

atten

above

0:8H

z

�m

inim

ize

m

axijhij

�som

e

solution

tim

es:

n=

255:

5

sec

n=

2047:

4

m

in

-1-0

.8-0

.6-0

.4-0

.20

0.2

0.4

0.6

0.8

1-3-2-101234 10

010

110

210

-3

10-2

10-1

100

t

h(t)

f

���H(e

2�jf)���

14

15

Beam

form

ing

om

nidirectionalantenna

elem

ents

at

positions

p1;:::;pn

2R

2

plane

w

ave

incident

from

angle

�:

exp

j(k(�)

Tp�!t);

k(�)

=

�[cos�sin

�]

T

p1

k(�)

�

dem

odulate

to

get

yi=

exp(jk(�)T

pi)

form

w

eighted

sum

y(�)

=

n X i=1

wiyi

design

variables:

x=

[R

ewT

Im

wT]T

(antenna

array

w

eights

or

shading

coe�

cients)

G(�)

� =

jy(�)jantenna

gain

pattern

16

Sam

ple

convex

constraints:

�y(� t)=

1

(target

direction

norm

alization)

�G(� 0)

=

0

(nullin

direction

� 0)

�wis

real(am

plitude

only

shading)

�jwij�1

(attenuation

only

shading)

Sam

ple

convex

objectives:

�m

ax

fG(�)

jj��� tj�5�g

(sidelobe

levelw

ith

10�

beam

w

idth)

��

2X i

jwij2(noise

pow

er

in

y)

17

Acoustic

array

�ltering

acoustic

�eld:

sum

ofplane

w

aves

(2D

for

sim

plicity)

om

nidirectionalm

icrophones,sam

pled

output

ui(t)

�

Multi-input

FIR�lter

y(t)

FIR

�lter

and

add:

y(t)

� =

nmic

X i=1

ntapX

�=1

hi�ui(t��)

design

variables:

x� =

[h11:::hnmicntap]T

incidence-angle

dependent

transfer

function:

H(�;!)

18

Sam

ple

(convex)

speci�cation:

�jH(� t;!)�Hdes(!)j=jHdes(!)j�0:1

(m

ax.10%

deviation

from

desired

T

F

in

target

direction)

Sam

ple

(convex)

objective:

�m

axfjH(�;!)j!L

�!�!U

;j��� tj�10�g

(broadband

attenuation

outside

20�

beam

w

idth)

19

Open-loop

trajectory

planning

D

iscrete-tim

e

linear

system

,input

u(t)2R

p,output

y(t)2R

q

Sam

ple

convex

constraints:

�jui(t)j�U(lim

it

on

input

am

plitude)

�jui(t+

1)�ui(t)j�S(lim

it

on

input

slew

rate)

�l i(t)�yi(t)�ui(t)(envelope

bounds

for

output)

Sam

ple

convex

objective:

�m

axt;ijyi(t)�ydesi

(t)j(peak

tracking

error)

20

Robustopen-loop

trajectory

planning

input

m

ust

w

ork

w

ellw

ith

m

ultiple

plants

one

input

uapplied

to

Lplants;outputs

are

y(1) ;:::;y(L)

constraints

are

to

hold

for

ally

(i)

sam

ple

objectives:

�w

eighted

sum

ofobjectives

for

each

i(average

perform

ance)

�m

ax

over

objectives

for

each

i(w

orst-case

perform

ance)

21

Exam

ple

�tw

o

plants

�0

�u(t)�1

�j�u(t)j�1:25=sec

�m

inim

ize

w

orst-case

peak

tracking

error

�128

tim

e

sam

ples

(variables)

00.

51

1.5

22.

53

3.5

44.

55

0

0.2

0.4

0.6

0.81

1.2

1.4

1.6

1.8

t

Plantstepresponses

00.

51

1.5

22.

53

3.5

44.

55

0

0.2

0.4

0.6

0.81

tu

opt

00.

51

1.5

22.

53

3.5

44.

55

0

0.2

0.4

0.6

0.81

t

y1;y2;y

des

22

Linear

controller

design

(static

case

for

sim

plicity)

KP

w

z y

u

exogenousinputs

actuatorinputs

sensedoutputs

criticaloutputs

linear

plant

Pgiven;design

linear

feedback

controller

K

closed-loop

I/O

relation:

z=

Hw,

H=

Pzw

+

PzuK(I�PyuK)�

1Pyw

m

ost

speci�cations,objectives

convex

in

H,notK

23

Transform

to

convex

problem

Linear-fractional(projective)

transform

ation:

Q� =

K(I�PyuK)�

1

H=

Pzw

+

PzuQPyw:

constraints,objectives

are

convex

in

Q!

�design

Qvia

convex

program

m

ing

�set

K

=

Q(I+

PyuQ)�

1

E

xtends

to

dynam

ic

case

:::

�tim

e

and

frequency

dom

ain

lim

its

on

actuator

e�ort,

regulation,tracking

error

�som

e

robustness

speci�cations

24

Otherexam

ples

�synthesis

ofLyapunov

functions,state

feedback

��lter/controller

realization

�system

identi�cation

problem

s

�truss

design

�V

LSI

transistor

sizing

�design

centering

�nonparam

etric

statistics

�com

putationalgeom

etry

25

Outline

�C

onvex

optim

ization

�Som

e

exam

ples

�Interior-point

m

ethods

26

Interior-pointconvex

program

m

ing

m

ethods

H

istory:

�D

ikin;Fiacco

&

M

cC

orm

ick's

SU

M

T

(1960s)

�K

arm

arkar's

LP

algorithm

(1984);m

any

m

ore

since

then

�N

esterov

&

N

em

irovsky's

generalform

ulation

(1989)

G

eneral:

�#

iterations

sm

all,grow

s

slow

ly

w

ith

problem

size

(typicalnum

ber:

10s)

�each

iteration

is

basically

least-squares

problem

27

Basic

idea

C

hoose

(potentialfct.)

's.t.

�'sm

ooth

�'(x)

!

+

1

as

x!

feasible

set

boundary

�'(x)

!

�1

as

x!

optim

al

M

inim

ize

'by

(m

odi�ed)

N

ew

ton

m

ethod

If'is

properly

chosen:

algorithm

is

polynom

ial,e�

cient

in

practice

28

Typicalexam

ple:m

atrix

norm

m

inim

ization

m

inim

ize

kA0+

x1A1+

���+

xkAkk

tw

o

speci�c

problem

s:

5

m

atrices,5�5;50

m

atrices,50�50

0

2

4

6

8

10

12

10�

4

10�

3

10�

2

10�

1

10

0

10

1

10

2

iteration

dualitygapvs.iteration

problem2

problem1

29

C

ost

per

iteration:

com

puting

N

ew

ton

direction,a

least

squares

problem

w

ith

sam

e

structure

as

originalproblem

(Toeplitz,etc.)

H

ence:

cost

ofsolving

convex

problem

�10

�cost

ofsolving

sim

ilar

least-squares

problem

H

ence:

can

solve

least-squares

problem

e�

ciently

=)

can

solve

convex

problem

e�

ciently

30

Exploiting

problem

structure

via

CG

C

onjugate

G

radients:

solve

m

inx

kAx�bk,x2R

m

via

m

evaluations

ofx!

Axand

y!

ATy

�roughly:

can

evaluate

response

and

adjoint

fast

=)

can

solve

least-squares

problem

fast

(=)

can

solve

convex

problem

fast)

�don't

need

exact

solution

for

interior-point

m

ethods

(allow

s

early

term

ination)

�preconditioning

(problem

speci�c)

E

xam

ples:

�FIR

�lter:

fast

(Nlog

N)

convolution

�Input

design:

system

state,co-state

sim

ulation

31

FIR

�lter

design

exam

ple

(M

.G

rant)

�forw

ard,adjoint

operator:

FFT

�#

taps

�2

�#

variables

�#

constraints

�10

�#

variables

�>1000

variables,

>10000

constraints

solved

in

4

m

in,4M

b

10-1

100

101

102

103

104

105

LSS

OL

MC

OE

101

102

103

101

102

103

104

105

Time

#variables

Memory32

M

IM

O

inputdesign

exam

ple

(M

.G

rant)

�3

inputs,8

outputs,

8

states

�am

plitude

lim

its

on

inputs

�slew

lim

its

on

3

outputs

�m

inim

ize

peak

tracking

error

on

5

outputs

-1

-0.50

0.51

1.52 0

12

34

56

78

910

-0.20

0.2

0.4

0.6

0.81

1.2

ui(t)

t

yi(t) 33

M

IM

O

inputdesign

exam

ple

(M

.G

rant)

�forw

ard,adjoint

operator:

state,co-state

sim

ulation

�#

vbles

=

3

�#tim

e

steps

�#

constr

�7

�#

vbles

�>1500

variables,

>10000

constraints

solved

in

12

m

in,5M

b

LSS

OL

MC

OE

100

101

102

103

104

105 10

210

310

2

103

104

105

Time

#variables

Memory

34

Exam

ple:m

ultiple

Lyapunov

inequalities

(L.V

andenberghe)

Problem

:m

inim

ize

linear

fct

ofm

atrix

P2R

n�

n

subject

to:

AT iP+

PAi+

Qi�0;

i=

1;:::;L

Ai;Qi

given;Pis

the

variable

�num

ber

ofvariables:

m

� =

n(n+

1)=2

�cost

ofsolving

sim

ilar

least-squares

problem

,not

exploiting

problem

structure:

O(Lm3)

�cost

ofsolving

problem

w

ith

prim

al-dualm

ethod,exploiting

problem

structure:

O(L1:2m2)

�problem

s

w

ith

>1000

variables,>10000

constraints

solved

on

sm

allw

orkstation

in

few

m

inutes

35

Exploiting

structure

in

convex

problem

s

can

evaluate

response,adjoint

fast

(exploitingstructure)

+

can

solve

least-squares

problem

fast

(usingconjgrad)

+

can

solve

convex

problem

fast

(usingint-ptmethods)

36

M

ain

point

�M

any

problem

s

arising

in

engineering

analysis

and

design

can

be

cast

as

convex

optim

ization

problem

s

�H

ence,can

be

e�

ciently

solved

by

interior-point

m

ethodsthat

exploit

problem

structure

�A

s

available

com

puting

pow

er

increases,this

observation

becom

es

m

ore

relevant

�convex

problem

s

not

w

idely

enough

recognized

37

(A

few)references

�

NesterovandNemirovsky,Interior-pointpolynomialalgorithmsinconvex

programming,SIAM,1994.

�

VandenbergheandBoyd,PositiveDe�niteProgramming,submitted,SIAM

Review,avail.anon.ftp.

�

BoydandBarratt,Linearcontrollerdesign:Limitsofperformance,

Prentice-Hall,1991.

�

Boyd,ElGhaoui,FeronandBalakrishnan,

Linearmatrixinequalitiesinsystem

andcontroltheory,SIAM,1994.

�

BenTalandNemirovskii,Interiorpointpolynomial-timemethodfortruss

topologydesign,Technionreport,1992.

�

VandenbergheandBoyd,Primal-dualpotentialreductionmethodfor

problemsinvolvingmatrixinequalities,Math.Programming1994.

�

Sapatnekar,A

ConvexProgrammingApproachtoProblemsinVLSI

Design,PhDthesis,Univ.ofIllinois,1992.

38

:::the

great

watershed

in

optim

ization

isn't

between

linearity

and

nonlinearity,but

convexity

and

nonconvexity.

|

R

.R

ockafellar,SIA

M

R

eview

1993

39