Convex Analysis and Optimization Mit

7/27/2019 Convex Analysis and Optimization Mit

1/300

LECTURE SLIDES ONNVEX ANALYSIS AND OPTIMIZATIONCO

BASED ON 6.253 CLASS LECTURES AT THEMASS. INSTITUTE OF TECHNOLOGY

CAMBRIDGE, MASS

SPRING 2010

BY DIMITRI P. BERTSEKAShttp://web.mit.edu/dimitrib/www/home.html

Based on the bookConvexOptimizationTheory,Athena Scientific,2009,includingtheon-lineChapter6andsupplementarymaterialat

http://www.athenasc.com/convexduality.html

All figures are courtesy of Athena Scientific, and are used with permission..
http://web.mit.edu/dimitrib/www/home.htmlhttp://www.athenasc.com/convexduality.htmlhttp://www.athenasc.com/convexduality.htmlhttp://web.mit.edu/dimitrib/www/home.html


2/300

LECTURE 1TRODUCTION TO THE COURSEAN IN

LECTURE OUTLINE TheRoleofConvexityinOptimization DualityTheory AlgorithmsandDuality CourseOrganization


3/300

HISTORY AND PREHISTORY Prehistory:Early1900s- 1949. Caratheodory,Minkowski,Steinitz,Farkas.

Propertiesofconvexsetsandfunctions.Fenchel- Rockafellarera:1949- mid1980s.

Dualitytheory.Minimax/gametheory(vonNeumann).(Sub)differentiability,optimalityconditions,

sensitivity.Modernera- Paradigmshift:Mid1980s- present.

Nonsmooth analysis (a theoretical/esotericdirection).

Algorithms (a practical/high impact direction).Achangeintheassumptionsunderlyingthefield.


4/300

OPTIMIZATION PROBLEMSricform:Gene

minimize f(x)subjectto xC

Costfunctionf :n ,constraintsetC,e.g.,C=Xx|h1(x) = 0, . . . , hm(x) = 0

x|g1(x)0, . . . , gr(x)0 Continuousvsdiscreteproblemdistinction Convex programming problems are those forwhichf andC areconvex Theyarecontinuousproblems

Theyarenice,andhavebeautifuland intuitivestructure

However, convexity permeates all of optimiza

tion,includingdiscreteproblems Principal vehicle for continuous-discrete connectionisduality:

The dual problem of a discrete problem iscontinuous/convex

Thedualproblemprovidesimportantinformationforthesolutionofthediscreteprimal(e.g., lowerbounds,etc)


5/300

WHY IS CONVEXITY SO SPECIAL?AconvexfunctionhasnolocalminimathatarenotglobalAnonconvexfunctioncanbeconvexifiedwhile

maintainingtheoptimalityof itsglobalminima Aconvexsethasanonemptyrelativeinterior

A convex set is connected and has feasible directionsatanypoint Theexistenceofaglobalminimumofaconvexfunctionoveraconvexsetisconvenientlycharacterized intermsofdirectionsofrecession A polyhedral convex set is characterized intermsofafinitesetofextremepointsandextremedirections

Areal-valuedconvexfunctioniscontinuousandhasnicedifferentiabilityproperties Closed convexcones are self-dualwith respecttopolarity

Convex,lowersemicontinuousfunctionsareselfdualwithrespecttoconjugacy


6/300

DUALITY Twodifferentviewsofthesameobject. Example: Dualdescriptionofsignals.

Time domain Frequency domain

Dualdescriptionofclosedconvexsets

A union of points An intersection of halfspaces


7/300

DUAL DESCRIPTION OF CONVEX FUNCTION Defineaclosedconvexfunctionbyitsepigraph. Describetheepigraphbyhyperplanes. Associatehyperplaneswithcrossingpoints(theconjugatefunction).

x

Slope =y

0

(y, 1)

f(x)

infxn

{f(x) xy}= f(y)

Primal Description Dual Description

Values f(x) Crossing pointsf(y)


8/300

FENCHEL PRIMAL AND DUAL PROBLEMS

x x

f1(x)

f2(x)

Slope yf

1(y)

f

2 (

y)

f1

(y) +f2

(y)

Vertical Distances Crossing Point Differentials

Primal Problem Description Dual Problem Description

Primalproblem:minf1(x) +f2(x)

x

Dualproblem:( ) f fy 1 2 (y)max

yaretheconjugateswheref1 andf2


9/300

FENCHEL DUALITY

f1(x)Slope y

x

x

f2(x)

f1

(y)

f2

(y)

f1

(y) +f2

(y)

Slope y

minx

f1(x) +f2(x)

= max

y

f

1(y) f

2(y)

Underfavorableconditions(convexity): Theoptimalprimalanddualvaluesareequal The optimal primal and dual solutions are

related


10/300

A MORE ABSTRACT VIEW OF DUALITY Despite its elegance, the Fenchel framework issomewhatindirect. Fromdualityofsetdescriptions,to

dualityoffunctionaldescriptions,to dualityofproblemdescriptions.

Amoredirectapproach: Startwithaset,then

Definetwosimpleprototypeproblems dualtoeachother.

Avoid functional descriptions (a simpler, lessconstrainedframework).


11/300

MIN COMMON/MAX CROSSING DUALITY

0!

"#$

%#0 )1*22&'3 ,*&'- 4/

%

!

"5$

%

%#0 )1*22&'3 ,*&'- 4/

7

!

"8$

9

6%

%

%#0 )1*22&'3 ,*&'- 4/

%&' )*++*' ,*&'- ./

.

7

70 0

0

u u

u

w

MM

M

MMin CommonPoint w

Max CrossingPoint q

Max CrossingPoint q Max Crossing

Point q

(a) (b)

(c)

Allofdualitytheoryandallof(convex/concave)minimax theory can be developed/explained intermsofthisonefigure. Themachineryofconvexanalysis isneededtoflesh out this figure, and to rule out the exceptional/pathologicalbehaviorshownin(c).


12/300

ABSTRACT/GENERAL DUALITY ANALYSIS

Minimax Duality Constrained OptimizationDuality

Min-Common/Max-CrossingTheorems

Theorems of the

Alternative etc( MinMax = MaxMin )

Abstract Geometric Framework

Special choices

ofM

(Set M)


13/300

EXCEPTIONAL BEHAVIORIfconvexstructure isso favorable,what isthesourceofexceptional/pathologicalbehavior?

Answer: Somecommonoperationsonconvexsetsdonotpreservesomebasicproperties. Example: A linearly transformedclosedcon

vex set need not be closed (contrary to compactandpolyhedralsets).

Alsothevectorsumoftwoclosedconvexsetsneednotbeclosed.

x1

x2

C1=

(x1, x2)| x1 >0, x2 >0, x1x2 1

C2=

(x1, x2)| x1= 0

Thisisamajorreasonfortheanalyticaldifficul

ties in convex analysis and pathological behaviorinconvexoptimization(andthefavorablecharacterofpolyhedralsets).


14/300

MODERN VIEW OF CONVEX OPTIMIZATIONTraditionalview:Pre1990s LPsaresolvedbysimplexmethod NLPsaresolvedbygradient/Newtonmeth

ods ConvexprogramsarespecialcasesofNLPs

LP CONVEX NLP

Duality Gradient/NewtonSimplex

Modernview:Post1990s LPs are often solved by nonsimplex/convex

methods Convexproblemsareoftensolvedbythesame

methodsasLPs

KeydistinctionisnotLinear-NonlinearbutConvex-Nonconvex(Rockafellar)

LP CONVEX NLP

Simplex Gradient/NewtonDuality

Cutting plane

Interior pointSubgradient


15/300

THE RISE OF THE ALGORITHMIC ERA ConvexprogramsandLPsconnectaround Duality

Large-scalepiecewiselinearproblems Synergyof:

Duality

Algorithms Applications

Newproblemparadigmswithrichapplications Duality-baseddecomposition

Large-scaleresourceallocation Lagrangianrelaxation,discreteoptimization Stochasticprogramming

Conicprogramming

Robustoptimization Semidefiniteprogramming

Machinelearning Supportvectormachines l1 regularization/Robustregression/Compressed

sensing


16/300

METHODOLOGICAL TRENDSwmethods,renewedinterestinoldmethods.InteriorpointmethodsNe

Subgradient/incrementalmethodsPolyhedralapproximation/cuttingplanemeth

ods Regularization/proximalmethods Incrementalmethods

Renewedemphasisoncomplexityanalysis

Nesterov,Nemirovski,andothers ... Optimalalgorithms(e.g.,extrapolatedgra

dientmethods) Emphasisoninteresting(oftenduality-related)large-scalespecialstructures


17/300

COURSE OUTLINEillfollowcloselythetextbookrtsekas, Convex Optimization Theory, Wew Be

Athena Scientific,2009,includingtheon-lineChapter6andsupplementarymaterialathttp://www.athenasc.com/convexduality.html

Additionalbookreferences: Rockafellar,ConvexAnalysis,1970. BoydandVanderbergue,ConvexOptimiza

tion,CambridgeU.Press,2004. (On-lineathttp://www.stanford.edu/~boyd/cvxbook/)

Bertsekas,Nedic,andOzdaglar,ConvexAnalysisandOptimization,Ath.Scientific,2003. Topics (the texts design is modular, and thefollowingsequenceinvolvesnolossofcontinuity):

BasicConvexityConcepts: Sect.1.1-1.4. Convexity and Optimization: Ch.3. Hyperplanes&Conjugacy: Sect.1.5,1.6. Polyhedral Convexity: Ch.2. Geometric Duality Framework: Ch.4.

Duality Theory: Sect.5.1-5.3. Subgradients: Sect.5.4. Algorithms: Ch.6.
http://www.athenasc.com/convexduality.htmlhttp://www.stanford.edu/~boyd/cvxbook/http://www.stanford.edu/~boyd/cvxbook/http://www.athenasc.com/convexduality.html


18/300

WHAT TO EXPECT FROM THIS COURSE Requirements: Homework(25%),midterm(25%),andatermpaper(50%)

Weaim:To develop insight and deep understanding

ofafundamentaloptimizationtopicTotreatwithmathematicalrigoran impor

tantbranchofmethodologicalresearch,andtoprovideanaccountofthestateoftheartinthefield

Togetanunderstandingofthemerits,limitations,

and

characteristics

of

the

rich

set

of

availablealgorithms

Mathematicallevel: Prerequisites are linear algebra (preferably

abstract)andrealanalysis(acourseineach) Proofswillmatter ... buttherichgeometryofthesubjecthelpsguidethemathematics

Applications: They are many and pervasive ... but dont

expect much in this course. The book byBoyd and Vandenberghe describes a lot ofpracticalconvexoptimizationmodels

Youcandoyourtermpaperonanapplicationarea


19/300

A NOTE ON THESE SLIDESheseslidesareateachingaid,notatextontexpectarigorousmathematicaldevelop T D

ment The statements of theorems are fairly precise,buttheproofsarenot Manyproofshavebeenomittedorgreatlyabbreviated Figures are meant to convey and enhance understandingofideas,nottoexpressthemprecisely Theomittedproofsandafullerdiscussioncanbe found in the Convex Optimization Theorytextbookanditssupplementarymaterial


20/300

LECTURE 2LECTURE OUTLINE

Convexsetsandfunctions Epigraphs

Closedconvexfunctions

RecognizingconvexfunctionsReading: Section1.1


21/300

SOME MATH CONVENTIONSllofourworkisdoneinn: spaceofn-tuples Ax= (x1, . . . , xn)

Allvectorsareassumedcolumnvectors denotestranspose,soweusex todenotearowvector

xy istheinnerproductni=1xiyi ofvectorsxandy

x=xx isthe(Euclidean)normofx. Weusethisnormalmostexclusively

Seethe

textbook

for

an

overview

of

the

linear

algebraandrealanalysisbackgroundthatwewilluse. Particularlythefollowing:

Definitionofsupandinfofasetofrealnumbers

Convergenceofsequences(definitionsofliminf,

limsup of a sequence of real numbers, anddefinitionof limofasequenceofvectors)

Open, closed, and compact sets and theirproperties

Definitionandpropertiesofdifferentiation


22/300

CONVEX SETS

x+ (1 )y, 0 1

yx x

y

x

y

x

y

AsubsetC ofn iscalledconvex ifx+(1)yC, x,yC, [0,1]

OperationsthatpreserveconvexityIntersection,scalarmultiplication,vectorsum,

closure, interior, lineartransformations Specialconvexsets: Polyhedralsets: Nonemptysetsoftheform

{x|ajxbj, j= 1, . . . , r}(alwaysconvex,closed,notalwaysbounded)

Cones: Sets C such that x C for all > 0 and x C (not always convex orclosed)


23/300

CONVEX FUNCTIONS

! "#$% ' #( ) ! %"#*%

$ *

+

"#! $ ' #( ) ! %*%

! $ ' #( ) ! %*

"#$%

"#*%

x+ 1 )y

C

x y

f(x)

f(y)

f(x) + (1 )f(y)

fx+ (1 )y

Let C be a convex subset of n. A functionf :C iscalledconvex ifforall[0,1]fx+(1)y

f(x)+(1

)f(y),

x,y

CIfthe inequality isstrictwhenevera(0,1)andx=y,thenf iscalledstrictlyconvexoverC.

Iff isaconvex function,thenall its levelsets{

x

C| f(x)

} and

{x

C

| f(x) <

},

where isascalar,areconvex.


24/300

EXTENDED REAL-VALUED FUNCTIONS!"#$

#

%&'()# !+',-.&'

!"#$

#

/&',&'()# !+',-.&'

01.23415 01.23415f(x) f(x)

xx

Epigraph Epigraph

Convex function Nonconvex function

dom(f) dom(f)

Theepigraphofafunctionf :X [,]isthesubsetofn+1 givenby

epi(f) =(x,w)|xX,w , f(x)wTheeffectivedomainoff istheset

dom(f) =xX |f(x)forallxX andX isconvex,thedefinitioncoincideswiththeearlierone. Wesaythatf isclosedifepi(f)isaclosedset. We say that f is lower semicontinuous at avectorxX iff(x)liminfkf(xk)foreverysequence{xk} X withxk x.


25/300

CLOSEDNESS AND SEMICONTINUITY I Proposition: Forafunctionf :n [,],thefollowingareequivalent:

(i) V ={x|f(x)} isclosedforall .(ii) f islowersemicontinuousatallx n.

(iii) f isclosed.f(x)

xx| f(x)

epi(f)

(ii) (iii): Let (xk, wk) epi(f) with (xk, wk) (x,w). Thenf(xk)wk,and

f(x)liminff(xk)w so(x,w)epi(f)k(iii) (i): Let{xk} V and xk x. Then

(xk, ) epi(f) and (xk, ) (x,), so (x,)epi(f),andxV.

(i)

(ii):

If

xk

x

and

f(x)

> >

lim

infk

f(xkconsidersubsequence{xk}K xwithf(xk)

- contradictsclosednessofV.


26/300

CLOSEDNESS AND SEMICONTINUITY II Lowersemicontinuityofafunctionisadomain-specificproperty,butclosenessisnot:

Ifwechangethedomainofthefunctionwithoutchangingitsepigraph,itslowersemicontinuitypropertiesmaybeaffected.

Example: Definef :(0,1) [,]andf :[0,1] [,]by f(x) = 0, x(0,1),

f(x) =0 ifx

(0,1),

ifx= 0orx=1.Thenf and fhavethesameepigraph, andbotharenotclosed. Butf islower-semicontinuouswhilefisnot.

Notethat: Iffislowersemicontinuousatallxdom(f),

it isnotnecessarilyclosed Iffisclosed,dom(f)isnotnecessarilyclosed

Proposition: Let f :X [,] be a function. If dom(f) is closedandf is lowersemicontinuousatallxdom(f),thenf isclosed.


27/300

ROPERANDIMPROPERCONVEXFUNCTION

f(x) f(x)

xdom(f) dom(f)

x

epi(f) epi(f)

NotClosedImproperFunction ClosedImproperFunction

Wesaythatf isproperiff(x)forallxX,andwewillcallf improperifit isnotproper. Notethatf isproperifandonlyifitsepigraphisnonemptyanddoesnotcontainaverticalline. Animproperclosedconvexfunctionisverype-culiar: it takes an infinite value ( or) at everypoint.


28/300

RECOGNIZING CONVEX FUNCTIONS Some important classes of elementary convexfunctions: Affine functions, positive semidefinitequadraticfunctions,normfunctions,etc. Proposition: Letfi :n (,],iI,begivenfunctions(I isanarbitraryindexset).(a)Thefunctiong:n (,]givenby

g(x) =1f1(x) + +mfm(x), i >0 isconvex(orclosed) iff1, . . . , f m areconvex(respectively,closed).(b)Thefunctiong:n (,]givenby

g(x) =f(Ax)whereAisanmnmatrix isconvex(orclosed)iff isconvex(respectively,closed).(c)Thefunctiong:n (,]givenby

g(x)=supfi(x)iI

isconvex(orclosed) ifthefi areconvex(respectively,closed).


29/300


DifferentiableConvexFunctionsConvexandAffineHulls

CaratheodorysTheorem

RelativeInteriorReading: Sections1.1,1.2,1.3.0


30/300

DIFFERENTIABLE CONVEX FUNCTIONS

zx

f(z)

f(x) +f(x)(z x)

LetC

n beaconvexsetandletf :

n

bedifferentiableovern.

(a) Thefunctionf isconvexoverC ifff(z)f(x) + (zx)f(x), x,zC

(b) If the inequality is strict whenever x = z,thenf isstrictlyconvexoverC.


31/300

PROOF IDEAS

z

x

x

f(x) + (z x)f(x)

f(z)

f(z)

f(x) + (1 )f(y)

f(x)

(y)

z = x+ (1 )y y

f(z) + (y z)f(z)

f(z) + (x z)f(z)

(a)

(b)

x+ (z x)

f(x) +fx+ (z x)

f(x)


32/300

OPTIMALITY CONDITION LetC beanonemptyconvexsubsetofn andlet f :n be convex and differentiable overanopensetthatcontainsC. Thenavectorx Cminimizesf overC ifandonly if

f(x)(x

x)

0,

x

C.

Proof: Iftheconditionholds,thenf(x)f(x)+(xx)f(x)f(x), xC,sox minimizesf overC.

Converse: Assumethecontrary,i.e.,x minimizesf overCandf(x)(xx)


33/300

TWICE DIFFERENTIABLE CONVEX FNS Let C be a convex subset of n and let f :n betwicecontinuouslydifferentiableover

.n(a) If2f(x) ispositivesemidefiniteforallx

C,thenf isconvexoverC.(b) If2f(x) is positive definite for all x C,

thenf isstrictlyconvexoverC.(c) If C is open and f is convex over C, then

2f(x)ispositivesemidefiniteforallxC.Proof: (a)Bymeanvaluetheorem,forx,y

C

f(y) =f(x)+(yx)f(x)+12(yx)2f

x+(yx)

(yx)

for some [0,1]. Using the positive semidefinitenessof2f,weobtain

f(y)f(x) + (yx)f(x), x,yCFromtheprecedingresult,f isconvex.(b) Similar to (a), we have f(y) > f(x) + (yx)

f(x) forallx,y

C withx=

y,andweuse

theprecedingresult.(c)Bycontradiction ... similar.


34/300

CONVEX AND AFFINE HULLS

GivenasetX n:A convex combination of elements of X is a

mvectoroftheform ixi,wherexi X,i 0,andmi=1i =1. i=1

TheconvexhullofX,denotedconv(X), isthe

intersectionofallconvexsetscontainingX. (CanbeshowntobeequaltothesetofallconvexcombinationsfromX). TheaffinehullofX,denotedaff(X),istheintersectionofallaffinesetscontainingX (anaffineset is a set of the form x+S, where S is a subspace). AnonnegativecombinationofelementsofX isavectoroftheformm ixi,wherexi Xandi=1i0foralli. TheconegeneratedbyX,denotedcone(X), isthesetofallnonnegativecombinationsfromX:

Itisaconvexconecontainingtheorigin. Itneednotbeclosed!

If X is a finite set, cone(X) is closed (nontrivialtoshow!)


35/300


36/300

PROOF OF CARATHEODORYS THEOREM(a) Let x be a nonzero vector in cone(X), andletmbethesmallest integersuchthatxhastheformm ixi, where i > 0 and xi X fori=1all i = 1, . . . , m. If the vectors xi were linearlydependent,therewouldexist1, . . . , m,with

mixi = 0i=1

andat leastoneofthei ispositive. Considerm

(ii)xi,i=1

where

is

the

largest

such

that

ii 0foralli. Thiscombinationprovidesarepresentation

ofxasapositivecombinationoffewerthanmvectorsofXacontradiction. Therefore,x1, . . . , xm,arelinearly independent.(b)Useliftingargument: applypart(a)toY =

(x,1)|xX.

Y

x

X

0

1(x, 1)

n


37/300

AN APPLICATION OF CARATHEODORY

Theconvexhullofacompactsetiscompact.Proof: Let X be compact. We take a sequenceinconv(X)andshowthatithasaconvergentsubsequencewhoselimit is inconv(X).

ByCaratheodory,asequenceinconv(X)canbeexpressedasn+1 ,whereforallkandi=1 ikxik

ki, k 0, x X, andn+1k =1. Since thei i i=1 isequence

k k k(1

k, . . . , n+1, x1, . . . , xn+1)

isbounded,ithasalimitpoint

(1, . . . , n+1, x1, . . . , xn+1)

,which must satisfy n+1i = 1, and i 0,i=1xi X foralli.

The vectorn+1ixi belongs to conv(X)i=1and is a limit point of n+1k k, showingi=1 ixithatconv(X)iscompact. Q.E.D.

Note that the convex hull of a closed set neednotbeclosed!


38/300

RELATIVE INTERIOR

a relative interior point of C, if x irpointofC relativetoaff(C). x is s aninterio ri(C)denotestherelativeinteriorofC,i.e.,thesetofallrelativeinteriorpointsofC. LineSegmentPrinciple: IfCisaconvexset,

x ri(C) and x cl(C), then all points on thelinesegmentconnectingxandx,exceptpossiblyx,belongtori(C).

x

C x = x+(1

)x

x

S

S

ProofofcasewherexC: Seethefigure. Proof of case where x

/ C: Take sequence

{xk} C withxk x. Argueasinthefigure.


39/300

ADDITIONAL MAJOR RESULTS LetC beanonemptyconvexset.

(a) ri(C)isanonemptyconvexset,andhasthesameaffinehullasC.

(b)Prolongation Lemma: x ri(C) if andonly if every line segment in C having xasoneendpointcanbeprolongedbeyondxwithout leavingC.

z2

C

X

z1

z1 and z2 are linearlyindependent, belong to

Cand span aff(C)

0

Proof: (a)Assumethat0C. Wechoosemlinearly independent vectors z1, . . . , zm

C, where

misthedimensionofaff(C),andwe letm m

X=

izi i 0, i= 1, . . . , m

i=1 i=1

(b)=>isclearbythedef.ofrel.interior. Reverse:takeanyxri(C);useLineSegmentPrinciple.


40/300

OPTIMIZATION APPLICATION

Aconcavefunctionf : n thatattainsits minimum over a convex set X at an x ri(X)mustbeconstantoverX.

X

x

x

x

aff(X)

Proof: (By contradiction) Let x X be suchthat f(x) > f(x). Prolong beyond x the linesegment x-to-x to a point x

X. By concavity

off,wehaveforsome(0,1)f(x)f(x)+(1)f(x),

and since f(x) > f(x), we must have f(x) >f(x)- acontradiction. Q.E.D. Corollary: Alinearfunctioncanattainamininumonlyattheboundaryofaconvexset.


41/300


Algebraofrelativeinteriorsandclosures Continuityofconvexfunctions

Closuresoffunctions

Recessionconesand linealityspaceReading: Sections1.31-1.3.3,1.4.0


42/300

CALCULUS OF REL. INTERIORS: SUMMARY The ri(C) and cl(C) of a convex set C differverylittle.

Any set between ri(C) and cl(C) has thesamerelativeinteriorandclosure.

Therelativeinteriorofaconvexsetisequalto

the

relative

interior

of

its

closure.

Theclosureoftherelative interiorofacon

vexset isequaltoitsclosure.Relative interior and closure commute with

Cartesianproductand inverse imageundera linear

transformation.

Relativeinteriorcommuteswithimageunderalineartransformationandvectorsum,butclosuredoesnot.

Neither relative interior nor closure commutewithset intersection.


43/300

CLOSURE VS RELATIVE INTERIOR Proposition:

(a) Wehavecl(C)=clri(C)andri(C)=ricl(C).(b) LetCbeanothernonemptyconvexset. Then

thefollowingthreeconditionsareequivalent:(i)

C

and

C

have

the

same

rel.

interior.

(ii) C andC havethesameclosure.

(iii) ri(C)Ccl(C).Proof: (a)Sinceri(C)C,wehavecl

ri(C)

cl(C). Conversely, let x

cl(C). Let x

ri(C).BytheLineSegmentPrinciple,wehave

x+(1)xri(C), (0,1].Thus,xisthelimitofasequencethatliesinri(C),so

x

cl

ri(C)

.

x

x

C

Theproofofri(C)=ricl(C) issimilar.


44/300

LINEAR TRANSFORMATIONS LetC beanonemptyconvexsubsetofn andletAbeanm nmatrix.

(a) WehaveA ri(C)=ri(A C). (b) WehaveA cl(C)cl(A C). Furthermore,

ifC isbounded,thenA cl(C)=cl(A C).

Proof: (a)Intuition: SphereswithinCaremappedonto spheres within A C (relative to the affinehull).(b)WehaveAcl(C)cl(A C),sinceifasequence {

xk}

C converges to some x

cl(C) then thesequence{Axk},whichbelongstoA C,convergestoAx,implyingthatAxcl(A C).

To show the converse, assuming that C isbounded, choose any z cl(A C). Then, thereexists{xk} C such that Axk z. Since C isbounded,

{xk}

has

asubsequence

that

converges

tosomexcl(C),andwemusthaveAx=z. ItfollowsthatzA cl(C). Q.E.D.

Notethat ingeneral,wemayhaveA int(C)=int(A C), A cl(C)=cl(A C)


45/300

INTERSECTIONS AND VECTOR SUMS LetC1 andC2 benonemptyconvexsets.

(a) Wehaveri(C1 +C2)=ri(C1)+ri(C2),cl(C1)

+

cl(C2)

cl(C1 +C2)IfoneofC1 andC2 isbounded,then

cl(C1)+cl(C2)=cl(C1 +C2)(b) Ifri(C1)ri(C2) =,then

ri(C1C2)=ri(C1)ri(C2),cl(C1C2)=cl(C1)cl(C2)

Proof of (a): C1 +C2 istheresultofthe lineartransformation(x1, x2)x1 +x2. Counterexamplefor(b):

C1 ={x|x0}, C2 ={x|x0}


46/300

CARTESIAN PRODUCT - GENERALIZATION LetC beconvexsetinn+m. Forx n,let

Cx ={y|(x,y)C},andlet

D={

x|Cx =

}.

Thenri(C) =(x,y)|xri(D), yri(Cx).

Proof:

SinceD

is

projection

of

C

on

x-axis,

ri(D) =x|thereexistsy m with(x,y)ri(C),sothat

ri(C) =xri(D)Mxri(C),

where Mx = (x,y) | y m. For every x ri(D),wehaveMx

ri(C)=ri(MxC) =(x,y)|yri(Cx).Combinetheprecedingtwoequations. Q.E.D.


47/300


48/300

CLOSURES OF FUNCTIONSTheclosure ofa function f :X [,n ] isefunctionclf : [,]with

thepi(clf)=clepi(f)

Theconvexclosureoff isthefunctionclf withepi(cl

f)=clconvepi(f)

Proposition: Foranyf :X [,]

inf f(x)= inf (clf)(x)= inf (clf)(x).xX xn xn

Also,anyvectorthatattainstheinfimumoffoverX alsoattainstheinfimumofclf andclf. Proposition: Foranyf :X [,]:

(a) clf (orclf)isthegreatestclosed(orclosedconvex,resp.) functionmajorizedbyf.

(b) If f is convex, then clf is convex, and it isproperifandonlyiff isproper. Also,(clf)(x) =f(x), xridom(f),andifx

ridom(f)

andy

dom(clf),

(clf)(y)=limfy+(xy). 0


49/300

RECESSION CONE OF A CONVEX SET

Given a nonempty convex set C, a vector d isadirectionof recession if starting at any x in Candgoing indefinitelyalongd,wenevercrosstherelativeboundaryofC topointsoutsideC:

x+d

C,

x

C,

0

x

C

0

d

x + d

Recession Cone RC

RecessionconeofC (denotedbyRC): Thesetofalldirectionsofrecession. RC isaconecontainingtheorigin.


50/300

RECESSION CONE THEOREM LetC beanonemptyclosedconvexset.

(a) The recession coneRC is a closed convexcone.

(b) AvectordbelongstoRC ifandonlyifthereexistssomevectorx

C suchthatx+d

C forall0.(c)RC containsanonzerodirection ifandonly

ifC isunbounded.(d) TherecessionconesofCandri(C)areequal.(e)

If

D

is

another

closed

convex

set

such

that

CD= ,wehave

RCD =RC RDMore generally, for any collection of closedconvexsetsCi,iI,whereI isanarbitraryindexsetandiICi isnonempty,wehave

RiICi =iIRCi


51/300

PROOF OF PART (B)

x

C

z1 = x+ d

z2

z3

x

x+ d

x+ d1

x+ d2

x+ d3

Let d = 0 be such that there exists a vectorx

C with x+d

C for all

0. We fix

xC and >0,andweshowthatx+dC.Byscalingd,itisenoughtoshowthatx+dC.

Fork= 1,2, . . .,let(zkx)

zk =x+kd, dk =zk

x

d

Wehavedk zk x d xx zk x xx

= + , 1, 0,d zk x d zk x zk x zk x

sodk dandx+dk x+d. Usetheconvexity andclosednessofC toconcludethatx+dC.


52/300


53/300

LECTURE 5

LECTURE OUTLINEDirectionsofrecessionofconvexfunctions

Localandglobalminima

ExistenceofoptimalsolutionsReading: Sections1.4.1,3.1,3.2


54/300

DIRECTIONS OF RECESSION OF A FNWeaimtocharacterizedirectionsofmonotonicdecreaseofconvexfunctions.

Somebasicgeometricobservations: Thehorizontal directions intherecession

coneoftheepigraphofaconvex functionfaredirectionsalongwhichthe levelsetsareunbounded.

Along these directions the level sets xf(x) are unbounded and f is mono-|tonicallynondecreasing.

Thesearethedirectionsofrecessionoff.

!

epi(f)

Level Set V!= {x | f(x) "!}

Slice {(x,!) | f(x) "!}

RecessionCone of f

0


55/300

RECESSION CONE OF LEVEL SETS Proposition: Letf :n (,]beaclosedproperconvexfunctionandconsiderthelevelsetsV =x|f(x),where isascalar. Then:

(a) AllthenonemptylevelsetsV havethesamerecessioncone:

RV =d|(d,0)Repi(f)(b) IfonenonemptylevelsetV iscompact,then

alllevelsetsarecompact.Proof: (a)JusttranslatetomaththefactthatRV =thehorizontaldirectionsofrecessionofepi(f)

(b)Followsfrom(a).


56/300

RECESSION CONE OF A CONVEX FUNCTION For a closed proper convex function f :n (,],the(common)recessionconeofthenonemptylevel sets V =x |f(x), , is the recessionconeoff,andisdenotedbyRf.

0

Recession ConeRf

Level Sets off

Terminology: dRf: adirectionofrecessionoff. Lf =Rf (Rf): the linealityspaceoff. dLf: adirectionofconstancyoff. Example: Forthepos.semidefinitequadratic

f(x) =xQx+ax+b,therecessionconeandconstancyspaceareRf={d|Qd= 0, ad0}, Lf ={d|Qd= 0, ad= 0}


57/300


58/300

DESCENT BEHAVIOR OF A CONVEX FN!"# % & '(

&

!"#(

"&(

!"# % & '(

&

!"#(

")(

!"# % & '(

&

!"#(

"*(

!"# % & '(

&

!"#(

"+(

!"# % & '(

&

!"#(

",(

!"# % & '(

&

!"#(

"!(

f(x)

f(x)

f(x)

f(x)

f(x)

f(x)

f(x+ d)

f(x+ d) f(x+ d)

f(x+ d)

f(x+ d)f(x+ d)

rf(d) = 0

rf(d) = 0 rf(d) = 0

rf(d)< 0

rf(d) > 0 rf(d)> 0

y isadirectionofrecession in(a)-(d). This behavior is independent of the startingpointx,as longasxdom(f).


59/300

LOCAL AND GLOBAL MINIMA

Considerminimizingf : n ( , ]overa setX n xisfeasibleifxXdom(f) x isa(global)minimumoff overX ifx isfeasibleandf(x)=infx

X f(x)

x isalocal minimumoff overX ifx isaminimumoff overasetX {x| xx }Proposition: If X is convex and f is convex,then:

(a)

Alocal

minimum

of

fover

X

is

also

aglobal

minimumoff overX.

(b) If f is strictly convex, then there exists atmostoneglobalminimumoff overX.

f(x)

f(x) + (1 )f(x)

fx + (1 )x

0 x

x x


60/300

The set of minima of a proper f : n EXISTENCE OF OPTIMAL SOLUTIONS

(,] isthe intersectionof itsnonempty levelsets. Thesetofminimaoff isnonemptyandcompactifthelevelsetsoff arecompact. (AnExtensionofthe)WeierstrassTheorem: Thesetofminimaoff overX isnonemptyandcompactifX isclosed,f islowersemicontinuous over X, and one of the following conditionsholds:

(1) X isbounded.(2) Some setx X | f(x) is nonempty

andbounded.(3) For every sequence{xk} X s. t.xk

,wehavelimk

f(xk) =

.(Coercivity

property).Proof: In all cases the level sets of f X arecompact. Q.E.D.


61/300


62/300


63/300


64/300


65/300

ROLE OF CLOSED SET INTERSECTIONS I A fundamental question: Givenasequenceofnonemptyclosedsets{Ck} inn withCk+1

forallk,whenis nonempty?Ck k=0Ck Set intersection theorems are significant in atleast three major contexts, which we will discussin

what

follows:

1. Does a function f : n (,] attain aminimumoverasetX? Thisistrueifandonlyif

Intersectionofnonempty

xX |f(x)k

isnonempty.

Optimal

Solution

Level Sets off

X


66/300

ROLE OF CLOSED SET INTERSECTIONS II2. IfC isclosedandAisamatrix,isA C closed?Specialcase:

IfC1 andC2 areclosed, isC1 +C2 closed?

x

Nk

AC

C

y yk+1 yk

Ck

3. If F(x,z) is closed, is f(x) = infzF(x,z)closed?

(Critical

question

in

duality

theory.)

Can

beaddressedbyusingtherelation

Pepi(F)epi(f)clPepi(F)

whereP() isprojectiononthespaceof(x,w).


67/300

ASYMPTOTIC SEQUENCES Given nested sequence {Ck} of closed convexsets,{xk}isanasymptoticsequenceif

= 0, k= 0,1, . . .xk Ck, xk xk d

xk

,

xkdwheredisanonzerocommondirectionofrecessionofthesetsCk. AsaspecialcasewedefineasymptoticsequenceofaclosedconvexsetC (useCk C). Every unbounded {xk} with xk Ck has anasymptoticsubsequence. {xk} iscalledretractiveifforsomek,wehave

xkdCk, kk.

x0

x1x2

x3

x4 x5

0d

Asymptotic Direction

Asymptotic Sequence


68/300

RETRACTIVE SEQUENCES A nested sequence{Ck} of closed convex setsisretractiveifallitsasymptoticsequencesarere-tractive.

x0!"

!#

!$

!"

!#

!$

%&' )*+,&-+./*

"

%0' 123,*+,&-+./*

4

!"

!#

!$

!"

!$

53+*,6*-+.2353+*,6*-+.23

"

4

4

!#!7

C0

C0

C1

C1

C2

C2x0

x1

x1x2

x2

x3

(a) Retractive Set Sequence (b) Nonretractive Set Sequence

Intersection k=0

Ck Intersection k=0Ck

d

d

0

0

A closed halfspace (viewed asa sequencewithidenticalcomponents) isretractive. IntersectionsandCartesianproductsofretractivesetsequencesareretractive. A polyhedral set is retractive. Also the vectorsumofaconvexcompactsetandaretractiveconvexset isretractive. Nonpolyhedralconesandlevelsetsofquadraticfunctionsneednotberetractive.


69/300

SET INTERSECTION THEOREM IProposition: If

{Ck

}isretractive,then

Ck

k=0isnonempty. Keyproof ideas:

(a) Theintersection Ck isemptyiffthesek=0quence

{x

k}ofminimumnormvectorsofC

k

is unbounded (so a subsequence is asymptotic).

(b) An asymptotic sequence{xk} of minimumnorm vectors cannot be retractive, becausesuch a sequence eventually gets closer to 0whenshiftedoppositetotheasymptoticdirection.

x0

x1

x2x3

x4 x5

0d

Asymptotic Direction

Asymptotic Sequence


70/300


71/300


72/300


73/300


74/300


75/300


76/300


PartialMinimization Hyperplaneseparation Properseparation

NonverticalhyperplanesReading: Sections3.3,1.5


77/300


78/300


79/300

Let F :n+m PARTIAL MINIMIZATION THEOREM

(,] be a closed properconvexfunction,andconsiderf(x)=infzm F(x,z). Everyset intersectiontheoremyieldsaclosed-nessresult. Thesimplestcaseisthefollowing:

Preservation of Closedness Under Com

pactness: Ifthereexistx n, suchthattheset

z|F(x,z)isnonemptyandcompact,thenfisconvex,closed,andproper. Also,foreachx

dom(f),thesetof

minimaofF(x, )isnonemptyandcompact.

x

z

w

x1

x2

O

F(x, z)

f(x) = infz

F(x, z)

epi(f)

x

z

w

x1

x2

O

F(x, z)

f(x) = infz

F(x, z)

epi(f)


80/300

HYPERPLANES

a

x

Negative Halfspace

Positive Halfspace

{x| ax b}

{x| ax b}

Hyperplane

{x| ax= b}= {x| ax= ax}

Ahyperplaneisasetoftheform

{x

|ax=b

},

whereaisnonzerovector inn andbisascalar. We say that two sets C1 and C2 areseparatedbyahyperplaneH={x|ax=b}ifeachliesinadifferentclosedhalfspaceassociatedwithH,i.e.,either ax1 bax2, x1 C1,x2 C2,

or ax2 bax1, x1 C1, x2 C2 IfxbelongstotheclosureofasetC,ahyperplane that separates C andthe singleton set

{x}issaidbesupportingC atx.


81/300

VISUALIZATION Separatingandsupportinghyperplanes:

a

(a)

C1 C2

x

a

(b)

C

Aseparating{x|ax=b}thatisdisjointfromC1 andC2 iscalledstrictlyseparating:

ax1 < b < ax2, x1 C1, x2 C2

(a)

C1 C2

x

a

(b)

C1

C2x1

x2


82/300


83/300


84/300


85/300


86/300


87/300

NONVERTICAL HYPERPLANES A hyperplane in n+1 with normal (,) isnonvertical if=0. Itintersectsthe(n+1)staxisat= (/)u+w,where(u,w)isanyvectoronthehyperplane.

0 u

w

(,)

(u, w)

u+w

NonverticalHyperplane

VerticalHyperplane

(, 0)

Anonverticalhyperplanethatcontainstheepigraphofafunction in itsupperhalfspace,provides lowerboundstothefunctionvalues. Theepigraphofaproperconvexfunctiondoesnotcontainaverticalline,soitappearsplausiblethat it is contained in the upper halfspace ofsomenonverticalhyperplane.


88/300

NONVERTICAL HYPERPLANE THEOREM

Let C be a nonempty convex subset of n+1 thatcontainsnovertical lines. Then:(a)C iscontainedinaclosedhalfspaceofanon-

verticalhyperplane, i.e.,thereexist n, with = 0, and such thatu

+

w

forall(u,w)C.(b) If (u,w)/ cl(C), there exists a nonvertical

hyperplanestrictlyseparating(u,w)andC.Proof: Notethatcl(C)containsnovert.line[sinceC contains no vert. line, ri(C) contains no vert.line,andri(C)andcl(C)havethesamerecessioncone]. Sowejustconsiderthecase: C closed.(a) C is the intersection of the closed halfspacescontainingC. Ifallthesecorrespondedtoverticalhyperplanes,C wouldcontainavertical line.(b)Thereisahyperplanestrictlyseparating(u,w)andC. Ifitisnonvertical,wearedone,soassumeit isvertical. Addtothisverticalhyperplaneasmall -multiple of a nonvertical hyperplane containingC inoneofitshalfspacesasper(a).


89/300


Convexconjugatefunctions Conjugacytheorem Examples

SupportfunctionsReading: Section1.6


90/300

CONJUGATE CONVEX FUNCTIONS Considerafunctionf anditsepigraphNonverticalhyperplanessupportingepi(f)

Crossingpointsofverticalaxis

f

(y)=

sup

xy

f(x)

, y

n.

xn

x

Slope =y

0

(y, 1)

f(x)

infxn

{f(x) xy}= f(y)

Foranyf :n [,],itsconjugateconvexfunction isdefinedby

f(y)= supxyf(x), y nxn


91/300


92/300

CONJUGATE OF CONJUGATEFromthedefinition

f(y)= supxyf(x), y n,xn

notethatf isconvexandclosed. Reason: epi(f)istheintersectionoftheepigraphsofthelinearfunctionsofy

xyf(x)asxrangesover

n.

Considertheconjugateoftheconjugate:f(x)= supyxf(y), x n.

yn

f isconvexandclosed. Important fact/Conjugacy theorem: If fisclosedproperconvex,thenf =f.


93/300

CONJUGACY THEOREM - VISUALIZATIONf(y)= sup x

y

f(x) , y

n

x n f(x)= supyxf(y), x n

yn

Iff isclosedconvexproper,thenf =f.

x

Slope =y

0

f(x)(y, 1)

infxn

{f(x) xy}= f(y)y

x

f

(y)

f(x) = supyn

yx f(y)

H=

(x,w)| w xy= f(y)

Hyperplane


94/300


95/300

PROOFOFCONJUGACYTHEOREM(A), (C) (a) For all x,y, we have f(y) yxf(x),implyingthatf(x)sup y{yxf (y)}=f (x). (c)Bycontradiction. Assumethere is(x,)epi(f)with(x,)/epi(f). Thereexistsanon-verticalhyperplanewithnormal(y,1)thatstrictlyseparates

(x,

)

and

epi(f).

(The

vertical

componentofthenormalvectorisnormalizedto-1.)

Consider two parallel hyperplanes, translatedto pass through x,f(x) and x,f(x) . Theirvertical crossing points are xyf(x) and xyf(x),andliestrictlyaboveandbelowthecrossingpointofthestrictlysep.hyperplane. Hence

xyf(x)> xyf(x)whichcontradictspart(a). Q.E.D.

x

epi(f)(y, 1)

x,f(x)

epi(f) (x,)

x,

f

(x)

0

xyf(x)xyf(x)


96/300


97/300


98/300


99/300


100/300


Mincommon/maxcrossingduality Weakduality SpecialCases

Constrainedoptimizationandminimax StrongdualityReading: Sections4.1,4.2,3.4


101/300


102/300


103/300

MATHEMATICAL FORMULATIONS

Optimal value of the min common problem:w = inf w

(0,w)M

u

w

M

M(,1)

(,1)

q

q() = inf (u,w)M

w+ u}

0

Dual function value

Hyperplane H, =

(u,w)| w + u=

w

Math formulation of the max crossingproblem: Focus on hyperplanes with normals(,1)whosecrossingpoint satisfies

w+u, (u,w)MMaxcrossingproblemistomaximizesubjecttoinf(u,w)M{w+u}, n,or

maximize q() = inf)M

{w+u}(u,w

subjectto . n


104/300


105/300


106/300

GENERAL OPTIMIZATION DUALITY Considerminimizingafunctionf :n [,]. LetF : n+r [ , ]beafunctionwith

f(x) =F(x,0), x n Considertheperturbationfunction

p(u)= inf F(x,u)xnandtheMC/MCframeworkwithM =epi(p)

Themincommonvaluew isw

=p(0)= inf F(x,0) = inf f(x)xn xn

Thedualfunction isq()= inf p(u)+u= inf F(x,u)+u

ur (x,u)n+rso

q(

) =F

(0,

),where

F istheconjugate

ofF,viewedasafunctionof(x,u) Since

q = sup q() = inf F(0,) = inf F(0, ),r r r

wehavew = inf F(x,0) inf F(0, ) =q

xn r


107/300

CONSTRAINED OPTIMIZATION Minimizef :n overtheset

C=xX |g(x)0,whereX n andg:n r. Introduceaperturbedconstraintset

Cu =xX |g(x)u, u r,andthefunction

f(x) ifxCu,F(x,u) = otherwise,

whichsatisfiesF(x,0)=f(x)forallxC. Considerperturbationfunction

p(u)=

inf

F

(x,

u)

=

inf

f(x),

xn xX,g(x)u

andtheMC/MCframeworkwithM =epi(p).


108/300

CONSTR. OPT. - PRIMAL AND DUAL FNS Perturbationfunction(orprimalfunction)

p(u)= inf F(x,u)= inf f(x),xn xX,g(x)u

0 u

(g(x), f(x))| x X

M= epi(p)

w =p(0)

p(u)

q

IntroduceL(x,) =f(x) +g(x). Thenq()= inf p(u) +u

r=

uinf f(x) +u

ur, xX,g(x)uinfxX L(x,) if0,= otherwise.


109/300

LINEAR PROGRAMMING DUALITY

Considerthelinearprogramminimize cxsubjectto ajxbj, j= 1,...,r,

wherec

n,aj

n,andbj

,j= 1, . . . , r. For0,thedualfunctionhastheform

q()= inf L(x,)xn

j(bj ajx)r

b ifj=1ajj =c,= otherwise

Thusthedualproblem ismaximize b

rsubjectto ajj =c, 0.

j=1

r

inf cx+=xn j=1


110/300

Given

MINIMAX PROBLEMS:X Z , where X n, Z m

considerminimize sup(x,z)

zZsubjectto xX

ormaximize

inf

(x,

z)

xX

subjectto zZ. Some importantcontexts:

Constrainedoptimizationdualitytheory

Zerosumgametheory Wealwayshave

sup inf (x,z) inf sup(x,z)zZxX xX zZ

Key question: Whendoesequalityhold?


111/300

CONSTRAINED OPTIMIZATION DUALITY Fortheproblem

minimize f(x)subjectto xX, g(x)0

introducetheLagrangianfunctionL(x,) =f(x) +g(x)

Primalproblem(equivalenttotheoriginal)

min supL(x,) =xX 0

f(x) ifg(x)

0, otherwise,

Dualproblemmax inf L(x,)

0 x

X

Keydualityquestion: Is ittruethat?

inf supL(x,) =w q =sup inf L(x,)xn

0 =

0xn


112/300


113/300


114/300

VISUALIZATION

x

z

Curve of maxima

Curve of minima

f (x,z)

Saddle point

(x*,z*)

^f(x(z),z)

f(x,z(x))^

The

curve

of

maxima

f(x,z(x))

lies

above

the

curveofminimaf(x(z), z),wherez(x)=argmaxf(x,z), x(z)=argminf(x,z)

z xSaddle points correspond to points where thesetwocurvesmeet.


115/300

MINIMAX MC/MC FRAMEWORK Introduce perturbation function p : m [,]

p(u)= inf sup(x,z)uz,xX zZ u m

ApplytheMC/MCframeworkwithM =epi(p)Introduceclf,theconcaveclosureoffWehave

sup(x,z)= sup(cl)(x,z),zZ zm

sow =p(0)= inf sup(cl)(x,z).

xX zm

Thedualfunctioncanbeshowntobe

q() =inf(cl)(x,),xX m

soif(x, )isconcaveandclosed,w

= inf sup (x,z), q = sup inf (x,z)xX zm zm xX


116/300


117/300


MinCommon/MaxCrossingdualitytheorems Strongdualityconditions Existenceofdualoptimalsolutions

NonlinearFarkas lemmaReading: Sections4.3,4.4,5.1

0!

"#$

%#0 )1*22&'3 ,*&'- 4/

%

!

"5$

%

%#0 )1*22&'3 ,*&'- 4/

7

!

"8$

9

6%

%

%#0 )1*22&'3 ,*&'- 4/

%&' )*++*' ,*&'- ./

.

7

70 0

0

u u

u

w

MM

M

MMin CommonPoint w

Max CrossingPoint q

Max CrossingPoint q Max Crossing

Point q

(a) (b)

(c)


118/300


119/300


120/300

PROOFOFTHEOREM I

Assumethatq =w . Let (uk, wk) M besuchthatuk 0. Then,

nq() = inf {w+u} wk+uk, k, (u,w)M

Taking the limit as k , we obtain q() n

liminfkwk,forall ,implyingthat w =q = sup q() liminfwk

n kConversely, assume that for every sequence

(uk, wk) M with uk 0, there holds w

liminfkwk. If w = , then q = , by weakduality,soassumethat < w . Steps:

Step1: (0, w )/cl(M)forany >0.

w

u

w

M

M(uk, wk)

(uk+1, wk+1)w (uk,

0

wk)(uk+1,liminfk wk+1) wk


121/300

PROOFOFTHEOREMI(CONTINUED) Step2: M doesnotcontainanyverticallines.If this were not so, (0,1) would be a directionof recession of cl(M). Because (0, w ) cl(M),theentirehalfline (0, w )|0 belongstocl(M),contradictingStep1. Step3: Forany >0,since(0, w )/cl(M),thereexistsanonverticalhyperplanestrictlysepa-rating(0, w )andM. Thishyperplanecrossesthe(n +1)staxisatavector(0, )withw

w, so w q w . Since can be arbitrarilysmall, itfollowsthatq =w .

u

w

M

M(0, w )

(0, w )0

q()(0, )

(, 1)

StrictlySeparatingHyperplane


122/300

PROOF OF THEOREM II

Notethat(0, w)isnotarelativeinteriorpointof M. Therefore, by the Proper Separation Theorem, there is a hyperplane that passes through(0, w),containsM inoneofitsclosedhalfspaces,butdoesnotfullycontainM,i.e.,forsome(,) =(0,0)

w u+w, (u,w)M ,w 0, and we canassume

that

=

1.

It

follows

that

w

(u,winf)M{u+w}=q()q

Since the inequality q w holds always, wemust

have

q() =

q

=

w.


123/300

NONLINEARFARKASLEMMAn : X Let X , f : X , and gj ,j= 1, . . . , r,beconvex. Assumethat

f(x)0, xX withg(x)0Let

Q = |0, f(x) + g(x)0, xX .ThenQ isnonemptyandcompact ifandonly if

there exists a vector x X such that gj(x)


124/300

PROOFOFNONLINEARFARKASLEMMA ApplyMC/MCtoM = (u,w)|there isxX s.t.g(x)u,f(x)w

(, 1)0 u

w

(0, w )

D

suchthatg(x)u, f(x)w

(g(x), f(x))|x X

h th tM =

(u, w)|thereexistsx X

g(x), f(x)

( )

M isequaltoM andisformedastheunionofpositiveorthantstranslatedtopoints g(x), f(x) ,xX. TheconvexityofX,f,andgj impliesconvexityofM. MC/MCTheoremIIapplies: wehaveD= u|thereexistsw with(u,w)M

and0int(D),because (g(x), f(x) M.


125/300


126/300

MC/MC TH. III - POLYHEDRAL Consider the MC/MC problems, and assumethat< w and:

(1) M isahorizontaltranslationofM byP,M =M (u,0)|uP ,

whereP: polyhedralandM: convex.

0} u

Mw

u0}

w

w(,1)

q()u0}

w

M =M (u,0)|uP

P

(2) Wehaveri(D)P =,whereD = u|thereexistsw with(u,w)M}

Then q = w, there is a max crossing solution,and all max crossing solutions satisfy d 0foralldRP. ComparisonwithTh.II:SinceD=DP,thecondition0ri(D)ofTheoremIIis

ri(D)ri(P) =


127/300

PROOF OF MC/MC TH. IIIonsiderthedisjointconvexsetsC1 = (u,v) |C v > w forsome(u,w)M andC2 =(u,w)

u P [u P and (u,w) M with w > w|contradictsthedefinitionofw]

(,)

0 u

v

C1

C2

M

w

P

Since C2 is polyhedral, there exists a separatinghyperplanenotcontainingC1, i.e.,a(,) =(0,0)suchthat

w +zv+x, (x,v)C1, zPinf v+x


128/300

PROOF (CONTINUED)Hence,w +z inf zP,

(u,v)C1{v+u},

sothatinf v+(uz)w (u,v)C1, zP

= inf(u,v)MP{v+u}

= inf(u,v)M

{v+u}=

q()

Using q (weak duality), we have q() = wq =w.

Proofthatallmaxcrossingsolutionssatisfyd

0foralld

RP: followsfrom

q()= inf v+(uz)(u,v)C1, zP

sothatq() =ifd >0. Q.E.D. Geometrical intuition: every (0,d) with dRP,isdirectionofrecessionofM.


129/300


130/300


131/300


132/300


133/300

PROOF OF LP DUALITY (CONTINUED)

Feasible Set

x

a1a2

c= 1a1+ 2a2

Cone D (translated to x)

Let x be a primal optimal solution, and let

J ={j |ajx =bj}. Then,cy0forally intheconeoffeasibledirections

D={y|ajy0,jJ}ByFarkasLemma,forsomescalars

j 0,ccan

beexpressedasr

c=jaj, j0, jJ, j = 0, j /J.j=1

Taking

inner

product

with

x,

we

obtain

cx

=

b,whichinviewofq f,showsthatq =fandthat isoptimal.


134/300

LINEAR PROGRAMMING OPT. CONDITIONSApairofvectors(x

,

)formaprimalanddualoptimal solution pair if and only if x is primal-feasible, isdual-feasible,and

j(bj ajx) = 0, j= 1,...,r. ()Proof: If x is primal-feasible and is dual-feasible,then

+j=1

j=1

r ()

=cx + j(bj ajx)j=1

SoifEq.(*)holds,wehaveb =cx,andweakduality implies that x is primal optimal and isdualoptimal.

Conversely,if(x, )formaprimalanddualoptimal solution pair, then x is primal-feasible, isdual-feasible,andbythedualitytheorem,wehave b = cx. From Eq. (**), we obtain Eq.(*).

r rc bjj ajjb x=


135/300


ConvexProgrammingDuality OptimalityConditions

MixturesofLinearandConvexConstraints

ExistenceofOptimalPrimalSolutions FenchelDuality ConicDualityReading: Sections5.3.1-5.3.6Line of analysis so far: Convexanalysis(rel. int.,dir.ofrecession,hyperplanes,conjugacy) MC/MC

NonlinearFarkasLemma Linearprogramming(duality,opt.conditions) We now discuss convex programming, and itsmanyspecialcases(relianceonNonlinearFarkasLemma)


136/300

CONVEX PROGRAMMINGidertheproblemCons

minimize f(x)subjectto xX, gj(x)0, j= 1,...,r,

where X

n is convex, and f : X

andgj :X areconvex. Assumef: finite. Recall the connection with the max crossingproblem in the MC/MC framework where M =epi(p)with

p(u)= inf f(x)xX,g(x)u ConsidertheLagrangianfunction

L(x,) =f(x) +g(x),thedualfunction

infxX L(x,) if0,q() = otherwiseandthedualproblemofmaximizinginfxX L(x,)over0.


137/300

STRONG DUALITY THEOREM

Assume that f is finite, and that one of thefollowingtwoconditionsholds:(1) ThereexistsxX suchthatg(x)


138/300


139/300

OPTIMALITY CONDITIONS Wehaveq =f,andthevectorsx and areoptimalsolutionsoftheprimalanddualproblems,respectively, iffx isfeasible, 0,and

x argminL(x,), jgj(x) = 0, j.xX

(1)

Proof: Ifq =f,andx, areoptimal,thenf =q =q()= inf L(x,)L(x, )

xXr

=f(x) +jgj(x)

f(x),

j=1wherethelastinequalityfollowsfromj 0andgj(x)0forallj. Henceequalityholdsthroughoutabove,and(1)holds.

Conversely,ifx, arefeasible,and(1)holds,q()= inf L(x,) =L(x, )

xXr

=f(x) +jgj(x) =f(x),j=1

soq =f,andx, areoptimal. Q.E.D.


140/300


141/300

LINEAR EQUALITY CONSTRAINTS Theproblemis

minimize f(x)subjectto xX, g(x)0, Ax=b,

whereX

is

convex,

g(x) =

g1(x), . . . , gr(x)

,f:

X andgj :X ,j= 1, . . . , r,areconvex. Convert the constraint Ax = b to Ax bandAx b,withcorrespondingdualvariables+ 0and 0. TheLagrangianfunctionis

f(x) +g(x) + (+)(Axb),andby introducingadualvariable=+,withnosignrestriction, itcanbewrittenas

L(x,,) =f(x) +g(x) +(Axb). Thedualproblemis

maximize q(,)

inf L(x,,)xX

subjectto 0, m.


142/300

DUALITY AND OPTIMALITY COND. Pure equality constraints:

(a) Assumethatf: finiteandthereexistsxri(X)suchthatAx=b. Thenf =q andthereexistsadualoptimalsolution.

(b)f =q,and(x, )areaprimalanddualoptimalsolutionpairifandonlyifx isfeasible,and

x argminL(x,)xX

Note: No complementary slackness for equalityconstraints.

Linear and nonlinear constraints:(a) Assume f: finite, that there exists x X

such that Ax = b and g(x)


143/300

COUNTEREXAMPLE IgDualityCounterexample: Consider Stron

minimize f(x) =ex1x2subjectto x1 = 0, xX={x|x0}

Heref

=1andf isconvex(itsHessianis>0intheinteriorofX). Thedualfunctionisq()= infex1x2 +x1=

0 if0,

x0 otherwise,(when 0, the expression in braces is nonnegative for x0 and can approach zero by takingx1 0andx1x2 ). Thusq =0. Therelativeinteriorassumption isviolated. As predicted by the corresponding MC/MC

framework,theperturbationfunction0 ifu >0,

p(u)= inf ex1x2 = 1 ifu=0,x1=u,x0 ifu


144/300


145/300

COUNTEREXAMPLE II Existence of Solutions Counterexample:LetX =, f(x) =x,g(x) =x2.Thenx =0 istheonlyfeasible/optimalsolution,andwehave

1q()= inf }= , >0,

x

{x+x24

and q() = for 0, so that q =f =0.However, there is no 0 such that q() =q =0. Theperturbationfunction is

u ifu0,p(u)= inf x=x2u ifu


146/300

FENCHEL DUALITY FRAMEWORK Considertheproblem

minimize f1(x) +f2(x)subjectto ,x n

wheref1 :

n

(

,

]andf2 :

n

(

,

]areclosedproperconvexfunctions. Converttotheequivalentproblemminimize f1(x1) +f2(x2)subjectto x1 =x2, x1

dom(f1), x2

dom(f2)

Thedualfunctionisq()= inf f1(x1) +f2(x2) +(x2x1)

x1dom(f1), x2dom(f2)= inf f1(x1)

x1+ inf f2(x2) +

x2

x1n x2n

Dual problem: max{f1()f2()} =min{q()}or

minimize f1() +f2()subject

to

,

n

wheref1 andf2 aretheconjugates.


147/300

FENCHEL DUALITY THEOREMConsidertheFenchelframework:

(a) Iff isfiniteandridom(f1)ridom(f2)=,thenf =q andthereexistsatleastonedualoptimalsolution.

(b) Thereholdsf =q,and(x, )isaprimalanddualoptimalsolutionpairifandonlyif

x arg minf1(x)x, x arg minf2(x)+xxn xn

Proof: For strong duality use the equality constrainedproblemminimize f1(x1) +f2(x2)subjectto x1 =x2, x1 dom(f1), x2 dom(f2)and

the

fact

ri

dom(f1)dom(f2)

=ridom(f1) dom(f2)tosatisfytherelativeinteriorcondition.

Forpart(b),applytheoptimalityconditions(primalanddualfeasibility,andLagrangianoptimality).


148/300

GEOMETRIC INTERPRETATION

f1(x)Slope

Slope

x x

f2(x)

q()

f =q

f1()

f2()

When dom(f1) = dom(f2) =n, and f1 andf2 are differentiable, the optimality condition isequivalentto

=f1(x) =f2(x) Byreversingtherolesofthe(symmetric)primalanddualproblems, weobtainalternativecriteriaforstrongduality: ifq isfiniteandridom(f1)ridom(f)= ,thenf =q andthereexists2

atleastoneprimaloptimalsolution.


149/300

Acotion f

CONIC PROBLEMSnicproblem istominimizeaconvexfunc: n ( , ] subject to a cone con

straint. Themostuseful/popularspecialcases:

Linear-conicprogramming Secondorderconeprogramming Semidefiniteprogramming

involveminimizationofalinearfunctionovertheintersectionofanaffinesetandacone. Can be analyzed as a special case of Fenchel

duality. Therearemanyinterestingapplicationsofconicproblems,including indiscreteoptimization.


150/300

CONIC DUALITYrminimizingf(x)overx C,wheref : Conside n (,]isaclosedproperconvexfunction

andC isaclosedconvexcone inn. WeapplyFencheldualitywiththedefinitions

f1(x) =f(x), f2(x) =0

ifif

x /xC, C.Theconjugatesaref1()= sup

xf(x), f2()= supx=

0 ifC,

xn xC if /C,

whereC ={|x0,xC}. Thedualproblemis

minimize f()subjectto C,

wheref istheconjugateoff andC ={|x0,xC}.

C andC arecalledthedualandpolarcones.


151/300


152/300

LINEAR CONIC PROGRAMMING Letf belinearoveritsdomain,i.e.,

cx ifxX,f(x) = ifx /X,

wherecisavector,andX=b+S isanaffineset. Primalproblemis

minimize cxsubjectto xbS, xC.

Wehave

f() = sup (c)x=sup(c)(y+b)xbS yS

(c)b ifcS,= ifc /S.

Dualproblem isequivalenttominimize b

subjectto cS, C. If Xri(C) =, there is no duality gap andthereexistsadualoptimalsolution.


153/300

ANOTHER APPROACH TO DUALITY Considertheproblem

minimize f(x)subjectto xX, gj(x)0, j= 1, . . . , r

andperturbationfnp(u)=infxX,g(x)uf(x) RecalltheMC/MCframeworkwithM =epi(p).Assuming thatp is convex and f


154/300


Subgradients Fenchelinequality Sensitivityinconstrainedoptimization

Subdifferentialcalculus Optimalityconditions


155/300

Let f :n SUBGRADIENTS (,] be a convex function.A vector g n is a subgradient of f at a point

xdom(f)iff(z)f(x) + (zx)g, z n

g isasubgradient ifandonlyiff(z)zgf(x)xg, z n

sog isasubgradientatxifandonlyifthehyperplanein

n+1 thathasnormal(

g,1)andpasses

throughx,f(x)supportstheepigraphoff.

0

(g, 1)

x, f(x)

z

Thesetofallsubgradientsatxisthesubdifferentialoff atx,denotedf(x).


156/300

EXAMPLES OF SUBDIFFERENTIALS Someexamples:

f(x)

f(x)

0 x x

xx

f(x) = max

0, (1/2)(x2 1)

f(x) =|x|

1

1

1-1

-1

-10

0

0

Iff isdifferentiable,thenf(x) ={f(x)}.Proof: Ifgf(x),then

f(x+z)f(x) +gz, z n.Applythiswithz=f(x)g, ,anduse1storderTaylorseriesexpansiontoobtain

f(x)g2 o(),


157/300

EXISTENCE OF SUBGRADIENTS NotetheconnectionwithMC/MC

M =epi(fx), fx(z) =f(x+z)f(x)

0

(g, 1)

f(z)

x, f(x)

z

0

z

(g, 1)

Epigraph offEpigraph off Translated

fx(z)

Let f : n (,] be a proper convexfunction. Foreveryxridom(f)),

f(x) =S +G,where:

Sisthesubspacethatisparalleltotheaffinehullofdom(f)

G isanonemptyandcompactset. Furthermore, f(x) isnonemptyandcompact

ifandonly ifxisinthe interiorofdom(f).


158/300

EXAMPLE:SUBDIFFERENTIALOF INDICATO LetC beaconvexset,andC be its indicatorfunction. For x /C,C(x) = ,byconvention. For xC, we have gC(x) iff

C(z)C(x) + g(zx), zC,orequivalentlyg(zx)0 forallzC. Thus C(x) is the normal cone of C at x, denotedNC(x):

NC(x) = g|g(zx)0,zC .

CNC(x)

x CNC(x)

x


159/300

EXAMPLE:POLYHEDRALCASE

NC(x)

Ca1

a2

x

ForthecaseofapolyhedralsetC={x|a

ixbi, i= 1, . . . , m},wehave

0 ifxNC(x) = { } int(C),cone {ai |aix=bi} ifx /int(C).


160/300

FENCHEL INEQUALITY Let f : , ] be proper convex andn (let f be its conjugate. Using the definition ofconjugacy,wehaveFenchels inequality:

nxy f(x) + f(y), x n, y . ConjugateSubgradientTheorem: Thefol-lowing two relations are equivalent for a pair ofvectors(x, y):

(i) xy=f(x) + f(y).(ii) y f(x).

Iff isclosed,(i)and(ii)areequivalentto(iii) x f(y).

f(x)

Epigraphoff

0 x 0 y(y,1)

(x, 1)

f(x)

x y0 0

i hEpgrap off

(x, 1)(y,1)

f(y)

Epigraphoff

f(y) f(x)


161/300

MINIMA OF CONVEX FUNCTIONSApplication: Let f be closed proper convex

and let X be the set of minima of f over n.Then:

(a)X =f(0).(b)X isnonempty if0ridom(f

).(c)X isnonempty andcompact if andonly if

0intdom(f).Proof: (a)Fromthesubgradient inequality,

x minimizesf iff 0

f(x),andsince

0f(x) iff x f(0),

wehave

x minimizesf iff x f(0),

(b)f(0)isnonemptyif0ri

dom(f)

.

(c)f

(0)is

nonempty

and

compact

if

and

only

if0intdom(f). Q.E.D.


162/300

SENSITIVITY INTERPRETATION ConsiderMC/MCforthecaseM =epi(p).

Dualfunction isq()= inf p(u) +u=p(),

mu

wherep istheconjugateofp. Assumep is proper convex and strong dualityholds,sop(0)=w =q =sup p().mLetQ bethesetofdualoptimalsolutions,

Q= |p(0)+p() = 0.

From Conjugate Subgradient Theorem, Qifandonly if p(0),i.e.,Q =p(0). Ifpisconvexanddifferentiableat0,p(0)isequal

to

the

unique

dual

optimal

solution

.

Constrainedoptimizationexample

p(u)= inf f(x),xX,g(x)u

Ifpisconvexanddifferentiable,p(0)j = uj , j= 1,...,r.


163/300

EXAMPLE: SUBDIFF. OF SUPPORT FUNCTIO Consider the support function X(y) of a setX. TocalculateX(y)atsomey,we introduce

r(y) =X(y+y), .y n

WehaveX(y) =r(0)=argminx

n r(x).

Wehaver(x)=supyn{yxr(y)},orr(x)= sup

yn{yxX(y+y)}=(x)yx,

where isthe indicatorfunctionofclconv(X). HenceX(y)=argminxn (x)yx,or

X(y)=arg maxxcl

conv(X)

yx

0

y1

y2

X

X(y2)

X(y1)


164/300

EXAMPLE: SUBDIFF. OF POLYHEDRAL FNLet

f(x)=max{a1x+b1, . . . , arx+br}. Forafixedx n,consider

Ax =j|ajx+bj =f(x)andthefunctionr(x)=maxajx|jAx.

f(x)

x0

Epigraph off

(g, 1)

x x0

(g, 1)r(x)

Itcanbeseenthatf(x) =r(0). Sincer isthesupportfunctionofthefiniteset{aj |jAx},weseethat

f(x) =r(0)=conv{aj |jAx}


165/300

CHAIN RULE Letf :

m

(

,

]beconvex,andAbeamatrix. ConsiderF(x) =f(Ax)andassumethatF is proper. If either f is polyhedral or else therangeofR(A)ri(dom(f))=,wehave

F(x) =Af(Ax), x n.Proof: ShowingF(x)Af(Ax)issimpleanddoesnotrequiretherelative interiorassumption.Forthereverseinclusion,letdF(x)soF(z)F(x) + (zx)d0orf(Az)zdf(Ax)xdforallz,so(Ax,x)solves

minimize f(y)zdsubjectto ydom(f), Az=y.

IfR(A)ri(dom(f))=,bystrongdualitytheorem,thereisadualoptimalsolution,suchthat(Ax,x)

arg min f(y)zd+(Azy)

ym, zn

Since the min over z is unconstrained, we haved=A,soAxargminymf(y)y,or

f(y)f(Ax) +(yAx), y m.Hencef(Ax),sothatd=AAf(Ax).ItfollowsthatF(x)Af(Ax). Inthepolyhedralcase,dom(f)ispolyhedral. Q.E.D.


166/300

SUM OF FUNCTIONS Letfi :n (,],i= 1, . . . , m,beproperconvexfunctions,andlet

F =f1 + +fm.

Assumethat

m ridom(fi)=

.1=1

ThenF(x) =f1(x) + +fm(x), . x n

Proof: WecanwriteF intheformF(x) =f(Ax),whereAisthematrixdefinedbyAx= (x,...,x),andf :mn (,]isthefunction

f(x1, . . . , xm) =f1(x1) + +fm(xm). Usetheproofofthechainrule.

Extension: Ifforsomek,thefunctionsfi,i=1,...,k,arepolyhedral,itissufficienttoassume

ki=1 dom(fi)

mi=k+1 ridom(fi)

=.


167/300

CONSTRAINED OPTIMALITY CONDITION

Letf : n ( , ]beproperconvex,letX beaconvexsubsetofn,andassumethatoneofthefollowingfourconditionsholds:

(i) ridom(f)ri(X) =.(ii) f ispolyhedralanddom(f)

ri(X) =.

(iii) X ispolyhedralandridom(f)X= .(iv) f andX arepolyhedral,anddom(f)X= .

Then, a vector x minimizes f over X iff thereexists g f(x) such that g belongs to thenormal

cone

NX(x),i.e.,

g(xx)0, xX.Proof: x minimizes

F(x) =f(x) +X(x)if and only if 0 F(x). Use the formula forsubdifferentialofsum. Q.E.D.


168/300

LLUSTRATIONOFOPTIMALITYCONDITION

LevelSetsoff

xf(x

)

LevelSetsoffx

NC(x)NC(x)

C Cg

f(x)

Inthefigureontheleft,f isdifferentiableandtheconditionisthat

f(x)NC(x),whichisequivalentto

f(x)(xx)0, xX. Inthefigureontheright,f isnondifferentiable,andtheconditionisthat

gNC(x) forsomegf(x).


169/300


Min-MaxDualityExistenceofSaddlePoints

Given :XZ , where X n, Z mconsider

minimize sup(x,z)zZ

subjectto x

Xand

maximize inf (x,z)xX

subjectto zZ.


170/300

REVIEW Minimax inequality(holdsalways)

sup inf (x,z) inf sup(x,z)zZxX xX zZ

Importantissueiswhetherminimaxequalityholds. Definition: (x, z) iscalledasaddlepointofif(x, z)(x, z)(x,z), xX,zZ

Proposition: (x, z) isasaddlepoint ifand

onlyiftheminimaxequalityholdsandx argminsup(x,z), z argmax inf (x,z)

xX zZ zZ xX

Connectionw/constrainedoptimization: Strongdualityisequivalentto

inf supL(x,)=sup inf L(x,)xX 0 0xX

whereListheLagrangianfunction.Optimal primal-dual solution pairs (x, )

arethesaddlepointsofL.


171/300

MC/MC FRAMEWORK FOR MINIMAX UseMC/MCwithM =epi(p)wherep:m [,] istheperturbationfunction

p(u)= inf sup(x,z)uz,xX zZ u m

Importantfact:pisobtainedbypartialmin. Note that w =p(0) = infsup and (, z):convexforallz impliesthatM isconvex. If(x,) isclosedandconvex,thedualfunctioninMC/MC is

q(z)= inf (x,z), q =supinfxX

u

w

(, 1)

q()

M= epi(p)

0

w = infxX

supzZ

(x, z)

q = supzZ

infxX

(x, z)

(, 1)

q()

u

w

0

M= epi(p)

w = infxX

supzZ

(x, z)

q = supzZ

infxX

(x, z)


172/300

MINIMAX THEOREM Ithat:andZ areconvex.

Assume

(1) X(2)p(0)=infxX supzZ(x,z)


173/300

MINIMAX THEOREM IIssumethat:(1) X andZ areconvex.(2)p(0)=infxX supzZ(x,z)>.

A

(3) ForeachzZ,thefunction(, z)isconvex.(4) Foreachx

X,thefunction

(x, ) :Z

isclosedandconvex.(5) 0liesintherelativeinteriorofdom(p).

Then,theminimaxequalityholdsandthesupremuminsupzZ infxX (x,z)isattainedbysomezZ. [Alsothesetofzwherethesupisattainediscompactif0 is intheinteriorofdom(p).]

Proof: Applythe2ndMinCommon/MaxCrossingTheorem. Counterexamples of strong duality and existenceofsolutions/saddlepointscanbeconstructedfromcorrespondingconstrainedminexamples.


174/300

EXAMPLE I

= (x1, x2) x 0 and Z = z Let X | { |z0},andlet(x,z) =ex1x2 +zx1,

whichsatisfytheconvexityandclosednessassumptions. Forallz0,

infex1x2 +zx1= 0,x0

sosupz0infx0(x,z) = 0.Also,forallx0,supex1x2 +zx1=

1 ifx1 =0,

z0 ifx1 >0,soinfx0supz0(x,z)=1.

Herep(u)= inf supex1x2 +z(x1u)

x0z0

epi(p)

u

p(u)

1

0


175/300

EXAMPLE II

,Z= z z 0 ,andlet LetX= { | }(x,z) =x+zx2,

whichsatisfytheconvexityandclosednessassumptions. Forallz0,

1/(4z) ifz >0,infx{x+zx2}= ifz=0,

sosupz0infx(x,z) = 0.Also,forallx ,

0

ifx

=

0,

z0{x+zx2sup }= otherwise,

so infxsupz0(x,z)=0. However,thesup isnotattained, i.e.,thereisnosaddlepoint.

Here

p(u)= inf sup uz}z0

{x+zx2x

u ifu0,= ifu


176/300

SADDLE POINT ANALYSIS Theprecedinganalysisindicatestheimportanceoftheperturbationfunction

p(u)= inf F(x,u),xn

where

F(x,u) =

supzZ

(x,z)uz ifxX, ifx /X.

Itsuggestsatwo-stepprocesstoestablishtheminimax

equality

and

the

existence

of

asaddle

point:

(1)Showthatpisclosedandconvex,thereby

showingthattheminimaxequalityholdsbyusingthefirstminimaxtheorem.

(2)VerifythattheinfofsupzZ(x,z)overxX,andthesupofinfxX (x,z)overzZ are attained, thereby showing thatthesetofsaddlepointsisnonempty.


177/300

SADDLE POINT ANALYSIS (CONTINUED) Step(1)requirestwotypesofassumptions:

(a) Convexity/concavity/semicontinuityconditionsofMinimaxTheoremI(sotheMC/MCframeworkapplies).

(b) Conditionsforpreservationofclosednessbythepartialminimization in

p(u)= inf F(x,u)xn

e.g.,forsomeu,thenonemptylevelsets

x|F(x,u)arecompact.

Step(2)requiresthateitherWeierstrassThe

oremcanbeapplied,orelseoneoftheconditionsforexistenceofoptimalsolutionsdevelopedsofarissatisfied.


178/300

CLASSICAL SADDLE POINT THEOREM Assumeconvexity/concavity/semicontinuityof and that X and Z are compact. Then the setofsaddlepointsisnonemptyandcompact. Proof: F isconvexandclosedbytheconvexity/concavity/semicontinuityof,sopisalsoconvex.

Using

the

compactness

of

Z,

F

is

real-valued

over X m, and from the compactness of X,it follows thatp is also real-valued and thereforecontinuous. Hence,theminimaxequalityholdsbythefirstminimaxtheorem.

Thefunctionsupz

Z(x,z)isequaltoF(x,0),soitisclosed,andthesetofitsminimaoverxXis nonempty and compact by Weierstrass Theorem. Similarlythesetofmaxima ofthe functioninfxX (x,z) over z Z is nonempty and compact. Hencethesetofsaddlepoints isnonemptyandcompact. Q.E.D.


179/300

ANOTHER THEOREM Use the theory of preservation of closednessunderpartialminimization. Assumeconvexity/concavity/semicontinuityof. Considerthefunctions

t(x) =F(x,0)=supzZ(x,z) ififx /xX, X,

andr(z) =

infxX (x,z) ifzZ,

if

z /

Z.

Ifthe levelsetsoftarecompact,theminimaxequalityholds,andtheminoverxof

sup(x,z)zZ

[which is t(x)] is attained. (Take u = 0 in thepartialmintheoremtoshowthatpisclosed.) Ifthelevelsetsoftandrarecompact,thesetofsaddlepoints isnonemptyandcompact. Various extensions: Use conditions for preservationofclosednessunderpartialminimization.


180/300

SADDLE POINT THEOREMssumetheconvexity/concavity/semicontinuityconA

ditions,andthatanyoneofthefollowingholds:(1)X andZ arecompact.(2)Z iscompactandthereexistsavectorzZ

and a scalar such that the level setxX |(x,z)

isnonemptyandcompact.

(3)XiscompactandthereexistsavectorxXand a scalar such that the level setz Z |(x,z)isnonemptyandcompact.

(4) Thereexistvectorsx

X andz

Z,andascalar suchthatthelevelsets

xX |(x,z), zZ |(x,z),arenonemptyandcompact.

Then,theminimaxequalityholds,andthesetofsaddlepointsofisnonemptyandcompact.


181/300


ProblemStructures Separableproblems Integer/discreteproblemsBranch-and-bound Largesumproblems Problemswithmanyconstraints

ConicProgramming SecondOrderConeProgramming

SemidefiniteProgramming


182/300

SEPARABLE PROBLEMS Considertheproblem

m

minimize fi(xi)i=1

ms.

t.

gji(xi)

0, j= 1,...,r, xi Xi, ii=1

where fi :ni and gji :ni are givenfunctions,andXi aregivensubsetsofni. Formthedualproblem

m m rmaximize inf fi(xi) +qi()

xiXijgji(xi)

i=1 i=1 j=1subjectto 0 Importantpoint: Thecalculationofthedualfunction has been decomposed into n simplerminimizations. Moreover, the calculation of dualsubgradients is a byproduct of these minimizations(thiswillbediscussedlater) Anotherimportantpoint: IfXi isadiscreteset (e.g., Xi ={0,1}), the dual optimal value isa lower bound to the optimal primal value. It isstillusefulinabranch-and-boundscheme.


183/300

LARGE SUM PROBLEMSConsidercostfunctionoftheform m

f(x) =fi(x), misvery large,i=1

wherefi :n areconvex. Someexamples: Dual cost of a separable problem.

Data analysis/machine learning: x is parametervectorofamodel;eachfi correspondstoerrorbetweendataandoutputofthemodel.

Leastsquaresproblems(fi quadratic).

1-regularization

(least

squares

plus

1

penalty):

m n

min(ajxbj)2 + xix | |j=1 i=1

Thenondifferentiablepenaltytendstosetalargenumberofcomponentsofxto0.

MinofanexpectedvalueEF(x,w),wherew is a random variable taking a finite but verylargenumberofvalueswi,i= 1, . . . , m,withcorrespondingprobabilitiesi. Stochastic programming:

minF1(x) +Ew{minF2(x,y,w)

x y Specialmethods,calledincrementalapply.


184/300

PROBLEMS WITH MANY CONSTRAINTSProblemsoftheform

minimize f(x)subjectto ajxbj, j= 1,...,r,

wherer: very large. One possibility isapenaltyfunctionapproach:Replaceproblemwith

rminf(x) +c

P(ajxbj)

x

n

j=1

where P() is a scalar penalty function satisfyingP(t)=0ift0,andP(t)>0ift >0,andcisapositivepenaltyparameter. Examples: ThequadraticpenaltyP(t) =max{0, t}2. ThenondifferentiablepenaltyP(t)=max{0, t}.

Another possibility: Initially discard some oftheconstraints, solvea less constrained problem,and

later

reintroduce

constraints

that

seem

to

be

violatedattheoptimum(outerapproximation). Alsoinnerapproximationoftheconstraintset.


185/300

CONIC PROBLEMS Aconicproblem istominimizeaconvexfunction f : n (,] subject to a cone constraint. Themostuseful/popularspecialcases:

Linear-conicprogramming Secondorderconeprogramming Semidefiniteprogramming

involveminimizationofalinearfunctionovertheintersectionofanaffinesetandacone. Can be analyzed as a special case of Fenchel

duality. Therearemanyinterestingapplicationsofconicproblems,including indiscreteoptimization.


186/300

PROBLEM RANKING INASING PRACTICAL DIFFICULTYINCRE

Linearand(convex)quadraticprogramming. Favorablespecialcases.

Second order cone programming. Semidefinite programming. Convexprogramming.

Favorablespecialcases. Geometricprogramming.

Quasi-convexprogramming.Nonlinear/nonconvex/continuousprogramming.

Favorablespecialcases. Unconstrained. Constrained.

Discreteoptimization/Integerprogramming Favorablespecialcases.


187/300

CONIC DUALITY Considerminimizingf(x)overxC,wheref :n (,]isaclosedproperconvexfunctionandC isaclosedconvexcone inn. WeapplyFencheldualitywiththedefinitions

f1(x) =f(x), f2(x) =0 ifx

C,

ifx /C.Theconjugatesare

0 ifC,

f1()= supxf(x), f2

()= supx=if /C,xn xC

where C ={ | x 0, x C} is the polarconeofC. Thedualproblemis

minimize f()subjectto C,

wheref istheconjugateoff andC ={|x0,xC}.

C =C iscalledthedualcone.


188/300

LINEAR-CONIC PROBLEMS Let f be affine, f(x) = cx, with dom(f) being an affine set, dom(f) = b+S, where S is asubspace. Theprimalproblem is

minimize cx

subjectto xbS, xC. Theconjugateis

f() = sup (c)x=sup(c)(y+b)x

b

S y

S(c)b ifcS,

= ifc /S,sothedualproblemcanbewrittenas

minimize bsubjectto cS, C.

Theprimalanddualhavethesameform. If C is closed, the dual of the dual yields theprimal.


189/300

SPECIAL LINEAR-CONIC FORMSmin c

x max b

,

Ax=b, xC cACmin cx max b,

AxbC A=c, Cwherex n, m,c n,b m,A:mn.

Forthefirstrelation,letxbesuchthatAx=b,andwritetheproblemonthe leftasminimize cxsubjectto xxN(A), xC

Thedualconicproblem is

minimize xsubjectto cN(A), C.

Using N(A) = Ra(A), write the constraintsasc Ra(A)=Ra(A),C,or

c=A, C, forsome m. Changevariables=cA,writethedualas

minimize x(cA)subject

to

cAC

discardtheconstantxc,usethefactAx=b,andchangefrommintomax.


190/300

SOME EXAMPLES Nonnegative Orthant: C={x|x0}.

The Second Order Cone: Let 2 2

C= (x1, . . . , xn)|xn x1 + +xn1

x1

x2

x3

The Positive Semidefinite Cone: Considerthespaceofsymmetricn

nmatrices,viewedas

thespacen2 withthe innerproductn n

=trace(XY) =xijyiji=1 j=1

Let C be the cone of matrices that are positivesemidefinite.

Alltheseareself-dual,i.e.,C=C =C.


191/300

SECOND ORDER CONE PROGRAMMING

Second order cone programming is the linear-conicproblemminimize cxsubjectto Aixbi Ci, i= 1,...,m,

where c,bi are vectors, Ai are matrices, bi is avector inni,and

Ci : thesecondorderconeofniTheconehereis

C=C1 Cm

x1

x2

x3


192/300

SECOND ORDER CONE DUALITY Usingthegenericspecialdualityform

min cx max b,AxbC A=c, C

andselfdualityofC,thedualproblem ismaximize m

i=1bii

subjectto mAii =c, i Ci, i= 1,...,m,

i=1where= (1, . . . , m). The duality theory is no more favorable thantheonefor linear-conicproblems. Thereisnodualitygapifthereexistsafeasible

solutionintheinteriorofthe2ndorderconesCi. Generally, second order cone problems can berecognized from the presence of norm or convexquadratic functions in the cost or the constraintfunctions. Therearemanyapplications.


193/300

EXAMPLE: ROBUST LINEAR PROGRAMMINGminimize cxsubjectto ajxbj, (aj, bj)Tj, j= 1,...,r,wherec n,andTj isagivensubsetofn+1. Weconverttheproblemtotheequivalentform

minimize cxsubjectto gj(x)0, j= 1,...,r,

wheregj(x)=sup(aj,bj)Tj{ajxbj}. ForspecialchoicewhereTj isanellipsoid,Tj =(aj+Pjuj, bj+qjuj)| uj 1, uj njwecanexpressgj(x)0intermsofaSOC:

gj(x)= sup (aj +Pjuj)x(bj +qjuj)uj1= sup (Pjxqj)uj +ajxbj,uj1

=Pjxqj+ajxbj.Thus,gj(x)0iff(Pjxqj, bjajx)Cj,whereCj istheSOCofnj+1.


194/300


Conicprogramming Semidefiniteprogramming Exactpenaltyfunctions

Descent methods for convex/nondifferentiableoptimization Steepestdescentmethod


195/300

LINEAR-CONIC FORMSmin c

x max b

,

Ax=b, xC

cACmin cx max b,

AxbC A=c, Cwherex n, m,c n,b m,A:mn. Secondorderconeprogramming:

minimize cxsubjectto Aixbi Ci, i= 1,...,m,

where c,bi are vectors, Ai are matrices, bi is avector inni,and

Ci : thesecondorderconeofni TheconehereisC=C1 Cm Thedualproblemis

mmaximize bii

i=1m

subjectto Ai =c, i

Ci, i= 1,...,m,i

i=1where= (1, . . . , m).


196/300

SEMIDEFINITE PROGRAMMING Considerthesymmetricnnmatrices. Innerproduct=trace(XY) =ni,j=1xijyij. LetCbetheconeofpos.semidefinitematrices. C isself-dual,anditsinterioristhesetofpositivedefinitematrices. Fix symmetric matrices D, A1, . . . , Am, andvectorsb1, . . . , bm,andconsiderminimize subjectto < Ai, X>=bi, i= 1, . . . , m , XC Viewingthisasalinear-conicproblem(thefirstspecial form), the dual problem (using also self-dualityofC)is

mmaximize

bii

i=1

subjectto D(1A1 + +mAm)C There is no duality gap if there exists primalfeasiblesolutionthat ispos.definite,orthereexistssuchthatD(1A1+ +mAm)ispos. definite.

7/27/2019 Convex Analysis and Optimizati

Documents

Convex Analysis and Optimization Mit