Chapter 4 Convex Optimization Problems

Shupeng Gui

Computer Science, UR

October 16, 2015

Shupeng Gui (Computer Science, UR) Convex Optimization Problems October 16, 2015 1 / 58

Outline

1 Optimization problems

2 Convex optimization problem

3 Quasiconvex optimization

4 Linear program(LP)

5 Quadratic program(QP)

6 Generalized inequality constraints

7 Vector optimization

Basic terminology

Optimization problem in standard form

min f0(x)

s.t. fi(x) ≤ 0, i = 1, 2, ...,m

hi(x) = 0, i = 1, 2, ..., p

x ∈ Rn is the optimization variable

f0 : Rn → R is the objective or cost function

fi : Rn → R, i = 1, 2, ...,m, are the inequality constraint functions

hi : Rn → R are the equality constraint functions

Optimal value

optimal value

p∗ = inf{f0(x)|fi(x) ≤ 0, i = 1, 2, ...,m, hi(x) = 0, i = 1, ..., p}

p∗ =∞ if problem is infeasible (no x satisfies the constraints)

p∗ = −∞ if problem is unbounded below

inf means the infimum.

Optimal value

Example What is the optimal value here?

min f0(x) = (x− x0)2

s.t. x ≥ −6

x ≤ 10

Figure 1: f0 which mentioned above,x0 = 4

Optimal value

min f0(x) =1

(x− x0)2

s.t. x = x0

Figure 2: f0 which mentioned above,x0 = 4

Optimal Value

min f0(x) = − log(x)

s.t. x > 0

Figure 3: f0 which mentioned above

Optimal and locally optimal points

x is feasible if x ∈ dom f0 and it satisfies the constraintsa feasible x is optimal if f0(x) = p∗; Xopt is the set of optimal points

Xopt = {x|fi(x) ≤ 0, i = 1, ...,m, hi(x) = 0, i = 1, ...p, f0(x) = p∗}

x is locally optimal if there is an R > 0 such that x is optimal for

min f0(z)

s.t. fi(z) ≤ 0, i = 1, ...,m

hi(z) = 0, i = 1, ..., p

‖z − x‖2 ≤ R

Figure 4: Local optimal points

example (with n=1, m=p=0)What is the optimal value and how about the optimal points?

f0(x) =1

x, domf0 = R++

Figure 5: f0(x) = 1/xShupeng Gui (Computer Science, UR) Convex Optimization Problems October 16, 2015 10 / 58

f0(x) = − log(x), domf0 = R++

Figure 6: f0(x) = − log(x)

f0(x) = x log(x), domf0 = R++

Figure 7: f0(x) = x log(x)

f0(x) = x log(x), domf0 = R++

Figure 8: f0(x) = x log(x)

p∗ = −1/e, x = 1/e is optimal.Shupeng Gui (Computer Science, UR) Convex Optimization Problems October 16, 2015 13 / 58

f0(x) = x3 − 3x

Figure 9: f0(x) = x3 − 3x

Implicit constraints

the standard form optimization problem has an implicit constraint

x ∈ D =

m⋂i=0

domfi ∩p⋂

Here include f0.

we call D the domain of the problem.

the constraints fi(x) ≤ 0, hi(x) = 0 are explicit constraints.

a problem is unconstrained if it has no explicit constraints(m=p=0);

Example

min f0(x) = −k∑

log(bi − aTi x)

is an unconstrained problem with implicit constraints aTi x < bi,

Feasibility problem

find x

s.t. fi(x) ≤ 0, i = 1, ...,m

hi(x) = 0, i = 1, ..., p

can be considered a special case of the general problem with f0(x) = 0

s.t. fi(x) ≤ 0, i = 1, ...,m

hi(x) = 0, i = 1, ..., p

p∗ = 0 if constraints are feasible; any feasible x is optimal

p∗ =∞ if constraints are infeasible

Convex optimization problem

standard form convex optimization problem

min f0(x)

s.t. fi(x) ≤ 0, i = 1, 2, ...,m

aTi x = bi, i = 1, 2, ..., p

f0, f1, ..., fm are convex; equality constraints are affine

problem is quasiconvex if f0 is quasiconvex(and f0, f1, ..., fmconvex)

Sometimes written as

min f0(x)

s.t. fi(x) ≤ 0, i = 1, 2, ...,m

Ax = b

important property: feasible set of a convex optimization problem isconvex.

Feasible set of a convex optimization problem is convex.ProofRecall the intersection of (any number of) convex sets is convex–Chapter 2It is the intersection of the domain of the problem

m⋂i=0

dom fi

which is a convex set, with m (convex) sublevel sets {x|fi(x) ≤ 0} andp hyperplanes {x|aTi x = bi}

Example

min f0(x) = x21 + x22

s.t. f1(x) = x1/(1 + x22) ≤ 0

hi(x) = (x1 + x2)2 = 0

f0 is convex; feasible set {(x1, x2)|x1 = −x2 ≤ 0} is convex

not a convex problem (according to our definition): f1 is notconvex, hi is not affine.

equivalent (but not identical) to the convex problem

min x21 + x22

s.t. x1 ≤ 0

x1 + x2 = 0

Figure 10: f0(x) = x21 + x22

Local and global optima

Any locally optimal point of a convex problem is (globally) optimalProof : suppose x is locally optimal, but there exists a feasible y withf0(y) < f0(x)x locally optimal means there is an R > 0 such that

z feasible, ‖z − x‖2 ≤ R⇒ f0(z) ≥ f0(x)

Consider z = θy + (1− θ)x with θ = R/(2‖y − x‖2)‖y − x‖2 > R, so 0 < θ < 1/2

z is a convex combination of two feasible points, hence also feasible

‖z − x‖2 = R/2 and

f0(z) ≤ θf0(y) + (1− θ)f0(x) < f0(x)

which contradicts our assumption that x is locally optimal

Local and global optima

Figure 11: f0(x) = x2

Optimality criterion for differentiable

Suppose that f0(x) is differentiable in a convex optimization problem,∀x, y ∈ dom f0

f0(y) ≥ f0(x) +∇f0(x)T (y − x) (first order condition)

Feasible set Xopt = {x|fi(x) ≤ 0, i = 1, ...,m, aTi x = bi, i = 1, ..., p}Criterion x is optimal if and only if it is feasible and

∇f0(x)T (y − x) ≥ 0 for all feasible y

If nonzero, ∇f0(x) defines a supporting hyperplane to feasible set X atx.

Figure 12: Supporting Hyperplane

1 Suppose x ∈ X and satisfies ∇f0(x)T (y−x) ≥ 0 for all feasible y.Then if y ∈ X, we have f0(y) ≥ f0(x), by the first order condition,f0(y) ≥ f0(x) +∇f0(x)T (y − x). This shows x is an optimal pointfor the problem.

2 Suppose x is optimal, but the condition∇f0(x)T (y − x) ≥ 0 for all feasible y does not hold, i.e., for somey ∈ X we have

∇f0(x)T (y − x) < 0

Consider z(t) = tx+ (1− t)y, t ∈ [0, 1]. Since z(t) is on the linesegment between x and y, and the feasible set is convex. z(t) isfeasible.

Claim that for small positive 1− t, we have f0(z(t)) < f0(x), sincewhich x is not optimal.

dxf0(z(t))|t=1 = ∇f0(x)T (y − x) < 0

so for small positive 1− t, we have f0(z(t)) < f0(x)

unconstrained problem: x is optimal if and only if

x ∈ dom f0, ∇f0(x) = 0

ProofSuppose x is optimal, which means that x ∈ dom f0, and for all feasibley we have ∇f0(x)T (y − x) ≥ 0. Since f0 is differentiable, its domain isopen, so all y sufficiently close to x are feasible.Take y = x− t∇f0(x), where x ∈ R is a parameter. For t small andpositive, y is feasible, and so

∇f0(x)T (y − x) = ∇f0(x)T (−t∇f0(x)) = −t‖∇f0(x)‖22 ≥ 0

Therefore, x is optimal if and only if x ∈ dom f0, ∇f0(x) = 0

equality constrained problem:

min f0(x) s.t.Ax = b

x is optimal if and only if there exists a v such that

x ∈ dom f0, Ax = b, ∇f0(x) +AT v = 0

Proof∀y, Ay = b. Since x is feasible, every feasible y has the form y = x+ vfor some v ∈ N (A). Then ∇f0(x)T v ≥ 0, ∀v ∈ N (A).If a linear function is nonnegative on a subspace, then it must be zeroon the subspace, so it follows that ∇f0(x)T v = 0,∀v ∈ N (A). In otherwords, ∇f0(x)⊥N (A). Using the fact N (A)⊥ = R(AT ),∇f0(x) ∈ R(AT ), there exists a v ∈ Rp such that

∇f0(x) +AT v = 0

minimization over nonnegative orthant

min f0(x) s.t. x � 0

x is optimal if and only if

x ∈ dom f0, x � 0, ∇f0(x) � 0, xi(∇f0(x))i = 0, i = 1, ..., n

Equivalent convex problems

two problems are (informally) equivalent if the solution of one isreadily obtained from the solution of the other, and vice-versaSome common transformations that preserve convexity.eliminating equality constraints

min f0(x)

s.t. fi(x) ≤ 0, i = 1, 2, ...,m

Ax = b

is equivalent to

f0(Fz + x0)

s.t. fi(Fz + x0) ≤ 0, i = 1, 2, ...,m

where F and x0 are such that

Ax = b⇔ x = Fz + x0 for some z

introducing equality constraints

min f0(A0x+ b0)

s.t. fi(Aix+ bi) ≤ 0, i = 1, 2, ...,m

is equivalent to

minx,yi

f0(y0)

s.t. fi(yi) ≤ 0, i = 1, 2, ...,m

yi = Aix+ bi, i = 0, 1, ...,m

introducing slack variables for linear inequalities

min f0(x)

s.t. aTi x ≤ bi, i = 1, 2, ...,m

is equivalent to

minx,s

s.t. aTi x+ si = bi, i = 1, 2, ...,m

si ≥ 0, i = 1, ...,m

epigraph form

minx,t

s.t. f0(x)− t ≤ 0

fi(x) ≤ 0, i = 1, 2, ...,m

Ax = b

minimizing over some variables

minx,t

f0(x1, x2)

s.t. fi(x1) ≤ 0, i = 1, 2, ...,m

is equivalent to

minx,t

f ′0(x1)

s.t. fi(x1) ≤ 0, i = 1, 2, ...,m

where f ′0(x1) = infx2 f0(x1, x2)

Quasiconvex optimization

minx,t

s.t. fi(x) ≤ 0, i = 1, 2, ...,m

Ax = b

with f0 : Rn → R quasiconvex, f1, ..., fm convexThis problem can have locally optimal points that are not (globally)optimal

Figure 13: Quasiconvex optimizationShupeng Gui (Computer Science, UR) Convex Optimization Problems October 16, 2015 36 / 58

Quasiconvex optimization

convex representation of sublevel sets of f0 if f0 is quasiconvex,there exists a family of functions Φt such that:

Φt(x) is convex in x for fixed t

t-sublevel set of f0 is 0-sublevel set of Φt, i.e.,

f0(x) ≤ t⇐ Φt(x) ≤ 0

example

f0(x) =p(x)

with p convex, q concave, and p(x) ≥ 0, q(x) > 0 on dom f0Take Φt(x) = p(x)− tq(x):

for t ≥ 0, Φt convex in x

p(x)/q(x) ≤ t if and only if Φt(x) ≤ 0

Linear program(LP)

min cTx+ d

s.t. Gx � hAx = b

convex problem with affine objective and constraint functions

feasible set is a polyhedron

Linear program(LP)

Formulate the LPs to standard form.

min cTx+ d

s.t. Gx+ s = h

Ax = b

s � 0

Hints: x+, positive part of x; x−, negative part of x. x = x+ − x−.

min cTx+ − cTx− + d

s.t. Gx+ −Gx− + s = h

Ax+ −Ax− = b

s � 0, x+ � 0, x− � 0

Linear-fractional program

min f0(x)

s.t. Gx � hAx = b

linear-fractional program

f0(x) =cTx+ d

eT + f,dom f0(x) = {x|eTx+ f > 0}

a quasiconvex optimization problem; can be solved by visectionalso equivalent to the LP (variables y,z)

min cT y + dz

s.t. Gy � hzAy = bz

eT y + fz = 1, z ≥ 0

Generalized linear-fractional program

f0(x) = maxi=1,...,r

cTi x+ di

eTi x+ fi,domf0(x) = {x|eTi x+ fi, i = 1, ..., r}

a quasiconvex optimization problem; can be solved by bisection.

Quadratic program(QP)

min (1/2)xTPx+ qTx+ r

s.t. Gx � hAx = b

P ∈ Sn+, so objective is convex quadratic

minimize a convex quadratic function over a polyhedron

Example: least-squares

min ‖Ax− b‖22

min xTATAx− 2bTAx+ bT b

Quadratically constained quadratic program(QCQP)

min (1/2)xTP0x+ qT0 x+ r0

s.t. (1/2)xTPix+ qTi x+ ri ≤ 0, i = 1, ...,m

Ax = b

Pi ∈ Sn+; objective and constraints are convex quadratic

if P1, ..., Pm ∈ Sn++, feasible region is intersection of m ellipsoids

and an affine set.

Second-order cone programming

min fTx

s.t. ‖Aix+ bi‖2 ≤ cTi x+ di, i = 1, ...,m

Fx = g

Ai ∈ Rni×n, F ∈ Rp×n

inequalities are called second-order cone (SOC) constraints:

(Aix+ bi, cTi x+ di) ∈ second− order cone inRni+1

for Ai = 0, reduces to an LP; if ci = 0, reduces to a QCQP

more general than QCQP and LP

Robust linear programming

the parameters in optimization problems are often uncertain, e.g., in anLP

min cTx

s.t. aTi x ≤ bi, i = 1, ...,m

there can be uncertainty in c, ai, bitwo common approaches to handling uncertainty(in ai, for simplicity)

deterministic model: constraints must hold for all ai ∈ ε〉min cTx

s.t. aTi x ≤ bi, ∀ai ∈ εi, i = 1, ...,m

stochastic model: ai is random variable; constraints must holdwith probability η

min cTx

s.t. prob(aTi x ≤ bi) ≥ η, i = 1, ...,m

deterministic approach via SOCP

choose an ellipsoid as εi:

εi = {ai + Piu|‖u‖2 ≤ 1} (ai ∈ Rn, Pi ∈ Rn×n)

center is ai, semi-axes determined by singular values/vectors of Pi

robust LP

min cTx

s.t. aTi x ≤ bi, ∀ai ∈ εi, i = 1, ...,m

is equivalent to the SOCP

min cTx

s.t. aTi x+ ‖P Ti x‖2 ≤ bi, i = 1, ...,m

Stochastic approach via SOCP

assume ai is Gaussian with mean ai, covariance Σi(ai ∼ N (ai,Σi))

aTi x is Gaussian r.v. with mean aTi x, variance xTΣix; hence

prob(aTi x ≤ bi) = Φ( bi − aTi x‖Σ1/2

i x‖2

)where Φ(x) = (1/

√2π)

∫ x−∞ e

−t2/2dt is CDF of N (0, 1)

robust LP

min cTx

s.t. prob(aTi x ≤ bi), i = 1, ...,m

with η ≥ 1/2, is equivalent to the SOCP

min cTx

s.t. aTi x+ Φ−1(η)‖Σ1/2i x‖2 ≤ bi, i = 1, ...,m

Geometric programming

monomial function

f(x) = cxa11 xa22 ...x

ann , dom f = Rn

with c > 0; exponent ai can be any real number posynomialfunction: sum of monomials

f(x) =K∑k=1

ckxa1k1 xa2k2 ...xank

n , dom f = Rn++, dom f = Rn

Geometric programming

geometric program (GP)

min f0(x)

s.t. fi(x) ≤ 1, i = 1, ...,m

hi(x) = 1, i = 1, ..., p

with fi posynomial, hi monomial

Geometric program in convex form

change variables t oyi = log xi, and take logarithm of cost, constraints

monomial f(x) = cxa11 xa22 ...x

ann transforms to

log f(ey1 , ..., eyn) = aT y + b (b = log c)

posynomial f(x) =∑K

k=1 ckxa1k1 xa2k2 ...xank

n transforms to

log f(ey1 , ..., eyn) = log( K∑

eaTk y+bk

)(bk = log ck)

geometric program transforms to convex problem

min log(K∑k=1

exp(aT0ky + b0k))

s.t. log(

K∑k=1

exp(aTiky + bik)) ≤ 0, i = 1, ...,m

Gy + d = 0

Generalized inequality constraints

convex problem with generalized inequality constraints

min f0(x)

s.t. fi(x) �Ki 0, i = 1, ...,m

Ax = b

f0 : Rn → R convex;fi : Rn → Rki Ki-convex w.r.t proper cone Ki

same properties as standard convex problem (convex feasible set,local optimum is global, etc.)

conic form problem: special case with affine objective andconstraints

min cTx

s.t. Fx+ g �K 0

Ax = b

Semidefinite program (SDP)

min cTx

s.t. x1F1 + x2F2 + ...+ xnFn +G � 0

Ax = b

with Fi, G ∈ Sk

inequality constraint is called linear matrix inequality (LMI)

includes problems with multiple LMI constraints: for example,

x1F1 + ...+ xnFn + G � 0, x1F1 + ...+ xnFn + G � 0

is equivalent to single LMI

]+...+xn

]� 0

LP and SOCP as SDP

LP and equivalent SDPLP:

min cTx

s.t. Ax � b

min cTx

s.t. diag(Ax− b) � 0

LP and SOCP as SDP

SOCP and equivalent SDPSOCP:

min fTx

s.t. ‖Aix+ bi‖2 ≤ cTi x+ di, i = 1, ...,m

min fTx

[(cTi x+ di)I Aix+ bi(Aix+ bi)

T cTi x+ di

]� 0, i = 1, ...,m

Vector optimization

general vector optimization problem

min(w.r.t.K) f0(x)

s.t. fi(x) ≤ 0, i = 1, ...,m

hi(x) = 0, i = 1, ..., p

vector objective f0 : Rn → Rq, minimized w.r.t. proper cone K ∈ Rq

convex vector optimization problem

min(w.r.t.K) f0(x)

s.t. fi(x) ≤ 0, i = 1, ...,m

hi(x) = 0, i = 1, ..., p

with f0 K-convex, f1, ..., fm convex.

Optimal and Pareto optimal points

set of achievable objective values

O = {f0(x)|x feasible}

feasible x is optimal if f0(x) is the minimum value of Ofeasible x is Pareto optimal if f0(x) is a minimal value of O

Multicriterion optimization

vector optimization problem with K = Rq+

f0(x) = (F1(x), ..., Fq(x))

q different objectives Fi: roughly speaking we want all Fi’s to besmall

feasible x∗ is optimal if

y feasible⇒ f0(x∗) � f0(y)

if there exists an optimal point, the objectives are noncompeting

feasible xpo is Pareto optimal if

y feasible, f0(y) �⇒ f0(xpo) = f0(y)

if there are multiple Pareto optimal values, there is a trade-offbetween the objectives

Scalarization

to find Pareto optimal points: choose λ �K∗ 0 and solve scalar problem

min λT f0(x)

s.t. fi(x) ≤ 0, i = 1, ...,m

hi(x) = 0, i = 1, ..., p

if x is optimal for scalar problem, then it is Pareto-optimal for vectoroptimization problem.For convex vector optimization problems, can find (almost) all Paretooptimal points by varying λ �K∗ 0

Chapter 4 Convex Optimization Problems - Computer Science … · Chapter 4 Convex Optimization...

Documents

Nearly convex segmentation of polyhedra through convex ...masc.cs.gmu.edu/wiki/uploads/GuilinLiu/CAD2016_Shape_Segmenta… · G.Liuetal./Computer-AidedDesign78(2016)137–146 139

Understanding Non-convex Optimization - Praneeth Netrapalli€¦ · •Convex optimization ()is a convex function, 𝒞is convex set •ut “today’s problems”, and this tutorial,

Variational Methods in Convex and Non-Convex Plasticity

CONVEX Remote Procedure Call Protocol Specification RPC Protocol Specificati… · CONVEX Remote Procedure Call Protocol Specification April 1988 CONVEX Computer Corporation Richardson,

Lecture 4 Convex Programming and Lagrange Duality...Lecture 4 Convex Programming and Lagrange Duality (Convex Programming program, Convex Theorem on Alternative, convex duality Optimality

1 Convex Sets, and Convex Functionsangell/ch1.pdf · 1 Convex Sets, and Convex Functions Inthis section, we introduce oneofthemostimportantideas inthe theoryofoptimization, that of

The convex geometry of inverse problemshelper.ipam.ucla.edu/publications/opws2/opws2_9055.pdf · The convex geometry of inverse problems Benjamin Recht Department of Computer Sciences

Piecewise Convex Contouring of Implicit Functions Tao Ju Scott Schaefer Joe Warren Computer Science Department Rice University

convex optimization solvers modeling systems disciplined ... › courses › electrical-engineering-and-computer... · • convex optimization solvers • modeling systems • disciplined

Convex Sets and Convex Functions 1 Convex Sets,

Convex and non-convex worlds in machine learning · Intro Convex solver: bound majorization (Convex) objective: multi-classi cation Non-convexity: deep learning Summary Convex and

Convex Hulls (3D) - Department of Computer Sciencemisha/Spring16/08.pdf · 2016. 2. 25. · Convex Hulls (3D) O’Rourke, Chapter 4. Outline •Polyhedra Polytopes Euler Characteristic

DAGM 2011 Tutorial on Convex Optimization for Computer ... · tugraz Graz University of Technology Overview 1 Introduction 2 Basics of convex analysis 3 Convex optimization 4 A class

DAGM 2011 Tutorial on Convex Optimization for …DAGM 2011 Tutorial on Convex Optimization for Computer Vision Part 3: Convex Solutions for Stereo and Optical Flow Daniel Cremers Computer

geometry of convex functions - CCRMAdattorro/gcf.pdfGeometry of convex functions The link between convex sets and convex functions is via the epigraph: A function is convex if and

Unconstrained convex optimization problems - … convex optimization problems Huanle Xu April 9, 2017 1 Basic Elements of Iterative Algorithms Huanle Xu is with the College of Computer

Computer Vision: Models, Learning and Inference Convexity ...cv192/wiki.files/CV192...CV/ML people) to focus on how to: recognize the problem is convex; or transform a non-convex problem

+ Convex Functions, Convex Sets and Quadratic Programs Sivaraman Balakrishnan

CONVEX BODIES ASSOCIATED WITH A CONVEX BODY 781

Convex Optimization — Boyd & Vandenberghe 3. Convex functions · Convex functions 3–13. Positive weighted sum & composition with aﬃne function nonnegative multiple: αf is convex