Convex Optimization Problems Prof. Daniel P. Palomar · Outline 1 Optimization Problems 2 Convex Optimization 3 Quasi-Convex Optimization 4 Classes of Convex Problems: LP, QP, SOCP,

Convex Optimization Problems

Prof. Daniel P. PalomarThe Hong Kong University of Science and Technology (HKUST)

MAFS6010R - Portfolio Optimization with RMSc in Financial Mathematics

Fall 2019-20, HKUST, Hong Kong

Outline

1 Optimization Problems

2 Convex Optimization

3 Quasi-Convex Optimization

4 Classes of Convex Problems: LP, QP, SOCP, SDP

5 Multicriterion Optimization (Pareto Optimality)

6 Solvers

Outline






6 Solvers

Optimization Problem in Standard Form

General optimization problem in standard form:

minimizex

f0 (x)

subject to fi (x) ≤ 0 i = 1, . . . ,mhi (x) = 0 i = 1, . . . , p

where

x = (x1, . . . , xn) is the optimization variablef0 : Rn −→ R is the objective functionfi : Rn −→ R, i = 1, . . . ,m are inequality constraintfunctionshi : Rn −→ R, i = 1, . . . , p are equality constraintfunctions.

Goal: find an optimal solution x? that minimizes f0 while satisfying allthe constraints.D. Palomar Convex Problems 4 / 48

Optimization Problem in Standard Form

Feasibility:a point x ∈ dom f0 is feasible if it satisfies all the constraints andinfeasible otherwisea problem is feasible if it has at least one feasible point and infeasibleotherwise.

Optimal value:

p? = inf {f0 (x) | fi (x) ≤ 0, i = 1, . . . ,m, hi (x) = 0, i = 1, . . . , p}

p? =∞ if problem infeasible (no x satisfies the constraints)p? = −∞ if problem unbounded below.

Optimal solution: x? such that f (x?) = p? (and x? feasible).

D. Palomar Convex Problems 5 / 48

Global and Local Optimality

A feasible x is optimal if f0 (x) = p?; Xopt is the set of optimal points.A feasible x is locally optimal if is optimal within a ball

Examples:f0 (x) = 1/x , dom f0 = R++: p? = 0, no optimal pointf0 (x) = x3 − 3x : p? = −∞, local optimum at x = 1.


Implicit Constraints

The standard form optimization problem has an explicit constraint:

x ∈ D =m⋂i=0

dom fi ∩p⋂

i=1

dom fi

D is the domain of the problemThe constraints fi (x) ≤ 0, hi (x) = 0 are the explicit constraintsA problem is unconstrained if it has no explicit constraintsExample:

minimizex

log(b − aT x

)is an unconstrained problem with implicit constraint b > aT x .


Feasibility Problem

Sometimes, we don’t really want to minimize any objective, just tofind a feasible point:

findx

x


This feasibility problem can be considered as a special case of ageneral problem:

minimizex

0


where p? = 0 if constraints are feasible and p? =∞ otherwise.


Outline






6 Solvers

Convex Optimization Problem

Convex optimization problem in standard form:

minimizex

f0 (x)

subject to fi (x) ≤ 0 i = 1, . . . ,mAx = b

where f0, f1, . . . , fm are convex and equality constraints are affine.Local and global optima: any locally optimal point of a convexproblem is globally optimal.Most problems are not convex when formulated.Reformulating a problem in convex form is an art, there is nosystematic way.


Example

The following problem is nonconvex (why not?):

minimizex

x21 + x2

2

subject to x1/(1 + x2

2)≤ 0

(x1 + x2)2 = 0

The objective is convex.The equality constraint function is not affine; however, we can rewriteit as x1 = −x2 which is then a linear equality constraint.The inequality constraint function is not convex; however, we canrewrite it as x1 ≤ 0 which again is linear.We can rewrite it as

minimizex

x21 + x2

2

subject to x1 ≤ 0x1 = −x2


Equivalent Reformulations

Eliminating/introducing equality constraints:

minimizex

f0 (x)


is equivalent to

minimizez

f0 (Fz + x0)

subject to fi (Fz + x0) ≤ 0 i = 1, . . . ,m

where F and x0 are such that Ax = b ⇐⇒ x = Fz + x0 for some z .



Introducing slack variables for linear inequalities:

minimizex

f0 (x)

subject to aTi x ≤ bi i = 1, . . . ,m

is equivalent to

minimizex ,s

f0 (x)

subject to aTi x + si = bi i = 1, . . . ,msi ≥ 0



Epigraph form: a standard form convex problem is equivalent to

minimizex ,t

t

subject to f0 (x)− t ≤ 0fi (x) ≤ 0 i = 1, . . . ,mAx = b

Minimizing over some variables:

minimizex ,y

f0 (x , y)

subject to fi (x) ≤ 0 i = 1, . . . ,m

is equivalent to

minimizex

f̃0 (x)

subject to fi (x) ≤ 0 i = 1, . . . ,m

where f̃0 (x) = infy f0 (x , y).


Outline






6 Solvers

Quasiconvex Optimization

Quasi-convex optimization problem in standard form:

minimizex

f0 (x)


where f0 : Rn −→ R is quasiconvex and f1, . . . , fm are convex.Observe that it can have locally optimal points that are not (globally)optimal:



Convex representation of sublevel sets of a quasiconvex function f0:there exists a family of convex functions φt (x) for fixed t such that

f0 (x) ≤ t ⇐⇒ φt (x) ≤ 0.

Example:

f0 (x) =p (x)

q (x)

with p convex, q concave, and p (x) ≥ 0, q (x) > 0 on dom f0. Wecan choose:

φt (x) = p (x)− tq (x)

for t ≥ 0, φt (x) is convex in xp (x) /q (x) ≤ t if and only if φt (x) ≤ 0.



Solving a quasiconvex problem via convex feasibility problems: theidea is to solve the epigraph form of the problem with a sandwichtechnique in t:

for fixed t the epigraph form of the original problem reduces to afeasibility convex problem

φt (x) ≤ 0, fi (x) ≤ 0 ∀i , Ax ≤ b

if t is too small, the feasibility problem will be infeasibleif t is too large, the feasibility problem will be feasiblestart with upper and lower bounds on t (termed u and l , resp.) anduse a sandwitch technique (bisection method): at each iteration uset = (l + u) /2 and update the bounds according to thefeasibility/infeasibility of the problem.


Outline






6 Solvers

Linear Programming (LP)

LP:minimize

xcT x + d

subject to Gx ≤ hAx = b

Convex problem: affine objective and constraint functions.Feasible set is a polyhedron:


`1- and `∞- Norm Problems as LPs

`∞-norm minimization:

minimizex

‖x‖∞subject to Gx ≤ h

Ax = b

is equivalent to the LP

minimizet,x

t

subject to −t1 ≤ x ≤ t1Gx ≤ hAx = b.


`1- and `∞- Norm Problems as LPs

`1-norm minimization:

minimizex

‖x‖1subject to Gx ≤ h

Ax = b

is equivalent to the LP

minimizet,x

∑i ti

subject to −t ≤ x ≤ tGx ≤ hAx = b.


Linear-Fractional Programming

Linear-fractional programming:

minimizex

(cT x + d

)/(eT x + f

)subject to Gx ≤ h

Ax = b

with dom f0 ={x | eT x + f > 0

}.

It is a quasiconvex optimization problem (solved by bisection).Interestingly, the following LP is equivalent:

minimizey ,z

cT y + dz

subject to Gy ≤ hzAy = bzeT y + fz = 1z ≥ 0


Quadratic Programming (QP)

Quadratic programming:

minimizex

(1/2) xTPx + qT x + r

subject to Gx ≤ hAx = b

Convex problem (assuming P ∈ Sn � 0): convex quadratic objectiveand affine constraint functions.Minimization of a convex quadratic function over a polyhedron:


Quadratically Constrained QP (QCQP)

Quadratically constrained QP:

minimizex

(1/2) xTP0x + qT0 x + r0

subject to (1/2) xTPix + qTi x + ri ≤ 0 i = 1, . . . ,mAx = b

Convex problem (assuming Pi ∈ Sn � 0): convex quadratic objectiveand constraint functions.


Second-Order Cone Programming (SOCP)

Second-order cone programming:

minimizex

f T x

subject to ‖Aix + bi‖ ≤ cTi x + di i = 1, . . . ,mFx = g

Convex problem: linear objective and second-order cone constraintsFor Ai row vector, it reduces to an LP.For ci = 0, it reduces to a QCQP.More general than QCQP and LP.


Semidefinite Programming (SDP)

SDP:minimize

xcT x

subject to x1F1 + x2F2 + · · ·+ xnFn � GAx = b

Inequality constraint is called linear matrix inequality (LMI).Convex problem: linear objective and linear matrix inequality (LMI)constraints.Observe that multiple LMI constraints can always be written as asingle one.



LP and equivalent SDP:

minimizex

cT x

subject to Ax ≤ bminimize

xcT x

subject to diag (Ax − b) � 0SOCP and equivalent SDP:

minimizex

f T x

subject to ‖Aix + bi‖ ≤ cTi x + di , i = 1, . . . ,m

minimizex

f T x

subject to[ (

cTi x + di)I Aix + bi

(Aix + bi )T cTi x + di

]� 0, i = 1, . . . ,m



Eigenvalue minimization:

minimizex

λmax (A (x))

where A (x) = A0 + x1A1 + · · ·+ xnAn, is equivalent to SDP

minimizex

t

subject to A (x) � tI

It follows from

λmax (A (x)) ≤ t ⇐⇒ A (x) � tI


Outline






6 Solvers

Scalarization for Multicriterion Problems

Consider a multicriterion optimization with q different objectives:

f0 (x) = (F1 (x) , . . . ,Fq (x)) .

To find Pareto optimal points, minimize the positive weighted sum:

λT f0 (x) = λ1F1 (x) + · · ·+ λqFq (x) .

Example: regularized least-squares:

minimizex

‖Ax − b‖22 + γ ‖x‖22 .


References on Convex Problems

Chapter 4 ofStephen Boyd and Lieven Vandenberghe, Convex Optimization.Cambridge, U.K.: Cambridge University Press, 2004.

https://web.stanford.edu/~boyd/cvxbook/


Outline






6 Solvers

Solvers

A solver is an engine for solving a particular type of mathematicalproblem, such as a convex program.Every programming language (e.g., Matlab, Octave, R, Python, C,C++) has a long list of available solvers to choose from.Solvers typically handle only a certain class of problems, such as LPs,QPs, SOCPs, SDPs, or GPs.They also require that problems be expressed in a standard form.Most problems do not immediately present themselves in a standardform, so they must be transformed into standard form.


Solver Example: Matlab’s linprog

A program for solving LPs:

x = linprog( c, A, b, A_eq, B_eq, l, u )

Problems must be expressed in the following standard form:

minimizex

cTx

subject to Ax ≤ bAeqx = beql ≤ x ≤ u


Conversion to Standard Form: Common Tricks

Representing free variables as the difference of nonnegative variables:

x free =⇒ x+ − x−, x+ ≥ 0, x− ≥ 0

Elimininating inequality constraints using slack variables:

aTx ≤ b =⇒ aTx + s = b, s ≥ 0

Splitting equality constraints into inequalities:

aTx = b =⇒ aTx ≤ b, aTx ≥ b


Solver Example: SeDuMi

A program for solving LPs, SOCPs, SDPs, and related problems:

x = sedumi( A, b, c, K )

Solves problems of the form:

minimizex

cTx

subject to Ax = bx ∈ K , K1 ×K2 × · · · × KL

where each set Ki ⊆ Rni , i = 1, 2, . . . , L is chosen from a very shortlist of cones.The Matlab variable K gives the number, types, and dimensions of thecones Ki .


Solver Example: SeDuMi

Cones supported by SeDuMi:free variables: Rni

a nonnegative orthant: Rni+ (for linear inequalities)

a real or complex second-order cone:

Qn , {(x, y) ∈ Rn × R | ‖x‖2 ≤ y}Qn

c , {(x, y) ∈ Cn × R | ‖x‖2 ≤ y}

a real or complex semidefinite cone:

Sn+ ,

{X ∈ Rn×n | X = XT , X � 0

}Hn

+ ,{X ∈ Cn×n | X = XH , X � 0

}The cones must be arranged in this order, i.e., the free variables first, thenthe nonnegative orthants, then the second-order cones, then thesemidefinite cones.


Example: Norm Approximation

Consider the norm approximation problem:

minimizex

‖Ax− b‖

An optimal value x? minimizes the residuals

rk = aTk x− bk , k = 1, 2, . . . ,m

according to the measure defined by the norm ‖·‖Obviously, the value of x? depends significantly upon the choice ofthat norm...... and so does the process of conversion to standard form


Example: Euclidean or `2-Norm

Norm approximation problem:

minimizex

‖Ax− b‖2 =(∑m

k=1(aTk x− bk

)2)1/2

No need to use any solver here: this is a least squares (LS) problem,with an analytic solution:

x? = (ATA)−1ATb

In Matlab or Octave, a single command computes the solution:

>‌> x = A \ b

Similarly, in R:>‌> x = solve(A, b)


Example: Chebyshev or `∞-Norm


minimizex

‖Ax− b‖∞ = minimizex

max1≤k≤m

∣∣aTk x− bk∣∣

This can be expressed as a linear program:

minimizex,t

t

subject to −t1 ≤ Ax− b ≤ t1

or, equivalently,

minimizex,t

[0T 1

] [ xt

]subject to

[A −1−A −1

] [xt

]≤[

b−b

]


Example: Chebyshev or `∞-Norm

Recall the final formulation:

minimizex,t

[0T 1

] [ xt

]subject to

[A −1−A −1

] [xt

]≤[

b−b

]

Matlab’s linprog call:

>‌> xt = linprog( [zeros(n,1); 1], ...[A,-ones(m,1); -A,-ones(m,1)], ...[b; -b] )

>‌> x = xt(1:n)


Example: Manhattan or `1-Norm


minimizex

‖Ax− b‖1 = minimizex

∑mk=1

∣∣aTk x− bk∣∣

This can be expressed as a linear program:

minimizex,t

1T t

subject to −t ≤ Ax− b ≤ t

or, equivalently,

minimizex,t

[0T 1T

] [ xt

]subject to

[A −I−A −I

] [xt

]≤[

b−b

]


Example: Manhattan or `1-Norm

Recall the final formulation:

minimizex,t

[0T 1T

] [ xt

]subject to

[A −I−A −I

] [xt

]≤[

b−b

]

Matlab’s linprog call:

>‌> xt = linprog( [zeros(n,1); ones(n,1)], ...[A,-eye(m,1); -A,-eye(m,1)], ...[b; -b] )

>‌> x = xt(1:n)


Example: Constrained Euclidean or `2-Norm

Constrained norm approximation problem:

minimizex

‖Ax− b‖2subject to Cx = d

l ≤ x ≤ u

This is not a least squares problem, but it is QP and an SOCP.This can be expressed as

minimizex,y,t,sl ,su

t

subject to Ax− b = yCx = dx− sl = lx + su = usl , su ≥ 0‖y‖2 ≤ t


Example: Constrained Euclidean or `2-Norm

Equivalently: minimizex,y,t,sl ,su

[0T 0T 0T 0T 1

]x̄

subject to

A −ICI −II I

x̄ ≤

bdlu

x̄ ∈ Rn × Rn

+ × Rn+ ×Qm

SeDuMi call:

>‌> AA = [ A, zeros(m,n), zeros(m,n), -eye(m), 0;C, zeros(p,n), zeros(p,n) zeros(p,n), 0;eye(n), -eye(n), zeros(n,n), zeros(n,n), 0;eye(n), zeros(n,n), eye(n), zeros(n,n), 0 ]

>‌> bb = [ b; d; l; u ]>‌> cc = [ zeros(3*n+m,1); 1 ]>‌> K.f = n; K.l = 2*n; K.q = m + 1;>‌> xsyz = sedumi( AA, bb, cc, K )>‌> x = xsyz(1:n)D. Palomar Convex Problems 46 / 48

Modeling Frameworks: cvx

A modeling framework simplifies the use of a numerical technology byshielding the user from the underlying mathematical details.Examples: SDPSOL, YALMIP, cvx, etc.cvx is designed to support convex optimization or, more specifically,disciplined convex programming (available in Matlab, Python, R, Julia,Octave, etc.)People don’t simply write optimization problems and hope that theyare convex; instead they draw from a "mental library"of functions andsets with known convexity properties and combine them in ways thatconvex analysis guarantees will produce convex results.Disciplined convex programming formalizes this methodology.Links:

cvx: http://cvxr.comcvx user guide: http://web.cvxr.com/cvx/doc/CVX.pdfcvx for R (cvxr): https://github.com/cvxgrp/CVXR


Thanks

For more information visit:

https://www.danielppalomar.com

Documents

Convex Optimization Problems Prof. Daniel P. Palomar · Outline 1 Optimization Problems 2 Convex Optimization 3 Quasi-Convex Optimization 4 Classes of Convex Problems: LP, QP, SOCP,