Upload
vandieu
View
230
Download
5
Embed Size (px)
Citation preview
Convex Optimization inCommunications and Signal Processing
Convex Optimization inCommunications and Signal Processing
Prof. Dr.-Ing. Wolfgang Gerstacker 1
University of Erlangen-NürnbergInstitute for Digital Communications
National Technical University of Ukraine, KPI, Kiev, April 2015
1These lecture slides are largely based on the book ”ConvexOptimization” by Steven Boyd and Lieven Vandenberghe and thecorresponding slides. Many thanks to Prof. Boyd for the permissionto use his materials for this course.
1 Introduction
2 Convex sets
3 Convex functions
4 Convex optimization problems
5 Duality
6 Applications in communications
7 Algorithms – Equality constrained minimization
8 Conclusions
Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)00
1. Introduction1. Introduction
OutlineOutline
1 Introduction
2 Convex sets
Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)11
1. Introduction1. Introduction
IntroductionIntroduction
mathematical optimization
least-squares and linear programming
convex optimization
course goals and topics
nonlinear optimization
brief history of convex optimization
Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)22
1. Introduction1. Introduction
Mathematical optimizationMathematical optimization
(mathematical) optimization problem
minimize f0(x)subject to fi(x) ≤ bi, i = 1, ...,m
x = (x1, ..., xn): optimization variables
f0 : Rn Ï R: objective function
fi : Rn Ï R, i = 1, ...,m: constraint functions
optimal solution x∗ has smallest value of f0 among allvectors that satisfy the constraints
Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)33
1. Introduction1. Introduction
ExamplesExamplesportfolio optimization
variables: amounts invested in different assetsconstraints: budget, max./min. investment perasset, minimum returnobjective: overall risk or return variance
device sizing in electronic circuitsvariables: device widths and lengthsconstraints: manufacturing limits, timingrequirements, maximum areaobjective: power consumption
data fittingvariables: model parametersconstraints: prior information, parameter limitsobjective: measure of misfit or prediction error
Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)44
1. Introduction1. Introduction
Examples (communications and signal processing)Examples (communications and signal processing)
channel estimation
detection
filter design
beamformer design
network optimization
power control
. . .
Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)55
1. Introduction1. Introduction
Solving optimization problemsSolving optimization problems
general optimization problem
very difficult to solve
methods involve some compromise, e.g., very longcomputation time, or not always finding the solution
exceptions: certain problem classes can be solvedefficiently and reliably
least-squares problems
linear programming problems
convex optimization problems
Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)66
1. Introduction1. Introduction
Least-squares ILeast-squares I
minimize ‖Ax − b‖22interpretation
linear system of equations Ax = b can be alsowritten as aT
i x = bi, i = 1, . . . , k, where aTi are the
rows of A ∈ Rk×n and bi are the entries of b ∈ Rk
‖Ax − b‖22 is equal tok∑
i=1(aTi x − bi)2, i.e. the sum of
squared equation errors
solving least-squares problems
analytical solution: x∗ = (ATA)−1ATbreliable and efficient algorithms and software
Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)77
1. Introduction1. Introduction
Least-squares IILeast-squares II
computation time proportional to n2k (A ∈ Rk×n); lessif structured
a mature technology
using least-squares
least-squares problems are easy to recognize
a few standard techniques increase flexibility (e.g .,including weights, adding regularization terms)
Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)88
1. Introduction1. Introduction
Calculation of least-squares solution ICalculation of least-squares solution Icost function:
J(x) = ‖Ax − b‖22 = (Ax − b)T (Ax − b)= xTATAx − bTAx − xTATb + bTb
we need the gradient vector
∂J∂x = [ ∂J
∂x1 ∂J∂x2 . . . ∂J
∂xn
]T
differentiation rules for some simple costfunctions:
J(x) = cTx = c1x1 + . . .+ cnxn∂J∂x = [ c1 . . . cn ]T = c
J(x) = xTCx∂J∂x = 2 Cx
Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)99
1. Introduction1. Introduction
Calculation of least-squares solution IICalculation of least-squares solution II
ÍÑ for the least-squares cost function we get
∂J∂x = 2 ATAx − 2 ATb
setting the gradient vector to zero yields the solution
x∗ = (ATA)−1ATb
this is the global minimum since the function isdifferentiable, there is only one local extremum andxTATAx = (Ax)T (Ax) ≥ 0 ∀x
Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)1010
1. Introduction1. Introduction
Modified least-squares cost functionsModified least-squares cost functions
weighted cost function
J(x) = k∑i=1 wi (aT
i x − bi)2with nonnegative weight factors wi
regularized cost function
J(x) = k∑i=1 (aT
i x − bi)2 + ρn∑
i=1 x2i
both cost functions can be minimized with similarcalculations as the original least-squares cost function
Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)1111
1. Introduction1. Introduction
Linear programmingLinear programming
minimize cTxsubject to aT
i x ≤ bi, i = 1, ...,msolving linear programs
no analytical formula for solutionreliable and efficient algorithms and software(e.g. Simplex algorithm by Dantzig)computation time proportional to n2m if m ≥ n; lessif structureda mature technology
using linear programmingnot as easy to recognize as least-squares problemsa few standard tricks used to convert problems intolinear programs (e.g ., problems involving `1- or`∞-norms, piecewise-linear functions)
Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)1212
1. Introduction1. Introduction
example for a linear programming problem: Chebyshevapproximation problem
minimize maxi=1,...,m |aTi x − bi|
Problem is equivalent to the linear program
minimize tsubject to aT
i x − t ≤ bi, i = 1, ...,m−aT
i x − t ≤ −bi, i = 1, ...,m−t ≤ 0
with variables x and t
Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)1313
1. Introduction1. Introduction
Convex optimization problemConvex optimization problem
minimize f0(x)subject to fi(x) ≤ bi, i = 1, ...,m
objective and constraint functions are convex:
fi(αx + βy) ≤ αfi(x) + βfi(y)if α+ β = 1, α ≥ 0, β ≥ 0includes least-squares problems and linearprograms as special cases
Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)1414
1. Introduction1. Introduction
solving convex optimization problems
no analytical solution
reliable and efficient algorithms (e.g. interior pointalgorithms)
computation time (roughly) proportional tomax{n3,n2m,F}, where F is cost of evaluating fi’sand their first- and second-order derivatives
almost a technology for subclasses of convexproblems like second-order cone programming
Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)1515
1. Introduction1. Introduction
using convex optimization
often difficult to recognize (is a given functionconvex?)
many tricks for transforming problems into convexform
surprisingly many problems can be solved viaconvex optimization
in particular, during the last 10-15 years, a varietyof problems in communications and signalprocessing could be solved via convex optimization
Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)1616
1. Introduction1. Introduction
Course goals and topicsCourse goals and topics
goals1 recognize/formulate problems as convex
optimization problems2 characterize optimal solution (optimal power
distribution), give limits of performance, etc.3 apply techniques to problems in communications
and signal processing4 understand the basic principles of convex
optimization algorithms
topics1 convex sets, functions, optimization problems2 examples and applications3 algorithms
Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)1717
1. Introduction1. Introduction
Nonlinear optimizationNonlinear optimizationtraditional techniques for general nonconvex problemsinvolve compromiseslocal optimization methods (nonlinear programming)
find a point that minimizes f0 among feasible pointsnear itfast, can handle large problemsrequire initial guessprovide no information about distance to (global)optimumExample: gradient search
global optimization methodsfind the (global) solutionworst-case complexity grows exponentially withproblem size
these algorithms are often based on solving convexsubproblems
Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)1818
1. Introduction1. Introduction
Brief history of convex optimization IBrief history of convex optimization I
theory (convex analysis): ca. 1900 - 1970
algorithms
1947: simplex algorithm for linear programming(Dantzig)
1960s: early interior-point methods (Fiacco &McCormick, Dikin, . . . )
1970s: ellipsoid method and other subgradientmethods
1980s: polynomial-time interior-point methods forlinear programming (Karmarkar 1984)
late 1980s–now: polynomial-time interior-pointmethods for nonlinear convex optimization(Nesterov & Nemirovski 1994)
Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)1919
1. Introduction1. Introduction
Brief history of convex optimization IIBrief history of convex optimization II
applications
before 1990: mostly in operations research; few inengineering
since 1990: many new applications in engineering(control, signal processing, communications, circuitdesign, . . . ); new problem classes (semidefinite andsecond-order cone programming, robustoptimization)
Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)2020
2. Convex sets2. Convex sets
OutlineOutline
1 Introduction
2 Convex sets
3 Convex functions
Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)2121
2. Convex sets2. Convex sets
Convex setsConvex sets
vector spaces and subspaces
affine and convex sets
some important examples
operations that preserve convexity
separating and supporting hyperplanes
Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)2222
2. Convex sets2. Convex sets
Vector spaces and subspacesVector spaces and subspaces
variables to be optimized are typically collected invectors
Ñ we need the concept of a vector space frommathematics: collection of vectors for which vectoraddition and multiplication of vectors with scalarsare defined; these operations must fulfill a numberof requirements (axioms) such as associativity ofaddition, commutativity of addition etc.
one important property of a vector space isclosure: the result of addition and scalarmultiplication, respectively, belongs also to thevector space
subspace: subset of a vector space that is closedunder addition and scalar multiplication
Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)2323
2. Convex sets2. Convex sets
Affine setAffine set
line through x1,x2: all points
x = θx1 + (1− θ)x2 = x2 + θ (x1 − x2) (θ ∈ R)
affine set: contains the line through any two distinctpoints in the set
Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)2424
2. Convex sets2. Convex sets
example: solution set of linear equations {x | Ax = b}assume that we have two solutions x1, x2
Ax1 = b, Ax2 = b
Ñ θAx1 = θ b, (1− θ) Ax2 = (1− θ) b
A (θ x1 + (1− θ) x2)) = (θ + (1− θ)) b = b
θ x1 + (1− θ) x2, θ ∈ R is also a solution, set is affine
(conversely, every affine set can be expressed assolution set of system of linear equations)
Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)2525
2. Convex sets2. Convex sets
Convex setConvex set
line segment between x1 and x2: all points
x = θx1 + (1− θ)x2with 0 ≤ θ ≤ 1convex set: contains line segment between any twopoints in the set
x1,x2 ∈ C, 0 ≤ θ ≤ 1 ÍÑ θx1 + (1− θ)x2 ∈ C
examples: (one convex, two nonconvex sets)
Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)2626
2. Convex sets2. Convex sets
Convex combination and convex hullConvex combination and convex hull
convex combination of x1, . . . ,xk: any point x of theform
x = θ1x1 + θ2x2 + . . .+ θkxk
with θ1 + . . .+ θk = 1, θi ≥ 0convex hull conv S: set of all convex combinations ofpoints in S
conv S is also smallest convex set that contains S
Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)2727
2. Convex sets2. Convex sets
Cones and convex conesCones and convex conesset C is called a cone if for any x ∈ C and θ ≥ 0, θ x ∈ C
conic (nonnegative) combination of x1 and x2: anypoint of the form
x = θ1x1 + θ2x2with θ1 ≥ 0, θ2 ≥ 0
two–dimensional pie slice with apex 0 and edgespassing through x1 and x2
Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)2828
2. Convex sets2. Convex sets
convex cone: set is convex and a cone; set thatcontains all conic combinations of points in the set
Ñ for any x1, x2 ∈ C, θ1 ≥ 0, θ2 ≥ 0, we haveθ1 x1 + θ2 x2 ∈ C
Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)2929
2. Convex sets2. Convex sets
Some important factsSome important facts
any line is affine; if it passes through zero, it is asubspace
any line segment is convex, but not affine
any subspace is affine, and a convex cone
any ray, i.e. a set {x0 + θ v | θ ≥ 0}, v 6= 0, is convexbut not affine; if x0 = 0, it is a convex cone
Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)3030
2. Convex sets2. Convex sets
Hyperplanes and halfspacesHyperplanes and halfspaceshyperplane: set of the form {x | aTx = b}(a 6= 0)
halfspace: set of the form {x | aTx ≤ b}(a 6= 0)
a is the normal vectorhyperplanes are affine and convex; halfspaces areconvex
Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)3131
2. Convex sets2. Convex sets
Alternative representation of hyperplanesAlternative representation of hyperplanes
hyperplane in a two-dimensional vector space(line):
x = x0 + θ c, θ ∈ R
c is orthogonal to a, cT a = 0Ñ aTx = aTx0︸ ︷︷ ︸
b
+θ aTc︸︷︷︸0hyperplane in a three-dimensional vector space(plane):
x = x0 + θ1 c1 + θ2 c2, θ1, θ2 ∈ R
c1, c2 are orthogonal to a, cT1 a = 0, cT2 a = 0Ñ aTx = aTx0︸ ︷︷ ︸
b
+θ1 aTc1︸ ︷︷ ︸0+θ2 aTc2︸ ︷︷ ︸0
Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)3232
2. Convex sets2. Convex sets
Euclidean balls and ellipsoidsEuclidean balls and ellipsoids(Euclidian) ball with center xc and radius r:
B(xc, r) = {x | ‖x − xc‖2 ≤ r} = {xc + ru | ‖u‖2 ≤ 1}ellipsoid: set of the form
{x | (x − xc)TP−1(x − xc) ≤ 1}with P ∈ Sn++ (Sn++: set of all symmetric positive definitematrices)
other representation: {xc + Au | ‖u‖2 ≤ 1} with A squareand nonsingular (A = P1/2)Euclidean balls and ellipsoids are convex sets
Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)3333
2. Convex sets2. Convex sets
Norm balls and norm conesNorm balls and norm cones
norm: a function ‖ · ‖ that satisfies
‖x‖ ≥ 0; ‖x‖ = 0 if and only if x = 0‖tx‖ = |t| ‖x‖ for t ∈ R‖x + y‖ ≤ ‖x‖+ ‖y‖
notation: ‖ · ‖ is general (unspecified) norm; ‖ · ‖symb isparticular norm
examples:
`1-norm: ‖x‖1 = |x1|+ |x2|+ . . .+ |xn|`2-norm: ‖x‖2 =√|x1|2 + |x2|2 + . . .+ |xn|2(Euclidean norm)
`∞-norm: ‖x‖∞ = maxi=1,...,n |xi|`p-norm: ‖x‖p = p
√|x1|p + |x2|p + . . .+ |xn|p
Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)3434
2. Convex sets2. Convex sets
norm ball with center xc and radius r: {x | ‖x−xc‖ ≤ r}
norm cone: {(x, t) | ‖x‖ ≤ t}Euclidean norm cone is calledsecond-order cone or ice-creamcone or quadratic cone (since itcan be defined by a quadraticinequality)
norm balls and cones are convex
Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)3535
2. Convex sets2. Convex sets
PolyhedraPolyhedrasolution set of finitely many linear inequalities andequalities
Ax � b, Cx = d
(A ∈ Rm×n,C ∈ Rp×n,� is componentwise inequality)
polyhedron is intersection of finite number ofhalfspaces and hyperplanes
Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)3636
2. Convex sets2. Convex sets
example: nonnegative orthant
Rn+ = {x ∈ Rn | xi ≥ 0, i = 1, . . . ,n} = {x ∈ Rn |x � 0}
Rn+ is polyhedron and convex cone
affine sets (e.g. subspaces, hyperplanes, lines),rays, line segments, halfspaces are all polyhedra
bounded polyhedron is called polytope
all polyhedra are convex
Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)3737
2. Convex sets2. Convex sets
Positive semidefinite conePositive semidefinite cone
notation:
Sn is set of symmetric n × n matrices and forms alinear space or vector space
Sn+ = {X ∈ Sn | X � 0}: positive semidefinite n × nmatrices
X ∈ Sn+ ⇐Ñ zTXz ≥ 0 for all z
Sn+ is a convex cone
Sn++ = {X ∈ Sn | X � 0}: positive definite n × nmatrices
X ∈ Sn++ ⇐Ñ zTXz > 0 for all z 6= 0
Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)3838
2. Convex sets2. Convex sets
Operations that preserve convexityOperations that preserve convexity
practical methods for establishing convexity of a set C1 apply definition
x1,x2 ∈ C, 0 ≤ θ ≤ 1 ÍÑ θx1 + (1− θ)x2 ∈ C
2 show that C is obtained from simple convex sets(hyperplanes, halfspaces, norm balls, . . . ) byoperations that preserve convexity
intersectionaffine functionsperspective functionlinear-fractional functions
Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)3939
2. Convex sets2. Convex sets
IntersectionIntersection
the intersection of (any number of) convex sets isconvex
Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)4040
2. Convex sets2. Convex sets
Affine functionAffine function
suppose f : Rn Ï Rm is affine (f (x) = Ax + b withA ∈ Rm×n,b ∈ Rm)
the image of a convex set under f is convex
S ⊆ Rn convex ÍÑ f (S) = {f (x) | x ∈ S} convex
the inverse image f−1(C) of a convex set under f isconvex
C ⊆ Rm convex ÍÑ f−1(C) = {x ∈ Rn | f (x) ∈ C} convex
examples
scaling, translation
projection
Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)4141
2. Convex sets2. Convex sets
Perspective and linear-fractional functionPerspective and linear-fractional function
perspective function P : Rn+1 Ï Rn:
P(x, t) = x/t, dom P = {(x, t) | t > 0}images and inverse images of convex sets underperspective functions are convex
Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)4242
2. Convex sets2. Convex sets
linear-fractional function f : Rn Ï Rm:
f (x) = Ax + bcTx + d , dom f = {x | cTx + d > 0}
f (·) may be viewed as concatenation of P(·) and afunction g(·), f (·) = P(·) ◦ g(·), with
g(x) = [ AcT
]x + [ b
d
]images and inverse images of convex sets underlinear-fractional functions are convex
Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)4343
2. Convex sets2. Convex sets
Separating hyperplane theoremSeparating hyperplane theoremif C and D are disjoint convex sets, then there existsa 6= 0, b such that
aTx ≤ b for x ∈ C, aTx ≥ b for x ∈ D
the hyperplane {x | aTx = b} separates C and D
strict separation (i.e., at least in one of bothinequalities, ”≤” resp. ”≥” can be replaced by ”<” resp.”>”) requires additional assumptions (e.g ., C is closed,D is a singleton)
Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)4444
2. Convex sets2. Convex sets
Supporting hyperplane theoremSupporting hyperplane theoremsupporting hyperplane to set C at boundary point x0:
{x | aTx = aTx0}where a 6= 0 and aTx ≤ aTx0 for all x ∈ C
hyperplane is tangent to C at x0halfspace defined by the hyperplane(in ”opposite direction” of a) contains C
Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)4545
2. Convex sets2. Convex sets
supporting hyperplane theorem: if C is convex,then there exists a supporting hyperplane at everyboundary point of C
Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)4646
3. Convex functions3. Convex functions
OutlineOutline
2 Convex sets
3 Convex functions
4 Convex optimization problems
Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)4747
3. Convex functions3. Convex functions
Convex functionsConvex functions
basic properties and examples
operations that preserve convexity
Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)4848
3. Convex functions3. Convex functions
DefinitionDefinition
f : Rn Ï R is convex if dom f is a convex set and
f (θx + (1− θ)y) ≤ θf (x) + (1− θ)f (y)for all x,y ∈ dom f , 0 ≤ θ ≤ 1
f is concave if −f is convex
f is strictly convex if dom f is convex and
f (θx + (1− θ)y) < θf (x) + (1− θ)f (y)for x,y ∈ dom f ,x 6= y, 0 < θ < 1
Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)4949
3. Convex functions3. Convex functions
convexity means that the line segment through any twopoints of the graph of the function is above the graph
consider a function of one scalar variable, f (z)straight line through (x, f (x)) and (y, f (y)): g(z) = a z + bg(x) = a x + b
!= f (x), g(y) = a y + b!= f (y)
Ñ a = (f (y)− f (x))/(y − x), b = f (x)− (f (y)− f (x))/(y − x) · xÑ g(z) = f (x) + (f (y)− f (x))/(y − x) · (z − x)g(θ x + (1− θ) y) = f (x) + (f (y)− f (x))/(y − x) (x (θ − 1) + (1− θ) y)= f (x) + (1− θ) (f (y)− f (x))= θ f (x) + (1− θ) f (y) ≥ f (θ x + (1− θ) y)
Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)5050
3. Convex functions3. Convex functions
Examples on RExamples on R
convex:
affine: ax + b on R, for any a,b ∈ Rexponential: eax, for any a ∈ Rpowers: xα on R++, for α ≥ 1 or α ≤ 0powers of absolute value: |x|p on R, for p ≥ 1negative entropy: x log x on R++
concave
affine: ax + b on R, for any a,b ∈ Rpowers: xα on R++, for 0 ≤ α ≤ 1logarithm: log x on R++
Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)5151
3. Convex functions3. Convex functions
Examples on Rn and Rm×nExamples on Rn and Rm×n
affine functions are convex and concave
all norms are convex due to the following inequalityvalid for any norm function f (x):
f (θ x + (1− θ) y) ≤ f (θ x) + f ((1− θ) y) = θ f (x) + (1− θ) f (y)examples on Rn
affine function f (x) = aTx + bnorms: ‖x‖p = (∑n
i=1 |xi|p)1/p forp ≥ 1; ‖x‖∞ = maxk |xk|
Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)5252
3. Convex functions3. Convex functions
examples on Rm×n (m × n matrices)affine function
f (X) = tr(ATX) + b = m∑i=1
n∑j=1 AijXij + b
spectral (maximum singular value) norm
f (X) = ‖X‖2 = σmax(X) = (λmax(XTX))1/2
Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)5353
3. Convex functions3. Convex functions
Restriction of a convex function to a lineRestriction of a convex function to a line
f : Rn Ï R is convex if and only if the function g : RÏ R,
g(t) = f (x + tv), dom g = {t | x + tv ∈ dom f}
is convex (in t) for any x ∈ dom f , v ∈ Rn
can check convexity of f by checking convexity offunctions of one variable
Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)5454
3. Convex functions3. Convex functions
First-order conditionFirst-order conditionf is differentiable if dom f is open and the gradient
∇f (x) = (∂f (x)∂x1 , ∂f (x)
∂x2 , · · · , ∂f (x)∂xn
),
exists at each x ∈ dom f
1st-order condition: differentiable f with convexdomain is convex iff
f (y) ≥ f (x) +∇f (x)T (y − x) for all x,y ∈ dom f
first-order Taylor series approximationof f is global underestimator
Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)5555
3. Convex functions3. Convex functions
first–order approximation establishes a global lowerbound
from local information about the function (functionvalue, gradient vector) we can derive global information(global underestimator)
∇f (x) = 0 Ñ f (y) ≥ f (x)Ñ x is global minimizer of f
Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)5656
3. Convex functions3. Convex functions
Second-order conditionsSecond-order conditions
f is twice differentiable if dom f is open and theHessian ∇2f (x) ∈ Sn,
∇2f (x)ij = ∂2f (x)∂xi∂xj
, i, j = 1, . . . ,n,exists at each x ∈ dom f
2nd-order conditions: for twice differentiable f withconvex domain
f is convex if and only if
∇2f (x) � 0 for all x ∈ dom f
if ∇2f (x) � 0 for all x ∈ dom f , then f is strictlyconvex
Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)5757
3. Convex functions3. Convex functions
ExamplesExamplesquadratic function: f (x) = (1/2)xTPx + qTx + r (withP ∈ Sn)
∇f (x) = Px + q, ∇2f (x) = Pconvex if P � 0least-squares objective: f (x) = ‖Ax − b‖22
∇f (x) = 2AT (Ax − b), ∇2f (x) = 2ATA
convex (for any A)
quadratic-over-linear:f (x, y) = x2/y∇2f (x, y) = 2
y3[
y−x
] [y−x
]T� 0
convex for y > 0Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)5858
3. Convex functions3. Convex functions
expression for the Hessian can be verified via thepartial derivatives:
∂2f (x, y)∂x∂x = 2
y
∂2f (x, y)∂y∂y = 2 x2
y3∂2f (x, y)∂x∂y = −2 x
y2∂2f (x, y)∂y∂x = −2 x
y2
Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)5959
3. Convex functions3. Convex functions
log-sum-exp: f (x) = log∑nk=1 exp xk is convex
∇2f (x) = 11Tz
diag(z)− 1(1Tz)2 zzT (zk = exp xk)to show ∇2f (x) � 0, we must verify that vT∇2f (x)v ≥ 0 forall v:
vT∇2f (x)v = (∑k zkv2k )(∑k zk)− (∑k vkzk)2(∑k zk)2 ≥ 0
since (∑k vkzk)2 ≤ (∑k zkv2k )(∑k zk) (from Cauchy-Schwarz
inequality (∑ni=1 ai bi)2 ≤∑n
i=1 a2i ·∑n
i=1 b2i , use ai = √zi,
bi = √zi vi)
geometric mean: f (x) = (∏nk=1 xk)1/n on Rn++ is concave
(similar proof as for log-sum-exp)
Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)6060
3. Convex functions3. Convex functions
Epigraph and sublevel setEpigraph and sublevel setα-sublevel set of f : Rn Ï R:
Cα = {x ∈ dom f | f (x) ≤ α}
sublevel sets of convex functions are convex (converseis false)
epigraph of f : Rn Ï R:
epi f = {(x, t) ∈ Rn+1 | x ∈ dom f , f (x) ≤ t}
(epi means ”above”)f is convex if and only if epi f is a convex set
Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)6161
3. Convex functions3. Convex functions
Jensen’s inequalityJensen’s inequality
basic inequality: if f is convex, then for 0 ≤ θ ≤ 1,
f (θx + (1− θ)y) ≤ θf (x) + (1− θ)f (y)extension: if f is convex, then
f (E z) ≤ E f (z)for any random variable z
Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)6262
3. Convex functions3. Convex functions
basic inequality is special case with discretedistribution
prob(z = x) = θ, prob(z = y) = 1− θgeneral discrete distribution
f( k∑
i=1 prob(zi) zi)≤
k∑i=1 prob(zi) f (zi)
Ñ f (θ1 x1+θ2 x2+. . .+θk xk) ≤ θ1 f (x1)+θ2 f (x2)+. . .+θk f (xk)with θi ≥ 0,
k∑i=1 θi = 1
Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)6363
3. Convex functions3. Convex functions
continuous distribution p(x)f(∫
S
p(x) x dx)≤∫S
p(x) f (x) dx
with p(x) ≥ 0,∫
S p(x) dx = 1, S ⊆ dom fdithering with respect to a deterministic vector x
Ef (x + z) ≥ f (x)with zero–mean random vector z
Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)6464
3. Convex functions3. Convex functions
Operations that preserve convexityOperations that preserve convexity
practical methods for establishing convexity of afunction
1 verify definition (often simplified by restricting to aline)
2 for twice differentiable functions, show ∇2f (x) � 03 show that f is obtained from simple convex
functions by operations that preserve convexitynonnegative weighted sumcomposition with affine functionpointwise maximum and supremumcompositionminimizationperspective
Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)6565
3. Convex functions3. Convex functions
Positive weighted sum & composition with affine functionPositive weighted sum & composition with affine function
nonnegative multiple: αf is convex if f is convex,α ≥ 0sum: f1 + f2 convex if f1, f2 convex (extends to infinitesums, integrals)
f1(θ x + (1− θ) y) + f2(θ x + (1− θ) y)≤ θ f1(x) + (1− θ) f1(y) + θ f2(x) + (1− θ) f2(y)= θ (f1(x) + f2(x)) + (1− θ) (f1(y) + f2(y))
Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)6666
3. Convex functions3. Convex functions
composition with affine function: f (Ax + b) isconvex if f is convex
f (A (θ x + (1− θ) y) + b)= f (θ (Ax + b) + (1− θ) (Ay + b))≤ θ f (Ax + b) + (1− θ) f (Ay + b)
Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)6767
3. Convex functions3. Convex functions
Pointwise maximum IPointwise maximum I
if f1, . . . , fm are convex, then f (x) = max{f1(x), . . . , fm(x)} isconvex
examples
piecewise-linear function: f (x) = maxi=1,··· ,m(aTi x + bi)
is convex
sum of r largest components of x ∈ Rn:
f (x) = x[1] + x[2] + · · ·+ x[r]is convex (x[i] is ith largest component of x)
proof:
f (x) = max{xi1 + xi2 + · · ·+ xir | 1 ≤ i1 < i2 < · · · < ir ≤ n}
Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)6868
3. Convex functions3. Convex functions
Pointwise maximum IIPointwise maximum II
consider e.g. x = [x1 x2 x3]T (n = 2),f (x) = x[1] + x[2] (r = 2)
equivalently, f can be expressed as
f (x) = max{x1 + x2, x2 + x3, x1 + x3}
Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)6969
3. Convex functions3. Convex functions
Pointwise supremumPointwise supremum
supremum: ”smallest upper bound”
if f (x,y) is convex in x for each y ∈ A, then
g(x) = supy∈A
f (x,y)is convex
Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)7070
3. Convex functions3. Convex functions
Composition with scalar functionsComposition with scalar functionscomposition of g : Rn Ï R and h : RÏ R:
f (x) = h(g(x))f is convex if
g convex, h convex, h nondecreasingg concave, h convex, h nonincreasing
proof (for n = 1, differentiable g ,h on R (functionsand derivatives exist for R))
f ′′(x) = h′′(g(x))g ′(x)2 + h′(g(x))g ′′(x)note: monotonicity must hold for extended-valueextension h
examplesexp g(x) is convex if g is convex1/g(x) is convex if g is concave and positive
Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)7171
3. Convex functions3. Convex functions
Vector compositionVector compositioncomposition of g : Rn Ï Rk and h : Rk Ï R:
f (x) = h(g (x)) = h(g1(x), g2(x), . . . , gk(x))f is convex if:
gi convex, h convex, h nondecreasing in eachargumentgi concave, h convex, h nonincreasing in eachargument
proof (for n = 1, differentiable g ,h)
f ′′(x) = g ′(x)T∇2h(g (x))g ′(x) +∇h(g (x))Tg ′′(x)examples∑m
i=1 log gi(x) is concave if gi are concave andpositivelog∑m
i=1 exp gi(x) is convex if gi are convex
Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)7272
3. Convex functions3. Convex functions
Minimization IMinimization I
if f (x,y) is convex in (x,y) and C is a convex set, then
g(x) = infy∈C
f (x,y)is convex; infimum: "greatest lower bound"
Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)7373
3. Convex functions3. Convex functions
PerspectivePerspective
the perspective of a function f : Rn Ï R is the functiong : Rn × RÏ R,
g(x, t) = t f (x/t), dom g = {(x, t) | x/t ∈ dom f , t > 0}g is convex if f is convex
this can be shown via the epigraph:
(x, t, s) ∈ epi g ⇐Ñ t f (x/t) ≤ s⇐Ñ f (x/t) ≤ s/t
⇐Ñ (x/t, s/t) ∈ epi f
thus, epi g is the inverse image of epi f under theperspective mapping (u, v,w)Ñ (u/v,w/v); inverseimages of convex sets under perspective are convex!
Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)7474
4. Convex optimization problems4. Convex optimization problems
OutlineOutline
3 Convex functions
4 Convex optimization problems
5 Duality
Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)7575
4. Convex optimization problems4. Convex optimization problems
Convex optimization problemsConvex optimization problems
optimization problem in standard form
convex optimization problems
linear optimization
quadratic optimization
semidefinite programming
Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)7676
4. Convex optimization problems4. Convex optimization problems
Optimization problem in standard formOptimization problem in standard form
minimize f0(x)subject to fi(x) ≤ 0, i = 1, ...,m
hi(x) = 0, i = 1, ..., px ∈ Rn is the optimization variablef0 : Rn Ï R is the objective or cost functionfi : Rn Ï R, i = 1, · · · ,m, are the inequality constraintfunctionshi : Rn Ï R are the equality constraint functions
optimal value:
p∗ = inf{f0(x) | fi(x) ≤ 0, i = 1, · · · ,m, hi(x) = 0, i = 1, · · · , p}p∗ =∞ if problem is infeasible (no x satisfies theconstraints)p∗ = −∞ if problem is unbounded below
Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)7777
4. Convex optimization problems4. Convex optimization problems
Optimal and locally optimal pointsOptimal and locally optimal pointsx is feasible if x ∈ domf0 and it satisfies the constraints
a feasible x is optimal if f0(x) = p∗; Xopt is the set ofoptimal points
x is locally optimal if there is an R > 0 such that x isoptimal for
minimize (over z) f0(z)subject to fi(z) ≤ 0, i = 1, ...,m, hi(z) = 0, i = 1, · · · , p
‖z− x‖2 ≤ R
examples (with n = 1,m = p = 0)f0(x) = 1/x, dom f0 = R++ : p∗ = 0, no optimal pointf0(x) = − log(x), dom f0 = R++ : p∗ = −∞f0(x) = x log x, dom f0 = R++ : p∗ = −1/e, x = 1/e isoptimalf0(x) = x3 − 3x, p∗ = −∞, local optimum at x = 1
Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)7878
4. Convex optimization problems4. Convex optimization problems
Implicit constraintsImplicit constraintsthe standard form optimization problem has an implicitconstraint
x ∈ D = m⋂i=0 dom fi ∩
p⋂i=1 dom hi
we call D the domain of the problemthe constraints fi(x) ≤ 0, hi(x) = 0 are the explicitconstraintsa problem is unconstrained if it has no explicitconstraints (m = p = 0)
example
minimize f0(x) = −∑k
i=1 log(bi − aTi x)
is an unconstrained problem with implicit constraintsaT
i x < biProf. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)7979
4. Convex optimization problems4. Convex optimization problems
Feasibility problemFeasibility problem
find xsubject to fi(x) ≤ 0, i = 1, ...,m
hi(x) = 0, i = 1, · · · , pcan be considered a special case of the generalproblem with f0(x) = 0
minimize 0subject to fi(x) ≤ 0, i = 1, ...,m
hi(x) = 0, i = 1, · · · , pp∗ = 0 if constraints are feasible; any feasible x isoptimal
p∗ =∞ if constraints are infeasible
Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)8080
4. Convex optimization problems4. Convex optimization problems
Convex optimization problemConvex optimization problemstandard form convex optimization problem
minimize f0(x)subject to fi(x) ≤ 0, i = 1, . . . ,m
aTi x = bi, i = 1, . . . , p
f0, f1, . . . , fm are convex; equality constraints areaffine
often written as
minimize f0(x)subject to fi(x) ≤ 0, i = 1, ...,m
Ax = b
important property: feasible set of a convexoptimization problem is convex
Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)8181
4. Convex optimization problems4. Convex optimization problems
example
minimize f0(x) = x21 + x22subject to f1(x) = x1/(1 + x22 ) ≤ 0
h1(x) = (x1 + x2)2 = 0f0 is convex; feasible set {(x1, x2) | x1 = −x2 ≤ 0} isconvex
not a convex problem (according to our definition):f1 is not convex, h1 is not affine
equivalent (but not identical) to the convex problem
minimize x21 + x22subject to x1 ≤ 0
x1 + x2 = 0Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)8282
4. Convex optimization problems4. Convex optimization problems
Local and global optimaLocal and global optimaany locally optimal point of a convex problem is(globally) optimal
proof: suppose x is locally optimal and y is optimalwith f0(y) < f0(x)x locally optimal means there is an R > 0 such that
z feasible, ‖z− x‖2 ≤ R ÍÑ f0(z) ≥ f0(x)consider z = θy + (1− θ)x with θ = R/(2‖y − x‖2)‖y − x‖2 > R, so 0 < θ < 1/2z is a convex combination of two feasible points,hence also feasible‖z− x‖2 = R/2 and
f0(z) ≤ θf0(y) + (1− θ)f0(x) < f0(x)which contradicts our assumption that x is locallyoptimal
Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)8383
4. Convex optimization problems4. Convex optimization problems
or use some simplified reasoning:consider z = θy + (1− θ)x with very small θÑ z lies in Euclidean ball around x with radius Rz is feasible, since feasible set is convex
f0(z) ≤ θ f0(y) + (1− θ) f0(x)< θ f0(x) + (1− θ) f0(x) = f0(x)
contradiction to assumption that x is locally optimum Ñf0(y) ≥ f0(x)
Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)8484
4. Convex optimization problems4. Convex optimization problems
Optimality criterion for differentiable f0Optimality criterion for differentiable f0x is optimal if and only if it is feasible and
∇f0(x)T (y − x) ≥ 0 for all feasible y
if nonzero, ∇f0(x) defines a supporting hyperplane tofeasible set X at x
Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)8585
4. Convex optimization problems4. Convex optimization problems
proof: ∇f0(x)T (y − x) ≥ 0 is validÑ f0(y) ≥ f0(x) +∇f0(x)T (y − x) ≥ f0(x)x is optimum point
converse: see Boyd book
Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)8686
4. Convex optimization problems4. Convex optimization problems
unconstrained problem: x is optimal if and only if
x ∈ dom f0, ∇f0(x) = 0equality constrained problem
minimize f0(x) subject to Ax = b
x is optimal if and only if there exists a ν such that
x ∈ dom f0, Ax = b, ∇f0(x) + ATν = 0proof: ∇f0(x)T (y − x) ≥ 0 must hold for optimumx for all y satisfying Ay = ball admissible vectors y can be expressed asy = F z + x0 where x0 is a particular solution to thelinear system of equations and the columns of Fspan the nullspace of A
Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)8787
4. Convex optimization problems4. Convex optimization problems
Ñ∇f0(x)T F z ≥ 0 ∀zÑ∇f0(x) is orthogonal to the nullspace of A;orthogonal complement of nullspace of A isidentical to the column space of AT Ñ∇f0(x) = ATvminimization over nonnegative orthant
minimize f0(x) subject to x � 0
x is optimal if and only if
x ∈ dom f0, x � 0, {∇f0(x)i ≥ 0 xi = 0∇f0(x)i = 0 xi > 0
proof: ∇f0(x)T y ≥ ∇f0(x)T x∀y � 0 for a particular x � 0∇f0(x)T y is unbounded below unless ∇f0(x) � 0Ñ demand: ∇f0(x)T x = 0
Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)8888
4. Convex optimization problems4. Convex optimization problems
Equivalent convex problemsEquivalent convex problemstwo problems are (informally) equivalent if thesolution of one is readily obtained from the solution ofthe other, and vice-versa
some common transformations that preserve convexity:
eliminating equality constraints
minimize f0(x)subject to fi(x) ≤ 0, i = 1, . . . ,m
Ax = bis equivalent to
minimize (over z) f0(Fz + x0)subject to fi(Fz + x0) ≤ 0, i = 1, . . . ,m
where F and x0 are such that
Ax = b ⇐Ñ x = Fz + x0 for some zProf. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)8989
4. Convex optimization problems4. Convex optimization problems
introducing equality constraints
minimize f0(A0x + b0)subject to fi(Aix + bi) ≤ 0, i = 1, . . . ,m
is equivalent to
minimize (over x,yi) f0(y0)subject to fi(yi) ≤ 0, i = 1, . . . ,m
yi = Aix + bi, i = 0, 1, . . . ,mintroducing slack variables for linearinequalities
minimize f0(x)subject to aT
i x ≤ bi, i = 1, . . . ,mis equivalent to
minimize (over x, s) f0(x)subject to aT
i x + si = bi, i = 1, . . . ,msi ≥ 0, i = 1, . . . ,m
Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)9090
4. Convex optimization problems4. Convex optimization problems
epigraph form: standard form convex problem isequivalent to
minimize (over x, t) tsubject to f0(x)− t ≤ 0
fi(x) ≤ 0, i = 1, . . . ,mAx = b
minimizing over some variables
minimize f0(x1,x2)subject to fi(x1) ≤ 0, i = 1, . . . ,m
is equivalent to
minimize f0(x1)subject to fi(x1) ≤ 0, i = 1, . . . ,m
where f0(x1) = infx2 f0(x1,x2)Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)9191
4. Convex optimization problems4. Convex optimization problems
Linear program (LP)Linear program (LP)
minimize cTx + dsubject to Gx � h
Ax = b
convex problem with affine objective and constraintfunctions
feasible set is a polyhedron
Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)9292
4. Convex optimization problems4. Convex optimization problems
Linear-fractional programLinear-fractional programminimize f0(x)subject to Gx � h
Ax = blinear-fractional program
f0(x) = cTx + deTx + f , dom f0(x) = {x | eTx + f > 0}
a quasiconvex optimization problem; can be solvedby bisectionalso equivalent to the LP (variables y, z )
minimize cTy + dzsubject to Gy � hz
Ay = bzeTy + fz = 1z ≥ 0
Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)9393
4. Convex optimization problems4. Convex optimization problems
Quadratic program (QP)Quadratic program (QP)
minimize (1/2)xTPx + qTx + rsubject to Gx � h
Ax = b
P ∈ Sn+, so objective is convex quadraticminimize a convex quadratic function over apolyhedron
Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)9494
4. Convex optimization problems4. Convex optimization problems
ExamplesExamplesleast-squares
minimize ‖Ax − b‖22analytical solution x∗ = A†b (A† is pseudo-inverse)can add linear constraints, e.g ., l � x � u
linear program with random cost
minimize cTx + γxTΣx = E (cTx) + γ Var (cTx)subject to Gx � h, Ax = b
c is random vector with mean c and covariance Σ
hence, cTx is random variable with mean cTx andvariance xTΣxγ > 0 is risk aversion parameter; controls thetrade-off between expected cost and variance (risk)
Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)9595
4. Convex optimization problems4. Convex optimization problems
Quadratically constrained quadratic program (QCQP)Quadratically constrained quadratic program (QCQP)
minimize (1/2)xTP0x + qT0 x + r0subject to (1/2)xTPix + qT
i x + ri ≤ 0, i = 1, . . . ,mAx = b
Pi ∈ Sn+; objective and constraints are convexquadratic
if P1, . . . ,Pm ∈ Sn++, feasible region is intersection ofm ellipsoids and an affine set (standard form ofellipsoid: {x | (x − xc,i)TB−1
i (x − xc,i) ≤ 1} withBi ∈ Sn++)
Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)9696
4. Convex optimization problems4. Convex optimization problems
Second-order cone programmingSecond-order cone programming
minimize fTxsubject to ‖Aix + bi‖2 ≤ cT
i x + di, i = 1, . . . ,mFx = q
(Ai ∈ Rni×n,F ∈ Rp×n)
closely related to quadratic programminginequalities are called second-order cone (SOC)constraints:(Aix + bi, cT
i x + di) ∈ second-order cone in Rni+1(affine function (Aix + bi, cT
i x + di) has to lie in thesecond–order cone in Rni+1)for Ai = 0, SOCP reduces to an LP; if ci = 0, itreduces to a QCQPmore general than QCQP and LP
Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)9797
4. Convex optimization problems4. Convex optimization problems
Semidefinite program (SDP)Semidefinite program (SDP)
minimize cTxsubject to x1F1 + x2F2 + · · ·+ xnFn + G � 0
Ax = b
with F i,G ∈ Sk
inequality constraint is called linear matrixinequality (LMI)
includes problems with multiple LMI constraints: forexample,
x1F1 + · · ·+ xnFn + G � 0, x1F1 + · · ·+ xnFn + G � 0
is equivalent to single LMI
x1[
F1 00 F1
]+x2[
F2 00 F2
]+· · ·+xn
[Fn 00 Fn
]+[ G 00 G
]� 0
Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)9898
4. Convex optimization problems4. Convex optimization problems
LP and SOCP as SDPLP and SOCP as SDP
LP and equivalent SDP
LP: minimize cTx SDP: minimize cTxsubject to Ax � b subject to diag(Ax − b) � 0
(note different interpretation of generalized inequality�)
SOCP and equivalent SDP
SOCP: minimize fTxsubject to ‖Aix + bi‖2 ≤ cT
i x + di, i = 1, . . . ,mSDP: minimize fTx
subject to[ (cT
i x + di)I Aix + bi(Aix + bi)T cTi x + di
]� 0, i = 1, . . . ,m
Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)9999
4. Convex optimization problems4. Convex optimization problems
proof of equivalence: we consider the block matrix
P = [ A bbT c
]Schur complement of A in P: S = c − bT A−1 b
P is positive semidefinite if and only if S is positivesemidefinite, provided A is positive definite
For
P = [ (cTi x + di)I Aix + bi(Aix + bi)T cT
i x + di
],
and a positive value of cTi x + di, the positive
definiteness of P implies
S = (cTi x + di)− 1
cTi x + di
‖Aix + bi‖22 ≥ 0Ñ ‖Aix + bi‖22 ≤ (cT
i x + di)2Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)100100
5. Duality5. Duality
OutlineOutline
4 Convex optimization problems
5 Duality
6 Applications in communications
Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)101101
5. Duality5. Duality
DualityDuality
Lagrange dual problem
weak and strong duality
geometric interpretation
optimality conditions
examples
Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)102102
5. Duality5. Duality
LagrangianLagrangianstandard form problem (not necessarily convex)
minimize f0(x)subject to fi(x) ≤ 0, i = 1, . . . ,m
hi(x) = 0, i = 1, . . . , pvariable x ∈ Rn, domain D, optimal value p∗
Lagrangian: L : Rn × Rm × Rp Ï R, withdom L = D × Rm × Rp,
L(x,λ, ν) = f0(x) + m∑i=1 λifi(x) + p∑
i=1 νihi(x)weighted sum of objective and constraint functionsλi is Lagrange multiplier associated with fi(x) ≤ 0νi is Lagrange multiplier associated with hi(x) = 0
Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)103103
5. Duality5. Duality
Lagrange dual functionLagrange dual functionLagrange dual function: g : Rm × Rp Ï R,
g(λ, ν) = infx∈D
L(x,λ, ν)= inf
x∈D
(f0(x) + m∑
i=1 λifi(x) + p∑i=1 νihi(x))
λ and ν are the dual variables or Lagrange multipliervectors
g is concave since it is defined as pointwise infimum ofaffine functions, and it can be −∞ for some λ, ν
lower bound property: if λ � 0, then g(λ, ν) ≤ p∗
proof: if x is feasible and λ � 0, then
f0(x) ≥ L(x,λ, ν) ≥ infx∈D
L(x,λ, ν) = g(λ, ν)minimizing over all feasible x gives p∗ ≥ g(λ, ν)
Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)104104
5. Duality5. Duality
Least-norm solution of linear equationsLeast-norm solution of linear equations
minimize xTxsubject to Ax = b
dual functionLagrangian is L(x, ν) = xTx + νT (Ax − b)to minimize L over x, set gradient equal to zero:
∇xL(x, ν) = 2x + ATν = 0 ÍÑ x = −(1/2)ATν
plug in L to obtain g:
g(ν) = L((−1/2)ATν, ν) = −14νTAATν − bTν
a concave function of ν
lower bound property: p∗ ≥ −(1/4)νTAATν − bTν forall ν
Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)105105
5. Duality5. Duality
Standard form LPStandard form LP
minimize cTxsubject to Ax = b, x � 0
dual functionLagrangian is
L(x,λ, ν) = cTx + νT (Ax − b)− λTx= −bTν + (c + ATν − λ)Tx
L is affine in x, hence
g(λ, ν) = infx
L(x,λ, ν) = { −bTν ATν − λ + c = 0−∞ otherwise
g is linear on affine domain {(λ, ν) | ATν − λ + c = 0},hence concave
lower bound property: p∗ ≥ −bTν if ATν + c � 0Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)106106
5. Duality5. Duality
The dual problemThe dual problemLagrange dual problem
maximize g(λ, ν)subject to λ � 0
finds best lower bound on p∗, obtained fromLagrange dual functiona convex optimization problem, also if originalproblem is not! optimal value denoted d∗λ, ν are dual feasible if λ � 0, (λ, ν) ∈ dom goften simplified by making implicit constraint(λ, ν) ∈ dom g explicit
example: standard form LP and its dual
minimize cTx maximize −bTνsubject to Ax = b subject to ATν + c � 0
x � 0Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)107107
5. Duality5. Duality
Weak and strong dualityWeak and strong duality
weak duality: d∗ ≤ p∗
always holds (for convex and nonconvex problems)
can be used to find nontrivial lower bounds fordifficult problems
strong duality: d∗ = p∗
does not hold in general
(usually) holds for convex problems
conditions that guarantee strong duality in convexproblems are called constraint qualifications
Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)108108
5. Duality5. Duality
Slater’s constraint qualificationSlater’s constraint qualificationstrong duality holds for a convex problem
minimize f0(x)subject to fi(x) ≤ 0, i = 1, . . . ,m
Ax = bif it is strictly feasible, i.e.,
∃ x ∈ intD : fi(x) < 0, i = 1, . . . ,m, Ax = b
also guarantees that the dual optimum is attained(if p∗ > −∞)can be sharpened: e.g., can replace intD withrelintD (interior relative to affine hull); linearinequalities do not need to hold with strictinequality, . . .there exist many other types of constraintqualifications
Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)109109
5. Duality5. Duality
Inequality form LPInequality form LPprimal problem
minimize cTxsubject to Ax � b
dual function
g(λ) = infx
((c + ATλ)Tx − bTλ) = { −bTλ ATλ + c = 0
−∞ otherwise
dual problem
maximize −bTλsubject to ATλ + c = 0, λ � 0
from Slater’s condition: p∗ = d∗ if Ax ≺ b for some xin fact, p∗ = d∗ except when primal and dual areinfeasible
Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)110110
5. Duality5. Duality
Quadratic programQuadratic programprimal problem (assume P ∈ Sn++)
maximize xTPxsubject to Ax � b
dual function
g(λ) = infx
(xTPx + λT (Ax − b)) = −14λTAP−1ATλ − bTλ
(infimum is attained at x for which 2 P x = −AT λÑ x = −12 P−1 AT λ)dual problem
maximize −(1/4)λTAP−1ATλ − bTλsubject to λ � 0
from Slater’s condition: p∗ = d∗ if Ax ≺ b for some xin fact, p∗ = d∗ always
Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)111111
5. Duality5. Duality
Geometric interpretationGeometric interpretationfor simplicity, consider problem with one constraintf1(x) ≤ 0interpretation of dual function:
g(λ) = inf(u,t)∈G(t + λu), where G = {(f1(x), f0(x) | x ∈ D}t + λu = [ λ 1 ] [ u
t
]
Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)112112
5. Duality5. Duality
λu + t = g(λ) is (non-vertical) supporting hyperplaneto Ghyperplane intersects t-axis at t = g(λ)
Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)113113
5. Duality5. Duality
Complementary slackness IComplementary slackness I
assume strong duality holds, x∗ is primal optimal, (λ∗, ν∗)is dual optimal
f0(x∗) = g(λ∗, ν∗) = infx∈D
(f0(x) + m∑
i=1 λ∗i fi(x) + p∑
i=1 ν∗i hi(x))
≤ f0(x∗) + m∑i=1 λ
∗i fi(x∗) + p∑
i=1 ν∗i hi(x∗)
≤ f0(x∗)(λ∗i ≥ 0, fi(x∗) ≤ 0, λ∗i fi(x∗) ≤ 0, hi(x∗) = 0)
Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)114114
5. Duality5. Duality
Complementary slackness IIComplementary slackness II
hence, the two inequalities hold with equality
x∗ minimizes L(x,λ∗, ν∗)λ∗i fi(x∗) = 0 for i = 1, . . . ,m (known as complementaryslackness):
λ∗i > 0 ÍÑ fi(x∗) = 0, fi(x∗) < 0 ÍÑ λ∗i = 0ith Lagrange multiplier λi can be only nonzero if ithconstraint is active at optimum
Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)115115
5. Duality5. Duality
Karush-Kuhn-Tucker (KKT) conditionsKarush-Kuhn-Tucker (KKT) conditions
the following four conditions are called KKT conditions(for a problem with differentiable fi,hi, D = Rn):
1 primal constraints:fi(x) ≤ 0, i = 1, . . . ,m, hi(x) = 0, i = 1, . . . , p
2 dual constraints: λ � 03 complementary slackness: λifi(x) = 0, i = 1, . . . ,m4 gradient of Lagrangian with respect to x vanishes:
∇f0(x) + m∑i=1 λi∇fi(x) + p∑
i=1 νi∇hi(x) = 0
from previous considerations: if strong duality holdsand x,λ, ν are optimal, then they must satisfy the KKTconditions
Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)116116
5. Duality5. Duality
KKT conditions for convex problem IKKT conditions for convex problem I
if x, λ, ν satisfy KKT for a convex problem, then they areoptimal:
from complementary slackness: f0(x) = L(x, λ, ν)from 4th condition (and convexity): g(λ, ν) = L(x, λ, ν)(L(x, λ, ν) is convex in x and differentiable)
hence, f0(x) = g(λ, ν)(g(λ, ν) = L(x, λ, ν) = f0(x) + m∑
i=1 λifi(x) + p∑i=1 νihi(x) = f0(x))
g(λ, ν) = f0(x) ≤ p∗ Ñ p∗ = f0(x)Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)117117
5. Duality5. Duality
KKT conditions for convex problem IIKKT conditions for convex problem II
if Slater’s condition is satisfied:x is optimal if and only if there exist λ, ν that satisfy KKTconditions
recall that Slater implies strong duality, and dualoptimum is attained
generalizes optimality condition ∇f0(x) = 0 forunconstrained problem
Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)118118
5. Duality5. Duality
example: water-filling (assume αi > 0)for signal transmission over n independent subchannelswith signal power xi and noise power αi, the totalchannel capacity is
C =∑n
i=1 log(1+xi/αi) =∑n
i=1(log(1+xi/αi)+log(αi))+const.
hence, for maximization of the channel capacity undera total power constraint the following optimizationproblem results
minimize −∑n
i=1 log(xi + αi)subject to x � 0, 1Tx = 1
x is optimal iff x � 0, 1Tx = 1, and there existλ ∈ Rn, ν ∈ R such that
λ � 0, λixi = 0, 1xi + αi
+ λi = ν
Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)119119
5. Duality5. Duality
if ν < 1/αi : λi = 0 and xi = 1/ν − αi (xi = 0 wouldcause negative λi)if ν ≥ 1/αi : λi = ν − 1/αi and xi = 0 (xi > 0 wouldcause λi > 0 violating complementary slackness)combining the above two results givesxi = max{0, 1/ν − αi}determine ν from 1Tx =∑n
i=1 max{0, 1/ν − αi} = 1interpretation
n patches; level of patch i is at height αiflood area with unit amount of waterresulting level is 1/ν∗
Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)120120
5. Duality5. Duality
Duality and problem reformulationsDuality and problem reformulations
equivalent formulations of a problem can lead tovery different duals
reformulating the primal problem can be usefulwhen the dual is difficult to derive, or uninteresting
common reformulations
introduce new variables and equality constraints
make explicit constraints implicit or vice-versa
transform objective or constraint functionse.g., replace f0(x) by φ(f0(x)) with φ convex,increasing
Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)121121
5. Duality5. Duality
Semidefinite programSemidefinite programprimal SDP (F i,G ∈ Sk)
minimize cTxsubject to x1F1 + · · ·+ xnFn � G
Lagrange multiplier is matrix Z ∈ Sk
Lagrangian
L(x,Z) = cTx + tr(Z(x1F1 + · · ·+ xnFn −G))= c1 x1 + . . .+ cn xn +tr(Z F1) x1 + . . .+ tr(Z Fn) xn − tr(Z G)
(scalar product between two symmetric matrices Aand B: tr(A B))dual function
g(Z) = infx
L(x,Z) = { −tr(GZ) tr(F iZ) + ci = 0, i = 1, . . . ,n−∞ otherwise
Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)122122
5. Duality5. Duality
dual SDP
maximize −tr(GZ)subject to Z � 0, tr(F iZ) + ci = 0, i = 1, . . . ,n
p∗ = d∗ if primal SDP is strictly feasible (∃ x withx1F1 + · · ·+ xnFn ≺ G)
Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)123123
6. Applications in communications6. Applications in communications
OutlineOutline
5 Duality
6 Applications in communications
7 Algorithms – Equality constrained minimization
Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)124124
6. Applications in communications6. Applications in communications
Examples for applications in communicationsExamples for applications in communications
downlink beamforming
uplink-downlink duality
multiuser detection
Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)125125
6. Applications in communications6. Applications in communications
Downlink beamforming problem [Luo et al. 2006]Downlink beamforming problem [Luo et al. 2006]
transmitter beamforming problem in downlink ofwireless communications
base station is equipped with multiple antennasand each of K mobile terminal with a single antenna
block diagram for K = 2 in equivalent complexbaseband:
z1~N (0,σ2)Base station Mobile terminals
u1
u2
w1
w2
x
h2H
h1H
z2~N (0,σ2)
y1
y2
Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)126126
6. Applications in communications6. Applications in communications
wi: transmit beamforming vector for ith user(NT × 1; NT : number of transmit antennas)
ui: information symbol of ith user of varianceE{|ui|2} = 1x: transmit vector of base station in current timestep (NT × 1)
Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)127127
6. Applications in communications6. Applications in communications
received signal of ith user:
yi = hHi x + zi, i = 1, . . . ,K
hi: channel vector of ith user(µth entry (µ = 1, . . . ,NT ): overall (conjugate)channel coefficient from µth transmit antennaof BS to user i)
hi is assumed to be known at BS and receiver
zi: i.i.d. additive complex Gaussian noise ofvariance σ2yi is a scalar (single antenna receivers)
Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)128128
6. Applications in communications6. Applications in communications
transmit vector of beamformer in the BS:
x = K∑i=1 uiwi
Ñ representation for received signal at `th terminal:
y` = hH`
( K∑i=1 uiwi
)+ z` , ` = 1, . . . ,K= u`hH
` w` + K∑i=1,i 6=` uihH
` wi + z`
signal-to-interference-and-noise ratio (SINR)of `th user:
SINR` = |hH` w` |2∑
k 6=` |hH` wk|2 + σ2
Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)129129
6. Applications in communications6. Applications in communications
design criterion for beamforming vectors:
minimization of total transmit power, whilesatisfying a given set of SINR constraints γ`for the users (assuming the set is feasible)
minimize E{xHx} = E{‖x‖22} =∑Ki=1 ‖wi‖22
subject towH` h`hH
` w`∑k 6=` wH
k h`hH` wk + σ2 ≥ γ` ∀`
however: SINR constraint is not convexÑ not aconvex optimization problem, but one that can berelaxed or transformed into a convex problem
Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)130130
6. Applications in communications6. Applications in communications
Relaxation approachRelaxation approach
reformulation
define Bi = wiwHi (NT ×NT positive semidef. matrix)
H i = hihHi
Bi is of rank 1 (dyadic product)
Ñ minimize∑K
i=1 tr(Bi)subject to tr(H`B` )− γ` ∑
k 6=` tr(H`Bk) ≥ γ`σ2 ∀`
Bi � 0, Bi Hermitian, rank (Bi) = 1 ∀i
Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)131131
6. Applications in communications6. Applications in communications
dropping the rank-1 constraint results in a convexsemidefinite programming (SDP) problem (SDPrelaxation)
it can be shown that the SDP relaxation isguaranteed to have at least one optimalsolution with rank 1Ñ it can be used to optimally solve to original,nonconvex problem
Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)132132
6. Applications in communications6. Applications in communications
Transformation into a convex problemTransformation into a convex problem
observe that an arbitrary phase rotation can beadded to the beamforming vectors withoutaffecting the transmit power or the constraintsÑ hH
` w` can be chosen to be real, ≥ 0 without anyloss in generality ∀`
constraints: (1 + 1γ` )|hH
` w` |2 ≥∑Kk=1 |hH
` wk|2 + σ2 ∀`
Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)133133
6. Applications in communications6. Applications in communications
define following vector and matrix
w = [wT1 wT2 . . .wT
K
]T
H` =
hH` 0 . . . 00 hH
` . . . 0...
......
0 0 . . . hH`
Ñ H`w =
hH` w1
hH` w2...
hH` wK
constraint can be written written as(1 + 1
γ`
)|hH
` w` |2 ≥∣∣∣∣∣∣∣∣[ H`w
σ
]∣∣∣∣∣∣∣∣22Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)134134
6. Applications in communications6. Applications in communications
hH` w` can be assumed to be real, non-negativeÑ taking the square root yields:√1 + 1
γ` hH` w` ≥
∣∣∣∣∣∣∣∣[ H`wσ
]∣∣∣∣∣∣∣∣2 ∀`
or ||Aw + b||2 ≤ cHw + d
with A = [ H`0T
] ; b =
0...0σ
cH = [ 0T 0T . . . hH
` 0T . . . 0T ] ·√1 + 1γ` ; d = 0
second-order cone constraint!original optimization problem is equivalent to thefollowing second-order cone program
Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)135135
6. Applications in communications6. Applications in communications
SOCPSOCP
minimize τ
subject to√1 + 1
γ` hH` w` ≥
∣∣∣∣∣∣∣∣[ H`wσ
]∣∣∣∣∣∣∣∣2 ∀`
||w||2 ≤ τ
(this minimizes√∑K
i=1 ||wi||22 under the constraints;√(·)
is a monotonically increasing function and does notchange the optimum solution for the beamformingfilters)
Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)136136
6. Applications in communications6. Applications in communications
Uplink-downlink duality via Lagrangian duality [Luo et al. 2006]Uplink-downlink duality via Lagrangian duality [Luo et al. 2006]
exploring the dual of convex optimization problemsin engineering often reveals the structure of theoptimum solution
Lagrangian dual of the above SOCP problem has anengineering interpretation, which is known asuplink-downlink duality
several different versions of uplink-downlink dualityhave been developed in the literature, referring todifferent figures of merit, e.g. channel capacity
Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)137137
6. Applications in communications6. Applications in communications
duality in beamforming context: minimumpower needed to achieve a set of SINR targets in adownlink multiple-input multiple-output (MIMO)channel is the same as the minimum power neededto achieve the same set of SINR targets in theuplink channel, where the uplink channel is derivedby reversing the input and output of the downlink
these results can be proven via Lagrangian dualityin convex optimization in a unified manner
Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)138138
6. Applications in communications6. Applications in communications
Lagrangian of the downlink beamforming problem:
L(w,λ)= K∑i=1 wH
i wi + K∑i=1 λi
− 1γi|hH
i wi|2 +∑k 6=i|hH
i wk|2 + σ2
= K∑i=1 λi σ2 + K∑
i=1 wHi
I − λiγi
hihHi +∑
µ 6=iλµhµhH
µ
wi
λ=[λ1, λ2, . . . , λK]TLagrangian dual function:
g(λ) = infw∈CNT ·K
L(w,λ)
Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)139139
6. Applications in communications6. Applications in communications
g(λ) =
K∑i=1 λi σ2 if
(I − λi
γihihH
i + ∑µ 6=i
λµhµhHµ
)� 0 ∀i
−∞ else
dual problem (SDP):
maximizeK∑
i=1 λi σ2subject to I + K∑
µ=1 λµhµhHµ �
(1 + 1γi
)λihihH
i ∀i
Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)140140
6. Applications in communications6. Applications in communications
dual problem can be shown to correspond to anuplink problem with λi as the the (scaled) uplinkpower, hi as the uplink channel, and γi as the SINRconstraintblock diagram of uplink transmission (K = 2)
Base stationMobile terminals
x1
x2
h1
h2 w2H
w1H ỹ1
ỹ2
z~N (0,σ2I)
wi: receiver filter vector of base station for ith user(N × 1)xi: information symbol of ith user of varianceE{|xi|2} = ρi
Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)141141
6. Applications in communications6. Applications in communications
hi: channel vector of ith user (µth entry: overallchannel coefficient from user i to µth receiveantenna of BS); N × 1z: i.i.d. additive complex Gaussian noise vectorwith covariance matrix σ2Ireceived signal for user ` after filtering at BS:
y` = wH`
( K∑i=1 xihi + z
)` = 1, . . . ,K
= x` wH` h` + K∑
i=1,i 6=` xi wH` hi + wH
` z
Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)142142
6. Applications in communications6. Applications in communications
SINR of `th user:
SINR` = ρ` |wH` h` |2∑
k 6=` ρk|wH` hk|2 + σ2wH
` w`
design criterion for receiver vectors:
minimize total sum transmit power of mobileterminals, while satisfying a given set of SINRconstraints γ` for the users
minimizeK∑
i=1 ρi
subject toρ`wH
` h`hH` w`∑
k 6=` ρkwH` hkhH
k w` + σ2wH` w`
≥ γ` ∀`
Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)143143
6. Applications in communications6. Applications in communications
optimal wi is the minimum mean-squared error(MMSE) filter
wi = ( K∑k=1 ρkhkhH
k + σ2I)−1
hi
substituting the MMSE filters into the constraintsand performing some matrix manipulations, it canbe shown that the uplink problem is equivalent to
minimizeK∑
i=1 ρi
subject to(1 + 1
γi
)ρihihH
i � σ2I + K∑µ=1 ρµhµhH
µ ∀i
Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)144144
6. Applications in communications6. Applications in communications
with λi = ρiσ2 , the dual downlink problem and
the uplink problem are identical, except thatmaximization and minimization are reversedand right hand side and left hand side in theconstraint inequalities have been interchanged
finally, both problems turn out to have the samesolution
strong duality holds for original and dual downlinkproblem (convex optimization problem)
Ñ original and dual problem must have the samesolution
Ñ solution of dual uplink problem solves alsooriginal downlink problem
Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)145145
6. Applications in communications6. Applications in communications
dual variables of the downlink problem can beinterpreted as uplink powers scaled by the noisevariance
there is a quite general uplink-downlink dualitywhich is useful because the uplink problem is easierto solve (e.g., iteratively)
Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)146146
6. Applications in communications6. Applications in communications
Multiuser detection [Luo et al. 2006]Multiuser detection [Luo et al. 2006]
consider a multiple-input multiple-output (MIMO)transmission with received vector
y =√ ρnHs + z
ρ: total signal-to-noise ratio, SNR
H: n ×m channel matrixentry (ν, µ): channel coefficient from µthtransmit antenna to νth receive antenna
m: number of transmit antennasn: number of receive antennas
s: transmitted symbol vector with BPSK entriess ∈ {−1,+1}mz: additive complex Gaussian noise vectorwith i.i.d. entries
Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)147147
6. Applications in communications6. Applications in communications
channel capacity of MIMO channel is known to beproportional to the number of transmit antennas(for sufficient number of receive antennas)
efficient detection algorithms are needed in order torealize the potential gains of a MIMO transmission
maximum-likelihood detection:
minimize∣∣∣∣∣∣y −√ ρ
n Hs∣∣∣∣∣∣22
subject to s ∈ {±1}mnonconvex problem!
known methods to find ML solution:1 full search2 sphere decoder
Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)148148
6. Applications in communications6. Applications in communications
in the following a semidefinite programmingrelaxation method with polynomial complexityis described
for simplicity, we consider the case m = nH, s and z are assumed to be real
rewrite decision metric as∣∣∣∣∣∣∣∣y −√ ρnHs
∣∣∣∣∣∣∣∣22 = tr (QxxT )with real-valued matrix Q and vector x,
Q = ( ρn)
HTH −√
ρn HTy
−√
ρn yTH ||y||22
, x = [ s1 ]
Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)149149
6. Applications in communications6. Applications in communications
define X = xxT ; X � 0, Xii = 1, rank(X) = 1 ifand only if X = xxT for some x with xi = ±1write original problem in terms of X and relaxthe rank-1 constraint
Ñ SDP relaxation:
minimize tr(Q · X)subject to X � 0, Xii = 1 ∀i
solve SDP which, however, in general will yield amatrix X with rank greater than one
Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)150150
6. Applications in communications6. Applications in communications
randomized procedure to generate a feasiblerank-1 solution xSDR
1 compute the largest eigenvalue of optimum solutionof SDP and associated unit-norm eigenvector
v = [v1 v2 . . . vn+1]T2 generate L i.i.d. binary vectors x` , ` = 1, . . . ,L, whose
ith entry (i = 1, . . . ,n + 1) follows the distribution
Pr{xi = +1} = (1 + vi)/2Pr{xi = −1} = (1− vi)/2
3 pick xSDR = argminx` xT` Qx`
set sSDR to be the first n entries of xSDR multipliedby the last entry (to correct the sign)
excellent performance - complexity tradeoff of SDPdetector in practical SNR ranges
Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)151151
6. Applications in communications6. Applications in communications
bit error rate vs. signal-to-noise ratio (SNR)
(from [Luo et al. 2006])
Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)152152
6. Applications in communications6. Applications in communications
ReferenceReference
[Luo et al. 2006] Z.Q. Luo and W. Yu, ”An introductionto convex optimization for communi-cations and signal processing”, IEEEJournal on Selected Areas in Com-munications, vol. 24, no. 8, pp.1426-1438, Aug. 2006
Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)153153
7. Algorithms – Equality constrained minimization7. Algorithms – Equality constrained minimization
OutlineOutline
6 Applications in communications
7 Algorithms – Equality constrained minimization
8 Conclusions
Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)154154
7. Algorithms – Equality constrained minimization7. Algorithms – Equality constrained minimization
Equality constrained minimizationEquality constrained minimization
equality constrained minimization
eliminating equality constraints
Newton’s method with equality constraints
Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)155155
7. Algorithms – Equality constrained minimization7. Algorithms – Equality constrained minimization
Equality constrained minimizationEquality constrained minimization
minimize f (x)subject to Ax = b
f convex, twice continuously differentiable
A ∈ Rp×n with rank A = pwe assume p∗ is finite and attained
optimality conditions x∗ is optimal iff there exists a ν∗such that
∇f (x∗) + ATν∗ = 0, Ax∗ = b
Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)156156
7. Algorithms – Equality constrained minimization7. Algorithms – Equality constrained minimization
equality constrained quadratic minimization (withP ∈ Sn+)
minimize (1/2)xTPx + qTx + rsubject to Ax = b
optimality condition:[P AT
A 0
] [x∗ν∗] = [ −q
b
]coefficient matrix is called KKT matrix
KKT matrix is nonsingular if and only if
Ax = 0, x 6= 0 ÍÑ xTPx > 0equivalent condition for nonsingularity: P + ATA � 0
Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)157157
7. Algorithms – Equality constrained minimization7. Algorithms – Equality constrained minimization
Eliminating equality constraintsEliminating equality constraintsrepresent solution of {x | Ax = b} as
{x | Ax = b} = {Fz + x | z ∈ Rn−p}
x is (any) particular solutionrange of F ∈ Rn×(n−p) is nullspace of A(rank F = n − p and AF = 0)reduced or eliminated problem
minimize f (Fz + x)an unconstrained problem with variable z ∈ Rn−p
from solution z∗, obtain x∗ and ν∗ as
x∗ = Fz∗ + x, ν∗ = −(AAT )−1A∇f (x∗)Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)158158
7. Algorithms – Equality constrained minimization7. Algorithms – Equality constrained minimization
example: optimal allocation with resource constraint
minimize f1(x1) + f2(x2) + . . .+ fn(xn)subject to x1 + x2 + . . .+ xn = b
eliminate xn = b − x1 − . . .− xn−1, i.e., choose
x = ben, F = [ I−1T
]∈ Rn×(n−1)
reduced problem:
minimize f1(x1) + . . .+ fn−1(xn−1) + fn(b − x1 − . . .− xn−1)(variables x1, . . . , xn−1)
Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)159159
7. Algorithms – Equality constrained minimization7. Algorithms – Equality constrained minimization
Newton stepNewton stepNetwton step ∆xnt of f at a feasible x is given bysolution v of[
∇2f (x) AT
A 0
] [vw
] = [ −∇f (x)0
]interpretations∆xnt solves second order approximation (with
variable v)
minimize f (x + v) = f (x) +∇f (x)Tv + (1/2)vT∇2f (x)vsubject to A(x + v) = b
∆xnt equations follow from linearizing optimalityconditions
∇f (x+v)+ATw ≈ ∇f (x)+∇2f (x)v+ATw = 0, A(x+v) = b
Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)160160
7. Algorithms – Equality constrained minimization7. Algorithms – Equality constrained minimization
Newton decrementNewton decrement
λ(x) = (∆xTnt∇2f (x)∆xnt )1/2 = (−∇f (x)T∆xnt )1/2
properties
gives an estimate of f (x)− p∗ using quadraticapproximation f:
f (x)− infAy=b
f (y) = 12λ(x)2directional derivative in Newton direction:
ddt f (x + t∆xnt )∣∣∣∣
t=0 = −λ(x)2in general, λ(x) 6= (∇f (x)T∇2f (x)−1∇f (x))1/2
Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)161161
7. Algorithms – Equality constrained minimization7. Algorithms – Equality constrained minimization
Newton’s method with equality constraintsNewton’s method with equality constraints
given starting point x ∈ dom f with Ax = b, toleranceε > 0.repeat
1 Compute the Newton step and decrement ∆xnt , λ(x).2 Stopping criterion. quit if λ2/2 ≤ ε.3 Line search. Choose step size t by backtracking line
search.4 Update. x := x + t∆xnt
a feasible descent method: x(k) feasible andf (x(k+1)) < f (x(k))affine invariant
Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)162162
7. Algorithms – Equality constrained minimization7. Algorithms – Equality constrained minimization
Newton’s method and eliminationNewton’s method and elimination
Newton’s method for reduced problem
minimize f (z) = f (Fz + x)variables z ∈ Rn−p
x satisfies Ax = b; rank F = n − p and AF = 0Newton’s method for f , started at z(0), generatesiterates z(k)
Newton’s method with equality constraints
when started at x(0) = Fz(0) + x, iterates are
x(k+1) = Fz(k) + x
hence, don’t need separate convergence analysis
Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)163163
8. Conclusions8. Conclusions
OutlineOutline
7 Algorithms – Equality constrained minimization
8 Conclusions
Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)164164
8. Conclusions8. Conclusions
ConclusionsConclusionsmathematical optimization
problems in engineering design, data analysis andstatistics, economics, management, etc., can oftenbe expressed as mathematical optimizationproblems
techniques exist to take into account multipleobjectives or uncertainty in the data
tractability
roughly speaking, tractability in optimizationrequires convexity
algorithms for nonconvex optimization find local(suboptimal) solutions, or are very expensive
surprisingly many applications can be formulatedas convex problems
Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)165165
8. Conclusions8. Conclusions
theoretical consequences of convexity
locally optimum solutions are globally optimum
set of globally optimum solutions is always convex
extensive duality theorysystematic way of deriving lower bounds on optimalvaluenecessary and sufficient optimality conditionssolution methods with polynomial worst-casecomplexity
Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)166166
8. Conclusions8. Conclusions
practical consequences of convexity: (most)convex problems can be solved globally andefficiently
interior-point methods require 20 – 80 steps inpractice
basic algorithms (e.g., Newton, barrier method,etc.) are easy to implement and work well for smalland medium size problems (larger problems ifstructure is exploited)
more and more high-quality implementations ofadvanced algorithms and modeling tools arebecoming available
high level modeling tools like cvx ease modelingand problem specification
Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)167167
8. Conclusions8. Conclusions
how to use convex optimization in some appliedcontext
use rapid prototyping, approximate modeling – startwith simple models, small problem instances,inefficient solution methods
if you don’t like the results, no need to expendfurther effort on more accurate models or efficientalgorithms
work out, simplify, and interpret optimalityconditions and dualeven if the problem is quite nonconvex, you can useconvex optimization
in subproblems, e.g., to find search directionby repeatedly forming and solving a convexapproximation at the current pointapproximate or even optimum solution viaproblem relaxation
Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)168168
8. Conclusions8. Conclusions
applications in communications and signalprocessing
power allocation
optimization of beamformers/filters and signalpowers in multiuser wireless networks (uplinkand downlink)
detection (SISO, MIMO)
calculation of channel capacity
design of source codes
design of transmission policies for sensornetworks with energy harvesting
signal recovery
blind source separation
radar signal design
optimization of power flow in smart grids
etc.Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)169169
8. Conclusions8. Conclusions
connections to current research topics forMaster’s / Ph.D. theses
knowledge on convex optimization is highlybeneficial for research projects on
cellular systems with buffer–aided relaying,wireless networks with information and powertransfer (energy harvesting),resource allocation,etc.
Prof. Dr.-Ing. Wolfgang Gerstacker – Convex Optimization (based on materials by S. Boyd)170170