Active-set method Frank-Wolfe method Penalty method Barrier methods
Solution methods forconstrained optimization problems
Mauro Passacantando
Department of Computer Science, University of [email protected]
Optimization MethodsMaster of Science in Embedded Computing Systems – University of Pisa
http://pages.di.unipi.it/passacantando/om/OM.html
M. Passacantando Optimization Methods 1 / 28 –
Active-set method Frank-Wolfe method Penalty method Barrier methods
Problems with linear equality constraints
Consider min f (x)Ax = b
with
I f strongly convex and twice continuously differentiable
I A matrix p × n with rank(A) = p
It is equivalent to an unconstrained problem:
write A = (AB ,AN) with det(AB) 6= 0, then Ax = b is equivalent to
ABxB + ANxN = b =⇒ xB = A−1B (b − ANxN),
thus min f (x)Ax = b
is equivalent to
min f (A−1
B (b − ANxN), xN)xN ∈ Rn−p
M. Passacantando Optimization Methods 2 / 28 –
Active-set method Frank-Wolfe method Penalty method Barrier methods
Problems with linear equality constraints
Example. Consider min x21 + x2
2 + x23
x1 + x3 = 1x1 + x2 − x3 = 2
Since x1 = 1− x3 and x2 = 2− x1 + x3 = 1 + 2x3, the original constrainedproblem is equivalent to the following unconstrained problem:
min (1− x3)2 + (1 + 2x3)2 + x23 = 6x2
3 + 2x3 + 2x3 ∈ R
Therefore, the optimal solution is x3 = −1/6, x1 = 7/6, x2 = 2/3.
M. Passacantando Optimization Methods 3 / 28 –
Active-set method Frank-Wolfe method Penalty method Barrier methods
Active-set method
Consider a quadratic programming problemmin 1
2xTQx + cTx
Ax ≤ b
where
I Q is positive definite
I for any feasible point x the vectors Ai : Aix = bi are linearly independent
The active-set method solves at each iteration a quadratic programming problemwith equality constraints only.
M. Passacantando Optimization Methods 4 / 28 –
Active-set method Frank-Wolfe method Penalty method Barrier methods
Active-set method
0. Choose a feasible point x0, set W0 = i : Aix0 = bi (working set) and
k = 0.
1. Find the optimal solution yk of the problemmin 1
2xTQx + cTx
Aix = bi ∀ i ∈Wk
2. If yk 6= xk then go to step 3else go to step 4
3. If yk is feasible then tk = 1
else tk = min
bi − Aix
k
Ai (yk − xk): i /∈Wk , Ai (y
k − xk) > 0
,
endxk+1 = xk + tk(yk − xk), Wk+1 = Wk ∪ i /∈Wk : Aix
k+1 = bi,k = k + 1 and go to step 1
4. Compute the KKT multipliers µk related to yk
If µk ≥ 0 then STOPelse xk+1 = xk , µk
j = mini∈Wk
µki , Wk+1 = Wk \ j, k = k + 1 and go to step 1
M. Passacantando Optimization Methods 5 / 28 –
Active-set method Frank-Wolfe method Penalty method Barrier methods
Active-set method
Example. Solve the problemmin 1
2 (x1 − 3)2 + (x2 − 2)2
−2x1 + x2 ≤ 0x1 + x2 ≤ 4−x2 ≤ 0
by means of the active-set method starting from x0 = (0, 0).
The working set W0 = 1, 3 hence y0 = x0; KKT multipliers are µ01 = −3/2,
µ03 = −11/2. The new point x1 = x0 with W1 = 1; y1 is the optimal solution of
min 12 (x1 − 3)2 + (x2 − 2)2
−2x1 + x2 = 0
which is equivalent to min 9
2x21 − 11x1
x1 ∈ Rthus y1 = (11/9, 22/9) which is feasible, therefore x2 = y1 and W2 = 1. Wealready know that y2 = x2; the KKT multiplier is µ2
1 = −8/9, hence x3 = x2 andW3 = ∅.
M. Passacantando Optimization Methods 6 / 28 –
Active-set method Frank-Wolfe method Penalty method Barrier methods
Active-set method
The optimal solution y3 = (3, 2) is not feasible, the step size is
t3 = min
b2 − A2x
3
A2(y3 − x3),
b3 − A3x3
A3(y3 − x3)
= min
1
4,
11
2
=
1
4,
x4 = x3 + t3(y3 − x3) = (5/3, 7/3) and W4 = 2. The optimal solutiony4 = (7/3, 5/3) is feasible, hence x5 = y4 and W5 = 2. Finally, y5 = x5 andµ5
2 = 2/3 > 0, thus x5 is the global minimum of the original problem.
M. Passacantando Optimization Methods 7 / 28 –
Active-set method Frank-Wolfe method Penalty method Barrier methods
Active-set method
Exercise 1. Solve the problemmin 2x2
1 + x22 − x1x2 − 2x1 + x2
−x1 ≤ 0−x2 ≤ 0x1 + x2 ≤ 12
by means of the active-set method starting from the point (0, 10).
M. Passacantando Optimization Methods 8 / 28 –
Active-set method Frank-Wolfe method Penalty method Barrier methods
Frank-Wolfe method
Consider the problem min f (x)Ax ≤ b
(P)
where f is convex and continuously differentiable and the feasible regionΩ = x : Ax ≤ b is bounded.
Linearize the objective function at xk , i.e., f (x) ' f (xk) +∇f (xk)T(x − xk) andsolve the linear programming problem:
min f (xk) +∇f (xk)T(x − xk)Ax ≤ b
The optimal solution yk of the linearized problem gives a lower bound for (P):
f (xk) ≥ v(P) = minx∈Ω
f (x) ≥ minx∈Ω
[f (xk) +∇f (xk)T(x − xk)
]= f (xk)+∇f (xk)T(y k − xk)︸ ︷︷ ︸
optimality gap
If ∇f (xk)T(yk − xk) = 0 then xk solves (P)else ∇f (xk)T(yk − xk) < 0, i.e., yk − xk is a descent direction for f at xk .
M. Passacantando Optimization Methods 9 / 28 –
Active-set method Frank-Wolfe method Penalty method Barrier methods
Frank-Wolfe method
Frank-Wolfe method
0. Choose a feasible point x0 and set k = 0
1. Compute an optimal solution yk of the LP problemmin ∇f (xk)TxA x ≤ b
2. if ∇f (xk)T(yk − xk) = 0 then STOPelse compute step size tk
3. Set xk+1 = xk + tk(yk − xk), k = k + 1 and go to step 1.
The step size tk can be predetermined, e.g., tk =1
k + 1or computed by an
(exact/inexact) line search along the direction yk − xk , with t ∈ [0, 1].
TheoremFor any starting point x0 the generated sequence xk is bounded and any of itscluster points is a global optimum.
M. Passacantando Optimization Methods 10 / 28 –
Active-set method Frank-Wolfe method Penalty method Barrier methods
Frank-Wolfe method
Example. Solve the problem min (x1 − 3)2 + (x2 − 1)2
0 ≤ x1 ≤ 20 ≤ x2 ≤ 2
by means of the Frank-Wolfe method starting from the point x0 = (0, 0) andusing an exact line search to compute the step size.
∇f (x) = (2x1 − 6, 2x2 − 2) and ∇f (x0) = (−6, −2), the optimal solution of thelinearized problem
min −6x1 − 2x2
x ∈ Ω
is y0 = (2, 2). Since ∇f (x0)T(y0 − x0) = −16, a line search is needed: the stepsize t0 is the optimal solution of
min (x1 − 3)2 + (x2 − 1)2
x1 = 2tx2 = 2tt ∈ [0, 1]
M. Passacantando Optimization Methods 11 / 28 –
Active-set method Frank-Wolfe method Penalty method Barrier methods
Frank-Wolfe method
which is equivalent to min 8t2 − 16tt ∈ [0, 1]
thus t0 = 1 and x1 = (2, 2). Since ∇f (x1) = (−2, 2), the optimal solution of thelinearized problem
min −2x1 + 2x2
x ∈ Ω
is y1 = (2, 0). Now, ∇f (x1)T(y1 − x1) = −4, the step size is the optimal solutionof
min (x1 − 3)2 + (x2 − 1)2
x1 = 2x2 = 2− 2tt ∈ [0, 1]
hence t1 = 1/2, x2 = (2, 1) and ∇f (x2) = (−2, 0), thus y2 = (2, 0) (analternative optimal solution is y2 = (2, 2)). Since ∇f (x2)T(y2 − x2) = 0, thepoint x2 is the global minimum of the original problem.
M. Passacantando Optimization Methods 12 / 28 –
Active-set method Frank-Wolfe method Penalty method Barrier methods
Frank-Wolfe method
Exercise 2. Implement in MATLAB the Frank-Wolfe method with exact linesearch for solving the problem
min 12x
TQx + cTxAx ≤ b
where Q is a positive definite matrix.
Exercise 3. Run the Frank-Wolfe method with exact line search for solving theproblem
min 12 (x1 − 3)2 + (x2 − 2)2
−2x1 + x2 ≤ 0x1 + x2 ≤ 4−x2 ≤ 0
starting from the point (0, 0). [Use ∇f (xk)T(yk − xk) > −10−2 as stoppingcriterion.]
Exercise 4. Solve the previous problem by means of the Frank-Wolfe methodwith predetermined step size tk = 1/(k + 1) starting from (0, 0).
M. Passacantando Optimization Methods 13 / 28 –
Active-set method Frank-Wolfe method Penalty method Barrier methods
Penalty method
Consider a constrained optimization problemmin f (x)gi (x) ≤ 0 ∀ i = 1, . . . ,m
(P)
Define the penalty function
p(x) =m∑i=1
(max0, gi (x))2
and consider the unconstrained penalized problemmin f (x) +
1
εp(x) := pε(x)
x ∈ Rn(Pε)
Note that
pε(x)
= f (x) if x ∈ Ω> f (x) if x /∈ Ω
M. Passacantando Optimization Methods 14 / 28 –
Active-set method Frank-Wolfe method Penalty method Barrier methods
Penalty method
Proposition
I If f , gi are continuously differentiable, then pε is continuously differentiableand ∇pε(x) = ∇f (x) + 2
ε
∑mi=1 max0, gi (x)∇gi (x)
I If f and gi are convex, then pε is convex
I Any (Pε) is a relaxation of (P), i.e., v(Pε) ≤ v(P) for any ε > 0
I If x∗ε solves (Pε) and x∗ε ∈ Ω, then x∗ε is optimal also for (P)If x∗ε solves (Pε) and x∗ε /∈ Ω, then v(Pε) > v(Pε′) for any ε′ > ε
Penalty method
0. Set ε0 > 0, τ ∈ (0, 1), k = 0
1. Find an optimal solution xk of the penalized problem (Pεk )
2. If xk ∈ Ω then STOPelse εk+1 = τεk , k = k + 1 and go to step 1.
TheoremIf f is coercive, then the sequence xk is bounded and any of its cluster points isan optimal solution of (P).
M. Passacantando Optimization Methods 15 / 28 –
Active-set method Frank-Wolfe method Penalty method Barrier methods
Penalty method
Exercise 5. Implement in MATLAB the penalty method for solving the problemmin 1
2xTQx + cTx
Ax ≤ b
where Q is a positive definite matrix.
Exercise 6. Run the penalty method withτ = 0.5 and ε0 = 5 for solving the prob-lem
min 12 (x1 − 3)2 + (x2 − 2)2
−2x1 + x2 ≤ 0x1 + x2 ≤ 4−x2 ≤ 0
[Use max(Ax − b) < 10−3 as stoppingcriterion.]
0 0.5 1 1.5 2 2.5 3 3.5 4
0
0.5
1
1.5
2
2.5
3
M. Passacantando Optimization Methods 16 / 28 –
Active-set method Frank-Wolfe method Penalty method Barrier methods
Barrier methods
Consider min f (x)g(x) ≤ 0
where
I f , gi convex and twice continuously differentiable
I there is no isolated point in Ω
I there exists an optimal solution (e.g. f coercive or Ω bounded)
I Slater constraint qualification holds: there exists x such that
x ∈ dom(f), gi (x) < 0, ∀ i = 1, . . . ,m
Hence strong duality holds.
Special cases: linear programming, convex quadratic programming
M. Passacantando Optimization Methods 17 / 28 –
Active-set method Frank-Wolfe method Penalty method Barrier methods
Unconstrained reformulation
The constrained problem min f (x)g(x) ≤ 0
is equivalent to the unconstrained problem min f (x) +m∑i=1
I−(gi (x))
x ∈ Rn
where
I−(u) =
0 if u ≤ 0
+∞ if u > 0
is called the indicator function of R−, that is neither finite nor differentiable.
M. Passacantando Optimization Methods 18 / 28 –
Active-set method Frank-Wolfe method Penalty method Barrier methods
Barrier function
However, I− can be approximated by the smooth convex function
u 7→ − εu, where u < 0,
and parameter ε > 0. Approximation improves as ε→ 0.Hence, we can approximate the problem min f (x) +
m∑i=1
I−(gi (x))
x ∈ Rn
with min f (x) + ε
m∑i=1
1
−gi (x)
x ∈ int(Ω)
B(x) =m∑i=1
1
−gi (x)is called barrier function.
dom(B) = int(Ω), B is convex and smooth.M. Passacantando Optimization Methods 19 / 28 –
Active-set method Frank-Wolfe method Penalty method Barrier methods
Barrier method
If x∗(ε) is the optimal solution of min f (x) + ε
m∑i=1
1
−gi (x)
x ∈ int(Ω)
then
∇f (x∗(ε)) + ε
m∑i=1
1
[gi (x∗(ε))]2∇gi (x∗(ε)) = 0.
Define λ∗i (ε) =ε
[gi (x∗(ε))]2> 0, for any i = 1, . . . ,m.
Then the Lagrangian function L(x , λ∗(ε)) = f (x) +m∑i=1
λ∗i (ε)gi (x) is convex and
∇xL(x∗(ε), λ∗(ε)) = 0, hence
f (x∗(ε)) ≥ v(P) ≥ ϕ(λ∗(ε)) = minx L(x , λ∗(ε))= L(x∗(ε), λ∗(ε))= f (x∗(ε))− εB(x∗(ε))︸ ︷︷ ︸
optimality gap
M. Passacantando Optimization Methods 20 / 28 –
Active-set method Frank-Wolfe method Penalty method Barrier methods
Barrier method
Barrier method
0. Set δ > 0, τ < 1 and ε1 > 0. Choose x0 ∈ int(Ω), set k = 1
1. Find the optimal solution xk of the unconstrained problem min f (x) + εk
m∑i=1
1
−gi (x)
x ∈ int(Ω)
using xk−1 as starting point
2. If εkB(xk) < δ then STOPelse εk+1 = τεk , k = k + 1 and go to step 1
TheoremIf Ω is bounded, then the sequence xk is bounded and any of its cluster pointsis an optimal solution of (P).
M. Passacantando Optimization Methods 21 / 28 –
Active-set method Frank-Wolfe method Penalty method Barrier methods
Logarithmic barrier
The indicator function I− can be also approximated by the smooth convexfunction:
u 7→ −ε log(−u), with ε > 0,
and the approximation improves as ε→ 0.
Hence, we can approximate the problem min f (x) +m∑i=1
I−(gi (x))
x ∈ Rn
with min f (x)− εm∑i=1
log(−gi (x))
x ∈ int(Ω)
M. Passacantando Optimization Methods 22 / 28 –
Active-set method Frank-Wolfe method Penalty method Barrier methods
Logarithmic barrier
Logarithmic barrier function
B(x) = −m∑i=1
log(−gi (x))
I dom(B) = int(Ω)
I B is convex
I B is smooth with
∇B(x) = −m∑i=1
1
gi (x)∇gi (x)
∇2B(x) =m∑i=1
1
gi (x)2∇gi (x)∇gi (x)T +
m∑i=1
1
−gi (x)∇2gi (x)
M. Passacantando Optimization Methods 23 / 28 –
Active-set method Frank-Wolfe method Penalty method Barrier methods
Logarithmic barrier
If x∗(ε) is the optimal solution of min f (x)− εm∑i=1
log(−gi (x))
x ∈ int(Ω)
then
∇f (x∗(ε)) +m∑i=1
ε
−gi (x∗(ε))∇gi (x∗(ε)) = 0.
Define λ∗i (ε) =ε
−gi (x∗(ε))> 0, for any i = 1, . . . ,m. Then the Lagrangian
function
L(x , λ∗(ε)) = f (x) +m∑i=1
λ∗i (ε)gi (x)
is convex and ∇xL(x∗(ε), λ∗(ε)) = 0, hence
f (x∗(ε)) ≥ v(P) ≥ ϕ(λ∗(ε)) = minx L(x , λ∗(ε))= L(x∗(ε), λ∗(ε))= f (x∗(ε))− m ε︸︷︷︸
optimality gapM. Passacantando Optimization Methods 24 / 28 –
Active-set method Frank-Wolfe method Penalty method Barrier methods
Interpretation via KKT conditions
The KKT system of the original problem is∇f (x) +
m∑i=1
λi∇gi (x) = 0
−λigi (x) = 0λ ≥ 0g(x) ≤ 0
Notice that (x∗(ε), λ∗(ε)) solves the system∇f (x) +
m∑i=1
λi∇gi (x) = 0
−λigi (x) = ελ ≥ 0g(x) ≤ 0
which is an approximation of the above KKT system.
M. Passacantando Optimization Methods 25 / 28 –
Active-set method Frank-Wolfe method Penalty method Barrier methods
Logarithmic barrier method
Logarithmic barrier method
0. Set δ > 0, τ < 1 and ε1 > 0. Choose x0 ∈ int(Ω), set k = 1
1. Find the optimal solution xk of min f (x)− εkm∑i=1
log(−gi (x))
x ∈ int(Ω)
using xk−1 as starting point
2. If m εk < δ then STOPelse εk+1 = τεk , k = k + 1 and go to step 1
Choice of τ involves a trade-off: small τ means fewer outer iterations, more inneriterations
M. Passacantando Optimization Methods 26 / 28 –
Active-set method Frank-Wolfe method Penalty method Barrier methods
Choice of starting point
How to find x0 ∈ int(Ω)?
Consider the auxiliary problem minx,s
s
gi (x) ≤ s
I Take any x ∈ Rn, find s > maxi=1,...,m
gi (x)
[(x , s) is in the interior of the feasible region of the auxiliary problem]
I Find an optimal solution (x∗, s∗) of the auxiliary problem using a barriermethod starting from (x , s)
I If s∗ < 0 then x∗ ∈ int(Ω)else int(Ω) = ∅
M. Passacantando Optimization Methods 27 / 28 –
Active-set method Frank-Wolfe method Penalty method Barrier methods
Logarithmic barrier method
Exercise 7. Implement in MATLAB the logarithmic barrier method for solving theproblem
min 12x
TQx + cTxAx ≤ b
where Q is a positive definite matrix.
Exercise 8. Run the logarithmic barriermethod with δ = 10−3, τ = 0.5, ε1 = 1and x0 = (1, 1) for solving the problem
min 12 (x1 − 3)2 + (x2 − 2)2
−2x1 + x2 ≤ 0x1 + x2 ≤ 4−x2 ≤ 0
0 0.5 1 1.5 2 2.5 3 3.5 4
0
0.5
1
1.5
2
2.5
3
M. Passacantando Optimization Methods 28 / 28 –