Upload
others
View
1
Download
0
Embed Size (px)
Citation preview
Introduction to Numerical Optimization (Part 1)Dr. José Ernesto Rayas-Sánchez
March 28, 2008
1
1
An Introduction to Numerical Optimization
(Part 1)
Dr. José Ernesto Rayas Sánchez
2Dr. J. E. Rayas Sánchez
Outline
Mathematical formulations
Types of optimization problems
Minimizers
Recognizing a minimizer
Types of optimization methods
Line search methods
Trust region methods
Introduction to Numerical Optimization (Part 1)Dr. José Ernesto Rayas-Sánchez
March 28, 2008
2
3Dr. J. E. Rayas Sánchez
Unconstrained Optimization Problem
Given u: ℜn→ℜ, find
nuu ℜ∈≤ xxx allfor )()( *
)(minarg* xx x u=
x* is a global minimizer ⇔
x* is a local minimizer ⇔
Replacing ≤ by < we have a strict global minimizer and a strict local minimizer, respectively
ℵ∈≤ xxx allfor )()( * uu:
2
* ε<−ℜ∈=ℵ xxx n
where u is the objective function (or cost function), and xcontains the optimization variables
4Dr. J. E. Rayas Sánchez
Example 1 – Quadratic
22
21 )12()1()( −+−= xxu x ⎥⎦
⎤⎢⎣
⎡=
2
1
xx
xwhere
-2-1
01
23
-2
0
2
40
10
20
30
40
x1
Quadratic Function
x2
-2 -1 0 1 2 3-2
-1
0
1
2
3
x1
x 2
Quadratic Function
Introduction to Numerical Optimization (Part 1)Dr. José Ernesto Rayas-Sánchez
March 28, 2008
3
5Dr. J. E. Rayas Sánchez
Example 2 – Rosenbrock
-2 -1.5 -1 -0.5 0 0.5 1 1.5 2-2
-1.5
-1
-0.5
0
0.5
1
1.5
2
x1
x 2
Rosenbrock Function
21
2212 )1()(100)( xxxu −+−=x ⎥⎦
⎤⎢⎣
⎡=
2
1
xx
xwhere
-2-1
01
2
-2-1
0
120
1000
2000
3000
4000
x1
Rosenbrock Function
x2
6Dr. J. E. Rayas Sánchez
Example 3 – Sinc
rru )sin()( =x ⎥⎦
⎤⎢⎣
⎡=
2
1
xx
xwhere
-10-5
05
10
-10-5
0
510
-0.5
0
0.5
1
x1
Sinc Function
x2
-10 -5 0 5 10-10
-5
0
5
10
x1
x 2
Sinc Function
22
21 xxr +=
Introduction to Numerical Optimization (Part 1)Dr. José Ernesto Rayas-Sánchez
March 28, 2008
4
7Dr. J. E. Rayas Sánchez
Example 4 – Styblinski and Tang
)5165160.5()( 222
421
21
41 xxxxxxu +−++−=x ⎥⎦
⎤⎢⎣
⎡=
2
1
xx
xwhere
-4 -3 -2 -1 0 1 2 3 4-4
-3
-2
-1
0
1
2
3
4
x1x 2
Styblinski and Tang (1990) Function
-4-2
02
4
-4-2
0
24
-80
-60
-40
-20
0
20
x1
Styblinski and Tang (1990) Function
x2
8Dr. J. E. Rayas Sánchez
Example 5 – Venkataraman
dcqxdpxcbxaxu ++−−+= )cos()cos()( 2122
21x
⎥⎦
⎤⎢⎣
⎡=
2
1
xx
xwhere ππ 4,3,4.0,3.0,2,1 ====== qpdcba
-1-0.5
00.5
1
-1-0.5
0
0.510
2
4
6
8
10
x1
Venkataraman (2002) Function
x2
-1 -0.5 0 0.5 1-1
-0.5
0
0.5
1
x1
x 2
Venkataraman (2002) Function
Introduction to Numerical Optimization (Part 1)Dr. José Ernesto Rayas-Sánchez
March 28, 2008
5
9Dr. J. E. Rayas Sánchez
Constrained Optimization Problem
Given u: ℜn→ℜ, find
Ω∈≤ xxx ),()( * uu
)(minarg* xxx
uΩ∈
=
x* is a global minimizer ⇔
x* is a local minimizer ⇔
u is the objective function (or cost function), x contains the optimization variables, Ω is the feasible set or region
ℵ∈≤ xxx allfor )()( * uu:
2
* εΩ <−∈=ℵ xxx
if u(x) is convex, a local minimizer is also the global minimizer
10Dr. J. E. Rayas Sánchez
Constrained Optimization Problem (cont)
Given u: ℜn→ℜ, find
⎩⎨⎧
∈≥∈=
=Ij
Ei
jcic
uΩΩ
0)(0)(
subject to )(minarg*
xx
xx x
ci are equality constrains, cj are inequality constrains
If u(x) and all the constraints are linear functions of x, the problem is a linear programming problem
If the objective function or at least one of the constraints is nonlinear, then the problem is a nonlinear programming problem
Introduction to Numerical Optimization (Part 1)Dr. José Ernesto Rayas-Sánchez
March 28, 2008
6
11Dr. J. E. Rayas Sánchez
Recognizing a Local Minimizer
xs is a stationary point if ∇u(xs) = 0
First order necessary condition: If x* is a local minimizer then ∇u(x*) = 0
Second order sufficient conditions: If ∇u(x*) = 0 and H(u(x*)) is positive definite then x* is a strict local minimizer
If a point is a stationary point and not a local minimizer or maximizer, the point is called a “saddle point”
12Dr. J. E. Rayas Sánchez
Optimization Methods: A Broad Classification
Non-descent methods
Descent methods, u(xi+1) < u(xi) for every i, or u(xi+1) < u(xi) for i > N, where N is a number of initial steps
Two fundamental strategies:
• Line search methods
• Trust region methods
Introduction to Numerical Optimization (Part 1)Dr. José Ernesto Rayas-Sánchez
March 28, 2008
7
13Dr. J. E. Rayas Sánchez
Line Search Methods
At the i-th iteration, the algorithm chooses a direction diand searches along this direction from the current iterate xi for a new iterate xi+1 with a lower function value
The search direction and the step size can be selected in several manners
14Dr. J. E. Rayas Sánchez
Trust Region Methods
At the i-th iteration, a model mi of the objective function u is created. The algorithm restricts the search for a minimizer of mi to some region around xi. If the minimizer of mi does not produce a sufficient decrease in u, the trust region is shrunk and the model is again minimized
There are several ways to construct the model mi, and to define the trust region (ball, elliptical, box-shaped, etc.)
Introduction to Numerical Optimization (Part 1)Dr. José Ernesto Rayas-Sánchez
March 28, 2008
8
15Dr. J. E. Rayas Sánchez
A Generic Line Search Algorithm
begini = 0, xi = x0repeat until stopping_criterion
di = SearchDirection(u, xi)αi = LineSearch(u, xi, di)xi+1 = xi + αιdii = i + 1
end
16Dr. J. E. Rayas Sánchez
Stopping Criteria
A maximum number of iterations has been reached
The objective function is practically not decreasing
The absolute change in the optimization variables is small enough
11)()( ε<− +ii uu xx
221 |||| ε<−+ ii xx
maxii >
Introduction to Numerical Optimization (Part 1)Dr. José Ernesto Rayas-Sánchez
March 28, 2008
9
17Dr. J. E. Rayas Sánchez
Stopping Criteria (cont)
The relative change in the optimization variables is small enough
The gradient is small enough
The Hessian is positive definite (near the solution, xi ≈ x*)
)||(|||||| 42321 εε +<−+ iii xxx
52||)(|| ε<∇ iu x
611 )))((()(21 ε<−− ++ iii
Tii u xxxHxx
18Dr. J. E. Rayas Sánchez
Search Directions for Line Search Methods
Steepest descent direction)( ii u xd −∇=
Introduction to Numerical Optimization (Part 1)Dr. José Ernesto Rayas-Sánchez
March 28, 2008
10
19Dr. J. E. Rayas Sánchez
Search Directions for Line Search Meth. (cont)
The steepest descent direction with exact line searches moves in orthogonal steps
20Dr. J. E. Rayas Sánchez
Search Directions for Line Search Meth. (cont)
The steepest descent direction can be extremely slow (zigzagging)
Introduction to Numerical Optimization (Part 1)Dr. José Ernesto Rayas-Sánchez
March 28, 2008
11
21Dr. J. E. Rayas Sánchez
Search Directions for Line Search Meth. (cont)
Newton direction
in this case the most used step length is 1
di is a linear transformation of −∇u(xi)
Quasi-Newton directions
Bi is an approximation of H(∇u(xi)) which is updated after each iteration to take into account the additional knowledge gained during the step
)())(( 1iii uu xxHd ∇−= −
)(1iii u xBd ∇−= −
22Dr. J. E. Rayas Sánchez
Search Directions for Line Search Meth. (cont)
The two most popular updating formulas for Bi in Quasi-Newton directions:
SR1 formula (Symmetric-Rank-One)
BFGS formula (Broyden, Fletcher, Goldfarb and Shanno)
where
iT
iii
Tiiiiii
ii ssBysBysByBB
)())((
1 −−−+=+
iTi
Tii
iiTi
iTiii
ii syyy
sBsBssBBB +−=+1
iii xxs −= +1 )()( 1 iii uu xxy ∇−∇= +
Introduction to Numerical Optimization (Part 1)Dr. José Ernesto Rayas-Sánchez
March 28, 2008
12
23Dr. J. E. Rayas Sánchez
Search Directions for Line Search Meth. (cont)
Conjugate Gradient direction
where βi is a scalar that ensures that di and di−1 are conjugate
Any two vectors a and b are conjugate with respect to a symmetric positive definite matrix A if aTAb = 0
1)( −+−∇= iiii u dxd β
24Dr. J. E. Rayas Sánchez
Finding the Step Size in Line Search Methods
Exact line search
The exact line search stops at a point where the local gradient is orthogonal to the search direction
Soft line search (See: P.E. Frandsen, K. Jonasson, H.B. Nielsen and O. Tingleff, Unconstrained Optimization. Lyngby, Denmark: Department of Mathematical Modeling, Technical University of Denmark, 1999, pp. 26-30)
)(minarg)(minarg00
ααααα
vu iii >>=+= dx
Introduction to Numerical Optimization (Part 1)Dr. José Ernesto Rayas-Sánchez
March 28, 2008
13
25Dr. J. E. Rayas Sánchez
A Generic Trust Region Algorithm
begini = 0, xi = x0, r = r0 > 0repeat until stopping_criterion
mi(s) = BuildModel(u, xi)
if a > 0.75 then r = 2rif a < 0.25 then r = r/3if a > 0 then xi+1 = xi + sii = i + 1
end
)]()(/[)]()([ iiiiii mmuua ssxx −+−= 0
rmii ≤==∈
sssss
: where)(minarg ΩΩ