View
267
Download
2
Category
Preview:
Citation preview
Multi-variable Unconstrained Optimization:
Unidirectional Search
Dr. Nasir M Mirza
Optimization TechniquesOptimization Techniques
Email: nasirmm@yahoo.com
In this Lecture …• In this lecture we will discuss two important methods
that deal with Multi-variable Unconstrained Optimization:
• Multi – Dimensional Plots and
• Uni-directional Search
Example 4.2The following function of two variables has distinct local minimum points and inflection point:
f = (x - 2)2 + (x - 2у2)2
Let us draw a mesh – grid graph in three dimensions using MATLAB.
% matlab function for 3D plot clear all [X,Y] = meshgrid(-2:.1:2);Z = ( X - 2.0).^2 + (X - 2.*Y.*Y).^2 ; mesh(X, Y, Z) %interpolatedaxis tight; hold onplot3(x,y,z,'.','MarkerSize',15)
% matlab function for 3D plot clear all [X,Y] = meshgrid(-2:.1:2);Z = ( X - 2.0).^2 + (X - 2.*Y.*Y).^2 ; surf(X,Y,Z,'FaceColor','red','EdgeColor','none')camlight left; lighting phong
SURFACE PLOT using MATLAB
Example 4.2 continued• Object function is: f = (x - 2)2+ (x - 2у2)2
⎟⎟⎠
⎞⎜⎜⎝
⎛
+−−+−
=
⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜
⎝
⎛
∂∂∂∂
=3
2
168444yxyyx
yfxf
vectorgradient
⎟⎟⎠
⎞⎜⎜⎝
⎛+−−−
=
⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢
⎣
⎡
∂∂
∂∂∂
∂∂∂
∂∂
= 2
2
22
2
2
2
488884
yxyy
yf
xyf
yxf
xf
H
32
22
168)2(8
444)2(2)2(2
yxyyxyyf
yxyxxxf
+−=−−=∂∂
−+−=−+−=∂∂
% matlab function for 3D plot clear all [X,Y] = meshgrid(-2:.05:5); Z = ( X - 2.0).^2 + (X - 2.*Y.*Y).^2 ; contour(X,Y,Z, 3000)
Next step is to draw a contour graph and identify points of interest.
(Here we see (2, -1)T and (2, 1)T as important points.
CONTOUR PLOT using MATLAB
% matlab function for 3D plot clear all [X,Y] = meshgrid(-2:.05:5); Z = ( X - 2.0).^2 + (X - 2.*Y.*Y).^2 ; surf(X,Y,Z,'FaceColor','red','EdgeColor','none')camlight left; lighting phonghold oncontour(X,Y,Z, 1000)
SURFACE PLOT using MATLAB
Example 4.2 continuedTwo types of contour plots showing possible maximum or minimum points
Example 4.2• Optimality condition (first order):
0168)2(8
0444)2(2)2(2
32
22
≡+−=−−=∂∂
≡−+−=−+−=∂∂
yxyyxyyf
yxyxxxf
Possible solutions (x, y) are following:
(1, 0) ; (2, -1); (2, 1)
Let us check with second optimality condition for each point:
For point (1, 0):
⎟⎟⎠
⎞⎜⎜⎝
⎛−
=80
04H
Principal minors are:
A1 = a11= 4, A2 = det(A)= -32
It is then inflection point and Function value = 2.
Example 4.2Let us check with second optimality condition for point (2, -1)
⎟⎟⎠
⎞⎜⎜⎝
⎛=
32884
HPrincipal minors are:
A1 = a11= 4, A2 = det(A)= 128 – 64 = 64
It is then minimum point and Function value = 0.
Let us check with second optimality condition for point (2, +1)
⎟⎟⎠
⎞⎜⎜⎝
⎛−
−=
32884
HPrincipal minors are:
A1 = a11= 4, A2 = det(A)= 128 – 64 = 64
It is then minimum point and Function value = 0.
Example 4.3• The following function of two variables has only one minimum point
but two distinct local maximum points and two inflection points:
f(x) = 25x2 – 12x4 – 6xy + 25y2 – 24x2y2 – 12y4.
% matlab function for 3D plot clear all [X,Y] = meshgrid(-1.5:.05:1.5); Z = 25.*X.^2-12.*X.^4-6.*X.*Y +25.*Y.^2 -24.*(X.^2).*(Y.^2) -12.*Y.^4. surf(X,Y,Z,'FaceColor','red','EdgeColor','none')camlight left; lighting phonghold oncontour(X,Y,Z, 1000)
Let us first draw 3D graph:
% matlab function for this graphclear all [X,Y] = meshgrid(-1.5:.1:1.5); Z = 25.*X.^2-12.*X.^4-6.*X.*Y +25.*Y.^2 -24.*(X.^2).*(Y.^2) - 12.*Y.^4. surf(X, Y, Z);colormap(jet);
Example 4.3
Example 4.3
% matlab function for 3D plot clear all [X,Y] = meshgrid(-1.5:.01:1.5); Z = 25.*X.^2-12.*X.^4-6.*X.*Y +25.*Y.^2 -24.*(X.^2).*(Y.^2) -12.*Y.^4. contour(X, Y, Z, 200);
Example 4.3• Object function is: f(x) = 25x2 – 12x4 – 6xy + 25y2 – 24x2y2 – 12y4.
⎟⎟⎠
⎞⎜⎜⎝
⎛
−−−−−−−−
=
⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢
⎣
⎡
∂∂
∂∂∂
∂∂∂
∂∂
=22
22
2
22
2
2
2
14448509669664814450
yxxyxyyx
yf
xyf
yxf
xf
H
32
23
4848506
4864850
yyxyxyf
xyyxxxf
−−+−=∂∂
−−−=∂∂
Example 4.3• Optimality condition (first order):
Possible solutions (x, y) are following:
(-0.764, 0.764) ; (-0.677, -0.677); (0, 0); (0.764, -0.764) ; (0.677, 0.677);
Let us check with second optimality condition for each point:
For point (-0.764, 0.764):
⎟⎟⎠
⎞⎜⎜⎝
⎛−
−=
62505062
H
Principal minors are:
A1 = a11= -62, A2 = det(A)= 1344
It is then maximum point and
Function value = 16.33
04848506
04864850
32
23
=−−+−=∂∂
=−−−=∂∂
yyxyxyf
xyyxxxf
Example 4.3Let us check with second optimality condition for point (-0.677, -0.677)
⎟⎟⎠
⎞⎜⎜⎝
⎛−−−−
=38505038
HPrincipal minors are:
A1 = a11= -38, A2 = det(A)= -1056
It is then inflection point and Function value = 10.08
Let us check with second optimality condition for point (0, 0)
⎟⎟⎠
⎞⎜⎜⎝
⎛−
−=
506650
HPrincipal minors are:
A1 = a11= 50, A2 = det(A)= 2464
It is then minimum point and Function value = 0.
Example 4.3Let us check with second optimality condition for point (+0.677, +0.677)
⎟⎟⎠
⎞⎜⎜⎝
⎛−−−−
=38505038
HPrincipal minors are:
A1 = a11= -38, A2 = det(A)= -1056
It is then inflection point and Function value = 10.08
Let us check with second optimality condition for point (0.764, -0.764)
⎟⎟⎠
⎞⎜⎜⎝
⎛−
−=
62505062
HPrincipal minors are:
A1 = a11= -62, A2 = det(A)= 1364
It is then maximum point and
Function value = 16.3
Example 4.3
The graph shows these points on contour plot as maximum, minimum and inflection points;
Maximum
Maximum
Inflection
Inflection
Minimum
Unidirectional Search Method• A unidirectional search is a one-dimensional search
performed by comparing function values only along a specific direction.
• Usually a unidirectional search is performed from a point x(t) in a specific direction s(t).
• Only points that lie on a line passing through x(t) in direction of s(t) are allowed.
• Any arbitrary point on that line can be expressed as
x(a) = x(t) + as(t).
• The parameter a is a scalar quantity and it is measure of distance of point x(a) from x(t).
Unidirectional Search Method
Once optimal a is found the corresponding point is found.
• The above equation is a vector equation in multi-dimensional space.
• Hence when a is known the x(a) can be estimated.
• Any positive or negative value of a will create x(a) on the line for search.
• When a is zero, the current point x(t) is obtained.
• When we want minimum point, we convert objective function in terms of single variable a by putting all xi as
x(a) = x(t) + as(t).
Example 3.2.1 Consider the objective function:
Minimize: f(x, y) = (x – 10)2 + (y – 10)2
• Step 1: First draw the contour plot of this function. The contour plot has lines where any two points on the line have the same function value.
• The optimal point is a minimum at (10, 10)T. The function value is zero at this point.
• Step 2: Let us say that a current point selected is x(t) = (2, 1)T. and the function value in a search direction s(t) = (2, 5)T from the current point. x(a) = x(t) + as(t) ; vector equation
x(a) = 2 + a2 ; y(a) = 1 + a5; scalar equations
Example 3.2.1 • Now we find the equation for the straight line passing
through (2, 1)T and direction (2, 5)T as:(x – 2)/2 = (y – 1)/5 ; y – 1 = (5/2)(x – 2).
• Now let us find a point on this line as x = 3 then y = 3.5 and we can estimate a from above equations as a = 0.5
• If the point on line is with x = 9 and y = 18.5, then a = 3.5
• The function value for (3, 3.5)T is 91.25 • And at point (9, 18.5)T the value is 73.25.• Then at a = (3.5 + 0.5)/2 = 2, then x = 6, y = 11 and
function value is 17. So, approx. opt. is (6, 11)T.
Example 3.3.1Consider the Himmelblau function: Minimize: f(x, y) = (x2 + y – 11)2 + (x + y2 –7)2
in the interval 0 < x < 5;
Step 1: This function is plotted as Figure. It appears from the plot that a minimum point is at (3, 2)T.
% matlab function for 3D plot clear all [X,Y] = meshgrid(0:.05:5); Z = ( X.*X + Y - 11.0).^2 + (X + Y.*Y - 7.0).^2 ; surf(X, Y, Z);colormap(jet);
% matlab function for contour plot clear all axis([0 5 0 5])hold on[X,Y] = meshgrid(-2:.05:5); Z = ( X.*X + Y - 11.0).^2 + (X + Y.*Y - 7.0).^2 ; contour(X, Y, Z, 200)
Example 3.3.1: contour linesA contour line is a collection of points having identical function values.
The contour plot for the object function is shown here.
The continual decrease in the function value of the successive contour lines as they approach a point means that the point is minimum point.
Example 3.3.1Step 1: We now choose an initial point x(0) = (1, 1)T
Size reduction parameter = (2, 2)T
Also choose e = 0.001
Step 2: We create a two dimensional hypercube (a square) around x(0) such that
X(1) = (0, 0)T , x(2) = (2, 0)T ,
X(3) = (0, 2)T , x(4) = (2, 2)T
Step 3: The function values at these points are:
f(0) = 106 ; f(1) = 170 ; f(2) = 74
f(3) = 90 ; f(4) = 26 The minimum is at (2, 2)T. Let us say this as x* =(2, 2)T
Example 3.3.1Step 4: Since x* is not x(0), we now set x(0) = (2, 2)T and got to step 2.
Step 2: We create a new square around this x(0) with Δ = 1 such that
X(1) = (1, 1)T , x(2) = (3, 1)T ,
X(3) = (1, 3)T , x(4) = (3, 3)T
Step 3: The function values at these points are:
f(0) = 26 ; f(1) = 106 ; f(2) = 10
f(3) = 58 ; f(4) = 26 The minimum is at (3, 1)T. The function value is 10.
Let us say this as x* =(3, 1)T
Example 3.3.1Step 4: Since x* is not x(0), we now set x(0) = (3, 1)T and go to step 2.
Step 2: We create a new square around this x(0) with Δ = 1 such that
X(1) = (2, 0)T , x(2) = (4, 0)T ,
X(3) = (2, 2)T , x(4) = (4, 2)T
Step 3: The function values at these points are:
f(0) = 10 ; f(1) = 74 ; f(2) = 34
f(3) = 26 ; f(4) = 50 The minimum is at (3, 1)T where function value is 10. It is same as x(0).
Let us say this as x* =(3, 1)T
Example 3.3.1Step 4: Since x* is same as x(0), we now set x(0) = (3, 1)T and go to step 2 after reducing step size = 0.5
Step 2: We create a new square around this x(0)=(3, 1)T with Δ = 0.5 such that
X(1) = (2.5, 0.5)T , x(2) = (3.5, 0.5)T
,
X(3) = (2.5, 1.5)T , x(4) = (3.5, 1.5)T
Step 3: The function values at these points are computed and minimum is at x(4) = (3.5, 1.5)T
And function value is 9.125.
The minimum is at (3.5, 1.5)T
Let us say this as x* =(3.5, 1.5)T
Example 3.3.1Step 4: Since x* is not same as x(0), we now set x(0) = (3.5, 1.5)T
and go to step 2 with step size = 0.5
Step 2: We create a new square around this x(0)=(3, 1)T with Δ = 0.5 such that
X(1) = (3, 1)T , x(2) = (4, 1)T ,
X(3) = (3, 2)T , x(4) = (4, 2)T
Step 3: The function values at these points are
f(0) =9.125; f(1) = 10 ;
f(2) = 40; f(3) = 0; f(4) = 50
the minimum is f(3) = 0.0
The minimum is at (3, 2)T
This is the best minimum so far as f=0.
Example 3.3.1It is important to see that although we have found a minimum point but the algorithm does not terminate at this step. We may continue.
It is clear that the convergence depends upon the initial cube size and location and the chosen size reduction parameter.
Small size may lead to premature convergence and large may never lead to convergence.
The minimum is at (3, 2)T
This is the best minimum so far as f=0.
Example: Suppose f(x, y) = 2xy + 2x – x2 – 2y2
Using the steepest ascent method to find the next point if we are moving from point (-1, 1).
yxyfxy
xf 42222 −=
∂∂
−+=∂∂
⎥⎦
⎤⎢⎣
⎡−
=⎥⎦
⎤⎢⎣
⎡−−
−−+=
⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢
⎣
⎡
−∂∂
−∂∂
=∇−6
6)1(4)1(2
)1(22)1(2
)1,1(
)1,1(),1,1(At
yfxf
f
)6161()(Let hh,-fhg −+=
Next step is to find h that maximize g(h)
2.0036072yields0)('Setting
180727...)6161()(222)(
2
22
=⇒=−=
−+−==−+=
−−+=
hhhg
hhhh,-fhgyxxxyxf
If h = 0.2 maximizes g(h), then x = -1+6(0.2) = 0.2 and y = 1-6(0.2) = -0.2 would maximize f(x, y).
So moving along the direction of gradient from point (-1, 1), we would reach the optimum point (which is our next point) at (0.2, -0.2).
Recommended