Optim Notes

8/12/2019 Optim Notes

1/19

Three general classes of nonlinear optimization problems can be identified, as

follows:1. One-dimensional unconstrained problems

2. Multidimensional unconstrained problems

3. Multidimensional constrained problems

Problems of the first class are the easiest to solve whereas those of the third classare the most difficult. In practice, multidimensional constrained problems are

usually reduced to multidimensional unconstrained problems which, in turn, arereduced to one-dimensional unconstrained problems. In effect, most of the

available nonlinear programming algorithms are based on the minimization of a

function of a single variable without constraints. Therefore, efficient onedimensional optimization algorithms are required, if efficient multidimensional

unconstrained and constrained algorithms are to be constructed.

The one-dimensional optimization problem isminimizeF =f(x)

where f(x) is a function of one variable. This problem has a solution if f(x) isunimodal in some range ofx, i.e.,f(x) has only one minimum in some rangexL x

xU, wherexL andxU are the lower and upper limits of the minimizerx .

Two general classes of one-dimensional optimization methods are available,

namely,search methods and approximation methods.

In search methods, an interval [xL, xU] containing x , known as a bracket, isestablished and is then repeatedly reduced on the basis of function evaluations until

a reduced bracket [xL,k, xU,k] is obtained which is sufficiently small. Theminimizer can be assumed to be at the center of interval [xL,k, xU,k]. These

methods can be applied to any function and differentiability of f(x) is not essential.

In approximation methods, an approximation of the function in the form of a low-

order polynomial, usually a second- or third-order polynomial, is assumed. This isthen analyzed using elementary calculus and an approximate value of x is

deduced. The interval [xL, xU] is then reduced and the process is repeated several

times until a sufficiently precise value of x is obtained. In these methods, f(x) isrequired to be continuous and differentiable, i.e.,f(x) C1.

Several one-dimensional optimization approaches are as follows:1. Dichotomous search

2. Fibonacci search3. Golden-section search


2/19

Dichotomous Search

Consider a unimodal function which is known to have a minimum in the interval

[xL, xU]. This interval is said to be the range of uncertainty. The minimizerx of

f(x) can be located by reducing progressively the range of uncertainty until a

sufficiently small range is obtained. In search methods, this can be achieved byusing the values off(x) at suitable points.

If the value of f(x) is known at a single point xa in the rangexL < xa < xU, point

x is equally likely to be in the range xL toxa orxa toxU as depicted in Fig. 1(a).

Consequently, the information available is not sufficient to allow the reduction of

the range of uncertainty. However, if the value off(x) is known at two points, say,xa and xb, an immediate reduction is possible. Three possibilities may arise,

namely,

(a)f(xa) < f(xb)(b)f(xa) > f(xb)(c)f(xa) =f(xb)

In case (a),x may be located in rangexL < x < xa orxa < x < xb, that is,xL

< x < xb , as illustrated in Fig. 4.1a. The possibility xb < x < xU is definitelyruled out since this would imply thatf(x) has two minima: one tothe left ofxb and

one to the right of xb. Similarly, for case (b), we must havexa < x < xU as in

Fig. 1b. For case (c), we must have xa < x < xb, thatis, both inequalities xL 0, such as 0.01. Select a small t

such that 0 < t < ba, called the length of uncertainty for the search. Calculate thenumber of iterations n using the formula

STEP 2: For k = 1 to n, do Steps 3 and 4.

STEP 3:

STEP 4: (For a maximization problem)

Iff (x1) f (x2), then b =x2 else a =x1. k = k + 1 Return to Step 3.

STEP 5: Let


5/19

In stead of determining the number of iterations, we may wish to continueuntil the change in the dependent variable is less than some predeterminedamount, say . That is continue to iterate untilf (a)f (b) .

To minimize a functiony =f (x), either maximizey or switch the directions

of the signs in Step 4.

Example

Maximize f (x) = x2 2x over the interval 3 x 6. Assume the optimaltolerance to be less than 0.2 and we choose = 0.01.

We determine the number of iterations to be


6/19

Fibonacci Search

Fibonacci search is a univariate search technique that can be used to find the

maximum (minimum) of an arbitrary unimodal, univariate objective function. The

name Fibonacci search has been attributed to this technique because of the search

procedures dependency on a numerical sequence called Fibonacci number.Xn= Xn-1+ Xn-2., n = 2, 3

Xn= 1 and X1= 1

The Fibonacci sequence

Identifier Sequence Fibonacci number

F0 0 1

F1 1 1

F2 2 2

F3 3 3

F4 4 5F5 5 8

F6 6 13

F7 7 21

F8 8 34

F9 9 55

F10 10 89

F11 11 144

F12 12 233

F13 13 377F14 14 610

F15 15 987

Etc. Etc. Etc.

Step-1: Define the end points of the search, A and B.


7/19

Step-2: Define the number of functional evaluations, N, that are to be used in the

search.

Step-3: Define the minimum resolution parameter, .

Step-4: Define the initial interval and first interval of uncertainity as (B-A)

Therefore, L0=L1= (B-A)Step-5: Define the second interval of uncertainty as follows:

, -

Step-6: Locate the first two functional evaluations at the two symmetric points X1

and X2, defined as follows:

X1= A + L2

X2= BL2

Step-7: Calculate f(X1) and f(X2), and eliminate the interval in which the optimum

cannot lie.

Step-8: Use the relationship Ln= Ln-2Ln-1to locate the subsequent points of

evaluation within the remaining interval of uncertainity.

Continue to repeat the steps 7 and 8 until N functional evaluations have been

executed. The final solution can either be an average of the last two points

evaluated (XNand XN-1) or the best (max/min) functional evaluation.

Example

Maximize the function f(x) = - 3x2 + 21.6x + 1, with a minimum resolution of

0.5 over six functional evaluations. The optimal value of f(x) is assumed to lie

in the range (0, 25)

Solution:

L0= 25, L1= 25

, - , -

The first two functional evaluations will be conducted over the range of 0, 25X1= 0 + 15.4231 = 15.4231, f (X1) = -379.477

X2= 2515.4231 = 9.5769, f (X2) = -67.233

0 9.58 15.42 25


8/19

Hence the region to the right of X1= 15.42 can be eliminated.

Similarly,

Functional

evaluations(n)

Interval of

Uncertainity

Xn-1 f(Xn-1) Xn f(Xn)

2 0, 15.4231 9.5769 -67.233 15.4231 -379.477

3 0, 9.5769 5.8462 24.744 9.5769 -67.233

4 0, 5.8462 3.731 39.83 5.8462 24.744

5 2.115,

5.8462

2.115 32.26 3.731 39.83

6 2, 4.2304 3.731 39.83 4.2304 38.688

At the sixth functional evaluation, the interval of uncertainity is established as2.115.

The best estimate of the optimal solution is given by X5*= 3.731, f (X5

*) = 39.83.

Golden Section Search Method

In performing the a Fibonacci search, the two primary drawbacks are the a priori

specification of the resolution factor and the number of experiments to be

performed.

*+ [

]The limit of the ratio of

goes to 0.618. This is known as the golden ratio or

golden section.

Step-1: Define the initial interval of uncertainity as L0= BA, where B is the

upper bound and A is the lower bound.

Step-2: Determine the first two functional evaluations at points X1 and X2 defined

by,X1= A + 0.618 (B - A)

X2= B0.618 (B - A)

Step-3: Eliminate the appropriate region in which the optimum cannot lie.

Step-4: Determine the region of uncertainity defined by

Lj+1= Lj-1Lj j = 2, 3, .


9/19

Where

L0= (B - A)

L1= (B - A)

L2= X1A

OrL2= BX2

Depending upon the region eliminated at step-3

Step-4: Establish a new functional evaluation using the result of step-4; Evaluate

f(x) at this point, and then go to step-3. Repeat this procedure until a specified

convergence criteria is satisfied.

Example

Minimize f(x) = x415x

3+ 72x

21135x

Terminate the search when

| | The initial range of x is 1 x 15.

Solution:

The first two points are placed symmetrically within the interval 1 x 15.The golden section ratio places at

X1= 1 + 0.618 (15 - 1) = 9.652, f(X1) = 595.70

And X2= 150.618 (15 - 1) = 6.348, f(X2) = - 168.82

Therefore the region to the right of X = 9.652 can be eliminated, and the interval of

uncertainity after two functional evaluations is given by 9.652 X 1.

Similarly,

Functionalevaluation

(n)

Xn-1(right)

f(Xn-1) Xn(left)

f(Xn) Interval ofuncertainity

Length

2 9.652 595.7 6.346 -168.8 9.652 X 1 8.652

3 6.346 -168.8 4.304 -100.6 9.652 X4.304

5.348

4 7.609 -

114.64

6.346 -168.8 7.609 X

4.304

3.305


10/19

5 6.346 -168.8 5.566 -147.61

7.609 X5.566

2.043

6 6.828 -

166.42

6.346 -168.8 6.828 X

5.566

1.262

7 6.346 -168.8 6.048 -163.25 6.828 X6.048 0.780

8 6.53 -169.83

6.346 -168.8 6.828 X6.346

0.482

9 6.643 -169.34

6.53 -169.83

6.643 X6.346

0.297

At iteration number 9, note that

f(X9) = -169.34 and f (X8)= -169.83

Hence, | | Since termination criteria are satisfied, the golden section search will stop at thispoint. The best answer is given by X* =6.643 and f(X*) = -169.34.


11/19

REGULA-FALSI METHOD

The convergence process is a bisection method is very low. It depends only on the

choice of end points of the interval [a, b]. The function f(x) does not have any role

in finding the point c (which is just mid-point of a and b). It is used only to decidethe next smaller interval [a, c] or [c, b]. A better approximation to c can be

obtained by taking the straight line L joining the points (a, f(a)) and (b, f(b))intersecting the x-axis. To obtain the value of c we can equate the two expressions

of the slope m of the line L.

( )

c = b -f(b) * (b-a)

f(b) - f(a)


12/19

Now the next smaller interval which brackets the root can be obtained by checking

f(a) * f(b) < 0 then b = c> 0 then a = c

= 0 then c is the root.

Selecting cby the above expression is called Regula-Falsi method or False position

method.

The false position method is again bound to converge because it brackets the root

in the whole of its convergence process.

Numerical Example :

Find a root of 3x + sin(x) - exp(x) = 0. The graph of this equation is given in the

figure. From this it's clear that there is a root between 0 and 0.5 and alsoanother root between 1.5 and 2.0. Now let us consider the function f (x)

in the interval [0, 0.5] where f (0) * f (0.5) is less than zero and use the regula-

falsi scheme to obtain the zero of f (x) = 0.

Algorithm - False Position Scheme

c =a*f(b) - b*f(a)

f(b) - f(a)

Given a function f (x) continuos on an interval [a,b] such that f (a) *

f (b) < 0 Do

if f (a) * f (c) < 0 then b = c

else a = cwhile (none of the convergence criterion C1, C2 or C3 is satisfied)

Iteration

No.a b c f(a) * f(c)


13/19

Example -2

Find the root of x * cos[(x)/ (x-2)]=0

a = 1 and b = 1.5

IterationNo.

a b c f(a) * f(c)

1 1 1.5 1.133 0.159 (+ve)

2 1.133 1.5 1.194 0.032 (+ve)3 1.194 1.5 1.214 3.192E-3 (+ve)4 1.214 1.5 1.22 2.586E-4(+ve)

5 1.22 1.5 1.222 1.646E-5 (+ve)

6 1.222 1.5 1.222 3.811E-9(+ve)So one of the roots of x * cos[(x)/ (x-2)]=0is approximately 1.222.

Example-3

Find the root of x2= (exp(-2x) - 1) / x for a = -0.5 and b = 0.5

Iteration


1 -0.5 0.5 0.209 -0.646 (-ve)

2 -0.5 0.208 0.0952 -0.3211 (-ve)

3 -0.5 0.0952 0.0438 -0.1547 (-ve)

4 -0.5 0.0438 0.0201 -0.0727 (-ve)5 -0.5 0.0201 9.212E-3 -0.0336 (-ve)

6 -0.5 9.212E-3 4.218E-3 -0.015 (-ve)

7 -0.5 4.218E-3 1.931E-3 -7.1E-3 (-ve)8 -0.5 1.931E-3 8.83E-4 -3.2E-3 (-ve)

So one of the roots of x2= (exp(-2x) - 1) / xis approximately 8.83E-4.

1 0 0.5 0.376 1.38 (+ve)

2 0.376 0.5 0.36 -0.102 (-ve)

3 0.376 0.36 0.36 -0.085 (-ve)


14/19

Example -4

Find the root of exp(x2-1)+10sin(2x)-5 = 0 for a = 0 and b = 0.5

IterationNo. a b c f(a) * f(c)

1 0 0.5 0.272 -2.637 (-ve)2 0 0.272 0.242 -0.210 (-ve)

3 0 0.242 0.24 -0.014 (-ve)4 0 0.24 0.24 -2.51E-3(-ve)

So one of the roots of exp[x2-1]+10sin(2x)-5 = 0is approximately 0.24.

Example5

Find the root of exp(x)-3x2=0 for a = 3 and b = 4

Iteration


1 3 4 3.512 24.137 (+ve)

2 3.512 4 3.681 3.375 (+ve)

3 3.681 4 3.722 0.211 (+ve)4 3.722 4 3.731 9.8E-3 (+ve)

5 3.731 4 3.733 3.49E-4 (+ve)

6 3.733 4 3.733 1.733*10-3 (+ve)

So one of the roots of exp(x)-3x2=0is approximately 3.733.


15/19

Example6

Find the root of tan(x)-x-1 = 0 for a = 0.5 and b = 1.5

Iteration


1 0.5 1.5 0.576 0.8836 (+ve)

2 0.576 1.5 0.644 0.8274 (+ve)3 0.644 1.5 0.705 0.762 (+ve)

4 0.705 1.5 0.76 0.692 (+ve)5 0.76 1.5 0.808 0.616 (+ve)

6 0.808 1.5 0.851 0.541 (+ve)

. . . . .33 1.128 1.5 1.129 1.859E-4(+ve)

34 1.129 1.5 1.129 2.947E-6(+ve)

So one of the roots of tan(x)-x-1 = 0is approximately 1.129.

Example - 7

Find the root of sin(2x)-exp(x-1) =0 for a = 0 and b = 0.5

IterationNo.

a b c f(a) * f(c)

1 0 0.5 0.305 -0.027 (-ve)

2 0 0.305 0.254 -4.497E-3(-ve)3 0 0.254 0.246 -6.384E-4 (-ve)

4 0 0.246 0.245 -9.782E-5 (-ve)5 0 0.245 0.245 -3.144E-5 (-ve)

So one of the roots of sin(2x)-exp(x-1) = 0is approximately 0.245.

Example 8

Find the root between (2,3) of x3+ - 2x - 5 = 0, by using regular falsimethod.Givenf(x) = x3- 2 x - 5f(2) = 23- 2 (2) - 5 = -1 (negative)f(3) = 33- 2 (3) - 5 = 16 (positive)Let us take a= 2 and b= 3.


16/19

The first approximation to root is x1and is given byx1= (a f(a) - b f(b))/(f(b)-f(a))=(2 f(3)- 3 f(2))/(f(3) - f(2))=(2 x 16 - 3 (-1))/ (16- (-1))= (32 + 3)/(16+1) =35/17= 2.058

Now f(2.058) = 2.0583- 2 x 2.058 - 5= 8.716 - 4.116 - 5= - 0.4The root lies between 2.058 and 3

Taking a = 2.058 and b = 3. we have the second approximation to the rootgiven by

x2= (a f(a) - b f(b))/(f(b)-f(a))= (2.058 x f(3) - 3 x f(2.058)) /(f(3) - f(2.058))= (2.058 x 16 -3 x -0.4) / (16 - (-0.4))= 2.081

Now f(2.081) = 2.0812- 2 x 2.081 - 5= -0.15The root lies between 2.081 and 3Take a = 2.081 and b = 3The third approximation to the root is given byx3= (a f(a) - b f(b))/(f(b)-f(a))= (2.089 X 16 - 3 x (-0.062))/ (16 - (-0.062))= 2.093The required root is 2.09

Practice Problems

Find the approximate value of the real root ofx log 10x = 1.2by regula falsimethod

Find the root of thex ex= 3by regula false method and correct to the threedecimal places

Find a root which lies between 1 and 2 of f(x) = x3+ 2 x2+ 10x -20(Leonardo's Equation) Using the regula falsi method


17/19

Newton-Raphson method

The Newton-Raphson method considers a linear approximation to the first

derivative of the function using the Taylors series expansion. Subsequently, this

expression is equated to zero to find the initial guess. If the current point at

iteration t is X(t), the point in the next iteration is governed by the nature of the

following expression.

The iteration process given by above equation is assumed to have converged when

the derivative, f(X(t+1)), is close to zero.


18/19

where is a small quantity. The following figure depicts the convergence process

in the Newton-Raphson method, where x*

is the true solution.

Example

Find the minimum of the function

using the Newton-Raphson method with the starting point x(1)

= 0.1. Use =

0.01 in equation (3) for checking the convergence.

Solution

The first and second derivatives of the function f(x) are given by


19/19

Documents

Optim Notes