Quasi-Newton Methods of Optimization

Lecture 2

General Algorithm

A Baseline Scenario• Algorithm U (Model algorithm for n-

dimensional unconstrained minimization). Let xk be the current estimate of x*.

– U1. [Test for convergence] If the conditions for convergence are satisfied, the algorithm terminates with xk as the solution.

– U2. [Compute a search direction] Compute a non-zero n-vector pk, the direction of the search.

General Algorithm

– U3. [Compute a step length] Compute a scalar ak, the step length, for which f(xk + akpk )<f(xk).

– U4. [Update the estimate of the minimum] Set xk+1 = xk + ak pk, k=k+1, and go back to step U1.

• Given the steps to the prototype algorithm, I want to develop a sample problem that we can compare the various algorithms against.

4321

1.4

4.3

3.2

2.1

32100

max

xxxxst

xxxxx

General Algorithm

– Using Newton-Raphson, the optimal point for this problem is found in 10 iterations using 1.23 seconds on the DEC Alpha.

Derivation of the Quasi-Newton Algorithm An Overview of Newton and Quasi-Newton

Algorithms• The Newton-Raphson methodology can be

used in U2 in the prototype algorithm. Specifically, the search direction can be determined by:

p f x f xk xx k x k 2 1

( ) ( )

Derivation of the Quasi-Newton Algorithm

• Quasi-Newton algorithms involve an approximation to the Hessian matrix. For example, we could replace the Hessian matrix with the negative of the identity matrix for the maximization problem. In this case the search direction would be:

p I n f xk x k ( ) ( )

Derivation of the Quasi-Newton Algorithm

• This replacement is referred to as the steepest descent method. In our sample problem, this methodology requires 990 iterations and 29.28 seconds on the DEC Alpha.

– The steepest descent method requires more overall iterations. In this example, the steepest descent method requires 99 times as many iterations as the Newton-Raphson method.

Derivation of the Quasi-Newton Algorithm

– Typically, the time spent on each iteration is reduced. Again, in the current comparison each the steepest descent method requires .123 seconds per iteration while Newton-Raphson requires .030 seconds per iteration.

Derivation of the Quasi-Newton Algorithm

• Obviously substituting the identity matrix uses no real information from the Hessian matrix. An alternative to this drastic reduction would be to systematically derive a matrix Hk which uses curvature information akin to the Hessian matrix. The projection could then be derived as:

p H f xk k x k 1 ( )

Derivation of the Quasi-Newton Algorithm Conjugate Gradient Methods

• One class of Quasi-Newton methods are the conjugate gradient methods which “build” up information on the Hessian matrix.

– From our standard starting point, we take a Taylor series expansion around the point xk + sk

x k k x k xx k kf x s f x f x s( ) ( ) ( )2

Derivation of the Quasi-Newton Algorithm

for some sk N(xk,). Solving this expression for the term involving the Hessian yields

x k k x k xx k k

k x k k x k k xx k k

f x s f x f x s

s f x s f x s f x s

( ) ( ) ( )

' ( ( ) ( )) ' ( )

2

2

Derivation of the Quasi-Newton Algorithm

y f x s f xk x k k x k ( ) ( )

y f x sk xx k k2 ( )

y B sk k k 1

Derivation of the Quasi-Newton Algorithm One way to generate Bk+1 is to start with the

current Bk and add new information on the current solution

B B uv

y B uv sk k

k k k

1 '

( ' )

Derivation of the Quasi-Newton Algorithm

u v s y B sk k k k( ' )

uv s

y B sk

k k k 1

'( )

B Bv s

y B s vk kk

k k k 1

1

'( ) '

Derivation of the Quasi-Newton Algorithm

• The Rank-One update then involves choosing v to be yk + Bksk. Among other things, this update will yield a symmetric Hessian matrix:

B By B s s

y B s y B sk kk k k k

k k k k k k

1

1

( )'( )( )'

Derivation of the Quasi-Newton Algorithm

• Other than the Rank-One update, no simple vector will result in a symmetric Hessian. An alternative is to reconfigure the Hessian by letting the numeric be the 1/2 the sum of a numeric approximation plus itself transposed. This procedure yields the general update:

B Bv s

y B s v v y B sy B s s

v sv vk k

kk k k k k k

k k k k

k

1 2

1

'( ) ' ( )'

( )'

( ' )'

DFP and BFGS

• Two prominent conjugate gradient methods are the Davidon-Fletcher-Powell (DFP) update and the Broyden-Fletcher-Goldfarb-Shanno (BFGS) update.

– In the DFP update v is set equal to yk yielding

B Bs B s

B s s By s

y y s B s w w

wy s

ys B s

B s

k kk k k

k k k kk k

k k k k k k k

kk k

kk k k

k k

1

1 1

1 1

''

'' ' '

' '

DFP and BFGS

– The BFGS update is then

B Bs B s

B s s By s

y yk kk k k

k k k kk k

k k 1

1 1

''

''

DFP and BFGS

A Numerical Example• Using the previously specified problem and

starting with an identity matrix as the original Hessian matrix, each algorithm was used to maximize the utility function.

Bt

1 0 0

1 0

1

Bt*

. . .

. .

.

5275 2885 0718

6085 0954

2244

DFP and BFGS

• In discussing the difference in step, I will focus on two attributes.

– The first attribute is the relative length of the step (the 2-norm).

– The second attribute is the direction of the step. Dividing each vector by its 2-norm yields yields a normalized direction of the search

st . . .7337 9766 2428

st* . . . 4 3559 4 3476 4 3226

DFP and BFGS

x1 x2 x3 x1 x2 x3

Newton-Raphson 4.36 4.35 4.32 7.52 0.58 0.58 0.57Conjugate Gradient 0.73 0.98 0.24 1.25 0.59 0.78 0.19

Relative Performance

– The Rank One Approximation• Iteration 1

B

B

t

t

. . .

. .

.

. . .

. .

.

*

6690 3842 1536

5540 1783

9287

2780 1276 0511

2501 0592

2280

x1 x2 x3 x1 x2 x3

Newton-Raphson 6.75 7.63 4.83 11.27 0.60 0.68 0.43Conjugate Gradient 4.32 5.01 2.00 6.91 0.62 0.73 0.29

Relative Performance

• Iteration 2

B

B

t

t

. . .

. .

.

. . .

. .

.

*

6315 4256 1776

5083 2048

9134

0628 0171 0106

0612 0113

0846

x1 x2 x3 x1 x2 x3

Newton-Raphson 10.85 11.69 6.08 17.07 0.64 0.68 0.36Conjugate Gradient 7.73 8.91 3.79 12.39 0.62 0.72 0.31

Relative Performance

– PSB• Iteration 1 B

B

t

t

. . .

. .

.

. . .

. .

.

*

6703 3860 1504

5565 1827

9365

2780 1276 0511

2501 0592

2280

x1 x2 x3 x1 x2 x3

Newton-Raphson 10.85 11.69 6.08 17.07 0.64 0.68 0.36Conjugate Gradient 7.73 8.91 3.79 12.39 0.62 0.72 0.31

Relative Performance

• Iteration 2

B

B

t

t

. . .

. .

.

. . .

. .

.

*

6328 4274 1745

5109 2096

9223

0629 0171 0106

0612 0114

0850

x1 x2 x3 x1 x2 x3

Newton-Raphson 10.85 11.70 6.07 17.07 0.64 0.69 0.36Conjugate Gradient 7.72 8.91 3.78 12.38 0.62 0.72 0.30

Relative Performance

– DFP• Iteration 1 B

B

t

t

. . .

. .

.

. . .

. .

.

*

7187 4517 0326

6455 3424

12232

2780 1276 0511

2501 0592

2280

x1 x2 x3 x1 x2 x3

Newton-Raphson 6.75 7.63 4.83 11.27 0.60 0.68 0.43Conjugate Gradient 4.14 5.01 1.76 6.73 0.61 0.74 0.26

Relative Performance

• Iteration 2

B

B

t

t

. . .

. .

.

. . .

. .

.

*

6788 4945 0589

6021 3766

12194

0653 0177 0119

0602 0124

0971

x1 x2 x3 x1 x2 x3

Newton-Raphson 6.75 7.63 4.83 11.27 0.60 0.68 0.43Conjugate Gradient 4.14 5.01 1.76 6.73 0.61 0.74 0.26

Relative Performance

– BFGS• Iteration 1

B

B

t

t

. . .

. .

.

. . .

. .

.

*

6771 3952 1338

5690 2051

9768

2780 1276 0511

2501 0593

2280

x1 x2 x3 x1 x2 x3

Newton-Raphson 10.81 11.93 5.80 17.11 0.63 0.70 0.34Conjugate Gradient 7.52 9.06 3.40 12.26 0.61 0.74 0.28

Relative Performance

• Iteration 2

B

B

t

t

. . .

. .

.

. . .

. .

.

*

6391 4369 1585

5238 2333

9644

0634 0172 0109

0610 0115

0871

x1 x2 x3 x1 x2 x3

Newton-Raphson 10.84 11.74 6.02 17.08 0.63 0.69 0.35Conjugate Gradient 7.69 8.94 3.71 12.36 0.62 0.72 0.30