Computers & Chemical Engiae&ng, Vol. IO, No. 3, pp. 241-241, 1986 0898-1354/86 83.00 + 0.00 Printed in Great Britain. All rights reserved Ccpyr@ht Q 1986 Pergamon Journals Ltd

THE UPDATING OF ILLJ FACTORS IN QUASI-NEWTON METHODS

J. R. PALoscmt and J. D. PERKINSS Department of Chemical Engineering and Chemical Technology, Imperial College,

Exhibition Road, London SW7 2BX, England

(Received 6 August 1984; revision received 13 September 1985; received for publication 18 October 1985)

Abntract-An algorithm is proposed for the updating of LU factors for use with rank-one quasi-Newton methods for the solution of sets of non-linear algebraic equations. The proposed method is a modification of an algorithm due to Bennett [l]. The modification is introduced to avoid singularity or ill-conditioning of the factorized Jacobian approximations generated by the quasi-Newton methods.

Soope-There are many applications in which a system of non-linear equations must be solved. ‘Ihe steady-state simulation of process flowsheets is probably the most common application in chemical engineering (see Refs [2,3] for recent reviews of these fields).

Although Newton’s method is attractive from the theoretical point of view, and computational experience shows it to be a reasonably reliable method [3], it cannot bc used efficiently where analytical derivatives for the equations to be solved are not available. Such cases are common in flowsheeting applications, where problems are often described using mixtures of equations and procedures (sub- routines) [2,3]. For this class of applications, quasi-Newton methods, in particular Broyden’s method [4] have been used successfully.

Recently, robust implementations of quasi-Newton methods have been produced based on updating an LU factorization of the Jacobian approximation [5,6] using an algorithm due to Bennett [l]. problems arise with this approach, either because the Jacobian approximation generated by the update becomes nearly singular or because the LIJ factorixation of the Jacobian approximation is based on an inappropriate choice of pivots. In this paper, a simple modification to Bennett’s algorithm is proposed which overcomes these two difficulties without incurring a large computational cost.

Colrlg;ioaaBd -Bennett’s algorithm [l] for updating an LU factorization of a matrix subject to a rank-one change can be modified to avoid singularity or ill-conditioning of a Jacobian approximation generated by a quasi-Newton method (e.g. Broyden’s method [4]). The modification is simple, and little computational cost is incurred by its implementation.

INTBODUCIION

Quasi-Newton methods are used to solve the problem

f(x)=O, f:R”+R”.

Having x, as an initial estimate for the solution vector x,, a sequence of vectors {x,} and a sequence of matrices {PI,} are generated. If the method succeeds in finding the solution then

limx,=x,. 1-m

The sequence of matrices {!I!,} is, for each i, an ap- proximation to the Jacobian of the function f(x) evaluated at the point x,. This sequence is obtained using an update formula as follows:

B ,+ 1 = b + (Y, - mh) &,. (1) I

tAutbor to whom all correspondence should be addressed at his present address: Planta Filoto de Ingenierla Quimica (UNS-CONICET), 12 de Octubre 1840, 8000 Bahia Blanca, Argentina.

$&sent address: Department of Chemical Engineering, University of Sydney, Sydney, NSW 2006, Australia.

To obtain p, the following linear system must be solved:

B,P, = -fi, (2) where

f, =f (Xi) -

Once equation (2) has been solved, xi+ 1 is obtained as

Xi+l =x,+pi

and then yt is evaluated as

y,=f,+*-f,.

(3)

The vector v, is determined according to the quasi- Newton method being used. There are two ap- proaches to implementing these methods in order to solve equation (2) without inverting Bi at each step. The tirst one makes use of the Sherman-Morrison [7] formula to obtain an update for the inverse of B. Defining

U-I, = Et;‘,

it can be shown that equation (1) implies

241

242 J. R. P~roscr-11 and J. D. PERKINS

If the matrix Bi is non-singular, it is possible to find a permutation matrix Pi, a lower triangular matrix I_, and a unit upper triangular matrix U,, such that

PilEI, = lL,U,.

If Bi+ , is non-singular, and it is possible to obtain

PiBi+I=Li+lu~+l~ (4)

then we can use Bennett’s [l] algorithm to obtain I_ ,+, and IJi+l from equation (1). The advantage of using Bennett’s algorithm is that it requires just 0(n2) operations against the 0(n3) needed for a new I.IJ decomposition.

It may happen that the matrix I$+ I is singular or that equation (4) is not possible with the permutation matrix Ip,. If the problem is caused by the use of Pi then we can abandon Bennett’s algorithm and do a full LU decomposition which will give us a new per- mutation matrix Pi+, . If Eli+, is singular the only options are not to use an lLU decomposition, or to modify Bi+ , so that it is non-singular.

If Bi is a matrix of rank m, then we can find a permutation matrix P,, an orthogonal matrix Q and an m x m upper triangular matrix I$ of rank m such that

Q,lT$P, = wi 0

[ 1 0 0’ The pseudo-inverse of lEIi is given by

IEq = P, R;’ 0 [ 1 0 0 62,.

Gill et al. [8] presented an algorithm for obtaining in O(n2) operations Pi+, , Ri+ , and Q,, , such that

QI+IBi+IPi+l= “ii 0

[ 1 0 .

As we can see the use of this QDB factorization avoids very nicely the problem found with the LU fac- torization. Unfortunately, in terms of storage, the QIW factorization is more expensive requiring $r2 storage locations as opposed to the n2 required for the LIJ decomposition. This difference becomes important when large numbers of equations are to be solved. Also, if the sparsity of the Jacobian is to be exploited in some way (e.g. by using the Schubert update), an I_U factorization is more appropriate. These reasons have led us to try a modification of Bennett’s algo- rithm which avoids the problems of singular Jacobian approximations.

The updating formula (1) represents a family of different quasi-Newton methods where each member of the family is obtained by a different choice of vector vi. The common property for all members of this fam- ily is that the approximations Eli+ 1 satisfy the secant relation

Bi+lPi=Yi* (5)

We will propose here a modification to Bennett’s algorithm which produces a non-singular matrix II,, , satisfying equation (5). In cases where the update (1)

and the original Bennett algorithm produce a non- singular matrix Bi+ ], our algorithm gives the same matrix. If equation (1) or the selected pivoting strategy gives a singular matrix our algorithm gives a matrix which is in some sense close to that matrix, but which is non-singular.

BENNETT’S ALGORITHM

Suppose we have an LU factorization of matrix A,

A=lLU

and a matrix B is obtained using the updating formula

B = A + abT, (6)

where a and b are two given vectors. If we are going to obtain the I_U factors of B using

a standard technique, without using the fact that I3 is obtained using equation (6), we need O(n’) oper- ations. Bennett [l] proposed an algorithm to obtain the LU factors of B satisfying equation (6) using just O(n2) operations. We will very briefly describe the algorithm.

Define the vectors

Ii = 0e,, i=l,2,...,n and

u:=efU, i=l,2 ,..., n.

We can express the matrix a as

A = i Iill:. (7) l-1

Assume that B, as given by equation (6), can be factorized as

B = lL*u* (8)

and define the vectors I,* and IIT such that

iB= i I:“:‘. (9) i-1

The following convention will be used:

E( )=0, ifp-zm. m

Define now the matrix k-l

Ck = c (&UT - 1: uFT) + abT. i-l

(10)

With this definition it follows that

C,+,+Itt$r=&+I tt* k k*

It is possible to show that

(11)

efCk=(Cke,)T=O, Vj<k.

Hence from equation (1 l), and since uzTek = u:ek = 1,

(12) and

IT=“: (e:b) : e:ck . uk

@D,*) @fW (13)

We will now show that Ck is a rank-one matrix.

Updating of LUJ factors in quasi-Newton methods 243

Lemma 1

The matrix Ck is such that

Ck=akb:, l<k<n. (14)

Proof. The proof is by induction on k.. Assume that Ck = akb:. From equation (11)

@ k+ I = 1,~: - ljt+ufT + a,bi, (15)

and from equation (12)

lt=&+a&ed. (16) hence

(e%!) = (e%) + (e:a,)(eZb& (17)

If we define (e:ak)

*“=m and

Bk = @Ze,),

then from equations (13) and (17)

ufT = uz(l - ak/YIk) + akb:. (18)

Hence, using equation (16)

1~~~~ = l&(1 - Q/&) + 1 bTa kkk

+ ak$bk(l- @kbk)+akbzak!k. (19)

Using equations (15) and (19)

ck+l =lku:crkBk-Ikb~ak-akU:Bk(l -akfik>

+ ak%(l - akflk)

= -IkCfk(--U;flk+ b;) + ak<-u:h + b:)

x (1 - akflk)

= [-lkak + ak(1 -akflk)l(-u~8k + b%

and then if we deline

ak+l= - lkak + ak(l - akflk)

and bT k+l= -U;&+b:

we can express Ck + , as

@k+t=ak+Ib:+I.

(20)

(21)

Since by taking a1 = a and bl = b,

C, =aLlbf,

it follows that equation (14) is true for 1 <k <n. We can then see that given a,, bk, lk and uk, we can

obtain ak + , and bk + , from equations (20) and (21), c and u$ from equations (16) and (18). Bennett’s algorithm consists of n stages, stage k calculating two n-vectors If and I$ ; B may then be calculated using equation (9).

PROPOSED MODIFICATION

In the special case of quasi-Newton methods, the vector b in equation (6) is varied to give different methods. In fact, varying b in equation (6), we can obtain a family of rank-one updates for quasi-

C.A.C.E. ,0,3--E

Newton methods. In general, all quasi-Newton methods satisfy the following secant relation:

BP=y, (22)

where p and y are two given vectors. Suppose we have obtained B for a particular

method and it is singular. Since B will later be used to solve a linear system, this particular method will clearly fail in this case. When this problem arises, we propose to modify B to Et such that

B’p=y. (23)

We will require B’ to be non-singular and also not too different from B.

To achieve this, we propose a modification to Bennett’s algorithm, to be added at stage k, the stage when singularity is detected.

The next theorem will show the only case when our proposed modification will not be applicable.

Theorem 2

If p and y are such that

and

0 Y' r1

(24)

(25) LYkJ

with pk being of order k and yk of order n - k and p # 0, y # 0, then it is not possible to find L and U, L non-singular, such that

L!Jp=y. (26)

Proof. Assume there exist L non-singular and U satisfying equation (26). Define the following partitions: L 0 L= I1

[ 1 %I b and

RJ= WI Q=,*

[ 1 0 UJ, ’

where I_,i and OJ ,i are of order k, L, and !Jr2 of order n - k, C2, is (n -k) x k and C,, is k x (n -k).

From equation (26) it follows that

and

Since Rll is non-singular then !_liipk = 0 and then yk = 0, which contradicts our hypothesis y # 0.

We will now show that it is possible to modify the algorithm to achieve what we want.

Define the matrices

Ak= ;kI,u:v

and

8, = i$k I+:‘.

We can see that Define now k-l

A = a, + 1 l,uT (ebb/) = @ - e:Akek) i= I e:ak

(35)

and k-l If 6 is such that 16 12 L it follows that inequality (33)

IEB=lB,+ 1 li*lq? (27) is true. - i=l We also require that I& satisfies equation (30) and

From equation (6) it follows that this is equivalent to asking that k-l k-l

Bk+ c &%,t’=Ak+ 1 l,u:+abT, biTp = b;p, (36) i-1 i-1 and since equation (24) is not true it is possible to find

and then from equations (10) and (14) at least one vector b; satisfying equations (35) and

B,=A,+a,bi. (28) (36). Then both required conditions are met for this case.

Suppose we detect a singularity in the matrix El at Case (b). Assume now that equation (24) is true, stage k of Bennett’s algorithm. We would like to we then have obtain a matrix El’ satisfying our conditions [non- singular and satisfying the secant relation (23)] with- P= i (eTP>ei.

i= I out having to alter what we have already obtained. We can achieve this if we can find a matrix 5; such The condition (23) becomes, if !Et’ is obtained using

that equation (29), k-l

&ek = i&&k

B’ = IEJ; + c I:@ (29) i=l

b%iUlSe A& = b;ej = bkej = 0, j < k. We can then

satisfies our conditions. In this case equation (23) see that inequality (33) cannot be satisfied because of equation (31).

becomes k-l

We have, in this case, to abandon equation (29)

wp=iB;p+ 1 l:lqTp=y, and define a new Et’ for which i=l k-l

and from equations (22) and (27) it follows that B’ = El; + c l;*uf*T. i-1

E&p = B,p. (30) Define as before (ezb;) with equation (35). It then

Hence equation (30) is equivalent to equation (23). follows if 16 I> E, and taking B; = A, + akbiT, that

Singularity at stage k means that e:lf = 0 and from equations (28) and (16) this is equivalent to

lez&ekl > 6. Assume now that

el Bkek = 0, hence to have B’ non-singular we require elB,e, # 0. u:‘p=O, 1 <i<k.

The following theorem will show that these two Then, from equation (27), conditions can be met.

BkP=Y

Theorem 3

If at stage k of Bennett’s algorithm we have for and since ef Etk = 0, if i < k, it follows that equation (25) is true, and this is a contradiction since equation

E > 0, leTl:l> c,

(24) is true. It then follows that for some q, 1 G q < k, i <k; ll?Tp # 0.

equations (24) and (25) are not true simultaneously We can then see that by taking uj* = uf, 1 =G i < k,

and l;* = Ii+, 1 < i < k, i # q and

ez&ek = 0 (31) and

I’* = 1+ + ak@k - WTp

cl q (37)

IeT&ekj 3 6. (32) (u:TP) ’

we will have B’ such that B’p = y and since s < k, Then there exist El; such that if !B’ is defined by equa- tion (29), then it satisfies equation (23), and le~l~*l=le,Tl:l>c;

le:&ekl 2 6. (33) and then B’ satisfies all the requested conditions.

By applying the above theorem to all stages of Proof: Case (a). Assume first that equation (24) is Bennett’s algorithm we will be left with a matrix El’

not true. We will show that it is possible to find b; which will satisfy the secant relation and which is such that

B; = Ak + akbiT non-singular.

(34)

satisfies the required conditions. From expressions (31) and (32) it follows that

eza,#O. .

IMPLEMENTATION

We see from the last section that it is possible to update LlJ factors achieving two of our objectives,

244 J. R. PALOSCHI and J. D. PERKINS

Updating of t_U factors in quasi-Newton methods 245

viz. to avoid singular Jacobian approximations while satisfying the secant relation. The necessary relations which the updating algorithm must satisfy are given in Theorem 3. In particular they are equations (35) and (36) in Case (a) and equations (35) and (37) in Case (b). A study of these conditions reveals that they define a whole family of algorithms. In this section we make suggestions of reasonable choices from the family. These choices minimize in some sense the cor- rection required to the original quasi-Newton update to remove the singularity.

In Case (a) of Theorem 3 we have that

B’-III=&-B,

and from equation (34)

II’ - If% = a,(& - bk)T.

It then follows that

IIB’-BII = IIa,It IlhL--h& (38)

The first component of the vector b; is uniquely defined by equation (35) but there is some freedom for the choice of the remaining components of b;.

We will make use of the following lemma due to Dennis and Schnabel [9].

Lemma 4

Let CL E R, v cR”, v # 0. Then the unique solution to

$2 II * II subject to vTx = a

is

x=cLv VTV’

If we define for a vector a~ R”, ii as the vector in which the first k elements are replaced by 0, it follows from equation (36) that

(eZb;)(e:p) + 6LTii = (elbd(eip) + hip

and we should choose b; satisfying

(6; - 6:)~ = el(b, - b;)e:p.

By applying Lemma 4, if we define

we will achieve the minimum value for II bi - bk II. It is possible to derive expressions for the relative

change in B necessary to achieve non-singularity. To do this, we note that

llb;-411”=[e~@;-bk)12+ IlY-hll’ and from equation (39)

ll6;_6,ll=le:(b~-b’~)lle:pl ll1ll

and then

(lb~-41~2=k:(b;-411z(l+~)

2 lIPlIZ = [er(b; - bdl JIPI~Z.

From equation (31) it follows that

ezAa,q + ezb,e:m, = 0

and, now using equation (35), it follows that

and then

161 IIPII Ilb;-bkll =mIIB(I

(40)

(41)

From equations (38) and (41)

IFBII_ Ilatll IIPII 161 IIBII 7i2miTl(IBII’

(42)

Equation (42) is an exact value for the relative change from B to B’ proportional to 6.

For case (b), we have that k-l

B’-B=5;-Bk+ 1 (l;*-l:)U:T. ,= I

Since in this case IEl; = A, + akbiT, k-l

E - B = 4(b; - b,)= + c (I;* - l:)UtT i-1

and since I;* = 17, if i # q,

US’ - 03 = ak(bi - bk)= + (li* - 1:)~:~.

Using now equation (37)

5’ - 5 = a&; - bk)= + akllp*T @k - Y )‘p

($‘P)

and then we can see that since

(see Paloschi [lo]),

11 B’ - B ll < ll a, 11 Ilb; - bk 11 ‘I ;$;i ‘I . ‘I

If we choose b; such that it only differs from bk in component k, then we can use equation (40), as in Case (a), to give

11 B’- B 11 ’ lezakl l1411gl IIPII IMI

lu;TpI ’

We should then choose q such that

min II ICq<k IU;=pl

and

u,l’p#O,

which gives the result

(43)

The cost of the proposed modification in Case (a) of Theorem 3 (which will be the most probable case in general) will be 2(n -k) operations, as can be de- duced from equation (39).

246 J. R. PALOSCHI and J. D. m

The application of Case (b) is more expensive if we want to solve equation (43). If not, the cost will just be the evaluation of equation (37). In our experience the application of Case (b) was found only in prob- lems where n = 2.

EXAMPLES OF USE

We will show with one example how our proposed modification improves the performance of quasi- Newton methods. Consider Broyden’s [3] good method [vi = pi in equation (l)]. We will take the function

f@)=x as the initial point

1 x0= [I 1 ’

and as the initial approximation to the Jacobian

-1 0 B,=

[ 1 0 1;

B0 is easily factorized as

Using Broyden’s method we obtain

po= -B;‘&= l -1

and then [ 1

2 x,=xiJ+pJ= Cl 0 *

From equation (1) it follows that

r&=8,+!& 0 -1

[ 1 0 1;

B, is singular and Broyden’s method will have to be aborted since we will not be able to solve equation (2) for i = 1.

If instead of the original Bennett’s algorithm, which will give us lE& as in equation (44), we apply our modification [Case (a)] of Theorem 3 for any L > 0, we will obtain

and then 4. I- * -I

xy2-z OJ,

1 E-l Bz=

[ 1 0 1 and

x, =

and then x, is the solution, obtained with any c > 0. This example shows how we can theoretically

improve the performance of quasi-Newton methods. We have also tested the proposed modification on

a large set of examples, which was also used by Chen

- Table I. Pcrfo mmux of modification in s&k precision

t

10-S 10-10 10-m 10-w 0

Perantagc of s-s 55 61 58 58 SO

Table 2. Performance of rc-initialization instead of using the mod&&ion kin&e utision)

c

10-J 10-10 10-m 10-m 0

Percentage of s- 52 58 58 58 SO

and Stadtherr [5] and Paloschi and Perkins [6]. It consists of a set of mathematical problems which is increased by considering the original scale, by forcing the variables to be badly scaled and also doing it with the equations.

We present in Table 1 the performance of the pro- posed modification using Broyden’s method in terms of percentage of success. We have used different values of L in equation (33), c = 0 corresponds to no modification. All results were obtained in single pre- cision on a VAX 780 under a VMS operating system. It can be seen that the proposed modification appears to be quite effective in terms of robustness (there was no difference in terms of efficiency). The optimum value for E seems to be around lo-” and is machine and precision dependent. We have also tested re- initialization of the Jacobian, as an alternative to the proposed modification, and the results can be found in Table 2. By comparison with Table 1 we can see that the proposed modification has better per- formance than reinitializing in terms of robustness. In terms of efficiency, the modification was slower in seven cases and faster in four cases.

1.

2.

3.

5.

6.

I.

8.

REFERENCES

J. M. Bennett, Triangular factors of modified matrices. Numer. Mar/t. 7, 217 (1965). L. T. Biegler, Simultaneous-modular simulation and optimisation. Paper presented at FOCAPD-83, Snow- mass, Colo. (19-24 June 1983). J. D. Perkins, Equation-oriented flowsheeting. Paper presented at FOCAPD-83, Snowmass, Colo. (1924 June 1983). C. G. Broyden, A class of methods for solving simnlta- neous nonlinear equations. Maths Comput. 19, 517 (1965). H. S. Chen and M. A. Stadtberr, A modification of Powell’s dogleg algorithm for solving systems of non- linear equations. Comput. chum. &gng 5, 143 (1981). J. R. Palo&i and J. D. Perkins, Robustness of non- linear equation solvers for flowsheeting packages. Paper presented at the IChemE A. Research Mtg, London (April 1982). J. Sherman and W. J. Morrison, Adjustment of an in- verse matrix corresponding to change in the elements of a given column or a given row of the original matrix. Ann. marh. Starist. 20, 621 (1949). P. E. Gill, 0. H. Golub, W. Murray and M. A. Saunders, methods for modifying matrix factorizations. Maths Comput. 2% 505 (1974).

Updating of LU factors in quasi-Newton methods 247

9. J. E. Dennis and R. B. Schnabel, Least change secant 10. J. R. Paloschi, The numerical solution of nonlinear updates for quasi-Newton methods. SIAM Reu. 21,443 equations representing chemical processes. Ph.D. (1979). Thesis, Univ. of London (1982).