Exercise 7: Newton’s method

CSElab Computational Science & Engineering Laboratory

CSElab www.cse-lab.ethz.chExercise 7 - Solving non-linear equations

• Learn about finding approximate numerical solutions to non-linear equations

• Newton-Raphson method: Simple, intuitive way of finding roots of non-linear equations

• Uses the first few terms of the Taylor-series expansion of a function in the vicinity of an ‘initial guess’

• Iterate numerically, until the correction |x(k) - x(k-1)| drops below threshold, or the maximum number of iterations is reached.

CSElab www.cse-lab.ethz.ch

The rationale behind Newton’s:

• The Taylor series of f(x) about the point x+Δx is given by

• Keeping only the first order terms, we get

• This equation forms the basis of the iterative algorithm presented in the lecture notes

• We iterate until the correction |x(k) - x(k-1)|drops below a threshold value

Theory: Newton’s method

f(x+�x) = f(x) + f

0(x)�x+ f

00(x)�x

2

2!+ f

000(x)�x

3

3!+ · · ·

f(x+�x) ⇡ f(x) + f

0(x)�x

4.5. Newton’s Method 7

f(x)

x

x*

x(0)x(1)

f(x(0))

x(2)θ

tan(�) =f(x(0))

x(0) � x(1)= f �(x(0))

) x(0) � x(1) =f(x(0))

f �(x(0))

) x(1) = x(0) � f(x(0))

f �(x(0))

Figure 4.3: Graphical example of Newton iterations as we follow the tangents of f(x).

4.5 Newton’s Method

In a linear problem the equivalent of the linear problem Ax � b = 0 is f(x) = 0 as in Eq. (4.2.1)

(or ~F (~x) = ~0 for a system of equations). We expect to solve f(x) = 0 by starting from a guess

x(k) and improving it by computing the next guess x(k+1).

Everything in a sense depends on what is known of x(k). Normally, we can calculate f(x(k)) and

see how close it is to 0. Can we estimate f(x(k+1)) from f(x(k)) too? Let’s use the first terms of

a Taylor series:

f(x(k+1)) = f(x(k)) + f 0(x(k))(x(k+1) � x(k)). (4.5.1)

Note that we dropped all higher terms from the Taylor series. We can now solve f(x(k+1)) = 0

in Eq. (4.5.1) and get that:

f 0(x(k))(x(k+1) � x(k)) = �f(x(k)) (4.5.2)

) x(k+1) = x(k) � f(x(k))

f 0(x(k))(4.5.3)

Graphically the procedure is visualized in Fig. 4.3. Newton’s method approximates the non-linear

function f near x(k) by the tangent line at f(x(k)).

Convergence for simple roots is quadratic. For roots of multiplicity m > 1, the convergence is

linear unless we modify the iteration (Eq. (4.5.3)) to x(k+1) = x(k) � mf(x(k))/f 0(x(k)).

4.5.1 Problems with Newton’s Method

Newton’s method requires evaluation of both function and derivative at each iteration.

Evaluating the derivative may be expensive or inconvenient so we replace this with a finite

di↵erence approximation.

CSElab www.cse-lab.ethz.chIntuition: Newton’s method

The algorithm intuitively:

Source: http://en.wikipedia.org/wiki/File:NewtonIteration_Ani.gif

CSElab www.cse-lab.ethz.chCaveats

• The tangent lines of x3 - 2x + 2 at 0 and 1 intersect the x-axis at 1 and 0.

• Newton’s method can oscillate indefinitely between these two values.

• Might diverge with a poorly chosen initial guess Source: Wikipedia

Source: http://qucs.sourceforge.net/tech/node16.html

Initial guess is important!

CSElab www.cse-lab.ethz.chTheory: Newton’s method for a system

CSElab www.cse-lab.ethz.chAlgorithm

10 Chapter 4. Solving Non-Linear Equations

which gives the same solution as when simply solving the system A~x = ~b.

To sum up: we start with an initial approximation ~x(0) and iteratively improve the approximation

by computing new approximations ~x(k) as in Algorithm 2.

Algorithm 2 Newton’s methodInput:

~x(0), {vector of length N with initial approximation}tol, {tolerance: stop if k~x(k) � ~x(k�1)k < tol}kmax

, {maximal number of iterations: stop if k > kmax

}Output:

~x(k), {solution of ~F (~x(k)) = ~0 within tolerance tol} (or a message if k > kmax

reached)

Steps:

k 1

while k kmax

do

Calculate ~F (~x(k�1)) and N ⇥N matrix J(~x(k�1))

Solve the N ⇥N linear system J(~x(k�1))~y = �~F (~x(k�1))

~x(k) ~x(k�1) + ~y

if k~yk < tol then

break

end if

k k + 1

end while

Computational Cost

For large N and dense Jacobians J , the cost of Newton’s method can be substantial. It costs

O(N2) to build the matrix J and solving the linear system in Eq. (4.5.10) costs O(N3) operations

(if J is a dense matrix).

Secant Updating Method

Reduce the cost of Newton’s Method by:

• using function values at successive iterations to build approximate Jacobians and avoid

explicit evaluations of the derivatives (note that this is also necessary if for some reason

you cannot evaluate the derivatives analytically)

• update factorization (to solve the linear system) of approximate Jacobians rather than

refactoring it each iteration

You must not take such shortcuts in the exam! Make

sure to provide complete steps.

For instance, you could write 3 explicit functions (with all loops and computations shown) for computing F, computing J, and

solving the system, which can then be called from the main body.

CSElab www.cse-lab.ethz.chExample I

Evaluate x(2) starting with x(0)=0 for the following system:

• The Jacobian:

• Initial guess:

• 1st iteration:

5x

21 � x

22 = 0

x2 � 0.25(sinx1 + cosx2) = 0

x

(0)1 = x

(0)2 = 0

�F (~x

0) = �

5 ⇤ 02 � 0

2

0� 0.25(sin(0) + cos(0))

�=

0

0.25

�

J(~x0)~y = �F (~x0)

)

0 0�0.25 1

�~y =

0

0.25

�

Note: y is the change in x.

J =

2

4@f1

x1

@f1

x2

@f2

x1

@f2

x1

3

5=

10x1 �2x2

�0.25 cosx1 1 + 0.25 sinx2

�J =

2

4@f1

@x1

@f1

@x2

@f2

@x1

@f2

@x2

3

5

y1 = 0, y2 = 0.25

CSElab www.cse-lab.ethz.ch

• Update the current guess:

• 2nd iteration:

�F (~x

1) = �

5 ⇤ 02 � 0.25

2

0.25� 0.25(sin(0) + cos(0.25))

�=

0.0625

�0.00777

�

y1 = 0, y2 = 0.25

x

(2)1 = x

(1)1 + y1

x

(2)2 = x

(1)2 + y2

) x

(1)1 = x

(0)1 + y1 = 0

x

(1)2 = x

(0)2 + y2 = 0.25

J(~x

1)~y = �F (~x

1)

)

0 �0.2

�0.25 1 + 0.24223 cosx2

�~y =

0.0625

�0.00777

�

Note the decrease in y from the previous iteration. It indicates that we are converging towards a solution.

Example II

CSElab www.cse-lab.ethz.chQuestion 1

• formulate the problem as and find the root of the function

• initial guess: use e.g.

CSElab www.cse-lab.ethz.ch

rp

free surface

d

D

soft soil

hard soil

Question 2 I

Pressure p to sink circular plate of radius r to a certain depth d in the soft soil:

with the unknown coefficients k1, k2, k3.

f1(~k) = 100� k1ek2·0.1 + k30.1

(a)p(r) = k1e

k2r + k3r

Formulate the system of equations, using the given data points: • For the first data point r=0.1 m, p=100 N/m2, the equation is

• For the second data point r=0.2 m, p=120 N/m2, the equation is …

Source: http://forum.unity3d.com/threads/what-type-of-joint-should-i-use-for-a-pressure-plate.126858/

CSElab www.cse-lab.ethz.chQuestion 2 II

Implement Newton’s method. Initial guess: Use e.g. k = (80,10,1).

(b)

Modify the code from question b to solve for r:

with k1, k2, k3 computed from (b) and the critical pressure.

(c)f(r) = k1e

k2r + k3r � pcrit = 0

Find the intersection between p(r) and pcrit(r) using a plotting tool.(d)

pcrit(r) =Load

Area(r)