This article was downloaded by: [Dicle University]On: 11 November 2014, At: 06:31Publisher: Taylor & FrancisInforma Ltd Registered in England and Wales Registered Number: 1072954 Registeredoffice: Mortimer House, 37-41 Mortimer Street, London W1T 3JH, UK

International Journal of ComputerMathematicsPublication details, including instructions for authors andsubscription information:http://www.tandfonline.com/loi/gcom20

Parallel iteratively regularizedGauss–Newton method for systems ofnonlinear ill-posed equationsPham Ky Anha & Vu Tien Dzunga

a Department of Mathematics, Vietnam National University, 334Nguyen Trai, Thanh Xuan, Hanoi, VietnamAccepted author version posted online: 19 Mar 2013.Publishedonline: 15 Apr 2013.

To cite this article: Pham Ky Anh & Vu Tien Dzung (2013) Parallel iteratively regularizedGauss–Newton method for systems of nonlinear ill-posed equations, International Journal ofComputer Mathematics, 90:11, 2452-2461, DOI: 10.1080/00207160.2013.782399

To link to this article: http://dx.doi.org/10.1080/00207160.2013.782399

PLEASE SCROLL DOWN FOR ARTICLE

Taylor & Francis makes every effort to ensure the accuracy of all the information (the“Content”) contained in the publications on our platform. However, Taylor & Francis,our agents, and our licensors make no representations or warranties whatsoever as tothe accuracy, completeness, or suitability for any purpose of the Content. Any opinionsand views expressed in this publication are the opinions and views of the authors,and are not the views of or endorsed by Taylor & Francis. The accuracy of the Contentshould not be relied upon and should be independently verified with primary sourcesof information. Taylor and Francis shall not be liable for any losses, actions, claims,proceedings, demands, costs, expenses, damages, and other liabilities whatsoever orhowsoever caused arising directly or indirectly in connection with, in relation to or arisingout of the use of the Content.

This article may be used for research, teaching, and private study purposes. Anysubstantial or systematic reproduction, redistribution, reselling, loan, sub-licensing,systematic supply, or distribution in any form to anyone is expressly forbidden. Terms &Conditions of access and use can be found at http://www.tandfonline.com/page/terms-and-conditions

International Journal of Computer Mathematics, 2013Vol. 90, No. 11, 2452–2461, http://dx.doi.org/10.1080/00207160.2013.782399

Parallel iteratively regularized Gauss–Newton method forsystems of nonlinear ill-posed equations

Pham Ky Anh and Vu Tien Dzung*

Department of Mathematics, Vietnam National University, 334 Nguyen Trai, Thanh Xuan, Hanoi, Vietnam

(Received 21 May 2012; revised version received 16 January 2013; accepted 26 February 2013)

We propose a parallel version of the iteratively regularized Gauss–Newton method for solving a system ofill-posed equations. Under certain widely used assumptions, the convergence rate of the parallel method isestablished. Numerical experiments show that the parallel iteratively regularized Gauss–Newton method iscomputationally convenient for dealing with underdetermined systems of nonlinear equations on parallelcomputers, especially when the number of unknowns is much larger than that of equations.

Keywords: ill-posed problem; IRGNM; parallel computation; componentwise source condition; undeter-mined system

2010 AMS Subject Classifications: 47J06; 47J25; 65J15; 65Y05

1. Introduction

Many parameter identification problems lead to a system of operator equations

Fi(x) = yi, 1 ≤ i ≤ N , (1)

where Fi, 1 ≤ i ≤ N , are possibly nonlinear operators mapping a Hilbert space X of unknownparameter x into Hilbert spaces Yi of observations yi. In the case of noisy data yδ

i with ‖yδi − y‖Yi ≤

δ, we have the perturbed system

Fi(x) = yδi , 1 ≤ i ≤ N . (2)

Clearly, systems (1) and (2) can be rewritten as operator equations in the product space

F(x) = y, (3)

and

F(x) = yδ , (4)

where F : X → Y = Y1 × Y2 × · · · × YN , F(x) = (F1(x), . . . , FN (x)), y = (y1, . . . , yN ) and yδ =(yδ

1, . . . , yδN ). For u = (u1, . . . , uN ) ∈ Y and v = (v1, . . . , vN ) ∈ Y , the inner product and the norm

in Y are defined as < u, v >= ∑Ni=1 < ui, vi >Yi and ‖u‖Y = (

∑Ni=1 ‖ui‖2

Yi)1/2, respectively.

*Corresponding author. Email: anhpk@vnu.edu.vn

© 2013 Taylor & Francis

Dow

nloa

ded

by [

Dic

le U

nive

rsity

] at

06:

31 1

1 N

ovem

ber

2014

International Journal of Computer Mathematics 2453

One of the most efficient regularization methods for nonlinear ill-posed problems is the iter-atively regularized Gauss–Newton method (IRGNM), proposed by Bakushinskii [4] in 1992.Convergence results of the IRGNM were obtained by Blaschke et al. [5], Hohage [10], Deuflhardet al. [7], Jin et al. [11] and others, see [9,12,14].

Recently, an IRGN–Kaczmarz method has been introduced by Burger and Kaltenbacher [6].The main idea of the last method is to perform a cyclic IRGN iteration over the equations.However, when the number of equations N is large, the Kaczmarz-like methods are costly on asingle processor.

In this note, we propose a parallel version of the IRGNM for the system of ill-posed operatorEquation (1). Other parallel methods for solving systems of ill-posed equations can be found in[1–3].

Suppose that the exact system (1) has a solution x†, which may not depend continuously onthe right-hand side y. Suppose that the operators Fi, 1 ≤ i ≤ N , are continuously differentiablein some set containing x† and x0– an initial approximation of x†. Let xδ

n be the nth approximationof x†. According to the IRGNM, for a fixed number n, we linearize the Tikhonov functional

Jδn(x) := ‖F(x) − yδ‖2

Y + αn‖x − x0‖2 =N∑

i=1

‖Fi(x) − yδi ‖2

Yi+ αn‖x − x0‖2

about xδn, and consider the unconstrained optimization problem

�δn(�x) :=

N∑i=1

‖Fi(xδn) − yδ

i + F ′i(x

δn)�x‖2

Yi+ αn‖xδ

n − x0 + �x‖2 → min�x∈X

,

where F ′i(x

δn) stands for the Frechet derivative of Fi(x) computed at xδ

n.Finding �x from the equation ∂�δ

n/∂(�x) = 0, we determine the next approximation as xδn+1 =

xδn + �x, or

xδn+1 = xδ

n −(

N∑i=1

F ′i(x

δn)

∗F ′i(x

δn) + αnI

)−1 (N∑

i=1

F ′i(x

δn)

∗(Fi(xδn) − yδ

i ) + αn(xδn − x0)

). (5)

However, in some cases, it is much more computationally convenient to apply the IRGNM toeach subproblem (2) synchronously, i.e. to find in parallel

xδn+1,i = xδ

n − (F ′i(x

δn)

∗F ′i(x

δn) + βnI)−1(F ′

i(xδn)

∗(Fi(xδn) − yδ

i ) + βn(xδn − x0

i )), (6)

i = 1, . . . , N , where βn := αn/N , and then define the next approximation as an average of theintermediate approximations xn+1,i, i.e.

xδn+1 = 1

N

N∑i=1

xδn+1,i. (7)

Although each step (6) consists exactly of one iterate of the IRGNM applied to subproblem (2),the convergence of the ordinary IRGNM (5) does not necessarily imply the convergence of theparallel iteratively regularized Gauss–Newton method (PIRGNM) (6), (7).

In the next section, we will study the stopping rule and the convergence of the PIRGNM (6), (7).

Dow

nloa

ded

by [

Dic

le U

nive

rsity

] at

06:

31 1

1 N

ovem

ber

2014

2454 P.K. Anh and V.T. Dzung

2. Convergence analysis

For convenience of the reader, we collect some facts, necessary for deriving an error estimateof approximate solutions. We begin with a particular case of Lemma 2.4 [5], whose proof isstraight-forward.

Lemma 2.1 Let {γn} be a sequence of nonnegative numbers satisfying the relations

γn+1 ≤ a + bγn + cγ 2n , n ≥ 0 for some a, b, c > 0.

Let M+ := (1 − b + √(1 − b)2 − 4ac)/2c, M− := (1 − b − √

(1 − b)2 − 4ac)/2c. If b +2√

ac < 1 and γ0 ≤ M+, then γn ≤ l := max{γ0, M−} for all n ≥ 0.

Proof Clearly, γ0 ≤ l. Suppose γk ≤ l, then γk+1 − l ≤ a + bγk + cγ 2k − l ≤ a + (b − 1)l +

cl2 ≤ 0 because l ∈ [M−, M+], hence γk+1 ≤ l. It follows by induction that γn ≤ l for n ≥ 0. �

Lemma 2.2 Let A be a bounded linear operator on a Hilbert space H. Then, for every β > 0,the following estimates hold.

(i) β1−μ‖(A∗A + βI)−1(A∗A)μv‖ ≤ μμ(1 − μ)1−μ‖v‖ ≤ ‖v‖, for any fixed μ ∈ (0, 1] and v ∈H .

(ii) ‖(A∗A + βI)−1‖ ≤ 1/β.(iii) ‖(A∗A + βI)−1A∗‖ ≤ 1

2β−1/2.(iv) ‖A(A∗A + βI)−1(A∗A)1/2‖ ≤ 1.

The proof of the estimates in Lemma 2.2 can be found in [12, pp. 72, 81, 82].In what follows, the parameters αn are chosen such that

αn > 0, αn → 0 and 1 ≤ αn

αn+1≤ ρ for some ρ > 1.

Let Br(x0) denote a closed ball centred at x0 and with radius r > 0 in X. Before stating aconvergence theorem, we make some widely used assumptions.

Assumption 2.1 System (1) has an exact solution x† ∈ Br(x0) and Fi, i = 1, 2, . . . , N , arecontinuously differentiable in B2r(x0).

Assumption 2.2 The following componentwise source condition (cf. [6, p. 8]) holds

x† − x0i = (F ′

i(x†)∗F ′

i(x†))μvi, (8)

where 0 < μ ≤ 1 and x0i ∈ B2r(x0), vi ∈ X , 1 ≤ i ≤ N. Moreover, suppose that

(i) If 0 < μ ≤ 12 , then Fi, i = 1, 2, . . . , N , satisfy the following condition (see [11,14]), ∀x, z ∈

B2r(x0); ∀v ∈ X, ∃hi(x, z, v) ∈ X

(F ′i(x) − F ′

i(z))v = F ′i(z)hi(x, z, v); ‖hi(x, z, v)‖ ≤ K0‖x − z‖‖v‖, (9)

(ii) If 12 < μ ≤ 1, then F ′

i are Lipschitz continuous, i.e.

‖F ′i(x) − F ′

i(x̃)‖ ≤ L‖x − x̃‖, 1 ≤ i ≤ N (10)

for all x, x̃ ∈ B2r(x0).

Dow

nloa

ded

by [

Dic

le U

nive

rsity

] at

06:

31 1

1 N

ovem

ber

2014

International Journal of Computer Mathematics 2455

The assumption (8) is rather restricting and it requires the choice of appropriate initial guessesx0

i , i = 1, . . . , N . Further, since the vectors vi in Equation (8) do not occur in the iterationprocess (6) and (7), they need not to be known explicitly.

Define the stopping index Nδ in the PIRGNM (6) and (7) as the first number n satisfying thecondition ηβ

μ+1/2n ≤ δ, i.e.

ηβμ+1/2Nδ

≤ δ < ηβμ+1/2n , 0 ≤ n < Nδ , (11)

where βn = αn/N and η > 0 is a fixed parameter. This stopping rule is an a priori one and has onlya theoretical meaning, since it depends on μ which is not often available in practice. However,presently an a posteriori stopping rule for the PIRGNM has not been established yet.

The proposed here parallel algorithm consists of the following steps:

(1) Give an initial approximation xδ0 and set n := 0.

(2) Compute in parallel the ith vectors xδn+1,i, 1 ≤ i ≤ N by Equation (6), where the given initial

guesses x0i , i = 1, . . . , N , are associated with the componentwise source condition (8).

(3) Define xδn+1 by Equation (7).

(4) if n > Nδ , where Nδ is the stopping index defined by Equation (11), then stop. Else putn := n + 1 and return to Step 2.

Theorem 2.3 Let the assumptions 2.1 and 2.2 hold and let the stopping index n∗ = Nδ be chosenaccording to Equation (11). If

∑Ni=1 ‖vi‖ and η are sufficiently small and xδ

0 = x0 is close enoughto x†, then there holds the estimate

‖xδn∗ − x†‖ = O(δ2μ/(2μ+1)). (12)

Proof We follow the techniques used in [5,6,11,12] to estimate the distance between xδn and x†.

Let xδn ∈ Br(x†) and denote Ai := F ′

i(x†), Ain := F ′

i(xδn); en := xδ

n − x† and ein+1 := xδ

n+1,i − x†.From Equation (6), we get ei

n+1 = en − (A∗inAin + βnI)−1(A∗

in(Fi(xδn) − yδ

i ) + βn(xδn − x0

i )), or

ein+1 = (A∗

inAin + βnI)−1[βn(x0i − x†) + A∗

in(yδi − yi) − A∗

in(Fi(xδn) − yi − Ainen)]. (13)

Depending on the value of μ, we consider two cases.

Case 1 Let μ ∈ ( 12 , 1]. Using the source condition (8) and taking into account

the identity (A∗i Ai + βnI)−1 − (A∗

inAin + βnI)−1 = −(A∗inAin + βnI)−1[(A∗

i − A∗in)Ai + A∗

in(Ai −Ain)](A∗

i Ai + βnI)−1, we can rewrite ein+1 as

ein+1 = −βn(A

∗i Ai + βnI)−1(A∗

i Ai)μvi

+ βn(A∗inAin + βnI)−1[A∗

in(Ai − Ain) + (A∗i − A∗

in)Ai](A∗i Ai + βnI)−1(A∗

i Ai)μvi

− (A∗inAin + βnI)−1A∗

in(Fi(xδn) − yi − Ainen) + (A∗

inAin + βnI)−1A∗in(y

δi − yi). (14)

According to Lemma 2.2, we have ωni(μ) := β1−μn ‖(A∗

i Ai + βnI)−1(A∗i Ai)

μvi‖ ≤ μμ(1 −μ)(1−μ)‖vi‖ ≤ ‖vi‖ for μ ∈ (0, 1]; ‖(A∗

inAin + βnI)−1‖ ≤ 1βn

, ‖(A∗inAin + βnI)−1A∗

in‖ ≤ 12β

−1/2n ;

‖Ai(A∗i Ai + βnI)−1(A∗

i Ai)1/2‖ ≤ 1 and ‖Ain − Ai‖ = ‖F ′

i(xδn) − F ′

i(x†)‖ ≤ L‖en‖, hence

Dow

nloa

ded

by [

Dic

le U

nive

rsity

] at

06:

31 1

1 N

ovem

ber

2014

2456 P.K. Anh and V.T. Dzung

‖Fi(xδn) − yi − Ainen‖Yi = ‖Fi(xδ

n) − Fi(x†) − F ′i(x

δn)en‖Yi ≤ 1

2 L‖en‖2, therefore

‖(A∗inAin + βnI)−1A∗

in(Fi(xδn) − yi − Ainen)‖Yi ≤ 1

2β−1/2n ( 1

2 L‖en‖2). (15)

Further,

T1 := βn‖(A∗inAin + βnI)−1)[A∗

in(Ai − Ain) + (A∗i − A∗

in)Ai](A∗i Ai + βnI)−1(A∗

i Ai)μvi‖

≤ βn‖(A∗inAin + βnI)−1A∗

in‖‖Ai − Ai,n‖‖(A∗i Ai + βnI)−1(A∗

i Ai)μvi‖

+ βn‖(A∗inAin + βnI)−1‖‖A∗

i − A∗in‖‖Ai(A

∗i Ai + βnI)−1(A∗

i Ai)12 )‖‖(A∗

i Ai)μ− 1

2 vi‖.

Thus,

T1 ≤ L‖en‖( 12βμ−1/2

n ωni(μ) + ‖(A∗i Ai)

μ−1/2vi‖). (16)

Besides,

‖(A∗inAin + βnI)−1A∗

in(yδi − yi)‖ ≤ 1

2β−1/2n δ. (17)

Finally,

βn‖(A∗i Ai + βnI)−1(A∗

i Ai)μvi‖ = βμ

n ωni(μ). (18)

Combining relations (13)–(18), we find

‖ein+1‖ ≤ βμ

n ωni(μ) + L‖en‖( 12βμ−1/2

n ωni(μ) + ‖(A∗i Ai)

μ−1/2vi‖) + 12β−1/2

n ( 12 L‖en‖2 + δ).

This and together with Equation (7) yields the estimate

‖en+1‖ = 1

N

∥∥∥∥∥N∑

i=1

ein+1

∥∥∥∥∥ ≤ 1

N

N∑i=1

(βμ

n ωni(μ) + L‖en‖(

1

2βμ−1/2

n ωni(μ) + ‖(A∗i Ai)

μ−1/2‖))

+ 1

2β−1/2

n

(1

2L‖en‖2 + δ

).

Now introducing the sequence γn := ‖en‖/βμn and observing that the stopping rule (11) implies

δ < ηβμ+1/2n for 0 ≤ n < Nδ , from the last inequality, we have

γn+1 ≤ 1

N

N∑i=1

(βn

βn+1

)μ

ωni(μ) + L

2N

‖en‖β

μn

(βn

βn+1

)μ

βμ−1/2n

N∑i=1

ωni(μ)

+ L

N

‖en‖β

μn

(βn

βn+1

)μ N∑i=1

‖(A∗i Ai)

μ−1/2vi‖ + L

4βμ−1/2

n

(‖en‖β

μn

)2 (βn

βn+1

)μ

+ η

2

(βn

βn+1

)μ

≤ 1

Nρμ

N∑i=1

ωni(μ) + ρμ η

2+ Lγn

2Nρμβ

μ−1/20

N∑i=1

ωni(μ)

+ Lγn

Nρμ

N∑i=1

‖(A∗i Ai)

μ−1/2vi‖ + L

4β

μ−1/20 ρμγ 2

n

≤ ρμ

(1

N

N∑i=1

‖vi‖ + η

2

)+ Lρμ

(1

2Nβ

μ−1/20

N∑i=1

‖vi‖ + 1

N

N∑i=1

‖(A∗i Ai)

μ− 12 vi‖

)γn

+ L

4β

μ−1/20 ρμγ 2

n .

Here, we use the inequality ωni(μ) ≤ ‖vi‖. Besides, the above-defined constant ρ satisfies theinequality ρ ≥ αn/αn+1 ≥ 1.

Dow

nloa

ded

by [

Dic

le U

nive

rsity

] at

06:

31 1

1 N

ovem

ber

2014

International Journal of Computer Mathematics 2457

Thus,

γn+1 ≤ a + bγn + cγ 2n , (19)

where

a =(

1

N

N∑i=1

‖vi‖ + η

2

)ρμ; b =

(1

2Nβ

μ−1/20

N∑i=1

‖vi‖ + 1

N

N∑i=1

‖(A∗i Ai)

μ−1/2vi‖)

ρμ

and c = L

4β

μ−1/20 ρμ.

If∑N

i=1 ‖vi‖ and η are small enough, then a and b will be small, hence b + 2√

ac ≤ 1, and

2aβμ0 ≤ r(1 − b +

√(1 − b)2 − 4ac). (20)

Now if x0 is sufficiently close to x†, then γ0 = β−μ0 ‖xδ

0 − x†‖ = β−μ0 ‖x0 − x†‖ ≤ M+ :=

(1 − b + √(1 − b)2 − 4ac)/2c. Lemma 2.1 applied to the inequality (19) ensures that γn :=

‖en‖/βμn ≤ l := max{γ0; M−} for 0 ≤ n ≤ Nδ , where M− = (1 − b − √

(1 − b)2 − 4ac)/2c =2a/(1 − b + √

(1 − b)2 − 4ac). In particular, ‖xδn+1 − x†‖ = ‖en+1‖ = γn+1β

μn+1 ≤ lβμ

0 .Observe that γ0β

μ0 = ‖x0 − x†‖ ≤ r. From Equation (20), we find M−β

μ0 = 2aβ

μ0 /(1 − b+√

(1 − b)2 − 4ac) ≤ r, therefore, lβμ0 ≤ r, hence xδ

n+1 ∈ Br(x†).Thus, for the case 1

2 < μ ≤ 1, the estimate γn ≤ l yields ‖en‖ ≤ lβμn = lαμ

n /Nμ = O(αμn )for

0 ≤ n ≤ n∗ := Nδ .

Case 2 Let μ ∈ (0, 12 ] and condition (9) hold. First observe that Fi(xδ

n) − yi − F ′i(x

δn)(x

δn −

x†) = ∫ 10 (F ′

i(x† + t(xδ

n − x†)) − F ′i(x

δn))(x

δn − x†) dt = ∫ 1

0 F ′i(x

δn)h

itdt = F ′

i(xδn)

∫ 10 hi

tdt, where

hit := hi(x† + t(xδ

n − x†), xδn, xδ

n − x†) and ‖ ∫ 10 hi

tdt‖Yi ≤ K0/2‖xδn − x†‖2.

From Equation (13), we find

‖ein+1‖ ≤ βn‖(A∗

inAin + βnI)−1(x0 − x†)‖ + ‖(A∗inAin + βnI)−1A∗

in(yδi − yi)‖

+ ‖(A∗inAin + βnI)−1A∗

inAin‖K0

2‖xδ

n − x†‖2.

Thus,

‖ein+1‖ ≤ βn‖(A∗

inAin + βnI)−1(x0 − x†)‖ + δ

2β−1/2

n + K0

2‖xδ

n − x†‖2.

This and together with the source condition (8) and the estimate

βn‖(A∗inAin + βnI)−1 − (A∗

i Ai + βnI)−1‖ ≤ 2K0‖xδn − x†‖

(see [11, Lemma 4.2, p. 1613]), gives

‖ein+1‖ ≤ βn‖(A∗

i Ai + βnI)−1(A∗i Ai)

μvi‖ + 2K0‖xδn − x†‖‖x0 − x†‖ + δ

2β−1/2

n

+ K0

2‖xδ

n − x†‖2 ≤ βμn ωni(μ) + 2K0‖en‖‖(A∗

i Ai)μvi‖ + δ

2β−1/2

n + K0

2‖en‖2.

Dow

nloa

ded

by [

Dic

le U

nive

rsity

] at

06:

31 1

1 N

ovem

ber

2014

2458 P.K. Anh and V.T. Dzung

Setting γn := ‖en‖/βμn , from the last relations, we find

γn+1 = ‖en+1‖β

μn+1

≤ 1

N

N∑i=1

‖ein+1‖

βμn+1

≤ 1

N

N∑i=1

(βn

βn+1

)μ

ωni(μ)

+ 2K0

N

‖en‖β

μn

(βn

βn+1

)μ N∑i=1

‖(A∗i Ai)

μvi‖

+ δ

2β−1/2

n .1

βμn+1

+ K0

2

(‖en‖β

μn

)2β2μ

n

βμn+1

.

The stopping rule (11) ensures that for 0 ≤ n < Nδ ,

γn+1 ≤ 1

Nρμ

N∑i=1

ωni(μ) + 2K0

Nρμγn

n∑i=1

‖(A∗i Ai)

μvi‖ + η

2β−1/2

n βμ+1/2n

1

βμn+1

+ K0

2γ 2

n ρμβμ0 .

Thus, γn+1 ≤ a + bγn + cγ 2n , where a = ρμ((1/N)

∑Ni=1 ‖vi‖ + η/2); b = (2K0ρ

μ/N)∑N

i=1‖(A∗i Ai)

μvi‖, and c = (K0/2)ρμβμ0 .

Again, if∑N

i=1 ‖vi‖ and η are sufficiently small and xδ0 = x0 is close enough to x†, then arguing

similarly as in Case 1, we can show that xδn+1 ∈ Br(x†) and ‖xδ

n − x†‖ = O(αμn ) for 0 ≤ n ≤ Nδ .

Thus, in both cases for 0 ≤ n ≤ Nδ , we have

‖xδn − x†‖ = O(αμ

n ).

Let n = n∗ := Nδ , then ηβμ+1/2n∗ = η(αn∗/N)μ+1/2 ≤ δ, hence, αμ

n∗ ≤ Nμ(δ/η)μ/(μ+1/2), therefore‖xδ

n∗ − x†‖ = O(δ2μ/(2μ+1)). Theorem 2.3 is proved. �

3. Numerical experiments

Underdetermined systems of equations arise in a variety of problems, such as, nonlinear comple-mentarity problems, problems of finding interior points of polytopes, image processing, etc.

We consider a simultaneous underdetermined system of nonlinear equations

Fi(x1, . . . , xm) = yi, i = 1, . . . , N , (21)

where Fi : Rm → R and m � N .

First we rewrite Equation (6) as

xδn+1,i = x0

i + (F ′i(x

δn)

∗F ′i(x

δn) + βnI)−1F ′

i(xδn)

∗(yδi − Fi(x

δn) − F ′

i(xδn)(x

0i − xδ

n)). (22)

Here, F ′i(x) = (∂Fi/∂x1, . . . , ∂Fi/∂xm); i = 1, . . . , N are row vectors.

Further, noting that (F ′i(x

δn)

∗F ′i(x

δn) + βnIX)−1F ′

i(xδn)

∗ = F ′i(x

δn)

∗(F ′i(x

δn)F

′i(x

δn)

∗ + βnIYi)−1,

where IX and IYi are the identity operators on spaces X and Yi, respectively, we have

xδn+1,i = x0

i + F ′i(x

δn)

∗(F ′i(x

δn)F

′i(x

δn)

∗ + βnIYi)−1(yδ

i − Fi(xδn) − F ′

i(xδn)(x

0i − xδ

n)). (23)

Taking into account that (F ′i(x

δn)F

′i(x

δn)

∗ + βnIYi)−1 = ‖F ′

i(xδn‖2 + βn, we can rewrite formula

(6) as

xδn+1,i = x0

i + F ′i(x

δn)

T(yδi − Fi(xδ

n) − F ′i(x

δn)(x

0i − xδ

n))

‖F ′i(x

δn)‖2 + βn

; i = 1, . . . , N , (24)

where the symbol T denotes transposition of a matrix or a vector and the Euclidean norm is used.The next approximation xδ

n+1 is defined by Equation (7) as before.

Dow

nloa

ded

by [

Dic

le U

nive

rsity

] at

06:

31 1

1 N

ovem

ber

2014

International Journal of Computer Mathematics 2459

Denoting F = (F1, . . . , FN )T; yδ = (yδ1, . . . , yδ

N )T and observing that F ′(x)TF ′(x) = ∑Ni=1

F ′i(x)

TF ′i(x), by a similar argument as in Equation (23), we can reduce Equation (5) to

xδn+1 = x0 + F ′(xδ

n)T(F ′(xδ

n)F′(xδ

n)T + αnI)−1(yδ − F(xδ

n) − F ′(xδn)(x

0 − xδn)). (25)

At each iteration step the IRGNM (5) requires to solve an m × m system of linear equations,which is time consuming when m is very large. On the other hand, using formula (25), we needto solve a N × N system of linear equations, where N m. Meanwhile, for the PIRGNM, all thecomponents xδ

n+1,i are computed by the explicit formula (24) in parallel, hence the algorithm (24),Equation (7) can give a satisfactory result within reasonable computing time.

For the sake of simplicity, we choose for our experiments m = 105, N = 64, x† =(1, 0, . . . , 0)T, xδ

0 = (0.5, 0, . . . , 0)T, ‖xδ0 − x†‖ = 0.5 and αn = 0.2 ∗ 64 ∗ (0.5)n.

In all the experiments, the matrix [F ′(x†)]T[F ′(x†)] will be singular, hence the Newton methodand its parallel modification (see [8,15]) may not converge, therefore, the IRGNM should be used.However, due to formula (25) at each step, the IRGNM requires to solve a 64 × 64 system oflinear equations. On the other hand, the application of the PIRGNM to Equation (21) leads tosimple explicit formulae (24).

All the numerical experiments will be performed on a LINUX cluster 1350 with eight computingnodes. Each node contains two Intel Xeon dual core 3.2 GHz, 2GBRam. All the programs arewritten in C.

We evaluate the accuracy of the IRGNM and PIRGNM using the relative error norm (REN),i.e. REN := ‖xδ

n − x†‖/‖x†‖. In our examples, ‖x†‖ = 1, hence REN = ‖xδn − x†‖.

The notations used in the tables are as follows:Tp : time of the parallel execution on p processors taken in seconds.Ts : time of the sequential execution taken in seconds.Sp = Ts/Tp : speed up.Ep = Sp/p : efficiency of parallel computation by using p processors.nmin : the first number n, where the REN of the corresponding method is less than a given

tolerance.Nδ : the stopping index defined by Equation (11).η : a fixed small positive parameter in stopping rule (11).For the first experiment, we consider the following system of equations

Fk(x) := xTAkx + bTk x = yk ,

where the matrices Ak are 2k − 1 diagonal with the entries

a(k)ij =

{1 |i − j| ≤ k − 1,

0 otherwise.

Further, let bk = (8, . . . , 8, 0, . . . , 0)T; k = 1, . . . , 64, where the component 8 in the vector bk

repeats exactly k times. Finally, the right-hand sides yi = 9 + χi, where the entries of χi arenormally distributed random numbers with zero mean, scaled to yield the noise level δ.

In this case, the source condition (8) holds with μ = 1 and the initial guesses are x0k =

(0.95, −0.05, . . . , −0.05, 0, . . . , 0)T, k = 1, . . . , 64, where the entry −0.05 in xk occurs exactlyk − 1 times. Moreover, all the derivatives F ′

i(x) are Lipschitz continuous.Table 1 gives the RENs of the PIRGNM and IRGNM as well as their execution times in

sequential mode. For solving systems of linear equations in IRGNM, we used the Choleskymethod. It shows that within a given tolerance, the PIRGNM is less time consuming than IRGNM.

Table 2 finds stopping indices of the PIRGNM and verifies the conclusion of Theorem 2.3 that‖xδ

n∗ − x†‖ = O(δ2/3), where n∗ = Nδ .

Dow

nloa

ded

by [

Dic

le U

nive

rsity

] at

06:

31 1

1 N

ovem

ber

2014

2460 P.K. Anh and V.T. Dzung

Table 1. RENs and execution times in sequential mode with η = 2.

IRGNM PIRGNM

δ REN nmin Ts nmin Ts

e−6 1e−4 11 27 4 6.21e−5 15 36.92 8 12.43

1e−6 18 44.25 12 18.65e−7 1e−4 11 27 4 6.21

e−5 15 36 8 12.431e−6 18 44.25 12 18.65

Table 2. Stopping indices of the PIRGNM with η = 0.02.

m δ Nδ REN REN/δ2/3

e−5 3 1.68e−4 0.36e−6 5 3.85e−5 0.38

105 e−7 8 6.37e−6 0.29e−8 10 2.13e−6 0.45e−9 12 7.4e−7 0.75

Finally, Table 3 gives the efficiency and the speed up of the PIRGNM in parallel mode.For our second experiment, we take F0(x) = x2

1 + x22 + · · · + x2

m + 8x1; Fi(x) = ∑m−ij=1 xjxj+i +

10∑i

j=1 xj + 9xi+1; i = 1, . . . , 63. The right-hand sides y0 = 9 + χ0; yi = 10 + χi, i = 1, . . . , 63and the entries of χi; i ≥ 0 are again normally distributed random numbers with zero mean, scaledto yield the noise level δ.

Clearly, in this case the source condition (8) is satisfied with an exponent μ = 1 and initialguesses x0

0 = (0.5, 0, . . . , 0)T; x0i = (0.5, −0.5, . . . , −0.5, 0, . . . , 0)T; i = 1, . . . , 63, where num-

ber −0.5 in x0i repeats exactly i times. Observe that in this example all the derivatives F ′

i(x) areLipschitz continuous and the initial guesses x0

i need not to be closed to the exact solution x†.Tables 4 and 5 for the second experiment are similar to Tables 1 and 2, respectively.

Table 3. Efficiency and speed up of the PIRGNM.

m Processors Tp Sp Ep

1 18.65100,000 2 9.5 1.96 0.98

4 5.6 3.3 0.82

Table 4. RENs and execution times in sequential mode with η = 0.4.

IRGNM PIRGNM

δ REN nmin Ts nmin Ts

e−4 11 24 3 0.71e−6 e−5 15 33 6 1.4

e−6 18 40 9 2.08e−4 11 24 3 0.71

e−7 e−5 15 33 6 1.4e−6 18 40 9 2.08

Dow

nloa

ded

by [

Dic

le U

nive

rsity

] at

06:

31 1

1 N

ovem

ber

2014

International Journal of Computer Mathematics 2461

Table 5. Stopping indices of the PIRGNM with η = 0.02.

m δ Nδ REN REN/δ2/3

e−5 3 4.5e−5 0.096e−6 5 1.1e−5 0.112

105 e−7 8 1.4e−6 0.065e−8 10 3.5e−7 0.07e−9 12 8.6e−8 0.08

4. Conclusion

In this article, a parallel version of the IRGNM for solving a system of nonlinear ill-posed operatorequations is proposed and its convergence is established. Based on parallel computation, we canreduce the overall computational effort without imposing any extra conditions than widely usedones on the nonlinearity of the operators (see [13]). Numerical experiments for underdeterminedsystems of nonlinear equations show the advantage of the proposed parallel method.

Acknowledgement

The authors are grateful to the anonymous referees and Professor Qin Sheng, Editor-in-Chief of IJCM for their commentswhich substantially improved the quality of this paper. The authors express their sincere thanks to the Advanced MathProgram of Ministry of Education and Training,Vietnam for sponsoring their working visit to University ofWashington andthe Department of Applied Mathematics, University of Washington for the hospitality. This work was partially supportedby the Vietnam National Foundation for Science and Technology (NAFOSTED).

References

[1] P.K. Anh and C.V. Chung, Parallel iterative regularization methods for solving systems of ill-posed equations, Appl.Math. Comput. 212 (2009), pp. 542–550.

[2] P.K.Anh and C.V. Chung, Parallel regularized Newton method for nonlinear ill-posed equations, Numer.Algorithms58 (2011), pp. 379–398.

[3] P.K. Anh and V.T. Dung, Parallel iterative regularization algorithms for large overdetermined linear systems, Int. J.Comput. Methods 7 (2010), pp. 525–537.

[4] A.B. Bakushinskhii, The problem of the convergence of the iteratively regularized Gauss-Newton method, Comput.Math. Math. Phys. 32 (1992), pp. 1353–1359.

[5] B. Blaschke, A. Neubauer, and O. Scherzer, On convergence rates for the iteratively regularized Gauss-Newtonmethod, IMA J. Numer. Anal. 17 (1997), pp. 421–436.

[6] M. Burger and B. Kaltenbacher, Regularizing Newton-Kaczmart methods for nonlinear ill-posed problems, SIAMJ. Numer. Anal. 44 (2006), pp. 153–182.

[7] P. Deuflhard, H.W. Engl, and O. Scherzer, A convergence analysis of iterative methods for the solution of nonlinearill-posed problems under affinely invariant conditions, Inverse Problems 14 (1948), pp. 1081–1106.

[8] M.A. Diniz-Ehrhardt, J.M. Martinez, and S.A. Santos, Parallel projection methods and the resolution of ill-posedproblems, Comput. Math. Appl. 27 (1994), pp. 11–24.

[9] H.W. Engl, K. Kunisch, and A. Neubauer, Convergence rates for Tikhonov regularization of nonlinear ill-posedproblems, Inverse Problems 5 (1989), pp. 523–540.

[10] T. Hohage, Logarithmic convergence rates of the iteratively regularized Gauss-Newton method for an inversepotential and inverse scattering problem, Inverse Problems 13 (1997), pp. 1279–1299.

[11] Q.N. Jin, On the iteratively regularized Gauss-Newton method for solving nonlinear ill-posed problems, Math.Comput. 69 (2000), pp. 1603–1623.

[12] B. Kaltenbacher, A. Neubauer, and O. Scherzer, Iterative Regularization Methods for Nonlinear Ill-posed Problems,Walterde Gruyter, Berlin, New York, 2008.

[13] T. Lu, P. Neittaanmaki, and X.C. Tai, A parallel splitting up method for partial differential equations and itsapplication to Navier-Stokes equations, RAIRO Math. Model. Numer. Anal. 26 (1992), pp. 673–708.

[14] O. Scherzer, H.W. Engl, and K. Kunisch, Optimal a posteriori parameter choice for Tikhonov regularization forsolving nonlinear ill-posed problems, SIAM J. Numer. Anal. 30 (1993), pp. 1796–1838.

[15] G. Zilli and L. Bergmaschi, Parallel Newton methods for sparse systems of nonlinear equations, Rend Circ. Mat.Palermo (2), Suppl. 58 (1999), pp. 247–257.

Dow

nloa

ded

by [

Dic

le U

nive

rsity

] at

06:

31 1

1 N

ovem

ber

2014