SOLVING SECOND ORDER CONE PROGRAMMING VIA A

REDUCED AUGMENTED SYSTEM APPROACH

ZHI CAI ∗ AND KIM-CHUAN TOH †

To the memory of Jos Sturm

Abstract. The standard Schur complement equation based implementation of interior-pointmethods for second order cone programming may encounter stability problems in the computationof search directions, and as a consequence, accurate approximate optimal solutions are sometimesnot attainable. Based on the eigenvalue decomposition of the (1, 1) block of the augmented equation,a reduced augmented equation approach is proposed to ameliorate the stability problems. Numericalexperiments show that the new approach can achieve more accurate approximate optimal solutionsthan the Schur complement equation based approach.

Key words. second order cone programming, augmented equation, Nesterov-Todd direction,stability

AMS subject classifications. 90C20, 90C22, 90C51, 65K05

1. Introduction . A second order cone programming (SOCP) problem is a lin-ear optimization problem over a cross product of second order convex cones. A widerange of problems can be formulated as SOCP problems; they include linear pro-gramming (LP) problems, convex quadratically constrained quadratic programmingproblems, filter design problems [5, 20], and problems arising from limit analysis ofcollapses of solid bodies [6]. An extensive list of applications problems that can be for-mulated as SOCPs can be found in [14]. For a comprehensive introduction to SOCP,we refer the reader to the paper by Alizadeh and Goldfarb [1].

SOCP itself is a subclass of semidefinite programming (SDP). In theory, SOCPproblems can be solved as SDP problems. However, it is far more efficient computa-tionally to solve SOCP problems directly. A few interior-point methods (IPMs) havebeen developed to solve SOCPs directly [3, 21, 25]. But these IPMs sometimes failto deliver solutions with satisfactory accuracy. The main objective of this paper isto propose a method that can solve an SOCP to high accuracy, but with comparableor moderately higher cost than the standard IPMs employing the Schur complementequation (SCE) approach. We note that global polynomial convergence results forIPMs for SOCP can be found in [15] and the references therein.

Given a column vector xi, we will write it as xi = [x0i ; xi] with x0

i being thefirst component and xi consisting of the remaining components. Given square ma-trices P, Q, the notation [P ; Q] means that Q is appended to the last row of P ;and diag(P, Q) denotes the block diagonal matrix with P, Q as its diagonal blocks.Throughout this paper, ‖ · ‖ denotes the matrix 2-norm or vector 2-norm, unless oth-erwise specified. For a given matrix M , we let λmax(M) and λmin(M) be the largestand smallest eigenvalues of M in magnitude, respectively. The condition numberof a matrix M (not necessarily square) is the number κ(M) = σmax(M)/σmin(M),where σmax(M) and σmin(M) are the largest and smallest singular values of M , re-spectively. For a matrix Mµ which depends on a positive parameter µ, the notation

∗High Performance Computing for Engineered Systems (HPCES), Singapore-MIT Alliance, 4Engineering Drive 3, Singapore 117576. (smap0035@nus.edu.sg).

†Department of Mathematics, National University of Singapore, 2 Science Drive 2, Singa-pore 117543, Singapore; Singapore-MIT Alliance, 4 Engineering Drive 3, Singapore 117576.(mattohkc@math.nus.edu.sg).

1

2 Z. CAI AND K. C. TOH

‖Mµ‖ = O(µ) (‖Mµ‖ = Ω(µ)) means that there is a positive constant c such that‖Mµ‖ ≤ cµ (‖Mµ‖ ≥ cµ) as µ ↓ 0; and ‖Mµ‖ = Θ(µ) means that there are positiveconstants c1, c2 such that c1µ ≤ ‖Mµ‖ ≤ c2µ as µ ↓ 0. More generally, for a func-tion Kµ depending on a positive parameter µ, the notation ‖Mµ‖ = Θ(1)Kµ meansthat there are positive constants c1, c2 such that c1K

µ ≤ ‖Mµ‖ ≤ c2Kµ for all µ

sufficiently small.Consider the following standard primal and dual SOCP problems:

(P) min∑Ni=1 cT

i xi :∑N

i=1 Aixi = b, xi 0, i = 1, . . . , N

(D) maxbT y : ATi y + zi = ci, zi 0, i = 1, . . . , N,

(1.1)

where Ai ∈ IRm×ni , ci, xi, zi ∈ IRni , i = 1, . . . , N , and y ∈ IRm. The constraintxi 0 is a second order cone constraint defined by x0

i ≥ ‖xi‖. In particular, if thecone dimension ni is 1, then the constraint is simply the standard non-negativityconstraint xi ≥ 0, and such a variable is called a linear variable. For convenience, wedefine

A = [A1 A2 · · · AN ] , c = [c1 ; c2 ; · · · ; cN ] ,

x = [x1 ; x2 ; · · · ; xN ] , z = [z1 ; z2 ; · · · ; zN ] , n =

N∑

i=1

ni.

The notation x 0 (x 0) means that each xi is in (the interior of) the ith secondorder cone.

In this paper, we will assume that A has full row rank, and that (P) and (D) in(1.1) are strictly feasible. Under these assumptions, the solutions to the perturbedKKT conditions of (1.1) form a path (known as the central path) in the interior ofthe primal-dual feasible region. At each iteration of an IPM, the Newton equationassociated with the perturbed KKT conditions needs to be solved. By performingblock eliminations, one can either solve a system of linear equations of size m + n orone of size m. These linear systems are known as the augmented equation and theSchur complement equation (SCE), respectively. The SCE has the obvious advantageof being smaller in size as well as being symmetric positive definite. Currently, mostimplementations of IPMs [3, 21, 25] are based on solving the SCE. However, as we shallsee in Section 3, the SCE can be severely ill-conditioned when the barrier parameteris close to 0. This typically causes numerical difficulties and imposes a limit on howaccurately one can solve an SOCP problem.

In the case of LP, the ill-conditioning of the augmented equation was analyzed byWright [28, 29]. Under certain assumptions including nondegeneracy, the computedsearch direction from the augmented equation is shown to be sufficiently accurate forthe IPM to converge to high accuracy. The structure of the ill-conditioning of the SCEarising from LP was analyzed in [13]. A stabilization method based on performingGaussian elimination with a certain pivoting order was also proposed to transformthe SCE into a better conditioned linear system of equations.

In nonlinear conic programming, however, the ill-conditioning of the augmentedequation and SCE are much more complicated than that in LP. The potential nu-merical difficulties posed by the ill-conditioned SCE in SOCP were recognized bydevelopers of solvers for SOCP such as [3, 4], [23], and [25]. It was also recognizedby Goldfarb and Scheinberg [9] and that motivated them to propose and analyze a

Solving SOCP via a reduced augmented system 3

product-form Cholesky factorization for the Schur complement matrix. Subsequently,Sturm [23] implemented the product-form Cholesky factorization [9] in his very popu-lar code SeDuMi to solve the SCE arising at each iteration of a homogeneous self-dual(HSD) IPM. SeDuMi also employed sophisticated techniques to minimize numericalcancelations when computing the SCE and its factorization [23]. These sophisticatedtechniques typically greatly improve the stability of the SCE approach. However, forcertain extreme cases, they do not entirely ameliorate the numerical difficulties causedby the inherently ill-conditioned SCE; see Section 4.

The IPM code SeDuMi differs from standard infeasible interior-point methodsin that it solves the homogeneous self-dual embedding model. A natural questionto ask is whether SeDuMi’s unusually good performance arises from the inherentstructure of the HSD model itself or from the sophisticated numerical techniques ituses in solving the SCE (or both). For a certain class of SOCPs with no strictlyfeasible primal/dual points, we show numerically in Section 4 that SeDuMi’s superiorperformance can be explained by the structure of the HSD model itself. For someSOCPs with strictly feasible points, we shall also see in Section 4 that the sophisticatednumerical techniques sometimes may offer only limited improvement in the attainableaccuracy when compared to simpler techniques used to solve the SCE.

Herein we propose a method to compute the search directions based on a reducedaugmented equation (RAE). This RAE is derived by applying block row operationsto the augmented equation, together with appropriate partitioning of the eigen-spaceof its (1,1) block. The RAE is generally much smaller in size compared to the originalaugmented equation. By their construction, RAE-based IPMs are computationallymore expensive than SCE-based IPMs. Fortunately, numerical experiments showthat if sparsity in the SOCP data is properly preserved when forming the RAE, it cangenerally be solved rather efficiently by a judicious choice of a symmetric indefinitesystem solver.

The RAE-based IPMs are superior to SCE-based IPMs in that the former canusually deliver approximate optimal solutions that are much more accurate than thelatter before numerical difficulties are encountered. For example, for the schedxxx

SOCP problems selected from the DIMACS library [17], our RAE-based IPMs are ableto obtain accuracies of 10−9 or better, while the SCE-based IPMs (SDPT3 version3.1 and SeDuMi) can only obtain accuracies of 10−3 or 10−4 in some cases.

The paper is organized as follows. In Section 2, we introduce the augmented andSchur complement equations. In Section 3, we analyse the conditioning and the growthin the norm of the Schur complement matrix. We also discuss how the latter affectsthe primal infeasibility as the interior-point iterates approach optimality. In Section4, we present numerical results obtained from two different SCE-based primal-dualIPMs. In Section 5, we derive the RAE. The conditioning of the reduced augmentedmatrix is analysed in Section 6. In Section 7, we discuss major computational issuesfor efficiently solving the RAE. Numerical results for an RAE-based IPM are presentedin Section 8. We conclude the paper in Section 9.

2. The augmented and Schur complement equations. In this section, wepresent the linear systems that need to be solved to compute the search direction ateach IPM iteration.

For xi in a second order cone, we define

aw(xi) =

[x0

i xTi

xi x0i I

], γ(xi) =

√(x0

i )2 − ‖xi‖2.(2.1)

4 Z. CAI AND K. C. TOH

For a given barrier parameter ν, the perturbed KKT conditions of (1.1) in matrixform are:

Ax = b, AT y + z = c, aw(x)aw(z)e0 = νe0,(2.2)

where e0 = [e1 ; e2 ; · · · ; eN ], with ei being the first unit vector in IRni . The matrixaw(x) = diag(aw(x1), · · · , aw(xN )) is a block diagonal matrix with aw(x1), · · · , aw(xN )as its diagonal blocks. The matrix aw(z) is defined similarly.

For reasons of computational efficiency that we will explain later, in most IPMimplementations for SOCP, a block diagonal scaling matrix is usually applied to thelast equation in (2.2). Here, we apply the Nesterov-Todd (NT) scaling matrix [25] toproduce the following equation:

aw(Fx)aw(F−1z)e0 = νe0,(2.3)

where F = diag(F1, · · · , FN ) is chosen such that Fx = F−1z =: v. For details onthe conditions that F must satisfy and on other scaling matrices, we refer the readerto [15]. Let

fi =

f0i

fi

:=

1√2(γ(xi)γ(zi) + xT

i zi

)

1

ωiz0

i + ωix0i

1

ωizi − ωixi

,

where ωi =√

γ(zi)/γ(xi). (Note that γ(fi) = 1.) The precise form of Fi is given by

Fi = ωi

f0i fT

i

fi I +fif

T

i

1+f0

i

.(2.4)

Let µ = xT z/N be the normalized complementarity gap. The Newton equationassociated with the perturbed KKT conditions (2.2) with NT scaling is given by

A∆x = rp, AT ∆y + ∆z = rd, V F∆x + V F−1∆z = rc,(2.5)

where V = aw(v), rp = b − Ax, rd = c − z − AT y, rc = σµe0 − V v. Note that wehave chosen ν to be ν = σµ for some parameter σ ∈ (0, 1).

The solution (∆x, ∆y, ∆z) of the Newton equation (2.5) is referred to as thesearch direction. At each IPM iteration, solving (2.5) for the search direction is com-putationally the most expensive step. Observe that by eliminating ∆z, the Newtonequation (2.5) reduces to the so-called augmented equation:

[−F 2 AT

A 0

] [∆x

∆y

]=

[rx

rp

],(2.6)

where rx = rd − FV −1rc. The augmented equation can further be reduced in size byeliminating ∆x in (2.6) to produce the SCE:

AF−2AT︸ ︷︷ ︸

M

∆y = ry := rp + AF−2rx = rp + AF−2rd − AF−1V −1rc.(2.7)

The coefficient matrix M in (2.7) is known as the Schur complement matrix. It issymmetric positive definite if x, z 0. The search direction corresponding to (2.5)

Solving SOCP via a reduced augmented system 5

always exists as long as x, z 0. Note that if the scaling matrix F is not appliedto the last equation in (2.2), the corresponding Schur complement matrix would beAaw(z)−1

aw(x)AT , which is a nonsymmetric matrix. This nonsymmetric coefficientmatrix is not guaranteed to be nonsingular even when x, z 0. Moreover, computingits sparse LU factorization is usually much more expensive than computing the sparseCholesky factorization of M .

In their simplest form, most current implementations of IPMs compute the searchdirection (∆x, ∆y, ∆z) based on the SCE (2.7) via the following procedure.

Simplified SCE approach:

(i) Compute the Schur complement matrix M and the vector ry ;

(ii) Compute the Cholesky or sparse Cholesky factor of M ;

(iii) Compute ∆y by solving 2 triangular linear systems involving the Cholesky factor;

(iv) Compute ∆z via ∆z = rd − AT ∆y; and ∆x via ∆x = F−2(AT ∆y − rx).

We should note that various heuristics to improve the numerical stability of the sim-plified SCE approach are usually incorporated in the actual implementations. Wewill describe in Section 4 variants of the above approach implemented in two publiclyavailable SOCP solvers, SDPT3, version 3.1 [26] and SeDuMi, version 1.05 [22].

The SCE is preferred because it is usually a much smaller system compared tothe augmented or Newton equations. Furthermore, the Schur complement matrix hasthe highly desirable property of being symmetric positive definite. (In contrast, thecoefficient matrix in (2.6) is symmetric indefinite while that of (2.5) is nonsymmetric.)Consequently, the SCE can be solved very efficiently via Cholesky or sparse Choleskyfactorization of M . We should mention that there are highly efficient and machineoptimized sparse Cholesky codes readily available in the public domain, the primeexample being the sparse Cholesky codes of Ng and Peyton [16]. Comparatively, thestate-of-the-art LDLT factorization codes (an example being the MA47 codes of Duffand Reid [18]) for a sparse symmetric indefinite matrix available in the public domainare less advanced.

3. Conditioning of M and the deterioration of primal infeasibility. De-spite the advantages of the SCE approach described in the last section, the SCE ishowever, generally severely ill-conditioned when the iterates (x, y, z) approach opti-mality, and this typically causes numerical difficulties. The most common numericaldifficulty one may encounter in practice is that the Schur complement matrix M isnumerically indefinite, although in exact arithmetic M is positive definite. Further-more, the computed solution ∆y from (2.7) may also be very inaccurate in that theresidual norm ‖ry −M∆y‖ is much larger than the machine epsilon, and this typicallycauses the IPM to stall.

In this section, we will analyze the relationship between the norm ‖M‖, theresidual norm ‖ry −M∆y‖ of the computed solution ∆y, and the primal infeasibility‖rp‖, as the interior-point iterates approach optimality.

3.1. Eigenvalue decomposition of F2. To analyze the norm ‖M‖ and the

conditioning of M , we need to know the eigenvalue decomposition of F 2. Recall thatF = diag(F1, · · · , FN ). Thus it suffices to find the eigenvalue decomposition of F 2

i ,where Fi is given in (2.4). By noting that for cones of dimensions ni ≥ 2, F 2

i can bewritten as F 2

i = ω2i

(I + 2(fif

Ti − eie

Ti )

), the eigenvalue decomposition of F 2

i canreadily be found. (The case where ni = 1 is easy, and F 2

i = zi/xi.) Without going

6 Z. CAI AND K. C. TOH

through the algebraic details, the eigenvalue decomposition of F 2i is given by

F 2i = QiΛiQ

Ti , Qi =

− 1√2

+ 1√2

0 · · · 0

1√2

gi1√2

gi q3i · · · qni

i

,(3.1)

with Λi = ω2i diag

((f0

i − ‖fi‖)2, (f0i + ‖fi‖)2, 1, · · · , 1

), and

gi := [g0i ; gi] = fi/‖fi‖ ∈ IRni−1.(3.2)

Notice that since γ(fi) = 1, the first eigenvalue is the smallest and the second is thelargest. The set q3

i , · · · , qni

i is an orthonormal basis of the subspace u ∈ IRni−1 :uT gi = 0. To construct such an orthonormal basis, one may first construct the(ni − 1)× (ni − 1) Householder matrix Hi [10] associated with the vector gi, then thelast ni − 2 columns of Hi is such an orthonormal basis. The precise form of Hi willbe given later in Section 6.

3.2. Analysis of ‖M‖ and the conditioning of M . Recall that M is de-pendent on the normalized complementarity gap µ. Here we analyze how fast thenorm ‖M‖ and the condition number of M will grow when µ ↓ 0, i.e., when theinterior-point iterates approach an optimal solution (x∗, y∗, z∗). To simplify the anal-ysis, we will assume that strict complementarity holds at the optimal solution. Unlessotherwise stated, we assume that ni ≥ 2 in this subsection.

Strict complementarity [2] implies that for each pair of the optimal primal anddual solutions, x∗

i and z∗i , we have γ(x∗i ) + ‖z∗i ‖ and γ(z∗i ) + ‖x∗

i ‖ both positive. Inother words, (a) either γ(x∗

i ) = 0 or z∗i = 0, but not both; and (b) either γ(z∗i ) = 0or x∗

i = 0, but not both. Under the strict complementarity assumption, we have thefollowing three types of eigenvalue structures (following the classification in [9]) forF 2

i when aw(xi)aw(zi)ei = µei and µ is small. Note that xTi zi = µ.

Type 1 solution: x∗i 0, z∗i = 0. In this case, γ(xi) = Θ(1), γ(zi) = Θ(µ), and

ωi = Θ(√

µ). Also, f0i , ‖fi‖ = Θ(1), implying that all the eigenvalues of F 2

i

are Θ(µ).Type 2 solution: x∗

i = 0, z∗i 0. In this case, γ(xi) = Θ(µ), γ(zi) = Θ(1), andωi = Θ(1/

õ). Also, f0

i , ‖fi‖ = Θ(1), implying that all the eigenvalues ofF 2

i are Θ(1/µ).Type 3 solution: γ(x∗

i ) = 0, γ(z∗i ) = 0, x∗i , z

∗i 6= 0. In this case, γ(xi), γ(zi) =

Θ(√

µ), and ωi = Θ(1). This implies that f0i , ‖fi‖ = Θ(1/

õ). Thus the

largest eigenvalue of F 2i is Θ(1/µ) and by the fact that γ(fi) = 1, the smallest

eigenvalue of F 2i is Θ(µ). The rest of the eigenvalues are Θ(1).

Let D be the diagonal matrix consisting of the eigenvalues of F 2 sorted in as-cending order. Then we have F 2 = QDQT , where the columns of Q are the sortedeigenvectors of F 2. Let D be partitioned into D = diag(D1, D2a, D2b) such thatdiag(D1) consists of all the small eigenvalues of F 2 of order Θ(µ), and diag(D2a),diag(D2b) consist of the remaining eigenvalues of order Θ(1) and Θ(1/µ), respec-tively. Note that the three groups of eigenvalues need not be all present. We alsopartition the matrix Q as Q = [Q(1) , Q(2a) , Q(2b)]. Then A := AQ is partitioned

as A = [A1 , A2a , A2b] = [AQ(1) , AQ(2a) , AQ(2b)]. With the above partitions, wecan express M as

M = A1D−11 AT

1︸ ︷︷ ︸M1

+ A2aD−12a AT

2a︸ ︷︷ ︸M2a

+ A2bD−12b AT

2b︸ ︷︷ ︸M2b

.(3.3)

Solving SOCP via a reduced augmented system 7

Lemma 3.1. (a) For the matrices M1, M2a and M2b in (3.3), we have

‖M1‖ = Θ(1/µ)‖A1‖2, ‖M2a‖ = Θ(‖A2a‖2), ‖M2b‖ = Θ(µ)‖A2b‖2.

(b) Suppose there are Type 1 or Type 3 solutions so that A1 is not a null matrix. Then

the following statments hold. [i] ‖M‖ = Θ(1/µ)‖A1‖2. [ii] If A1 does not have full row

rank, then ‖M−1‖ =(O(1)‖A2a‖2 + O(µ)‖A2b‖2

)−1

= Ω(1)(‖A2a‖2 + ‖A2b‖2

)−1

.

Proof. (a) We shall only prove the first result since the other two can be provedsimilarly. By the definition of D1, there are positive constants c1, c2 such that(c1/µ)I D−1

1 (c2/µ)I. Thus (c1/µ)A1AT1 M1 (c2/µ)A1A

T1 . This implies

that (c1/µ)‖A1AT1 ‖ ≤ ‖M1‖ ≤ (c2/µ)‖A1A

T1 ‖, and the required result follows by not-

ing that ‖A1AT1 ‖ = ‖A1‖2.

(b)[i] From (3.3), it is clear that ‖M‖ = Θ(1/µ)‖A1‖2. (b)[ii] If A1 does not have full

row rank, then the null space N (AT1 ) is non-trivial. Let U be a matrix whose columns

form an orthonormal basis of N (AT1 ) and W := UT (M2a + M2b)U . Since AT

1 U = 0,we have UT MU = UT (M2a + M2b)U = W . By the Courant-Fischer Theorem, itis clear that λmin(M) ≤ λmin(W ). Thus ‖M−1‖ = 1/λmin(M) ≥ 1/λmin(W ) =

‖W−1‖ ≥ 1/‖W‖. Since ‖W‖ ≤ ‖M2a‖ + ‖M2b‖ = O(1)‖A2a‖2 + O(µ)‖A2b‖2 =

O(1)(‖A2a‖2 + ‖A2b‖2), the required result follows.

Remark 3.1. (a) Lemma 3.1 implies that the growth in ‖M‖ is caused by F 2

having small eigenvalues of order Θ(µ).

(b) If A1 is present and does not have full row rank, then κ(M) = Ω(1/µ)‖A1‖2(‖A2a‖2+

‖A2b‖2)−1

. On the other hand, if A1 has full row rank (which implies that the number

of eigenvalues of F 2 of order Θ(µ) is at least m), then κ(M) = Θ(1)κ(A1)2.

(c) If there are only Type 2 solutions (thus x∗ = 0 and z 0), then M = AQDQT AT

with D = Θ(µ). In this case, we have κ(M) = Θ(1)κ(A)2.

Based on the results in [2], we have the following theorem concerning the rank

of A1 and A2a. We refer the reader to [2] for the definitions of primal and dualdegeneracies.

Theorem 3.2. Suppose that (x∗, y∗, z∗) satisfies strict complementarity. If the

primal optimal solution x∗ is primal nondegenerate, then[A1 , A2a

]has full row rank

when µ is small. If the dual optimal solution (y∗, z∗) is dual nondegenerate, then A1

has full column rank when µ is small.

Proof. The result follows from Theorems 20 and 21 in [1].

Remark 3.2. We should emphasize that while Theorem 3.2 said that [A1, A2a]has full row rank when the optimal solution is strictly complementary and primal anddual nondegenerate, the matrix A1, however, does not necessarily have full row rankunder the same condition. In the event that A1 is present and does not have full rowrank, Remark 3.1 (b) said that M is ill-conditioned with κ(M) = Ω(1/µ). Thus evenif the optimal solution is strict complementary and primal and dual nondegenerate,M does not necessarily have bounded condition number when µ ↓ 0. In contrast, fora LP problem (for which all the cones have dimensions ni = 1 and A2a is absent),

primal and dual nondegeneracy ensure that A1 has full column and row rank, and asa result, M has bounded condition number when µ ↓ 0.

8 Z. CAI AND K. C. TOH

3.3. Analysis of the deterioration of primal infeasibility. Although Choleskyfactorization is stable for any symmetric positive definite matrix, the conditioning ofthe matrix may still affect the accuracy of the computed solution of the SCE. It is acommon phenomenon that for SOCP, the accuracy of the computed search directiondeteriorates as µ decreases due to an increasingly ill-conditioned M . As a result ofthis loss of accuracy in the computed solution, the primal infeasibility ‖rp‖ typicallyincreases or stagnates when the IPM iterates approach optimality.

With the analysis of ‖M‖ given in the last subsection, we will now analyze why theprimal infeasibility may deteriorate or stagnate as interior-point iterations progress.

Lemma 3.3. Suppose at the kth iteration, the residual vector in solving the SCE(2.7) is ξ = ry − M∆y. Assuming that ∆x is computed exactly via the equation∆x = F−2(AT ∆y−rx), then the primal infeasibility for the next iterate x+ = x+α∆x,α ∈ [0, 1], is given by

r+p := b − Ax+ = (1 − α)rp + αξ.

Proof. We have r+p = (1−α)rp +α(rp −A∆x). Now A∆x = AF−2(AT ∆y− rx) =

M∆y − ry + rp, thus rp − A∆x = ry − M∆y = ξ and the lemma is proved.Remark 3.3. (a) In Lemma 3.3, we assume for simplicity that the component

direction ∆x is computed exactly. In finite precision arithmetic, errors will be intro-duced in the computation of ∆x and that will also worsen the primal infeasibility r+

p

of the next iterate besides ‖ξ‖.(b) Observe that if the SCE is solved exactly, i.e., ξ = 0, then ‖r+

p ‖ = (1 − α)‖rp‖,and the primal infeasibility should decrease monotonically.

Lemma 3.3 implies that if the SCE is not solved to sufficient accuracy, then theinaccurate residual vector ξ may worsen the primal infeasibility of the next iterate.By standard perturbation error analysis, the worst-case residual norm of ‖ξ‖ can beshown to be proportional to ‖M‖‖∆y‖ times the machine epsilon u. The precisestatement is given in the next lemma.

Lemma 3.4. Let u be the machine epsilon. Given a symmetric positive definitematrix B ∈ IRn×n with (n + 1)2u ≤ 1/3, if Cholesky factorization is applied to B tosolve the linear system Bx = b to produce a computed solution x, then (B + ∆B)x =b, for some ∆B with ‖∆B‖ satisfying the following inequality: ‖∆Bx‖ ≤ 3(n +1)2u‖B‖‖x‖. Thus

‖b − Bx‖ = ‖∆Bx‖ = O(n2)u‖B‖‖x‖.

Proof. The lemma follows straightforwardly from Theorem 10.3 and 10.4, and theirextensions in [12].

Remark 3.4. Lemma 3.4 implies that if ‖B‖‖x‖ is large, then in the worst casescenario, the residual norm ‖b − Bx‖ is expected to be proportionately large.

By Lemma 3.3 and the application of Lemma 3.4 to the SCE, we expect in theworst case the primal infeasibility ‖rp‖ to grow to some extent that is proportional to‖M‖‖∆y‖u. We end this section by presenting a numerical example to illustrate therelation between ‖rp‖ and ‖M‖‖∆y‖u in the last few iterations of an SCE-based IPMwhen solving the SOCP problems rand200 800 1 and sched 50 50 orig (describedin Section 4).

The IPM we use is the primal-dual path-following method with Mehrotra predictor-corrector implemented in the Matlab software SDPT3, version 3.1 [26]. But we

Solving SOCP via a reduced augmented system 9

should mention that to be consistent with the analysis presented in this section, thesearch directions are computed based on the simplified SCE approach presented inSection 2, not the more sophisticated variant implemented in SDPT3.

Table 3.1 shows the norms ‖M‖, ‖M−1‖, ‖ry − M∆y‖ when solving the SCE(2.7). For this problem, ‖M‖ and (hence κ(M)) grows like Θ(1/µ) because its optimalsolutions x∗

i , z∗i are all of Type 3. The fifth and sixth columns in the table show that

the residual norm in solving the SCE and ‖rp‖ deteriorate as ‖M‖ increases. Thisis consistent with the conclusions of Lemmas 3.3 and 3.4. The last column furthershows that ‖rp‖ increases proportionately to ‖M‖‖∆y‖u, where the machine epsilonu is approximately 2.2 × 10−16.

Figure 3.1 illustrates the phenomenon graphically for the SOCP problems rand200 800 1

and sched 50 50 orig. The curves plotted correspond to the relative duality gap(relgap), and the relative primal and dual infeasibility (p-inf and d-inf), definedby

relgap =|cT x − bT y|

1 + (|cT x| + |bT y|)/2, p-inf =

‖rp‖1 + ‖b‖ , d-inf =

‖rd‖1 + ‖c‖ .(3.4)

2 4 6 8 10 12 1410

−15

10−10

10−5

100

iteration

rand200_800_1: SCE approach

relgap

p−inf

d−inf

5 10 15 20 25 3010

−10

10−5

100

105

iteration

sched_50_50_orig: SCE approach

relgap

d−inf

p−inf

Fig. 3.1. Convergence history of the SOCPs problems rand200 800 1 and sched 50 50 orig

when solved by the SCE-based IPM in SDPT3, version 3.1. Notice that the relative primal infeasi-bility p-inf deteriorates as interior-point iterates approach optimality, while relgap may stagnate.

4. Computational results of two SCE-based IPMs on solving some

SOCP problems. Here we present numerical results for the SCE-based IPMs imple-mented in the public domain solvers, SDPT3, version 3.1 [26], and SeDuMi, version1.05 [22]. In this paper, all the numerical results are obtained in Matlab 6.5 from aPentium IV 2.4GHz PC with 1G RAM running a Linux operating system.

Before we analyze the performance of the SCE-based IPMs implemented in SDPT3and SeDuMi, we must describe the methods employed to solve the SCE in both solvers.The IPM in SDPT3 is an infeasible path-following method that attempts to solve thecentral path equation based on (2.2), even if this path does not exist. It solves the re-sulting SCE at each IPM iteration as follows. First it computes the Cholesky or sparseCholesky factor of the Schur complement matrix M . Then the computed Choleskyfactor is used to construct a preconditioner within a preconditioned symmetric quasi-minimal residual (PSQMR) Krylov subspace iterative solver employed to solve theSCE for ∆y. The computations of ∆z and ∆x are the same as in the simplified SCEapproach presented in Section 2.

10 Z. CAI AND K. C. TOH

Table 3.1

The norm of the Schur complement matrix and ‖rp‖ associated with the last few IPM iterationsfor solving the SOCP problems rand 200 800 1 and sched 50 50 orig.

Iter ‖M‖ ‖M−1‖ µ := xT z/N ‖∆y‖ ‖ry − M∆y‖ ‖rp‖‖rp‖

‖M‖‖∆y‖urand 200 800 1

9 1.8e+13 4.9e+02 9.2e-07 2.0e-02 8.7e-06 7.1e-06 8.9e-02

10 2.9e+14 2.3e+02 1.2e-07 2.7e-03 2.5e-05 1.8e-05 1.0e-01

11 3.8e+15 4.0e+01 1.1e-08 5.1e-04 4.9e-05 6.1e-05 1.4e-01

12 1.9e+17 6.8e+00 1.2e-09 9.4e-05 2.2e-04 1.4e-04 3.5e-02

13 1.2e+18 3.8e+01 1.8e-10 2.5e-03 3.7e-02 9.0e-04 1.4e-03

sched 50 50 orig

25 5.0e+08 1.5e+04 1.5e-01 3.3e+02 7.5e-09 1.1e-06 3.0e-02

26 3.0e+09 2.1e+04 4.3e-02 2.2e+02 2.1e-07 1.4e-05 9.8e-02

27 2.7e+10 1.7e+04 9.9e-03 4.8e+01 1.9e-07 1.5e-05 5.4e-02

28 4.9e+11 1.9e+04 1.4e-03 1.0e+01 1.5e-06 1.0e-05 9.3e-03

29 5.3e+12 2.1e+04 3.3e-04 2.1e+00 1.2e-05 2.9e-04 1.2e-01

SeDuMi is a very well implemented SCE-based public domain solver for bothSOCP and SDP. The IPM in SeDuMi is not based on the central path for the originalprimal and dual problems (1.1), but that of the homogeneous self-dual (HSD) modelof Ye, Todd, and Mizuno [30]. The HSD model has the nice theoretical property thata strictly feasible primal and dual point always exists even if the original problems donot have one, and as a result the central path for the HSD model always exists, whichis not necessarily true for the original problems in (1.1). As a consequence of this niceproperty, its solution set is always bounded. The same cannot be said for the originalproblems. For a problem that models an unrestricted variable by the difference of 2nonnegative variables, the solution set for the original primal SOCP (P) is unbounded,and the feasible region of (D) has an empty interior, implying that the primal-dualcentral path does not exist. The HSD model, on the other hand, does not suffer fromthese defects. Thus the IPM in SeDuMi will not feel the effect of the unboundedsolution set and nonexistence of the central path in the original problems in (1.1), butthe effect of the unboundedness of the solution set on the infeasible path-followingIPM in SDPT3 can be substantial and it often causes serious numerical difficulties.

The computation of the search direction in SeDuMi is based on the SCE associatedwith the HSD model. But it employs sophisticated numerical techniques to minimizenumerical cancelations in its implementation of the SCE approach [23]. It computesthe Schur complement matrix in the scaled space (called the v-space) framework, andtransforms back and forth between quantities in the scaled and original spaces. Itemploys the sparse Cholesky codes adapted from Ng and Peyton [16] to compute thefactorization. It also employs the product-form Cholesky factorization [9] to handledense columns. If the computed Cholesky factor is deemed sufficiently stable, SeDuMiwill proceed to compute ∆y by solving two triangular linear systems involving theCholesky factor; otherwise, it will solve the SCE by using the preconditioned conjugategradient iterative method with a preconditioner constructed from the Cholesky factor.Note that the Cholesky factorization has been shown in [9] to produce stable triangularfactors for the Schur complement matrix if the iterates are sufficiently close to thecentral path and strict complementarity holds at optimality. It is important to note,

Solving SOCP via a reduced augmented system 11

however, that using a stable method to solve the SCE does not necessarily imply thatthe computed direction (∆x, ∆y, ∆z) based on the SCE approach will produce a smallresidual norm with respect to the original linear system (2.5); see Theorem 3.2 of [11]for the case of SDP.

We tested the SCE-based IPMs in SDPT3 and SeDuMi on the following set ofSOCP problems. The statistics for the test problems are shown in Table 4.1.(a) The first set consists of 18 SOCPs in the DIMACS library collected by Pataki and

Schmieta [17], available at http://dimacs.rutgers.edu/Challenges/Seventh/Instances/(b) The second set consists of 10 SOCPs from the FIR Filter Optimization Tool-

box of Scholnik and Coleman, available at http://www.csee.umbc.edu/~

dschol2/opt.html

(c) The last set consists of 10 randomly generated SOCPs. These random problemsrandxxx are generated to be feasible and dominated by Type 3 solutions. Foreach problem, the constraint matrix A has the form V1ΣV T

2 , where V1, V2 arematrices whose columns are orthonormal, and Σ is a diagonal matrix withrandom diagonal elements drawn from the standard normal distribution, buta few of the diagonal elements are set to 105 to make A moderately ill-conditioned.

In our experiments, we stop the IPM iteration in SDPT3 when any of the follow-ing situations are encountered: (1) max(relgap, p-inf, d-inf) ≤ 10−10; (2) incurablenumerical difficulties (such as the Schur complement matrix being numerically indef-inite) occur; (3) p-inf has deteriorated to the extent that p-inf > relgap. SeDuMialso has a similar set of stopping conditions but based on the variables of the HSDmodel. In SeDuMi, the dual conic constraints are not strictly enforced, thus themeasure d-inf for SeDuMi is defined to be d-inf = max(‖rd‖, ‖z−‖), where ‖z−‖measures how much the dual conic constraints are violated. We define

φ := log10(maxrelgap, p-inf, d-inf).(4.1)

Table 4.1 shows the numerical results for SDPT3 and SeDuMi on 36 SOCP problems.Observe that the accuracy exponent (φ) for many of the problems fall short of thetarget of −10. For the sched-xxx problems, the accuracy exponents attained areespecially poor, only −3 or −4 in some cases. We should mention that the resultsshown in Table 4.1 are not isolated to just the IPMs implemented in SDPT3 orSeDuMi; similar results were also reported in the SCE-based IPM implemented byAndersen et al. [3]. For example, for the problem sched 50 50 orig, the IPM in [3]reported the values 0.9 and 0.002 for the maximum violation of certain primal boundconstraints and the dual constraints, respectively.

From Table 4.1, we have thus seen the performance of SCE-based IPMs for tworather different implementations in SDPT3 and SeDuMi. It is worthwhile to analyzethe performance of these implementations to isolate the factor contributing to thegood performance in one implementation, but not the other. On the first 10 SOCPproblems, nbxxx, nqlxxx and qsspxxx in the DIMACS library, SeDuMi performsmuch better than the IPM in SDPT3 in terms of accuracy. We hypothesize that Se-DuMi is able to obtain accurate approximate optimal solutions for these test problemsprimarily because of nice theoretical properties (existence of a strictly feasible point,and boundedness of solution set) of the HSD model. These problems contain linearvariables that are the results of modeling unrestricted variables as the difference oftwo nonnegative vectors. Consequently, the resulting primal SOCPs have unboundedsolution sets and the feasible regions of the dual SOCPs have empty interior. It should

12 Z. CAI AND K. C. TOH

come as no surprise that the IPM in SDPT3 has trouble solving such a problem tohigh accuracy since the ill-conditioning in the Schur complement matrix is made worseby the growing norm of the primal linear variables as the iterates approach optimality.On the other hand, for the IPM in SeDuMi, the ill-conditioning of the Schur comple-ment matrix is not amplified since the norm of the primal variables in the HSD modelstays bounded.

To verify the above hypothesis, we solve the nbxxx, nqlxxx and qsspxxx problemsagain in SDPT3, but at each IPM iteration, we trim the growth in the primal linearvariables, xu

+, xu−, arising from unrestricted variables xu using the following heuristic

[26]:

xu+ := xu

+ − 0.8 min(xu+, xu

−), xu− := xu

− − 0.8 min(xu+, xu

−).(4.2)

This modification does not change the original variable xu but it slows down thegrowth of xu

+, xu−. After these modified vectors have been obtained, we also modify

the associated dual linear variables zu+, zu

− as follows if µ ≤ 10−4:

(zu+)i :=

0.5µ

max(1, (xu+)i)

, (zu−)i :=

0.5µ

max(1, (xu−)i)

.(4.3)

Such a modification in zu+, zu

− ensures that they approach 0 at the same rate as µ,and thus prevents the dual problem from attaining the equality constraints in (D)prematurely.

The results shown in Table 4.2 supported our hypothesis. Observe that with theheuristic in (4.2) and (4.3) to control the growth of (xu

+)i/(zu+)i and (xu

−)i/(zu−)i,

the IPM in SDPT3 can also achieve accurate approximate solutions, just as the IPMbased on the HSD model in SeDuMi is able to achieve. It is surprising that such asimple heuristic to control the growth can result in such a dramatic improvement onthe achievable accuracy, even though the problems (P) and (D) in (1.1) do not havea strictly feasible point and the corresponding central path does not exist.

On other problems such as schedxxx, firxxx, and randxxx, the performanceof SDPT3 and SeDuMi is quite comparable in terms of accuracy attained, althoughSeDuMi is generally more accurate on the schedxxx problems, while SDPT3 performssomewhat better on the randxxx problems. On the firxxx problems, SDPT3 seemsto be more robust whereas SeDuMi runs into numerical difficulties quite early whensolving firL1Linfalph and firL2L1alph.

The empirical evidence of Table 4.1 shows that even though sophisticated numer-ical techniques used to solve the SCE in SeDuMi can help to achieve better accuracy,sometimes these techniques give limited improvement over simpler techniques em-ployed in SDPT3. On SOCP problems where the two solvers have vastly differentperformance in terms of accuracy, the difference can be attributed to the inherentIPM models used in the solvers rather than the numerical techniques employed tosolve the SCE. The conclusion we may draw here is that the SCE is generally inher-ently ill-conditioned, and if our wish is to compute the search direction of (2.5) tohigher accuracy, a new approach other than the SCE is necessary.

Solv

ing

SO

CP

via

ared

uced

augm

ented

system

13

Table 4.1

Accuracy attained by 2 SCE-based IPMs for solving SOCP problems. The timings reported are in seconds. A number of the form ”1.7-4” means 1.7 × 10−4. An

entry of the form “793x3” in the ”SOC” column means that there are 793 3-dimensional second order cones. The numbers under the ”LIN” column are the number

of linear variables.

SDPT3 SeDuMi

problem m SOC LIN φ Time relgap p-inf d-inf φ Time relgap p-inf d-inf

nb 123 793 × 3 4 -3.5 6.5 2.2-4 3.3-4 8.3 -9 -11.3 12.8 6.5-13 4.8-12 2.8-15

nb-L1 915 793 × 3 797 -4.9 13.0 7.1-7 1.4-5 9.6-11 -12.2 14.7 6.2-13 1.2-14 1.5-14

nb-L2 123 1 × 1677 ; 838 × 3 4 -5.5 10.9 5.1-8 3.1-6 8.9-12 -9.3 33.9 5.4-10 3.1-12 9.7-12

nb-L2-bessel 123 1 × 123 ; 838 × 3 4 -6.4 6.2 3.1-7 4.3-7 8.8-11 -10.5 20.1 3.3-11 8.0-14 1.7-13

nql30 3680 900 × 3 3602 -4.8 5.1 1.7-5 2.6-6 3.6-13 -10.2 2.5 6.8-11 3.4-11 3.4-11

nql60 14560 3600 × 3 14402 -6.5 21.1 3.4-7 1.3-7 2.8-12 -10.0 11.8 1.0-10 1.1-11 1.1-11

nql180 130080 32400 × 3 129602 -5.3 278.3 4.9-6 8.5-7 5.7-12 -9.2 229.8 5.8-10 1.9-11 1.9-11

qssp30 3691 1891 × 4 2 -8.7 4.5 2.0 -9 2.3-10 2.2-14 -11.1 4.5 7.1-13 4.8-12 7.5-12

qssp60 14581 7381 × 4 2 -7.9 24.1 1.4-8 1.7 -9 3.4-15 -10.6 26.5 3.3-12 1.7-11 2.7-11

qssp180 130141 65341 × 4 2 -7.6 493.1 2.8-8 4.8-10 1.7-14 -11.2 665.9 7.0-12 1.2-12 1.8-12

sched-50-50-o 2527 1 × 2474 ; 1 × 3 2502 -4.5 5.0 1.4-5 3.2-5 2.3-7 -7.0 6.2 1.0-12 1.0-7 4.0-14

sched-100-50-o 4844 1 × 4741 ; 1 × 3 5002 -3.7 11.7 1.8-4 4.2-6 9.7 -9 -6.0 14.4 2.9-13 1.0-6 1.4-12

sched-100-100-o 8338 1 × 8235 ; 1 × 3 10002 -2.8 20.8 1.2-3 1.6-3 1.5-5 -3.3 31.0 6.6-11 4.6-4 4.1-11

sched-200-100-o 18087 1 × 17884 ; 1 × 3 20002 -3.8 77.8 2.6-5 1.7-4 3.5-8 -3.9 66.4 4.8-12 1.2-4 2.3-11

sched-50-50-s 2526 1 × 2475 2502 -7.2 5.0 6.0-8 8.5 -9 4.5-15 -8.2 7.7 1.0-13 7.0 -9 1.1-14

sched-100-50-s 4843 1 × 4742 5002 -7.7 11.7 1.0-8 2.1-8 7.2-14 -8.9 21.3 1.1-11 1.3 -9 1.2-11

sched-100-100-s 8337 1 × 8236 10002 -6.2 21.2 5.5-8 7.0-7 2.8-14 -7.1 34.9 3.2-12 7.8-8 5.3-15

sched-200-100-s 18086 1 × 17885 20002 -6.5 61.3 3.1-7 3.2-7 2.2-13 -7.8 115.8 1.2-12 1.7-8 3.8-14

firL1Linfalph 3074 5844 × 3 -10.0 235.6 7.3-11 1.1-10 0.8-15 -4.7 286.8 5.2-7 1.8-5 0.0-16

firL1Linfeps 7088 4644 × 3 1 -9.9 255.5 1.1-10 1.2-10 6.8-16 -10.4 106.5 3.4-13 4.3-11 1.2-14

firL1 6223 5922 × 3 -10.1 612.6 3.0-11 7.3-11 1.0-15 -9.0 598.4 1.8-11 1.0 -9 8.1-12

firL2a 1002 1 × 1003 -10.3 35.1 5.0-11 7.2-16 0.8-16 -12.6 21.3 7.8-15 2.7-13 4.5-15

firL2L1alph 5868 1 × 3845 ; 1922 × 3 1 -10.1 117.6 8.0-11 1.2-11 6.2-16 -3.3 197.6 6.2-7 5.1-4 4.4-11

firL2L1eps 4124 1 × 203 ; 3922 × 3 -10.4 181.3 3.6-11 2.1-11 0.9-15 -9.3 198.9 1.3-11 4.8-10 1.1-11

firL2Linfalph 203 1 × 203 ; 2942 × 3 -10.0 127.7 7.8-11 9.3-11 7.4-16 -9.5 237.4 4.0-12 3.5-10 8.1-14

firL2Linfeps 6086 1 × 5885 ; 2942 × 3 -10.1 369.7 7.1-11 3.8-11 6.7-16 -9.1 262.2 8.1-10 5.7-10 0.0-16

firL2 102 1 × 103 -11.3 0.3 5.2-12 4.6-16 1.5-16 -13.1 0.1 1.3-15 7.3-14 3.3-15

firLinf 402 3962 × 3 -8.9 465.4 1.2 -9 1.1 -9 1.0-15 -9.3 936.5 6.6-13 5.4-10 2.4-13

rand200-300-1 200 20 × 15 -7.2 2.9 5.6-8 3.0 -9 6.4-15 -6.4 7.6 3.3-7 4.3-7 0.0-16

rand200-300-2 200 20 × 15 -6.3 3.0 5.6-7 1.4-8 5.6-14 -5.0 12.8 7.8-6 9.0-6 0.0-16

rand200-800-1 200 20 × 40 -6.1 5.5 8.0-7 1.6 -9 2.0-14 -5.0 25.4 1.0-5 1.1-6 0.0-16

rand200-800-2 200 20 × 40 -4.7 6.0 1.9-5 1.0-8 6.8-14 -5.8 56.2 8.8-7 1.6-6 0.0-16

rand400-800-1 400 40 × 20 -6.5 19.0 2.9-7 1.8-8 8.7-12 -5.1 29.0 7.1-6 2.7-6 0.0-16

rand400-800-2 400 40 × 20 -5.6 17.7 2.6-6 8.2 -9 1.3 -9 -4.5 56.6 3.5-5 2.2-5 0.0-16

rand700-1e3-1 700 70 × 15 -7.1 74.4 8.1-8 1.8-8 3.0-14 -5.7 142.6 1.6-6 1.9-6 0.0-16

rand700-1e3-2 700 70 × 15 -5.3 80.4 5.0-6 1.1-7 6.6-14 -4.6 199.4 1.4-5 2.6-5 0.0-16

rand1000-2e3 1000 100 × 20 -5.7 230.4 1.9-6 1.3-8 9.4-10 -5.0 600.0 7.4-6 9.8-6 0.0-16

rand1500-3e3 1500 150 × 20 -7.0 812.3 1.2-8 1.0-7 8.7-14 -7.0 2119.6 9.7-8 8.8-8 0.0-16

14 Z. CAI AND K. C. TOH

Table 4.2: Performance of the SCE-based IPM in SDPT3 in solving SOCPproblems with linear variables coming from unrestricted variables. The heuris-tics in (4.2) and (4.3) are applied at each IPM iteration.

SDPT3 SeDuMi

problem φ Time p-inf d-inf relgap φ Time p-inf d-inf relgap

nb-u -10.2 14.2 6.4-11 1.1-13 5.2-16 -11.1 13.6 6.5-13 8.4-12 0.0-16

nb-L1-u -10.0 28.2 9.9-11 1.1-11 2.2-16 -12.2 15.1 6.1-13 1.0-14 1.0-14

nb-L2-u -10.2 16.9 5.8-11 1.6-11 6.6-16 -9.3 33.8 5.4-10 3.1-12 6.5-12

nb-L2-bessel-u -10.2 12.9 6.7-11 3.3-11 3.3-16 -10.5 20.6 3.3-11 7.9-14 1.7-13

nql30-u -10.1 7.1 8.7-11 2.4-12 8.0-13 -10.2 3.5 6.8-11 3.4-11 2.8-11

nql60-u -10.4 29.9 4.4-11 2.0-11 2.8-13 -10.0 12.0 1.0-10 1.1-11 8.9-12

nql180-u -9.7 455.7 2.1-10 1.2-11 8.4-14 -9.2 263.8 5.8-10 1.9-11 1.1-11

qssp30-u -10.0 4.3 7.4-11 9.2-11 2.7-15 -11.3 4.0 7.1-13 4.8-12 5.2-12

qssp60-u -8.8 21.7 1.4 -9 1.8 -9 4.0-14 -10.8 26.5 3.3-12 1.7-11 1.7-11

qssp180-u -9.0 560.9 1.1 -9 6.7-10 9.5-15 -11.2 694.2 7.0-12 1.2-12 9.9-13

5. Reduced augmented equation. In this section, we present a new approachto compute the search direction via a potentially better-conditioned linear system ofequations. Based on the new approach, the accuracy of the computed search directionis expected to be better than that computed from the SCE when µ is small. In thisnew approach, we assume that the iterate (x, y, z) is sufficiently close to the centralpath so that the eigenvalues of F 2 separate into three distinct groups as described inSection 3.2.

In this approach, we start with the augmented equation in (2.6). By usingthe eigenvalue decomposition, F 2 = QDQT presented in Section 3.1, where Q =diag(Q1, . . . , QN ) and D = diag(Λ1, . . . , ΛN). We can diagonalize the (1,1) block andrewrite the augmented equation (2.6) as follows:

[−D AT

A 0

] [∆x

∆y

]=

[r

rp

],(5.1)

where

A = AQ, ∆x = QT ∆x, r = QT rx.(5.2)

The augmented equation (5.1) has dimension m + n, which is usually much largerthan m, the dimension of the SCE. We can try to reduce its size while overcomingsome of the undesirable features of the SCE such as the growth of ‖M‖ when µ ↓ 0.

Let the diagonal matrix D be partitioned into two parts as D = diag(D1, D2)with diag(D1) consisting of the small eigenvalues of F 2 of order Θ(µ) and diag(D2)consisting of the remaining eigenvalues of order Θ(1) or Θ(1/µ). We partition the

eigenvector matrix Q accordingly as Q = [Q(1) , Q(2)]. Then A is partitioned as A =

[A1 , A2] = [AQ(1) , AQ(2)] and r = [r1 ; r2] = [(Q(1))T rx ; (Q(2))T rx]. Similarly, ∆xis partitioned as ∆x = [∆x1 ; ∆x2] = [(Q(1))T ∆x ; (Q(2))T ∆x].

By substituting the above partitions into (5.1), and eliminating ∆x2, it is easy toshow that solving the system (5.1) is equivalent to solving the following:

[A2D

−12 AT

2 A1

AT1 −D1

] [∆y

∆x1

]=

[rp + A2D

−12 r2

r1

],(5.3)

Solving SOCP via a reduced augmented system 15

∆x2 = D−12 (AT

2 ∆y − r2) = D−12 (Q(2))T (AT ∆y − rx).(5.4)

By its construction, the coefficient matrix in (5.3) does not have large elements whenµ ↓ 0. But its (1,1) block is generally singular or nearly singular, especially when µ isclose to 0. Since a singular (1,1) block is not conducive for symmetric indefinitefactorization of the matrix or the construction of preconditioners for the matrix,we will construct an equivalent system with a (1,1) block that is less likely to besingular. Let E1 be a given positive definite diagonal matrix with the same dimensionas D1. Throughout this paper, we take E1 = I. Let S1 = E1 + D1. By addingA1S

−11 times the second block equation in (5.3) to the first block equation, we get

Adiag(S−1, D−12 )AT ∆y + A1S

−11 E1∆x1 = rp + Adiag(S−1, D−1

2 )r. This, together

with the second block equation in (5.3) but scaled by S−1/21 , we get the following

equivalent system:

[M A1S

−1/21

S−1/21 AT

1 −D1E−11

]

︸ ︷︷ ︸B

[∆y

S−1/21 E1∆x1

]=

[q

S−1/21 r1

],(5.5)

where

M = Adiag(S−11 , D−1

2 )AT , q = rp + Adiag(S−11 , D−1

2 )r.(5.6)

We call the system in (5.5) the reduced augmented equation (RAE). Note thatonce ∆y and ∆x1 are computed from (5.5) and ∆x2 is computed from (5.4), ∆x canbe recovered through the equation ∆x = Q[∆x1 ; ∆x2].

Remark 5.1. (a) If the matrix D1 is null, then the RAE (5.5) is reduced to theSCE (2.7).

(b) B is a quasi-definite matrix [8, 27]. Such a matrix has the nice property thatany symmetric reordering ΠBΠT has a “Cholesky factorization” LΛLT where Λ isdiagonal with both positive and negative diagonal elements.

Observe that the (1, 1) block, M , in (5.5) has the same structure as the Schur

complement matrix M = Adiag(D−11 , D−1

2 )AT . But for M , ‖diag(S−11 , D−1

2 )‖ =O(1), whereas for M , ‖diag(D−1

1 , D−12 )‖ = O(1/µ). Because of this difference, the

reduced augmented matrix B has bounded norm as µ ↓ 0, but ‖M‖ is generallyunbounded. Under certain conditions, B can be shown to have a condition numberthat is bounded independent of the normalized complementarity gap µ. The precisestatements are given in the following theorems.

Theorem 5.1. Suppose in (5.5) we use a partition such that diag(D1) consistof all the eigenvalues of F 2 of order Θ(µ). If the optimal solution of (1.1) satisfiesstrict complementarity, then ‖B‖ satisfies the following inequality: ‖B‖ = O(1) ‖A‖2.Thus ‖B‖ is bounded independent of µ (as µ ↓ 0).Proof. It is easy to see that

‖B‖ ≤√

2 max( ‖M‖ + ‖A1S−1/21 ‖, ‖S−1/2

1 AT1 ‖ + ‖D1E

−11 ‖ ).

Under the assumption that the optimal solution of (1.1) satisfies strict complemen-tarity, then as µ ↓ 0, ‖D1‖ ↓ 0, and ‖D−1

2 ‖ = O(1), so it is possible to find aconstant (independent of µ) τ ≥ 1 such that: max( ‖S−1

1 ‖, ‖D−12 ‖, ‖D1E

−11 ‖ ) ≤ τ .

16 Z. CAI AND K. C. TOH

Now ‖M‖ ≤ ‖A‖max(‖S−11 ‖, ‖D−1

2 ‖)‖A‖ ≤ τ‖A‖2 and ‖S−1/21 AT

1 ‖ = ‖A1S−1/21 ‖ ≤

τ‖A1‖, thus we have

‖B‖ ≤ τ√

2max( ‖A‖2 + ‖A1‖, ‖A1‖ + 1 ) ≤ τ√

2(‖A‖ + 1)2.

From here, the required result follows.

Lemma 5.2. The reduced augmented matrix B in (5.5) satisfies the followinginequality:

‖B−1‖ ≤ 2√

2max(‖M−1‖ , ‖W−1‖),

where W = BT1 M−1B1 + D1E

−11 with B1 = A1S

−1/21 .

Proof. From [19, p. 389], it can be deduced that

B−1 =

[M−1/2(I − P )M−1/2 M−1B1W

−1

W−1BT1 M−1 −W−1

],

where P = M−1/2B1W−1BT

1 M−1/2. Note that P satisfies the condition 0 P I,i.e., P and I − P are positive semidefinite. By the definition of W , we have 0 W−1/2BT

1 M−1B1W−1/2 I, and thus ‖M−1/2B1W

−1/2‖ ≤ 1. This implies that

‖M−1B1W−1‖ ≤ ‖M−1/2‖ ‖M−1/2B1W

−1/2‖ ‖W−1/2‖ ≤ max(‖M−1‖ , ‖W−1‖

).

It is easy to see that

‖B−1‖ ≤√

2max(‖M−1/2(I − P )M−1/2‖ + ‖M−1B1W

−1‖ , ‖W−1BT1 M−1‖ + ‖W−1‖

).

From here, the required result follows.

Theorem 5.3. Suppose in (5.5) we use a partition such that diag(D1) consist ofall the eigenvalues of F 2 of order Θ(µ). If the optimal solution of (1.1) satisfies strictcomplementarity and the primal and dual nondegeneracy conditions defined in [2],then the condition number of the coefficient matrix in (5.5) is bounded independent ofµ (as µ ↓ 0).

Proof. Let D2 be further partitioned into D2 = diag(D2a, D2b) where diag(D2a)and diag(D2b) consist of eigenvalues of F 2 of order Θ(1) and Θ(1/µ), respectively.

Let Q(2) and A2 be partitioned accordingly as Q(2) = [Q(2a), Q(2b)] and A2 =

[AQ(2a), AQ(2b)] =: [A2a, A2b]. By Theorems 20 and 21 in [1], dual nondegener-

acy implies that A1 = AQ(1) has full column rank and primal nondegeneracy impliesthat [A1, A2a] has full row rank. Since ‖M − [A1, A2a]diag(S−1

1 , D−12a )[A1 , A2a]T ‖ =

O(µ), thus σmin(M) is bounded away from 0 even when µ ↓ 0. This, together

with the fact that A1S−1/21 has full column rank, implies that the matrix W :=

S−1/21 AT

1 M−1A1S−1/21 +D1E

−11 has σmin(W ) bounded away from 0 even when µ ↓ 0.

By Lemma 5.2, ‖B−1‖ is bounded independent of µ. By Theorem 5.1, ‖B‖ is alsobounded independent of µ, and the required result follows.

Solving SOCP via a reduced augmented system 17

6. Reduced augmented equation and primal infeasibility. Let [ξ ; η] bethe residual vector for the computed solution of (5.5).

Lemma 6.1. Let u be the machine epsilon and l be the dimension of ∆x1. Suppose(l+m)u ≤ 1/2 and we use Gaussian elimination with partial pivoting (GEPP) to solve(5.5) to get the computed solution (∆y ; ∆x1), then the residual vector [ξ ; η] for thecomputed solution satisfies the following inequality:

‖(ξ ; η)‖∞ ≤ 4(l + m)3uρ ‖B‖∞‖(∆y ; ∆x1)‖∞where ρ is the growth factor associated with GEPP.Proof. This lemma follows from Theorem 9.5 in [12].

Remark 6.1. Theorem 5.1 stated that if strict complementarity holds at theoptimal solution, then ‖B‖∞ will not grow as µ ↓ 0 in contrast to ‖M‖, which usuallygrows proportionately to Θ(1/µ). Now because the growth factor ρ for GEPP is usuallyO(1), Lemma 6.1 implies that the residual norm ‖(ξ ; η)‖∞ will be maintained at somelevel proportional to u‖A‖2 even when µ ↓ 0.

Now we establish the relationship between the residual norm in solving (5.5)and the primal infeasibility associated with the search direction computed from theRAE approach. Suppose that in computing ∆x2 from (5.4), a residual vector δ isintroduced, i.e.,

∆x2 = D−12 (Q(2))T (AT ∆y − r) − δ.

Then we have the following lemma for the primal infeasibility of the next iterate.Lemma 6.2. Suppose ∆x is computed from the RAE approach. Then the primal

infeasibility ‖r+p ‖ for the next iterate x+ = x + α∆x, α ∈ [0, 1], satisfies the following

inequality:

‖r+p ‖ ≤ (1 − α)‖rp‖ + α‖ξ + A2δ − A1S

−1/21 η‖.

Proof. The proof is quite routine and we omit it.Remark 6.2. From Lemma 6.2, we see that if the RAE returns a small residual

norm, then the primal infeasibility of the next iterate would not be seriously worsenedby the residual norm. From Theorem 5.1 and Lemma 6.1, we expect the residual norm‖[ξ; η]‖ to be small since the upper bound on ‖B‖ is independent of µ. Also, since byits construction, D−1

2 does not have large elements, ‖δ‖ is expected to be small aswell.

Figure 6.1 shows the convergence behavior of the IPM in SDPT3, but withsearch directions computed from the RAE (5.5) for the problems ran200 800 1 andsched 50 50 orig. As can be seen from the relative primal infeasibility curves, theRAE approach is more stable than the SCE approach. It is worth noting that underthe new approach, the solver is able to deliver 10 digits of accuracy, i.e. φ ≤ −10.This is significantly better than the accuracy φ ≈ −6 attained by the SCE approach.Note that we use a partition such that eigenvalues of F 2 that are smaller than 10−3

are put in D1.In Table 7.1, we show the norms ‖B‖, ‖B−1‖ and the residual norm in solving the

RAE (5.5) for the last few IPM iterations in solving the problem2 rand200 800 1 andsched 50 50 orig. Observe that ‖B‖ and κ(B) do not grow when µ ↓ 0 in contrastto ‖M‖ and κ(M) in Table 3.1. The residual norm for the computed solution of (5.5)remains small throughout, and in accordance with Lemma 6.1, the residual norm isapproximately equal to u‖B‖ times the norm of the computed solution. By Lemma6.2, the small residual norm in solving the RAE explains why the primal infeasibilitycomputed from the RAE approach does not deteriorate as in the SCE approach.

18 Z. CAI AND K. C. TOH

2 4 6 8 10 12 14 1610

−15

10−10

10−5

100

iteration

rand200_800_1: RAE approach

relgap

p−inf

d−inf

5 10 15 20 25 30 35 40

10−10

10−5

100

105

iteration

sched_50_50_orig: RAE approach

relgap

d−inf

p−inf

Fig. 6.1. Same as Figure 3.1 but for the RAE approach in computing the search directionsfor the problems rand200 800 1 and sched 50 50 orig. Notice that the primal infeasibility doesnot deteriorate when the iterates approach optimality. Both problems are primal and dual non-degenerate, and strict complementarity holds at optimality.

7. Computational issues. The theoretical analysis in the last section indicatesthat the RAE approach is potentially more stable than the standard SCE approach,but the trade-off is that the former needs to solve a larger indefinite linear system.Thus, how to efficiently solve (5.5) is one of our major concerns in the implementation.

In forming the reduced augmented matrix B, those operations involving Q (theeigenvector matrix of F 2) must be handled carefully by exploiting the structure of Qto avoid incurring significant storage and computational cost. Also, the sparsity ofAAT must be properly preserved when computing M .

Table 7.1

Condition number of the reduced augmented matrix B associated with the last few IPM iterationsfor solving the SOCP problem rand200 800 1 and sched 50 50 orig. The maximum number of

columns in A1 for the former problem is 19, and that for the latter is 82.

Iter ‖B‖ ‖B−1‖ xT z/N ‖[∆y; ∆x1]‖ residualnorm

‖rp‖‖rp‖

‖B‖‖[∆y; ∆x1]‖urand200 800 1

12 3.7e+11 2.7e+02 1.3e-09 4.6e-04 6.9e-09 1.8e-08 4.7e-01

13 3.4e+11 2.8e+02 1.9e-10 2.3e-04 4.3e-09 8.1e-09 4.7e-01

14 2.6e+11 3.7e+02 2.5e-11 1.0e-04 1.2e-09 2.9e-09 4.9e-01

15 2.3e+11 4.2e+02 4.0e-12 3.6e-05 3.4e-10 9.4e-10 5.1e-01

16 2.0e+11 5.2e+02 5.6e-13 1.3e-05 1.4e-10 4.8e-10 8.6e-01

sched 50 50 orig

33 1.1e+08 3.9e+04 7.9e-07 1.4e-02 9.3e-13 1.2e-12 3.6e-03

34 6.2e+07 1.4e+04 1.6e-07 2.1e-03 3.9e-14 1.9e-11 6.8e-01

35 6.2e+07 2.7e+04 3.7e-08 1.7e-03 8.4e-15 1.2e-12 5.0e-02

36 6.2e+07 1.5e+04 7.5e-09 3.8e-04 2.8e-15 2.7e-11 5.1e+0

37 6.2e+07 2.2e+04 2.8e-09 2.2e-04 6.5e-16 2.5e-11 8.1e+0

7.1. Computations involving Q. The operations involving Q in assemblingthe RAE (5.5) are as follows:

Solving SOCP via a reduced augmented system 19

• Computation of the (1,1) block M = AQdiag(S−11 , D−1

2 )QT AT ;

• Computation of the (1,2) block A1 = AQ(1) and the right hand side vector r = QT rx.

To carry out the above operations efficiently, we need to derive an explicit formulafor Q to facilitate such calculations. Recall the eigenvector matrix Qi (3.1) associatedwith the ith second order cone. To get an explicit description of Qi, we need toconstruct the Householder matrix Hi explicitly. Without going into the algebraicdetails, the precise form of Hi is given as follows:

Hi = I − hihTi , hi :=

[h0

i

hi

]=

1

τi

[τ2i sign(g0

i )

gi

]∈ IRni−1, τi :=

√1 + |g0

i |.(7.1)

With some algebraic manipulations, the eigenvector Qi can be rewritten in theform given in the next lemma.

Lemma 7.1. Let βi = −sign(h0i )/

√2. We have Qi = diag(Ki, I) − uiv

Ti ,where

Ki =

[ − 1√2

1√2

βi βi

], ui =

[0

hi

], vi =

βih0i

βih0i

hi

.(7.2)

Proof. Note that by construction, the first column of Hi is given by −sign(g0i )gi. Let

α = 1√2

+ sign(g0i ). From (3.1), we have

Qi =

− 1√2

1√2

0 · · · 0

1√2gi αgi 0 · · · 0

+

0 0

0 I − hihTi

=

− 1√2

1√2

0 . . . 0

− sign(g0

i)

√2

− sign(g0

i)

√2

− 1 0 . . . 0

......

......

0 0 0 . . . 0

+

[0 0

0 I

]−

[0

hi

]

− τi√2

h0i − ατi

hi

T

.

It is readily shown that h0i − ατi = −τi/

√2 = βih

0i . Now it is easy to see that the

required results hold.

Observe that each Qi is a rank-one perturbation of a highly sparse block di-agonal matrix. Based on the above lemma, those operations listed at the begin-ning of this subsection, except the first one, can be computed straightforwardly.To compute the matrix M , we have to further analyze the structure of the matrixQidiag(S−1

1i , D−12i )QT

i .Let Gi = diag(S−1

1i , D−12i ) and Σi = diag(Ki, I), then Qi = Σi − uiv

Ti and

Qi Gi QTi = Σi Gi ΣT

i − ΣiGiviuTi − uiv

Ti GiΣ

Ti + uiv

Ti Gi viu

Ti . By setting ρi =

vTi Givi and vi = ΣiGivi/

√ρi, we have Qi Gi QT

i = ΣiGiΣTi + lil

Ti − viv

Ti , where

li = vi −√ρiui. Thus each component matrix Mi in M =

∑Ni=1 Mi can be expressed

as:

Mi = AiQiGiQTi AT

i = Ai (ΣiGiΣTi )AT

i + (Aili)(Aili)T − (Aivi)(Aivi)

T .(7.3)

20 Z. CAI AND K. C. TOH

Since ΣiGiΣTi is a highly sparse block diagonal matrix, Mi is a symmetric rank-

two perturbation to a sparse matrix if AiATi is sparse. Hence, the computational

complexity of M is only slightly more expensive than that for the Schur complementmatrix M .

7.2. Handling dense columns. Let Σ = diag(Σ1, . . . , ΣN ), and

Al = [A1l1 , . . . , AN lN ] , Av = [A1v1 , . . . , AN vN ] .

Then it is readily shown that

M = AΣdiag(S−11 , D−1

2 )ΣT AT + AlATl − AvAT

v .(7.4)

If AAT is sparse, then the first matrix in (7.4) is sparse as well. For an SOCPproblem where all the cones are low dimensional, typically the matrices Al and Av

are also sparse. In that case, the RAE (5.5) may be solved directly. However, if highdimensional cones exist, then Al and Av invariably contain dense columns. Moreover,when A is sparse but has dense columns, AAT will also be dense. In order to preservethe sparsity in M , it is necessary to handle the dense columns separately when theyexist.

Let P1 be the dense columns in AΣdiag(S−1/21 , D

−1/22 ) and Al, and P2 be the

dense columns in Av. Let Ms = M − P1PT1 + P2P

T2 be the “sparse part” of M . It is

well known that by introducing the following auxiliary variables, t1 = PT1 ∆y, t2 =

−PT2 ∆y, the dense columns can be removed from M ; see [4]. The precise form of the

RAE (5.5) with dense column handling is as follows:

[Ms U

UT −C

] [∆y ; S

−1/21 E1∆x1 ; t1 ; t2

]=

[q ; S

−1/21 r1 ; 0 ; 0

],(7.5)

where q is defined in (5.6) and U =[A1S

−1/21 , P1, P2

], C = diag(D1E

−11 , I1,−I2).

Here I1, I2 are identity matrices.

7.3. Direct solvers for symmetric indefinite systems. Solving the sparsesymmetric indefinite system (7.5) is one of the most expensive step at each IPMiteration. Thus, it is critical that the solver used must be as efficient as possible.

We consider two methods for solving (7.5). The first is the Schur complementmethod, which is also equivalent to the Sherman-Morrison-Woodbury formula. Thesecond is the LDLT factorization implemented in MA47 [18]. Each of these methodshas its own advantages under different circumstances.

Schur complement method. This method is widely used for dense columnhandling in IPM implementations; see [4] and the references therein. It uses the

sparse matrix Ms as the pivoting matrix to perform block eliminations in (7.5). It isreadily shown that solving (7.5) is equivalent to solving the following systems:

(UT M−1

s U + C) [

S−1/21 E1∆x1 ; t1 ; t2

]= UT M−1

s q −[

S−1/21 r1 ; 0 ; 0

](7.6)

∆y = M−1s q − M−1

s U[

S−1/21 E1∆x1 ; t1 ; t2

].

Note that since M is symmetric positive definite, its “sparse part”, Ms, is typicallyalso positive definite if the number of dense columns removed from M is small. If Ms

Solving SOCP via a reduced augmented system 21

is indeed positive definite, then (7.6) can be solved by Cholesky or sparse Choleskyfactorization. As mentioned before, highly efficient and optimized Cholesky solvers arereadily available in the public domain. Another advantage of the Schur complementmethod is that the symbolic factorization of Ms and the pivoting order of the Choleskyfactorization needs only be computed once or twice during the initial phase of the IPMiteration and it can be re-used for subsequent IPM iterations even when the partitionin D changes.

But the Schur complement method does have a major disadvantage in that thematrix UT M−1

s U+C is typically dense. This can lead to a huge computational burdenwhen U has a large number of columns, say, more than a few hundreds. Furthermore,the Schur complement method is numerically less stable than a method that solves(7.5) directly.

Roughly speaking, the Schur complement method is best suited for problemswith U having a small number of columns. When U has a large number of columns orwhen Ms is not positive definite, we have to solve (7.5) directly by the second methoddescribed below.

MA47. MA47 is a direct solver developed by Reid and Duff [18] for sparsesymmetric indefinite systems. This is perhaps the only publicly available state-of-the-art direct solver for sparse symmetric indefinite systems. It appears not to be asefficient as the sparse Cholesky codes of Ng and Peyton [16].

The MA47 solver implements the multi-frontal sparse Gaussian elimination de-scribed in [7]. In the algorithm, the pivots used are not limited only to 1× 1 diagonalpivots but also 2 × 2 block diagonal pivots. The solver performs a pre-factorizationphase (called symbolic factorization) on the coefficient matrix to determine a pivotingorder so as to minimize fill-ins. In the actual factorization process, this pivoting ordermay be modified to obtain better numerical stability. Note that in sparse Choleskyfactorization, the pivoting order is not modified after the symbolic factorization phase.Because significant overhead may be incurred when the pivoting order is modified inthe factorization process, running MA47 is sometimes much more expensive than thesparse Cholesky routine of Ng and Peyton on matrices with the same dimensions andsparsity patterns.

The advantage of using MA47 to solve (7.5) is that it does not introduce a fullydense matrix in the solution process. Thus it is more suitable for SOCP problemswith U having a relatively large number of columns.

However, the MA47 method does have a disadvantage in that the symbolic fac-torization of the reduced augmented matrix needs to be re-computed whenever thepartition in D changes.

7.4. Partitioning Strategy. As shown in Section 6, the RAE approach forcomputing the search directions has the potential to overcome certain numerical insta-bilities encountered in the SCE approach. The RAE was derived from the augmentedequation (2.6) by modifying the part of the coefficient matrix involving the smalleigenvalues of F 2. Here we will describe the partition we use in D = diag(D1, D2).

The choice of D1 is dictated by the need to strike a balance between our desire tocompute more accurate search directions and to minimize the size of the RAE to besolved. For computational efficiency, it is better to have as few columns in the matrixU (7.5) as possible, thus suggesting that the threshold for labelling an eigenvalue as”small” should be low. But to obtain better accuracy, it is beneficial to partitioneigenvalues that are smaller than, say 10−3, into D1 to improve the conditioning ofthe reduced augmented matrix.

22 Z. CAI AND K. C. TOH

With due consideration in balancing the two issues just mentioned, we adopt ahybrid strategy in computing the search direction at each IPM iteration. If κ(F 2) ≥106, put the eigenvalues of F 2 that are smaller than 10−3 in D1, and the rest in D2;Otherwise, put all the eigenvalues of F 2 in D2.

Some of our test problems also contain linear blocks (i.e., cones with dimensionsni = 1). In this case, F 2

i = zi/xi is a scalar, and we put F 2i in D1 if it is smaller than

10−3, otherwise, we put it in D2.As noted in Remark 3.1, when A1 has full row rank (for which a necessary con-

dition is that the number of small eigenvalues put into D1 is at least m), the Schurcomplement matrix M is not highly ill-conditioned, and it is not necessary to use theRAE approach to compute the search directions. When such a situation occurs, weuse the SCE approach.

8. Numerical experiments. The RAE (5.5) or (7.5) is more expensive to solvethan the SCE (2.7) because it is larger in size. As we have discussed in the last section,we can try to minimize the additional computational cost by a judicious choice of thesolver used. If the number of columns in U is small, then using the Schur complementmethod to solve (7.5) should not be much more expensive than solving the SCE. Weadopt the following heuristic rule to select the solver used to solve (7.5). If the numberof columns in U is less than 200, we use the Schur complement method; otherwise,we use the MA47 method.

The RAE approach is implemented in Matlab based on the IPM in SDPT3,version 3.1; see [25]. But the search direction at each iteration is computed based onthe RAE (7.5). We use the same stopping criteria mentioned in Section 4. Again, thenumerical results are obtained from a Pentium IV 2.4GHz PC with 1G RAM.

We consider the same SOCP problems in Section 4. But in order to focus onthe comparison between the SCE and RAE approaches without the complication ofunbounded primal solution sets, we exclude the nbxxx, nqlxxx, and qsspxxx problemsfrom the numerical experiments in this section. Our major concern in the experimentsare efficiency and accuracy. We measure efficiency by the total CPU time taken; whileaccuracy is again measured by accuracy exponent defined in (4.1).

The numerical results for the RAE-based IPM is presented in Table 9.1. In thetable, Titer denotes the average CPU time taken per iteration. For the RAE-basedIPM, the number of IPM iterations taken for each problem is given under the column“iter”. The number in each bracket gives the number of iterations using the RAEapproach. The total CPU time taken to solve each problem is given under the column“Time”. The number in each bracket gives the CPU time taken by the iterations usingthe RAE approach.

The numerical results in Table 4.1 show that the SCE-based IPMs may not deliverapproximate optimal solutions with small primal infeasibilities. In Table 9.1, we seethat the RAE-based IPM can drive the primal infeasibilities of all the problems to alevel of 10−9 or smaller. For the schedxxx and randxxx problem sets, both the SCE-based IPMs in SDPT3 and SeDuMi cannot deliver accurate approximate solutionswhere the accuracies attained range from φ = −2.5 to φ = −8.9 for the schedxxx setand from φ = −4.5 to φ = −7.6 for the randxxx set. The RAE-based IPM, however,can achieve solutions with accuracy φ ≤ −9.1 for all the problems in these 2 sets. Theimprovement in the attainable accuracy is more than 5 orders of magnitudes in somecases. For the firxxx problems, the SCE approach can already produce accurateapproximate solutions, and the RAE approach produces comparable accuracies.

The good performance in terms of accuracy of the RAE-based IPM on the schedxxx

Solving SOCP via a reduced augmented system 23

and randxxx problem sets is consistent with the theoretical results established in Sec-tion 6. The SOCP problems in the schedxxx set are primal and dual nondegenerate,and strict complementarity holds at optimality. For the randxxx set, all the problemsare primal non-degenerate, but 4 of the problems are dual degenerate. It is interestingto note that dual degeneracy does not seem to affect the performance of RAE on thesedegenerate problems. This fact is consistent with the observation we made in Remark6.2.

By Theorem 5.3, the condition number of the reduced augmented matrix for theproblems in schedxxx set is bounded when µ ↓ 0. But as noted in Remark 3.2, strictcomplementarity, and primal and dual nondegeneracy in an SOCP does not necessarilyimply that the associated Schur complement matrix has bounded condition numberswhen µ ↓ 0. The numerical results produced by the schedxxx problems concretelyshow the difference in numerical stability between the SCE and RAE approaches.

From the average CPU time taken per IPM iteration for the RAE and SCEapproaches in Table 9.1, we see that the RAE approach is reasonably efficient in thatthe ratio (compared with SDPT3) is at most 6.0 for all the test problems, and 78%of them have ratios between 1.0 and 2.0.

The objective values obtained by the RAE-based IPM are given in Table 9.2.

As we are able to compute rather accurate approximate solutions for (1.1), itis worthwhile to gather information such as primal and dual degeneracy, and strictcomplementarity for some of the smaller SOCP problems we have considered in thispaper. Such information is given in Table 9.3. We note that the degeneracies of theproblems are determined by computing the numerical row and column rank (via theMatlab command rank) of the matrices in Theorems 20 and 21 in [1], respectively.

9. Conclusion. We analyzed the accuracy of the search direction computedfrom the SCE approach, and how the residual norm in the computed solution affectsthe primal infeasibility and hence the achievable accuracy in the approximate optimalsolution.

We also discussed the factors contributing to the good numerical performance ofthe very well implemented SCE-based IPM in the software SeDuMi.

A reduced augmented equation is proposed to compute the search direction ateach IPM iteration when the SCE cannot be solved to sufficient accuracy. The pro-posed RAE approach can improve the robustness of IPM solvers for SOCP. It canbe implemented efficiently by carefully preserving the sparsity structure in the prob-lem data. Numerical results show that the new approach can produce more accurateapproximate optimal solutions compared to the SCE approach.

Acknowledgment. The authors are grateful to Professor Robert Freund forvaluable suggestions on the manuscript. The authors also thank the Associate Editor,Professor M. J. Todd, and the Referees for many helpful comments and suggestionsto improve the paper.

REFERENCES

[1] F. Alizadeh and D. Goldfarb, Second-order cone programming, Mathematical Programming,95 (2003), pp. 3–51.

[2] F. Alizadeh and S.H. Schmieta, Optimization with Semi-definite, Quadratic and Linear Con-straints, Report 23-97, Rutgers Center for Operations Research, Rutgers University, 1997.Available from http://rutcor.rutgers.edu/pub/rrr/reports97/23.ps

[3] E.D. Andersen, C. Roos, and T. Terlaky, On implementing a primal-dual interior-point method

24 Z. CAI AND K. C. TOH

for conic quadratic optimization, Mathematical Programming, Series B, 95 (2003),pp. 249-277.

[4] K.D. Andersen, A modified Schur complement method for handling dense columns in interiorpoint methods for linear programming, ACM Transactions on Mathematical Software, 22(1996), pp. 348–356.

[5] K.M. Tsui, S.C., Chan, and K.S. Yeung, Design of FIR digital filters with prescribed flatnessand peak error constraints using second order cone programming, IEEE Transactions onCircuits and Systems II, 52 (2005), pp. 601–605.

[6] H. Ciria and J. Peraire, Computation of upper and lower bounds in limit analysis using second-order cone programming and mesh adaptivity, 9th ASCE Specialty Conference on Proba-bilistic Mechanics and Structural Reliability, 2004.

[7] I.S. Duff and J.K. Reid, The multifrontal solution of indefinite sparse symmetric linear equa-tions, ACM Transactions on Mathematical Software, 9 (1983), pp. 302–325.

[8] A. George, and K.H. Ikramov, On the condition of symmetric quasi-definite matrices, SIAMJ. Matrix Analysis and Applications, 21 (2000), pp. 970–977.

[9] D. Goldfarb, and K. Scheinberg, A product-form Cholesky factorization implementation of aninterior-point method for second order cone programming, Mathematical Programming, toappear.

[10] G. Golub and C. Van Loan, Matrix Computations, 3rd ed., Johns Hopkins University Press,Baltimore, USA, 1996.

[11] M. Gu, Primal-dual interior-point methods for semidefinite programming in finite precision,SIAM J. Optimization, 10 (2000), pp. 462–502.

[12] N.J. Higham, Accuracy and stability of numerical algorithms, SIAM, Philadelphia, 1996.[13] V. Kovacevic-Vujcic and M.D. Asic, Stabilization of interior-point methods for linear program-

ming, Computational Optimzation and Applications, 14 (1999), pp. 331–346.[14] M. S. Lobo, L. Vandenberghe, S. Boyd and H. Lebret, Applications of Second-order Cone

Programming, Linear Algebra Appl., 284 (1998), pp.193–228.[15] R.D.C. Monteiro and T. Tsuchiya, Polynomial Convergence of Primal-Dual Algorithms for

the Second-Order Cone Program Based on the MZ-Family of Directions, MathematicalProgramming, 88 (2000), pp. 61–83.

[16] E.G. Ng and B.W. Peyton, Block sparse Cholesky algorithms on advanced uniprocessor com-puters, SIAM J. Scientific and Statistical Computing, 14 (1993), pp. 1034–1056.

[17] G. Pataki and S.H. Schmieta, The DIMACS Library of Mixed Semidefinite-Quadratic-LinearPrograms,http://dimacs.rutgers.edu/Challenges/Seventh/Instances/

[18] J. Reid, and I.S. Duff, MA47, a Fortran code for direct solution of indefinite sparse sym-metric linear systems, Report RAL-95-001, Rutherford Appleton Laboratory, Oxfordshire,England, January, 1995.

[19] Y. Saad, Iterative Methods for Sparse Linear Systems, PWS Publishing Company, Boston,1996.

[20] D. Scholnik and J. Coleman, An FIR Filter Optimization Toolbox for Matlab 5 & 6, availablefrom http://www.csee.umbc.edu/~dschol2/opt.html

[21] J.F. Sturm, Using SeDuMi 1.02, a Matlab toolbox for optimization over symmetric cones,Optimization Methods and Software, 11 & 12 (1999), pp. 625–653.

[22] J.F. Sturm, Implementation of interior point methods for mixed semidefinite and second ordercone optimization problems, Optimization Methods and Software, 17 (2002), pp. 1105–1154.

[23] J.F. Sturm, Avoiding numerical cancelation in the interior point method for solving semidefi-nite programs, Mathematical Programming, Series B, 95 (2003), pp. 219–247.

[24] K.C. Toh, M.J. Todd, and R.H. Tutuncu, SDPT3- A Matlab Software package for SemidefiniteProgramming, Optimization Methods and Software, 11 (1999), pp. 545–581.

[25] R.H. Tutuncu, K.C. Toh and M.J. Todd, Solving Semidefinite-Quadratic-Linear ProgrammingUsing SDPT3, Mathematical Programming, Series B, 95 (2003), pp. 189–217.

[26] K.C. Toh, R.H. Tutuncu, and M.J. Todd, On the implementation of SDPT3 (version 3.1) –a Matlab software package for semidefinite-quadratic-linear programming, invited paper,2004 IEEE Conference on Computer-Aided Control System Design, Taipei, Taiwan.

[27] R.J. Vanderbei, Symmetric Quasidefinite Matrices, SIAM J. Optimization, 5 (1995), pp. 100–113.

[28] S.J. Wright, Stability of linear equations solvers in interior-point methods, SIAM J. MatrixAnalysis and Applications, 16 (1995), pp. 1287–1307.

[29] S.J. Wright, Stability of augmented system factorizations in interior-point methods, SIAM J.Matrix Analysis and Applications, 18 (1997), pp. 191–222.

Solving SOCP via a reduced augmented system 25

Table 9.1

A comparison between 2 SCE-based IPMs and the RAE-based IPM for solving SOCP problems.

The last column in the table gives the maximum number of columns in the matrix U in (7.5). The

numbers in round brackets are those taken by the iterations using the RAE approach.

SDPT3 SeDuMi RAE approach

problem φ Titer φ Titer φ Titer iter Time relgap p-inf d-inf nc(U)

sched-50-50-o -4.5 0.18 -7.0 0.17 -10.0 0.26 37 (27) 9.7 (7.2) 10.0-11 1.2-12 1.2-11 86

sched-100-50-o -4.3 0.39 -6.0 0.37 -10.6 0.90 41 (26) 36.9 (30.3) 2.6-11 1.7-11 1.7-11 494

sched-100-100-o -2.5 0.76 -3.3 0.71 -9.1 1.94 41 (24) 79.5 (65.2) 7.5-10 7.1-11 1.2-10 245

sched-200-100-o -4.0 2.08 -3.9 1.60 -10.1 7.30 49 (38) 357.8 (323.4) 8.2-11 4.7-11 1.9-11 578

sched-50-50-s -6.2 0.17 -8.2 0.24 -10.6 0.28 31 (26) 8.6 (7.3) 2.6-11 4.6-12 4.2-15 85

sched-100-50-s -7.0 0.41 -8.9 0.41 -10.4 1.15 33 (24) 37.9 (33.2) 4.4-11 9.8-12 6.6-14 495

sched-100-100-s -5.9 0.76 -7.1 0.63 -9.4 2.12 33 (28) 70.0 (63.7) 3.4-11 4.2-10 2.5-14 254

sched-200-100-s -6.2 2.23 -7.8 2.33 -10.5 13.32 31 (25) 412.9 (388.8) 2.9-11 3.8-12 2.0-13 578

firL1Linfalph -9.9 6.92 -4.7 10.71 -9.9 8.23 35 (18) 288.0 (153.8) 4.5-11 1.4-10 0.8-15 10

firL1Linfeps -10.2 6.67 -10.4 3.83 -10.2 8.22 39 (32) 320.7 (232.9) 5.8-11 1.6-12 7.3-16 646

firL1 -10.1 26.65 -9.0 24.35 -10.1 28.00 23 (0) 644.0 (0.0) 2.3-11 7.3-11 1.0-15 0

firL2a -10.3 4.37 -12.6 4.29 -10.3 4.57 8 (0) 36.6 (0.0) 5.0-11 0.8-15 0.9-16 2

firL2L1alph -10.1 5.13 -3.3 5.24 -10.1 5.27 23 (18) 121.1 (72.4) 8.0-11 1.6-11 6.2-16 4

firL2L1eps -10.4 9.56 -9.3 9.51 -10.5 13.86 19 (13) 263.3 (185.0) 3.4-11 2.4-11 0.9-15 1

firL2Linfalph -10.1 3.79 -9.5 8.88 -10.1 6.60 34 (29) 224.5 (201.6) 7.8-11 6.4-14 7.3-16 19

firL2Linfeps -10.2 16.86 -9.1 7.74 -10.1 17.23 22 (0) 379.1 (0.0) 7.1-11 4.2-12 6.7-16 2

firL2 -11.3 0.03 -13.1 0.02 -11.3 0.04 8 ( 3) 0.3 (0.1) 5.2-12 3.3-16 2.0-16 1

firLinf -8.9 17.30 -9.3 34.48 -8.9 22.40 29 (18) 649.6 (443.4) 3.6-10 1.3 -9 1.0-15 170

rand200-300-1 -7.6 0.22 -6.4 0.59 -10.5 0.26 14 ( 5) 3.6 (1.5) 3.3-11 1.9-13 5.9-15 47

rand200-300-2 -6.0 0.23 -5.0 1.07 -10.1 0.28 16 ( 6) 4.5 (2.0) 8.1-11 4.6-14 5.3-14 62

rand200-800-1 -5.6 0.45 -5.0 2.11 -10.1 0.54 16 ( 8) 8.7 (4.7) 7.6-11 1.2-14 1.8-14 19

rand200-800-2 -4.9 0.45 -5.8 4.67 -10.1 0.55 18 ( 8) 9.9 (4.8) 8.1-11 1.0-13 7.6-14 19

rand400-800-1 -5.8 1.60 -5.1 2.63 -10.3 1.80 14 ( 6) 25.3 (11.5) 5.3-11 5.4-14 2.6-14 40

rand400-800-2 -5.8 1.68 -4.5 5.66 -10.4 1.80 15 ( 6) 27.0 (11.6) 3.9-11 2.0-13 5.1-14 40

rand700-1e3-1 -6.1 5.67 -5.7 10.93 -10.1 6.42 17 ( 8) 109.1 (55.1) 7.3-11 4.5-14 2.4-14 123

rand700-1e3-2 -5.3 5.84 -4.6 18.07 -10.1 6.59 20 (10) 131.8 (71.6) 8.6-11 5.2-13 5.8-14 151

rand1000-2e3 -5.7 20.95 -5.0 50.00 -11.3 24.66 14 ( 6) 345.3 (151.3) 4.7-12 1.6-13 6.9-14 100

rand1500-3e3 -7.0 67.69 -7.0 192.69 -10.1 69.27 15 ( 6) 1039.0 (388.9) 8.5-11 6.9-13 9.2-14 450

[30] Y. Ye, M.J. Todd, and S. Mizuno, An O(√

nL)-iteration homogeneous and self-dual linearprogramming algorithm, Math. Oper. Res., 19 (1994), pp. 53–67

26 Z. CAI AND K. C. TOH

Table 9.2

Primal and dual objective values obtained by the IPM using the RAE approach.

Problemprimal

objectivedual

objectiveProblem

primalobjective

dualobjective

sched-50-50-o 2.6673000979 4 2.6673000977 4 firL2Linfalph -7.0591166471 -3 -7.0591167258 -3

sched-100-50-o 1.8188993937 5 1.8188993936 5 firL2Linfeps -1.4892049051 -3 -1.4892049762 -3

sched-100-100-o 7.1736778669 5 7.1736778615 5 firL2 -3.1186645862 -3 -3.1186645914 -3

sched-200-100-o 1.4136044650 5 1.4136044649 5 firLinf -1.0068176528 -2 -1.0068176895 -2

sched-50-50-s 7.8520384401 0 7.8520384399 0 rand200-300-1 -1.5094030119 2 -1.5094030119 2

sched-100-50-s 6.7165031103 1 6.7165031100 1 rand200-300-2 -1.2861024800 2 -1.2861024801 2

sched-100-100-s 2.7330785593 1 2.7330785592 1 rand200-800-1 1.8086048337 0 1.8086048335 0

sched-200-100-s 5.1811961028 1 5.1811961027 1 rand200-800-2 -2.3277765222 1 -2.3277765220 1

firL1Linfalph -3.0673166232 -3 -3.0673166686 -3 rand400-800-1 6.6607764189 0 6.6607764185 0

firL1Linfeps -2.7112896665 -3 -2.7112897249 -3 rand400-800-2 6.3708631137 1 6.3708631135 1

firL1 -2.9257813804 -4 -2.9257816083 -4 rand700-1e3-1 -7.1501954797 1 -7.1501954802 1

firL2a -7.1457742547 -4 -7.1457747536 -4 rand700-1e3-2 -5.5374169002 1 -5.5374169007 1

firL2L1alph -5.7634914619 -5 -5.7634994782 -5 rand1000-2e3 -2.4138366508 4 -2.4138366508 4

firL2L1eps -8.4481294535 -4 -8.4481297976 -4 rand1500-3e3 1.7396653464 4 1.7396653465 4

Table 9.3

Primal nondegeneracy (“p.n.d”) and dual nondegeneracy (“d.n.d”), and strict complementarity(“s.c.”) information of approximate solutions of some SOCPs. A “1” means true and a “0” meansfalse. A number of the form (34/35) in the second column means that at the computed approximate

optimal solution, the column rank of A1 is 34, and the number of columns in A1 is 35.

Problem p.n.d d.n.d s.c. Problem p.n.d d.n.d s.c.sched-50-50-orig 1 1 (79/79) 1 rand200 800 1 1 1 (19/19) 1

sched-50-50-scaled 1 1 (83/83) 1 rand200 800 2 1 1 (19/19) 1

firL2a 1 1 (1/1) 1 rand400 800 1 1 1 (40/40) 1

firL2Linfalph 1 1 (15/15) 1 rand400 800 2 1 1 (40/40) 1

firL2 1 1 (1/1) 1 rand700 1e3 1 1 0 (84/85) 1

rand200 300 1 1 0 (34/35) 1 rand700 1e3 2 1 0 (126/130) 1

rand200 300 2 1 0 (62/65) 1