Towards Practical Algorithm Based Fault Tolerance inDense Linear Algebra

Panruo WuDepartment of ComputerScience and EngineeringUniversity of California,

Riversidepwu011@cs.ucr.edu

Qiang Guan,Nathan DeBardeleben and

Sean BlanchardUltrascale Systems Research

CenterLos Alamos National

Laboratory∗

{qguan,ndebard,seanb}@lanl.gov

Dingwen Tao, Xin Liang,Jieyang Chen and

Zizhong ChenDepartment of ComputerScience and EngineeringUniversity of California,

Riverside{dtao001,xlian007,jchen098,chen}

@cs.ucr.edu

ABSTRACTAlgorithm based fault tolerance (ABFT) attracts renewedinterest for its extremely low overhead and good scalabil-ity. However the fault model used to design ABFT has beeneither abstract, simplistic, or both, leaving a gap betweenwhat occurs at the architecture level and what the algorithmexpects. As the fault model is the deciding factor in choos-ing an effective checksum scheme, the resulting ABFT tech-niques have seen limited impact in practice. In this paperwe seek to close the gap by directly using a comprehensivearchitectural fault model and devise a comprehensive ABFTscheme that can tolerate multiple architectural faults of var-ious kinds. We implement the new ABFT scheme into highperformance linpack (HPL) to demonstrate the feasibility inlarge scale high performance benchmark. We conduct archi-tectural fault injection experiments and large scale experi-ments to empirically validate its fault tolerance and demon-strate the overhead of error handling, respectively.

1. INTRODUCTIONThe extreme scale high performance computing (HPC)

systems that are expected by the end of this decade posesseveral challenges including performance, power efficiency,and reliability. Due to the large amount of components inthese systems and the shrinking feature size, the probabilitythat an extreme scale application experiences faults duringits execution is projected to be non negligible. Resilienceto faults have been widely accepted as critical for exascaleHPC applications[21, 6, 3].

∗This work was performed at the Ultrascale Systems Re-search Center (USRC) at Los Alamos National Laboratory.The publication has been assigned the LANL identifier LA-UR-16-20226.

ACM acknowledges that this contribution was authored or co-authored by an em-ployee, or contractor of the national government. As such, the Government retainsa nonexclusive, royalty-free right to publish or reproduce this article, or to allow oth-ers to do so, for Government purposes only.

Permission to make digital or hard copiesfor personal or classroom use is granted. Copies must bear this notice and the full ci-tation on the first page. Copyrights for components of this work owned by others thanACM must be honored. To copy otherwise, distribute, republish, or post, requires priorspecific permission and/or a fee. Request permissions from permissions@acm.org.

HPDC’16, May 31-June 04, 2016, Kyoto, Japanc© 2016 ACM. ISBN 978-1-4503-4314-5/16/05. . . $15.00

DOI: http://dx.doi.org/10.1145/2907294.2907315

Faults are malfunctions of the hardware or software, andare the underlying causes for observable errors. When thefault does not interrupt the execution of a process the pro-gram can continue execution normally, but the results maybe corrupted. Such silent data corruptions cannot be tol-erated by checkpoint/restart (C/R) alone unless they canbe frequently detected. Silent data corruptions may be theconsequence of soft faults caused by cosmic rays and radi-ation from packaging materials, and are usually one timeevents that corrupt the state of the machine but not itsoverall functionality. We restrict our scope to silent datacorruptions (SDC) in this work. Note that since soft errorswhich are caused by single event upset frequently corruptdata silently, SDC handling is also often discussed in con-text of soft errors.

Faults in storage and communication systems are often ef-fectively tolerated by error correction codes (ECC) becausethe data stored or communicated are not changing. How-ever, faults in logic units that transform the data are harderto detect and tolerate. Typically some kind of double mod-ular redundancy (DMR) is needed to detect soft faults inlogic units and triple modular redundancy (TMR) is neededto tolerate SDCs. Although modular redundancy requiresat least 100% resource overhead and often incurs significantexecution time overhead, it is sometimes the only generalsystem level solution to tolerate SDCs [9, 22].

System level SDC solutions can be prohibitively expensivefor HPC systems. An alternative solution is to implementfault tolerance in applications, which can take advantageof the semantics and structure of a specific application re-sulting in much lower cost. Algorithm based fault toler-ance (ABFT) represents a middle ground between applica-tion specific fault tolerance and architecture fault tolerance.At one end application specific fault tolerance is highly di-verse that often require ad-hoc solutions, at the other endsystem fault tolerance is general but too costly and unscal-able. Algorithms thus presents just enough semantics totake advantage and structure to be generally useful.

ABFT has first been proposed in a seminal work by Huangand Abraham [13] for matrix-matrix multiplication on sys-tolic arrays. The idea of ABFT can be seen as an adap-tion of ECC to numeric structures like matrices or vectors.The significant difference is that for ECC the data is staticbut for ABFT the data is under transformation. In ABFT

1

31

the central problem is that the codes must maintain aftertransformation in order to be able to detect errors using thecodes. The fault model is a deciding factor in the designof ABFT codes and adaption to the associated algorithm.However the fault models used in existing ABFT researchare either too abstract [8, 14, 10] or too simplistic [5, 24,25] limiting their use where the architectural fault modelsdo not fit. In this work we rethink the fault model andexplore the challenges if we use a comprehensive architec-tural fault model that allows both logic/arithmetic faultsand storage faults in main memory, on-chip memory, andother datapaths. We demonstrate that with this fault modelwe still can design highly efficient and resilient ABFT tech-niques for dense linear algebra and use high performancelinpack (HPL) to show that the new techniques can be im-plemented efficiently in complex real world high performanceand highly scalable applications. The design is validated em-pirically by a QEMU [2] based architectural fault injector, F-SEFI [1], which implements the comprehensive fault model.We incorporate the new ABFT techniques into the latestNetlib HPL-2.1 and empirically show that the resulting FT-HPL incurs low overhead and maintains high scalability ofthe original HPL.

The contributions of this paper are:

New fault model We use a fault model that allows logicfaults and memory system faults that are comprehen-sive temporally and spatially and design ABFT schemesthat can effectively detect and correct errors caused bythese faults.

New checksum scheme We propose a novel process localchecksum scheme, multiple checksums for error detec-tion and correction by studying the syndrome (errorpatterns) caused by the faults.

Validation and software implementation We test andvalidate the resilience using an architectural fault in-jector. We implement the new ABFT schemes in thelatest Netlib HPL-2.1.

The rest of the paper is organized as follows. In section2 we survey the techniques to handle SDCs in computingsystems especially the algorithm based approach. In section3 we propose the architectural fault model and the errors itcauses in the eyes of application. In section 4, we present ournew designs to handle the proposed fault model. In section5, fault tolerance capability, various sources of overheads,and optimization methods are discussed. In section 6, wepresent empirical study of the fault tolerance of the proposeddesign and implementation through error injection, and theoverheads in large scale runs. Section 7 concludes the paper.

2. RELATED WORKThe first report on soft errors due to alpha particles in

computer chips was from Intel in 1978 [15]. The first reporton soft errors due to cosmic radiations in computer chipswas in 1984 [27]. In 1996, Norman [18] studied error logsof several large computer systems and reported a number ofincidents of cosmic ray strikes. In 2005, Hewlett-Packardadmitted that the ASC Q supercomputer located in LosAlamos National Laboratory experienced frequent crashesbecause of cosmic ray strikes to its parity protected cachetag arrays. The machine is particularly susceptible because

of the 7000ft altitude of the installation location [16]. Thebook by Mukherjee [17] surveys extensively the architecturaltechniques to design architectures for soft errors.

In the HPC context much effort has been spent on tech-niques to detect and tolerate soft errors. System level ap-proaches usually involves some kind of modular redundancy.RedMPI [9] is a general MPI level solution that replicateseach MPI rank to form double modular redundancy (DMR)for soft error detection or triple modular redundancy (TMR)for error correction. The difficulty is the silent nature ofsoft errors; error detection must be active and in a timelymanner. RedMPI does the error detection when MPI rankscommunicate: the replicas should send out the same mes-sage otherwise a soft error is detected. According to thepaper, MPI rank level replication incurs 20% to 60% execu-tion time overhead in addition to 100% to 200% computingresource overheads. Another approach is algorithmic errordetection coupled with checkpointing for recovery. In [4],the intrinsic orthogonality of some Krylov linear solvers isused for error detection. A study by [20] proposes to turnmany interesting problems into optimization problems thatcan be solved iteratively which is naturally resilient to softerrors.

In the following texts the most relevant related works arediscussed and special attention is paid to the fault mod-els and the influence fault model on the design of algorithmbased fault tolerant schemes. ABFT has been researched ex-tensively for many algorithms but we will narrow our scopeto those that are checksum based and applicable on densematrix multiplication and triangularization.

Algorithm based fault tolerance was first proposed by Abra-ham and Huang [13]. The original ABFT was proposed formatrix multiplication and LU on systolic arrays for real timesignal processing. The fault model used is logic faults thatproduces erroneous results. Storage cell faults such as inmemory, latch, and registers are assumed to be handled bytraditional error correction codes. In matrix multiplication,as a single arithmetic fault causes only a single error in theresult matrix, this ABFT scheme can effectively detect andcorrect it. In LU decomposition, because of error propa-gation, a single fault will cause an overwhelmingly largeamount of errors in the results, thus making this ABFTscheme unable to tolerate a single fault algorithmically. Thelimited correction capability is due to three factors: 1) in-ability to tolerate multiple errors in the checksum scheme, 2)massive error propagation in matrix triangularization, and3) offline error correction. These three factors conspire tomake algorithmic error correction difficult in matrix trian-gularization.

Later Luk and Park [14] described an elegant analyticalmodel for ABFT in matrix triangularization. The analyti-cal model assumes an abstract fault model that a transienterror occurs at some intermediate iteration in the triangular-ization. Even though the single error will propagate in laterstages and become uncorrectable at the end, it can be shownthat the error can be cast back as a single rank perturbationto the original input matrix, much like the widely used back-ward error analysis [23]. Then assuming two row checksumsthe correct result can be derived based on the backwardfault model. This is a powerful technique that avoids theerror propagation problem but it has three limitations: 1)the fault model assumes single error not necessarily singlefault, as we have seen that single fault may cause multiple

2

32

errors; 2) this checksum scheme has no column checksumsthus may fail to even detect certain faults as pointed outby a recent work by Yao [26]; and 3) the method can onlytolerate at most one fault during the decomposition. Asthe scale of supercomputing marches towards exascale, faulttolerance is becoming a key aspect in achieving the requiredperformance at reasonable cost [21, 6, 3]. And assumingonly one fault during the application run seems not appro-priate in future large scale systems any more. To addressmore than one error in matrix triangularization, Du [8, 7]proposed a technique to tolerate two errors in solving lin-ear system using partial pivoting LU decomposition. In thiscase, the decomposition cannot be corrected, but the resultto the linear system can be recovered using the Sherman-Morrison-Woodbury formula. Handling beyond two errorswould be more expensive than the LU decomposition itself.The fault model used is the same as in Luk and Park [14]thus suffers from the same problem.

Some researchers went in another direction in order totolerate more faults effectively. Realizing that the offlineapproach taken by the traditional ABFT techniques have toface catastrophic error propagation at the end, researchersattempted to adapt checksum schemes for online error de-tection and correction [5, 25, 24]. The idea is that onlineABFT catches errors early on when they are not propa-gated far away, therefore making it easier to correct. OnlineABFT also can tolerate more errors that spread in time byavoiding errors compounding each other. The fault modelused however is still arithmetic faults, and there still is nocolumn checksums due to the difficulty in row pivoting.

A recent study [26] discovers that the fault models used inthe previous ABFT works are not adequate even in detectingfaults (Section 3 in [26]). This work proposes a global rowand column checksums that can effectively detect errors andit is also an online approach. However error correction is notconsidered.

In this work we do not use an abstract fault model; ratherwe assume an architectural fault model and aim to detectand correct multiple errors. The architectural fault modelis closer to what happens in real world and not only includeall the fault models discussed above but also more improve-ments.

3. FAULT MODELThe fault model for silent soft errors includes arithmetic

faults that result in a wrong answer, for example 1+1=3.The other important fault is the memory system fault, man-ifesting as corrupted bits in storage cells. Memory faultscould happen in main memory, in caches, registers, andother datapaths. We suppose one memory fault only af-fects one memory word; it can be multiple bits or single bitcorruption.

It is useful to see how the architectural level faults man-ifest themselves in the algorithm level. Typically numericalalgorithms deal with scalar numbers, vectors, and matrices.A variable may be mapped to multiple memory devices. Forexample the variable may be mapped to main memory, andcached in on-chip cache. It may also live in a register tem-porarily. The fault that affects the variable may be causedby corruptions in one of the mapped physical devices, andmanifest themselves differently. For example if the mainmemory is corrupted, the mapped variable may read thecorrupted value continuously until the memory is overwrit-

ten. If the corruption happens in cache, the variable mayread incorrect value until the cache line is flushed. There-fore, a corrupted data element in program may sometimesread correct value but at other times read corrupted value.

4. THE CHECKSUM SCHEMEIt is important to make a distinction between fault and

error. For our purpose, a fault is a malfunction in the archi-tecture, such as a bit flip in memory, cache, or registers. Anerror is the symptom due to the fault. Thus faults are thecause and errors are what we observe that are not correct. Indesigning numerical algorithms, errors are erroneous float-ing point variables. A single bit fault may lead to multipleerrors, depending on how the faulty value is used. For al-gorithm designers and implementers, the problem to designfault tolerant algorithms is to find ways to detect and tol-erate errors. In online ABFT framework, the problem canbe further specified as to detect and tolerate errors resultingfrom one for every error handling interval. In this section,we will first study the error patterns of a single fault and howto tolerate them; then we will discuss how to design check-sum schemes in LU decomposition; we will discuss how toput this technique in use in the very high performance LUdecomposition package HPL; last we drop the assumptionof precise arithmetic and deal with finite precision floatingpoint arithmetic.

4.1 Error patterns and correctionWe begin by studying the error patterns caused by a single

fault in matrix multiplication, as matrix multiplication is thesimplest dense matrix operation and it is an important partof the LU decomposition. We will see that memory faultsmay lead to multiple errors, while in contrast one arithmeticfault will only lead to one error in matrix multiplication.

Figure 1 shows four cases when one fault strikes. Thefault could be an arithmetic fault or a silent data corrup-tion (SDC). The red elements indicate errors. In subfigure(a), a single arithmetic error can only corrupt one elementin the result, because the intermediate value produced bythe faulty arithmetic operation is only used to calculate oneelement. In subfigure (b), a SDC in matrix A corrupts thewhole row in the result C, because the corrupted elementin A is used to calculate the whole row. In subfigure (c),the SDC occurs not in memory but in for example cache, oroccurs later during the matrix multiplication. In this casea single SDC in matrix A causes partial row corruptions inC. In subfigure (d) a single SDC in matrix B causes partialcolumn corruption in C. The important observation here isthat a single fault cannot cause errors in more than one rowor column. This observation enables us to design checksumsthat can correct all the error patterns caused by a singlefault.

Next we discuss how to design checksum schemes to detectand correct up to one fault based on the fault patterns infigure 1. A matrix can have two types of checkums alongits two dimensions: the checksum at the bottom of a matrixis called column checksum and the checksum to the rightof a matrix is called row checksum. The column checksumencoded matrix is often denoted by a superscript Ac and therow checksum encoded matrix by superscript Ar. If a matrixhas both then it is called fully checksummed and denoted byAf . Mathematically, let e be the weight vector (or matrix

3

33

(a)  One  arithmetic  fault  corrupts  at  most  one  element  in  result  matrix  C.

(b)  One  SDC  in  matrix  A  will  cause  a  row  corruption  in  result  matrix  C.

(c)  One  SDC  in  matrix  A  causes  a    partial  row  corruption  in  result  matrix  C.

(d)  One  SDC  in  matrix  B  causes  a    partial  column  corruption  in  result  matrix  C.

x =

x = =

=

x

x

Figure 1: Error patterns for a single fault in matrix multiplication

x =

(a)  Row  +  column  checksum  locates  and  corrects  single  error.

x =

(b)  Double  checksums  locates  and  corrects  single  error.

x =

(d)  Double  row  checksums  cannot  detect  whole  row  corruption  caused  by  single  error  in  A.

x =

(c)  Row  +  column  checksum  detects  error  but  cannot  correct  the  error.

Figure 2: Checksums for matrix multiplication

in the case of multiple checksums) then:

Ac =

[AeTA

], Br =

[B Be

], Cf =

[C CeeTC

]As shown in figure 2, we have multiple configurations ofchecksums. The yellow blocks are row or column check-sums associated with the matrix. The red block indicatesan incorrect element, and a black cross on a row/columnchecksum indicates that the row/column checksum is incon-sistent with the respective row/column in matrix. We needat least two checksums to correct up to one error because thelocation and the magnitude of the error are two unknowns.For a single error in a matrix, either two row checksum, twocolumn checksum, or one row plus one column checksumscan detect and correct one error in matrix C. In subfigure(a), the error can be located at the intersection of the incon-sistent row and column. The error can be recovered usingeither the row or column checksum [13], because the check-sums are correct. In subfigure (b), a single error in matrixC can be detected and corrected using two row (weighted)checksums with different weights [24]. The location of theerror and the magnitude of the error can be solved from thetwo checksums. In subfigure (c), a single SDC in matrix Acauses a whole row corruption that result in an incorrectbut consistent row. Because the row checksum is corrupted,

it leaves us with only one column checksum which is inade-quate to correct the errors. In subfigure (d), a single SDCin matrix A causes a whole row corruption with incorrectbut consistent checksums. In this case the checksum schemecannot detect the errors.

It is now clear that we need both row and column check-sums to avoid the error detection failure. And to correctrow/column corruptions we need two row checksums andcolumn checksums, as shown in figure 3. In figure 3, a SDCin matrix A causes a whole row corruption in C detectableby the column checksums. The errors can be located andcorrected on per column basis using the two correct columnchecksums. The row checksums are neither able to locatethe errors nor correct them.

x =

Figure 3: The checksum scheme that can tolerate singlearithmetic fault or memory fault

4

34

Specifically, how do we locate and correct one erroneouselement using two checksums? There is an easy to use encod-ing method. Suppose we encode a vector using two differentweights e1 = [1, 1, . . . , 1]T , e2 = [1, 2, . . . , n]T . The vector isa = [a1, . . . , an] and we have two correct encoded checksumsof a:

r1 = ae1 =

n∑i=1

ai, r2 = ae2 =

n∑i=1

iai

Now suppose the computed a′ = [a′1, . . . , a′n] has up to one

erroneous element a′j 6= aj , where the location j is unknownto us. However when we verify the checksums:

δ1 =

n∑i=1

a′i − r1 = a′j − aj 6= 0

δ2 =

n∑i=1

iai − r2 = j(a′j − aj) 6= 0

Then a simple division δ2/δ1 gives us the location j. Thecorrect value of aj can then be recovered using the correctchecksum and the other correct elements of a: aj = a′j −∑n

i=1,i6=j a′i.

In this subsection the error patterns in matrix multiplica-tion are discussed and checksums are devised to detect andcorrect errors, given that we have the desired checksumsavailable. In the following subsection, how to maintain thechecksums online is discussed in LU decomposition. Notethat in LU decomposition the matrix multiplication is actu-ally C ← C − A × B instead of C ← A × B so correctionthrough re-computation cannot be used because the originalC is overwritten.

4.2 Checksum scheme in LU decompositionIn this subsection the right-looking LU decomposition is

briefly introduced. We first show that LU decompositionmaintains global row and column checksums. Then we dis-cuss the two adaptions to the LU decomposition that areessential in achieving good performance on modern cachebased system and parallel computing.

LU decomposition factors a matrix A into the productof two triangular matrices (lower) L and (upper) U : A →L× U . The tiled right-looking variant of the LU algorithmworks as shown in figure 4.

𝑨𝟏𝟏 𝑨𝟏𝟐

𝑨𝟐𝟏 𝑨𝟐𝟐

𝑼𝟏𝟐

𝑳𝟐𝟏 𝑨𝟐𝟐&

𝑳𝟏𝟏𝑼𝟏𝟏

Figure 4: Tiled right-looking LU algorithm, one iteration

Figure 4 shows the state before and after an iteration inthe algorithms. The algorithm is a series of iterations thatkeeps shrinking the trailing matrix until done. The yellowparts of the matrix indicate areas that have been factoredand not active. For a certain iteration, the algorithms fol-lows three steps: left panel factorization, top panel update,

and trailing matrix update, described by the following equa-tions:

[A11

A21

]→[L11

L21

]× U11 (1)

A12 → L11 × U12 (2)

A′22 ← A22 − L21 × U12 (3)

The maintenance of checksums offline: In the origi-nal Huang and Abraham ABFT paper [13] it has been shownthat if we LU decompose a full checksummed matrix Af , wewill end up with column checksummed Lc and row check-summed Ur:

[A AeeTA

]→[LeTL

]×[U Ue

](4)

where vector e is the checksum weights vector. This re-lationship can only be used to detect errors but not correcterrors because in LU the errors will propagate to checksumstoo.

The maintenance of checksums online: If LU decom-pose a full checksum matrix, we will end up with a columnchecksummed L and row checksummed U . However multi-ple errors compound each other resulting in algorithmicallyuncorrectable errors. It would be desirable to detect andcorrect errors frequently during the factorizations to handleerrors in a timely manner. In fact, we will show that at theend (or beginning) of each iteration, the factored left paneland top panel will be column checksummed and row check-summed, and the trailing matrix will be fully checksummed.We will show this claim inductively by first assuming thecondition holds at the beginning of a certain iteration andprove that the condition holds at the end of the iteration.The initial condition clearly holds as we have a fully check-summed initial matrix.

𝑨𝟏𝟏 𝑨𝟏𝟐 𝑨𝟏$𝒆

𝑨𝟐𝟏 𝑨𝟐𝟐 𝑨𝟐$𝒆

𝒆𝑻𝑨$𝟏 𝒆𝑻𝑨$𝟐

𝑼𝟏𝟐 𝑼𝟏$𝒆

𝑳𝟐𝟏 𝑨𝟐𝟐) 𝑨𝟐𝟐) 𝒆

𝒆𝑻𝑳$𝟏 𝒆𝟐𝑻𝑨𝟐𝟐)

𝒊𝒕𝒆𝒓𝒂𝒕𝒊𝒐𝒏  𝟏

𝑳𝟏𝟏𝑼𝟏𝟏

Figure 5: Tiled right-looking LU algorithm with checksums,one iteration

For simplicity we only examine the first iteration. Asshown in figure 5, before the iteration we have the full check-sums. After the left panel has been factorized accordingto equation (1), the column checksum associated with theleft panel turns into the checksum of the factorized panel:eTA·1 → eTL·1. To see why this is true one only has to ob-serve: 1) the factorized left panel will not be updated againtherefore will stay unchanged through the end; 2) from equa-tion (4) we know that at the end the left panel will be columnchecksummed. Thus we proved that the left panel factoriza-tion maintains column checksum. Similarly, the second stepaccording to equation (2) maintains the row checksum ofthe top panel. Next we need to prove that after the trailing

5

35

matrix update according to equation (5), the trailing ma-trix will be fully checksummed. To see that we only have toapply the matrix multiplication to the checksums. Take thecolumn checksums for example. The transformation done tothe column checksums is depicted by:

= eT[A12

A22

]− eT

[L11

L21

]× U12

= eT1 (A12 − L11U12) + eT2 (A22 − L21U12)

= eT2 (A′22)

(5)

which proves that the trailing matrix is fully checksummedby the second part of the checksum weights vector e2.

4.3 The complete picture as in HPLThe previous subsection discusses the algorithmic struc-

ture of tiled right-looking LU decomposition, and the main-tenance of checksums at each iteration in fault free execu-tion. In this subsection we discuss what happens when faultsstrikes, namely the error patterns. Once we know the errorpatterns we can describe correction procedures. We will alsodeal with two more complications in HPL: partial row pivot-ing for numerical stability and 2d cyclic block distributionsof matrix for load balance in distributed computing.

Error patterns: We examine the error patterns in thethree steps during one iteration, and discuss detection andcorrection procedures. First, we look at the first step andthe second step according to equations (1) and (2), namelythe left and top panel factorization. Our first claim is thatany single fault that occur during the left and top panel fac-torization will lead to inconsistent checksums, provided thatthe arithmetic are precise, i.e. no round-off errors. In otherwords, the error detection by checksums is precise. The rea-son that the error detection is precise is because we haveboth row and column checksums. If for example only rowchecksums are used, as pointed out by figure 5 in [26], cer-tain faults strike in lower triangular L will not be detected.In our case the fault will be detected by the inconsistent col-umn checksums. Depending on the location and timing ofthe fault, the error pattern could be very complex and boththe row and column checksums will be contaminated andthere is no easy algorithmic corrections, as shown in figure 6(a). For this case we can use in-memory checkpointing androllback specifically for the left and top panels. Once thechecksum inconsistency is detected the computation can berolled back to the beginning of the iteration. In HPL thein-memory checkpoint can be stored in the communicationbuffer for broadcasting L thus do not consume extra mem-ory space. The overhead of memory copy of two panels isnot significant.

For the trailing matrix update, as discussed earlier a sin-gle arithmetic fault only affects one element in the resultthus easily correctable. More interesting cases are memoryfaults within L21 or U12. For a single SDC in L21 or U12, theerrors cannot be in more than one row or one column. As-suming precise arithmetic, a single fault will trigger at leastone row checksum inconsistency and one column checksuminconsistency. Therefore the error detection in trailing ma-trix update is precise, and furthermore the error patternsare within our capability to correct. For example in thecase shown in figure 6 (b) a memory fault associated withan element in L21 causes partial row corruptions. In this

√  √  √  √    X  X X

√√√

X√

X  X X X  X X X X

X●

(a) (b)

Figure 6: Tiled right-looking LU algorithm with checksums,one iteration. Shaded area are incorrect due to error propa-gation. Note that the affected checksums are also incorrectbut the checksums are inconsistent therefore can be used todetect errors.

case the errors are easily located by the intersection of theinconsistent row and column checksums and corrected by thecorrect column checksums. It seems that one row checksumand one column checksum is sufficient to locate and correctany single fault in the trailing matrix update. However thisis not true and will be explained next.

Parallel LU decomposition and 2d cyclic block dis-tribution: On a multiprocessor machine a matrix is usu-ally distributed onto a PxQ grid of processes according to2d block cyclic scheme for load balance and scalability. Asshown in figure 7, a 4x4 block matrix is distributed onto fourprocesses. In the previous discussion we only look at the log-ical (global) view of the matrix and the checksum scheme isapplied to the whole matrix. This view has some drawbacks.First, the fault tolerance capability is not scalable with thesize of the matrix. Second, as the checksums are associatedwith the global matrix that are distributed, the error detec-tion and correction requires inter process communication.To avoid these two drawbacks, we instead apply checksumsto the process local matrix rather than the global matrix. Inthis way, the fault tolerance capability is fixed per process,and increases proportionally with the number of processesor the size of the matrix. Error detection and correctiononly involve local information.

Logical  view  of  the  matrix

Process  0 Process  1

Process  2 Process  3

Process  view  of  the  matrix

Figure 7: 2D block cyclic matrix distribution.

The online maintenance of the process local checksums arevery similar to the global checksums. The error patterns canexhibit more patterns than that of global checksums. For

6

36

example, consider the first iteration and the matrix distri-bution in figure 7. In the trailing matrix update, for process0 and 2, a memory fault in left panel will always produce oneinconsistent row checksum but that is not the case for pro-cess 1 and 3. For process 1 and 3, a persistent memory cor-ruption in L causes the trailing matrix update to exhibit theerror pattern shown in figure 2 (d) where all row checksumsare incorrect but consistent. In this case a single columnchecksum can only detect error; two column checksums arerequired to correct the errors. For process 0 and 2 we showthat even a persistent memory fault in L can produce oneinconsistent incorrect checksums. Similar to equation (5)and figure 5, suppose after the left panel is factorized it is

corrupted in one element L21 → L21 := L21 + αeieTj . Then

the trailing matrix A22 and its row checksums will be up-

dated by the corrupted L21 in the following way (the symbolwith a hat indicates a corruption):

A′22 ← A22 − L21U12

CS(A′22)←[A21 A22

] [e1e2

]= (A21 − L21U11)e1 + (A22 − L21U12)e2

CS(A′22) = A′22e2 = (A22 − L21U12)e2

CS(A′22)− CS(A′22) = (A21 − L21U11)e1

= αeieTj U11e1

(6)

with the last equation indicating one inconsistent rowchecksum. Note that the equations confirm that only onerow in the trailing matrix will be affected; the whole row iscorrupted and so is the associated row checksum, but theyare corrupted in a way that makes them inconsistent. Thesingle row corruption can be handled by the double columnchecksum effectively. The above analysis also shows that theExample 3 in [26] is incorrect.

Partial row pivoting in LU: In practice unpivoted LUcan easily break down due to numerical instability. To re-duce the instability while not incur prohibitively high over-head, partial row pivoting is commonly used. However therow swapping in the pivoting disrupts the maintenance ofthe checksums. If the row checksums are swapped togetherwith their respective rows the row checksums still maintain.Column checksums need to be fixed and not swapped. Inprocess local checksums however, maintaining column check-sums require more care. One row may be swapped with arow from another process, thus invalidating both checksums.Therefore the checksums must be updated when inter pro-cess swapping happens.

Putting them together The pseudo code algorithm 1summarizes the error detection and correction logic. Forbrevity it is in the point of view of global matrix.

4.4 Round-off error boundsIn the last section we have shown that if we limit the faults

to one per error handling interval and assume precise arith-metic, the error detection is both sound and precise. In prac-tice the floating point arithmetic are not precise, soundnessand precision cannot be attained simultaneously. As lack ofsoundness is not acceptable in fault tolerance, we thus striveto maintain soundness at some expense of precision. To dothat, we derive a priori norm based error bounds for the

Algorithm 1 The fault tolerant HPL algorithm, globalview.

Require: Fully checksummed matrix Af and right handside b

Ensure: x = A−1b in the presence of floating point softerrors, or signal errorsn is the size of A, B the blocking factorfor i = 0 to n step B do

A(i : n, i : n) =:

[A11 A12

A21 A22

]Factorize left panel

A11

A21

CS(A·1)

→L11\U11

L21

CS(L·1)

Factorize top panel

[A12 CS(A12)

]→[

U12 CS(U12)]

Check column checksums for L and row checksums Uif Errors not algorithmically correctable then

Rollback to the start of this iterationend ifUpdate the trailing matrix Af

22 ← Af22 − Lc

21Ur12

Check and correct full checksum matrix Af22

end for

round-off error, and use the upper bound as the thresholdto distinguish architectural faults from floating point round-off errors. If the architectural faults alters the less significantbits in a floating point number and the result is still withinround-off error bounds, no errors will be detected and thefault is deemed indistinguishable from round-off errors.

Specifically, when we are verifying the checksums we needto compare the calculated sums to the checksums. Becausefloating point arithmetic has finite precision, those two maydiffer even in fault free execution. Our problem is now tobound the difference that round-off errors such that round-off errors alone would not violate the bound. Consider thematrix multiplication C = AB. A well known norm boundof the round-off errors in matrix multiplication is as fol-lows [11].

||fl(AB)−AB||∞ ≤ γn||A||∞||B||∞ (7)

Assuming that the encoded matrix multiplication Cf =AcBr is carried correctly, and the variable with a hat repre-sents its floating point representation, we have the followingresult:∣∣∣∣∣

n∑j=1

cij − ci,n+1

∣∣∣∣∣ =

∣∣∣∣∣n∑

j=1

(cij − cij)− (ci,n+1 − ci,n+1)

∣∣∣∣∣≤

∣∣∣∣∣n∑

j=1

(cij − cij)

∣∣∣∣∣+ |(ci,n+1 − ci,n+1)|

≤ ||fl(Cf )− Cf ||∞≤ γn||Ac||∞||Br||∞

(8)

where γn = nu/(1 − nu) and u is the unit round-off errorof the machine. For IEEE 754 64bit floating point numberu = 10−16. We thus obtained a bound of round-off errorsthat can be used as a threshold to distinguish architecturalfaults from floating point round-off errors. There is a similarbound to verify the row checksums.

7

37

5. OVERHEAD, PERFORMANCE, SCALA-BILITY, AND FAULT TOLERANCE CA-PABILITY

In this section we model the fault tolerance capability, theexecution time overhead, the scalability, and optimization ofthe proposed fault tolerant HPL.

5.1 Fault tolerance capabilityFor the error correction capability provided that errors

can be detected, a natural question is how many errors orfaults can be corrected? For each process in each error han-dling interval, any number of errors during the left and toppanel factorization can be tolerated by the rollback. Multi-ple errors or one fault can be tolerated during the trailingmatrix update, provided that the errors are within one rowor one column. Note that the number of faults that canbe tolerated is scalable with the number of processes andproblem size, so at large scale enormous number of errorsor faults can be tolerated as long as the faults do not burstinto one error handling interval.

Compared to online ABFT (FT-ScaLAPACK) [5, 24]: On-line ABFT may fail to detect memory error in the trailingmatrix update where the process is not engaging in the leftpanel factorization. FT-ScaLAPACK cannot correct the er-rors caused by faults in the left panel during the matrixmultiplication.

Compared to offline ABFT (Du, Luk) [7, 8, 14] : Our FT-HPL is resilient to much more faults. For non permanentlysticky memory fault, for example faults in cache or registers,offline ABFT correction based on casting the fault back tolow rank perturbations to the initial matrix no longer work.In fact, any fault that do not corrupt a variable for its entirelifespan will fail in offline ABFT fault tolerance scheme, asthe fault do not fit in the abstract fault model. Thus thetolerable faults in offline ABFT schemes is a small subsetof the more comprehensive fault models considered in thispaper.

5.2 Execution time overheadThe fault tolerant LU decomposition introduces overheads

in maintaining checksums, checking checksums periodically,and correcting errors if detected. As the analysis here onlyserves as a first order approximation of the performance, weuse a widely used simple machine model. The communica-tion time is modeled as T = α + βL where α is networklatency and β is the reciprocal of network bandwidth. Thecomputation of matrices and vectors can be modeled by theproduct of compute rate γ and number of floating point op-erations (FLOPs). The compute rate of BLAS3 operationsuch as matrix multiplication is γ3 and the compute rateof BLAS2 operation such as matrix vector multiplication isγ2. On modern architectures γ2 is much lower than γ3 soit is important to make the distinction. Let N be the sizeof the matrix A, B be the blocking factor, P × Q be thedimension of the process grid, then the run time of HPL LUdecomposition is as follows [19]:

Thpl =2γ3N3

3PQ+ βN2 3P +Q

2PQ+

αN(B + 1) logP + P

B

(9)

Checksum maintenance overhead: The overhead of

checksum maintenance can be considered as the effectivelyincreased matrix size. Adding two row checksums and twocolumn checksums to the process local matrix, the globalchecksum matrix is bounded by max(N(1 + 2P/N), N(1 +2Q/N)). In a reasonable configuration of HPL, N/P andN/Q are the local dimensions of process local matrix thatare around 10,000 therefore the enlargement of the globalmatrix size is around 0.02%. The resulting relative increaserun time in equation 9 will be less than 0.1%, thus not asignificant contribution to the run time overheads.

Checksum verification overhead: The periodical ver-ification of the checksums is one major contribution to therun time overhead. The verification of checksums is a BLAS2operation. The overhead of the verifications are:

Tcheck =4γ2PQ

(N2 + (N −B)2 + (N − 2B)2 + · · ·+B2)

= 4γ2N3

3BPQ(10)

Compared to equation 9 the relative overhead is

TcheckThpl

<2γ2Bγ3

(11)

Assuming a blocking factor B around 200 and BLAS2 op-eration is 5x slower than BLAS3 operations, the overheadis less than 5%. Different machines will have different ratioand different relative overhead.

5.3 Error correction overheadThis overhead is only present when errors are detected and

correctable. The algorithmic error correction using check-sums are non-significant. For the errors that are not algo-rithmically correctable by the checksums, the overhead is thelost work and rollback and recompute of the left panel fac-torization, which is empirically a small relative to the wholefactorization.

5.4 Memory overheadThe fault tolerance needs extra memory space to store the

checksums and the left panels. The extra space to store thechecksums are less than 0.1% so not a significant overhead.The memory overhead of storing the left panel is more signif-icant at B

N/Q. Again assuming a typical HPL configuration

B = 200, N/Q = 10, 000 the overhead is at 2%.

5.5 Impact on scalabilityIf we measure scalability by the parallel efficiency

TserPQThpl

which indicates how close it is to ideal parallel speedup, be-cause the execution time overhead is bounded if memoryusage per process is fixed and regardless of P,Q, the scala-bility of the fault tolerant HPL will remain the same as theoriginal HPL which is excellent.

5.6 Tradeoffs between resilience and overheadAccording to the overhead analysis and the detailed tim-

ing result from the experiments we found that the verifi-cation of the trailing matrix is one major overhead to theexecution time. In fact when the trailing matrix verificationis disabled the fault free execution time overhead droppedby half. In this section we discuss the tradeoff between fault

8

38

tolerance and overhead, and the insights to allow such trade-offs to happen.

Let us take the point of view of one particular process.Suppose there is a grid of P ×P processes and the matrix isdistributed in 2d cyclic blocked manner. Since the LU de-composition works factorizes left and top panel sequentiallyfrom left to right and from top to bottom, the particular pro-cess engages into panel factorization every P iterations (infigure 8 P = 4). As we have discussed in the error patternsin matrix-matrix multiplication (TU), the errors propagatein a controlled way. In fact, if we skip the trailing matrixverification procedure at the end of iteration 1 and 2, westill can correct up to 1 fault happening during iteration1,2, and 3 at the end of iteration 3. In this way we tradefault tolerance for reduced error checking overhead. The ob-servation that allows us to make this tradeoff is that faultsduring iteration 1 will not propagate during iteration 2 and3. However this is not true for PF as one fault in PF willpropagate and cause massive errors in subsequent TU mak-ing the single fault uncorrectable. Thus the error handlingprocedure after each PF cannot be skipped to reduce faulttolerance.

The overhead analysis in the last section takes a globalworkload approach and assumes perfect load balance be-tween the processes. But in parallel LU there is load imbal-ance during the panel factorizations where only a columnof processes engage and other processes are waiting for thefactorization result. It can be seen that PF is likely to beon the critical path. As PF depends on the TU immediatelybefore, that TU is also likely to be on the critical path. TheTU verification before PF thus is likely to be on the crit-ical path. In fact from the experiments we found that bydisabling only the TU verification immediately before a PF(shown in figure 8 OPT) the overall execution time dropsalmost as much as by disabling all TU verifications alto-gether. This significant reduction in overhead is thereforehighly desirable, however it seems to break the promise thatsingle fault during one error handling interval is tolerable.To remedy this problem, we only need to observe that, onefault in the last TU will cause the immediate subsequent PFverification to fail. The PF can be made non-destructiveand once the PF fails the checksum verification, the errorhandling procedure for the previous TU is automatically in-voked and the PF will restart. Therefore, the best tradeoff

Iteration

PF+TU TU TU TU PF+TU TU TU TU PF+TU

0 1 2 3 54 76 8

PF+TU TU TU TU PF+TU TU TU TU PF+TU

PF+TU TU TU TU PF+TU TU TU TU PF+TU

ORIG

FULL

OPT

Figure 8: One process view in a 4x4 process grid: PF standsfor (left and top) panel factorization and TU for trailingmatrix update. The red diamond represents checksum veri-fication point.between fault tolerance and overhead is to disable only theTU verification immediately before a PF for every process.In this way the error handling interval remains short and thecritical TU verification overhead is reduced significantly.

6. EXPERIMENTAL STUDYIn this section we empirically evaluate: 1) the fault cov-

erage of the proposed FT-HPL in comparison to the state-of-the-art ABFT techniques by targeted fault injection; 2)the resilience of the FT-HPL scheme and implementation byrandomly injecting various faults; 3) the cost of introducingsuch fault tolerance by measuring large scale executions.

6.1 Fault injection for fault coverageIn this subsection we experimentally compare the fault

coverage of the state-of-the-art ABFT techniques that canapply to LU decomposition and HPL. We inject both arith-metic faults and memory faults to various locations in codeand data at various times during the execution. We selectseveral representative stages in one iteration to inject faults.Specifically, during the first iteration of LU algorithm, weinject faults right before the iteration and in the middle ofthe iteration (at iteration 2 of trailing matrix update). Thearithmetic fault is simulated by modifying the output of afloating point multiplication. The memory fault is injectedto matrix element (2,1) by modifying the data value. Toprecisely control where and how to inject a fault, we use thedebugger GDB to stop the program and modify the programand data. During each run, we only inject one fault.

The fault coverage are summarized in table 1. As canbe seen in table 1, no previous ABFT techniques provideas complete coverage to both arithmetic faults and memoryfaults happening at any time.

Table 1: Fault coverage for different ABFT techniques. “Be-fore” means the fault affects data that is produced but notyet used. “Middle” means the fault affects data that is un-dergoing repeated use.

Fault category Arithmetic MemoryFault timing Before MiddleFT-HPL (this paper) 3 3 3

FT-ScaLAPACK[24]/FTLU[5]

3 7 7

FT-DGESV[8, 7] 7 3 7

6.2 Fault injection experimentsWe use a architectural fault injector F-SEFI [1] to imple-

ment the fault model and reveal the resilience of the FT-HPL implementation. Faults are injected at random timeto a random instruction or memory locations that is to beused. Note that we inject faults into active memory to avoidmasked faults that are never used. We model both floatingpoint arithmetic faults and memory system faults. F-SEFIis based on QEMU, an architecture emulator. It works byintercepting the instructions of the application and alter theeffect of the instructions to simulate arithmetic faults andmemory faults. The application runs unmodified in the vir-tual machine and F-SEFI effectively simulate architecturecorrect execution (ACE) faults [17]. Memory system faultsare modeled in detail: different level of stickiness associatedwith a memory address is used. In a cache based archi-tecture, a variable in the program is mapped to multiplephysical spaces in the memory hierarchy. When the imageof the variable in different physical spaces is corrupted, theprogram perceives a certain stickiness of the error. For ex-ample, a corrupted main memory word is very sticky as it

9

39

Table 2: Fault tolerant for dense linear algebra: costs and fault tolerance capability. “Yes” means the faults canbe tolerated; “No” means otherwise. The percentage indicates the execution time overhead against non faulttolerant LU implementation (PDGESV/PDGESV in ScaLAPACK, HPL pdgesv in HPL).

Fault category No Error Arithmetic Faults Memory FaultsNumber of faults 0 ≤ 2 many ≤ 2 many

FT-HPL 5% Yes, 5% Yes, 5-35%a,b Yes, 5% Yes, 5-35%a,b

FT-ScaLAPACK[24]/FTLU[5] 8% Yes, 8% Yes, >8%b No NoFT-DGESV[8, 7] 1% Partialc, 1% No Partialc, 1% No

RedMPI[9]d ≥ 20% Yes, ≥ 20% Yes, ≥ 20% Yes ≥ 20% Yes,≥ 20%a Overhead depends on the impacted phase in HPL.b To tolerate multiple faults they must be spaced out in time thus not overwhelming one error handling interval.c The fault must happen in specific time and location to fit the algebraic model in [8, 7]. See table 1.d To tolerate faults RedMPI need 200% more processors to form TMR at MPI rank level.

will read corrupted value until overwritten. On contrast, acorrupted cache word may only read corrupted value tem-porarily until it is flushed out, and subsequent read to thevariable will read from main memory or lower level cachewhich has the correct value.

The configurations of the fault injection experiments areas follows. Four virtual machines are used with one MPIrank in each virtual machine. The problem size is 200x200with blocking factor B = 5, which means that there are40 intervals. During each run of the experiment, 5 faultsare injected at random times to a random memory locationsthat are active. We take care not to inject two faults intoone error handling interval which our FT-HPL cannot han-dle. Note that this setting injects a considerable amount offaults into a small problem size to stress the fault tolerancemechanism.

In total 300 repetitions of the experiment are performed.Among them, 252 cases (84%) successfully tolerated the in-jected faults and passed the residual check of the HPL appli-cation. In all passed cases, the injected faults are detectedand corrected by our algorithms. Another 21 cases (7%)run to completion but failed to pass the residual check be-cause in HPL application not all data structures and opera-tions can be protected by our algorithm. The remaining 27(9%) cases crashed or hung. In contrast, when subject to5 random memory faults both FT-ScaLAPACK/FTLU andFT-DGESV would have success rate of 0%.

6.3 Overheads of fault free execution and er-ror correction

In this section we evaluate how much execution time over-head is during fault free execution, and the cost of errorcorrection in the presence of faults. The experiments areconducted on two clusters: 1) a small cluster TARDIS (upto 512 cores) for detailed overhead reduction experiments,and 2) TACC Stampede for large scale (up to 4096 cores)scalability and overhead experiments. The TARDIS is a 16node cluster; each node is equipped with two sockets AMD6272 processors (32 cores) clocked at 2.1GHz. Each nodehas 64 GB memory. The interconnect is Mellanox QDRInfiniBand. The TACC Stampede is currently the #10 onTop500.org November 2015 list. Each node has two IntelE5 8-core (Sandy Bridge) processors with core frequency2.7GHz and 32 GB memory. Each core is capable to de-liver 21.6GFLOP/s at maximum. The interconnect is FDR56Gbps InfiniBand Mellanox switches using the 2-level Closfat tree topology. Table 2 provides summarized comparison

to state-of-the-art ABFT techniques in terms of overheadand fault coverage.

6.3.1 Overhead reduction and correction overheadThis set of experiments are done on TARDIS to investi-

gate the overhead reduction effect discussed in subsection5.6. In the fault free execution mode, four variants of imple-mentations are measured: ORIG is the original unmodifiedNetlib HPL-2.1[19]; FULL implements the fault tolerancedescribed in the last section; OPT implements an optimiza-tion technique that partially removes the trailing matrixchecksum verification from the critical path; and FAULTis essentially FULL plus injected error that triggers all er-ror correction procedures. In the non fault free execution,faults are injected via source code instrumentation to trig-ger all error checking and correction, thus demonstrating themaximum overhead of error correction. The process localmatrix size is fixed at around 3000x3000, lower than a typ-ical 10000x10000 configuration which will take much longerto complete. We use process grids N × 32, with the numberof nodes N being from 2 to 10, and the matrix size N beingfrom 24000 to 51000 The block size is fixed at B = 200.

Figure 9 thus shows the execution time in weak scalingexperiments. It can be seen that with fault free execution,the execution time overhead can be as low as 6% comparedto the non fault tolerant original HPL implementation. Thisis the cost paid to be able to tolerate faults that can occurduring the execution. Also the error correction proceduresare very cheap and cost between 25% to 35% execution over-head at the maximum of its fault tolerance capability. Notethat this is the time it takes to handle hundreds of faults orthousands of errors caused by the faults.

It is also worth noting that the OPT configuration has al-most 50% overhead reduction over the FULL configurationwhich confirms the analysis that the trailing matrix verifica-tion immediately before panel factorization is on the criticalpath.

6.3.2 Scalability experimentsIn the following texts we adopt the OPT strategy and look

at the fault free overhead at large scale on TACC Stampedeusing up to 4096 cores (256 nodes, the maximum allowedscale without special request). For HPL the efficiency interms of floating point operations per second (FLOP/s) percore increases when memory usage per process increases.In the first set of experiments we use only a small fractionof memory available to avoid exceedingly long experiment

10

40

0

10

20

30

40

50

60

70

80

90

2 3 4 5 6 7 8 9 10

Execution  tim

e  (s)

Number  of  nodes

FAULT

FULL

OPT

ORIG

0.00%

5.00%

10.00%

15.00%

20.00%

25.00%

30.00%

35.00%

2 3 4 5 6 7 8 9 10

Execution  tim

e  overhead

Number   of  nodes

FAULT  overhead

FULL  overhead

OPT  overhead  

ORIG  overhead

Figure 9: The execution time of FAULT, FULL, OPT, andORIG HPL with varying number of nodes as X-axis. Eachnode comes with 32 computing cores.

execution time (a single HPL run at its maximum problemsize could take hours for 4096 cores). In the second setof experiments we fix the number of computing elementsat 1024 cores and increase the problem size to observe thetrend of overhead. From these two sets of experiments wecan get an empirical idea of the overhead in introducing theresilience into HPL. The results are shown in figure 10

Reproducing large scale parallel experiments is difficult;so we strive to improve the interpretability [12] by pro-viding more contexts and data. Since the execution timeof HPL on a typical computing cluster is slightly indeter-ministic on Stampede, we collected enough measurementsuntil the 99% confidence interval is around 5% of the re-ported mean measurements, following the recommendationsfrom [12]. Also for this particular experiments on TACCStampede we strongly suspect that there was an abnormalnode with significantly slower network interface. If suchnode is included in the resource allocation the job will besignificantly delayed by at least 20%. We base this con-clusion on the following two reasons: 1) the measurementsstrongly exhibit two clusters around two modes. Any onemeasurement belonging to one cluster will appear as out-lier for the other cluster using Tukey’s outlier classificationmethod. 2) jobs involving more nodes have a higher portionof such abnormally slow measurements: for 1024 cores wegot 1 every 20 measurements; for 2048 cores we got 1 every10 measurements; for 4096 cores 1 every 2 measurements.To eliminate the interference of such slow node we removethe measurements that are abnormally slow.

7. CONCLUSIONFault model is the deciding factor on design of ABFT

algorithms. In this work we seek to close the gap between

5

10

15

20

25

30

35

40

45

50

55

256 512 1024 2048 4096 0

20

40

60

80

100

execu

tion t

ime(s

)

execu

tion t

ime o

verh

ead(%

)

number of processes/cores

OPT timeORIG time

FT Overhead

5.5% 6.1% 6.5%1.5% 3.3%

(a) Fault free execution time for optimized fault-tolerant HPL(OPT) and the original HPL (ORIG). The process local matrixsize is fixed at 2000 × 2000 while the number of processes/coresis scaling from 256 to 4096.

0

200

400

600

800

1000

1200

1400

2000 4000 6000 8000 10000 0

20

40

60

80

100

execu

tion t

ime(s

)

execu

tion t

ime o

verh

ead(%

)

process local matrix size

OPT timeORIG time

FT Overhead

6.5% 4.6% 6.7% 8.1% 8.3%

(b) Fault free execution time for optimized fault-tolerant HPL(OPT) and the original HPL (ORIG). The number of pro-cesses/cores is fixed at 1024 while the process local matrix sizescales from 2000 × 2000 to 10000 × 10000.

Figure 10: Fault free execution time for optimized fault-tolerant HPL.

what occurs at the architecture level and what the algorithmexpects. We explore the challenges in designing ABFT algo-rithms under a general architectural fault model that allowsboth arithmetic and memory system faults comprehensiveboth temporally and spatially. By dividing the executioninto many error handling intervals and aim at tolerating sin-gle fault in each error handling interval, we build a processlocal checksum scheme that achieves scalable fault tolerance(one fault per iteration per process) at around 5% fault freeexecution time overhead and less than 35% execution timeoverhead when facing maximum number of faults. Targetedfault injection shows that the comprehensive fault modelcannot be handled by existing state-of-the-art ABFT tech-niques but will be effectively tolerated by FT-HPL scheme.Random fault injection shows that our FT-HPL implemen-tation can tolerate 84% of the cases where 5 faults occurwithin less than 1 second. Such low overhead and high faulttolerance under comprehensive fault model makes the newABFT in dense linear algebra practical and attractive in ex-treme scale systems, on unreliable commodity hardwares, orin hostile environments.

11

41

AcknowledgmentsThe authors would like to thank the anonymous reviewersfor their insightful comments and valuable suggestions. Thiswork is partially supported by the NSF grants CCF-1305622,ACI-1305624, CCF-1513201, the SZSTI basic research pro-gram JCYJ20150630114942313, and the Special Programfor Applied Research on Super Computation of the NSFC-Guangdong Joint Fund (the second phase).

8. REFERENCES[1] F-SEFI: A Fine-Grained Soft Error Fault Injection

Tool for Profiling Application Vulnerability. InIPDPS’14.

[2] F. Bellard. QEMU, a Fast and Portable DynamicTranslator. In USENIX Annual Technical Conference,FREENIX Track, pages 41–46, 2005.

[3] F. Cappello, A. Geist, W. Gropp, S. Kale, B. Kramer,and M. Snir. Toward Exascale Resilience: 2014update. Supercomputing frontiers and innovations,1(1):5–28, June 2014.

[4] Z. Chen. Online-ABFT: An Online Algorithm BasedFault Tolerance Scheme for Soft Error Detection inIterative Methods. In Proceedings of the 18th ACMSIGPLAN Symposium on Principles and Practice ofParallel Programming, PPoPP ’13, pages 167–176,New York, NY, USA, 2013. ACM.

[5] T. Davies and Z. Chen. Correcting soft errors online inLU factorization. In Proceedings of the 22ndinternational symposium on High-performance paralleland distributed computing, pages 167–178. ACM, 2013.

[6] N. DeBardeleben, J. Laros, J. T. Daly, S. L. Scott,C. Engelmann, and B. Harrod. High-end computingresilience: Analysis of issues facing the HECcommunity and path-forward for research anddevelopment. Whitepaper, 2009.

[7] P. Du, P. Luszczek, and J. Dongarra. HighPerformance Dense Linear System Solver with SoftError Resilience. In 2011 IEEE InternationalConference on Cluster Computing (CLUSTER), pages272–280, Sept. 2011.

[8] P. Du, P. Luszczek, and J. Dongarra. HighPerformance Dense Linear System Solver withResilience to Multiple Soft Errors. Procedia ComputerScience, 9:216–225, 2012.

[9] D. Fiala, F. Mueller, C. Engelmann, R. Riesen,K. Ferreira, and R. Brightwell. Detection andCorrection of Silent Data Corruption for Large-scaleHigh-performance Computing. SC ’12, pages78:1–78:12, Los Alamitos, CA, USA, 2012.

[10] P. Fitzpatrick and C. Murphy. Fault tolerant matrixtriangularization and solution of linear systems ofequations. In Proceedings of the InternationalConference on Application Specific Array Processors,1992, pages 469–480, Aug. 1992.

[11] G. H. Golub and C. F. V. Loan. Matrix Computations.JHU Press, Dec. 2012.

[12] T. Hoefler and R. Belli. Scientific Benchmarking ofParallel Computing Systems: Twelve Ways to Tell theMasses when Reporting Performance Results. SC ’15,pages 73:1–73:12, New York, NY, USA, 2015.

[13] K.-H. Huang and J. Abraham. Algorithm-Based Fault

Tolerance for Matrix Operations. IEEE Transactionson Computers, C-33(6):518–528, June 1984.

[14] F. T. Luk and H. Park. An Analysis ofAlgorithm-based Fault Tolerance Techniques. J.Parallel Distrib. Comput., 5(2):172–184, Apr. 1988.

[15] T. May and M. H. Woods. Alpha-particle-induced softerrors in dynamic memories. IEEE Transactions onElectron Devices, 26(1):2–9, Jan. 1979.

[16] S. E. Michalak, K. W. Harris, N. W. Hengartner,B. E. Takala, S. Wender, and others. Predicting thenumber of fatal soft errors in Los Alamos NationalLaboratory’s ASC Q supercomputer. Device andMaterials Reliability, IEEE Transactions on,5(3):329–335, 2005.

[17] S. Mukherjee. Architecture design for soft errors.Morgan Kaufmann, 2011.

[18] E. Normand. Single event upset at ground level. IEEEtransactions on Nuclear Science, 43(6):2742–2750,1996.

[19] A. Petitet, R. C. Whaley, J. Dongarra, and A. Cleary.HPL - A Portable Implementation of theHigh-Performance Linpack Benchmark forDistributed-Memory Computers.

[20] J. Sloan, D. Kesler, R. Kumar, and A. Rahimi. Anumerical optimization-based methodology forapplication robustification: Transforming applicationsfor error tolerance. In 2010 IEEE/IFIP InternationalConference on Dependable Systems and Networks(DSN), pages 161–170, June 2010.

[21] M. Snir and et. al. Addressing failures in exascalecomputing. International Journal of High PerformanceComputing Applications, 28(2):129–173, May 2014.

[22] J. Von Neumann. Probabilistic logics and thesynthesis of reliable organisms from unreliablecomponents. Automata studies, 34:43–98, 1956.

[23] J. H. Wilkinson, J. H. Wilkinson, and J. H. Wilkinson.The algebraic eigenvalue problem, volume 87.Clarendon Press Oxford, 1965.

[24] P. Wu and Z. Chen. FT-ScaLAPACK: Correcting SoftErrors On-line for ScaLAPACK Cholesky, QR, andLU Factorization Routines. In Proceedings of the 23rdInternational Symposium on High-performanceParallel and Distributed Computing, HPDC ’14, pages49–60, New York, NY, USA, 2014. ACM.

[25] P. Wu, C. Ding, L. Chen, T. Davies, C. Karlsson, andZ. Chen. On-line soft error correction inmatrix–matrix multiplication. Journal ofComputational Science, 4(6):465–472, Nov. 2013.

[26] E. Yao, J. Zhang, M. Chen, G. Tan, and N. Sun.Detection of soft errors in LU decomposition withpartial pivoting using algorithm-based fault tolerance.International Journal of High Performance ComputingApplications, page 1094342015578487, Apr. 2015.

[27] J. F. Ziegler and H. Puchner. SER–history, Trendsand Challenges: A Guide for Designing with MemoryICs. Cypress, 2004.

12

42