Trading Redundancy for Communication:Speeding up Distributed SGD for Non-convex Optimization

Farzin Haddadpour 1 Mohammad Mahdi Kamani 2 Mehrdad Mahdavi 1 Viveck R. Cadambe 1

AbstractCommunication overhead is one of the key chal-lenges that hinders the scalability of distributedoptimization algorithms to train large neural net-works. In recent years, there has been a greatdeal of research to alleviate communication costby compressing the gradient vector or using localupdates and periodic model averaging. In this pa-per, we advocate the use of redundancy towardscommunication-efficient distributed stochastic al-gorithms for non-convex optimization. In partic-ular, we, both theoretically and practically, showthat by properly infusing redundancy to the train-ing data with model averaging, it is possible tosignificantly reduce the number of communica-tion rounds. To be more precise, we show thatredundancy reduces residual error in local averag-ing, thereby reaching the same level of accuracywith fewer rounds of communication as comparedwith previous algorithms. Empirical studies onCIFAR10, CIFAR100 and ImageNet datasets ina distributed environment complement our theo-retical results; they show that our algorithms haveadditional beneficial aspects including toleranceto failures, as well as greater gradient diversity.

1. IntroductionThe Stochastic Gradient Descent (SGD) is a fundamentaltool for solving numerous optimization problems in machinelearning. SGD is an iterative method in which the modelparameters x(t),x(t+1) respectively at the tth and (t+ 1)stiteration are related as:

x(t+1) = x(t) − ηg(t), (1)

where g(t) is the stochastic gradient of the cost function. Re-quiring immense computational resources, large scale ma-

1 School of Electrical Engineering and Computer Science2College of Information Sciences and Technology, The Pennsylva-nia State University, University Park, PA, USA. Correspondenceto: Farzin Haddadpour <fxh18@psu.edu>.

Proceedings of the 36 th International Conference on MachineLearning, Long Beach, California, PMLR 97, 2019. Copyright2019 by the author(s).

chine learning models such as deep neural networks whichuse SGD are, nowadays, trained in distributed systems onseveral computation nodes.

In particular, the dataset D is partitioned into shards anddistributed to computation nodes. Each node evaluates thepartial gradients over a randomly sampled mini-batch fromits shard. In fully synchronous SGD, the computation nodes,after evaluation of gradients over the sampled mini-batch,exchanges their updates in every iteration to ensure that allnodes have the same updated model. Thus, implementingfully synchronous SGD to update the model as (1) incurssignificant amount of communication both in terms of num-ber of rounds and data exchange. This communication costis, in fact, among the primary obstacles towards scalingdistributed SGD.

A promising strand of research that aims at mitigating com-munication bottlenecks in large scale SGD is model averag-ing, where each node locally updates the model sequentiallyon different data partitions, and the models of the differentnodes are averaged periodically (McDonald et al., 2010;Wang & Joshi, 2018; Yu et al., 2018; Zhang et al., 2016;Zhou & Cong, 2017; Zinkevich et al., 2010). Because oflocal updates, the model averaging approach reduces thenumber of rounds of communication in training and cantherefore be much faster in practice. However, since themodel for every iteration is not updated based on the entiredata, it suffers from a residual error with respect to fullysynchronous SGD; this residual error lowers the accuracyin training for model averaging. In this paper, we developa new approach towards model averaging by infusing re-dundancy in the data stored at the computation nodes. Ourapproach retains the benefits of model averaging - lowernumber of rounds of communication - and yet leads to im-proved model accuracy via residual error reduction.

In summary, the main contributions of the present work areas follows:

• We present in Section 3, the Redundancy-Infused SGD(RI-SGD) algorithm, which is a variation of distributedSGD. In RI-SGD, the data is partitioned into shards;each node has its own shard of data, and is allowed topartially access the shards of a subset of other workers

Speeding up Distributed SGD for Non-convex Optimization

Figure 1. Training CIFAR10 using ResNet44 on 4 parallel GPUs,using a generic Model Averaging introduced in (Yu et al., 2018),and our framework with 25% redundancy infused. In both runsthe number for local updates is 50. For more details, see theexperiment setup in Section 5.

to the level of imposed redundancy budget (either ex-plicitly by data replication or implicitly by samplingdata from other workers’ shard). RI-SGD operates byperforming local updates and periodically averagingthe models across the nodes.

• In Section 4, we perform a convergence analysis of RI-SGD for non-convex Lipschitz-continuous loss func-tions. Under certain assumptions, our bounds show thatthe error converges inversely proportional to the num-ber of local updates. Importantly, for a fixed number ofnumber of iterations and local updates, our algorithmcan be designed to have lower residual error as com-pared with the previous algorithms where there is noredundancy. The main technical contribution of ourpaper is an error analysis in distributed local SGD withnon-identical and non-uniform sampling at the differ-ent nodes. As a special case, this includes the practicalcase where data is partitioned among the computationnodes, and each node draws mini-batches from thelocally stored data. The important consequences ofour analysis are (a) the role of diversity in the residualerror, (b) the role of redundancy reducing this error tospeed up convergence and (c) fault tolerance.

• In Section 5, we demonstrate the accuracy and conver-gence properties empirically - see for e.g., Fig. 1. Ourresults show that with some redundancy, both train-ing and testing can be significantly more accurate as

compared with conventional algorithms for the samecommunication. Depending on the minibatch size, RI-SGD can incur an additional computation cost due tothe redundancy in the data. Despite this, for a fixedlevel of accuracy, our experiments demonstrate thatRI-SGD trains faster (lower wall clock times) as com-pared with the no-redundancy case. This speed up isdue to communication being a more severe bottleneckthan computation. Furthermore, even when we reducethe minibatch size of RI-SGD inversely proportionalto the amount of redundancy to make its computationburden equal to the no-redundancy counterpart, ourexperiments reveal that RI-SGD can still lead to in-creased accuracy due to greater diversity (see Section 5for more details). We also show a positive byproductof redundancy in the form of increased fault-tolerancein our paper.

2. Related workThe primary approach to distribute the training processacross several workers is the parameter-server framework(Cui et al., 2014; Dean et al., 2012; Li et al.). These methodssuffer from synchronization delay and high communicationcost. A line of work is to mitigate the synchronization delayvia asynchronous model update methods (Cui et al., 2014;Gupta et al., 2016; Mitliagkas et al., 2016; Recht et al.,2011). These methods, though faster than synchronizedmethods, lead to convergence error issues due to stale gra-dients. This line of work is orthogonal to our work as wefocus on communication issues.

Communication bottlenecks in distributed SGD manifest intwo ways; first, the number of bits communicated can besignificant, and second, the number of rounds of commu-nication can be large. Several studies aim to minimize thenumber of transmitted bits in each iteration by quantizing(Alistarh et al., 2017; Bernstein et al., 2018; Seide et al.,2014; Wangni et al., 2018) or sparsifying (Aji & Heafield,2017; Dryden et al., 2016; Strom, 2015) the gradient vector.Another approach to reducing communication is decentral-ized parallel SGD (Jiang et al., 2017; Jin et al., 2016; Lianet al., 2017), which performs decentralized training withsparse-connected network of computation nodes. All thesemethods are complementary to ours, as we focus on ap-proaches that have fewer number of communication roundsrelative to the number of iterations. Similar to us, refer-ences (Haddadpour et al., 2018; Ye & Abbe, 2018) useredundancy to reduce communication costs. However theirtechnical content is orthogonal to ours since they focus onexact gradient descent and quadratic cost functions.

We focus on model averaging methods where nodes to per-form updates locally based on their own model. Thesemethods were first introduced under the name of one-shot

Speeding up Distributed SGD for Non-convex Optimization

averaging in (McDonald et al., 2010; Zinkevich et al., 2010),wherein they show numerically that model averaging workswell in several optimization problems. Model averaging isshown to experimentally work well in (Chen & Huo, 2016;Kamp et al., 2018; Lin et al., 2018; McMahan et al., 2016;Povey et al., 2014; Su & Chen, 2015) by reducing com-munication costs; it is also shown to improve privacy andsecurity in the federated learning (McMahan et al., 2016).However, it is also known (Zhang et al., 2016) that one-shotaveraging leads to inaccurate models in non-convex opti-mization settings. The study of the benefits and trade-offsin model averaging, in particular its convergence properties,have been explored from a theoretical/analytical viewpointin (Chaudhari et al., 2017; Wang & Joshi, 2018; Yu et al.,2018; Zhang et al., 2015; Zhou & Cong, 2017).

Periodic averaging, a generalization of one-shot averagingwhere the models are averaged across nodes afters every τlocal updates; here we focus on a common periodic aver-aging technique sometimes referred to as parallel restartedSGD (PR-SGD)1 (Yu et al., 2018). Reference (Zhou &Cong, 2017) demonstrated that the convergence error ofPR-SGD grows with the number of local updates. Thismethod is empirically shown to work well in (Chaudhariet al., 2017). Recently, (Yu et al., 2018) improved the con-vergence rate of (Zhou & Cong, 2017) showing that thesame convergence rate as fully synchronous SGD can beachieved by properly tuning the number of local updates.Nonetheless, in (Yu et al., 2018; Zhou & Cong, 2017), theresidual error grows quadratically in τ . Recently, (Wang& Joshi, 2018) demonstrated that the residual error can belinear in the number of local updates τ , if every node cansample mini-batches from the entire data, rather than theirown locally stored partition. Interestingly, (Wang & Joshi,2018) also removed the bounded gradient assumption whichwas used in (Yu et al., 2018; Zhou & Cong, 2017).

Our algorithm can be viewed as a redundancy-infused vari-ant of PR-SGD discussed before. In our work, we showa residual error that grows linearly in τ similar to (Wang& Joshi, 2018). Importantly, we do not assume that eachnode draws samples from the entire data. Rather, we makethe more practical assumption that each node only samplesfrom the partitions that it stores locally. We discuss furthercomparisons of our work with (Wang & Joshi, 2018; Zhou& Cong, 2017) in Appendix C.

3. RI-SGD: Redundancy Infused SGDConsider a setting with p distributed worker nodes, each ofwhich has a disjoint chunk of data, Dj for 1 ≤ j ≤ p. Weare interested in solving the following optimization problem

1The term PR-SGD also helps distinguish the technique westudy from elastic-averaging SGD (Zhang et al., 2015), wherenodes perform a weighted average rather than a simple average.

in a distributed manner

minx∈Rd

F (x) ,1

p

p∑j=1

f(x;Dj), (2)

where the function f(·;Dj) is the training loss over thesamples in jth chunk Dj . Recall that a commonly studiedcentralized approach towards solving this optimization prob-lem is mini-batch stochastic gradient descent (SGD) for ran-domly sampled mini-batches ξ(t) ⊂ D = {D1 ∪ . . . ∪ Dp},which requires the same update rule as in (1), whereg(t) = 1

|ξ(t)|∇f(x(t); ξ(t)) is the stochastic unbiased gradi-

ent over a uniformly sampled mini-batch ξ(t) ⊆ D from allthe data samplesD. Here we study a decentralized approachto the above problem in homogeneous redundancy settingand defer the heterogeneous setting to Appendix D.

In homogeneous sample size setting, we assume that thedata budget at each worker node is γ , q

p , where q isa predetermined integer satisfying p ≥ q ≥ 1 and p isthe total number of chunks of the data set as well as thenumber of workers. The assumption that q is an integeris a mathematical simplification, as it is indicated in theexperiments with non-integer choice of q. We assume thatthe jth worker receivesDj and q−1 other distinct chunks ofdata from D1,D2, . . . ,Dj−1,Dj+1, . . .Dp. We define theredundancy factor as µ , q−1

p = γ− 1p . We denote the data

chunks at the jth worker as Dj , {Dj,1, . . . ,Dj,q} where1 ≤ j ≤ p and Dj,` ∈ {D1,D2, . . . ,Dp}, ` = 1, 2, . . . , q.We assume that the qth data chunk at each worker j is itsown local data Dj,q = Dj . In our algorithm, denoted byRI-SGD(τ) where τ represents the number of local updates,at iteration t worker j, 1 ≤ j ≤ p samples mini-batchesξ(t)j,i identically and independently from data chunk Dj,i for

1 ≤ i ≤ q. Then the samples are used to compute thestochastic gradient as follows:

g(t)j,i = ∇f(x

(t)j ; ξ

(t)j,i ) , g

(t)j ,

q∑i=1

g(t)j,i , (3)

Next, each worker, updates its own local version of themodel x(t)

j using x(t+1)j = x

(t)j − η g

(t)j . After every τ iter-

ations, we do the model averaging, where we average localversions of the model in all p workers. The pseudocode ofRI-SGD is shown in Algorithm 1. We note that the intro-duced algorithm does not necessarily imply data replicationin practice, although data replication leads to one naturalway of implementing the algorithm. Specifically, as long asevery worker can sample mini-batch samples from the datachunk of other workers (e.g., via network communication),our algorithm and results apply.

A few remarks and comparison with related works.The case of τ = 1 corresponds to one round of commu-nication for every iteration. The case of τ > 1 is referred to

Speeding up Distributed SGD for Non-convex Optimization

as periodic averaging in literature, and leads to lower com-munication cost albeit, with a residual error in the model. Inour results in the next section, we show that the residual errorgrows linearly with τ, the number of local updates.The caseof γ = 1/p corresponds to PR-SGD (Yu et al., 2018); thisalgorithm is shown to have a residual error that is quadraticin τ . Thus, our analysis can be viewed as an improvement ofthe analysis of (Yu et al., 2018). In addition, we can reducethe residual error by tuning the parameter γ.

If γ = 1, every node has access to all the chunks. Thiscase is comparable to (Wang & Joshi, 2018) as well to fullysynchronous mini-batch SGD where τ = 1, and the workerssample mini-batches from the entire data in every iteration.However, a subtle difference is that in our approach, wesample the data differently. The workers first sample mini-batches the various partitions of the data and then performupdates on the various mini-batches. Thus our approachleads to the samples being more uniformly spread across thedifferent partitions across different iterations; the effect ofthis difference is explored empirically in Section 5. Similarto (Wang & Joshi, 2018), we show that our residual erroronly linear in τ , even if γ < 1 meaning that the workersdo not have access to the entire data. The main technicaldifference between the γ = 1 case (e.g., (Wang & Joshi,2018)) and the γ < 1 case is that in the latter case, sinceeach worker samples a mini-batches from the its data chunks,instead of entire dataset, the mini-batch gradients are biased.This technical difference is the main challenge we overcomein our proofs of convergence analysis.

4. Convergence AnalysisIn this section, we present the convergence analysis of theproposed algorithm. We first state the main assumptionsand then present the convergence rates. All the proofs aredeferred to the appendix. In what follows, we use gj ,∑qi=1 gj,i to denote the full batch gradients over all q data

chunks at jth worker, where gj,i , ∇f(·;Dj,i) is the fullgradient over ith chunk.

4.1. Assumptions

Our convergence analysis is based on the following standardassumptions in non-convex optimization.

Assumption 1 (Unbiased estimation over chunks). Thestochastic gradient evaluated on a mini-batch ξj,i ⊂ Dj,iis an unbiased estimator of the partial full gradient, i.e.Eξj,i [gj,i|xj ] = gj,i for 1 ≤ j ≤ p and 1 ≤ i ≤ q.

Assumption 2 (Lipschitz continuity and lower bounded-ness). The objective function F (x) is differentiable and L-smooth, i.e., ‖∇F (x)−∇F (y)‖ ≤ L‖x−y‖, ∀x,y ∈ Rd.The function value is bounded below by a scalar F ∗.

Assumption 3 (Bounded variance). The variance of

Algorithm 1 RI-SGD(τ): Redundancy Infused SGD

1: Offline phase:2: The worker j receives set of random chunks Dj ,{Dj,1, . . . ,Dj,q}, where 1 ≤ j ≤ p and Dj,k 6= Dj,lfor k 6= l and 1 ≤ k, l ≤ q and Dj,q = Dj for 1 ≤ j ≤p.

3: Online-phase:4: Inputs: {D1, . . . , Dp} and x(1) as an initial global

model.5: for t = 1, 2, . . . , T do6: parallel for j = 1, 2, . . . , p do7: j-th worker forms the set of mini-batches as ξ(t)j ,

{ξ(t)j,1, . . . , ξ(t)j,q}, where ξ(t)j,k, k = 1, . . . , q is uniformly

sampled from kth chunk8: Evaluates stochastic gradient over each mini-

batches, g(t)j,i for i = 1, . . . , q, and computes g

(t)j as

in (3)9: if t divides τ do

10: x(t+1)j = 1

p

∑pj=1

[x(t)j − η g

(t)j

]11: else do12: x

(t+1)j = x

(t)j − η g

(t)j

13: end if14: end parallel for15: end16: Output: x(T ) = 1

p

∑pj=1 x

(T )j

stochastic gradients evaluated on a mini-batch ξj,i for all1 ≤ j ≤ p and 1 ≤ i ≤ q is bounded as

Eξj,i[‖gj,i − gj,i‖2|

]≤ C1‖gj,i‖2 +

C22

p(4)

where C1 and C2 are non-negative constants and inverselyproportion to the mini-batch size.

Remark 1. We note that, as p increases, the data chunksize reduces, and therefore, the mini-batch size increasesrelative the the size of the data chunk. As a result, the vari-ance reduces; this justifies the assumption that the varianceis inversely proportional to p. For further clarification, sup-pose that total number of the data points is n and let thedata chunk at each worker node have n

p data points. Then,let p = n

b where is the mini-batch size at each worker node.In this case, each data chunk at workers will have n

p = bdata points and therefore gj,i = gj,i. Therefore, for largerp and fixed number of the data points the variance reduces.

We make the following assumption about the partial fullgradients among different workers.

Assumption 4. We assume that the local full gradientsover pair of chunks for every pair of workers satisfies|〈gj,i,gj′,i′〉| ≤ β, ∀1 ≤ i, j, i′, j′ ≤ p, i 6= i′, j 6= j′.

We note that Assumption 4 is easily implied by the as-

Speeding up Distributed SGD for Non-convex Optimization

sumption of uniformly bounded gradient. To see this, wenote that if ‖gj,i‖ ≤

√β holds, a simple application of

Cauchy-Schwartz inequality implies that |〈gj,i,gj′,i′〉| ≤‖gj,i‖‖gj′,i′‖ = β. However, Assumption 4 does not nec-essarily imply uniformly bounded gradient assumption.

Finally, we note that since F (x) is a non-convex function,SGD may converge to a local minimum or a saddle point.We say the algorithm achieves an ε-submoptimal (Alistarhet al., 2017; Ghadimi & Lan, 2013; Lian et al., 2017) solu-tion if

E[ 1

T

T∑t=1

‖∇F (x(t))‖2]≤ ε,

where x(t) is average model at tth iteration. This conditionguarantees convergence of the algorithm to a stationarypoint, which will be used to show the convergence rate.

4.2. Main results

We now turn to provide the convergence analysis for Al-gorithm 1. To illustrate our key technical contribution inconvergence analysis, let us define an auxiliary variablex(t) = 1

p

∑pj=1 x

(t)j , which is the average model across p

different workers at iteration t. Using the definition of x(t),the update rule in Algorithm 1, becomes:

x(t+1) = x(t) − η[1

p

p∑j=1

g(t)j

](5)

As mentioned earlier, when γ < 1, the mini-batch stochasticgradients used in local updates are a biased estimate of thefull gradient. Specifically, the update rule in (5) can bere-written as

x(t+1) = x(t) − η∇F (x(t)) + η[∇F (x(t))− 1

p

p∑j=1

g(t)j

],

which establishes a connection between our algorithmand the perturbed SGD with deviation

(∇F (x(t)) −

1p

∑pj=1 g

(t)j

). We show that by introducing data redun-

dancy among workers, we can reduce the variance of biasedgradients and obtain the desired convergence rates.

We start by stating the convergence rate of RI-SGD(τ) algo-rithm.

Theorem 4.1. For RI-SGD(τ), under Assumptions 1 - 4,

if the learning rate satisfies η2L2(

2τC1(p+1p ) + τ(τ −

1))

+ ηL(C1+γp2)

p ≤ 1 and all local model parameters

are initialized at the same point x(1), for the given budget1p ≤ γ ≤ 1 the average-squared gradient after T iterations

is bounded by:

E[ 1

T

T∑t=1

‖∇F (x(t))‖2]≤

2[F (x(1))− F ∗

]ηT

+ ηLC22γ

p

+ γ(p+ 1

p)η2L2(τ − 1)C2

2 + (γ − 1)1

T

T∑t=1

G(t)

+p2

2

[3γ2 − 4γ + (3− 1

p)]β, (6)

where G(t) ,∑p

j=1 G(t)j and G

(t)j , minDi∈D\Dj ‖g

(t)j,i‖

2 for1 ≤ j ≤ p.

A few remarks regarding the convergence rate in Theo-rem 4.1 are in place. First, the obtained bound depends onthe diversity of local gradients parameter β which is alsoillustrated in our experimental results as well, in particu-lar Figure 3. Specifically, the rate increases with β, whichmeans that for larger values of β, more rounds of com-munication are needed to achieve the same error. This isconsistent with the result of (Yin et al., 2018), because thelarger β implies smaller gradient diversity. We also notethat due to our data allocation policy where each workersamples from multiple chunks of data instead of samplingfrom the entire dataset, the possibility of sampling the samedata by multiple worker nodes decreases, leading to fasterconvergence rate.

Corollary 4.1.1. The learning rate condition in Theorem4.1 implies that η ≤ 1

L√

2τC1(p+1p )+τ(τ−1)

. Therefore, for

the larger τ we need to choose smaller learning rate.

The following theorem strengthens Theorem 4.1 under amild condition on the gradients, which specializes to thefully synchronous distributed SGD in case of full data re-dundancy.

Theorem 4.2. For RI-SGD(τ), if for all 1 ≤ t ≤ T ,there exists constants ρ(t) > 0 such that

∑pj=1 ‖

∑i g

(t)j,i +∑

i g(t)j,i‖2 ≥ ρ(t)

∑pj=1

(∑i ‖g

(t)j,i‖2

), under Assump-

tions 1-4, the learning rate satisfies η2L2(

2τC1(p+1p ) +

τ(τ−1))

+ ηL(C1+γp2)

p < ρ(t) for 1 ≤ t ≤ T and all local

model parameters are initialized at the same point x(1), forthe given budget 1

p ≤ γ ≤ 1 the average-squared gradientafter τ iterations is bounded as follows:

E[ 1

T

T∑t=1

‖∇F (x(t))‖2]≤

2[F (x(1))− F ∗

]ηT

+ ηLC22

γ

p

+ (p+ 1

p)γC2

2η2L2(τ − 1) + 2p2

[(1− γ)

]β (7)

Remark 2. In Theorem 4.2, if we let τ = 1 and γ = 1, weget the convergence error terms reduce to the convergence

Speeding up Distributed SGD for Non-convex Optimization

error of fully synchronous SGD (Bottou et al., 2018). Ifwe let γ = 1 (i.e. full redundancy), the error bound in(7) reduces to the error bound provided by (Wang & Joshi,2018) which is proved to achieve a convergence rate ofO( 1√

pT). Therefore, by letting γ = 1 − O( 1

p2√pT

), and

η =√p

L√T

, we can achieve the convergence rate of O( 1√pT

).This is shown more formally in Appendix G. On the otherhand, if we let γ = 1

p , (7) gives the error for the casewhere there is no redundancy and each worker applies localupdates.

These theoretical bounds are validated through experimentalresults in Section 5 where we compare RI-SGD to PR-SGDalgorithm (µ = 0) proposed in (Yu et al., 2018) and (Wang& Joshi, 2018), in which it is assumed that each workercan sample its mini-batches from the entire dataset. Thedetailed comparison of convergence of various algorithmcan be found in Appendix C.

4.3. Distribution-aware convergence

Based on Assumption 4, we consider an upper bound β forthe correlation between each pair of local full gradients ofeach data chunk gj,i. Hence, convergence rates presentedin Theorems 4.1 and 4.2 do not explicitly reflect the effectsof each pair individually. In order to further elaborate onthe impact of distribution of data chunks on the final boundwe need to bound each pair independently. To that end,we introduce an upper bound on the correlation betweengradients of the main chunk at the jth worker (i.e. gj =∇f(·,Dj,q), recalling that the qth data chunk is local data)and gradients of data chunks not present in the jth worker(i.e. g

j,i= ∇f(·,Di),Di 6∈ Dj), as stated below.

Assumption 5. For 1 ≤ j ≤ p we assume an upper boundon the inner products of gradients of each worker’s datachunk and data chunks not presented in that worker asfollows:

|〈gj ,gj,i〉| ≤ βj,i, (8)

We note that by setting maxj,i βj,i = β, we can get theresults which only depends on the number of mini-batchesrather than the chunks. In Appendix F we derive the con-vergence of RI-SGD based on Assumption 5, where theconvergence is stated in terms of gradient dissimilarity be-tween different shards of data.

4.4. Robustness to node failure

While performing distributed optimization using iterativemethods such as distributed fully synchronous scheme,worker nodes have to exchange their gradients. If at eachround of communication certain straggling workers havedelays (or failed) in their computational task, this can causehuge delay in entire optimization process. One popular ap-proach (Chen et al., 2016) to deal with straggler nodes is

to ignore them. Our single and multi-round approach canoutperform a strategy that ignores stragglers by tuning thedegree of redundancy. To see this, let us assume the numberof failures is s. If we ignore failures in algorithm proposedin (Wang & Joshi, 2018), the convergence error upper bound

becomes E(τ, s) ,2[F (x(1))−F∗

]η(τ+T ) +

ηLC22

p−s +η2L2C22 (τ−1)

Now, if we consider the convergence error of RI-SGD(τ),we can see that it introduces the additional parameter γ, andignoring the stragglers in this algorithm results in conver-gence error

ERI-SGD(τ, s, γ) =2[F (x(1))− F ∗

]ηT

+ ηLC22

γ

(p− s)

+(γ − 1)

p− sp

T

T∑t=1

G(t) +q(p− s+ 1)

(p− s)2η2C2

2L2(τ − 1)

+p2

2

[3γ2 − 4γ + (3− 1

p)]β

where G(t) =∑p−sj=1 Gj . Then, by proper choice of γ = γs,

we can satisfy E(τ, s) > ERI-SGD(τ, s, γs), which indicatesthat taking advantage of the data redundancy, RI-SGD(τ) isresilient against failures.

4.5. Redundancy and intra-node gradient diversity

As we discussed in the convergence of RI-SGD, by inducingredundancy among workers, we can effectively reduce thevariance in local updates and reduce the number of commu-nication rounds and yet enjoying faster convergence. Wecan also investigate the effectiveness of redundancy infusionfrom gradient diversity perspective introduced in (Yin et al.,2018). In Appendix E, we mathematically demonstratethe intimate connection between redundancy and gradientdiversity and show that under mild assumptions, infusingredundancy increases the intra-node gradient diversity– theeffective quantity for the mini-batch size used for local up-dates at each worker node. A direct implication is that theeffective size of mini-batch can be increased without anysaturation or decay in performance.

5. ExperimentsIn this section, we experimentally evaluate the performanceof our RI-SGD algorithm with parameters redundancy µ =γ − 1/p = q−1

p and step size τ. We compare our algorithmwith fully synchronous SGD (SyncSGD), (τ = 1, µ = 0),PR-SGD (µ = 0, variable τ) and cooperative SGD (Wang& Joshi, 2018), where they consider each node has access tothe whole dataset for mini-batch sampling. We first describethe setup in which we conduct the experiment.

5.1. Experimental setup

We conduct different types of experiments on clusters with4 and 8 GPUs (Tesla V 100 and GeForce GTX 1080 Ti), to

Speeding up Distributed SGD for Non-convex Optimization

Figure 2. Training error and test accuracy for CIFAR10 and CIFAR100 on ResNet using syncSGD and RI-SGD with µ ∈ {0.0, 0.1, 0.25}and τ = 50 for both datasets. Note that RI-SGD(µ = 0) is PR-SGD. Top row is for CIFAR10 dataset, and bottom row is for CIFAR100dataset.

test the performance of the algorithm. We use CIFAR10and CIFAR1002, as well as ImageNet3 datasets, and chooseResNet (He et al., 2016) as the base model for training. Weuse Tensorflow (Abadi et al., 2016) to develop RI-SGD andSync-SGD in the distributed environment. In each experi-ment, the batch size for original Sync-SGD is 128 (unlessotherwise stated), and we will do the training on 45K and80K iterations for CIFAR10 and CIFAR100 datasets, re-spectively. For ImageNet dataset, we do the training on both4 and 8 GPUs, with batch size of the Sync-SGD set to 128and 256, and for 1.1M and 550K iterations, respectively.We present the results of experiments on CIFAR datasets onthe next section, and you can find the results on ImageNetdataset on Appendix A. All the experiments are based onheterogeneous mini-batch samples introduced in Section D.Hence, the redundancy value for each worker node is pickedfrom a normal distribution with the mean of the reportedredundancy value in the experiments (except for µ = 0 orno redundancy).

5.2. Experimental results

For RI-SGD we have two hyperparameters we can tune,the redundancy µ and number of local updates τ . Oneexploration is to see the effect of redundancy alone withoutany averaging step in the middle of training. This scenariohas been explained in Appendix A, and the results indicate

2Available at https://www.cs.toronto.edu/˜kriz/cifar.html

3Available at https://www.kaggle.com/c/imagenet-object-localization-challenge

that redundancy can, indeed, improve the convergence ratewith much less time compared to SyncSGD. After that, weshould find the optimum number for local updates, τ . Thisnumber introduces a trade-off in training, in which as itgoes up, the accuracy goes down, however the number ofcommunication rounds and hence the time for executionreduces. Based on this experiment’s results in Appendix A,it seems that in CIFAR datasets, this number should bearound τ = 50 to have the optimal trade-off.

Now, in the main experiment, we will set τ = 50 forboth datasets. We change the redundancy with valuesµ ∈ {0.0, 0.1, 0.25, 0.5} in order to see its effects onthe convergence. In our experiment we replicate datachunks cyclically, however, in general we can followany replication strategy as long as the redundancy bud-get is satisfied for all workers. For instance, in oursetup because we have 4 GPUS, in µ = 0.25, thenodes have {D1,D2}, {D2,D3}, {D3,D4}, {D4,D1} re-spectively. For µ = 0.1, the redundant part is obtainedas random sub-sampling, e.g., the first node has D1 and arandom subset of D2 to ensure that µ = 0.1. As it is shownin Figure 2, by increasing redundancy rate we get quiteclose to the baseline SyncSGD in terms of convergence, inan accelerated rate. We beat SyncSGD, not only in training,but also in test accuracy. We reach almost the same accu-racy level way sooner. Note that singificant improvementsin speed of training and testing occur even though RI-SGDhas higher computation cost, due to significantly lower timespent in communication.

Speeding up Distributed SGD for Non-convex Optimization

Figure 3. Training CIFAR10 using ResNet44 with τ = 50 for bothcases. Comparing the case, where each node has access to theentire dataset versus RI-SGD with full redundancy. Redundancycan lead to a better performance by introducing diversity in trainingdata input. Note that in this experiment, both runs have the batchsize of 512.

In the next experiment, we show that redundancy can bebeneficial in introducing gradient diversity (Yin et al., 2018)in training. To that end, we compare RI-SGD with twodifferent extreme cases of distributed local SGD. These twoextreme cases are when each node has a full access to thedataset and samples mini-batches from the entire dataset(e.g. cooperative SGD (Wang & Joshi, 2018)) or each nodehas only access to its chunk of the data for mini-batch sam-pling (e.g. PR-SGD (Yu et al., 2018)). For the first one, wecompare it with RI-SGD with full redundancy, where eachnode not only has access to its portion of data, but also canget mini-batches from every other node’s data chunk. Notethat the main difference between the two schemes is thenature of the sampling; in RI-SGD, each node’s data chunkcontributes to the mini-batch equally, while in cooperativeSGD we randomly sample mini-batches from the entiredataset. In this systematic redundancy infusion regime ofRI-SGD using non-i.i.d. sampling, the chances of havingoverlap in input data for different nodes are less than theformer case. In Figure 3, it is shown that RI-SGD withfull redundancy can outperform the simple model averagingwith every node having access to all data. To compare RI-SGD with PR-SGD, which is basically RI-SGD with µ = 0,we choose mini-batch size to be 512 and then compare itwith RI-SGD(µ = 0.25), with the same mini-batch size. Asit is depicted in Figure 4, redundancy can help to escape sat-uration by introducing gradient diversity, hence, can achievea better performance.

Figure 4. Training CIFAR10 using ResNet44 with τ = 50 for bothcases. Comparing PR-SGD with RI-SGD(µ = 0.25). Redundancycan lead to a better performance by introducing diversity usingredundancy in training data input and escape the saturation intraining. Note that in this experiment, both runs have the batchsize of 512.

6. ConclusionIn this paper, we advocate the use of data redundancy inperiodic model averaging techniques for distributed SGD.Through a theoretical convergence proof, we show that re-dundancy reduces residual error as compared with conven-tional algorithms where there is no redundancy. Throughexperimental results, we show that the redundancy, is wellworth the implied higher computation cost. The residualerror reduction leads to lower communication overheadsand therefore faster convergence. Furthermore, we alsodemonstrate empirically that our redundancy infused ap-proach has higher diversity, leading to superior algorithmsas compared to the previous ones where nodes can samplefrom the complete data.

This work leaves some interesting directions as future work.First, it would be interesting to investigate the significanceof redundancy in federated optimization to reduce sensitivityof SGD to the variance in data. Furthermore, understandingmore effective replication schemes, e.g. sketching methods,when data in different workers has some low-rank structureis worthy of investigation.

AcknowledgementThis work was partially supported by the NSF CCF 1553248and NSF CCF 1763657 grants. The first author would liketo thank Dr. Kang Wook Lee to bring the paper (McMahanet al., 2016) to his attention.

Speeding up Distributed SGD for Non-convex Optimization

ReferencesAbadi, M., Barham, P., Chen, J., Chen, Z., Davis, A., Dean,

J., Devin, M., Ghemawat, S., Irving, G., Isard, M., et al.Tensorflow: a system for large-scale machine learning.In OSDI, volume 16, pp. 265–283, 2016.

Aji, A. F. and Heafield, K. Sparse communica-tion for distributed gradient descent. arXiv preprintarXiv:1704.05021, 2017.

Alistarh, D., Grubic, D., Li, J., Tomioka, R., and Vojnovic,M. Qsgd: Communication-efficient sgd via gradient quan-tization and encoding. In Advances in Neural InformationProcessing Systems, pp. 1709–1720, 2017.

Bernstein, J., Wang, Y.-X., Azizzadenesheli, K., and Anand-kumar, A. Signsgd: Compressed optimisation for non-convex problems. In International Conference on Ma-chine Learning, pp. 559–568, 2018.

Bottou, L., Curtis, F. E., and Nocedal, J. Optimizationmethods for large-scale machine learning. SIAM Review,60(2):223–311, 2018.

Chaudhari, P., Baldassi, C., Zecchina, R., Soatto, S., Tal-walkar, A., and Oberman, A. Parle: parallelizing stochas-tic gradient descent. arXiv preprint arXiv:1707.00424,2017.

Chen, J., Pan, X., Monga, R., Bengio, S., and Jozefowicz, R.Revisiting distributed synchronous sgd. arXiv preprintarXiv:1604.00981, 2016.

Chen, K. and Huo, Q. Scalable training of deep learning ma-chines by incremental block training with intra-block par-allel optimization and blockwise model-update filtering.In Acoustics, Speech and Signal Processing (ICASSP),2016 IEEE International Conference on, pp. 5880–5884.IEEE, 2016.

Cui, H., Cipar, J., Ho, Q., Kim, J. K., Lee, S., Kumar, A.,Wei, J., Dai, W., Ganger, G. R., Gibbons, P. B., et al.Exploiting bounded staleness to speed up big data analyt-ics. In USENIX Annual Technical Conference, pp. 37–48,2014.

Dean, J., Corrado, G., Monga, R., Chen, K., Devin, M.,Mao, M., Senior, A., Tucker, P., Yang, K., Le, Q. V., et al.Large scale distributed deep networks. In Advances inneural information processing systems, pp. 1223–1231,2012.

Dryden, N., Moon, T., Jacobs, S. A., and Van Essen, B.Communication quantization for data-parallel trainingof deep neural networks. In Machine Learning in HPCEnvironments (MLHPC), Workshop on, pp. 1–8. IEEE,2016.

Ghadimi, S. and Lan, G. Stochastic first-and zeroth-ordermethods for nonconvex stochastic programming. SIAMJournal on Optimization, 23(4):2341–2368, 2013.

Gupta, S., Zhang, W., and Wang, F. Model accuracy andruntime tradeoff in distributed deep learning: A system-atic study. In Data Mining (ICDM), 2016 IEEE 16thInternational Conference on, pp. 171–180. IEEE, 2016.

Haddadpour, F., Yang, Y., Cadambe, V., and Grover, P.Cross-iteration coded computing. In 2018 56th AnnualAllerton Conference on Communication, Control, andComputing (Allerton), pp. 196–203. IEEE, 2018.

He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learn-ing for image recognition. In Proceedings of the IEEEconference on computer vision and pattern recognition,pp. 770–778, 2016.

Jiang, Z., Balu, A., Hegde, C., and Sarkar, S. Collaborativedeep learning in fixed topology networks. In Advances inNeural Information Processing Systems, pp. 5904–5914,2017.

Jin, P. H., Yuan, Q., Iandola, F., and Keutzer, K. Howto scale distributed deep learning? arXiv preprintarXiv:1611.04581, 2016.

Kamp, M., Adilova, L., Sicking, J., Huger, F., Schlicht, P.,Wirtz, T., and Wrobel, S. Efficient decentralized deeplearning by dynamic model averaging. arXiv preprintarXiv:1807.03210, 2018.

Li, M., Andersen, D. G., and Park, J. W. Scaling distributedmachine learning with the parameter server.

Lian, X., Zhang, C., Zhang, H., Hsieh, C.-J., Zhang, W., andLiu, J. Can decentralized algorithms outperform central-ized algorithms? a case study for decentralized parallelstochastic gradient descent. In Advances in Neural Infor-mation Processing Systems, pp. 5330–5340, 2017.

Lin, T., Stich, S. U., and Jaggi, M. Don’t use large mini-batches, use local sgd. arXiv preprint arXiv:1808.07217,2018.

McDonald, R., Hall, K., and Mann, G. Distributed train-ing strategies for the structured perceptron. In HumanLanguage Technologies: The 2010 Annual Conferenceof the North American Chapter of the Association forComputational Linguistics, pp. 456–464. Association forComputational Linguistics, 2010.

McMahan, H. B., Moore, E., Ramage, D., Hampson, S., et al.Communication-efficient learning of deep networks fromdecentralized data. arXiv preprint arXiv:1602.05629,2016.

Speeding up Distributed SGD for Non-convex Optimization

Mitliagkas, I., Zhang, C., Hadjis, S., and Re, C. Asynchronybegets momentum, with an application to deep learning.In Communication, Control, and Computing (Allerton),2016 54th Annual Allerton Conference on, pp. 997–1004.IEEE, 2016.

Povey, D., Zhang, X., and Khudanpur, S. Parallel trainingof dnns with natural gradient and parameter averaging.arXiv preprint arXiv:1410.7455, 2014.

Recht, B., Re, C., Wright, S., and Niu, F. Hogwild: A lock-free approach to parallelizing stochastic gradient descent.In Advances in neural information processing systems,pp. 693–701, 2011.

Seide, F., Fu, H., Droppo, J., Li, G., and Yu, D. 1-bit stochas-tic gradient descent and its application to data-paralleldistributed training of speech dnns. In Fifteenth AnnualConference of the International Speech CommunicationAssociation, 2014.

Strom, N. Scalable distributed dnn training using commod-ity gpu cloud computing. In Sixteenth Annual Conferenceof the International Speech Communication Association,2015.

Su, H. and Chen, H. Experiments on parallel training of deepneural network using model averaging. arXiv preprintarXiv:1507.01239, 2015.

Wang, J. and Joshi, G. Cooperative sgd: Aunified framework for the design and analysis ofcommunication-efficient sgd algorithms. arXiv preprintarXiv:1808.07576, 2018.

Wangni, J., Wang, J., Liu, J., and Zhang, T. Gradient spar-sification for communication-efficient distributed opti-mization. In Advances in Neural Information ProcessingSystems, pp. 1306–1316, 2018.

Ye, M. and Abbe, E. Communication-computation efficientgradient coding. arXiv preprint arXiv:1802.03475, 2018.

Yin, D., Pananjady, A., Lam, M., Papailiopoulos, D., Ram-chandran, K., and Bartlett, P. Gradient diversity: a keyingredient for scalable distributed learning. In Interna-tional Conference on Artificial Intelligence and Statistics,pp. 1998–2007, 2018.

Yu, H., Yang, S., and Zhu, S. Parallel restarted sgd for non-convex optimization with faster convergence and lesscommunication. arXiv preprint arXiv:1807.06629, 2018.

Zhang, J., De Sa, C., Mitliagkas, I., and Re, C. Paral-lel sgd: When does averaging help? arXiv preprintarXiv:1606.07365, 2016.

Zhang, S., Choromanska, A. E., and LeCun, Y. Deep learn-ing with elastic averaging sgd. In Advances in NeuralInformation Processing Systems, pp. 685–693, 2015.

Zhou, F. and Cong, G. On the convergence propertiesof a k-step averaging stochastic gradient descent al-gorithm for nonconvex optimization. arXiv preprintarXiv:1708.01012, 2017.

Zinkevich, M., Weimer, M., Li, L., and Smola, A. J. Paral-lelized stochastic gradient descent. In Advances in neuralinformation processing systems, pp. 2595–2603, 2010.