CATERPILLAR: Coarse Grain Reconfigurable Architecturefor Accelerating the Training of Deep Neural Networks

Yuanfang Li∗, Ardavan Pedram∗†∗Stanford University, †Cerebras Systems{yli03,perdavan}@stanford.edu

Abstract—Accelerating the inference of a trained DNN is a wellstudied subject. In this paper we switch the focus to the trainingof DNNs. The training phase is compute intensive, demandscomplicated data communication, and contains multiple levelsof data dependencies and parallelism. This paper presents analgorithm/architecture space exploration of efficient accelera-tors to achieve better network convergence rates and higherenergy efficiency for training DNNs. We further demonstratethat an architecture with hierarchical support for collectivecommunication semantics provides flexibility in training vari-ous networks performing both stochastic and batched gradientdescent based techniques. Our results suggest that smallernetworks favor non-batched techniques while performance forlarger networks is higher using batched operations. At 45nmtechnology, CATERPILLAR achieves performance efficienciesof 177 GFLOPS/W at over 80% utilization for SGD trainingon small networks and 211 GFLOPS/W at over 90% utilizationfor pipelined SGD/CP training on larger networks using a totalarea of 103.2 mm2 and 178.9 mm2 respectively.

1. Introduction

State of the art Deep Neural Networks (DNNs) are be-coming deeper and can be applied to a range of sophisticatedcognitive tasks such as image recognition [1] and naturallanguage processing [2]. Convolutional Neural Networks(CNNs) [1] and Recurrent Neural Networks (RNNs) [2] aresome of the commonly used network architectures that areinspired by the Multilayer Perceptron (MLP) [3]. Most ofthe community has focused on acceleration of the forwardpath/inference for DNNs, neglecting the acceleration fortraining [4], [5]. Training DNNs is a performance and energycostly operation that routinely takes weeks or longer onservers [6]. This makes the task of navigating the hyper pa-rameter space for network architecture expensive. Howeverthe nature of computation in training DNNs makes it an ex-cellent candidate for specialized acceleration if the necessarycomputation/communication functionality is supported [7].

Today, acceleration of the training process is primarilyperformed on GPUs [6]. However, GPUs suffer from funda-mental computation, memory, and bandwidth imbalance intheir memory hierarchy [5]. Thus, the fundamental questionis what are the compute architecture, memory hierarchy, andalgorithmic tradeoffs for an accelerator designed to traindeep neural networks. In this paper we aim to address thedesign tradeoffs by introducing a Coarse Grain Reconfig-urable Architecture (CGRA) for training MLPs.

We focus on training MLPs, an important class of DNNscurrently used on state of the art servers [5], with severalvariants of Backpropagation (BP) [8] learning. Although

Figure 1. An MLP with 2 hidden layers h1 and h2.

the computation and the communication graph for CNNsand RNNs are more complex compared to MLPs, we choseMLPs for the following reasons. First, MLPs are inherentlymemory bound and more challenging to accelerate [9].Second, several research studies on the principles of BP andoptimization in DNNs investigate MLPs because of theirsimpler network-architecture [10], [11], [12]. Finally, MLPsrepresent the fully-connected layers of CNNs [1]. Hence,we believe this effort establishes a platform for future workon accelerating the training of CNNs and RNNs.

The challenge in training DNNs is that the fastest con-verging gradient descent algorithms in terms of passes overthe dataset are not efficient in conventional architecturesas they are based on GEneral Matrix-Vector multiplication(GEMV), which is a bandwidth limited operation. For sparsenetwork weights, sparse matrix-vector multiplication is evenless compute intensive. However, as the size of the networkshrinks, and is able to fit on local storage, the overallcost of communication drops [13]. GPUs use a variant ofDNN training that groups samples into batches in order toovercome bandwidth limitations at the cost of more epochsto convergence and perform GEneral Matrix-Matrix multi-plication (GEMM), which is a compute intensive kernel.

Given the fundamental differences in various techniquesfor training MLPs, we aim to compare the design spaceof accelerators and the memory hierarchy for various con-figurations of networks and training algorithms. This papermakes the following contributions:

• Techniques to decrease dependencies, and increaselocality and parallelism for training DNNs and theireffect on convergence.

• Exploration of the design space of accelerators forvarious BP algorithms.

• CATERPILLAR: an efficient architecture for train-ing DNNs targeting both GEMV and GEMM kernelsexploiting optimized collective communications.

• Evaluation of the costs of convergence of variousnetworks, training algorithms, and accelerators withrespect to both performance and energy efficiency.

arX

iv:1

706.

0051

7v2

[cs

.DC

] 8

Jun

201

7

time

t

x h1 h2 h3 ŷh11 h21 h31

δ31δ21δ11

h12 h22 h32

δ32δ22δ12

h1i h2i h3i

δ3iδ2iδ1i

x h1 h2 h3 ŷh11:4 h21:4 h31:4

δ31:4δ21:4

δ11:4

h15:8 h25:8 h35:8

δ35:8δ25:8δ15:8

x h1 h2 h3 ŷh11 h21 h31

x h1 h2 h3 ŷh11:4 h21:4 h31:4

h15:8 h25:8 h35:8

δ35:8

b c daFigure 2. Methods of learning. Each arrow demonstrates one sample’s movement through the forward and backward path in time for a network with threehidden layers. (a) SGD, (b) MBGD, (c) DFA, (d) CP.

The rest of the paper is organized as follows: Sec-tion 2 describes various BP algorithms for training DNNs.Section 3 discusses the proposed CGRA, and the algo-rithm/architecture tradeoffs. Section 4 presents the exper-imental results. Section 5 reviews related work. Finally, weconclude in Section 6.

2. Multilayer Perceptron (MLP)

The key purpose of the MLP is to output a prediction(generally a classification label) for a given input. Figure 1shows a three layered MLP, where x is the input, y is theoutput, and h1 and h2 are the activations of the first andsecond hidden layers respectively. Wi is the set of weightssuch that Wi(j, k) is the weight from the jth element of theinput to the kth element of the output at layer i. At eachneuron of a hidden layer a nonlinear activation function fis performed on the sum of the weighted inputs. At theoutput layer the softmax function turns y into a vector ofprobabilities for the possible classes. A bias can be addedby appending a +1 term to the input vector at each layer.

The network is trained on input-label pairs (x, y) wherex is the input as before and y is the correct label representedas a one-hot vector. Training is performed in two stages.During the forward pass the prediction, y, is calculated:

a1 = xTW1, h1 = f(a1)

ai = hTi Wi, hi+1 = f(ai)

ay = hT2W3, y = softmax(ay)

During the backward pass, the error in the prediction iscalculated at the output and backpropagated to previouslayers to provide an estimate of the weight’s gradient withrespect to the error. The weights are then updated usinggradient descent:

e = y − yδ2 = e� f ′(h2)WT

3

δ1 = δ2 � f ′(h1)WT2

W3 =W3 − ηhT2 eW2 =W2 − ηhT1 δ2W1 =W1 − ηxT δ1

It is apparent that MLP training consists of a series of largeGEMV and GEMM operations, making it an ideal candidatefor specialized hardware acceleration. Once the MLP modelis trained, inference is performed on an unseen sample byrunning the forward pass only. Although MLP is a simplerneural network compared to CNNs and RNNs, by increasingthe size and number of the hidden layers as well as thenumber of training samples, it performs as well as othercomplex models on tasks like digit classification [14].

2.1. Stochastic/Minibatch Gradient Descent(SGD/MBGD)

In SGD (Figure 2(a)), each training sample goes throughthe forward and backward passes and updates the weights.Thus, a single pass through a training set of size K resultsin K updates to the weights [8]. MBGD (Figure 2(b)) isa variant of SGD that groups training samples into ”mini-batches”. The weights are updated after each minibatchgoes through the network so that a single pass through thetraining set results in K/b weight updates, where b is thesize of the minibatch. This allows some parallelization ofthe training process as several samples can be processed atonce. However, the sequential nature of BP still preventsparallelization across layers.

2.2. Feedback Alignment (FA)

MLPs are meant to mimic how the brain works, but aBP algorithm such as SGD is not biologically plausible asit requires weight symmetry in the forward and backwardpass. To circumvent this, Lillicrap et al. propose the use ofa fixed random feedback weight, Bi, for each layer duringthe backward pass [15]. The change in learning is:

e = y − yδ2 = eBT

2 � f ′(h2)δ1 = δ2B

T1 � f ′(h1)

Despite the use of random feedback weights, FA can per-form as well or better than standard MBGD. However, it isnecessary to either use batch normalization or significantlylower the learning rate to prevent the gradients from growingtoo large, especially for deep networks [10].

2.3. Direct Feedback Alignment (DFA)

DFA (Figure 2(c) [16]) backpropagates the last layer’sweights to all previous layers as follows:

e = y − yδ2 = eBT

2 � f ′(h2)δ1 = eBT

1 � f ′(h1)

Like FA, DFA also requires either batch normalization or asmaller learning rate. However for a typical neural network,the dimension of the output layer is significantly smallerthan the hidden layers. This means that the Bi’s will besignificantly smaller than their corresponding Wi’s, reducingthe number of computations required to backpropagate theerror by up to several orders of magnitude. Figure 2(c)shows DFA also introduces parallelism during the backwardpass as the error for all layers can be calculated at once.

2.4. Pipelined/Continuous Propagation ((MB)CP)

Continuous propagation allows parallelization acrosslayers by relaxing the constraint that a full forward andbackward pass through the network is required for eachset of samples before the weights can be updated [17]. InCP, forward and backward passes through a layer can occursimultaneously - in particular the weights can be applied tothe current sample on its forward pass at the same time thatthe previous sample updates the weights on its backwardpass. Figure 2(d) demonstrates the CP algorithm as samplesare propagated through the layer and time. Once the pipelinehas been initialized, all layers are simultaneously working.

3. CATERPILLAR Architecture

This section motivates the design of the CATERPILLARarchitecture by demonstrating the available parallelism andlocality for each of the existing MLP training algorithmsin Section 2. We then build on these insights and proposean architecture that provides the required computation andcommunication functionalities.

3.1. Discussion of Various Algorithms with Respectto Locality and Parallelism

Training the neural network requires a series of GEMVand/or GEMM operations. There are two cases to consider:(1) the entire network’s weights fits in the local memoriesand (2) it does not fit and some of the weights must bestored off-core. Below we briefly describe various methodsof exploiting parallelism and locality in the different trainingmethods.

In the SGD algorithm, only a single sample passesthrough the entire network at once. Thus, the compute kernelof this algorithm is GEMV, which is memory bound andinefficient, especially when the entire network does not fitin the aggregate local memories of each core. For a matrix of

size m×n GEMV requires mn weight accesses to producen elements of the output. Thus each of the proposed variantson BP aims to overcome this drawback by introducing andexploiting various sources of parallelism and locality.

Note that even in case (1) when the entire networkis stored locally, SGD remains inefficient as GEMV itselfis inherently an inefficient operation especially when per-formed on a 2D array of processing elements. In additionto broadcasting the input vector across one dimension, areduction must be performed across the opposite dimensionto sum up the partial sums into a single output vector.Consequently, there are m broadcasts and n reductions totransform an input of size m to an output of size n.

Data parallelization: Algorithms that use minibatches(MBGD, MBCP, DFA) exploit parallelism in sample data bypassing several samples through the network at once. Thistransforms the computations in each layer of the networkinto a series of GEMM operations. Furthermore, by accu-mulating the gradient estimates from several samples intoone weight update, the number of memory accesses neededfor the weights is divided by the batch size. The further theminibatch size is increased, the greater parallelism achiev-able, but if the minibatch size is too large, the algorithmwill fail to converge to an acceptable value. In practice,minibatch sizes on nodes range from 2 to 100 [19] and cango up to 10,000+ on clusters [20].

An important caveat of data parallelization is that for allBP algorithms, the activations calculated during the forwardpass must be stored for use during BP. Thus as the minibatchsize grows, more memory is needed to store the activations.

Activation locality: During the backward pass, the acti-vations, hi, computed during the forward pass are needed inorder to update the weights, introducing activation localitybetween the forward and backward pass. However, layers arevisited in the reverse order during BP so that for a singlesample, the first activation to be produced is the last oneto be consumed. For SGD/CP, the size of the activations isnegligible compared to the weights, in which case, the acti-vations can be stored without incurring significant memoryoverhead. While activation size is still smaller than weightsize for algorithms using minibatches, as the network growsdeeper, the total size of activations to be stored grows andthe memory overhead becomes a concern. For a networkwith L layers trained using minibatches of size b, if thefirst layer is m × n, the size of activations to be stored isLbn. If Lb ≥ n, this becomes larger than the size of thelayer’s weights. This issue is more evident in CNNs thanMLPs, as they are usually deeper and have huge samplesizes [21]. This issue is mitigated with reverse checkpointingand recomputing the activations in earlier layers [22]. Here,only the activations for some layers are saved. During BPwhen an activation that has not been saved is needed, thenetwork propagates the last saved activation for that inputthrough the forward path again.

Layer parallelization and weight locality: CP intro-duces layer parallelization by pipelining the samples throughthe network. Instead of distributing the weights for a layerto all cores, as in SGD, they can be distributed to a subset

0,0 0,1 0,15

15,0SRAMSPAD7

SRAMSPAD6

SRAMSPAD5

SRAMSPAD4

Core0

Core1

Core2

Core3

7 6 5 4

SRAMSPAD0

SRAMSPAD1

SRAMSPAD2

SRAMSPAD3

PE(0,0)

PE(0,1)

PE(0,2)

PE(0,3)

PE(1,0)

PE(1,1)

PE(1,2)

PE(1,3)

PE(2,0)

PE(2,1)

PE(2,2)

PE(2,3)

PE(3,0)

PE(3,1)

PE(3,2)

PE(3,3)

`

MEM B

Address Regs

Row Bus Write

Column Bus Write

A B

µ programmed Controller

Column Bus Read

Row Bus Read

MACAccumulator

Cin

Memory Interface

RFMEM A

`

MEM B

Address Regs

Column Bus Write

A B

µ programmed Controller

MACAccumulator

Cin

RFMEM A

a) b) c)

16Columns

16rows

to/fromCoreInSameColumn

to/fromCoreInSam

erow

Figure 3. CATERPILLAR: (a) Array of cores with ring communication; (b) core with 16× 16 PEs connected to column and row broadcasts; (c) PE [18].

of cores, creating a pipeline of layers mapped on the coresand allowing all layers to be processed simultaneously. Witheach GEMV operation distributed to a smaller number ofPEs, the number of reductions needed decreases, increasingutilization. Additionally, the ability to perform the forwardand backward passes through a layer simultaneously allowsthe weights updated by the backward pass to be appliedimmediately to the forward activations, thus decreasing thememory accesses for the weights by half compared to SGD.

Dependency elimination: By propagating the outputlayer’s error to all previous layers, DFA eliminates thedependency between layers during the backward pass, al-lowing all weight updates to proceed in parallel. If the sizeof the network is large enough that all compute units arebusy, this provides little advantage. However, in the caseof small networks, this allows parallelization of the layersduring the backward pass.

3.2. CGRA for Training

The presented locality and parallelism explorationdemonstrates that for networks which do not fit in the localmemory of the cores, an architecture optimized for GEMMwill perform well by performing a minibatch learning algo-rithm. However, the same architecture must also support thehigh communication demands of GEMV operations if thenetwork is small enough to be stored locally, in which caseeither SGD or CP can be used to train the network withoutincurring memory access and communication overheads.

Since GEMM and GEMV inherently use the same innerkernel, an architecture inspired by an array of Linear Alge-bra Cores (LACs) [18], [23] will perform well. The LACconsists of nr × nr Processing Elements (PEs). Each PEcontains a half precision floating point multiply-accumulateunit (FPU) and a local SRAM. PEs are connected acrosscolumns and rows by low-overhead broadcast buses. Thearchitecture uses a 2 × C array of these cores connectedin a systolic fashion that can support unidirectional ringcommunication. Communication between cores is systolicsuch that the number of cycles to pass data from one coreto another is equal to the distance between the cores. Eachcore also has its own private off-core memory. Figure 3shows the multicore architecture for an array of 2× 4 coreswith 16 × 16 PEs each. The following section presents adetailed description of how different training methods mapto the architecture.

3.3. Mapping of Various Learning Methods

CP/MBCP: To perform CP, the architecture must sup-port fast broadcast and reduction of partial products betweenand within the core for GEMV. To compute the matrix-vector multiplication within each core, the weights are dis-tributed to the array of PEs in 2D round robin fashion andthe input vector to the layer is broadcast across the rowbuses. Each PE performs a MAC operation and produces apartial sum of the output vector. To sum the partial products,each PE broadcasts its value along the column bus and thesevalues are summed together in the diagonal PEs, whichbroadcasts the final output vector out along the row buses.

Layers of the network may not be the same size, thuslarger layers must be assigned to more cores to keep thecores busy with a stream of activations. To address the lackof symmetry between the number of cores across layers, amethod of reducing partial sums across a non-square arrayof cores is demonstrated in Figure 4 for the case of twocores exploiting the diagonal PEs in each core.

In the forward pass (a), each core receives a portion ofthe input activation and calculates a partial product of theoutput. Each core then performs the reduction internally,and subsequently passes its result to the other to sum itto the final output (c). In the backward pass (b), the inputis broadcast across both cores and each core reduces withinitself to produce a portion of the output (d). Note that GEMVproduces a transposed output, thus in (a) the reduction mustoccur in the diagonal PEs and in (b) the diagonal PEsbroadcast to produce an untransposed output.

The LAC architecture contains fast broadcast buses.However, the reduction operation remains expensive as eachPE in a column must broadcast its partial sum, with (nr−1)cycles required to produce one output for an array of nr×nrPEs. As the reduction is performed on the diagonal PEs,the remaining PEs are idle during this time. Fortunately, thepipelined nature of CP allows an overlap in computationof the next sample either in the forward or backward passwith reduction of the current sample. There is no conflictin the broadcast buses, as reductions use the column busesand broadcasts during computation use the row buses (this isreversed for the backward pass). Thus the only overhead inCP is the extra cycles required to fill and empty the pipeline,which is proportional to the depth of the network.

SGD: For the SGD algorithm, since there is limitedparallelism between layers, a single GEMV operation is

InputActivation

OutputActivation

Broadcast

Reduce

Transpose&sendintime

Core1Partition

Core2Partition

CurrentLayer’sweights

tonextlayer

Frompreviouslayer

Outputdelta

Inputdelta

Backtopreviouslayer

Reduce

Transpose

Core1Partition

Core2Partition

Reduce

Broadcast

Backfromnextlayer

Broadcast,Transpose,andreduceonthecore

Reduce1

Reduce2

Broadcast

Broadcast

Reduce

(a)

(b)

(c) (d)Broadcast

Broadcast

Figure 4. GEMV across multiple cores: (a), (c) forward pass through layer;(b), (d) backward pass through layer.

performed by all of the PEs. This means the computation forall layers will be performed sequentially and by the samePEs. The mapping of CP and SGD are therefore similarwith one major difference: Instead of passing the result ofthe current layer to the next set of cores, it is rebroadcastto perform the GEMV operation for the next layer.

The expensive overhead of reduction can drop the uti-lization to half. To address the reduction overhead, directcommunication is added between neighboring PEs. Thisdrops the cost of reduction of nr elements in nr PEsfrom nr − 1 to log(nr) − 1 as it allows more parallelcommunication between short distance PEs.

MBGD: For MBGD with small batch sizes, trainingoccurs in the same method as SGD. During minibatchtraining with larger batches, all cores are working on thesame layer and produce a single matrix output. Within cores,GEMM occurs as described in [18], but cores must also nowbe able to pass results to each other between layers.

In the forward pass, the ith core contains a row panelWi, of the weights and accesses all elements of the inputactivation X , to produce a row block Yi of the outputactivation. Below we show an example for three cores:[

Y1

Y2

Y3

]=

[W1

W2

W3

][X]

In order to make the complete output Y available to allcores as input for the next layer, an all-gather [24] operation

is performed between layers where each core passes its Yito the next, using the ring communication. For a ring of2 × C cores with n2r PEs per core,(nb − nb/c)/nr cyclesare required to communicate an output of size n× b.

In the backward pass, the errors need to be multipliedby the transpose of the weights from the forward pass. Aseach core’s off-core memory contains only a portion of theweights, a transpose is performed with the weights in placeby changing the input. Each core now contains a columnpanel of the weights WT

i and receives a row block of theinput error, Y T

i , then calculates a partial product of theoutput error:

[X1] + [X2] + [X3] = [WT1 WT

2 WT3 ]

[Y1

Y2

Y3

]To obtain the final output a reduce-scatter [24] operation isperformed. The systolic ring communication between coresis used with the same communication overhead as for all-gather.

FA: FA is best used when memory limitations are not anissue, since twice the weight accesses and storage is requiredfor both Wi and Bi compared to SGD. In addition, the needfor batch normalization makes FA impractical architecturallysince the inputs to each layer must either be normalized orthe network must be trained for more epochs to achievesimilar convergence as MBGD. Liao et al. also showedthat the use of feedback alignment led to no performanceimprovement over traditional gradient descent when appliedto a MLP network trained on TIMIT dataset [10]. Thusfeedback alignment is not further considered in the currentstudy on MLPs although it may be revisited for CNNs.

DFA: The difference between DFA and MBGD is thatduring the backward pass, error is not propagated betweenlayers, i.e., the output error of the current layer is notneeded as input to the next layer. However the reduce-scatteroperation is still required to sum the partial sums to obtainthe final output, thus the mapping to the architecture is thesame.

Activation Function: For all algorithms, to calcu-late the nonlinear activations at each layer, Goldschmidt’smethod [25] [26] [27] is used, which can be implementedwith a lookup table and the existing FPU in the PE. Calcula-tion of the activation thus requires a local memory access tothe lookup table and a few iterations of multiply and accu-mulate operations. Note that the derivative of the activationrequired during the backward pass can be easily calculated,as for typical nonlinearities it is a linear function of theactivation itself, i.e. for the sigmoid activation function it isσ′(x) = σ(x)(1− σ(x)).

3.4. Architectural Tradeoffs

In a single epoch, each sample passes through thenetwork once on the forward pass and once on backwardpass for all BP algorithms, but the number and manner ofweight updates varies. Thus the number of FPU operationsrequired for each algorithm is the same and only the amount

of memory accesses, local storage and overhead differs.The exception is the DFA algorithm, which calculates eachhidden layer’s error using the last layer’s error. For all otheralgorithms, each sample has a forward pass, a backwardpass and a gradient calculation for all layers, resulting in

a total of 3KL∑

i=1

mini MAC operations for a network of

size L with mi×ni layers trained on K samples. For DFA,

the backward pass requires only KL∑

i=1

minL operations as

each set of weights is applied to the last layer’s error. Theactivation function for each layer must also be applied toeach sample, but this is negligible compared to the cost ofthe matrix multiplications.

For SGD and CP, which use GEMV operations, toavoid high communication cost with off-core memory, thenetwork must be able to fit onto the local core memory.In addition to storing the weights for each layer, the in-put and activation for the layer must also be stored foruse during BP. When performing the GEMV operation,extra memory is also required to store the partial sums,thus the total memory required for the network to fit isL∑

i=1

(L − i + 1)(mi + ni +max(mi, ni) +mini)/(2Cn2r),

where n2r is the number of PEs/core and 2C is the numberof cores. If this is larger than the available memory, partof the network must be stored off-core, which will impactthe effective utilization if the off-core memory bandwidth isinsufficient.

For MBGD, the weights can be stored off-core and thuslocal memory is only required to store the activations, which

are sizeL∑

i=1

(L− i+ 1)(mi + ni)b where b is the minibatch

size. If this is larger than the available memory, only partof the activations can be stored and reverse checkpointingmust be used.

In SGD, DFA and MBGD, the weights are accessed onceduring the forward pass and once during the backward passfor each weight update performed. Thus for a single epoch

over K samples, SGD requires 2KL∑

i=1

mini accesses while

MBGD requires (2K/b)L∑

i=1

mini accesses. DFA also re-

quires an additional (K/b)L∑

i=1

minL accesses to the random

weights Bi during the backward pass. Additional memoryaccesses are required to apply the activation function duringthe forward pass and access the activation during the back-

ward pass but these are on the order of KL∑

i=1

ni and thus

negligible compared to weight accesses.The CP and MBCP algorithms allow the weights to be

accessed only once for both the forward and backward pass,thus the number of memory accesses is halved compared to

SGD/MBGD to (K/b)L∑

i=1

mini.

TABLE 1. ENERGY AND AREA FOR FPU AND SRAM BLOCKS.Energy/Op Area

Half-precision FPU 2.63pJ 0.0056mm2

16KB Local SRAM 3.5pJ (per 2 bytes) 0.0617mm2

512KB Off-Core SRAM 16pJ (per 2 bytes) 1.948mm2

4. Evaluations

To study the interplay between algorithms and archi-tecture we perform two classes of studies. First, we studythe convergence rate of various methods compared to eachother. Next, we investigate how this rate is translated inthe architecture mapping and how existing parallelism andlocality affect the energy and speed to convergence. Thisalso allows for evaluation of the proposed architecture andits various characteristics such as memory size, number ofPEs per core, memory per PE, and number of cores withregard to various learning approaches.

4.1. Methodology

Networks, Dataset, and Algorithms: We explore dif-ferent network sizes and learning methods tested on a subsetof the MNIST dataset in order to determine convergence andaccuracy results. All networks use ReLU as the hidden layeractivation. As the networks are trained on only a subset ofthe complete MNIST dataset, the accuracies achieved arelower. However, experimentation with the complete datasetand comparisons with existing results show that the rela-tive rates of convergence and accuracies achieved by thedifferent networks and learning algorithms behave similarlyfor the complete dataset. As the purpose of this study is tocompare different networks and learning methods and not toachieve the best possible accuracy, the difference betweenresults for the complete dataset and the subset are negligible.

Four different sized networks are trained using SGD, CP,MBGD and DFA with batch sizes of 2, 4, 8, 50 and 100 foreach. To demonstrate the effect of increasing network depth,4, 5, and 6 layer networks with hidden layers of size 500×500 are chosen. A 2500-2000-1500-1000-500-10 network isused to represent a network that is both deep and wide andwith varying hidden layer dimensions.

Architecture: Software studies showed no discernibledifference between training with 16bit floating point and32bit floating point, thus we choose to use half-precisionFloating Point Units (FPUs) in the PEs. Each PE has 16KBof local memory [18] and each core has 512 KB of privateSPAD memory. Table 1 shows the energy per operation forthe FPU and energy per access for the local and off-corememory, as well as respective areas of the units. Energyand area values for memories, wires, and look-up tableswere obtained and estimated from [9] and CACTI [28]respectively. The Half-Precision FPU area and energy wereobtained from [29]. All estimates are for implementationin bulk CMOS operating at 1 GHz frequency. These valuesare used to analytically derive time, energy and performanceresults for the proposed architecture.

We consider two arrangements of PEs: 2×16 cores with16×16 PEs each, and 2×4 cores with 4×4 PEs each, result-ing in a total area of 103.2mm2 and 178.9mm2 respectively.

0

10

20

30

40

50Ep

ochs

Accuracyin50Epochs(500-500-500-10)

80% 85% 88% 90% 91% 92% 93%

92%

88% 90% 91%

90% 90%

93%

88%

88%91%

90%

88% 88%

90%

88%

(a)

0

10

20

30

40

50

Epochs

Accuracyin50Epochs(500-500-500-500-10)

80% 85% 88% 90% 91% 92% 93%

93%

90%91%

92%

91%

92%

93%88%

90%

90%

88%

88%

91%

85%

85%

(b)

0

10

20

30

40

50

Epochs

Accuracyin50Epochs(500-500-500-500-500-10)

80% 85% 88% 90% 91% 92% 93% 94%

93%

91%

91%

92%

91%

93%

94%

90%90%

88%

88%88%

90%

88%85%

(c)

0

10

20

30

40

50

Epochs

Accuracyin50Epochs(2500-2000-1500-1000-500-10)

80% 85% 88% 90% 91% 92% 93% 94%

93%92%

92%

93% 93%93% 94% 92% 91%

88%

90%

88%

90% 90%88%

(d)

Figure 5. Epochs for each network to reach different accuracies for each of the training methods applied to four different neural networks. (a) a 500-500-500-10 network trained for 50 epochs, (b) a 500-500-500-500-10 network trained for 50 epochs, (c) a 500-500-500-500-500-10 network trained for 50epochs, (d) a 2500-2000-1500-1000-500-10 network trained for 30 epochs.

We choose to use energy required for convergence to givenaccuracies as the comparison unit, because of the need fora uniform measure between all networks and algorithms.

4.2. Software Experimental Results

Figure 5 shows the validation accuracy achieved for thefour chosen networks. Each epoch constitutes a single passover the entire dataset. The network in Figure 5(a) is smallenough that even SGD and CP require many epochs to reachconvergence, although the accuracy reached is higher thanfor other algorithms. In Figure 5(b), the additional hiddenlayer causes the epochs to convergence for SGD and CP todrop by 60% so that they converge faster and to a higheraccuracy than the minibatched algorithms. In general, SGDand CP are able to achieve the highest accuracy in the fewestepochs of all algorithms as the weights are updated oncefor each sample in an epoch. CP also performs as well orbetter than SGD in all cases, although this is not true for theMBCP. Further for small minibatches, minibatches of sizeeight outperform those of size two and four for all networksas it can support a slightly higher learning rate.

For the same learning method compared across thedifferent networks, larger networks are able to converge tohigher accuracies in fewer epochs. This behavior is espe-cially evident between the network in Figure 5(c) and thelargest network in Figure 5(d). Here, instead of increasingdepth of the network, the size of each layer is increased.However, the larger size of the network means more aggre-gate calculations and weight updates, which will have an

impact on the energy and time performance when mappedto the architecture.

The difference in convergence rate and highest achiev-able accuracy between learning methods also becomes lessevident as the network size increases. Thus as network sizeincreases, MBGD’s performance approaches that of SGD interms of accuracy and is also able to reach this accuracy in acomparable number of epochs. The architectural implicationis that minibatch training can be used for larger networksthat do not fit on local memory without sacrificing accuracy.Note that in the same number of epochs, DFA alwaysachieves a lower accuracy than other learning methods dueto the lower learning rate.

4.3. Architecture Experimental Results

Figures 6-8 demonstrate the energy required for threenetworks as well as the breakdown into FPU energy andmemory access energy. The energy of broadcasts was foundto be negligible and is not included here. Network 1 inFigure 6 is small and fits completely on the local corememories for all configurations. For the same 90% accu-racy, SGD requires 70% of the energy as MBGD for largeminibatches while CP requires 30%. Examination of energybreakdowns shows that for minibatch algorithms, energyusage is dominated by the FPU while for SGD memoryaccess energy is 1.5 times higher than FPU energy. Theenergy cost for CP is split evenly between FPU and memoryaccesses. This is due to the fact that both minibatched

0

1

2

3

4

5

6TO

TAL

FPU

PEM

EM

TOTA

LFPU

PEM

EM

TOTA

LFPU

PEM

EM

TOTA

LFPU

PEM

EM

TOTA

LFPU

PEM

EM

TOTA

LFPU

PEM

EM

TOTA

LFPU

PEM

EM

TOTA

LFPU

PEM

EM

TOTA

LFPU

PEM

EM

TOTA

LFPU

PEM

EM

TOTA

LFPU

PEM

EM

TOTA

LFPU

PEM

EM

TOTA

LFPU

PEM

EM

TOTA

LFPU

PEM

EM

TOTA

LFPU

PEM

EM

SGD BGD2 BGD4 BGD8 BGD50 BGD100 CP CP2 CP4 CP8 DFA2 DFA4 DFA8 DFA50 DFA100

Energy(J)

EnergyBreakdown(500-500-500-10)4x4PEs/core,2x16Cores,16KBMemory/PE

80% 85% 88% 90% 91% 92% 93%

Figure 6. Energy required to achieve accuracy for various learning methods on a small 500-500-500-10 network that fits on 2× 16 cores with 4× 4 PEseach.

0

5

10

15

20

25

30

TOTA

LFPU

PEM

EMOFFM

EM

TOTA

LFPU

PEM

EMOFFM

EM

TOTA

LFPU

PEM

EMOFFM

EM

TOTA

LFPU

PEM

EMOFFM

EM

TOTA

LFPU

PEM

EMOFFM

EM

TOTA

LFPU

PEM

EMOFFM

EM

TOTA

LFPU

PEM

EMOFFM

EM

TOTA

LFPU

PEM

EMOFFM

EM

TOTA

LFPU

PEM

EMOFFM

EM

TOTA

LFPU

PEM

EMOFFM

EM

TOTA

LFPU

PEM

EMOFFM

EM

TOTA

LFPU

PEM

EMOFFM

EM

TOTA

LFPU

PEM

EMOFFM

EM

TOTA

LFPU

PEM

EMOFFM

EM

TOTA

LFPU

PEM

EMOFFM

EM

SGD BGD2 BGD4 BGD8 BGD50 BGD100 CP CP2 CP4 CP8 DFA2 DFA4 DFA8 DFA50 DFA100

Energy(J)

EnergyBreakdown(2500-2000-1500-1000-500-10)4x4PEs/core,2x16Cores,16KBMemory/PE

80% 85% 88% 90% 91% 92% 93% 94%

35.4566.55 38.38

Figure 7. Energy required to achieve accuracy for various learning methods on a large 2500-2000-1500-1000-500-10 network that does not fit in local PEmemory. High communication cost is required for bringing in weights from off-core memory for SGD and CP; these two algorithms would not be usedin practice but energy results are presented here for completeness

0

5

10

15

20

25

30

35

TOTA

LFPU

PEM

EM

TOTA

LFPU

PEM

EM

TOTA

LFPU

PEM

EM

TOTA

LFPU

PEM

EM

TOTA

LFPU

PEM

EM

TOTA

LFPU

PEM

EM

TOTA

LFPU

PEM

EM

TOTA

LFPU

PEM

EM

TOTA

LFPU

PEM

EM

TOTA

LFPU

PEM

EM

TOTA

LFPU

PEM

EM

TOTA

LFPU

PEM

EM

TOTA

LFPU

PEM

EM

TOTA

LFPU

PEM

EM

TOTA

LFPU

PEM

EM

SGD BGD2 BGD4 BGD8 BGD50 BGD100 CP CP2 CP4 CP8 DFA2 DFA4 DFA8 DFA50 DFA100

Energy(J)

EnergyBreakdown(2500-2000-1500-1000-500-10)16x16PEs/core,2x4Cores,16KBMemory/PE

80% 85% 88% 90% 91% 92% 93% 94%

38.85

Figure 8. Energy required to achieve accuracy for various learning methods on a large 2500-2000-1500-1000-500-10 network that fits on 2× 4 cores with16× 16 PEs each.

and CP algorithms reduce the number of weight accessesrequired each epoch compared to SGD.

Network 2 in Figure 7 is representative of a large net-work that does not fit in local core memories. Althoughthe minibatch algorithms require more epochs to convergeto the same accuracy as SGD and CP, their total energyconsumption is lower due to the smaller number of weightaccesses. For networks that do not fit on the core, SGDand CP must access weights from off-core, substantiallyincreasing energy usage. As discussed previously, SGD andCP also require higher bandwidth to access the weights ifthey do not fit locally. These results suggest to performtraining using minibatch algorithms for networks that donot fit.

Network 3 in Figure 8 is the same as network 2 buttrained on an architecture with more PEs such that the

network fits locally. Unlike the smaller network 1, MBGDwith batch size of 50 now performs better than SGD in termsof energy cost, although still not as well as CP. Comparisonof FPU energy only shows that batched algorithms have ahigher energy cost due to the greater number of epochs, butthis is balanced out by the lower memory access energy.As discussed in the previous section, MBGD’s performancein terms of epochs increases with network size. Here, thedifference in energy consumption for FPU operation be-tween minibatched and non-minibatched methods is smallenough that memory access energy becomes the dominantfactor differentiating the two methods. However, the fasterconvergence and higher accuracy of CP causes it to performbetter than MBGD even for large networks.

These results suggest that for large networks, MBGD canperform better in terms of energy than SGD even when there

0

5

10

15

N1

N2

N3

N1

N2

N3

N1

N2

N3

N1

N2

N3

N1

N2

N3

N1

N2

N3

N1

N2

N3

N1

N2

N3

N1

N2

N3

N1

N2

N3

N1

N2

N3

N1

N2

N3

N1

N2

N3

N1

N2

N3

N1

N2

N3

SGD BGD2 BGD4 BGD8 BGD50 BGD100 CP CP2 CP4 CP8 DFA2 DFA4 DFA8 DFA50 DFA100

Time(s)

TimetoAccuracyforDifferentNetworksandArchitectures

80% 85% 88% 90% 91% 92% 93% 94%

N1 - (500-500-500-10),4x4PEs/core,2x16Cores,16KBMemory/PEN2- (2500-2000-1500-1000-500-10),4x4PEs/core,2x16Cores,16KBMemory/PEN3- (2500-2000-1500-1000-500-10),16x16PEs/core,2x4Cores,16KBMemory/PE

17.1 17.7

Figure 9. Time required to achieve accuracy for various learning methods and networks.

0

5

10

15

20

25

SGD

BGD2

BGD4

BGD8

BGD5

0

BGD1

00 CP CP2

CP4

CP8

DFA2

DFA4

DFA8

DFA5

0

DFA1

00

GFLO

PS/m

m2

GFLOPS/mm2 forDifferentNetworksandArchitectures

(500-500-500-10) 4x4PEs/core,2x16Cores,16KBMemory/PE

(2500-2000-1500-1000-500-10) 4x4PEs/core, 2x16Cores, 16KBMemory/PE

(2500-2000-1500-1000-500-10) 16x16PEs/core, 2x4Cores, 16KBMemory/PE

Figure 10. GFLOPS/mm2 for various learning methods and networks.

is enough local memory to store the entire network. Further,CP consistently outperforms all other training methods. Theenergy to convergence results for DFA indicate that althoughdirect propagation of the error from the last layer leads tofewer FPU operations, thus less energy required per epoch,the slower convergence rate causes the algorithm to consumemore energy to reach the same accuracy.

Figure 9 illustrates the time required for each networkto reach given accuracies on the architecture operating at1GHz. As in the energy performance comparisons, CP per-forms better than SGD and requires less time to convergeto the same accuracy. While software studies showed thatthe epochs required to converge to the same accuracy wassimilar for the two algorithms, the pipelining and weightlocality utilized by CP allows it to achieve better perfor-mance on the architecture. When the network does notfit in local memory for CP and SGD, the need to fetchweights from off-core memory can significantly increase thetime to convergence. For Network 2 which does not fit onthe core, the utilization drops from 99% to 75% for CPand from 81% to 47% for SGD. From software studies,the epochs to convergence for minibatched algorithms issimilar to batched for large network sizes. Architecturally,however, minibatched algorithms can converge faster thannon-minibatched algorithms if the network does not fit evenwhen it requires more epochs.

Figure 10 and Table 2 show the performance inGFLOPS/W and GFLOPS/mm2 for SGD, CP and MBGDapplied to networks of different sizes respectively. CP con-sistently outperforms SGD in all cases. For networks thatdo not fit on core, MBGD demonstrates the highest per-formance, followed by CP and SGD, while for networks

TABLE 2. GFLOPS/W FOR (A) A SMALL NETWORK THAT FITS ON2× 16 CORES OF 4× 4 PES; (B) A LARGE NEURAL NETWORK THAT

DOES NOT FIT ON THE SAME ARCHITECTURE; (C) THE SAME NEURALNETWORK THAT FITS ON 2× 4 CORES OF 16× 16 PES.

Network Dimensions BP Method GFLOPS/WSGD 177

500-500-500-10 CP 204MBGD 195SGD 98

2500-2000-1500-1000-500-10 CP 127MBGD 187SGD 185

2500-2000-1500-1000-500-10 CP 211MBGD 195

that do fit, CP outperforms MBGD. Although performanceof MBGD can be greater than CP/SGD due to the reducedenergy cost of accessing fewer weights, it is not as accu-rate as either, especially for small networks (90% vs. 92%accuracy for small networks and 93% vs 94% accuracy forlarge networks) and also takes longer to converge. Thus,there is a tradeoff between performance and accuracy/timeto convergence that must be considered when determiningwhich training method to use. When comparing betweennetworks of different sizes, it can be seen that for the samearchitecture size, while the larger network reaches higheraccuracy, the time to convergence to a lower accuracy issmaller for smaller networks. The flexibility of the archi-tecture in supporting both batched and non-batched trainingalgorithms provides the user with freedom to determine thelearning method to use based on the time and accuracyconstraints of their network application.

The best overall performance of the architecture occursfor a network of size 2500-2000-1500-1000-500-10 mappedto a 2 × 4 array of cores with 16 × 16 PEs, with trainingdone using CP. From an algorithmic perspective, the size anddepth of the network leads to higher accuracy and fewerepochs to convergence while the greater number of PEsboth increases the effective utilization of the architectureand eliminates costly accesses to external memory.

CATERPILLAR achieves 98% effective utilization ofthe FPUs and performance of 211 GFLOPS/W for networksthat fit on cores using CP. Further, when the same networkdoes not fit on the cores, using minibatched algorithmsMBGD can achieve 187 GFLOPS/W at 94% utilization.

5. Related Work

Several FPGA implementation efforts have been per-formed to accelerate the training of neural networks[30] [31] [32]. Maximum performance of up to 10 G Mul-

tiply Accumulates (GMACs) is achieved in [32]. However,these works are limited in scope as they focus on eitherretraining [30] or shallow 2-layer neural networks [31].Furthermore, there is no support for performing differentlearning algorithms as for Caterpillar architecture.

Gupta et al. have shown that 16-bit fixed precision canachieve the same accuracy as floating-point if stochasticrounding is used [11]. Our preliminary studies suggests thatthe convergence rate and accuracy decreases for networksdeeper than two layers with stochastic rounding. Furtherstudy is required to completely characterize the performanceof stochastic rounding compared to floating point.

The work in [33] showed that CP can outperformMBGD’s speed and accuracy for CNNs. In this paper weapply and evaluate CP for MLPs.

6. Conclusion

Our investigation for training MLPs demonstrates thatfor various networks sizes, the target architecture shouldsupport both GEMV (for pipelined backpropagation),GEMM (for minibatched algorithms), and hierarchical col-lective communications. For networks that do not fit onchip, minibatched algorithms have comparable performanceto pipelined backpropagation, however for networks that fit,pipelined backpropagation consistently performs the best.Fast convergence on the algorithmic side in tandem withlayer parallelization and weight locality from an architec-tural perspective allows Pipelined Continuous Propagationto outperform all other training methods in terms of energyand time to convergence, distinguishing it as a promisingtraining method for use with specialized deep learning ar-chitectures.

Acknowledgments

We thank Hadi Esmaeilzadeh, Michael James, DavidKoeplinger, Ilya Sharapov, Vijay Korthikanti, and SaraO’Connell for their feedback on the manuscript. Thisresearch was partially sponsored by NSF grants CCF-1563113. Any opinions, findings and conclusions or rec-ommendations expressed in this material are those of theauthors and do not necessarily reflect the views of theNational Science Foundation (NSF).

References

[1] A. Krizhevsky et al., “Imagenet classification with deep convolutionalneural networks,” in NIPS, 2012.

[2] T. Mikolov et al., “Recurrent neural network based language model.”in Interspeech, vol. 2, 2010.

[3] B. Widrow et al., “30 years of adaptive neural networks: perceptron,madaline, and backpropagation,” Proceedings of the IEEE, vol. 78,no. 9, 1990.

[4] V. Sze et al., “Efficient processing of deep neural networks: A tutorialand survey,” arXiv preprint arXiv:1703.09039, 2017.

[5] N. Jouppi et al., “In-datacenter performance analysis of a tensorprocessing unit,” in ISCA44. IEEE Press, 2017.

[6] J. Dean et al., “Large scale distributed deep networks,” in NIPS, 2012.

[7] D. Zhang et al., “Neural networks and systolic array design,” Seriesin Machine Perception and Artificial Intelligence, vol. 49, 2002.

[8] D. E. Rumelhart et al., “Learning internal representations by errorpropagation,” DTIC Document, Tech. Rep., 1985.

[9] A. Pedram et al., “Dark memory and accelerator-rich system opti-mization in the dark silicon era,” IEEE Design & Test, vol. 34, no. 2,pp. 39–50, 2017.

[10] Q. Liao et al., “How important is weight symmetry in backpropaga-tion?” arXiv preprint arXiv:1510.05067, 2015.

[11] S. Gupta et al., “Deep learning with limited numerical precision.” inICML, 2015.

[12] B. Recht et al., “Hogwild: A lock-free approach to parallelizingstochastic gradient descent,” in NIPS, 2011.

[13] S. Han et al., “EIE: efficient inference engine on compressed deepneural network,” in ISCA43. IEEE Press, 2016.

[14] D. C. Ciresan et al., “Deep big simple neural nets excel on handwrit-ten digit recognition,” Neural Computation, vol. 22, 2010.

[15] T. P. Lillicrap et al., “Random feedback weights support learning indeep neural networks,” arXiv preprint arXiv:1411.0247, 2014.

[16] A. Nøkland, “Direct feedback alignment provides learning in deepneural networks,” in NIPS, 2016.

[17] R. G. Girones et al., “Systolic implementation of a pipelined on-linebackpropagation,” in MicroNeuro. IEEE, 1999, pp. 387–394.

[18] A. Pedram et al., “A high-performance, low-power linear algebracore,” in ASAP. IEEE, 2011, pp. 35–42.

[19] M. Li et al., “Efficient mini-batch training for stochastic optimiza-tion,” in 20th ACM SIGKDD. ACM, 2014.

[20] S. Zhang et al., “Asynchronous stochastic gradient descent for DNNtraining,” in IEEE ICASSP, 2013, pp. 6660–6663.

[21] K. Simonyan et al., “Very deep convolutional networks for large-scaleimage recognition,” arXiv preprint arXiv:1409.1556, 2014.

[22] A. Gruslys et al., “Memory-efficient backpropagation through time,”CoRR, 2016. [Online]. Available: http://arxiv.org/abs/1606.03401

[23] A. Pedram et al., “Codesign tradeoffs for high-performance, low-power linear algebra architectures,” IEEE Transactions on Computers,Special Issue on Power efficient computing, vol. 61, no. 12, pp. 1724–1736, 2012.

[24] E. Chan et al., “Collective communication: theory, practice, and ex-perience,” Concurrency and Computation: Practice and Experience,vol. 19, no. 13, 2007.

[25] J. Cao et al., “High-performance hardware for function generation,”in IEEE ARITH. IEEE, 1997, pp. 184–188.

[26] J.-A. Pineiro et al., “Faithful powering computation using table look-up and a fused accumulation tree,” in IEEE ARITH, 2001, pp. 40–47.

[27] A. Pedram et al., “Floating point architecture extensions for optimizedmatrix factorization,” in IEEE ARITH. IEEE, 2013, pp. 49–58.

[28] N. Muralimanohar et al., “Architecting efficient interconnects forlarge caches with CACTI 6.0,” IEEE Micro, vol. 28, 2008.

[29] S. Galal et al., “FPU generator for design space exploration,” in IEEEARITH, 2013, pp. 25–34.

[30] J. Park et al., “FPGA based implementation of deep neural networksusing on-chip memory only,” in IEEE ICASSP, 2016, pp. 1011–1015.

[31] V. T. Huynh, “Design of artificial neural network architecture forhandwritten digit recognition on FPGA,” 2017.

[32] H. P. Graf et al., “A massively parallel digital learning processor,” inNIPS, 2009, pp. 529–536.

[33] M. James et al., “Continuous propagation: layer parallelism fortraining deep networks,” 2017.