1

Design Enhancements for In-Cache ComputationsPatrick A. La Fratta Peter M. Kogge

University of Notre Dame

plafratt@nd.edu kogge@cse.nd.edu

Abstract—New programming models and increased off-chip bandwidth demands of multicore processors havechanged the nature of the memory wall problem, and in-novative hardware and software solutions are emerging tocombat the resulting latency and power issues. Emergingprocessor designs include organizations of on-chip resourcesthat diverge from the classical multicore model. One ex-ample of a such a system increases core counts along a newdesign dimension by embedding simple cores into the cachehierachy of more complex, conventional cores. Developmentof new execution models based on In-Cache Computations(ICCs) for execution on such a system is underway. Thiswork presents several enhancements to the ICC design, in-cluding an improved style of execution, co-processor archi-tecture, and ICC code generation techniques. Simulationresults highlight promising opportunities for significant ad-vantages over conventional architectures, including up to61% average improvement in L1 cache miss rate for a con-ventional host processor. We project an average potentialspeedup of 1.32 due to increased L1 hit rate allowed byICCs.

I. Introduction

Many issues with single-core processors such as designcomplexity and power constraints have resulted in theemergence of multicore processors in high performancecomputing [4]. These chip multiprocessor (CMP) designsfrequently replicate a core that is simpler and runs at alower clock rate than the cores found on single-chip pro-cessors. At the same time, the increase in computationaland storage resources on the chip due to improvementsin fabrication technology has lead to advanced heteroge-neous CMP designs, such as STI’s Cell [16]. System-on-a-chip (SoC) designs can include three levels of cache andeven main memory embedded on the same die as the cores[18], [12]. Hence, the increase in core count is occur-ring across multiple design dimensions to accommodatea wide variety of workloads. Along with these innova-tive designs, microarchitectural features and programmingtechniques have emerged for efficiently utilizing these het-erogeneous computational resources while facing worseningperformance and power problems in the memory hierarchy.

One example of such an innovative chip-multiprocessoris based on the integration of computational logic nextto the on-chip caches of conventional out-of-order (OoO)cores. This type of system extends previous work withprocessing-in-memory (PIM) [2], [22], [13] and IRAM [8]architectures to account for architectural and technologicalimprovements in the multicore processing era. A recentlyproposed execution model, named In-Cache Computations(ICCs), seeks to improve performance and lower powerconsumption by utilizing the simple in-cache cores [14].We present improvements to the execution model, architec-tural support, and translation tools for running ICCs. Us-

Fig. 1. A conventional core backed by DCPs in upper-level caches.

The upper-level caches are assumed to be unified. This work studies

the effects of DCPs in the L2 cache only.

ing execution traces from applications in the SPEC bench-mark suite and scientific applications at Sandia Labs, ourstudies illustrate noticeable performance potential for thedesign across a variety of workloads.

The paper is organized as follows. The next section offersa presentation of the ICC execution model, architecturaldesign extensions, and code translation tools. The thirdsection presents results of application analyses showing theICC content in applications, followed by simulation results.The fourth section discusses related work, and the finalsection gives conclusions for this work and implications forfuture endeavors.

II. ICC Design

ICCs are a lightweight unit of computation formed tobe spawned from and execute in parallel with one or moreheavyweight threads. The processing architecture assumedfor executing ICCs consists of one or more conventionalcores (called the hosts) supporting out-of-order executionfor executing the heavyweight threads along with a set ofsimpler cores called distributed co-processors (DCPs). TheDCPs are embedded next to the unified upper-level caches(L2, L3, etc.), as shown in figure 1, and possibly in mainmemory. By offloading computations to the DCPs, ICCsincrease both parallelism and the L1 hit rate of the host.A compiler or binary translator assists in forming ICCs –which execute on the DCPs – with consideration to bothparallelism and data locality. This section presents thedefinition of the ICC execution model, architectural designof the DCP, and ICC code translation tools used in thisanalysis.

A. Execution Model

The ICC execution model consists of three phases: theprefetch phase, the setup phase, and the execution phase.

2

Fig. 2. Commands for a DCP-enhanced cache.

In the prefetch phase, the host issues a special directive,shown in figure 2b as the Prefetch Directive, out to thememory hierarchy for increasing the probability that bothinstruction and data cache lines are positioned in the uni-fied L2 cache in a manner that will enable efficient ICC ex-ecution. This directive seeks to fulfill two conditions. Thefirst is that copies of the instruction cache lines containingthe code for an ICC are located in either the memory orthe instruction cache line overflow buffer (IOB) next to theDCP on which the ICC will execute. The second, for thepurpose of maintaining consistency with the host is thatthe L1 cache does not contain lines that ICCs will modify.In the ideal case, the L1 requires no invalidation duringthe prefetch phase.1

The directive is encoded as a memory access with supple-mentary information, including a flag that identifies it asa ICC prefetch directive, the addresses of the needed datalines and the instruction lines, and optionally (as an opti-mization) the correlation between the two. To avoid con-sistency issues, data lines are left in their home set while in-struction lines are broadcast to DCPs if not present where

1 The frequent occurrence of L1 cache line invalidation during theprefetch phase is an indication of poor ICC formation. In this work,we take necessary steps to avoid this case, and leave the implemen-tation of the necessary exception handler as future work.

needed. The data lines should not be evicted before theend of the execution phase, so the L2 cache controller by-passes accesses to any lines that would evict them (causedby memory accesses from the host).2 The directive is is-sued prior to the time when the host reaches the basic blockcontaining the ICC, similar to a prefetch. Prefetch mecha-nisms given in prior work will likely be useful in optimizingimplementations of the prefetch directive [23].

The ISA of the DCP will likely differ from that of thehost. In the case that run-time code translation at theDCP is used, this would occur during the prefetch phase.

During the setup phase, the host constructs the “ICC ac-tivator”, shown in figure 2c, (similar to the continuationdescribed in previous work [2], [22]) which it will injectinto the memory hierarchy for initiating the execution ofone or more ICCs. The ICC activator contains the regis-ter values, data addresses, starting PC of each ICC, andcontrol vectors needed to initiate and carry out ICC exe-cution. The control vectors are bit vectors that indicatewhich instructions following the starting IP the DCP willexecute.

When constructing the ICC, the host accumulates theregister values along with the ICC data and instructionaddresses. These values are accumulated as they are pro-duced by the main thread. The instructions for inser-tion into the ICCs are removed from the host’s instructionstream.

Once the activator construction is complete, the execu-tion phase begins. In the execution phase, the activatoris injected into the memory hierarchy. The L2 cache con-troller executes the activator and routes the register valuesand addresses to the appropriate DCPs, based on the dataaddresses of each ICC. The ICCs communicate across a businternal to the cache via special instructions which enforcea static ordering, similar to techniques used in previousVLIW architectures [21]. The instructions in the ICCsmay have dependences at the host, and the ICCs will re-turn register values as necessary.

Consider an example of ICC execution, as illustrated infigure 3. Instruction lines X, Y , and Z contain the ICC’sinstructions. These instructions include a load and a storeto line L, which maps to a physical block next to DCP 0in L2, and a reference to data on line M which maps toa remote physical block. Prior to the prefetch phase, L1contains a partial copy of line L.

When the host reaches the prefetch instruction, it issuesthe prefetch directive to the cache hierarchy. The L1 con-troller invalidates its partial copy of line L, and the copy ofL is then fetched from L3 (we assume multi-level inclusionis not maintained here to serve the purpose of the exam-ple). Data line M is then moved from L3 to remote DCPn. Instruction line Z is fetched from DCP n into DCP 0’sIOB. Line Y is fetched from L3 into IOB 0, and line X isfetched from main memory into IOB 0.

During the setup phase, the host accumulates the reg-ister values needed for the execution of the ICC. When

2 In future work, we will consider the effectiveness of this techniqueand consider alternatives if appropriate.

3

Fig. 3. ICC Example.

the host issues a memory access to a line in L3 that con-flicts with L, the L2 controller bypasses the access to avoidthe eviction of line L. When the ICC’s register values areready, the host issues the ICC activator to the hierarchy.

The execution phase begins, and the activator reachesDCP 0. As specified by the control vectors, the DCP be-gins executing instructions from line W in its local mem-ory, and then lines X, Y , and Z in its IOB. Instructionsfrom line Z contain loads from line L – in local memory –and line M – located at DCP n. A request is sent on theintra-cache bus to retrieve the remote data from line M .A value is written back to line L using the store on line Z.Upon completion, the ICC returns a set of register valuesback to the host.

B. DCP Architecture

The DCP architecture is responsible for ICC execution.The DCP has its roots in previous work in the designof cores for processing-in-memory systems [22], [2], espe-cially those using dense memory caches implemented with

Fig. 4. DCP Architecture.

eDRAMs [10]. Figure 4 shows the organization of the com-ponents at the DCP node. Each DCP in the L2 has a num-ber of cache sets associated with it. The design must main-tain the functionality of a conventional L2 cache, while atthe same time providing an interface to the extended pro-cessing capabilities of the DCPs. Hence, the L2 operatesin a dual-mode fashion which is governed by the L2 cachecontroller (labeled Master BIU/Controller in figure 4), andinstructed by the commands issued from the host.

During the prefetch phase, the controller ensures thatthe data lines are present in L2. The controller must thenensure that the instruction lines containing the ICC codeare also present at the DCPs that will modify the datalines. In the case that the sets at the DCP do not includethe necessary instruction lines, the controller must dis-tribute them to a special buffer (labeled I$ Line OverflowBuffer in figure 4) near the DCP. The slave BIU recordswhich instruction lines are located in the IOB, as this in-formation is necessary during the execution phase.

During the execution phase, the controller distributesthe control vectors that specify which instructions will beexecuted on each DCP. The cache controller and compilercan uniquely identify an ICC by the addresses of the datait uses. The DCP uses the control vectors to fetch andparse instruction lines, which direct the flow of executionand data from the cache sets. ICC communication occursover a special communication bus local to the L2 cache(labeled Intra-Cache Bus in figure 4). Sequential consis-tency is enforced through this bus, as guided by the schemedescribed in the next section.

C. Code Translation

Computational partitioning towards increased paral-lelism and reduced data communication is a key challengeof ICCs. In this study, we analyzed the potential of data-centric ICCs through simulation. The key concept is ICC-

4

compatible data cache lines (ICLs). An ICL is defined asa line that, if evicted from the L1 cache of a conventionalprocessor, was touched by memory accesses in only onebasic block of the program. This type of line has the prop-erties that both lend itself well to parallelism and local-ity that can be exploited by ICCs. Since the instructionsdo not cross basic blocks, we avoid the complications ofcontrol speculation. Implementing misspeculation recov-ery would both complicate the DCP design and requireincreased traffic in the cache hierarchy, resulting in over-heads in power and performance. Furthermore, such lineswill tend to hurt performance on a system using conven-tional caching by using a physical block for a cache linewith limited temporal locality.

Through trace analysis, we determine which lines areICC-compatible. This information is then given to a codetranslator which it uses for computational partitioning. Infuture work, we plan to consider advanced compiler tech-niques for prefetching along with statistical analysis at run-time to avoid trace analysis.

The translation scheme used is given in algorithm 1.Prior to entering the algorithm, a set of memory accesseshas been selected for each ICC using trace analysis. Thepurpose of the algorithm is two-fold. The first objectiveis to partition the dependence graph of the basic blockinto sets of instructions for inclusion in each ICC. Afterlabeling each ICC memory access, the algorithm removesbranch critical ICCs (BCIs). A BCI is defined as an ICCwhose memory accesses include a load with a branch as adescendant. One objective of ICC execution is to increasesystem-wide parallelism, and this implies increasing hostutilization. The partitioning scheme assigns all branchesto the host, and we want to avoid causing processor stallsor speculation due to missed branch critical loads.3 Oncethe BCIs are removed, the current scheme simply givespriority to the first instructions in program order, using adepth-first search from each ICC memory access and la-beling every unlabeled instruction that is reached.

The second objective is to determine the register valuesfor each ICC that must be included in the ICC activatorlaunched from the host. In the current design, all modi-fied registers are returned to the host. Future work willinclude load balancing and live register analysis to furthereliminate traffic.

III. Evaluation

This section presents the simulation results and analy-ses illustrating the potential performance increase offeredby ICCs. The first section gives the simulation frameworkand overview of applications. The second section offersthe results of ICC content analysis, to quantifying the fre-quency of access patterns conducive to ICC formation inthe applications. The third section presents cache simula-

3 Future work will consider the optimization of the inclusion ofcertain types of BCIs. As an example, consider a BCI leading toconditional basic blocks that each return to a common PC. However,in the applications studied here, the results in the next section showthat BCIs are generally very uncommon.

Algorithm 1: ICC Computational PartitioningInput: G, S, L/* G: dependence graph of a basic block.Each instruction is represented as a vertex u in G.v is a child of u if and only if v has an inputdependence from u. v is an ancestor of u if andonly if ∃ a path from v to u.S: A set of ICC IDs of ICCs in G.L: A set of |S| sets of vertices in G.Each set l∈L corresponds to an ICC ID s∈S.l contains the vertices corresponding to memoryaccesses in G assigned to the ICC with ID s. */index=0.1foreach l∈L do2

foreach g∈G do3if g is in l then4

Label g with index.5Depth-first search on g, labelling all6unlabeled descendants of g with index.7

end8

end9index=index+1.10

end11Remove labels of branch critical ICCs.12curCommIndex=0.13foreach g∈G do14

if g has label x then15foreach Incoming edge (i,g) do16

if i is an instruction outside the basic block then17Mark i as an external ICC input.18

end19

end20foreach Outgoing edge (g,o) do21

Mark o as an external ICC output.22if o has label y and y 6=x then23

Schedule bus event curCommIndex from g to o.24curCommIndex=curCommIndex+1.25

end26

end27

end28

end29

tion results and estimates of how ICCs may affect cacheperformance of the host.

A. Methodology

The simulation framework consists of a two-stage sim-ulation process based on the Structural Simulator Toolkit(SST) [20], which partitions simulation infrastructure be-tween frontend and backend components. In the first stage,the frontend reads in the instruction trace and runs a cachesimulation to determine which data line evictions are ICCcompatible. In the second stage, the trace is read a secondtime along with the record of memory accesses that are toICLs. The computational partitioning takes place in thisstage, along with the recording of the numbers of BCIs,which are given in the next section. The second stage alsoruns a second set of cache simulations to compare the cacheperformance of various cache configurations to produce theresults given in the Cache Simulation section. In the finalset of experiments, we configure the backend of SST forSimpleScalar to produce timed simulations.

The first set of traces were taken from the SPEC-Int andSPEC-FP suites. Additionally, the Sandia-Int suite is a setof traces of seven commonly used integer-intensive appli-cations run at Sandia National Labs. Sandia-FP is a setof eighteen traces from eight of Sandia’s floating point-intensive scientific applications run with varied inputs.The applications cover a wide range, including molecular

5

dynamics (LAMMPS), shock mechanics of solids (CTH),radiation transport (ITS), and circuit simulation (Xyce).The code analysis and cache simulations overall used 47traces of 100 million instructions each generated from theapplications. The SimpleScalar simulations use 20 millioninstructions from the same 47 traces. [17] includes moredetails of these applications.

B. ICC Content Analysis

Table I presents the first set of analysis results using aconventional 16KB direct-mapped L1 cache with a 64B linesize. The second column of table I shows the number ofL1 evictions that occur and the percentage of these evic-tions that are ICLs. The next column, labeled % ICC Con-tent, shows the percentage of instructions from the originaltrace offloaded from the host to the DCPs. The memoryaccesses to lines that are branch critical are used as in-put L to algorithm 1. The algorithm partitions each basicblock’s instructions among the ICCs and the host. The to-tal number of instructions inserted into ICCs throughoutthe trace divided by the length of the trace (100,000,000)gives the % ICC Content. The right-most column givesthe percentage of ICLs that are branch critical.

The percentage of ICLs is significant, with a minimumof 5.3% (sppm.2B), a maximum of 93% (sp.4B), and ageometric mean of 43%. A high percentage of ICLs and ahigh number of evictions means that there is large oppor-tunity for both an increase in L1 performance at the hostand increased parallelism through the use of the DCPs.Although the amount of ICC content is low across all thetraces (1.5% on average), improved partitioning algorithmswill likely improve these numbers, resulting in increasedparallelism through increased DCP utilization. Several av-enues exist for possible improvements to the partitioningalgorithm. One option is to increase the average number ofinstructions between branches through the use of compilertechniques (loop unrolling), or speculatively at run-timewith branch prediction. Another option is to adjust theL1 cache parameters such that a cache line is more likelyto be touched by instructions in only one basic block. Theresults in the right-most column are quite encouraging, re-vealing that branch-criticality is a rare property amongICCs. We found the average percentage of ICCs that arebranch-critical across all traces to be much less than 1%.

Consider the first trace in the table, 164.gzip.1.b. Thereare 1,661,259 L1 evictions, 70% (1,162,881) of which areICLs. Intuitively, the system can achieve performance im-provement by leaving these lines in L2 and working onthem with the DCPs. Although it is likely that the ICCswill still result in traffic back to the host, the host canlikely hide this latency since virtually none of these ac-cesses are branch critical (as given by the right-most col-umn). Furthermore, it is likely that this traffic will beless than that required by the fetch-modify-flush approachof conventional caches, and the accesses to specific wordsacross multiple lines can be bundled together after the ex-ecution phase of a group of ICCs to mitigate latency costs.Both of these features show promise for improvements in

TABLE I

ICC Content Analysis Results

Application # of L1 % % ICC %Evictions ICLs Content BCIsSPEC-Int

164.gzip.1.1b 1661259 70% 2% 0%164.gzip.2.1b 880566 47% 1% 0%164.gzip.3.1b 1605856 57% 1% 0%164.gzip.4.1b 2139275 74% 2% 0%164.gzip.5.1b 1465704 60% 1% 0%175.vpr.1.1b 2964109 23% 1% 0%176.gcc.1.1b 97623 14% 0% 6%176.gcc.3.1b 2272139 37% 2% 8%

SPEC-FP168.wupwise.1.1b 506879 12% 0% 0%171.swim.1.1b 8139834 55% 5% 0%172.mgrid.1.1b 3579451 49% 3% 0%173.applu.1.1b 3729154 68% 17% 0%177.mesa.1.1b 1175265 21% 1% 1%178.galgel.1.1b 10924676 69% 8% 0%179.art.1.1b 8515087 82% 8% 0%179.art.2.1b 8499246 82% 8% 0%183.equake.1.1b 2368176 45% 2% 0%187.facerec.1.1b 4106654 77% 5% 0%189.lucas.1.1b 3711088 54% 6% 0%191.fma3d.1.1b 2203690 36% 2% 2%200.sixtrack.1.1b 2542373 44% 1% 0%301.apsi.1.1b 4131489 78% 4% 0%

Sandia-Intblast.4B 2107204 25% 1% 0%chaco.4B 11807395 78% 10% 0%dfs.4B 3181081 65% 2% 16%iso.4B 7817602 47% 4% 0%kmetis.4B 2534815 38% 1% 1%sp.4B 4557051 93% 6% 0%zchaff.4B 2195427 35% 1% 1%

Sandia-FPalegra.4B.WES04 4563413 61% 8% 7%alegra.4B.hemipen 3019218 54% 6% 2%cth.4B.2gas 4316113 74% 4% 0%cth.4B.amr-2d-cust4 458864 40% 0% 0%cth.4B.efp 724509 38% 0% 0%cth.4B.sc 1411917 40% 1% 0%cube3.4B.crs 3291162 19% 1% 0%cube3.4B.vbr 1884920 21% 1% 0%its.4B.cad.brems 497149 41% 0% 12%its.4B.cg.brems 348060 46% 0% 25%lmp.4B.flow.langevin 2074179 67% 2% 0%lmp.4B.lj.fix 1491885 56% 2% 0%lmp.4B.poly.chain.a 1310525 55% 2% 0%mpsalsa.4B.airport3d 3 3713702 40% 10% 0%mpsalsa.4B.thermal... 1466939 12% 0% 5%sppm.2B 1445981 5% 0% 0%xyce.4B.invertgen75 3438999 49% 3% 2%xyce.4B.invertgen v3 75 4120789 53% 3% 2%

both execution time and power over conventional cachemechanisms.

C. Cache Simulation

Figures 5–8 present changes in L1 miss rates with re-spect to a conventional 64KB cache (henceforth, the base-line cache). For each application, four sets of three barsare given, where each set represents a different cache con-figuration and each bar within a set represents a differentcache size. The four configurations are a conventional LRUcache, a column-associative cache [1], an ICC-enhancedcache hierarchy, and an ICC-enhanced cache hierarchywith column-associative L1. The reason for the column-associative cache option will be discussed later. We willfirst explain the configurations that produced these resultsand the reasoning behind them, followed by an analyticalsummary and an explanation of their significance.

6

Fig. 5. SPEC-Int: L1 misses expressed as normalized fraction of misses to conventional 64KB cache.

Each bar expresses a fraction m/n, where m is the number of L1 misses observed for a particular cache policy and configuration and n is

the number of L1 misses observed using a conventional 64KB cache. Four cache configurations were studied: a conventional LRU cache,

a column-associative cache as given by Agarwal et al. [1], an ICC-enhanced cache hierarchy, and an ICC-enhanced hierarchy run with a

column-associative L1. Each configuration was simulated with a 16KB, 32KB, and 64KB L1 size, using a 64B line size. The maximum

value of the y-axis is limited to 2 in these figures to improve readability.

Fig. 6. SPEC-FP: L1 misses expressed as normalized fraction of misses to conventional 64KB cache.

ICCs free cache lines in the L1 by moving the computa-tions that use them out to the DCPs. Because we assumea direct-mapped L1, the use of an ICC will improve themiss rate only if it results in the avoidance of a conflictbetween the ICL and another line. It is desirable that theblock can be used by lines that map to other blocks. Oneapproach to dealing with this situation is to increase theassociativity of the L1, but this will increase the hit time.Another approach is to consider cache configurations, suchas hash-rehash caches, that allow more flexible line place-ment. Agarawal et al. presented an improvement to hash-rehash caches called column-associative caches [1], which isthe configuration used here to exploit L1 blocks freed fromICLs. The second cache configuration in figures 5–8 showsthe change in miss rates with only the change from con-

ventional L1 cache to column-associative L1 cache. In thismanner, the figures distinguish between misses avoided bythe column-associative cache alone, the ICCs alone, andthe additive effects of both together.

Across all 47 benchmarks, halving the size of the base-line cache increases the number of misses by 38% on av-erage (geometric mean). The conventional 16KB cachehas an average increase of 81% misses over the baselinecache. In only one case (171.swim.1.1b) did the 64KBcolumn-associative cache not perform better than the base-line cache. In 34 cases, the 32KB column-associative cachedecreased the number of misses from the baseline, and in13 cases the 16KB cache also demonstrated a decrease inthe number of misses. The average changes in miss ratesfrom the baseline for the column-associative caches were

7

Fig. 7. Sandia-Int: L1 misses expressed as normalized fraction of misses to conventional 64KB cache.

Fig. 8. Sandia-FP: L1 misses expressed as normalized fraction of misses to conventional 64KB cache.

-33%, -13%, and 17%, for the 64KB, 32KB, and 16KBcases, respectively.

The 64KB ICC-enhanced system decreased the numberof misses from the baseline in all cases, while the 32KBICC-enhanced system showed an increase in only threecases (175.vpr.1.1b, 177.mesa.1.1b, 183.equake.1.1b). The16KB ICC-enhanced system showed an increase in numberof misses over the baseline in 11 cases. The average changesin miss rates for the 64KB, 32KB, and 16KB ICC-enhancedsystems were -53%, -43%, and -37%, respectively. Finally,ICCs combined with the column-associative L1 showed animprovement in L1 miss rate over the baseline for all ap-plications for both the 64KB and 32KB cases. And in only9 of the 47 cases did the 16KB combined configurationshow more misses than the baseline. The average changesin miss rates for the 64KB, 32KB, and 16KB cases were-61%, -54%, and -46%, respectively.

Hence, the 16KB cache size when using ICCs with thecolumn-associative L1 decreases the number of misses at

the host significantly from the baseline case. However,it is important to note that these results indicate a hugeamount of potential performance improvement due to ICCenhancements. The actual performance improvement re-alized is contigent on the architecture’s ability to exploitlocality and extract parallelism necessary to take advan-tage of this potential. However, due to the definition ofICCs discussed in previous sections, it is apparent that thedesign of such an architecture seems quite feasible.

Another important highlight of these results is that forthe ICC configurations, the difference in number of missesas cache size is increased is significantly smaller than inthe conventional cases. A good example of this is the caseof 183.equake.1.1b, for which there is a factor of 6.1 (6.1 &1 in figure 6) between the number of misses with a 64KBcache and 16KB cache. However, for the ICC + column-associative case, this factor is only 2 (1.2 & 0.6 in fig-ure 6). Across all benchmarks, the average of this factorfor the conventional, column-associative, ICC, and ICC +

8

TABLE II

Baseline Host Configuration

Register Update Unit Size 64 entriesLoad-Store Queue Size 32 entriesFetch Queue Size 4 entriesDecode Width 4Issue Width 8Functional Units Integer ALU - 3

Integer Mult/Div - 1FP Add - 2

FP Mult/Div - 1Branch Predictor Bimodal, 2048 entriesBranch Target Buffer 512 entries, 4-wayBranch Mispredict Penalty 3 cyclesCaches DL1 - 1024 sets, 64-byte blocks, direct-mapped, 4-cycle latency

DL2 - 2048 sets, 64-byte blocks, 16-way, 27-cycle latencyTLBs ITLB - 16 entries, 4-way

DTLB - 32 entries, 4-way30-cycle miss penalty

Main Memory Latency - 150 cyclesBandwidth - 64 bits/cycle

column-associative cases are 1.81, 1.74, 1.35, 1.39, respec-tively. This result is significant in the consideration of L1cache design tradeoffs. Because the factor of increase inL1 misses with decrease in cache size is much lower whenusing ICCs, the architect is more free to choose the smallerL1 cache size and gain the highly desirable benefits of de-creased hit time and energy costs than when using conven-tional caches.

In the next set of simulations, we used SimpleScalar toproject a bound on the speedup due to the improvementin the host’s L1 hit rate (called the λ-speedup). That is,the λ-speedup does not include the performance improve-ment due to the offloading of instructions from the host tothe DCPs. Table II gives the baseline SimpleScalar config-uration, and table III gives the speedup numbers. Thesespeedup numbers are an optimistic estimate for λ-speedupin that we free L1 blocks from ICLs by converting ICLs’memory accesses to single-cycle non-memory access ops.However, the speedup numbers do not take into accountbenefits due to the column-associative L1, which, giventhe results in figures 5–8, we predict will have significantperformance benefits. The speedup estimates vary widelyfrom negligible speedup (cube3.4B.crs) to 6.82 (sp.4B). Asexpected, high speedups tend to correlate with a high %of ICLs and a high number of evictions as given in tableI (sp.4B has the highest % ICLs of 94%). However, notethat this analysis covers only a portion of the trace usedin table I, which may result in disparity. The averages forSPEC-Int, SPEC-FP, Sandia-Int, and Sandia-FP are 1.15,1.54, 1.87, and 1.08, respectively, and the average acrossall traces is 1.32.

IV. Related Work

Previous work has presented a DCP architecture, ar-chitectural extensions to the host for supporting ICC ex-ecution, and definitions of two types of ICCs and corre-sponding code translation algorithms [14]. This past workfocused primarily on computation-centric ICCs – that is,

with data locality as a secondary consideration. Becausethe ICC formation did not take into account working setpartitioning between the DCPs and the host, the workingset of a given ICC might land at any position in the cachehierachy. Because of this, the design included DCPs ateach level in the cache hierarchy.

Previous studies have considered augmenting a conven-tional processor with compute logic in either the memorycontrollers or next to memory macros [22] using embeddedDRAM technology [10]. Examples of the former are theImpulse Memory Controller [25] and Active Memory Op-erations (AMOs) [7]. Impulse adds a layer of indirectionbetween main memory and the processor at the memorycontroller, allowing the application or OS to reorganizedata in memory to improve memory bandwidth utilizationand cache performance. AMOs extend Impulse by allow-ing computations at the memory controllers of distributedshared memory systems, as opposed to just providing amechanism for remapping of data addresses. AMOs workat a coarser level of granularity than ICCs, as they aremanaged with system calls and always execute at the mem-ory controller rather than within the caches, as do ICCs.Impulse and AMOs are not as architecturally radical oraggressive as the DCP design, which seeks to exploit con-tinuing improvements in merged logic and dense memoryfabrication with parallelism at fine levels of granularity.

Li et al. proposed an architecture utilizing logic nextto memory macros [15]. This architecture replicates alightweight processor (LWP) that supports large numbersof simultaneous threads. The programmer creates thesethreads through a special language supporting this styleof massive multithreading. An even simpler core is asso-ciated with each memory bank to assist in thread man-agement and atomic operations. The ICC design has keydifferences from LWPs, including the use of OoO cores inconjunction with the in-memory cores and also considera-tion to parallel computations created by a code translator.

The DCP-based architecture has some similarities to the

9

TABLE III

ICC Projected λ-Speedup Bound

Application Est. Projectedλ-Speedup Bound

SPEC-Int164.gzip.1.1b 1.16164.gzip.2.1b 1.33164.gzip.3.1b 1.15164.gzip.4.1b 1.23164.gzip.5.1b 1.19175.vpr.1.1b 1.04176.gcc.1.1b 1.08176.gcc.3.1b 1.03

SPEC-FP168.wupwise.1.1b 1.05171.swim.1.1b 3.75172.mgrid.1.1b 1.01173.applu.1.1b 2.45177.mesa.1.1b 1.00178.galgel.1.1b 1.01179.art.1.1b 2.23179.art.2.1b 3.66183.equake.1.1b 1.03187.facerec.1.1b 1.22189.lucas.1.1b 1.08191.fma3d.1.1b 1.97200.sixtrack.1.1b 1.04301.apsi.1.1b 1.90

Sandia-Intblast.4B 1.02chaco.4B 1.06dfs.4B 1.77iso.4B 5.58kmetis.4B 1.01sp.4B 6.82zchaff.4B 1.06

Sandia-FPalegra.4B.WES04 1.57alegra.4B.hemipen 1.29cth.4B.2gas 1.18cth.4B.amr-2d-cust4 1.04cth.4B.efp 1.01cth.4B.sc 1.01cube3.4B.crs 1.00cube3.4B.vbr 1.17its.4B.cad.brems 1.05its.4B.cg.brems 1.04lmp.4B.flow.langevin 1.05lmp.4B.lj.fix 1.04lmp.4B.poly.chain.a 1.04mpsalsa.4B.airport3d 3 1.02mpsalsa.4B.thermal... 1.04sppm.2B 1.01xyce.4B.invertgen75 1.02xyce.4B.invertgen v3 75 1.02

clustered VLIW architecture [21], [9], [26]. Chu et al. con-sider a multicore that resembles a clustered VLIW with dis-tributed data caches, using profile-guided analysis to par-titioning memory accesses and assigning operations acrosscores to reduce memory stall time [6]. However, their workconsiders only homogeneous cores with uniform cache dis-tribution at the L1-level, and also does not consider scien-tific workloads.

Studies focusing on addressing memory performancespecifically in multicores have included prefetching [5],cache partitioning [3], [19], and advanced insertion poli-cies [11]. We believe many of these techniques, especially

those supporting software caching for the Cell’s SPU’s, canbe combined with ICCs to achieve additional performancegains.

V. Conclusions and Future Work

The early results for ICCs are quite promising, revealinghigh percentages of potential ICC content across standardand scientific workloads. In summary, the paper has pre-sented the following key results:

• ICCs significantly decrease the number of host L1misses – up to 61% when used with a column-associative L1.

• The problem of branch-criticality in ICCs occurs veryrarely – less than 1% of ICCs are branch critical onaverage for the traces considered.

• A naive computational partitioning algorithm resultsin 1.5% of instructions offloaded from the host to theDCPs.

• Potential IPC improvement due to improved cachemiss rate observed was 1.32 on average.

These improvements will provide for more flexibility inthe design space that will likely allow for decreased L1cache size and reduction of traffic in the memory hier-archy, both highly desirable characteristics in the high-performance, low-power architecture design space.

Future work will be driven by tradeoff analysis of con-ventional processor design and ICC-enhanced computing.At the microarchitectural level, of particular interest arepower/performance tradeoffs at the host, cache controllers,on-chip interconnect, and DCP design for supporting thethree-phase ICC execution model, and how this designspace differs from that of conventional processors. For theISA and code translation, improved computational par-titioning schemes will provide decreased traffic throughpruning of ICC return register values and load balancingbetween the host and among the DCPs. Implementationdetails including the handling of frequent L1 cache lineinvalidations during the prefetch phase, L2 ICL evictionduring the setup phase, handling of IOB overflows, paral-lel ICC activators, and BCI inclusion will also be consid-ered. Microarchitectural and compilation components forstatistical analysis of computational and memory accesspatterns will also be useful, so as to eliminate the need fortrace analysis.

There is also a great interest in the use of ICCs to im-prove parallelization and cache performance when used inconjunction with other emerging multicore processor de-signs, especially through the decrease of coherence traffic.Alternative cache architectures – for example, caches withprogrammable decoders [24] – when used in tandem withICCs, may provide further improvement through increasedutilization of L1 cache lines freed from ICL usage with lit-tle or no increase in hit time. Finally, plans are in placefor the investigation of ICC effectiveness with other classesof codes, such as database applications involving frequentuse of spatially isolated writes to memory.

10

References

[1] A. Agarwal and S. D. Pudar, Column-associative caches: atechnique for reducing the miss rate of direct-mapped caches, inISCA ’93: Proceedings of the 20th annual international sym-posium on Computer architecture, New York, NY, USA, 1993,ACM, pp. 179–190.

[2] J. B. Brockman, S. Thoziyoor, S. K. Kuntz, and P. M.Kogge, A low cost, multithreaded processing-in-memory sys-tem, in WMPI ’04: Proceedings of the 3rd workshop on Memoryperformance issues, New York, NY, USA, 2004, ACM, pp. 16–22.

[3] J. Chang and G. S. Sohi, Cooperative cache partitioning forchip multiprocessors, in ICS ’07: Proceedings of the 21st annualinternational conference on Supercomputing, New York, NY,USA, 2007, ACM, pp. 242–252.

[4] S. Chaudhry, R. Cypher, M. Ekman, M. Karlsson,A. Landin, S. Yip, H. Zeffer, and M. Tremblay, Rock: Ahigh-performance sparc cmt processor, Micro, IEEE, 29 (2009),pp. 6–16.

[5] T. Chen, T. Zhang, Z. Sura, and M. G. Tallada, Prefetch-ing irregular references for software cache on cell, in CGO ’08:Proceedings of the sixth annual IEEE/ACM international sym-posium on Code generation and optimization, New York, NY,USA, 2008, ACM, pp. 155–164.

[6] M. Chu, R. Ravindran, and S. Mahlke, Data access parti-tioning for fine-grain parallelism on multicore architectures, inMICRO ’07: Proceedings of the 40th Annual IEEE/ACM In-ternational Symposium on Microarchitecture, Washington, DC,USA, 2007, IEEE Computer Society, pp. 369–380.

[7] Z. Fang, L. Zhang, J. B. Carter, A. Ibrahim, and M. A.Parker, Active memory operations, in ICS ’07: Proceedingsof the 21st annual international conference on Supercomputing,New York, NY, USA, 2007, ACM, pp. 232–241.

[8] R. Fromm, S. Perissakis, N. Cardwell, C. Kozyrakis,B. McGaughy, D. Patterson, T. Anderson, and K. Yelick,The energy efficiency of iram architectures, in ISCA ’97: Pro-ceedings of the 24th annual international symposium on Com-puter architecture, New York, NY, USA, 1997, ACM, pp. 327–337.

[9] E. Gibert, J. Sanchez, and A. Gonzalez, Distributed datacache designs for clustered vliw processors, Computers, IEEETransactions on, 54 (2005), pp. 1227–1241.

[10] S. Iyer, J. J.E. Barth, P. Parries, L. Norum, J. Rice, L. Lo-gan, and D. Hoyniak, Embedded dram: Technology platformfor the blue gene/l chip, IBM Journal of Research and Develop-ment, 49 (2005), pp. 333–350.

[11] A. Jaleel, W. Hasenplaugh, M. Qureshi, J. Sebot,S. Steely, Jr., and J. Emer, Adaptive insertion policies formanaging shared caches, in PACT ’08: Proceedings of the 17thinternational conference on Parallel architectures and compila-tion techniques, New York, NY, USA, 2008, ACM, pp. 208–219.

[12] M. Kaku, H. Iwai, T. Nagai, M. Wada, A. Suzuki, T. Takai,N. Itoga, T. Miyazaki, T. Iwai, H. Takenaka, T. Hojo,S. Miyano, and N. Otsuka, An 833mhz pseudo-two-port em-bedded dram for graphics applications, Feb. 2008, pp. 276–613.

[13] P. M. Kogge, Of piglets and threadlets: Architectures for self-contained, mobile, memory programming, in IWIA ’04: Pro-ceedings of the Innovative Architecture for Future GenerationHigh-Performance Processors and Systems, Washington, DC,USA, 2004, IEEE Computer Society, pp. 130–138.

[14] P. La Fratta and P. M. Kogge, Instructing the memory hier-archy with in-cache computations, in Workshop on InteractionBetween Compilers and Computer Architecture, February 2009.

[15] S. Li, A. Kashyap, S. Kuntz, J. Brockman, P. Kogge,P. Springer, and G. Block, A heterogeneous lightweightmultithreaded architecture, Parallel and Distributed ProcessingSymposium, 2007. IPDPS 2007. IEEE International, (2007),pp. 1–8.

[16] S. Maeda, S. Asano, T. Shimada, K. Awazu, and H. Tago, Areal-time software platform for the cell processor, Micro, IEEE,25 (2005), pp. 20–29.

[17] R. Murphy, A. Rodrigues, P. Kogge, and K. Underwood,The implications of working set analysis on supercomputingmemory hierarchy design, in ICS ’05: Proceedings of the 19thannual international conference on Supercomputing, New York,NY, USA, 2005, ACM Press, pp. 332–340.

[18] M. Ohmacht, D. Hoenicke, R. Haring, and A. Gara, Theedram based l3-cache of the bluegene/l supercomputer processornode, Oct. 2004, pp. 18–22.

[19] M. K. Qureshi and Y. N. Patt, Utility-based cache partition-ing: A low-overhead, high-performance, runtime mechanism topartition shared caches, Microarchitecture, 2006. MICRO-39.39th Annual IEEE/ACM International Symposium on, (2006),pp. 423–432.

[20] A. F. Rodrigues, Programming future architectures: dustydecks, memory walls, and the speed of light, PhD thesis, NotreDame, IN, USA, 2006. Adviser-Peter Kogge.

[21] J. Sanchez and A. Gonzalez, Instruction scheduling for clus-tered vliw architectures, in ISSS ’00: Proceedings of the 13th in-ternational symposium on System synthesis, Washington, DC,USA, 2000, IEEE Computer Society, pp. 41–46.

[22] S. Thoziyoor, J. Brockman, and D. Rinzler, Pim lite: amultithreaded processor-in-memory prototype, in GLSVSLI ’05:Proceedings of the 15th ACM Great Lakes symposium on VLSI,New York, NY, USA, 2005, ACM, pp. 64–69.

[23] S. P. Vanderwiel and D. J. Lilja, Data prefetch mechanisms,ACM Comput. Surv., 32 (2000), pp. 174–199.

[24] C. Zhang, Reducing cache misses through programmable de-coders, ACM Trans. Archit. Code Optim., 4 (2008), pp. 1–31.

[25] L. Zhang, Z. Fang, M. Parker, B. Mathew, L. Schaelicke,J. Carter, W. Hsieh, and S. McKee, The impulse mem-ory controller, Computers, IEEE Transactions on, 50 (2001),pp. 1117–1132.

[26] Y. Zhao, C. Xue, M. Li, and B. Hu, Energy-aware registerfile re-partitioning for clustered vliw architectures, Jan. 2009,pp. 805–810.