1

The Memory

HierarchyIn the book: 5.1-5.3, 5.7, 5.10

2

• Understand how CPUs run programs• How do we express the computation the CPU?

• How does the CPU execute it?

• How does the CPU support other system components (e.g., the OS)?

• What techniques and technologies are involved and how do they work?

• Understand why CPU performance (and other metrics) varies• How does CPU design impact performance?

• What trade-offs are involved in designing a CPU?

• How can we meaningfully measure and compare computer systems?

• Understand why program performance varies• How do program characteristics affect performance?

• How can we improve a programs performance by considering the CPU running it?

• How do other system components impact program performance?

Goals for this Class

3

memory

Abstraction: Big array of bytes

Memory

CPU

Memory

4

Main points for today• What is a memory hierarchy?

• What is the CPU-DRAM gap?

• What is locality? What kinds are there?

• Learn a bunch of caching vocabulary.

5

Processor vs Memory

Performance • Memory is very slow compared to

processors.

Pe

rfo

rma

nce

vs 1

98

0

SRAM and DRAM

6

7

Silicon Memories

• Why store things in silicon?• It’s fast!!!

• Compatible with logic devices (mostly)

• The main goal is to be cheap• Dense -- The smaller the bits, the less area you

need, and the more bits you can fit on a chip/wafer/through your fab.

• Bit sizes are measured in F2 -- the smallest feature you can create.

• The number of F2 /bit is a function of the memory technology, not the manufacturing technology.

• i.e. an SRAM in todays technology will take the same number of F2 in tomorrow’s technology

8

Questions

• What physical quantity should represent the bit?• Voltage/charge -- SRAMs, DRAMs, Flash memories

• Magnetic orientation -- MRAMs

• Crystal structure -- phase change memories

• The orientation of organic molecules -- various exotic technologies

• All that’s required is that we can sense it and turn it into a logic one or zero.

• How do we achieve maximum density?

• How do we make them fast?

9

Anatomy of a Memory

• Dense: Build a big array• bigger the better

• less other stuff

• Bigger -> slower

• Row decoder• Select the row by

raising a “word line”

• Column decoder• Select a slice of the row

• Decoders are pretty big.

10

The Storage Array

• Density is king.• Highly engineered, carefully tuned, automatically

generated.

• The smaller the devices, the better.

• Making them big makes them slow.• Bit/word lines are long (millimeters)

• They have large capacitance, so their RC delay is long

• For the row decoder, use large transistors to drive them hard.

• For the bit cells...• There are lots of these, so they need to be as small as

possible (but not smaller)

11

Measuring Memory Density

• We use a “technology independent” metric to measure the inherent size of different memory cells.• F == the “feature size” == the smallest dimension a CMOS

process can create (e.g., the width of the narrowest wire).

• In a 22nm process technology, F = 22nm.

• F2 (F-squared) is the smallest 2D feature we can manufacture.

• A single bit of a given type of memory (e.g., SRAM or DRAM) requires a fixed number of F2

• This number doesn’t change with process technology.

• e.g., NAND flash memory is 4F2 in 90nm and in 22nm.

• Using this metic is useful because the relative sizes of different memory technologies don’t change much, although absolute densities do.

12

Sense Amps

• Sense amplifiers take a difference between two signals and amplify it

• Two scenarios• Inputs are initially equal (“precharged”) -- they

each move in opposite directions

• One input is a reference -- so only one signal moves

• Frequently used in memories• Storage cells are small, so the signals they produce

are inherently weak

• Sense amps can detect these weak, analog signals and convert them into a logic one or logic zero.

13

Static Random Access Memory (SRAM)

• Storage• Voltage on a pair of cross-

coupled inverters

• Durable in presence of power

• To read• Pre-charge two bit lines to

Vcc/2

• Turn on the “word line”

• Read the output of the sense-amp

1 0

1 0

1

14

1 0

SRAM Writes

• To write• Turn off the sense-amp

• Turn on the wordline

• Drive the bitlines to the correct state

• Turn off the wordline

1 0

0

10

15

Building SRAM

• This is “6T SRAM”• 6 transistors is pretty

big

• SRAMs are not dense

16

SRAM Density

• At 65nm: 0.52um2

• 123-140 F2

• [ITRS 2008]

65nm TSMC 6T SRAM

17

SRAM Ports

• Add word and bit lines

• Read/write multiple things at once

• Density decreases quadratically

• Bandwidth increase linearly

18

SRAM Performance

• Read and write times• 10s-100s of ps

• Bandwidth• Registers -- 324GB/s

• L1 cache -- 128GB/s

19

DRAM

20

Dynamic Random Access Memory (DRAM)

• Storage• Charge on a capacitor

• Decays over time (us-scale)

• This is the “dynamic” part.

• About 6F2: 20x better than SRAM

• Reading• Precharge

• Assert word line

• Sense output

• Refresh data Only one bit line is read at a time.

The other bit line serves as a reference.

The bit cells attached to Wordline 1 are not shown.

21

DRAM: Write and Refresh

• Writing• Turn on the wordline

• Override the sense amp.

• Refresh• Every few milli-seconds,

read and re-write every bit.

• Consumes power

• Takes time

22

DRAM Lithography:

How do you get a big capacitor?

C ~ Area/dielectric-thickness

Stacked Capacitors

23

DRAM Lithography

Trench Capacitors

24

Accessing DRAM• Apply the row address

• “opens a page”

• Slow (~12ns read + 24 ns precharge)

• Contents in a “row buffer”

• Apply one or more column addrs

• fast (~3ns)

• Reads and/or writes

16k Rows

One DD3

DRAM bank

25

DRAM Devices

• There are many banks per die (16 at left)• Multiple pages can be open at once.

• Can keep pages open longer

• Parallelism

• Example• open bank 1, row 4

• open bank 2, row 7

• open bank 3, row 10

• read bank 1, column 8

• read bank 2, column 32

• ...

Micron 78nm 1Gb DDR3

26

DRAM: Micron MT47H512M4

27

DRAM: Micron MT47H512M4

28

DRAM Variants

• The basic DRAM technology has been wrapped in several different interfaces.

• SDRAM (synchronous)

• DDR SDRAM (double data-rate)• Data clocked on rising and falling edge of the

clock.

• DDR2 -- faster, lower voltage DDR

• DDR3 -- even faster, even lower-voltage

• GDDR2-5 -- For graphics cards.

29

Current State-of-the-art: DDR3 SDRAM

• DIMM data path is 64bits (72 with ECC)

• Data rate: up to 1066Mhz DDR (2133Mhz effective)

• Bandwidth per DIMM GTNE: 16GB/s• guaranteed not to exceed

• Multiple DIMMs can attach to a bus• Reduces bandwidth/GB (a good idea?)

Each chip provides one

8-bit slice.

The chips are all

synchronized and

received the same

commands

30

DRAM Scaling

• Long term need for performance has driven DRAM hard

• complex interface.

• High performance

• High power.

• DRAM used to be the main driver for process scaling, now it’s flash.

• Power is now a major concern.

• Scaling is expected to match CMOS tech scaling

• F2 cell size will probably not decrease

• Historical foot note: Intel got its start as a DRAM company, but got out of it when it became a commodity.

31

A Typical Hierarchy: Costs and Speeds

On-chip L1 cache

SRAM

KBs

main memory

GBs

Disk

TBs

Cost

0.009 $/MB

0.00004 $/MB

Access time

60ns

10,000,000ns

< 1ns

On-chip L2 cache

SRAM

KBs

On-chip L3 cache

SRAM

MBs

< 2-3ns

< 10ns

???

???

???

SSDs

GB0.0006 $/MB 20,000ns

32© 2004 Jim Gray, Microsoft Corporation

Los Angeles

32

How far away is the data?

33

Typical Hierarchy: Architecture

34

The Principle of Locality

• “Locality” is the tendency of data access to be predictable. There are two kinds:

• Spatial locality: The program is likely to access data that is close to data it has accessed recently

• Temporal locality: The program is likely to access the same data repeatedly.

35

Memory’s impactM = % mem ops

Mlat (cycles) = average memory latency

BCPI = base CPI with single-cycle data memory

CPI =

36

Memory’s impact

M = % mem ops

Mlat (cycles) = average memory latency

TotalCPI = BaseCPI + M*Mlat

Example:

BaseCPI = 1; M = 0.2; Mlat = 240 cycles

TotalCPI = 49

Speedup = 1/49 = 0.02 => 98% drop in performance

Remember!: Amdahl’s law does not bound the slowdown. Poor memory performance can make your program arbitrarily slow.

37

• Why did branch prediction work?

Why should we expect caching to work?

38

• Why did branch prediction work?

• Where is memory access predictable• Predictably accessing the same data• In loops: for(i = 0; i < 10; i++) {s += foo[i];}

• foo = bar[4 + configuration_parameter];

• Predictably accessing different data• In linked lists: while(l != NULL) {l = l->next;}

• In arrays: for(i = 0; i < 10000; i++) {s += data[i];}

• structure access: foo(some_struct.a, some_struct.b);

Why should we expect caching to work?

39

The Principle of Locality• “Locality” is the tendency of data access to

be predictable. There are two kinds:

• Spatial locality: The program is likely to access data that is close to data it has accessed recently

• Temporal locality: The program is likely to access the same data repeatedly.

40

Locality in Action• Label each access

with whether it has temporal or spatial locality or neither• 1

• 2

• 3

• 10

• 4

• 1800

• 11

• 30

• 1

• 2

• 3

• 4

• 10

• 190

• 11

• 30

• 12

• 13

• 182

• 1004

41

Locality in Action• Label each access

with whether it has temporal or spatial locality or neither• 1 n

• 2 s

• 3 s

• 10 n

• 4 s

• 1800 n

• 11 s

• 30 n

• 1 t

• 2 s, t

• 3 s,t

• 4 s,t

• 10 s,t

• 190 n

• 11 s,t

• 30 s

• 12 s

• 13 s

• 182 n?

• 1004 n

There is no hard and fast rule here. In practice, locality

exists for an access if the cache performs well.

42

Cache Vocabulary• Hit - The data was found in the cache

• Miss - The data was not found in the cache

• Hit rate - hits/total accesses

• Miss rate = 1- Hit rate

• Locality - see previous slides

• Cache line - the basic unit of data in a cache. generally several words.

• Tag - the high order address bits stored along with the data to identify the actual address of the cache line.

• Hit time -- time to service a hit

• Miss time -- time to service a miss (this is a function of the lower level caches.)

43

Cache Vocabulary• There can be many caches stacked on top of each

other

• if you miss in one you try in the “lower level cache” Lower level, mean higher number

• There can also be separate caches for data and instructions. Or the cache can be “unified”

• In the 5-stage MIPS pipeline• the L1 data cache (d-cache) is the one nearest processor. It

corresponds to the “data memory” block in our pipeline diagrams

• the L1 instruction cache (i-cache) corresponds to the “instruction memory” block in our pipeline diagrams.

• The L2 sits underneath the L1s.

• There is often an L3 in modern systems.

44

Typical Cache Hierarchy

45

Data vs Instruction Caches• Why have different I and D caches?

46

Data vs Instruction Caches• Why have different I and D caches?• Different areas of memory

• Different access patterns• I-cache accesses have lots of spatial locality. Mostly sequential

accesses.

• I-cache accesses are also predictable to the extent that branches are predictable

• D-cache accesses are typically less predictable

• Not just different, but often across purposes.• Sequential I-cache accesses may interfere with the data the D-

cache has collected.

• This is “interference” just as we saw with branch predictors

• At the L1 level it avoids a structural hazard in the pipeline

• Writes to the I cache by the program are rare enough that they can be slow (i.e., self modifying code)