Upload
others
View
8
Download
0
Embed Size (px)
Citation preview
CIS371 (Roth/Martin): Caches 1
CIS 371 Computer Organization and Design
Part III: Memory Hierarchy
Unit 9: Caches
CIS371 (Roth/Martin): Caches 2
Recall: Binary Tree Performance vs Size
CIS371 (Roth/Martin): Caches 3
Recall: Binary Tree Performance vs Size
5x
1M
What is going on here?
5x difference
CIS371 (Roth/Martin): Caches 4
Average Instructions per Lookup
So number of instructions isn’t the problem
CIS371 (Roth/Martin): Caches 5
This Unit: Caches
•! Basic memory hierarchy concepts
•! Speed vs capacity
•! Caches
•! Later •! Organizing an entire memory hierarchy
•! Main memory
•! Virtual memory
CPU Mem I/O
System software
App App App
CIS371 (Roth/Martin): Caches 6
Readings
•! P+H
•! Chapter 7
•! Except 7.4
CIS371 (Roth/Martin): Caches 7
Motivation: Types of Memory
•! Static RAM (SRAM) •! 6 transistors per bit (two inverters, two other transistors for off/on)
•! Optimized for speed (first) and density (second)
•! Fast (sub-nanosecond latencies for small SRAM)
•! Speed proportional to its area
•! Mixes well with standard processor logic
•! Dynamic RAM (DRAM) •! 1 transistor + 1 capacitor per bit
•! Optimized for density (in terms of cost per bit)
•! Slow (>40ns internal access, ~100ns pin-to-pin)
•! Different fabrication steps (does not mix well with logic)
•! Nonvolatile storage: Magnetic disk, Flash RAM
CIS371 (Roth/Martin): Caches 8
Memory & Storage Technologies
•! Cost - what can $200 buy today? •! SRAM - 4MB
•! DRAM - 1,000MB (1GB) --- 250x cheaper than SRAM
•! Disk - 500,000MB (500GB) --- 500x cheaper than DRAM
•! Latency •! SRAM - <1 to 5ns (on chip)
•! DRAM - ~100ns --- 100x or more slower
•! Disk - 10,000,000ns or 10ms --- 100,000x slower (mechanical)
•! Bandwidth •! SRAM - 10-100GB/sec
•! DRAM - ~1GB/sec
•! Disk - 100MB/sec (0.1 GB/sec) - sequential access only
•! Aside: Flash, a non-traditional (and nonvolatile) memory •! 16GB for $200, 16x cheaper than DRAM! (But 30x more than disk)
CIS371 (Roth/Martin): Caches 9
Storage Technology Trends
Cost
Access Time Copyright Elsevier Scientific 2003
CIS371 (Roth/Martin): Caches 10
The “Memory Wall”
•! Processors are get faster more quickly than memory (note log scale)
•! Processor speed improvement: 35% to 55%
•! Memory latency improvement: 7%
Copyright Elsevier Scientific 2003
Log scale
+35 to 55%
+7%
CIS371 (Roth/Martin): Caches 11
Locality to the Rescue
•! Locality of memory references
•! Property of real programs, few exceptions
•! Books and library analogy
•! Temporal locality
•! Recently referenced data is likely to be referenced again soon
•! Reactive: cache recently used data in small, fast memory
•! Spatial locality
•! More likely to reference data near recently referenced data
•! Proactive: fetch data in large chunks to include nearby data
•! Holds for data and instructions
CIS371 (Roth/Martin): Caches 12
Known From the Beginning
“Ideally, one would desire an infinitely large memory capacity such that any particular word would be immediately available … We are forced to recognize the possibility of constructing a hierarchy of memories, each of which has a greater capacity than the preceding but which is less quickly accessible.”
Burks, Goldstine, VonNeumann
“Preliminary discussion of the logical design of an electronic computing instrument”
IAS memo 1946
Library Analogy
•! Consider books in a library
•! Library has lots of books, but it is slow to access
•! Far away (time to walk to the library)
•! Big (time to walk within the library)
•! How can you avoid these latencies?
•! Check out books, take them home with you
•! Put them on desk, on bookshelf, etc.
•! But desks & bookshelves have limited capacity
•!Keep recently used books around (temporal locality)
•!Grab books on related topic at the same time (spatial locality)
•!Guess what books you’ll need in the future (prefetching)
CIS371 (Roth/Martin): Caches 13 CIS371 (Roth/Martin): Caches 14
Exploiting Locality: Memory Hierarchy
•! Hierarchy of memory components
•! Upper components
•! Fast ! Small ! Expensive
•! Lower components
•! Slow ! Big ! Cheap
•! Connected by “buses”
•! Which also have latency and bandwidth issues
•! Most frequently accessed data in M1
•! M1 + next most frequently accessed in M2, etc.
•! Move data up-down hierarchy
•! Optimize average access time •! latencyavg = latencyhit + %miss * latencymiss
•! Attack each component
CPU
M1
M2
M3
M4
CIS371 (Roth/Martin): Caches 15
Concrete Memory Hierarchy
•! 0th level: Registers
•! 1st level: Primary caches •! Split instruction (I$) and data (D$)
•! Typically 8KB to 64KB each
•! 2nd level: Second-level cache (L2$)
•! On-chip, certainly on-package (with CPU)
•! Made of SRAM (same circuit type as CPU)
•! Typically 512KB to 16MB
•! 3rd level: main memory •! Made of DRAM (“Dynamic” RAM)
•! Typically 1GB to 4GB for desktops/laptops
•! Servers can have 100s of GB
•! 4th level: disk (swap and files) •! Uses magnetic disks
Processor
D$
L2$
Main Memory
I$
Disk
Compiler Managed
Hardware Managed
Software Managed (by OS)
Regs
CIS371 (Roth/Martin): Caches 16
Evolution of Cache Hierarchies
Intel 486
8KB I/D$
1.5MB L2
L3 tags
64KB D$
64KB I$
IBM Power5 (dual core)
•! Chips today are 30–70% cache by area
Library Analogy Revisited
•! Registers ! books on your desk
•! Actively being used, small capacity
•! Caches ! bookshelves
•! Moderate capacity, pretty fast to access
•! Main memory ! library
•! Big, holds almost all data, but slow
•! Disk (swap) ! inter-library loan
•! Very slow, but hopefully really uncommon
CIS371 (Roth/Martin): Caches 17 CIS371 (Roth/Martin): Caches 18
This Unit: Caches
•! “Cache”: hardware managed •! Hardware automatically retrieves missing data
•! Built from fast SRAM, usually on-chip today
•! In contrast to off-chip, DRAM “main memory”
•! Cache organization •! ABC
•! Miss classification
•! High-performance techniques •! Reducing misses
•! Improving miss penalty
•! Improving hit latency
•! Some example performance calculations
CPU
D$
L2
Main Memory
I$
Disk
CIS371 (Roth/Martin): Caches 19
Looking forward: Memory and Disk
•! Main memory
•! DRAM-based memory systems
•! Virtual memory
•! Disks and Storage
•! Properties of disks
•! Disk arrays (for performance and reliability)
CPU
Main Memory
Disk
D$
L2$
I$
CIS371 (Roth/Martin): Caches 20
Basic Memory Array Structure
•! Number of entries
•! 2n, where n is number of address bits
•! Example: 1024 entries, 10 bit address
•! Decoder changes n-bit address to 2n bit “one-hot” signal
•! One-bit address travels on “wordlines”
•! Size of entries •! Width of data accessed
•! Data travels on “bitlines”
•! 256 bits (32 bytes) in example
0
1
1021
1022
1023
2
3
1024*256bit SRAM
bitlines
wo
rdlin
es
10 bits
CIS371 (Roth/Martin): Caches 21
FYI: Physical Memory Layout
•! Logical layout
•! Arrays are vertically contiguous
•! Physical layout - roughly square •! Vertical partitioning to minimize wire lengths
•! H-tree: horizontal/vertical partitioning layout
•!Applied recursively
•!Each node looks like an H
512
513
1022
1023
767
data address
0
1
510
511
255
256 768
CIS371 (Roth/Martin): Caches 22
Physical Cache Layout
•! Arrays and h-trees make caches easy to spot in µgraphs
CIS371 (Roth/Martin): Caches 23
Caches: Finding Data via Indexing
•! Basic cache: array of block frames •! Example: 32KB cache (1024 frames, 32B blocks)
•! “Hash table in hardware”
•! To find frame: decode part of address •! Which part?
•! 32-bit address
•! 32B blocks " 5 lowest bits locate byte in block
•!These are called offset bits
•! 1024 frames " next 10 bits find frame
•!These are the index bits
•! Note: nothing says index must be these bits
•! But these work best (think about why)
0
1
1021
1022
1023
2
3
[4:0] [31:15] index [14:5] <<
1024* 256bit SRAM
bitlines
wo
rdlin
es
data address CIS371 (Roth/Martin): Caches 24
Knowing that You Found It: Tags
•! Each frame can hold one of 217 blocks
•! All blocks with same index bit pattern
•! How to know which if any is currently there? •! To each frame attach tag and valid bit
•! Compare frame tag to address tag bits
•!No need to match index bits (why?)
•! Lookup algorithm •! Read frame indicated by index bits
•! “Hit” if tag matches and valid bit is set
•! Otherwise, a “miss”. Get data from next level
0
1
1021
1022
1023
2
3
[4:0] tag [31:15]
data
index [14:5] <<
address
=
hit?
wo
rdlin
es
CIS371 (Roth/Martin): Caches 25
Calculating Tag Overhead
•! “32KB cache” means cache holds 32KB of data
•! Called capacity
•! Tag storage is considered overhead
•! Tag overhead of 32KB cache with 1024 32B frames
•! 32B frames " 5-bit offset
•! 1024 frames " 10-bit index
•! 32-bit address – 5-bit offset – 10-bit index = 17-bit tag
•! (17-bit tag + 1-bit valid)* 1024 frames = 18Kb tags = 2.2KB tags
•! ~6% overhead
•! What about 64-bit addresses?
•! Tag increases to 49bits, ~20% overhead (worst case)
CIS371 (Roth/Martin): Caches
Handling a Cache Miss
•! What if requested data isn’t in the cache?
•! How does it get in there?
•! Cache controller: finite state machine
•! Remembers miss address
•! Accesses next level of memory
•! Waits for response
•! Writes data/tag into proper locations
•! All of this happens on the fill path
•! Sometimes called backside
26
CIS371 (Roth/Martin): Caches 27
Cache Misses and Pipeline Stalls
•! I$ and D$ misses stall pipeline just like data hazards
•! Stall logic driven by miss signal
•!Cache “logically” re-evaluates hit/miss every cycle
•!Data is filled " miss signal de-asserts " pipeline restarts
I$ Regfile D$
a
d
+ 4
nop nop
CIS371 (Roth/Martin): Caches 28
Cache Performance Equation
•! For a cache •! Access: read or write to cache
•! Hit: desired data found in cache
•! Miss: desired data not found in cache
•!Must get from another component
•!No notion of “miss” in register file
•! Fill: action of placing data into cache
•! %miss (miss-rate): #misses / #accesses
•! thit: time to read data from (write data to) cache
•! tmiss: time to read data into cache
•! Performance metric: average access time
tavg = thit + %miss * tmiss
Cache
thit
tmiss
%miss
CIS371 (Roth/Martin): Caches 29
CPI Calculation with Cache Misses
•! In a pipelined processor, I$/D$ thit is “built in” (effectively 0) •! High thit will simply require multiple F or M stages (deeper pipeline)
•! Parameters •! Simple pipeline with base CPI of 1
•! Instruction mix: 30% loads/stores
•! I$: %miss = 2%, tmiss = 10 cycles
•! D$: %miss = 10%, tmiss = 10 cycles
•! What is new CPI? •! CPII$ = %missI$*tmiss = 0.02*10 cycles = 0.2 cycle
•! CPID$ = %load/store*%missD$*tmissD$ = 0.3 * 0.1*10 cycles = 0.3 cycle
•! CPInew = CPI + CPII$ + CPID$ = 1+0.2+0.3 = 1.5
CIS371 (Roth/Martin): Caches 30
Measuring Cache Performance
•! Ultimate metric is tavg
•! Cache capacity and circuits roughly determines thit
•! Lower-level memory structures determine tmiss
•! Measure %miss
•!Hardware performance counters (Pentium)
•! Simulation (homework)
•! Paper simulation (next)
CIS371 (Roth/Martin): Caches 31
Cache Miss Paper Simulation
•! 4-bit addresses " 16B memory •! Simpler cache diagrams than 32-bits
•! 8B cache, 2B blocks •! Figure out number of sets: 4 (capacity / block-size)
•! Figure out how address splits into offset/index/tag bits
•!Offset: least-significant log2(block-size) = log2(2) = 1 " 0000
•! Index: next log2(number-of-sets) = log2(4) = 2 " 0000
•!Tag: rest = 4 – 1 – 2 = 1 " 0000
•! Cache diagram •! 0000|0001 are addresses of bytes in this block, values don’t matter
Cache contents Address Outcome
Set00 Set01 Set10 Set11
0000|0001 0010|0011 0100|0101 0110|0111
CIS371 (Roth/Martin): Caches 32
Cache Miss Paper Simulation
•! 8B cache, 2B blocks
Cache contents (prior to access) Address Outcome
Set00 Set01 Set10 Set11
0000|0001 0010|0011 0100|0101 0110|0111 1100 Miss
0000|0001 0010|0011 1100|1101 0110|0111 1110 Miss
0000|0001 0010|0011 1100|1101 1110|1111 1000 Miss
1000|1001 0010|0011 1100|1101 1110|1111 0011 Hit
1000|1001 0010|0011 1100|1101 1110|1111 1000 Hit
1000|1001 0010|0011 1100|1101 1110|1111 0000 Miss
0000|0001 0010|0011 1100|1101 1110|1111 1000 Miss
•! How to reduce %miss? And hopefully tavg?
1 bit tag (1 bit) index (2 bits)
CIS371 (Roth/Martin): Caches 33
Capacity and Performance
•! Simplest way to reduce %miss: increase capacity +!Miss rate decreases monotonically
•!“Working set”: insns/data program is actively using
•!Diminishing returns
–! However thit increases
•! Latency proportional to sqrt(capacity)
•! tavg ?
•! Given capacity, manipulate %miss by changing organization
Cache Capacity
%miss “working set” size
CIS371 (Roth/Martin): Caches 34
Block Size
•! Given capacity, manipulate %miss by changing organization
•! One option: increase block size •! Exploit spatial locality
•! Notice index/offset bits change
•! Tag remain the same
•! Ramifications +!Reduce %miss (up to a point)
+!Reduce tag overhead (why?)
–! Potentially useless data transfer
–! Premature replacement of useful data
–! Fragmentation
0
1
510
511
2
[5:0] [31:15]
data
[14:6]
address
=
hit?
<<
512*512bit SRAM
9-bit
block size#
CIS371 (Roth/Martin): Caches 35
Block Size and Tag Overhead
•! Tag overhead of 32KB cache with 1024 32B frames
•! 32B frames " 5-bit offset
•! 1024 frames " 10-bit index
•! 32-bit address – 5-bit offset – 10-bit index = 17-bit tag
•! (17-bit tag + 1-bit valid) * 1024 frames = 18Kb tags = 2.2KB tags
•! ~6% overhead
•! Tag overhead of 32KB cache with 512 64B frames •! 64B frames " 6-bit offset
•! 512 frames " 9-bit index
•! 32-bit address – 6-bit offset – 9-bit index = 17-bit tag
•! (17-bit tag + 1-bit valid) * 512 frames = 9Kb tags = 1.1KB tags
+!~3% overhead
CIS371 (Roth/Martin): Caches 36
Block Size Cache Miss Paper Simulation
•! 8B cache, 4B blocks
+!Spatial “prefetching”: miss on 1100 brought in 1110
–! Conflicts: miss on 1000 kicked out 0011
Cache contents (prior to access) Address Outcome
Set0 Set1
0000|0001|0010|0011 0100|0101|0110|0111 1100 Miss
0000|0001|0010|0011 1100|1101|1110|1111 1110 Hit (spatial locality)
0000|0001|0010|0011 1100|1101|1110|1111 1000 Miss
1000|1001|1010|1011 1100|1101|1110|1111 0011 Miss (conflict)
0000|0001|0010|0011 1100|1101|1110|1111 1000 Miss (conflict)
1000|1001|1010|1011 1100|1101|1110|1111 0000 Miss
0000|0001|0010|0011 1100|1101|1110|1111 1000 Miss
2 bits tag (1 bit) index (1 bit)
CIS371 (Roth/Martin): Caches 37
Effect of Block Size on Miss Rate
•! Two effects on miss rate +!Spatial prefetching (good)
•! For blocks with adjacent addresses
•! Turns miss/miss into miss/hit pairs
–! Interference (bad)
•! For blocks with non-adjacent addresses (but in adjacent frames)
•! Turns hits into misses by disallowing simultaneous residence
•! Consider entire cache as one big block
•! Both effects always present •! Spatial prefetching dominates initially
•! Depends on size of the cache
•! Good block size is 16–128B
•! Program dependent
Block Size
%miss
CIS371 (Roth/Martin): Caches 38
Block Size and Miss Penalty
•! Does increasing block size increase tmiss?
•! Don’t larger blocks take longer to read, transfer, and fill?
•! They do, but…
•! tmiss of an isolated miss is not affected
•! Critical Word First / Early Restart (CRF/ER)
•! Requested word fetched first, pipeline restarts immediately
•! Remaining words in block transferred/filled in the background
•! tmiss’es of a cluster of misses will suffer
•! Reads/transfers/fills of two misses can’t happen at the same time
•! Latencies can start to pile up
•! This is a bandwidth problem (more later)
CIS371 (Roth/Martin): Caches 39
Conflicts
•! 8B cache, 2B blocks
Cache contents (prior to access) Address Outcome
Set00 Set01 Set10 Set11
0000|0001 0010|0011 0100|0101 0110|0111 1100 Miss
0000|0001 0010|0011 1100|1101 0110|0111 1110 Miss
0000|0001 0010|0011 1100|1101 1110|1111 1000 Miss
1000|1001 0010|0011 1100|1101 1110|1111 0011 Hit
1000|1001 0010|0011 1100|1101 1110|1111 1000 Hit
1000|1001 0010|0011 1100|1101 1110|1111 0000 Miss
0000|0001 0010|0011 1100|1101 1110|1111 1000 Miss
•! Pairs like 0000/1000 conflict
•! Regardless of block-size (assuming capacity < 16)
•! Q: can we allow pairs like these to simultaneously reside?
•! A: yes, reorganize cache to do so
1 bit tag (1 bit) index (2 bits)
CIS371 (Roth/Martin): Caches 40
Set-Associativity
•! Set-associativity •! Block can reside in one of few frames
•! Frame groups called sets
•! Each frame in set called a way
•! This is 2-way set-associative (SA)
•! 1-way " direct-mapped (DM)
•! 1-set " fully-associative (FA)
+!Reduces conflicts
–! Increases latencyhit:
•! additional tag match & muxing
•! Note: valid bit not shown
512
513
1022
1023
514
data
<<
address
=
hit?
0
1
510
511
2
=
ways
sets
[4:0] [31:14] [13:5]
9-bit
associativity#
CIS371 (Roth/Martin): Caches 41
Set-Associativity
•! Lookup algorithm
•! Use index bits to find set
•! Read data/tags in all frames in parallel
•! Any (match and valid bit), Hit
•! Notice tag/index/offset bits
•!Only 9-bit index (versus 10-bit for direct mapped)
•! Notice block numbering
512
513
1022
1023
514
data
<<
address
=
hit?
0
1
510
511
2
=
ways
sets
[4:0] [31:14] [13:5]
9-bit
associativity#
CIS371 (Roth/Martin): Caches 42
Associativity and Miss Paper Simulation
+!Avoid conflicts: 0000 and 1000 can both be in set 0
–! Introduce some new conflicts: notice address re-arrangement
•!Happens, but conflict avoidance usually dominates
•! 8B cache, 2B blocks, 2-way set-associative
Cache contents (prior to access) Address Outcome
Set0.Way0 Set0.Way1 Set1.Way0 Set1.Way1
0000|0001 0100|0101 0010|0011 0110|0111 1100 Miss
1100|1101 0100|0101 0010|0011 0110|0111 1110 Miss
1100|1101 0100|0101 1110|1111 0110|0111 1000 Miss
1100|1101 1000|1001 1110|1111 0110|0111 0011 Miss (new conflict)
1100|1101 1000|1001 1110|1111 0010|0011 1000 Hit
1100|1101 1000|1001 1110|1111 0010|0011 0000 Miss
0000|0001 1000|1001 1110|1111 0010|0011 1000 Hit (avoid conflict)
CIS371 (Roth/Martin): Caches 43
Replacement Policies
•! Set-associative caches present a new design choice •! On cache miss, which block in set to replace (kick out)?
•! Some options •! Random
•! FIFO (first-in first-out)
•! LRU (least recently used)
•! Fits with temporal locality, LRU = least likely to be used in future
•! NMRU (not most recently used)
•!An easier to implement approximation of LRU
•! Is LRU for 2-way set-associative caches
•! Belady’s: replace block that will be used furthest in future
•!Unachievable optimum
•! Which policy is simulated in previous example?
CIS371 (Roth/Martin): Caches 44
NMRU and Miss Handling
•! Add MRU field to each set •! MRU data is encoded “way”
•! Hit? update MRU
•! MRU/LRU bits updated on each access
512
513
1023
data
<<
address
=
hit?
0
1
511
=
WE
data from memory
[4:0] [31:15] [14:5]
CIS371 (Roth/Martin): Caches 45
Parallel or Serial Tag Access?
•! Note: data and tags actually physically separate
•! Split into two different arrays
•! Parallel access example:
data
<<
= = = =
offset tag 2-bit index
2-bit
2-bit
Four blocks transferred
CIS371 (Roth/Martin): Caches 46
Serial Tag Access
•! Tag match first, then access only one data block
•! Advantages: lower power, fewer wires/pins
•! Disadvantages: slow
<<
= = = =
offset tag 2-bit index
2-bit
2-bit
4-bit
Only one block transferred
CPU Data
Tags
Serial
CPU Data
Tags
Parallel
Chip boundary
Chip boundary
data
CIS371 (Roth/Martin): Caches 47
Best of Both? Way Prediction
•! Predict “way” of block •! Just a “hint”
•! Use the index plus some tag bits
•! Table of n-bit entries for 2n associative cache
•! Update on mis-prediction or replacement
•! Advantages •! Fast
•! Low-power
•! Disadvantage •! More “misses”
<< = = = =
offset tag 2-bit index
2-bit
2-bit
4-bit
Way Predictor
=
data hit CIS371 (Roth/Martin): Caches 48
Associativity And Performance
•! Higher associative caches +!Have better (lower) %miss
•!Diminishing returns
–! However thit increases
•!The more associative, the slower
•! What about tavg?
•! Block-size and number of sets should be powers of two •! Makes indexing easier (just rip bits out of the address)
•! 3-way set-associativity? No problem
Associativity
%miss ~5
CIS371 (Roth/Martin): Caches 49
Classifying Misses: 3(4)C Model
•! Divide cache misses into three categories •! Compulsory (cold): never seen this address before
•!Would miss even in infinite cache
•! Identify? easy
•! Capacity: miss caused because cache is too small
•!Would miss even in fully associative cache
•! Identify? Consecutive accesses to block separated by access to at least N other distinct blocks (N is number of frames in cache)
•! Conflict: miss caused because cache associativity is too low
•! Identify? All other misses
•! (Coherence): miss due to external invalidations
•!Only in shared memory multiprocessors (later)
•! Who cares? Different techniques for attacking different misses
CIS371 (Roth/Martin): Caches 50
Miss Rate: ABC
•! Capacity +!Decreases capacity misses
–! Increases latencyhit
•! Associativity +!Decreases conflict misses
–! Increases latencyhit
•! Block size –! Increases conflict/capacity misses (fewer frames)
+!Decreases compulsory/capacity misses (spatial locality)
•! No significant effect on latencyhit
•! Why do we care about 3C miss model? •! So that we know what to do to eliminate misses
•! If you don’t have conflict misses, increasing associativity won’t help
CIS371 (Roth/Martin): Caches 51
Reducing Conflict Misses: Victim Buffer
•! Conflict misses: not enough associativity
•! High-associativity is expensive, but also rarely needed
•! 3 blocks mapping to same 2-way set and accessed (XYZ)+
•! Victim buffer (VB): small fully-associative cache
•! Sits on I$/D$ miss path
•! Small so very fast (e.g., 8 entries)
•! Blocks kicked out of I$/D$ placed in VB
•! On miss, check VB: hit? Place block back in I$/D$
•! 8 extra ways, shared among all sets
+!Only a few sets will need it at any given time
+!Very effective in practice
•! Does VB reduce %miss or latencymiss?
I$/D$
L2
VB
CIS371 (Roth/Martin): Caches 52
Lockup Free Cache
•! Lockup free: allows other accesses while miss is pending •! Consider: Load [r1] -> r2; Load [r3] -> r4; Add r2, r4 -> r5
•! Handle misses in parallel
•! “memory-level parallelism”
•! Makes sense for…
•! Processors that can go ahead despite D$ miss (out-of-order)
•! Implementation: miss status holding register (MSHR)
•!Remember: miss address, chosen frame, requesting instruction
•!When miss returns know where to put block, who to inform
•! Common scenario: “hit under miss”
•!Handle hits while miss is pending
•!Easy
•! Less common, but common enough: “miss under miss”
•!A little trickier, but common anyway
•!Requires multiple MSHRs: search to avoid frame conflicts
CIS371 (Roth/Martin): Caches 53
Software Restructuring: Data
•! Capacity misses: poor spatial or temporal locality
•! Several code restructuring techniques to improve both
–! Compiler must know that restructuring preserves semantics
•! Loop interchange: spatial locality
•! Example: row-major matrix: X[i][j] followed by X[i][j+1]
•! Poor code: X[I][j] followed by X[i+1][j] for (j = 0; j<NCOLS; j++)
for (i = 0; i<NROWS; i++)
sum += X[i][j]; // say
•! Better code
for (i = 0; i<NROWS; i++)
for (j = 0; j<NCOLS; j++)
sum += X[i][j];
CIS371 (Roth/Martin): Caches 54
Software Restructuring: Data
•! Loop blocking: temporal locality •! Poor code
for (k=0; k<NITERATIONS; k++)
for (i=0; i<NELEMS; i++)
sum += X[i]; // say
•! Better code
•!Cut array into CACHE_SIZE chunks
•!Run all phases on one chunk, proceed to next chunk
for (i=0; i<NELEMS; i+=CACHE_SIZE)
for (k=0; k<NITERATIONS; k++)
for (ii=0; ii<i+CACHE_SIZE-1; ii++)
sum += X[ii];
–! Assumes you know CACHE_SIZE, do you?
•! Loop fusion: similar, but for multiple consecutive loops
CIS371 (Roth/Martin): Caches 55
Software Restructuring: Code
•! Compiler an layout code for temporal and spatial locality •! If (a) { code1; } else { code2; } code3;
•! But, code2 case never happens (say, error condition)
•! Fewer taken branches, too
•! Intra-procedure, inter-procedure
Better locality
Better locality
CIS371 (Roth/Martin): Caches 56
Prefetching
•! Prefetching: put blocks in cache proactively/speculatively
•! Key: anticipate upcoming miss addresses accurately
•!Can do in software or hardware
•! Simple example: next block prefetching
•!Miss on address X " anticipate miss on X+block-size
+!Works for insns: sequential execution
+!Works for data: arrays
•! Timeliness: initiate prefetches sufficiently in advance
•! Coverage: prefetch for as many misses as possible
•! Accuracy: don’t pollute with unnecessary data
•! It evicts useful data
I$/D$
L2
prefetch logic
CIS371 (Roth/Martin): Caches 57
Software Prefetching
•! Use a special “prefetch” instruction
•! Tells the hardware to bring in data, doesn’t actually read it
•! Just a hint
•! Inserted by programmer or compiler
•! Example
for (i = 0; i<NROWS; i++)
for (j = 0; j<NCOLS; j+=BLOCK_SIZE) {
prefetch(&X[i][j]+BLOCK_SIZE);
for (jj=j; jj<j+BLOCK_SIZE-1; jj++)
sum += x[i][jj];
}
•! Multiple prefetches bring multiple blocks in parallel
•! Using lockup-free caches
•! “Memory-level” parallelism
CIS371 (Roth/Martin): Caches 58
Hardware Prefetching
•! What to prefetch?
•! Stride-based sequential prefetching
•!Can also do N blocks ahead to hide more latency
+!Simple, works for sequential things: insns, array data
+!Works better than doubling the block size
•! Address-prediction
•!Needed for non-sequential data: lists, trees, etc.
•!Use a hardware table to detect strides, common patterns
•! When to prefetch?
•! On every reference?
•! On every miss?
CIS371 (Roth/Martin): Caches 59
[This slide intentionally blank]
CIS371 (Roth/Martin): Caches 60
[This slide intentionally blank]
Extra Cache Examples
•! 4-bit address (16-byte memory)
•! 8-byte cache
CIS371 (Roth/Martin): Caches 61
4-bit Address, 8B Cache, 2B Blocks 0000 A
0001 B
0010 C
0011 D
0100 E
0101 F
0110 G
0111 H
1000 I
1001 J
1010 K
1011 L
1100 M
1101 N
1110 P
1111 Q
CIS371 (Roth/Martin): Caches 62
Data
Set Tag 0 1
00
01
10
11
1 bit tag (1 bit) index (2 bits) Main memory
4-bit Address, 8B Cache, 2B Blocks 0000 A
0001 B
0010 C
0011 D
0100 E
0101 F
0110 G
0111 H
1000 I
1001 J
1010 K
1011 L
1100 M
1101 N
1110 P
1111 Q
CIS371 (Roth/Martin): Caches 63
Data
Set Tag 0 1
00 0 A B
01 0 C D
10 0 E F
11 0 G H
1 bit tag (1 bit) index (2 bits) Main memory
Load: 1110 Miss
4-bit Address, 8B Cache, 2B Blocks 0000 A
0001 B
0010 C
0011 D
0100 E
0101 F
0110 G
0111 H
1000 I
1001 J
1010 K
1011 L
1100 M
1101 N
1110 P
1111 Q
CIS371 (Roth/Martin): Caches 64
Data
Set Tag 0 1
00 0 A B
01 0 C D
10 0 E F
11 1 P Q
1 bit tag (1 bit) index (2 bits) Main memory
Load: 1110 Miss
Larger Block Size
2-byte to 4-byte blocks
CIS371 (Roth/Martin): Caches 65 CIS371 (Roth/Martin): Caches 66
[This slide intentionally blank]
4-bit Address, 8B Cache, 4B Blocks 0000 A
0001 B
0010 C
0011 D
0100 E
0101 F
0110 G
0111 H
1000 I
1001 J
1010 K
1011 L
1100 M
1101 N
1110 P
1111 Q
CIS371 (Roth/Martin): Caches 67
Data
Set Tag 00 01 10 11
0 0 A B C D
1 0 E F G H
2 bit tag (1 bit) index (1 bits) Main memory
Load: 1110 Miss
4-bit Address, 8B Cache, 4B Blocks 0000 A
0001 B
0010 C
0011 D
0100 E
0101 F
0110 G
0111 H
1000 I
1001 J
1010 K
1011 L
1100 M
1101 N
1110 P
1111 Q
CIS371 (Roth/Martin): Caches 68
Data
Set Tag 00 01 10 11
0 0 A B C D
1 1 M N P Q
2 bit tag (1 bit) index (1 bits) Main memory
Load: 1110 Miss
Set Associative Cache
•! Direct mapped cache to 2-way set associative cache
CIS371 (Roth/Martin): Caches 69
4-bit Address, 8B Cache, 2B Blocks, 2-way 0000 A
0001 B
0010 C
0011 D
0100 E
0101 F
0110 G
0111 H
1000 I
1001 J
1010 K
1011 L
1100 M
1101 N
1110 P
1111 Q
CIS371 (Roth/Martin): Caches 70
Way 0 LRU Way 1
Data Data
Set Tag 0 1 Tag 0 1
0
1
1 bit tag (2 bit) index (1 bits) Main memory
4-bit Address, 8B Cache, 2B Blocks, 2-way 0000 A
0001 B
0010 C
0011 D
0100 E
0101 F
0110 G
0111 H
1000 I
1001 J
1010 K
1011 L
1100 M
1101 N
1110 P
1111 Q
CIS371 (Roth/Martin): Caches 71
Way 0 LRU Way 1
Data Data
Set Tag 0 1 Tag 0 1
0 00 A B 0 01 E F
1 00 C D 1 01 G H
1 bit tag (2 bit) index (1 bits) Main memory
Load: 1110 Miss
4-bit Address, 8B Cache, 2B Blocks, 2-way 0000 A
0001 B
0010 C
0011 D
0100 E
0101 F
0110 G
0111 H
1000 I
1001 J
1010 K
1011 L
1100 M
1101 N
1110 P
1111 Q
CIS371 (Roth/Martin): Caches 72
Way 0 LRU Way 1
Data Data
Set Tag 0 1 Tag 0 1
0 00 A B 0 01 E F
1 00 C D 0 11 P Q
1 bit tag (2 bit) index (1 bits) Main memory
Load: 1110 Miss
LRU updated on each access (not just misses)
CIS371 (Roth/Martin): Caches 73
Write Issues
•! So far we have looked at reading from cache
•! Instruction fetches, loads
•! What about writing into cache •! Stores, not an issue for instruction caches (why they are simpler)
•! Several new issues •! Tag/data access
•! Write-through vs. write-back
•! Write-allocate vs. write-not-allocate
•! Hiding write miss latency
CIS371 (Roth/Martin): Caches 74
Tag/Data Access
•! Reads: read tag and data in parallel
•! Tag mis-match " data is garbage (OK, stall until good data arrives)
•! Writes: read tag, write data in parallel?
•! Tag mis-match " clobbered data (oops)
•! For associative caches, which way was written into?
•! Writes are a pipelined two step (multi-cycle) process •! Step 1: match tag
•! Step 2: write to matching way
•! Bypass (with address check) to avoid load stalls
•! May introduce structural hazards
CIS371 (Roth/Martin): Caches 75
Write Propagation
•! When to propagate new value to (lower level) memory?
•! Option #1: Write-through: immediately •! On hit, update cache
•! Immediately send the write to the next level
•! Option #2: Write-back: when block is replaced •! Requires additional “dirty” bit per block
•!Replace clean block: no extra traffic
•!Replace dirty block: extra “writeback” of block
+!Writeback-buffer: keep it off critical path of miss
•! Step#1: Send “fill” request to next-level
•! Step#2: While waiting, write dirty block to buffer
•! Step#3: When new blocks arrives, put it into cache
•! Step#4: Write buffer contents to next-level
2 1
4
$
Next-level-$
WBB
3
CIS371 (Roth/Martin): Caches 76
Write Propagation Comparison
•! Write-through –! Requires additional bus bandwidth
•!Consider repeated write hits
–! Next level must handle small writes (1, 2, 4, 8-bytes)
+!No need for valid bits in cache
+!No need to handle “writeback” operations
•! Simplifies miss handling (no WBB)
•! Sometimes used for L1 caches (for example, by IBM)
•! Write-back +!Key advantage: uses less bandwidth
•! Reverse of other pros/cons above
•! Used by Intel and AMD
•! Second-level and beyond are generally write-back caches
CIS371 (Roth/Martin): Caches 77
Write Miss Handling
•! How is a write miss actually handled?
•! Write-allocate: fill block from next level, then write into it +!Decreases read misses (next read to block will hit)
–! Requires additional bandwidth
•! Commonly used (especially with write-back caches)
•! Write-non-allocate: just write to next level, no allocate
–! Potentially more read misses
+!Uses less bandwidth
•! Use with write-through
CIS371 (Roth/Martin): Caches 78
Write Misses and Write Buffers
•! Read miss? •! Load can’t go on without the data, it must stall
•! Write miss? •! Technically, no instruction is waiting for data, why stall?
•! Write buffer: a small buffer •! Stores put address/value to write buffer, keep going
•! Write buffer writes stores to D$ in the background
•! Loads must search write buffer (in addition to D$)
+!Eliminates stalls on write misses (mostly) –! Creates some problems (later)
•! Write buffer vs. writeback-buffer •! Write buffer: “in front” of D$, for hiding store misses
•! Writeback buffer: “behind” D$, for hiding writebacks
Cache
Next-level cache
WBB
WB
Processor
Write Buffer Examples
•! Example #1:
•! Store “1” into address A
•!Miss in cache, put in store buffer (initiate miss)
•! Load from address B
•!Hit in cache, read value from cache
•! Wait for miss to fill, write a “1” to A when done
•! Example #2: •! Store “1” into address A
•!Miss, put in store buffer (initiate miss)
•! Load from address A
•!Miss in cache, but do we stall? Don’t need to stall
•! Just bypass load value from the store buffer
CIS371 (Roth/Martin): Caches 79
Write Buffer Examples
•! Example #3:
•! Store byte value “1” into address A
•!Miss in cache, put in store buffer (initiate miss)
•! Load word from address A, A+1, A+2, A+3
•!Hit in cache, read value from where?
•!Read both cache and store buffer (byte-by-byte merge)
•! Store buffer holds address, data, and per-byte valid bits
•! Example #4:
•! Store byte value “1” into address A (initiate miss)
•! Store byte value “2” into address B (initiate miss)
•! Store byte value “3” into Address A (???)
•!Can the first and last store share the same entry?
•! What if “B” fills first?
•!Can the second store leave before the first store? CIS371 (Roth/Martin): Caches 80
CIS371 (Roth/Martin): Caches 81
Memory Performance Equation
•! For memory component M •! Access: read or write to M
•! Hit: desired data found in M
•! Miss: desired data not found in M
•!Must get from another (slower) component
•! Fill: action of placing data in M
•! %miss (miss-rate): #misses / #accesses
•! thit: time to read data from (write data to) M
•! tmiss: time to read data into M
•! Performance metric •! tavg: average access time
tavg = thit + %miss * tmiss
CPU
M
thit
tmiss
%miss
CIS371 (Roth/Martin): Caches 82
Hierarchy Performance
tavg
tavg-M1
thit-M1 + (%miss-M1*tmiss-M1)
thit-M1 + (%miss-M1*tavg-M2)
thit-M1 + (%miss-M1*(thit-M2 + (%miss-M2*tmiss-
M2)))
thit-M1 + (%miss-M1* (thit-M2 + (%miss-M2*tavg-
M3)))
…
tmiss-M3 = tavg-M4
CPU
M1
M2
M3
M4
tmiss-M2 = tavg-M3
tmiss-M1 = tavg-M2
tavg = tavg-M1
CIS371 (Roth/Martin): Caches 83
Local vs Global Miss Rates
•! Local hit/miss rate:
•! Percent of references to cache hit (e.g, 90%)
•! Local miss rate is (100% - local hit rate), (e.g., 10%)
•! Global hit/miss rate:
•! Misses per instruction (1 miss per 30 instructions)
•! Instructions per miss (3% of instructions miss)
•! Above assumes loads/stores are 1 in 3 instructions
•! Consider second-level cache hit rate
•! L1: 2 misses per 100 instructions
•! L2: 1 miss per 100 instructions
•! L2 “local miss rate” -> 50%
CIS371 (Roth/Martin): Caches 84
Performance Calculation I
•! In a pipelined processor, I$/D$ thit is “built in” (effectively 0)
•! Parameters
•! Base pipeline CPI = 1
•! Instruction mix: 30% loads/stores
•! I$: %miss = 2%, tmiss = 10 cycles
•! D$: %miss = 10%, tmiss = 10 cycles
•! What is new CPI? •! CPII$ = %missI$*tmiss = 0.02*10 cycles = 0.2 cycle
•! CPID$ = %memory*%missD$*tmissD$ = 0.30*0.10*10 cycles = 0.3 cycle
•! CPInew = CPI + CPII$ + CPID$ = 1+0.2+0.3= 1.5
CIS371 (Roth/Martin): Caches 85
Performance Calculation II
•! Parameters
•! Reference stream: all loads
•! D$: thit = 1ns, %miss = 5%
•! L2: thit = 10ns, %miss = 20%
•! Main memory: thit = 50ns
•! What is tavgD$ without an L2?
•! tmissD$ = thitM
•! tavgD$ = thitD$ + %missD$*thitM = 1ns+(0.05*50ns) = 3.5ns
•! What is tavgD$ with an L2?
•! tmissD$ = tavgL2
•! tavgL2 = thitL2+%missL2*thitM = 10ns+(0.2*50ns) = 20ns
•! tavgD$ = thitD$ + %missD$*tavgL2 = 1ns+(0.05*20ns) = 2ns
CIS371 (Roth/Martin): Caches 86
Performance Calculation III
•! Memory system parameters •! D$: thit = 1ns, %miss = 10%, 50% dirty, writeback-buffer, write-buffer
•! Main memory: thit = 50ns
•! 32-byte block size
•! Reference stream: 20% stores, 80% loads
•! What is tavgD$? •! Write-buffer " hides store misses latency
•! Writeback-buffer " hides dirty writeback latency
•! tmissD$ = thitM
•! tavgD$ = thitD$ + %loads * %missD$*thitM = 1ns+(0.8*0.10*50ns) = 5ns
CIS371 (Roth/Martin): Caches 87
Bandwidth Calculation
•! Memory system parameters •! D$: thit = 1ns, %miss = 10%, 50% dirty, writeback-buffer, write-buffer
•! Main memory: thit = 50ns
•! 32-byte block size
•! Reference stream: 20% stores, 80% loads
•! What is the average bytes transferred per miss? •! All misses: 32-byte blocks
•! Dirty evictions: 50% of the time * 32-byte block
•! 48B per miss
•! Average bytes transfer per memory operation? •! 10% of memory operations miss, so 4.8B per memory operation
CIS371 (Roth/Martin): Caches 88
Designing a Cache Hierarchy
•! For any memory component: thit vs. %miss tradeoff
•! Upper components (I$, D$) emphasize low thit
•! Frequent access " thit important
•! tmiss is not bad " %miss less important
•! Low capacity/associativity (to reduce thit)
•! Small-medium block-size (to reduce conflicts)
•! Moving down (L2, L3) emphasis turns to %miss •! Infrequent access " thit less important
•! tmiss is bad " %miss important
•! High capacity/associativity/block size (to reduce %miss)
CIS371 (Roth/Martin): Caches 89
Memory Hierarchy Parameters
•! Some other design parameters •! Split vs. unified insns/data
•! Inclusion vs. exclusion vs. nothing
•! On-chip, off-chip, or partially on-chip?
•! SRAM or embedded DRAM?
Parameter I$/D$ L2 L3 Main Memory
thit 2ns 10ns 30ns 100ns
tmiss 10ns 30ns 100ns 10ms (10M ns)
Capacity 8KB–64KB 256KB–8MB 2–16MB 1-4GBs
Block size 16B–32B 32B–128B 32B-256B NA
Associativity 1–4 4–16 4-16 NA
CIS371 (Roth/Martin): Caches 90
Split vs. Unified Caches
•! Split I$/D$: insns and data in different caches
•! To minimize structural hazards and thit
•! Larger unified I$/D$ would be slow, 2nd port even slower
•! Optimize I$ for wide output (superscalar), no writes
•! Why is 486 I/D$ unified?
•! Unified L2, L3: insns and data together •! To minimize %miss
+!Fewer capacity misses: unused insn capacity can be used for data
–! More conflict misses: insn/data conflicts
•!A much smaller effect in large caches
•! Insn/data structural hazards are rare: simultaneous I$/D$ miss
•! Go even further: unify L2, L3 of multiple cores in a multi-core
CIS371 (Roth/Martin): Caches 91
Hierarchy: Inclusion versus Exclusion
•! Inclusion
•! A block in the L1 is always in the L2
•! Good for write-through L1s (why?)
•! Exclusion
•! Block is either in L1 or L2 (never both)
•! Good if L2 is small relative to L1
•!Example: AMD’s Duron 64KB L1s, 64KB L2
•! Non-inclusion •! No guarantees
CIS371 (Roth/Martin): Caches 92
Summary •! Average access time of a memory component
•! latencyavg = latencyhit + %miss * latencymiss
•! Hard to get low latencyhit and %miss in one structure " hierarchy
•! Memory hierarchy •! Cache (SRAM) " memory (DRAM) " swap (Disk)
•! Smaller, faster, more expensive " bigger, slower, cheaper
•! Cache ABCs (capacity, associativity, block size) •! 3C miss model: compulsory, capacity, conflict
•! Performance optimizations •! %miss: victim buffer, prefetching
•! latencymiss: critical-word-first/early-restart, lockup-free design
•! Write issues •! Write-back vs. write-through/write-allocate vs. write-no-allocate