CECS 440 Pipelining.1 (c) 2014 – R. W. Allison

CSE 440Computer Architecture

Pipelining

[slides adapted from D. Patterson slides with additional credits to M.J. Irwin]

CECS 440 Pipelining.2 (c) 2014 – R. W. Allison

Actual time saved

Review: Single Cycle vs. Multiple Cycle Timing

Clk Cycle 1

Multiple Cycle Implementation:

IFetch Dec Exec Mem WB

Cycle 2 Cycle 3 Cycle 4 Cycle 5 Cycle 6 Cycle 7 Cycle 8 Cycle 9Cycle 10

IFetch Dec Exec Mem

lw sw

IFetch

R-type

Clk

Single Cycle Implementation:

lw sw Waste

Cycle 1 Cycle 2

multicycle clock slower than 1/5th of single cycle clock due to stage register overhead

CECS 440 Pipelining.3 (c) 2014 – R. W. Allison

How Can We Make It Even Faster?

Split the multiple instruction cycle into smaller and smaller steps

There is a point of diminishing returns where as much time is spent loading the state registers as doing the work

Start fetching and executing the next instruction before the current one has completed

Pipelining – (all?) modern processors are pipelined for performance

Fetch (and execute) more than one instruction at a time (out-of-order superscalar and VLIW)

Fetch (and execute) instructions from more than one instruction stream (multithreading [hyperthreading])

CECS 440 Pipelining.4 (c) 2014 – R. W. Allison

A Pipelined MIPS Processor

Start the next instruction before the current one has completed

improves throughput - total amount of work done in a given time

instruction latency (execution time, delay time, response time, i.e. the time from the start of an instruction to its completion) is not reduced

Cycle 1 Cycle 2 Cycle 3 Cycle 4 Cycle 5

IFetch Dec Exec Mem WBlw

Cycle 7Cycle 6 Cycle 8

sw IFetch Dec Exec Mem WB

R-type IFetch Dec Exec Mem WB

- clock cycle (pipeline stage time) is limited by the slowest stage- for some instructions, some stages are wasted cycles

CECS 440 Pipelining.5 (c) 2014 – R. W. Allison

Actual time saved

Actual time saved

Single Cycle, Multiple Cycle, vs. Pipeline

Multiple Cycle Implementation:

Clk

Cycle 1

IFetch Dec Exec Mem WB

Cycle 2 Cycle 3 Cycle 4 Cycle 5 Cycle 6 Cycle 7 Cycle 8 Cycle 9Cycle 10

IFetch Dec Exec Mem

lw sw

IFetch

R-type

lw IFetch Dec Exec Mem WB

Pipeline Implementation:

IFetch Dec Exec Mem WBsw

IFetch Dec Exec Mem WBR-type

Clk

Single Cycle Implementation:

lw sw Waste

Cycle 1 Cycle 2

CECS 440 Pipelining.6 (c) 2014 – R. W. Allison

Write Data

Read Addr 1

Read Addr 2

Write Addr

Register

File

Read Data 1

Read Data 2

16 32

De

c/E

xe

c

ID:Dec

SignExtend

MIPS Pipeline Datapath Modifications What do we need to add/modify in our MIPS datapath?

State registers between each pipeline stage to isolate them

DataMemory

Address

Write Data

ReadData

MEM:MemAccess

Me

m/W

BReadAddress

InstructionMemory

Add

PC

4

IFe

tch

/De

c

IF:IFetch

System Clock

WB:WriteBack

Ex

ec

/Me

m

EX:Execute

ALU

Shiftleft 2

Add

CECS 440 Pipelining.7 (c) 2014 – R. W. Allison

MIPS Pipeline Control Path Modifications

All control signals can be determined during Decode and held in the state registers between pipeline stages

ReadAddress

InstructionMemory

Add

PC

4

Write Data

Read Addr 1

Read Addr 2

Write Addr

Register

File

Read Data 1

Read Data 2

16 32

ALU

Shiftleft 2

Add

DataMemory

Address

Write Data

ReadData

IF/ID

SignExtend

ID/EXEX/MEM

MEM/WB

Control

CECS 440 Pipelining.8 (c) 2014 – R. W. Allison

Pass control signals along just like the data

Pipeline Control

Execution/Address Calculation stage control lines

Memory access stage control lines

Write-back stage control

lines

InstructionReg Dst

ALU Op1

ALU Op0

ALU Src Branch

Mem Read

Mem Write

Reg write

Mem to Reg

R-format 1 1 0 0 0 0 0 1 0lw 0 0 0 1 0 1 0 1 1sw X 0 0 1 0 0 1 0 Xbeq X 0 1 0 1 0 0 0 X

Control

EX

M

WB

M

WB

WB

IF/ID ID/EX EX/MEM MEM/WB

Instruction

CECS 440 Pipelining.9 (c) 2014 – R. W. Allison

Pipelining the MIPS ISA

What makes it easy all instructions are the same length (32 bits)

can fetch in the 1st stage and decode in the 2nd stage few instruction formats (three) with symmetry across

formatscan begin reading register file in 2nd stage

memory operations can occur only in loads and storescan use the execute stage to calculate memory

addresses each MIPS instruction writes at most one result (i.e.,

changes the machine state) and does so near the end of the pipeline (MEM and WB)

What makes it hard structural hazards: what if we had only one memory? control hazards: what about branches? data hazards: what if an instruction’s input operands

depend on the output of a previous instruction?

CECS 440 Pipelining.10 (c) 2014 – R. W. Allison

Graphically Representing MIPS Pipeline

Can help with answering questions like: How many cycles does it take to execute this code? What is the ALU doing during cycle 4? Is there a hazard, why does it occur, and how can it

be fixed?ALUIM Reg DM Reg

CECS 440 Pipelining.11 (c) 2014 – R. W. Allison

Why Pipeline? For Performance!

Instr.

Order

Time (clock cycles)

Inst 0

Inst 1

Inst 2

Inst 4

Inst 3

Once the pipeline is full, one instruction is completed every

cycle so complete an instruction every

cycle (CPI = 1)

Time to fill the pipeline

ALUIM Reg DM Reg

ALUIM Reg DM Reg

ALUIM Reg DM Reg

ALUIM Reg DM Reg

ALUIM Reg DM Reg

CECS 440 Pipelining.12 (c) 2014 – R. W. Allison

Can Pipelining Get Us Into Trouble?

Yes: Pipeline Hazards structural hazards: attempt to use the same resource

by two different instructions at the same time data hazards: attempt to use data before it is ready

An instruction’s source operand(s) are produced by a prior instruction still in the pipeline

control hazards: attempt to make a decision about program control flow before the condition has been evaluated and the new PC target address calculated

branch and jump instructions, exceptions

Can always resolve hazards by waiting pipeline control must detect the hazard and take action to resolve hazards

CECS 440 Pipelining.13 (c) 2014 – R. W. Allison

ALUIM Reg DM Reg

ALUIM Reg DM RegInstr.

Order

Time (clock cycles)

lw

Inst 1

Inst 2

Inst 4

Inst 3

A Single Memory Would Be a Structural Hazard

Reading data from memory

Reading instruction from memory

Can fix with separate instruction and data memories

ALUIM Reg DM Reg

ALUIM Reg DM Reg

ALUIM Reg DM Reg

CECS 440 Pipelining.15 (c) 2014 – R. W. Allison

How About Register File Access?

Instr.

Order

Time (clock cycles)

Inst 1

Inst 2

Fix register file access hazard by doing

writes in the first half of the cycle and reads

in the second half

add $1,

add $2,$1,

clock edge that controls register writing

clock edge that controls loading of pipeline state registers

ALUIM Reg DM Reg

ALUIM Reg DM Reg

ALUIM Reg DM Reg

ALUIM Reg DM Reg

CECS 440 Pipelining.17 (c) 2014 – R. W. Allison

Register Usage Can Cause Data Hazards

add $1,

sub $4,$1,$5

and $6,$1,$7

xor $4,$1,$5

or $8,$1,$9

Read before write data hazard

ALUIM Reg DM RegALUIM Reg DM Reg

ALUIM Reg DM Reg

ALUIM Reg DM Reg

ALUIM Reg DM Reg

Dependencies going “backward in time” cause hazards

CECS 440 Pipelining.19 (c) 2014 – R. W. Allison

Loads Can Cause Data Hazards

Instr.

Order

lw $1,4($2)

sub $4,$1,$5

and $6,$1,$7

xor $4,$1,$5

or $8,$1,$9

Load-use data hazard

ALUIM Reg DM RegALUIM Reg DM Reg

ALUIM Reg DM Reg

ALUIM Reg DM Reg

ALUIM Reg DM Reg

Dependencies going “backward in time” cause hazards

CECS 440 Pipelining.20 (c) 2014 – R. W. Allison

noop (stall)

noop (stall)

One Way to “Fix” a Data Hazard

Instr.

Order

add $1,

sub $4,$1,$5

and $6,$1,$7

ALUIM Reg DM Reg

ALUIM Reg DM Reg

Can fix data hazard by

waiting – stall

ALUIM Reg DM Reg

CECS 440 Pipelining.22 (c) 2014 – R. W. Allison

Another Way to “Fix” a Data Hazard

Fix data hazards by forwarding

results as soon as they are available to where they are

needed

Instr.

Order

add $1,

sub $4,$1,$5

and $6,$1,$7

xor $4,$1,$5

or $8,$1,$9

ALUIM Reg DM Reg

ALUIM Reg DM Reg

ALUIM Reg DM Reg

ALUIM Reg DM Reg

ALUIM Reg DM Reg

CECS 440 Pipelining.23 (c) 2014 – R. W. Allison

How to Implement Forwarding

The main idea (some details not shown)

Rs, Rt from “Instr”

Rd from “Instr-1”

Rd from “Instr-2”

Forwarding Unit

must detectdata hazards

Forward Source

0 0 ID/EX 1 0 EX/MEM 0 1 MEM/WB

CECS 440 Pipelining.24 (c) 2014 – R. W. Allison

Forwarding with Load-use Data Hazards

Will we still need one stall cycle even with forwarding?

Instr.

Order

lw $1,4($2)

sub $4,$1,$5

and $6,$1,$7

xor $4,$1,$5

or $8,$1,$9ALUIM Reg DM Reg

ALUIM Reg DM Reg

ALUIM Reg DM Reg

ALUIM Reg DM Reg

ALUIM Reg DM Reg

CECS 440 Pipelining.25 (c) 2014 – R. W. Allison

Hazard Detection Unit

Stalling by creating a no-operation (noop) instructionConsider this sequence:

0x100 lw $t1, 24[$s1]0x104 add $t3, $t1, $t20x108 sub $t4, $t1, $t2

PC=0x108.Fetching the

“sub” instruction

EA calc for instruction at 0x100 (lw) is being done in this stage; thus, MemRead=1.

Reg

cod

es a

nd

op

era

nd

s f

or

“lw

”

Getting $t2 and $t1 for the “add”

instruction at 0x104

ad

d $

t3,

$t2

, $

t1

Hazard detected, so (1) the mux in ID stage selects all “0’s” (i.e nop) (2) PCWrite=0, so PC not change, remaining at 0x108(3) IF/Dwrite=0, so the add instruction remains.Thus, the sub instruction will be

fetched after “stalling” for one clock cycle

Rt = $t1Rs = $t1Rt = $t2

MemRead=1

Hazard Detection:If ( Ex.MemRead & (Ex.Rt==ID.rs | Ex.Rt == ID.Rt) )

CECS 440 Pipelining.26 (c) 2014 – R. W. Allison

Control Hazards

When the flow of instruction addresses is not sequential (i.e., PC = PC + 4)

Conditional branches (beq, bne) Unconditional branches (j, jal, jr) Exceptions

Possible “solutions” Stall (impacts performance) Move branch decision point as early in the pipeline as

possible, thereby reducing the number of stall cycles Delay decision (requires compiler support) Predict and hope for the best !

Control hazards occur less frequently than data hazards, but there is nothing as effective against control hazards as forwarding is for data hazards

CECS 440 Pipelining.27 (c) 2014 – R. W. Allison

flush

Jumps Incur One Stall

Instr.

Order

j

j target

Fortunately, jumps are very infrequent – only 3% of the SPECint instruction mix

Jumps not decoded until ID, so one flush is needed To flush, set IF.Flush=1 to “zero” the instruction field of

the IF/ID pipeline register (turning it into a noop)

Fix jump hazard by waiting –

flush

ALUIM Reg DM Reg

ALUIM Reg DM Reg

ALUIM Reg DM Reg

CECS 440 Pipelining.28 (c) 2014 – R. W. Allison

Review: MIPS Pipeline Control & Datapath

All control signals can be determined during Decode and held in the state registers between pipeline stages

ReadAddress

InstructionMemory

Add

PC

4

Write Data

Read Addr 1

Read Addr 2

Write Addr

Register

File

Read Data 1

Read Data 2

16 32

ALU

Shiftleft 2

Add

DataMemory

Address

Write Data

ReadData

IF/ID

SignExtend

ID/EXEX/MEM

MEM/WB

Control

CECS 440 Pipelining.29 (c) 2014 – R. W. Allison

ALUIM Reg DM Reg

ALUIM Reg DM Reg

Branches Cause Control Hazards

Instr.

Order

lw

Inst 4

Inst 3

Dependencies backward in time cause hazards

ALUIM Reg DM Reg

ALUIM Reg DM Reg

beq

CECS 440 Pipelining.30 (c) 2014 – R. W. Allison

flush

flush

flush

One Way to “Fix” a Branch Control Hazard

Instr.

Order

beq

beq target

ALU

Inst 3 IM Reg DM

Fix branch hazard by waiting –

flush – but affects CPI

ALUIM Reg DM Reg

ALUIM Reg DM Reg

ALUIM Reg DM Reg

ALUIM Reg DM Reg

ALUIM Reg DM Reg

CECS 440 Pipelining.31 (c) 2014 – R. W. Allison

flush

Another Way to “Fix” a Branch Control Hazard

Instr.

Order

beq

beq target

Inst 3

Fix branch hazard by waiting –

flush

Move branch decision hardware back to as early in the pipeline as possible – i.e., during the decode cycle

ALUIM Reg DM Reg

ALUIM Reg DM RegALUIM Reg DM Reg

ALUIM Reg DM Reg

CECS 440 Pipelining.32 (c) 2014 – R. W. Allison

8 sub $4,$1,$5flush

Yet Another Way to “Fix” a Control Hazard

4 beq $1,$2,2Instr.

Order

ALUIM Reg DM Reg

16 and $6,$1,$7

20 or r8,$1,$9

ALUIM Reg DM Reg

ALUIM Reg DM Reg

ALUIM Reg DM Reg

To flush, set IF.Flush=1 to zero the instruction field of the IF/ID pipeline register (turning it into a noop)

“Predict branches are always not taken – and take corrective action when wrong (i.e., taken)

CECS 440 Pipelining.33 (c) 2014 – R. W. Allison

Two “Types” of Stalls

Noop instruction inserted between two instructions in the pipeline (e.g., load-use hazards)

Keep the instructions earlier in the pipeline (later in the code) from progressing down the pipeline for a cycle (“bounce” them in place with write control signals)

Insert noop instruction by zeroing control bits in the pipeline register at the appropriate stage

Let the instructions later in the pipeline (earlier in the code) progress normally down the pipeline

Flushes (or instruction squashing) where an instruction in the pipeline is replaced with a noop instruction (as done for instructions located sequentially after j and beq instructions)

Zero the control bits for the instruction to be flushed

CECS 440 Pipelining.34 (c) 2014 – R. W. Allison

Many Other Pipeline Structures Are Possible

What about the (slow) multiply operation? Make the clock twice as slow or … let it take two cycles (since it doesn’t use the DM

stage)

ALUIM Reg DM Reg

MUL

ALUIM Reg DM1 RegDM2

What if the data memory access is twice as slow as the instruction memory?

make the clock twice as slow or … let data memory access take two cycles (and keep the

same clock rate)

CECS 440 Pipelining.35 (c) 2014 – R. W. Allison

Pipelining Summary

All modern day processors use pipelining Pipelining doesn’t help latency of single task, it helps

throughput of entire workload Potential speedup: a really fast clock cycle and able

to complete one instruction every clock cycle (CPI) Pipeline rate limited by slowest pipeline stage

Unbalanced pipe stages makes for inefficiencies The time to “fill” pipeline and time to “drain” it can

impact speedup for deep pipelines and short code runs

Must detect and resolve hazards Stalling negatively affects CPI (makes CPI less than the

ideal of 1)