I/O Interfaces, A Little Queueing Theory RAID

ENGS 116 Lecture 18 1

I/O Interfaces,A Little Queueing Theory

RAID

Vincent H. Berk

November 11, 2005

Reading for Today: Sections 7.1 – 7.4

Reading for Monday: Sections 7.5 – 7.9, 7.14

Homework for Friday Nov 18: 5.4, 5.17,

6.4, 6.10, 7.3, 7.21, 8.9/8.10, 8.17


Common Bus Standards

• ISA

• PCI

• AGP

• PCMCIA

• USB

• FireWire/IEEE 1394

• IDE

• SCSI

3ENGS 116 Lecture 18

Programmed I/O (Polling)

CPU

IOC

Device

Memory

Is thedata

ready?

readdata

storedata

yesno

done? no

yes

Busy wait loopnot an efficientway to use the CPUunless the deviceis very fast!

However, checks for I/O completion can be dispersed among computationallyintensive code.


User program progress only halted during actual transfer

1000 transfers at 1 ms each:1000 interrupts @ 2 µ sec per interrupt1000 interrupt service @ 98 µ sec each = 0.1 CPU seconds

Device transfer rate = 10 MBytes/sec 0.1 x 10-6 sec/byte 0.1 µsec/byte 1000 bytes = 100 µsec 1000 transfers 100 µsecs = 100 ms = 0.1 CPU seconds

Still far from device transfer rate! 1/2 time in interrupt overhead

Interrupt-Driven Data TransferCPU

IOC

device

Memory

addsubandornop

readstore...rti

memory

userprogram

(1) I/Ointerrupt

(2) save PC

(3) interruptservice addr

interruptserviceroutine(4)


Direct Memory Access (DMA)Time to do 1000 transfers at 1 msec each:

1 DMA set-up sequence @ 50 µ sec1 interrupt @ 2 µ sec1 interrupt service sequence @ 48 µ sec

.0001 second of CPU time

CPU sends a starting address, direction, and length count to DMAC. Then issues "start".

DMAC provides handshake signals for peripheralcontroller and memory addresses and handshakesignals for memory.

CPU

IOC

device

Memory DMAC

0ROM

RAM

Peripherals

DMACn

Memory Mapped I/O


Input/Output Processors

CPU IOP

Mem

D1

D2

Dn

. . .main memory

bus

I/Obus

CPU

IOP

issues instruction to IOP

interrupts when done(1)

memory

(2)

(3)

(4)

Device to/from memorytransfers are controlledby the IOP directly.

IOP steals memory cycles.

OP Device Address

target devicewhere commands are

looks in memory for commands

OP Addr Cnt Other

whatto do

whereto putdata

howmuch

specialrequests


Summary

• Disk industry growing rapidly, improves bandwidth and areal density

• Time = queue + controller + seek + rotate + transfer

• Advertised average seek time much greater than average seek time in practice

• Response time vs. bandwidth tradeoffs

• Processor interface: today peripheral processors, DMA, I/O bus, interrupts


A Little Queueing Theory

• More interested in long-term, steady-state behavior

Arrivals = Departures

• Little’s Law: mean number of tasks in system

= arrival rate mean response time

– Observed by many, Little was first to prove

– Applies to any system in equilibrium as long as nothing inside the system (black box) is creating or destroying tasks

• Queueing models assume state of equilibrium:

input rate = output rate

Arrivals

System

Departures



• Avg. arrival rate

• Avg. service rate • Avg. number in system N

• Avg. system time per customer T = avg. waiting time + avg. service time

• Little’s Law: N = T

• Service utilization = /

Departures

ServerQueue

Arrivals



• Server spends a variable amount of time with customers

– Service distribution characterized by mean, variance, squared coefficient of variance (C)

– Squared coefficient of variance: C = variance/mean2, unitless measure

• Exponential distribution: C = 1; most short relative to average

• Hypoexponential distribution: C < 1; most close to average

• Hyperexponential distribution: C > 1; most further from average

• Disk response times: C ≈ 1.5, but usually pick C = 1 for simplicity


Average Residual Service Time

How long does a new customer wait for the current customer to finish service?

Average residual time =

If C = 0, average residual service time = 1/2 mean service time

1

2Mean service time 1 C


Average Wait Time in Queue

• If something at server, it takes average residual service time to complete

• Probability that server is busy is • All customers in line must complete, each averaging Tservice

• If exponential distribution, C = 1 and

Tq Tservice 1 C

2 1

Tq Tservice

1


M/M/1 and M/G/1

• Assumptions so far:

– System in equilibrium

– Time between two successive arrivals in line are random

– Server can start on next customer immediately after prior customer finishes

– No limit to the queue, FIFO service

– All customers in line must complete, each averaging Tservice

• Exponential distribution (C = 1) is “memoryless” or Markovian, denoted by M

• Queueing system with exponential arrivals, exponential service times, and 1 server: M/M/1

• General distribution is denoted by G: can have M/G/1 queue


An Example

Processor sends 10 8-KB disk I/Os per second, requests and service times exponentially distributed, avg. disk service = 20 ms.


Another Example

Processor sends 20 8-KB disk I/Os per second, requests and service times exponentially distributed, avg. disk service = 12 ms.


Yet Another Example

Processor sends 10 8-KB disk I/Os per second, C = 1.5, avg. disk service = 20 ms.


Manufacturing Advantages of Disk Arrays

3.5”

Disk Array: 1 disk design

14”10”5.25”3.5”

Conventional: 4 disk designs

Low End High End

Disk Product Families14”


Replace Small # of Large Disks withLarge # of Small Disks!

Data Capacity

Volume

Power

Data Rate

I/O Rate

MTBF

Cost

IBM 3390 (K)

20 GBytes

97 cu. ft.

3 KW

15 MB/s

600 I/Os/s

250 KHrs

$250K

IBM 3.5" 0061

320 MBytes

0.1 cu. ft.

11 W

1.5 MB/s

55 I/Os/s

50 KHrs

$2K

x70

23 GBytes

11 cu. ft.

1 KW

120 MB/s

3900 IOs/s

??? Hrs

$150K

Disk Arrays have potential for

large data and I/O rates

high MB per cu. ft., high MB per KW

awful reliability


Array Reliability

• Reliability of N disks = Reliability of 1 Disk ÷ N

50,000 Hours ÷ 70 disks = 700 hours

Disk system MTTF: Drops from 6 years to 1 month!

• Arrays without redundancy too unreliable to be useful!

Hot spares support reconstruction in parallel with access: very high media availability can be achieved


Mirroring/Shadowing (high capacity cost)

Horizontal Hamming Codes (overkill)

Parity & Reed-Solomon Codes

Failure Prediction (no capacity overhead!)VaxSimPlus — Technique is controversial

Techniques:

Redundant Arrays of Disks

• Files are "striped" across multiple spindles

• Redundancy yields high data availabilityDisks will failContents reconstructed from data redundantly stored in the

array Capacity penalty to store it Bandwidth penalty to update


Redundant Arrays of Disks (RAID) Techniques

• Disk Mirroring, ShadowingEach disk is fully duplicated onto its "shadow" Logical write = two physical writes100% capacity overhead

• Parity Data Bandwidth ArrayParity computed horizontallyLogically a single high data bandwidth disk

• High I/O Rate Parity ArrayInterleaved parity blocksIndependent reads and writesLogical write = 2 reads + 2 writesParity + Reed-Solomon codes

10010011

11001101

10010011

00110010

10010011

10010011

Parity disk


RAID 0: Disk Striping

• Data is distributed over disks

• Improved bandwidth and seek time on read and write• Larger virtual disk

• No redundancy


RAID 1: Disk Mirroring/Shadowing

• Each disk is fully duplicated onto its "shadow" Very high availability can be achieved

• Bandwidth sacrifice on write: Logical write = two physical writes• Half seek time on reads

• Reads may be optimized

• Most expensive solution: 100% capacity overhead

Targeted for high I/O rate, high availability environments

recoverygroup


RAID 3: Parity Disk

P

100100111100110110010011

. . .logical record 1

0010011

11001101

10010011

00110000

Striped physicalrecords

• Parity computed across recovery group to protect against hard disk failures 33% capacity cost for parity in this configuration wider arrays reduce capacity costs, decrease expected availability, increase reconstruction time

• Arms logically synchronized, spindles rotationally synchronized logically a single high capacity, high transfer rate disk

Targeted for high bandwidth applications


RAID 4 & 5: Block-Interleaved Parity and Distributed Block-Interleaved Parity

• Similar to RAID 3, requiring same number of disks.• Parity is computed over blocks and stored in blocks.• RAID 4 places parity on last disk• RAID 5 places parity blocks distributed over all disks:

Advantage: parity block is always accesses on read/writes• Parity is updated by reading the old-block and parity-block, and writing

the new-block and the new parity-block. (2 reads, 2 writes)


Disk access advantage RAID 4/5 over RAID 3


Problems of Disk Arrays: Small Writes

D0 D1 D2 D3 PD0'

+

+

D0' D1 D2 D3 P'

newdata

olddata

old parity

XOR

XOR

(1. Read) (2. Read)

(3. Write) (4. Write)

RAID-5: Small Write Algorithm1 Logical Write = 2 Physical Reads + 2 Physical Writes


I/O Benchmarks: Transaction Processing

• Transaction Processing (TP) (or On-line TP = OLTP)– Changes to a large body of shared information from many terminals, with

the TP system guaranteeing proper behavior on a failure

– If a bank’s computer fails when a customer withdraws money, the TP system would guarantee that the account is debited if the customer received the money and that the account is unchanged if the money was not received

– Airline reservation systems & banks use TP

• Atomic transactions makes this work

• Each transaction 2 to 10 disk I/Os and 5,000 to 20,000 CPU instructions per disk I/O

– Efficient TP software, avoid disks accesses by keeping information in main memory

• Classic metric is Transactions Per Second (TPS) – Under what workload? How is machine configured?


I/O Benchmarks: Transaction Processing

• Early 1980s great interest in OLTP– Expecting demand for high TPS (e.g., ATM machines, credit cards)

– Tandem’s success implied medium range OLTP expands

– Each vendor picked own conditions for TPS claims, reported only CPU times with widely different I/O

– Conflicting claims led to disbelief of all benchmarks chaos

• 1984 Jim Gray of Tandem distributed paper to Tandem employees and 19 in other industries to propose standard benchmark

• Published “A measure of transaction processing power,” Datamation, 1985 by Anonymous et. al

– To indicate that this was effort of large group

– To avoid delays of legal department of each author’s firm

– Author still gets mail at Tandem


I/O Benchmarks: TP by Anon et. al

• Proposed 3 standard tests to characterize commercial OLTP– TP1: OLTP test, DebitCredit, simulates ATMs (TP1)

– Batch sort

– Batch scan

• Debit/Credit:– One type of transaction: 100 bytes each

– Recorded 3 places: account file, branch file, teller file; events recorded in history file (90 days)

• 15% requests for different branches

– Under what conditions, how to report results?


I/O Benchmarks: TP1 by Anon et. al

• DebitCredit Scalability: size of account, branch, teller, history function of throughput TPS Number of ATMs Account-file size

10 1,000 0.1 GB

100 10,000 1.0 GB

1,000 100,000 10.0 GB

10,000 1,000,000 100.0 GB

– Each input TPS 100,000 account records, 10 branches, 100 ATMs

– Accounts must grow since a person is not likely to use the bank more frequently just because the bank has a faster computer!

• Response time: 95% transactions take ≤ 1 second

• Configuration control: just report price (initial purchase price + 5 year maintenance = cost of ownership)

• By publishing, in public domain

Documents

I/O Interfaces, A Little Queueing Theory RAID