Hosted by IP Storage: Best Practices Marc Staimer, President & CDS Dragon Slayer Consulting...

Hosted by

IP Storage: Best Practices

Marc Staimer, President & CDSDragon Slayer Consultingmarcstaimer@earthlink.net503-579-3763

Hosted by

Agenda

IP Storage Level Setting

File vs. Block Storage

FCIP, iFCP & iSCSI

Fiction & Facts about iSCSI

Storage Replication over WANs

Considerations for Designing IP Storage Networks

Questions

Hosted by

IP Storage Level Setting

There are 3 types:• Block Storage

iSCSI

• File Storage NAS

• Storage over WAN for business continuity = Block iSCSI FCIP iFCP

Hosted by

IP Storage Level Setting

iSCSI Storage• Block-based external storage on Ethernet

Vs. SCSI, USB, 1394, or FC

NAS or Network Attached Storage• File-based storage = NFS & CIFS

• No different than any other file server

• Requires block storage behind it

Hosted byHosted by

By show of hands, what is acceptable packet loss for IP Block Storage on a LAN/WAN?

1) 1%

2) 5%

3) 10%

4) 0%

Hosted by

Acceptable Packet Loss for IP Block Storage on a LAN/WAN

The Answer is:

Hosted by

Block vs. File Storage

IP block data is unlike any other IP data

• Overwhelms most current LAN/WAN environments

• Incredible amounts of traffic

• Tolerates “ZERO” packet loss

• Very low latency

File storage = specialized file server

• NFS & CIFS

• Higher prioritization is required depending on app

• Volume of data may overwhelm untuned LAN/WAN

Hosted by

FCIP, iFCP, iSCSI

FCIP

• Fibre Channel tunneled in TCP/IP

• IP transport between FC switches

iFCP

• IP header put on Fibre Channel frames for routing

• IP connection services for FC devices

iSCSI

• SCSI-3 mapped to TCP

Hosted by

FCIPPt-to-Pt: Becomes “One” FC SAN• Disruptions pass SAN-to-SAN• Large FSPF database• PSS between SAN sites• Gateways between fabrics (blades or boxes)

Ethernet LAN/WAN Switches

Cisco MDS 9216

CNT UESR 3000

Brocade 3xxx

SAN 1

SAN 2

SAN 3

SAN 4

TCP/IPWAN

Hosted by

iFCPPt-to-Multi-Pt• Device specific passing only the data that is required• Devices can appear in multiple individual SANs

The SANs themselves remain independent

Ethernet LAN/WAN Switches

SAN 1

SAN 2

SAN 3

SAN 4

TCP/IPWAN

McDATA/Nishan 3300

Hosted by

Parable

Hosted by

Hosted by

iSCSI: Ethernet-Based SAN

The Hype• Block storage on Ethernet

• Leverage current infrastructure investment &

knowledge base

• Lowers cost

• Eliminates headaches

• Ubiquitous

• Makes FC another Ethernet Road Kill

Hosted by

iSCSI Defined IETF standard protocol

• Establishes & manages connections

• Carries storage (SCSI) blocks From initiators to storage

targets

Encapsulates SCSI blocks in TCP/IP

• Then tunneled in Ethernet

• iSCSI is to Ethernet as FCP is to Fibre Channel

Network application

• One infrastructure for LANs, NAS, & SANs

Ethernet Ethernet

FrameFrame

TCP/IP TCP/IP

PacketPacket

SCSI-3SCSI-3

TCP/IPTCP/IP

iSCSIiSCSI

NAS: File Storage

GbE Switch

Mission CriticalIA App Servers

C O M P A C T

Power

C O M P A C T

Power

C O M P A C T

Power

C O M P A C T

Power

C O M P A C T

Power

C O M P A C T

Power

C O M P A C T

Power

C O M P A C T

Power

C O M P A C T

Power

C O M P A C T

Power

C O M P A C T

Power

C O M P A C T

Power

C O M P A C T

Power

C O M P A C T

Power

C O M P A C T

Power

C O M P A C T

Power

C O M P A C T

Power

C O M P A C T

Power

C O M P A C T

Power

C O M P A C T

Power

C O M P A C T

Power

C O M P A C T

Power

C O M P A C T

Power

C O M P A C T

Power

C O M P A C T

Power

C O M P A C T

Power

C O M P A C T

Power

C O M P A C T

Power

C O M P A C T

Power

C O M P A C T

Power

C O M P A C T

Power

C O M P A C T

Power

C O M P A C T

Power

C O M P A C T

Power

C O M P A C T

Power

C O M P A C T

Power

Hosted by

iSCSI Applications

NAS/SAN combined storage units

Entry level SANs

Limited budget SANs

Hosted by

SAN Benefit Assertions of iSCSI

Reduced costs

• Professional services, implementation,

management & IT Staff time

Reduced complexity

Reduced Management

Increased Interoperability

Elimination of Multiple Networks

Unlimited SAN Distance

Equal or Better Performance

Hosted by

Fiction & Facts about iSCSI Fiction

• Cost is lots < Fibre Channel (FC)

• Complexity is much < than FC

• Uses current infrastructure

• Requires no storage knowledge

• Is as fast as FC

• Will replace FC in Enterprise

• Easier to manage than FC

• Eliminates SAN distance limits Latency (delay) is not an

issue

Facts

• Known technology

• Costs are relatively < FC

Cycles or hardware

• Doesn’t require special HW

But benefits from it

• Latency (Delay) matters

Can’t be > 1 millisecond

• Deterministic routing

• Doesn’t require any-to-any

Hosted by

Rating the iSCSI Value Props

Reduces or Eliminates SAN Professional Services

Lowers SAN Hardware Costs

Simplifies SAN Management

Eliminates Interoperability Issues

Converges SAN/LAN/MAN/WAN Fabric Infrastructure

Extends SANs over unlimited distances

Equal or better performance than FC SANs

Hosted by

iSCSI Reality Check There are some real cost benefits for:

• NAS/SAN on the same fabric infrastructure

• SANs that don’t need the performance of FC

• Entry SANs that may not even need GigE and TOEs

Hype overshadows reality:

• GigE NICs with iSCSI and TOEs cost ~ same as FC HBAs

• FC ports & GigE ports on server motherboards

Makes port cost differences higher for GigE w/TOEs

• Very low cost simplified FC switches

Have erased much of the infrastructure HW differences

Hosted by

Sample iSCSI Vendors

Switches

• Cisco

• Extreme

• Foundry

• Enterasys

• Nortel

• Lucent

• 3Com

Gateways

• Cisco

• McDATA

Silicon

• Adaptec

• Alacritech

• Intel

• Siliquent

• QLogic

Storage

NICs

• QLogic

• Intel

• Alacrite

ch

• Adaptec

• Emulex

Hosted byHosted by

By show of hands, is latency (delay) important to iSCSI block storage?

Hosted by

Is Latency Important to IP Block Storage?

The Answer is:• Yes, for the most part

• It also depends on application

Hosted byHosted by

By show of hands, who believes that TOEs & iSOEs are an iSCSI block storage requirement?

Hosted by

Are TOEs & iSOEs an iSCSI block storage requirement?

The Answer is: Not necessarily

Hosted by

Spectrum of iSCSI Adapter Solutions

Ho

stA

dap

ter

Adapter Driver

TCP/IP

iSCSI

SCSI Port to OS

Software iSCSI“NIC + Driver”

Media Interface

Ethernet

Media Interface

Ethernet

Fast Path TCP/IP

Software iSCSIWith Partial TCP Off-load

TCP/IP

iSCSI

SCSI Port to OS

Media Interface

Ethernet

TCP/IP

iSCSI

Firmware TCPand iSCSI Off-load

SCSI Port to OS

= SW or FW

= Hardware

Media Interface

Ethernet

TCP/IP

iSCSI

Hardware TCP and Firmware iSCSI

Off-load

SCSI Port to OS

Hosted by

iSCSI: No TOE

Definition• Std Ethernet NIC

TCP/IP & iSCSI Host-based in drivers

Who• Microsoft & Cisco

Advantages• Lowest cost• NICs available today• Easy integration with OS

Disadvantages • Lowest performance• High CPU load • High interrupts

Once/packet Many/ TCP segment

Adapter Driver

TCP/IP

iSCSI

SCSI Port to OS

Software iSCSI“NIC + Driver”

Media Interface

Ethernet

= SW or FW

= Hardware

Ho

stA

dap

ter

Hosted by

iSCSI: Little TOE Definition• NIC w/limited TOE

Packets in order & no frags Out of order etc. go to OS

Who• Alacritech

Advantages• Relatively low cost• Small layout (low profile card)• Good throughput w/pristine

Ethernet Disadvantages

• Out-of-order & frags < performance

• Interrupts Once/TCP segment Many/IO

• OS interface challenges

Media Interface

Ethernet

Fast Path TCP/IP

Software iSCSIW/Partial TCP Off-load

TCP/IP

iSCSI

SCSI Port to OS

Ho

stA

dap

ter

= SW or FW

= Hardware

Hosted by

iSCSI: Firmware TOE + iSOE

Definition• TCP/IP & iSCSI firmware

On-board processors Who• Adaptec, Intel, Emulex,

QLogic Advantages• Flexibility to change code• Low CPU load• Low interrupt load: < 1/IO

Disadvantages• No 10Gb scaling• Performance • Power, size

Media Interface

Ethernet

TCP/IP

iSCSI

Firmware TCPand iSCSI Off-load

SCSI Port to OS

Ho

stA

dap

ter

= SW or FW

= Hardware

Hosted by

iSCSI: Hardware TOE + iSOE Definition

• Hardware ASIC TCP/IP bulk data path iSCSI digest (CRC)

• iSCSI in processors

Who• QLogic, iReady

Advantages• Flexibility to change iSCSI code• Low CPU load• Low interrupt load < 1/IO• Performance, scaling to 10G

Disadvantages• Complex chip • Lack flexibility to change TCP

code

Media Interface

Ethernet

TCP/IP

iSCSI

Hardware TCP & Firmware iSCSI Off-load

SCSI Port to OS

= SW or FW

= Hardware

Ho

stA

dap

ter

Hosted by

A TOEs Impact on iSCSI

Hosted byHosted by

By show of hands, who believes that iSCSI allows block storage to go unlimited distance?

Hosted by

Will iSCSI allow block storage to go unlimited distance?

The Answer is: Yes & No• Latency is the limiting factor

• Application dependent Transactions cannot exceed 1ms one way (100

miles) Asynch replication is not distance dependent

“The speed of light, is not just a limit, it’s a law.”

Hosted by

Key Block IP Storage Issues

Distance

• Latency

• WAN bandwidth utilization of IP

Security

• Encryption

• Access

Performance

• Must be = to, or > than current expectations

Hosted by

Storage Replication over WANs

Issues• Good Citizen on Shared TCP/IP WANs

• Filling the pipe > 50%

• End-to-end throughput

Compression

TCP latency

Hosted by

Changing Paradigm for Asynch Storage Replication

Native Storage GigE interfaces emerging

• EMC Symm5 and DMX are available today

• EMC CLARiiON in development

• Hitachi developing GigE for Lightning and Thunder

Software Replication Apps over native IP

• Leverages IP WAN already in place

• Eliminates SAN gateway requirement (FCIP or iFCP)

Significant < cost Mirror/Replication apps

Hosted by

High Speed TCP/IP Data Transport Challenges

Optimized for

• Small payloads & relatively short distances

Employs inefficient

• Error recovery & session management techniques

Delivers poor bandwidth utilization

• For most high performance applications

• Usually < 30% efficiency at extended distances

• Even less as distance and bit errors increase

Hosted by

Cost of Inefficiency

Higher Bandwidth Cost

• Despite < costs, high speed (DS3, OC3, etc) circuits = expensive DSC survey of 200 end-users• BW = 50-70 % of storage replication costs

Operational Inefficiencies

• Can’t complete within time window delaying production ops

• Explosion in data exacerbates the problem

• Current = specialized equipment & separate networks Can’t fully leveraging IP infrastructure = > costs

Hosted by

Native GigE Replication: SRDF Adaptive Copy & SNAP/Asynch• Performance degrades starting at ~ 300 miles

At 500 miles performance degradation is noticeable & significant

Ethernet LAN/WAN Switches

TCP/IPWAN

EMC DMX

SAN

Hosted by

NetEx HyperIP

Native GigE Replication: SRDF + RFC 3135

Adaptive Copy & SNAP/Asynch• RFC 3135 = TCP/IP Performance Enhancing Proxy• Up to satelite distances (46K miles roundtrip)• 90% + bandwidth utilization (T1/E1, DS3, OC3, OC12)• Plus 2 to 4 to 1 compression

Who• NetEx (HyperIP), Expand, NetCera, Digital Fountain

Ethernet LAN/WAN Switches

TCP/IPWAN

EMC DMX

SAN

Hosted by

EMC SRDF Replication over WANs Replication Methodologies Illustrated

Ethernet LAN/WAN Switches

Cisco MDS 9216

CNT UESR 3000

Brocade 3xxx

TCP/IPWAN

McDATA/Nishan 3300

EMC DMX

NetEx HyperIP

= GigE

= FC

Hosted by

Considerations for Designing IP Storage Networks

Separate LAN fabric• Minimally, separate VLAN

Layer 2 switching• Best latency for Ethernet switching

• Nothing less than GigE

Understand LUNs• Mapping and Masking

Hosted by

Questions?