View
5
Download
0
Category
Preview:
Citation preview
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
1© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
STORAGE AREA NETWORKING PROTOCOLS AND ARCHITECTURESESSION OPT-2T01
222© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Morning Schedule
• 9:00am–10:30am Introduction to Storage Area Networking
Storage Terms and Acronyms
Storage Networking Devices (Switches, HBAs, Disk)
Storage Networking ApplicationsStorage Networking TopologiesIntro to Storage Protocols (SCSI, FC, FCIP, iSCSI)
• 10:30am–10:45am Break
• 10:50am–12:30pm Storage Protocols in-depth
Introduction to the Standards
SCSIFibre Channel
• 12:30pm–1:30pm Lunch
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
333© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Afternoon Schedule
• 1:45pm–3:30pm Storage Protocols In-Depth (Cont.)Fibre Channel ServicesiSCSI FCIPiFCPiSNS and SLP
• 3:30pm–3:45pm Break • 3:50pm–6:00pm Storage Network Troubleshooting
Required ToolsRequired Technical Skill SetsStorage Network Architecture
Design PracticesFC Network DesignsIP SANsSAN ExtensionImplementation and Management
444© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Associated Sessions
• OPT-1051 Introduction to Storage Topologies and Applications
• OPT-2051 Fibre Channel Storage Area Network Design • OPT-2052 FCIP Design and Implementation • OPT-2053 iSCSI Design and Implementation • OPT-2054 Storage Networking Security • OPT-3051 Troubleshooting MDS9000 Fibre Channel SAN• OPT-3052 Troubleshooting MDS9000 IP Storage
Area SAN• OPT-4051 Design and Architecture of Storage
Networking Platforms• OPT-4052 Case Study: Cisco IT Storage Strategy
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
555© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Reference Materials
• Cisco Storage Networkingwww.cisco.com/go/storagenetworking
• Cisco AVVID Storage Networking Partner Programwww.cisco.com/go/partners
• Cisco Metro Optical Product Informationwww.cisco.com/go/comet
• Storage Network Industry Association (SNIA)www.snia.org
• IETF—IP Storagewww.ietf.org/html.charters/ips-charter.html
• ANSI T11—Fibre Channelwww.t11.org/index.htm
INTRODUCTION TO STORAGE AREA NETWORKING
666© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
777© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Section Agenda
• Storage Terms and Acronyms
• Storage Networking Devices
• Storage Networking Applications
• Storage Networking Topologies
• Introduction to Storage Protocols
888© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
STORAGE TERMS AND ACRONYMS
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
999© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Technologies Overview (or “Storage in a Nutshell”)
FSPF
Databases
Servers andMainframes
BackupApps
Mirroring
IPCLOUD
IPCLOUD
IPCLOUD
FC Switch FC Switch
Virtual SAN
JBODsand NAS
Storage AreaNetwork (SAN)Technologies
TAPE
EmbeddedManagement
SANApplications
SANProtocols
RAID &VirtualRAID
iSCSI Drivers
FCIP
iSCSI
iSCSI
GenericFibre Channel
EnhancedFibre Channel
SAN
FCHA
Call Home
SupportCenter
StorageVirtualization
101010© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Introduction to SAN Terminology
• Block Level I/O
• File Level I/O
• SCSI—Small Computer Systems Interface
• FC—Fibre Channel
• RAID—Redundant Array of Inexpensive Disks
• iSCSI—Internet SCSI
• FCIP—Fibre Channel over TCP/IP
• iFCP—Internet Fibre Channel Protocol
• iSNS—Internet Storage Name Service
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
111111© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
RAID Levels
3 without log 4 with log
Stripe with distributed parity with Random I/O
5
N/AFix parity with Random I/O4
N/AFix parity with concert I/O3
N/AHamming Code2
4Mirror then Striping/Concatenation1+0
4Striping/Concatenation then Mirror0+1
2Mirror1
2Striping/Concatenation0
Min DisksDescriptionRAID Level
121212© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Terminology Direct Attached Storage (DAS)
• Block level I/O
• Can be internal or external
• Typically SCSI or FC
• Limited scalability
• High cost due to management
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
131313© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Terminology Network Attached Storage (NAS)
• File level I/O
• Used for file sharing applications
• IP-based
• Deployed over existing low-cost Ethernet networks
• Redundant links
• Scalable
• Multiple servers can share same file system
IP
NAS NAS NAS
141414© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Terminology Storage Area Network (SAN)
• Block level I/O
• Deployed as separate network
• Servers share storage subsystem
• Scalable
• Multiple paths for high availability
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
STORAGE NETWORKING DEVICES
151515© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
161616© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
SAN ComponentsHost Bus Adapter (HBA)
• Interface between host and storage
• Supports copper or optical
• Typically one port; Can be multiple ports
• 1Gb, 2Gb and 4Gb
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
171717© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
SAN ComponentsFabric Switch
• 1Gb, 2Gb, and 4Gb
• 8-40 ports
• Low latency
• Can be copper or optical
181818© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
SAN ComponentsDirector Class Switch
• 1Gb, 2Gb, 4Gb and 10Gb
• FC and FICON
• 256 ports and growing
• Low latency
• Can be copper or optical
• Multi-service platforms
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
191919© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
SAN ComponentsJBOD
• Just a bunch of disks
• Limited scalability
• Typically 2 FC ports
• SCSI or FC disks
• Basic controllers
• No caches
202020© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
SAN ComponentsStorage Arrays
• 36GB to many TB
• Typically 2 to many interfaces
• Subsystems may mix interfaces
• ESCON/FICON, SCSI, FC, or iSCSI
• SCSI or FC disks
• Intelligent controllers
• Large caches
iSCSI
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
212121© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
SAN ComponentsTape Arrays
• Tape speed vary 5MBs—30MBs+
• Capacity vary 20GB—300GB+
• Deployed in servers or external libraries
• SCSI, FC, Ethernet interface
• DLT most common; LTO gaining traction
222222© 2004 Cisco Systems, Inc. All rights reserved.
STORAGE NETWORKING APPLICATIONS
OPT-2T019899_06_2004_X
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
232323© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
IT Storage Requirements
• ScalabilityMeet high growth demand for storage capacity (>80% per year)Increase capacity utilization rates
• AvailabilityShare data across distributed data centers via fast speed, long distance connectivity links Provide effective disaster recoveryImprove interoperability across heterogeneous equipmentEnhance security
• ManageabilityAutomate storage management functionsProvide cross-vendor management tools Managing heterogeneous environments
242424© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Storage Network Build-Out
• Application-specific islands of networked storage
• iSCSIConvenient extension of existing FC SAN to IP-attached servers
• Extensive IP services for NAS environments
Homogenous Infrastructure“Isolated Islands”
Starting Point
DAS
SAN
NAS
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
252525© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Storage Network Interconnection
• SAN interconnection forBusiness continuanceUnified management
Remote backup
• Metro DWDM solutionsLow-latency option for synch replication
• FCIPLower-cost option for asynch replication and backup consolidation
SAN Interconnectivity
Present Trend
Optical
FCIP
FCIP
262626© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Intelligent SAN
• Intelligent services into the network
• Common management framework
• Content, file, and block awareness
• Transport independent
Storage Utility SAN
Data MgmtServices
StorageRouting
HostAwareness
StorageManagement
StorageSwitching
ContentDelivery
StorageVirtualization
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
STORAGE NETWORKING TOPOLOGIES
272727© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
282828© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
SCSI I/O Topology
• SCSI is the protocol used to communicate between servers and storage devices
• SCSI I/O channel provides a half-duplex pipe for SCSI commands and data
• Parallel implementationBus width: 8, 16 bitsBus speed: 5–80 MhzThroughput: 5–320 MBpsDevices/bus: 2–16 devicesCable length: 1.5m–25m
• A network approach can scale the I/O channel in many areas (length, devices, speed)
SCSI
Target
SCSI Adapter
Host System
Initiator
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
292929© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Fibre Channel Topology
• Very common method for networking SCSI
• Fibre Channel provides high-speed transport for SCSI payload
• Fibre Channel overcomes many shortcomings of DAS including:
Addressing for up to 16 million nodes (24 bits)Loop (shared) and Fabric (switched) transportSpeeds of 100 or 200 Mbps (1 or 2 Gbps)Distance of up to 10km (without extenders)Support for multiple protocols
• Combines best attributes of a channel and a network
SCSI
Target
FibreChannel HBA
Host System
Initiator
Fibre ChannelFabric
303030© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
iSCSI Storage Topology
• IP access to open storage sub-systems
• iSCSI driver is loaded onto hosts on ethernet network
• Able to consolidate servers via iSCSI onto existing storage arrays
• Able to build ethernet-based SANs using iSCSI arrays
• Storage assigned by iSCSI instance
iSCSI-EnabledHosts (Initiators)
iSCSI Array
(Target)
StoragePool (Target)
iSCSIRouter
IPNetwork
FCFabric FC HBA
Attached Host
(Initiator)
iSCSIiSCSI
iSCSI
iSCSI
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
313131© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
FCIP SAN Extension Topology
• FCIP gateways perform Fibre Channel encapsulation process into IP packets and reverse that process at the other end
• FC Switches connect to the FCIP gateways through an E_Port for SAN fabric extension to remote location
• A tunnel connection is set up through the existing IP network routers and switches across LAN/WAN/MAN
StandbyStorage
Production
Storage
Production SiteProduction Site
Database Servers
EMC SRDFExisting IP
NetworkLAN/WAN/MAN
FC Switch
FCSAN
Backup Server
FCIPFCIPGatewayGateway
FCIPFCIPGatewayGateway
ServersFC
SAN
FC Switch
Backup, R&D, Shared Storage,Data Warehousing, Etc.
Backup, R&D, Shared Storage,Data Warehousing, Etc.
323232© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
FCIP and iSCSI: Complementary
• FCIP: SAN-to-SAN over IP• iSCSI: Host to storage over IP
IP Network
FC SAN
Storage Router
FC SAN
iSCSIiSCSIiSCSI
iSCSI
Storage Router
FCIPGateway
FCIPGateway
FCIPFCIP
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
INTRODUCTION TO STORAGE PROTOCOLS
333333© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
343434© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Introducing SCSI
• SCSI = Small Computer System Interface
• SCSI is a standard that defines an interface between an initiator (usually a computer) and a target (usually a storage device such as a hard disk)
• INTERFACE refers to connectors, cables, electrical signals, optical signals and the command protocol that allow initiators and targets to communicate
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
353535© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
SunSun
Opcode (2A = Write 10)
Reserved
LBA
LBALBA
(0010E43)
Reserved
LBA
LBALen
(128)
Control
SCSI Example
Initiator Target 2Target 1
SCSI Connector SCSI Cable
The SCSI Command Protocol Is Used to Communicate Between SCSI Devices
SCSI Command
In this Case, a File is Being Written to the Hard Drive By an Application on the Workstation
TapeDisk
363636© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Why Is SCSI Important for SANs?
• SCSI command protocol is the de facto standard that is used extensively in high-performance storage applications
• The command part of SCSI can be encapsulated in FCP—Fibre Channel Protocol or IP and carried across internetworks; This is the core concept behind storage area networking
• To understand the finer points involved with transporting SCSI across a network with FC or ethernet, the basics of SCSI must be well understood
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
373737© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Standards
• SCSI has evolved since it was introduced as SASI in 1979 by Shugart Associates—it was approved as a standard by ANSI in 1986 and is now referred to as SCSI-1
• SCSI-2 was approved by X3 in 1990 and by ANSI in 1994
• SCSI-3 refers to a collection of standards, each of which defines a very specific part of SCSI: physical interface, transport interface, command interface, architecture model, programming interface, etc.
383838© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Sample SCSI Standard ComponentsSCSI Parallel Interface: SPI
Sun
Sun
Initiator
Target 1
Target 2
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
393939© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Sample SCSI Standard ComponentsSCSI Primary Commands: SPC
SunSun
Initiator Target 2Target 1
SCSI Primary Commands(SPC-2)
SCSI Block Commands(SBC)
SCSI Stream Commands(SSC)
404040© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
SCSI Standards: The Big Picture
FC-xx 1394
SPI-x
FCP SBP
ATAPI
SPC-2 / SPC-3
CAM ASPI Generic
SBC SSC SES More…
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
414141© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
“This specification describes a reference model for the coordination of standards applicable to SCSI-3 I/O systems and a set of common behavioral requirements which are essential for the development of host software and device firmware that can interoperate with any SCSI-3 interconnect or protocol.”
SCSI Architecture Model November 1995
SCSI Architecture Model
424242© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
SCSI Architecture Model
• The SCSI architecture model defines generic requirements and implementation requirements
• Each SCSI implementation standard must fulfill the requirements set forth by SAM
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
434343© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
SAM Highlights: Client-Server
• SCSI is a client-server protocol
• The client is called the initiator (this is usually the OS I/O subsystem) and issues requests to the server
• The server is called the target (this is usually the SCSI controller that is part of a storage device) and receives, executes and returns initiator requests and their associated responses
444444© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
SAM Highlights: Initiator: Target
• A single initiator can have multiple application clients
• Targets have ONE task manager and one OR MORE Logical Units (LU), which are numbered (LUN)
• The task manager has the authority to modify service requests that have already been received by the target
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
454545© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
SAM Highlights: Logical Units
• Each logical unit within a target is numbered; that number is called a LUN and is the only way to refer to that logical unit
• The device server is the entity that receives, executes and returns requests that are made to its logical unit
• The concept of task setis beyond the scope of this presentation
464646© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
SAM Highlights: Command Model
• SAM defines two categories of protocol services:Execute command/confirmation services;Data transfer services
• This leads to the three main phases of a data transfer:1. Execute: Send required command and parameters
via CDB;2. Data: Transfer data in accordance with the command;
3. Confirmation: Receive confirmation of command execution
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
474747© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
SAM Highlights: Sample Data READ
1. Send SCSI Cmd issued by initiator—the command sent is READ;
2. SCSI command receivedby target;
Data transfers occur during the ‘working’ phase between initiator and target;
3. Send command completeis returned by the target;
4. Command complete received by target
…
484848© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
SAM Highlights: Parameters
• The data transfer model reflects parameters that will be used by SCSI commands
• This model illustrates that a complete data transfer (right) can be broken up into multiple parts (left)
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
494949© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
SAM Highlights: Communication Model
SAM Defines a Hierarchy of Protocols
Let’s Expand on this Portion
505050© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
SCSI Transport Protocol
IPIP
TCPTCP
iSCSIiSCSI
SCSI ProtocolSCSI Protocol
FibreChannelFibreChannel
FCPFCP Parallel BusParallel Bus
EthernetEthernet
iFCPiFCPFCIPFCIP
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
515151© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
SCSI Transport Protocol
IPIP
TCPTCP
iSCSIiSCSI
SCSI ProtocolSCSI Protocol
FibreChannelFibreChannel
FCPFCP Parallel BusParallel Bus
EthernetEthernet
iFCPiFCPFCIPFCIP
Today’s In-Depth Protocol
Discussions
STORAGE PROTOCOLS IN-DEPTH
52© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
535353© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Section Agenda
• Introduction to Standards
• SCSI Protocol
• Fibre Channel Protocol
• Internet SCSI (iSCSI)
• Fibre Channel over IP (FCIP)
• Internet Fibre Channel Protocol (iFCP)
• iSNS and SLP
INTRODUCTION TO STANDARDS
54© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
555555© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Standards Groups: Storage
ISO / IECJTC-1
InterNational Committee for InformationTechnology Standards
(INCITS)
American NationalStandards Institute
(ANSI)
Techincal Committee onLower-Level Interfaces
(T10)
Information TechnologyIndustry Council
(ITI)
Techincal Committee onDevice-Level Interfaces
(T11)
CJ11
C++J16
Techincal Committee on ATAttachment Interfaces
(T13)
SCSI Fibre ChannelHIPPI
IPI
ATA (IDE)ATAPI
www.t10.org www.t11.org www.t13.org
565656© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Standards Process
• Technical Committees (T10) write drafts
• Drafts are sent to INCITS for approval
• Once approved by INCITS, drafts become standards and are published by ANSI
• ANSI promotes american national standards to ISO as a Joint Technical Committee member (JTC-1)
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
575757© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Standards Work Group: IP Storage
IP Storage Technical Work Group Acts as Primary Technical Focal Point of the Storage Networking Industry Association (SNIA) on IP Storage Issues, Coordinating with the SNIA IP Storage Forum
Transport Area—Has 23 WGs, One which Is the IP Storage WG
ISOC
Internet Society
IESGInternet Engineering
Steering Group
IETF
Internet EngineeringTask Force
Transport Area
IEFT Is the Organization Ratifying the IPS Standards
FIBRE CHANNEL IN-DEPTHFIBRE CHANNEL IN-DEPTH
58© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
595959© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Fibre Channel Protocol Agenda
• FC Introduction• Fibre Channel Communications Model• Protocol Constructs• FC-PH (Fibre Channel—Physical and Signaling
Interface)• Login Parameters• Frame Processing• Arbitrated Loop• Switch Fabric Operation• Switch and Hub Mixed Topology Network Operations• FC Error Management
606060© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Fibre Channel Environment
• Channel reliabilityMultiprotocol supportOvershared serial media
With networking capability and functionality
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
616161© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Fibre Channel Environment
• High bandwidth
• High data integrity
• Highly reliable
• Destination pacedBuffer credits
• Scalable
• High availability
• Shared media
• Circuit/packet
• Multiple protocol support
• Transport flexibilityDedicated conn—Class 1Multiplexed—Class 2
Datagram—Class 3
• Configuration flexibilitySwitch
Loop
626262© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
What Is It?
Channels• Connection service
• Physical circuits
• Reliable transfers
• High speed
• Low latency
• Short distance
• Hardware intense
Networks• Connectionless
• Logical circuits
• Unreliable transfers
• High connectivity
• Higher latency
• Longer distance
• Software intense
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
636363© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
What Is It? Fibre Channel
Channels• Connection service• Physical circuits• Reliable transfers• High speed• Low latency• Short distance• Hardware intense
Networks• Connectionless• Logical circuits• Unreliable transfers• High connectivity• Higher latency• Longer distance• Software intense
Fibre Channel• Circuit and packet
switched
• Reliable transfers
• High data integrity• High data rates
• Low latency• High connectivity
• Long distance
646464© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Fibre Channel Protocol LevelsLevels
HIPPI Cluster 370 OEM
SCSI IP ATM
Common Services
FC-4’s
FC-3
Signaling Protocol
Transmission Code
Physical Interface
FC-2
FC-1
FC-0
FC-PHN_Port
F_Port
PC-PH = Physical and Signaling Layer
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
656565© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Fibre Channel Functions
Structure Is Divided into 5 Levels of Functionality• FC-0 defines the physical interface characteristics
Signaling rates, cables, connectors, distance capabilities, etc.
• FC-1 defines how characters are encoded/decoded for transmission
Transmission characters are given desirable characters
• FC-2 defines how information is transportedFrames, sequences, exchanges, login sessions
• FC-3 is a place holder for future functions
• FC-4 defines how different protocols are mapped to use Fibre Channel
SCSI, IP, virtual interface architecture, others
666666© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Fibre Channel Topologies
• Point to point
• Arbitrated loop
• Switched fabricFC
N
N N
L
L
L L
L
L
N
N
N
FF
FF
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
676767© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Point to Point
• Dedicated connection between ‘N’ port FibreChannel devices
• All link bandwidth is dedicated to communication between the two nodes
• Suitable for small scale scenarios when storage devices are dedicated to file servers
NNN
N
686868© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Arbitrated Loop (FC-AL)
• TX of each node is connected to the RX of the next node until a closed loop is formed
• Maximum bandwidth: 100 MB/sec. (shared amongst all nodes on loop)
• 126 nodes max on loop• Not a token passing scheme—no limit
on how long a device may retain control• Operational sequence:
Arbitrate for control of loopOpen channel to targetTransfer dataClose
• Number of nodes on loop directly affects performance Fibre Channel Hub
FC
L
L
L
L
L
L
LLL
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
696969© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Data Integrity
Upper Level Protocol
Signaling Protocol
Transmission Code
Physical Media
Operation Control and Byte Counts
• Operation• Frame counts• CRC (32 bit)• Frame delimiters
8b/10b Code
Fibre Reliability
707070© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Flow Control
• Back pressure technique
• Frame credit Established by receiver during LOGIN
• TransmitterMust have credit to transmit
• ReceiverReinstates credit with ACK
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
FIBRE CHANNEL COMMUNICATIONS MODEL
71© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
727272© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
The Model
• The Fibre Channel communications model is based on the definition of:
Physical objectsProtocol construct
• These objects and constructs:Define the behavior of the physical elementsControl the transfer on informationProvide for “link” managementProvide the basis for:
HardwareFirmwareSoftware
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
737373© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Physical
• The fundamental physical objects in FibreChannel are:
PortsLink
Nodes
Fabric
Some Logical Items Used in These Discussion Are:• Addressing
• Communications Model
747474© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Fibre Channel: Port Types
• ‘N’ port: Node ports used for connecting peripheral storage devices to switch fabric or for point to point configurations; can be considered the end port
• ‘F’ port: Fabric ports reside on switches and allow connection of storage peripherals (‘N’ port devices)
• ‘L’ port: Loop ports are used in arbitrated loop configurations to build storage peripheral networks without FC switches; these ports often also have ‘N’ port capabilities and are called ‘NL’ ports
• ‘E’ port: Expansion ports are essentially trunk ports used to connect two Fibre Channel switches
• ‘G’ port: A generic port capable of operating as either an ‘E’ or ‘F’ port; its also capable of acting in an ‘L’ port capacity; Auto Discovery
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
757575© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
N_Port
Host / Device
N_Port
Host/ Device Interface
Serial Data Out Serial Data In
767676© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Link
• A link consists of2 unidirectional “fibers” transmitting in opposite directionsMay be either:
Optical fiber
Copper
• Transmitters may be:Long wave laser
Short wave laserLED
Electrical
Link Transfer Rates
25265.5100106.25
Mbytes/secClock Mbaud/sec
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
777777© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Link
Link
Host / Device
N_Port
Host/ Device interface
Serial Data Out Serial Data In
787878© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Node
• The equipment which contains one or moreN_Port or NL_Port (topology dependent)May be
Computer
ControllerDevice
Is NOT a switch fabric
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
797979© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Node
N_Port
Link
N_Port
Link
N_Port
Link
N_Port
Link
Controller
808080© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Communications Model
• Point to point
• N_Port to N_Port
• Flow control
• AcknowledgedNode Node
Transmitter
Receiver Transmitter
Receiver
N_Port
Node Node
Link
Transmitter
N_Port
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
818181© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Fabric
• FabricThe entity which interconnects N_PortsProvides routing based on destination address
Fabric may be:
Point to point—No routing requiredSwitched—Routing provided by switch
Arbitrated loop—Routing is distributed throughout attached L_Ports
828282© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Terms
• TopologyThe physical structure of the interconnect of portsDefines the logical behavior of transactionsFibre channel has 3 topologies
Pt to PtSwitchedArbitrated loop
• FabricThe fabric is the generic item that interconnects nodesA fabric is made of Fibre Channel topologies like Pt to Pt, switches and loops
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
838383© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Point to Point
Communications Model
• Source to destination
• Based on address routing through the fabric
Node Node
Transmitter
Receiver
Receiver
Node Node
Fabric
N_PortN_Port
Transmitter
848484© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Switched Fabric
N_Port N_PortN_Port N_Port N_PortN_Port
Fabric
Switch
Communications Model—Source to Destination Based on Address Routing through the Fabric
A B
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
858585© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Communications Model—Source to Destination Based on Address Routing Distributed in the NL_Ports on the Loop
Arbitrated Loop
NL_Node “A”
NL_Node “B”
Link
A B
FC PROTOCOL CONSTRUCTS
86© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
878787© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
What Are Protocol Constructs
• The fundamental protocol structures in the FibreChannel are called constructs, and they are:
FramesSequences
Exchanges
Information Units (IU)Procedures
Upper Layer Protocols (ULP’s)
888888© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Construct Introduction
• FC-2 defines these constructs that allow the related information to be:
Grouped togetherCoordinatedHandled in an efficient manner
• To accomplish this we define the notion of:FramesSequencesExchanges
• Also defined are means for the Upper Level Protocols ULP’s to communicate with FC-2:
Information Units (IU)• A procedure called the login defines the operating
environment between the N_PortsExchange of the data describing the characteristics of the ports
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
898989© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Chunks
• The ULP’s deal with “chunks” of data that are moved across the network
• These chunks of data may be eitherControlStatus
Real data
909090© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Frames
Frame• FC-2 layer will take this
chunk of data and move it from
Transmitting node to receiving node In the units of what Fibre Channel calls frames
Frame Size• FC-2 will determine the size
of the frames based on operating environment established between the two communicating nodes
Common Services
Signaling Protocol
Transmission CodePhysical Interface
FC-3
FC-2
FC-1
FC-0
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
919191© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Frame Structure
24* 4 24 0-2114 4 4 Bytes
Idles SOF Frame Header Data Field CRC EOF Idles
* 6 Idle Words (24 bytes) Requires by TX
2 Idle Words (8 bytes) Guaranteed to RX
General FC-2 Frame Format
Frame Format
CRC Calculated on Frame Header and Data Field Only
929292© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Frame Header
R_CTL Routing D_ID 24bits Destination
07815
16
23
24
31
Word
0
1
2
3
4
5
S_ID 24 bits SourceCS_CTL 8 bits Class Spec TYPE 8 bitsData structure
SEQ_ID 8 bits DF_CTL 8 bitsData field
F_CTL 24 bits Frame Control
SEQ_CNT 8 bits Sequence Count
OX_ID 16 bits Orig Exch ID RX_ID 8 bits Respon Exch ID
Parameter Specific to frame type
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
939393© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Data Field
Optional Headers PayloadFILL
Data Field 0-2114
0 - 64 0 - 2112
1 - 3
0–2048 Typical MTU
949494© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Sequence
• SequencesEach chunk of Upper Level Protocol (ULP) data is moved within the envelope of what Fibre Channel calls a Sequence (SEQ)A sequence consists of a set of related framesAs expected there are lots of rules governing sequences
• Information Units (IU)The ULP tells the FC-2 how to transfer theses chunks of data through a structure called a information unitVery few rules for IU’sIU is a convention defined outside of FC-PHIU’s are unique to each upper level protocol
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
959595© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Sequence
• Sequence Initiator (SI)The N_Port which is transmitting the data frames
• Sequence Recipient (SR)The N_Port which is receiving the data frames
FabricSI N_Port
SR N_Port
Data
Chunk
Data
Chunk
Data Frame
969696© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Sequence
Fabric
Initiator (SI)
Target
Data Frame
Sequence
Read Command (Chunk)
(SR)
(SI)
Sequence
(SR) Data (Chunk)
Status
Data Frame
Data Frame
Sequence
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
979797© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Sequence Identifier
• Sequence initiator assigns an “identifier” to each sequence
This “identifier” is called the Sequence_Identifier or Seq_IDThe Seq_ID uniquely identifies a given sequence within the context of the operation
Each frame is identified within this operation by Seq_ID and Seq_CNT
989898© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Sequences: Active and Open
• Sequence Initiator (SI)A sequence is ACTIVE
From the time the first frame of the sequence is transmitted until the frame with the end sequence flag is sent
A Sequence is OPENFrom the time the first frame is transmitted until the reception of the ACK to the last frame
• Sequence Recipient (SR)A sequence is ACTIVE and OPEN
From the time of the first frame of the sequence is received until the transmission of the ACK to the last frame of that sequence
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
999999© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Sequences: Active and Open
Originator (SI)Responder (SR)First Data_Frame
SOF Received
EOF Transmitted
Frame with End_SEQ set
ACK to first Frame
ACK to last Frame
EOT Received
Open
Active
Active & Open
100100100© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Sequence Streaming
• Sequence streaming is the ability to Begin transmission of the next sequence while one or more previous sequences are OPEN
• Sequence Recipient (SR) grants permission to have up to “n” streaming sequences; This is determined at N_Node login time
Must Support “n=1” sequence status blocks (state info)
(This Allows for More Data in the Pipe for Distant Connections)
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
101101101© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Exchange
• Upper level protocols frequently deal with related bits of data as:
Request/replyCommand/data/status
• These relationships are called “operations”Exchanges• “Operations” of data grouped together into what
Fibre Channel call exchangesAn exchange consists of a set of related sequences
Exchanges are bi-directionalSequences are unidirectional and sequential
• There are other rules that govern exchanges
102102102© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Exchange
Fabric
Initiator (SI)
TargetData Frame
Sequence
Read Command (Chunk)
(SR)
(SI)
Sequence
(SR) Data (Chunk)
Status
Data Frame
Data Frame
Sequence
Exchange
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
103103103© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Exchange
• Exchange originatorThe N_Port which transmitted the FIRST data frame for this exchange
• Exchange responderThe N_Port which is the destination of the FIRST data frame of this exchange
The designation for the originator and responder are fixed for the duration of the exchange
Unlike the SI and SR Which Change Roles Within the Exchange
104104104© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Exchange Identifiers X_ID
• An exchange has two “identifiers” associated with it
Exchange originator:Assigns an OX_ID which is meaningful to it
Exchange responder:Assigns a RX_ID which is meaningful to it
In general terms it is called the X_ID• Meaningful is that in the exchange there is
“context” with information like state, control, and status with regards to the exchange
• An N_Port will save, create and update this information throughout the exchange based on the assigned X_ID’s
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
105105105© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Information Unit
• Upper Level Protocols (ULP’s) know about Information Units (IU’s) but know nothing about:
FramesSequencesExchanges
• A ULP deals with units like:Order of events within the operationWhich node will transmit in the next “phase”
(Command phase, data phase, status phase)Is required to have some knowledge about Fibre Channel
• An information unit is a Fibre Channel sequence
106106106© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Information Unit
• The IU contains information sets with such items as LUN, task attributes, CDB and the command byte count
• The IU’s are used in protocol mapping from FC-4 to FC-2 and are assigned an identifier that is useful to humans not used by the machine
• All the information needed to support a ULP is formed into a IU table and is listed as a first , middle or last IU in the exchange
We Will See More of these Tables when We Cover SCSI Mapping onto Fibre Channel
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
107107107© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
FC-2 Hierarchy
Construct Meaning
OX_ID / RX_IDExchange
SEQ_ID
Information Unit
Sequence
Frame SEQ_CNT
Frame Fields
Consists of one or more Sequences for ULP Operation
The structure used by the ULP to define a Sequence (not visible over link)
Consists of one or more related Frames
Contains in its Payload a ULP “chunk” of data
The Hierarchy of Constructs
108108108© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
FC-2 Hierarchy
Frame Fields
OX_ID & RX_ID
…...
…...
EXCHANGE
SEQ_ID
SEQ_CNT
SEQUENCE
Frame FrameFrame
Information Unit Per ULP Terms
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
FC-PH (FIBRE CHANNEL: PHYSICAL AND SIGNALING INTERFACE) STRUCTURE, PROCEDURES, AND PROTOCOLS
109© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
110110110© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Transmission Code
• Fibre Channel uses a 8b/10b transmission codeEach 8 bit data byte to be transmitted is converted into a 10 bit quantity
The 10 bit quantity is then transmitted over the FC media
The 10 bit quantity is then converted back to the 8 bit data byte by the receiving node
• The 10 bit quantities are called transmission characters
• Transmission characters come in two formsData chartersSpecial characters
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
111111111© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
8b/10b Code
Why 8b/10b
1. To ensure the sufficient transitions are present in the serial bit stream to make clock recovery possible at the receiver
2. Increase the likelihood of detecting any single or multiple bit errors
3. To provide special characters with distinctive and easily recognizable characters to achieve word alignment on the incoming bit stream
112112112© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
8b/10b Code
Characteristics of 8b/10b• The 10 bit transmission code
Supports all 256 values of the 8 bit data byte
Contains unused code points
Illegal codes(called code violations)
Detection of code violations
May occur on the transmission character in which the error occurred or may be detected on a subsequent character
Contains “special” characters
Running “disparity” with DC balance
(Count of 0’s and 1’s Equal the Same over Time)
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
113113113© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
8b/10b Code
Running DisparityDisparity: The Difference Between the Number of Ones and
Zeros in a Transmission CharacterRunning Disparity: A Binary Parameter Indicating the Cumulative
Disparity of All Previously Issued Transmission Characters
Transmission Characters Always Have Either:6 Ones and 4 Zeros = Positive Disparity4 Ones and 6 Zeros = Negative Disparity5 Ones and 5 Zeros = Neutral Disparity
Rules: A Positive Disparity Transmission Character Can Not Be Followed By Another Positive Transmission Character
A Negative Disparity Transmission Character Can Not Be Followed By Another Negative Transmission CharacterAt Transmission Character Boundaries the Difference between the Number of Ones and Zeros is + or – 1
114114114© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
8b/10b Code
Code Notation• Each valid transmission character has been
assigned a name in the form of:
Zxx.y“Z” = K or D D=Data K=Special Character
“xx” = Decimal Value of the 5 LSb bits“y” = Decimal Value of the 3 MSb bits
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
115115115© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Conversion Table
ABCDEiFGHj
01234567 FC-2 Bits
FC-1 Code Bit
LSBMSB
D or K .
j and i are add as part of the 10b conversion process
0 0 0 0 1 0 0 0
Example D1.0
FC-1 Transmission Character
0 1 1 1 0 1 0 1 0 0 Neg Disp Value
Transmission Order
116116116© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Special Characters
• K28.5 only special character used in FibreChannel out of the 12 set aside
Has no 8 bit representationThe only FC transmission character with 5 consecutive 1’s or 0’s
Used to find word boundaries and syncUsed in ordered sets
0 0 1 1 1 1 1 0 1 0 + Current Running Disparity
1 1 0 0 0 0 0 1 0 1 - Current Running Disparity
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
117117117© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Transmission
• Transmission word consists of 4 continuous transmission characters treated as a unit
40 bits longAligned on a word boundaryThere is a ordered set and a data word
Encoded Data Byte
Encoded Data Byte
Encoded Data ByteK28.5
Byte 0 1 2 3
Ordered Set
Encoded Data Byte
Encoded Data Byte
Encoded Data Byte
Encoded Data ByteData Word
Transmission Order
118118118© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Ordered Set
• Transmission word starting with the K28.5 special character
• Three classifications of ordered sets are definedDelimitersPrimitive signals
Primitive sequences
Dxx.yDxx.yDxx.yK28.5MSB LSB
The Three Data Characters Define the Meaning of the Ordered Set and Are Repeated for the Third and Fourth Character
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
119119119© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Primitive Signals
• Primitive signals are ordered setsTransmission of primitive signals are interrupted occasionally to transmit frames
• Three basic typesReceiver_Ready (R_Rdy)Idle (idle or I)
Arbitrate (ARBx)
120120120© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Delimiters
• Delimiters are ordered sets that delineate a frameImmediately preceding and following the contents of a frame
• Two basic typesStart_of_Frame (SOF)End_of_Frame (EOF)
• SOF delimitersIdentify the start of a frameIdentify the transmission classUsed to establish a Class_1 connectionIdentify the beginning and continuation of a sequence
• EOF delimitersTerminate framesIdentify the end of a sequenceTerminate connectionsIndicate known frame errors
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
121121121© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
FC-1 Synchronization
• ProceduresSync acquireInitialization
Loss of sync procedure
• Primitive sequences
122122122© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Sync Procedures
• Bit synchronizationThe state in which a receiver is delivering retimed serial data at the required bit error rate
• Transmission word synchronizationAchieved when the receiver identifies the same transmission word boundary on the receive bit stream as the established by transmitter at the other end of linkAcquired by detection of three consecutive ordered sets without errors
• Loss of synchronization procedureThe receiver shall enter the loss-of-sync state upon detection of the fourth invalid transmission word
• Synchronization acquired procedureThe receiver shall enter the synchronization-acquired state when it has achieved both bit and transmission word sync
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
123123123© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Synch AcquiredLoss of Sync State
Bit Sync Acquired
Rx Ordered set #1
Rx Ordered set #2
Rx Ordered set #3
Sync Acquired
Data Word
Data Word
Data Word
Waiting on Bit Synchronization
124124124© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Loss-of-Sync ProcedureSync Acquire State
No Invalid Words Detected
First Invalid Word
Second Invalid Word
Third Invalid Word
Loss Of SyncFourth Invalid Word
One Invalid Word in Next
2 Words
One Invalid Word in Next
2 Words
One Invalid Word in Next
2 Words
One Invalid Word in Next
2 Words
Two Consecutive Valid Words
Two Consecutive Valid Words
Two Consecutive Valid Words
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
125125125© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
FC-1 Constructs
• Port states
• Primitive sequencesNOS/OLS/LR/LRR
• Primitive sequence protocolsSequence flows
• Relationships
• Port state transition table
126126126© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Port States
• Four primary operational statesActive stateLink recovery stateLink failure stateOffline state
• Operational states of a portN_PortsF_Ports
• Port state changes occur as a result ofConditions detected within the portIn response to reception of primitive sequencesIn response to upper level controlling entity
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
127127127© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Primitive Sequences
• Ordered set that is transmitted continuously to indicate that specific conditions within the port are encountered
• Transmitted while the condition exist
• Four primitive sequencesNot Operational Sequence (NOS)Offline Sequence (OLS)
Link Reset Sequence (LR)
Link Reset Response Sequence (LRR)
128128128© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Primitive Sequence NOS
Not_Operational Sequence
• Transmitted by the port to indicate thatLink failure had been detected
Loss of sync
Loss of signal
Port is offline
D5.2D31.5D21.1K28.5
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
129129129© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Primitive Sequence OLS
Offline Sequence
• Transmitted by port to indicate that it is:Initiating the link initialization protocol
Receiving NOS
Entering the Offline state
D21.2D10.4D21.2K28.5
130130130© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Primitive Sequence LR
Link Reset Sequence
• Transmitted by port to indicate that it is:Initiating the link reset protocol
To recover from a link timeout
To remove a Class_1 connection
D9.2D31.5D9.2K28.5
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
131131131© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Primitive Sequence LLR
Link Reset Response Sequence
• Transmitted by port to indicate that:Link reset is being received
D9.2D31.5D21.1K28.5
132132132© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Primitive Sequence Protocols
Link Initialization Protocols• Required after
Port power-on
Port internal resetPort has been in offline state
Online to offline protocols• Required to enter offline state
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
133133133© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Primitive Sequence Protocols
Link Failure Protocol• Required after
Detection of loss of synchronization for a period of time greater than 100ms which is the receiver-transmitter time-out value (R_T_TOV)
Loss of signal while not in the offline state
Link Reset Protocol• Required after
Link resetLink timeout
134134134© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Primitive Sequence Flows
Link Failure State (LF)
Offline State (OL)
Link Recovery State (LR)
Active State(AC)
NOS
Link Failure Protocol
Li nk Init iali zat ion Pr oto co l
Link Reset Protocol
Idle
Online t o O
f fl in e Prot ocol
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
135135135© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Primitive Sequence Meanings
Idles or R_RDY
Operational Link
• Idles and R_RDY recognizedIDLE
IdlesLink Reset Response
• Link Reset RecognizedLRR
LRRLink Reset
• Remove class_1 Conn
• Reset F_Port
• OLS recognized
LR
LROffline State
• Internal port failure
• Transmitter power down, perform diags, or perform initialization
• Receiver shall ignore Link error or Link Failure
OLS
OLSNot Operational
• Link FailureNOS
Transmit in Response
MeaningCurrently Transmitting
136136136© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
AC = Activity State
LR = Link Recovery State
LF = Link Failure State
OL = Offline State
Link Failure ConditionAC AC
LFLF
LROL
LR
AC
AC
NOS
OLS
LR
Idle
LRR
Idle
Port A Port B
Link Failure
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
137137137© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Offline
AC AC
OLOL
LRLR
AC
AC
OLS
LR
LRR
Idle
Idle
Idle
Port A Port BRequest to Go Offline
After 5ms Minimum Diags May Be Preformed
Request to Go Online
AC = Activity State
LR = Link Recovery State
LF = Link Failure State
OL = Offline State
138138138© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Frame Header Detail
• Routing control (R_CTL)
• Addressing (D_ID) (S_ID)
• Type (TYPE)
• Frame control (F_CTL)
• Sequence identifier (SEQ_ID)
• Sequence count (SEQ_CNT)
• Exchange identifiers (OX_ID) (RX_ID)
• Parameter field
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
139139139© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Frame Detail: Routing Control
R_CTL Routing D_ID 24bits Destination
07815
16
23
24
31
Word
0
1
2
3
4
5
S_ID 24 bits SourceCS_CTL 8 bits Class Spec
TYPE 8 bitsData structure
SEQ_ID 8 bits DF_CTL 8 bitsData field
F_CTL 24 bits Frame Control
SEQ_CNT 16 bits Sequence Count
OX_ID 16 bits Orig Exch ID RX_ID 16 bits Respon Exch ID
Parameter Specific to frame type
140140140© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Routing Control
• The Routing control field is an 8 bit field
• R_CTL consist of two 4 bit sub-fieldsRoutingInformation category
Info CategoryRouting
31 242728
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
141141141© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Routing Control
• The R_CTL is used to direct the frame to the process the frame is directed to; For example:
Frames directed to the fabric for extended link services (0x22)
Indication of the function or purpose of the frame payload from the upper level protocol at FC-4 (0x01)
142142142© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Port Addressing
R_CTL Routing D_ID 24 bits Destination
07815
16
23
24
31
Word
0
1
2
3
4
5
S_ID 24 bits SourceCS_CTL 8 bits Class Spec
TYPE 8 bitsData structure
SEQ_ID 8 bits DF_CTL 8 bitsData field
F_CTL 24 bits Frame Control
SEQ_CNT 16 bits Sequence Count
OX_ID 16 bits Orig Exch ID RX_ID 16 bits Respon Exch ID
Parameter Specific to frame type
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
143143143© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Port Addressing
• D_ID and S_ID fields are 24 bits each
• They provide the address or identifier of theSource and destination port of a frame
• Although the address map is flat, there are several formats depending on:
Topology
Location
144144144© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Port Address Identifiers
• Applicable to all topologiesPoint to pointSwitched
Loop
• Dynamically assigned or administratively assigned
• Used for frame routingUnique within Fibre Channel network
• Assigned by the “fabric”
• Some address reserved for special functions
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
145145145© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Port Address Identifiers
Acquired During Loop InitializationArbitrated Loop
By Switch During Fabric Logon
• Bound to Physical Port on Switch
Switched
By N_Port with Higher Worldwide Name (MAC)
Point To Point
AssignmentTopology
146146146© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Address Identifiers
Arbitrated Loop Physical Address (AL_PA)0000
AL_PAAreaDomain
DeviceAreaSwitch Domain
8 bits 8 bits 8 bits
Switch Topology Model
Private Loop (Not Connected to a Switch)
Public Loop (Connected to Switch)
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
147147147© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Reserved Addresses
• FC-PH has defined a block of addresses for special functions:
High order 16 addresses in the 24 bit address spaceCalled the well known addresses
Main Address Used Today
FF FF FC Directory Server
FF FF FD Fabric Controller
FF FF FE Fabric F_Port which N_Port is attached to
148148148© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Data Structure Type
R_CTL Routing D_ID 24bits Destination
07815
16
23
24
31
Word
0
1
2
3
4
5
S_ID 24 bits SourceCS_CTL 8 bits Class Spec
TYPE 8 bitsData structure
SEQ_ID 8 bits DF_CTL 8 bitsData field
F_CTL 24 bits Frame Control
SEQ_CNT 16 bits Sequence Count
OX_ID 16 bits Orig Exch ID RX_ID 16 bits Respon Exch ID
Parameter Specific to frame type
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
149149149© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Type
• The TYPE is a 8 byte field
• Indicates the upper level carried in the payload of the frame
• Examples:SCSI ‘08h’IP ‘05h’
SNMP ‘24h’Fibre Channel services ‘20h’
150150150© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Frame Control
R_CTL Routing D_ID 24bits Destination
07815
16
23
24
31
Word
0
1
2
3
4
5
S_ID 24 bits SourceCS_CTL 8 bits Class Spec
TYPE 8 bitsData structure
SEQ_ID 8 bits DF_CTL 8 bitsData field
F_CTL 24 bits Frame Control
SEQ_CNT 16 bits Sequence Count
OX_ID 16 bits Orig Exch ID RX_ID 16 bits Respon Exch ID
Parameter Specific to frame type
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
151151151© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Frame_Control
• The frame control is a 24 bit field
• It contains a number of flags that are used to control the flow of the sequence
• The more common flags are exchange and sequence management, acknowledgement control and error conditions
Bits 16-23 deal with the sequence and exchange settings
Bits 14-15 deal with X_IDBits 13-12 form the ACK level for class 1 & 2
Bits 5-4 used for aborting the sequence
152152152© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Frame Control Bits 12-13
• Acknowledgment CapabilityProvide assistance to Sequence Recipient (SR) by translating the ACK capabilities bits in the N_Port class parameters
Meaningful only in Class 1 and 2 data frames0 0 = No ACK
0 1 = ACK level 1 –one for every frame1 0 = ACK level “N” N = number of frames
1 1 = ACK Level 0—single ACK for complete exchange, used in video streaming
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
153153153© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Sequence Identifier
R_CTL Routing D_ID 24bits Destination
07815
16
23
24
31
Word
0
1
2
3
4
5
S_ID 24 bits SourceCS_CTL 8 bits Class Spec
TYPE 8 bitsData structure
SEQ_ID 8 bits DF_CTL 8 bitsData field
F_CTL 24 bits Frame Control
SEQ_CNT 16 bits Sequence Count
OX_ID 16 bits Orig Exch ID RX_ID 16 bits Respon Exch ID
Parameter Specific to frame type
154154154© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Sequences
Sequences• Deal with chunks of upper level protocol
• Are made up of one or more frames which transport the ULP
• The data phase may be subdivided into multiple sequences
• Uniquely identifiable with SEQ_ID
• The command, data, and status phases of SCSI are examples of sequences
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
155155155© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Sequence Identifier
• The Sequence Identifier (SEQ_ID) is a 8 bit field
• All Frames of a sequence will carry the same SEQ_ID value
Data content of these frames are related in some way by the ULP
156156156© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Sequence Count
R_CTL Routing D_ID 24bits Destination
07815
16
23
24
31
Word
0
1
2
3
4
5
S_ID 24 bits SourceCS_CTL 8 bits Class Spec
TYPE 8 bitsData structure
SEQ_ID 8 bits DF_CTL 8 bitsData field
F_CTL 24 bits Frame Control
SEQ_CNT 16 bits Sequence Count
OX_ID 16 bits Orig Exch ID RX_ID 16 bits Respon Exch ID
Parameter Specific to frame type
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
157157157© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Sequence Count
• Sequence count (SEQ_CNT) is a 16 bit field
• Identifies the order of the transmission of frames within this sequence
• Used by Sequence Recipient (SR) to account for all transmitted frames
• Used by Sequence Initiator (SI) to account for all transmitted acknowledges (ACK’s) in Class 1 and 2
158158158© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Sequence Count
• Within a Sequence_InitiativeThe SEQ_CNT of the first data frame will be zero
The SEQ_CNT of each subsequent data frame in the sequence will be incremented by 1
The first data frame of the next sequence may be either zero or one more then the last data frame, this is called “continuously increasing SEQ_CNT”If streamed sequences is used, continuously increasing SEQ_CNT is required
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
159159159© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Sequence Count
• Sequence initiatorAssigns SEQ_CNT to data framesKeeps a record of ACK frames received
• Sequence recipient Records SEQ_CNT of data framesTransmits an ACK frame for each valid data frame when Rx buffer is availableKnows that sequence was received without error if all Frames are Rx without errors and are accounted for
• Sequence initiator Knows the sequence was received without error if it has Rx an ACK frame to all frames within the sequence
160160160© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Exchange Identifiers
R_CTL Routing D_ID 24bits Destination
07815
16
23
24
31
Word
0
1
2
3
4
5
S_ID 24 bits SourceCS_CTL 8 bits Class Spec
TYPE 8 bitsData structure
SEQ_ID 8 bits DF_CTL 8 bitsData field
F_CTL 24 bits Frame Control
SEQ_CNT 16 bits Sequence Count
OX_ID 16 bits Orig Exch ID RX_ID 16 bits Respon Exch ID
Parameter Specific to frame type
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
161161161© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
OX_ID and RX_ID
• 2 byte fields each
• Contain the originator exchange identifier and responder exchange identifier
• They point to state and context information regarding the exchange in the originator port and responder port
OX_ID’s are reused after each exchange is over
162162162© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Parameter Field
R_CTL Routing D_ID 24bits Destination
07815
16
23
24
31
Word
0
1
2
3
4
5
S_ID 24 bits SourceCS_CTL 8 bits Class Spec
TYPE 8 bitsData structure
SEQ_ID 8 bits DF_CTL 8 bitsData field
F_CTL 24 bits Frame Control
SEQ_CNT 16 bits Sequence Count
OX_ID 16 bits Orig Exch ID RX_ID 16 bits Respon Exch ID
Parameter Specific to frame type
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
163163163© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Parameter Field
• The parameter is a 4 byte field
• The content of the parameter field is dependent on the specific frame type as identified in the routing field
FC-4 data frames
ACK link controlPort reject and frame reject frames
Port busy and fabric busy frames
LOGIN PARAMETERS
164© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
165165165© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Login
• Exchange “service parameters” done with login frame PLOGI or FLOGI
• Required before communications can be established between the two ports
• Applies to all topologies• Applies to all ports, node and fabric• Bi-directional
ACCEPT Frame contains service parameters of the port addressed
Procedure to Determine the Operating Environment for Communications between Two Ports
166166166© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Login
• Version of Fibre Channel support• N_Port or F_Port functionality• Service classes supported• Size of receive buffers• Number of sequences supported• Support for Intermix• ACK capability• Error policy supported• Others
Service Parameters Contain the Following “Type” of Information
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
167167167© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
ACK’s
Informs Transmitter that:• One or more valid data frames were received by the
sequence recipient for the corresponding sequence qualifier
• Interface buffer is available for another data frame, this only applies to class 1 and class 2
Class 3 are not ACK’ed
• Flow controlRe-instates end-to-end credit
168168168© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
ACK’s
• Frame HeaderConstructed from the data frame which is being acknowledged
S_ID and D_ID are swappedF_CTL with both exchange and sequence context bit invertedSEQ_ID is unchangedSEQ_CNT is set to the sequence count of the highest data frame being replied to by the ACK
Parameter FieldBit 16 = History bitBits 0-15 are ACK type specific
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
169169169© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
ACK’s
• Again there are three types of ACK’sACK_1 default for class 1 and 2 one ACK sent for each SEQ_CNT
ACK_N Class 1 or 2 N=ACK sent by recipient for the support indicated during port loginACK_0 class 1 or 2 single ACK sent at end of sequence
We could spend a lot more time discussing ACK’s but there is little or no class 1 or 2 used in networks today and doubt if we will see any soon
170170170© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Busy and Reject
Port Reject P_RJT Fabric Reject F_RJT
• Transmitted by destination port or fabric in response to a specific data frame
• Applicable to only Class 1 and 2
• Sent in reply to valid frames
• Transmitted by the “receiver” of the data frame with reason code
• Indicated that the corresponding data frame was NOT delivered to the ULP
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
171171171© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Busy and Reject
• Busy sent by fabric if unable to deliver frame due to busy condition
• Busy sent by port if temporarily busy and unable to process a frame
• If F_BSY or P_BSY is sent, fabric or port give reason code
Class 1 busy only allowed on the connection request
Class 2 any frame may Rx busyClass 3, busy is not sent; If a frame can not be delivered it is discarded without notification
172172172© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Flow Control and Credit
• Flow modelFrames are moved:
From one BufferTo another Buffer
Frame Flow is:From the Source buffer andTo the destination buffer
Depending on the class of serviceMultiple intermediate buffers may be involved
Applies to:All topologies
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
173173173© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Flow Control and Credit
• Frame flow is controlled by the receiverBack-pressure mechanismACK’s class 1 and 2, RDY’s class 3
• Flow control is based on frame flowWhich frames are flow controlled is dependent on class of service
• Receiver defined parameters during the login procedure
Maximum frame size
Number of buffers
174174174© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Flow Control and Credit
Receiver• Establishes operating environment through login
Size of buffers
Number of buffers (credits) allocated to this transmitting port
• Pumps-up these creditsBy ACK’s when buffer is available
• A receive Buffer is available afterThe frame was verified to be valid, no errors
And the frame has been moved off the interface buffer
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
175175175© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Flow Control and Credit
Transmitter• Keeps
Credit maximum valueCredit_Count
• Consumes one credit for each “frame” it transmitsCredit_CNT = Credit_CNT –1 for each Data_Frame Tx
• Regenerates credit for each ACK Rx’edCredit_CNT = Credit_CNT + N
• Stops transmitting whenCredit_CNT = 0
176176176© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Flow Control and Credit
• FC-2 defines two type of creditBuffer to Buffer (BB)End-to-End (EE)
• BB credit is the flow of connectionless trafficOver a LINK from Tx to RxClass 2 and 3Signal used = R_RDY
• EE credit is the flow on connection trafficSource to destination nodeClass 1 and 2Signal used = ACK
• Both based onCreditCredit_CNT
• Differ inFrames controlled and acknowledgement signal
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
177177177© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Flow Control and Credit
Sequence Initiator
Sequence Recipient
Fabric
ACK
R_RDYR_RDY
R_RDY
R_RDYACK
EE_Credit
BB_C BB_C
TX Buf
TX Buf
TX Buf
TX Buf
RX Buf
RX BufRX Buf
RX Buf
178178178© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Class of Service
• Applicable to all fabric topologiesSwitchedPoint to point
Arbitrated loop
• These three classes of service areClass 1 dedicated connection
Class 2 connectionless multiplexed
Class 3 datagram
• Delimiters used to set required class for a sequence
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
179179179© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Class of Service
• SOF delimiterThe required class of service along with basic sequence management are specified in the SOF delimiter of every frame
The SOF delimiter dedicate basic link management functions within the fabric
The SOF delimiter identifies basic Sequence management functions within the destination N_Port in the initial frame of the sequence and the last frame of the sequence
• EOF delimiterLast frame of a sequence is terminated by a special EOF
Dedicated connections are removed by a special EOF
180180180© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Class of Service
Class 1• Dedicated connection service
Connection oriented service between two N_Ports
Frames received in order transmittedGuaranteed delivery with notification of non-delivery
Guaranteed throughput
Optional IntermixCan mix Class 2 and 3 frames if allowed
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
181181181© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Class of Service
Class 1• Requires explicit connection establishment
SOF(C1) delimiter
• Requires explicit removal of connectionACK with EOF(DT) delimiter
• Once connection is establishedBSY and RJT will not occur
• Flow controlBuffer to buffer on SOF(C1) frame: R_RDYEnd to end for all other data frames: ACK
182182182© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Class of Service: Class 1 Flow
FabricInitiator Recipient
SOF(C1)
R_RDYACK
SOF(n1)
SOF(n1)
SOF(n1) ACK
ACK
R_RDY
Connection Requested
Connection Established
EOF(t)Conn Removed
ACK
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
183183183© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Class of Service
Class 2• Multiplexed connectionless service
Connectionless oriented service between two N_Ports
Order of frame reception not guaranteedGuaranteed delivery
Notification of non-delivery
No throughput guaranteesOptional intermix
184184184© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Class of Service
Class 2• Multiplex on a frame-by-frame basis
Between different destination N_Ports
Among different sequences
• BSY and RJT may occur on any frame
• Flow ControlBuffer-to-buffer for all frames: R_RDY
End-to-end for all data frames: ACK
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
185185185© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Class of Service: Class 2 Flow
FabricInitiator Recipient
SOF(C2)
R_RDYACK
SOF(n2)
SOF(n2)
ACK
ACK
R_RDY
R_RDY
R_RDY
R_RDY
R_RDY
R_RDYR_RDY
R_RDY
186186186© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Class of Service
Class 3• Datagram multiplexed connectionless service
Connectionless oriented service between two N_Ports
Order of frame reception not guaranteedUnacknowledged
Delivery NOT guaranteed
No throughput guaranteesOptional intermix
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
187187187© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Class of Service
Class 3• Multiplex on a frame-by-frame basis
Between different destination N_Ports
Among different sequences
• BSY and RJT will not occur on any frame
• Flow controlBuffer-to-buffer for all data frames: R_RDY
188188188© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Class of Service: Class 3 Flow
FabricRecipientInitiator
Data Frame
Data Frame
R_RDY
R_RDY
R_RDY
R_RDY
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
189189189© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
EE Credit
NL_Node “A”
NL_Node “B”
EE_Credit
Switch
EE_Credit
EE_Credit
Applies Only to Class 1 and Class 2 Frames for All Topologies
190190190© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
For All Class 2 and Class 3 Frames for All Topologies
BB Credit
BB_Credit
Switch
NL_Node “A”
NL_Node “B”
BB_Credit
BB_Credit
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
FRAME PROCESSING
191© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
192192192© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Tables
• The N_Port will keep the following informationAvailable X_ID tableExchange context table
Login table
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
193193193© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Tables
Available X_ID Table• This table contains a list of available X_ID’s
Can be used for OX_IDs or RX_IDs
A given implementation may choose to keep two tables one for OX_ID and RX_ID
• When a device driver sends a request to transmit a frame, a value will be taken for the OX_ID
• When a port receives a frame for a new exchange, a value will be taken for the RX_ID
194194194© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Tables
Exchange Context Table• Each exchange ID points to a unique entry in the
exchange context table• Each entry contains the context and state
information for the particular exchangePort_ID involved in exchangeX_ID it assigned to exchangeULP and phase within the operationData source or destination addressData frames transmitted or received (SEQ_CNT)ACK frames transmitted or received (SEQ_CNT)
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
195195195© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Tables
Login Table• This table contains one entry for each port to
which this port is logged in with
• Each entry contains service parameters and working EE_Credit count value
196196196© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Data Frames: Putting It All Together
Data Frame Transmission• Request for a ULP
Initiate some operation with a specific destination port
• Login processIf you are not logged in, initiate login process
Build logging table entry for destination port
• Assign OX_ID if neededGet a value from the available X_ID Table
Build the exchange context table
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
197197197© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Data Frames
Data Frame Transmission (Cont.)• Gather information
Exchange context tableReceive buffer size and destination port
Login tableWorking credit count of destination port
Set-up frame header
• Data frame transmissionSegmentation processCredit management
198198198© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Data Frames
Transmit Request
• ULP passes a request to transmit a chunk of data to the N_Port
Destination Port D_ID is made
• The N_Port must access the “login table” to determine the service parameters on the destination port
Number of Rx buffersValue of the working credit count
And the rest
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
199199199© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Data Frames
The Data Transmission• ULP data chunk is moved in frames with the
use of the sequenceAll within the context of the exchange
• A number of processes are involvedInitialization of the frame header fieldsSegmentation and reassemble
200200200© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
First and Last Data Frames
• The first data frame of a sequence is identified bySOF(Ix) Delimiter, where ‘x’ is the Class of Service
• The last data frame of a sequence is identified byF_CTL bit 19, End_SEQ=‘1’
• A sequence consists of all data framesStarting at the SEQ_CNT for the first frame through the SEQ_CNT of the last frame
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
201201201© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Sequence Processing
Sequence Count• ULP chunk of data is transmitted IN ORDER
All Frames are sent in order
• Sequence_Count (SEQ_CNT)Frames are assigned sequentially increasing numbers as they are sent
The receiving N_Port will use the SEQ_CNT to insure that Frames are reassembled in order and back in its original chunk
202202202© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Sequence Initiator (SI)
• Sets F_CTL bit 23“0” If it is the exchange originator“1” If it is the exchange responder
OX_ID and RX_ID set to assigned values
RX_ID = “FFFF” if first sequence of exchangeRouting field (R_CTL) set to “0000” to indicate FC-4 data frameInformation category field of R_CTL set according to payload
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
203203203© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Sequence Initiator: Frame Header
• Sequence ID (SEQ_ID)Any value select that is not used
• Sequence count (SEQ_CNT)Assign sequentially as frames are sent
Starts with “0” on first frame of sequenceIncrements by ‘1’ while sequence initiative is held
• ParameterSet to ‘offset’ of the first byte of payload with respects to entire chunkOffset = ‘0’ on first frame and ‘1’ + for second and subsequent frames
204204204© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Sequence Initiator: Frame Header
• Other important F_CTL bitsBit 23, exchange contextBit 21, first sequence
Bit 20, last sequence
Bit 19, end sequenceBit 16, sequence initiative
Used to pass initiative to other device
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
205205205© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Automatic Processes
• These processes are automatic and are performed by the protocol chip
Segmentation and reassemblySEQ_CNT assignment
Higher layers are unaware of these processes
206206206© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
ULP Processing
• The Upper Level Protocol (ULP) uses these fieldsRouting ‘0000’ = FC-4 data frameType ’08 = SCSI/FCP
Info category
Identifies Specific Function of Payload‘01’ = Solicited Data
‘06’ = Unsolicited Command
‘05’ = Data Descriptor
‘07’ = Command Status
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
ARBITRATED LOOP
207© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
208208208© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Fibre Channel Arbitrated Loop (FC-AL)
• Maximum bandwidth: 100 MB/sec. (shared amongst all nodes on loop)
• 126 nodes max on loop
• Can be combined with switches
• Attaches “NL_Ports”
• Number of nodes on loop directly affects performance
• Defined in it’s own standard
Fibre Channel Hub
FC
L
L
L
L
L
L
LLL
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
209209209© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Loop Advantages
• Low cost solution with copper transceivers
• Eliminates the need for a discrete “fabric”Fabric routing decision distributed around the loop
• Compatible with all FC- 0 variantsCopper within a box
Optical between boxes
• Self discovery procedure
• Simple additions to FC-PH
210210210© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Loop Advantages
• Port bypass network
• High availability configurations possible
• Supports both public and private loops
• Provides access fairness
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
211211211© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
NL_Port
• N_PortAttaches to the physical transport mediaProvides the Fibre Channel control and protocol
Provides the termination point for Fibre Channel
Resides within the node
• NL_PortProvides all functionality on N_Port with additional function of the loopAn NL_Port can function as a N_Port
212212212© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
FL_Port
• F_PortAttaches to the physical transport media at the edge of the switched fabric
• FL_PortThe switched fabric port which attaches to a loopF_Port functionality with additional function of the loop
G and GL Ports Will Do Both N and F
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
213213213© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Private and Public
• Private LoopContains no FL_PortCommunications outside the loop via Fibre Channel is not possible
• Public LoopContains an FL_Port
Communications outside loop via Fibre Channel is possible
• Private DevicesDevices on a public loop may be private, i.e. do not login
214214214© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Addressing
• Arbitrated Loop Physical Address (AL-PA or PA)Assigned during the loop initialization (soft addressing)A unique 8 bit value
127 valid values
• Arbitrated Loop Destination Address (AL-PD or PD)The AL_PA used to identify the destination L_Port
Target of a primitive signal or D_ID of a frame
• Arbitrated Loop Source Address (AL_PS or PS)The AL_PA used to identify the source L_PortSource of a primitive signal or S_ID of a frame
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
215215215© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
The Fabric Definition
• The entity that interconnects attached N_Ports
• Provides ‘routing’ based on destination address
• Fabric may be:Point to point—No routing requiredSwitched—Routing provided by the Switch
Arbitrated loop—Routing is distributed throughout the attached NL_Ports
216216216© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Switched Fabric
N_Port N_PortN_Port N_Port N_PortN_Port
Fabric
Switch
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
217217217© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Loop
NL_Port
Node
NL_Port
Node
NL_Port
Node
Node
NL_Port
NodeNL_Port
NodeNL_Port
Fabric
LOOP
= Arbitrated Loop Additional Function
218218218© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Routing Process: Loop
• The routing function is distributedEach L_Port performs a portion of routing
• Routing is performed through out-of-band signaling using primitive signals
• Connection oriented independent of class of service
Obtain ownership of the loop (Arbitration)Establish a connection (Open)Transfer frames (Data)Remove the connection (Close)Relinquish the loop
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
219219219© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Processes and Procedures
• InitializationThe process by which addresses are assigned and recovery is performed
• ArbitrationThe process by which an L_Port acquires ownership of the loop
• OpenThe process by which the L_Port which owns the Loop uses to select the L_Port to which it wants to communicate with
• CloseThe process by which the L_Port which owns the Loop releases control
220220220© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Fill Words
• FC-PH defines two signals that may be transmitted between frames (when no other information is being transmitted)
IdleR_RDY
• FC-AL defines several additional signals that may be transmitted between frames
• FC-AL defines the “fill word” to beARB(F0)ARB(x)Idle
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
221221221© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Primitive Signals and Sequences
• Primitive signalsArbitrateOpenCloseMark
• Primitive sequencesPort bypass enablePort bypass disableLoop initialization
FC-AL Defined the Following Unique Signals and Sequences
222222222© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Credits Buffers
• Alternate BB_Credit management requested during login
• When activated service parameter BB_Credit = number of buffers available when circuit is established
• The receiving L_Port shall transmit R_RDYs for the additional buffers at anytime when “opened”
Used to pump up BB_Credit_CNT
• Transmitting L_PortDecrements BB_Credit by ‘1’ for each data frame TxIncrements BB_Credit by ‘1’ for each R_RDY RxStops transmitting when BB_Credit =‘0’
Loop Uses Same Credit Method as Previously Discussed But Also Has an Alternate Credit Model
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
223223223© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Arbitrated Loop Initialization Procedure
Purpose• An L_Port will perform the loop initialization
procedure to:Determine the Operating environment for the L_Port; Is this a loop?
Acquire an address. AL_PA (Physical Address)Report that an error has been detected
224224224© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Loop Commands
Command & AL_PA Distribute Position map
132Link Initialization – Loop PositionLILP
Command & AL_PA Collect Position map
132Link Initialization – Report PositionLIRP
Command & AL_PA bit map
20Link Initialization – Soft AssignedLISA
Command & AL_PA bit map
20Link Initialization – Hard AssignedLIHA
Command & AL_PA bit map
20Link Initialization – Previously AssignedLIPA
Command & AL_PA bit map
20Link Initialization – Fabric AssignedLIFA
Command & WWN12Link Initialization – Select MasterLISM
Payload ContentsBytesCommand
Loop Initialization Procedure—LIP Is an Ordered Set
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
225225225© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
LIP: Initialization Procedure
Phase A
Phase B
Phase C
Phase D
Phase E
Close
Start The Initialization Procedure
Select Temporary Loop
Master
AL_PA Mapping
Phase
Reporting Phase
Distribute AL_PA Map Phase
LIP
LISM
LIFA, LIPA, LIHA, LISA
LIRP
LILP
Start The Initialization Procedure
FL_Port Wins if Present Otherwise Lowest WWN Wins
Build the AL_PA bit Map in 4 Steps
Collect the AL_PA Position Map
Distribute the AL_PA Position Map
226226226© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
LIP: Phase A
Loop Initialization Primitive Sequence• Transmitted continuously by L_Port until it
receives the same LIP configurationLIP (F7F7) the L_Port is attempting to determine if this is a loop and to acquire an AL_PA
LIP (F8F7) the L_Port has detected a loop failure at its receiver prior to acquiring an AL_PA
LIP (F8) the L_Port (AL_PS) had detected a loop failure at its receiver
LIP (F7) the L_Port (AL_PS) has detected a performance degradation
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
227227227© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
LIP: Phase B
• Each L_Port will build the LISM with:AL_PA = ’00’ hex if FL_Port
’EF’ hex if NL_Port
D_ID = ‘0000’hex + AL_PA Example (0000EF)
S_ID = “0000’hex + AL_PAPayload = Command + WWN
Current Fill Word = Idle
• Each L_Port will continuously transmit a LISM
• Normal flow control rules are not in effect during initialization
228228228© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
LIP: Phase B (Cont.)
• Each L_Port monitors its receiverWill continue to transmit LISM if Your AL_PA + WWN is less then received AL_PA + WWNOtherwise pass the received LISM
• You are temporary loop masterIf the device receives a LISM identical to the one transmittedFL_Ports always win; If two or more FL _Ports; Lowest WWN wins and the others go non-participatingIf no FL_Port the NL_Port with lowest WWN wins
• Loop masterCurrent fill word would be ARB(F0)When ARB(F0)’s are received, go to phase C
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
229229229© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
LIP: Phase C
Loop Master Will Form the Initial “Bit” Map as Shown:
000000000000000000000000000000003
000000000000000000000000000000002
000000000000000000000000000000001
0000000000000000000000000000L0000
0781516232431Word
Where L = 1 Requesting F_Login of all NL_Ports
Bit Position = 127 vector corresponding to valid AL_PA’s
Word 0 bit 30 = lowest number ’00’hex
Word 3 bit 0 = high number AL_PA value ‘EF’hex
Set the bit = 1 that corresponds to it’s Fabric Assigned AL_PA
Bit Position
Lowest AL_PA
Highest AL_PA
230230230© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
LIP: Phase C
• Loop master will transmit the following three commands allowing an L_Port to choose a desired AL_PA
LIFA bit map primed with initial value
LIPA bit map primed with results of LIFALIHA bit map primed with results of LIPA
• Loop master will then transmit the LISA commandLISA bit map primed with results of LIFA allowing L_Ports which were unable to obtain their desired AL_PA to get a “soft assigned” AL_PA
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
231231231© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
LIP: Phase C
• Each NL_Port willReceive, possibly modify and retransmit the four Initialization Command framesSet the Current Fill Word (CFW) = ARB(F0)
• Modify the AL_PA bit map as followsSet one bit of the initialization command AL_PA bit maps based on history of AL_PA assignmentIf the bit map corresponding to a “desired” AL_PA has been set by an up-stream L_Port, this L_Port assumes a soft AL_PA by setting the first “0” bit=1 in the bit map of the LISA frameIf no bit positions were available in the LISA bit map, the L_Port will remain in non-participating modeAt most the bit map of one command will be modified by each L_port
232232232© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
LIP: Phase D
• The loop master will prime the AL_PA position map to:
Byte 0 = ‘01’ hexByte 1 = it’s AL_PABytes 2-127 = ‘FF’ hexThen transmit the LIRP with this position map
• Each NL_Port will:Increment the offset by one and store the offsetStore its AL_PA at the offsetRetransmit the updated LIRP frame
• The loop master will save the resulting loop position map
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
233233233© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
LIP: Phase E
• The loop master will transmit the LILP command with
Payload = AL_PA position map• Each NL_port will
Save the loop position mapRetransmit the LILP command
• When the loop master receives the LIILP command it will
Transmit a CLS and go to monitoring stateWhen each NL_Port receives a CLS they willRetransmit the CLS and go to monitoring state
Initialization Complete
234234234© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
LIP: Summary
A. LIP starts the initialization procedure
B. Select a temporary loop masterLowest AL_PA | WWN wins
C. Build a AL_PA bit mapEach L_Port indicates the AL_PA it selected in one of 4 requests by the loop master
D. Collect a AL_PA position mapEach L_Port reports its relative position from master and it’s AL_PA
E. Distribute the resulting AL_PA position map to each L_Port
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
235235235© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Arbitration
• The process by which L_port request ownership of the loop based on primitive signals
AL_PAAL_PAD20.4K28.5MSB LSB
ARB(x)
Ordered Set
236236236© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Arbitration
Loop Owner• The current loop owner is responsible for
Seeds the arbitration process with ARB(F0)
Blocks propagation of the received ARB(x) until it relinquishes the loop
• Initiates a new arbitration “window”If ARB(F0) is received by setting current fill word = IDLE
• Fairness variablesAccess
ARB_WON
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
237237237© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Arbitration Process
• When a port is arbitrating it enters the arbitrating state
• The CFW is updated to the ports ARB(AL_PA) if the CFW is:
1. IDLE2. ARB(F0)3. ARB(FF)4. Lower-priority ARB (higher value AL_PA)
• Arbitration occurs even if a loop circuit exists between another pair of ports
• Once a port starts arbitrating itMust continue to arbitrate until it winsWithdraw if it knows that another port is arbitrating
238238238© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Fairness
Access Fairness• Ports with higher-Priority AL_PA values could lock out lower
priority portsWhen they ARB they will always winLower Priority ports might never win Arbitration
• Access fairness limits how often a port can arbitrateThis is done by not arbitrating the loop until all other ports on the loop that are arbitrating have won; This is called a fair port
• Access fairness is based on “access” not “duration of usage”
Does not limit how long a port uses the loop• Fairness is recommended by the standard but not
mandatoryFL_Ports may be unfair but NL_Ports should be fair
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
239239239© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Fairness
• The fairness is controlled by the FC-AL fairness algorithm called a fairness window
Window begins when the first port wins arbitration Ends when a port discovers that it was the last arbitrating portIDLE resets the fairness windowThe variables used are
Access = 0 for fairness window openAccess = 1 when NL_Port has won arbitration
• Fair ports can only arbitrate once per windowAfter winning arbitration they wait for the end of the window before arbitrating again
• Unfair ports can arbitrate at anytime
240240240© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Open
If the Port Requires the Loop when It Wins ARB
• It sends an OPN(yx) or OPN(yy)y=destination port x=source portFull-Duplex establishes a point to point like circuit between the loop portsHalf-duplex restricts open recipient to transmit link control frames only
Cannot transmit device data framesUsed by designs that can not support simultaneous data frames Tx and Rx
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
241241241© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Open
Selecting the Destination Port• Is the intended destination port on same loop or
connected via fabric switch?If the upper 16 bits of destination field (D_ID) are all zeros the port is on this private loop
If the upper 16 bits of the source(S_ID)are all zeros then the source port is a private port and can only talk to ports on same loop
If the upper 16 bits of the D_ID are the same as the upper 16 bits of the S_ID then they are both on the same loop or both are public and attached to the same FL_Port
• If none of these are true, the destination port is not on the same loop and must be accessed via FL_Port
242242242© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Opening a Port on Same Loop
• Open Originator inserts the destination AL_PD in the OPN
• The AL_PD is obtained from the low-order 8 bits of the destination address in the frame header
• This process can be entirely by hardware
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
243243243© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Opening a Port Off the Loop
• Originator inserts AL_PD of the FL_Port ’00’ in the AL_PD field of the OPN
• The FL_Port is opened and frames are sent to the FL_Port
• FL_Port and fabric forwards the frames using the destination address field
• FL_Port can send to multiple destination ports on the loop during this OPN
244© 2004 Cisco Systems, Inc. All rights reserved.
SWITCH FABRIC OPERATION
OPT-2T019899_06_2004_X
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
245245245© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Switch Model
Port Port
Port Port
Connection Matrix
Fabric Controller
Connectionless Switch Matrix
246246246© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Worldwide Names
• Each switch element is assigned a WWN at time of manufacture
• Each switch port is assigned a WWN at the time of manufacture
• During FLOGI the switch identifies the WWN in the service parameters of the accept frame
Fabric port and
Switch element
• These address assignments can then correlate each fabric port with the switch element
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
247247247© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Switch Ports
• Four basic types of switch portsF_Port—Uses NOS/LOS to attach to single N_PortFL_Port—Uses LIP to attach 1 to 126 NL_Ports
E_Port—Uses NOS/LOS to interconnect switches (inter-link switch ISL)G_Port—Uses NOS/LOS can be a F or E port
248248248© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Fabric Addressing
• The 24 bit address is partitioned into 3 fieldsDeviceArea
Domain
• This partitioning helps speed up routing
• Switch element assigns the address to N_Ports
• Address portioning is transparent to N_Ports
DeviceAreaSwitch Domain
8 bits 8 bits 8 bits
Switch Topology Model
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
249249249© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Directory Server
• Repository of information regarding the components that make up the FibreChannel network
• Located at address ‘FF FF FC’ (Some readings call this the name server)
• Components can register their characteristics with the directory server
• An N_Port can query the directory server for specific information
Query can be the address identifier, WWN and volume names for all SCSI targets
250250250© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Directory ServerCommand Requests
• Get objectsGA_NXT—Get all next
GFT_ID—Get FC-4 types
• Register objectsRFT_ID—Register FC-4 types
• Deregister objectsDA_ID—Deregister all
These Are Some of the More Used Commands Used to Query the Directory Server
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
251251251© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Fabric Controller
• Each switch has a fabric controller
• Assigned address ‘FF FF FD’Every fabric controller in the fabric has the same addressIt is the N_Port within the switch
Responsible for managing fabric, initialization, routing, setup and teardown of Class-1 connections
• Responsible to receive request and generate responses for the switch fabric
Information must be consistent independent of which fabric controller responds to a request
252252252© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Extended Link Services
• Extended link services provide a set of protocol functions used by the port to specify a function or service at another port
Usually sent from N_Port to F_port to perform needed requestThe R_CTL field of the first word will be set to 0x22 to indicate an extend link service requestMany ELS services will return a payload in response some have no reply
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
253253253© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Extended Link Services
• Some of the more important and most used ELS commands are:
FLOGI F_Port Login PLOGI N_Port Login
FAN Fabric Address Notification
PRLI Process Login PRLO Process Logout
SCN State Change Notification SCR State Change Registration
RSCN Registered State Change Notification
254254254© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
ELS: FLOGI
• FLOGI—Fabric loginIssued by N_Port to destination ‘FF FF FE’ to
Determine if fabric is present
Establish a session with the fabric
Exchange service parameters with the fabricFLOGI assigns N_Ports 24 bit address to N_Port or AL_PA to loop ports
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
255255255© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
ELS: PLOGI
• PLOGI—N_Port loginEstablished sessions between two N-PortsRequired before upper level protocol operations can begin
N_Port will register to the name server ‘FF FF FC” in fabric with all required login parameters N_Port will then query name server for other N_Ports on the fabric
256256256© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
ELS: PRLI
• PRLI—Process LoginAllows the FC-4 levels to exchange service parameters for communications between each other
Process is protocol specific (type field)
SCSI-3 FCP mapping requires PRLI
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
257257257© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
ELS: FAN
FAN—Fabric Address Notification• Used in fabric loop attached topology
• Provides mechanism for FL_Port to notify NL_Ports of addresses and names of FL_Ports along with fabric name
• Allows NL_Ports to verify configuration following a loop initialization
258258258© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
ELS: SCN
SCN—State Change Notification• Provides notification to ports of events that may
effect logins or process logins to ports on the fabric• SCN can be sent from
N_Port to N_PortN_Port to fabric controllerFabric controller to N_Ports
• Notification may indicate login session is no longer valid
Loss of signal (NOS, LOS, FLOGI)LIP has occurredSCN sent to fabric controller
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
259259259© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
ELS: RSCN
RSCN—Registered State Change Notification• Similar to SCN but only sends change notice
to those ports registered
• SCN did not define a registration method
260260260© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Class_F Service
• Communications between switch elements use Class_F Service
Unique SOF delimiter and normal EOF delimiter
• Used to pass control information within the switch
• Highest priority within switch
• Connectionless service
• Has no meaning outside switch, N_Port will discard if received
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
261261261© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Inter-Switch Link
• The interconnection between switches is called the inter-switch link
E_Port to E_Port
• Supports all classes of serviceClass 1, 2, 3, and switch to switch control traffic, class F
• FC-PH permits consecutive frames of a sequence to be routed over different ISL links for maximum throughput
262262262© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Interswitch Links (ISLs)
• Inter-switch link (ISL) connects switches
• Fabric parameters must match on both switch otherwise link would not come up and fabric will be segmented
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
263263263© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Principal Switch Selection
• Only one switch is designated principal switch in a fabric
• The switch with the lowest WWN becomes the principal switch originally
• Principal switch makes sure that no new switch is added to the fabric if it has a domain ID conflict with an existing switch in the fabric
Switch 1
Switch 3
Switch 4 Switch 5
Switch 6
Switch 2
264264264© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Fabric Configuration Process
• The fabric configuration process enables a switch port to determine its operating mode, exchange operating parameters, and provides for distribution of addresses
• The process is summarized in the following stepsEstablish link parameters and switch port operating mode
Principal switch selectionDomain ID distribution
Path selection
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
265265265© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Fabric Configuration Stages
Switch is operations with routes established
Path selection (FSPF) is defined in the next section
Switch has a Domain_ID
Path Selection
Switch has a Domain_IDSwitch requests a Domain_ID from the Domain Address Manager
Domain Address Manager had been selected
Domain ID Acquisition
The Principle Switch is selected
Switch_Names are exchanged over all ISLs to select a Principle Switch, which becomes the Domain Address Manager
BF or RCF SW_ILS transmitted or received
Select Principle Switch
Switch Port mode is known. If a Port is an E port, link parameters have been exchanged & Credit has been initialized.
The Switch Port attempts to discover whether it is an FL, F, or E port.
Switch Port has achieved word synchronization
Establish Link Parameters and Switch Port Operating Mode
Ending
Condition
ProcessStarting
Condition
Operation
266266266© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Fabric Configuration: PS Selection
• A principal switch shall be selected whenever at least one inter-switch link (A link between two E_Port) is established
• The selection process chooses a principal switch, which is then designated to assign domain identifier to all the switches in the fabric, and any who join later the fabric later on
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
267267267© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Fabric Configuration: PS Selection
• The principal switch selection can be triggered by anyone of the following events
Switch boot and EFPBuild Fabric (BF)
Reconfigure Fabric (RCF)
268268268© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Fabric Build Process
• When the switch first boots up and the first E_Port of a switch becomes operational, the switch starts 2xF_S_TOV timer and then sends out an exchange fabric parameters (EFP) from that port containing its own Destination ID (DoID) in the list trying to become Principle Switch (PS)
• The switch receiving the Exchange Fabric Parameter (EFP) replies with either ACCept or ReJecT after comparing the priority and WWN
0x11 Record Len(0x10) Payload Len
Reserved Priority
Principal Switch WWN (Word 0)
Principal Switch WWN (Word 1)
Domain_ID record 0
Domain_ID record M
A (0) (FF, Aa)
B (0) (FF, Bb) C (0) (FF, Cc)
E_Port E_Port
Domain_id
Priority
WWN
EFP EFP
E_Port
EFP
E_Port
EFP
EFP
D (0) (FF, Dd)
EFP EFPEFP
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
269269269© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Fabric Build Process
• If the received information has a lower value, the switch keeps the received information and considers sending switch as potential principal switch and also consider that link to be potential upstream link
• At that point switch generates another EFP for all other links with the updated potential principal switch
• When 2x F_S_TOV expired, all switches in the fabric consider the information collected for the principal switch to be definitive; At that point the principal switch is responsible for assigning the Domain_IDs
A (0) (FF, Aa)
B (0) (128, Aa) C (0) (FF, Aa)
E_Port E_Port
Domain_id
Priority
WWN
SW_RJT
SW_RJT
E_Port
SW_ACC
Potential Upstream port
SW_ACC
E_Port
EFP EFPEFP
Potential Upstream port
D (0) (FF, Bb)
270270270© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Fabric Configuration Details
• After the principal switch selection, the PS will change its priority to 0x02 (PS Priority) and then assign itself a domain ID and then the process of domain distribution starts
• The principal switch will initialize the process by sending Domain ID Assigned (DIA) SW_REQ out of all its E_Port
• The intermediate switch is actively involved in this process• Each switch will reply back with Request Domain ID (RDI)
To allow each switch to request for one or more domain ID
The neighboring switch receiving RDI will be able to identify its downstream principal ISL
• Each switch can send many RDI but once the principal switch has granted the domain ID to the switch, the following RDI from the switch must request the same set of domain_id
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
271271271© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
A (1) (XX, Aa)
B (2) (FF, Aa) C (4) (FF, Aa)
E_Port E_Port
DIA
DIA
E_PortUpstream port
E_PortUpstream port
D (3) (FF, Aa)
A B DDIA (SW_REQ)
SW_ACCSW_RJT
RDI (SW_REQ)
SW_ACCSW_RJT DIA (SW_REQ)EFP (SW_REQ)Contains DoID list
SW_ACCSW_RJT
RDI (SW_REQ)
RDI (SW_REQ)
SW_ACCSW_RJT SW_ACC
SW_RJTEFP (SW_REQ)Contains DoID list EFP (SW_REQ)Contains DoID list
Fabric Configuration Flows ID Assignment
SW_ACC
SW_ACC
SW_ACC
BD
A
272272272© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Fabric Configuration: The PS Battle
• After the principal switch selection and domain id assignment, all switches in the fabric will start two processes
FC_ID assignmentFSPF path selection
• When a new switch is added to the fabric, it will send out an EFP with its local value (I am PS); the fabric rejects that EFP and replies with DIA telling the new switch to send RDI; the RDI is then routed to the current PS
• If the new switch is part of another fabric (it also has a PS) then both fabric sends out an EFP and after comparing the DoID list the fabric enters one of the following states
BF state: If the DoID list does not overlapRCF State: If the DoID list overlapIsolation: No auto-reconfigure state or RCF disabled
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
273273273© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Fabric Configuration Disruptive/Non-Disruptive
• One of the following three conditions can trigger BF (non-disruptive) or RCF (disruptive)
Two disjoints fabric are combined togetherA principal ISL fails (upstream or downstream)A switch with Domain_ID request for another Domain_ID
• Whenever a switch receives a BF/RCF, the switch starts F_S_TOV timer and enters the BF/RCF state; It forwards BF/RCF out of all E_ports except the incoming port (only once) and wait for the timer to expire
• When the timer expires, BF/RCF propagation state is left and principal switch selection begins
• BF is not a disruptive process• RCF is a disruptive process
274274274© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Propagation of BF or RCF Requests
Fabric Configuration Distribution
Switch Starts theReconfig
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
275275275© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Fabric Configuration: Reserve ID’s
• N_ports and E_ports get one port ID; F ports don’t get any ID’s; FL ports in public AL gets 0x00 port ID
Fabric F_Port, Fabric Login databaseFEFFFF
N_Port of fabric controllerFDFFFF
Well Known AddressF0-FC,FFFFFF
Reserved00-EFFFFF
Reserved00-FFFD-FEFF
ReservedF0-FFFCFF
N_Port of domain controller. Port ID is the domain ID01-EFFCFF
Reserved00FCFF
Multicast & Broadcast00-FFFBFF
Reserved00-FF00-FAFF
Reserved00-FF00-FFF0-FE
N_Port & E_Port. Port ID=00 for FL port for public devices 255 address00-FF00-FF01-EF
Reserved00-FF01-FF00
ReservedNonAL_PA0000
Private Loop NL_PortAL_PA0000
Used during FLOGI000000
DescriptionPort_IDArea_IDDomain_ID
276276276© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Fabric Configuration: FSPF
• FSPF stands for fabric shortest path first
• Based on link state protocol
• Begins after domain ID assignment is completed
• Conceptually based on open shortest path first (OSPF) internet routing protocol
• Currently a standard defined in FC-SW-2
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
277277277© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Fabric Configuration: FSPF
• FSPF has four major componentsHello protocolReplicated topology database
A path computation algorithm
Routing table update
• FSPF discovers the paths to switches using Domain—Ids
• Each switch performs its own shortest path calculations
278278278© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Fabric Configuration: FSPF
• For FSPF a domain ID identifies a single switchThis limits the max number of switches that can support in the Fabric to 239 when FSPF is supported
• FSPF performs hop-by-hop routing
• FSPF supports hierarchical path selectionProvides the scalable routing tables in large topologies
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
279279279© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Fabric Configuration: FSPF
• Everyone says HELLO to their neighbor, on all initialized ISLs
• The neighbors say HELLO back, unless they are dead
• When the HELLO packet is received with both originator and recipient domain id, the two way communication is done and:
The ISL is active
The ISL may be available as a two-way path for frames
280280280© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Fabric Configuration: Hellos
• Point to Point Only• Default Hello Int = 20 S• Default HelloDead Int = 80 S
Hello Protocol
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
281281281© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Fabric Configuration LinkState Update and Ack
• After a 2-way HELLO is established on a Link, each switch exchanges its entire database with its neighbor using the LSU service
• When the recipient of the LSU has processed the database, it sends back the LSA service
BALSU(DB-A)
LSU(DB-B)
LSA(DB-B)
LSA(DB-A)
282282282© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Fabric ConfigurationLink State Record
• When the databases are in sync, each switch sends its LSR with the new link included using the LSU service
• The LSU is flooded to the entire fabric
• Each Switch retransmits the LSU by a mechanism called “reliable flooding”
BALSU(LSR-A)
LSU(LSR-B)
LSA(LSR-B)
LSA(LSR-A)
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
283283283© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Fabric Configuration
• Link cost is calculated based on baud rate of the link, plus an administratively set factor
• Link cost = S * (1.0625E12/baud rate)S is administrative factor defaults to 1Ex: Link cost of 1G port = 1000
• Path cost is the sum of the traversed link costs
• Lower metric more desirable
284284284© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Fabric Configuration
FSPF routes are fully functional
Routes are calculated and set up within each switch. Links are maintained by sending HELLOs every Hello_Interval. Link databases are maintained by flooding link updates as appropriate
Initial Database Exchange has been completed
Running State
Link State Databases have been exchanged
LSU SW_ILS frames are exchanged containing the initial database
Two communication has been established
Perform Initial Database Exchange
Two way communication has been established
HLO SW_ILS frames are exchanged on the link until each switch has received a HELLO with a valid neighbor Domain field
The switch sending HELLO has a valid Domain_ID
Perform initial HELLO Exchange
Ending Condition
ProcessStarting Condition
Operation
Path Selection (FSPF) Operation Summary
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
285285285© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
FSPF Characteristics
• Uses FSPF as the routing algorithm
• FSPF routes traffic based on destination domain ID
• FSPF uses total cost as the metric to determine most efficient path
• Static routes can be applied
286286286© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
FSPF Characteristics
Paths:• Finds the shortest path to each domain, then programs the
hardware routing tablesRoutes:• Dynamically
Round robin• Statically
Administrator can configure the routeAutomatically re-routes upon ISL going away and static routing will again take effect upon ISL return
• Automatic failover• Fault detection 150 ms• Self heals in 500 ms• So, alternate route is live in 650 ms
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
287287287© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Routing Software Configurable Parameters
• Link cost
• Static routes
• In Order Delivery (IOD)
• Timers (be careful)
288288288© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
What Is a Route and Path?
• A route is map between the input and output E_port used to reach the next switch
• A path is a map through the topology between a source and destination
FCFC
FCFCFCRoutePathISL
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
289289289© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Selecting a Path
• Each inter switch link has a cost metric• The cost of an ISL is related to the bandwidth• The total cost of a path between two switches is the sum
of the cost of all the traversed ISLs• The path to a destination switch is the one with the
minimum total cost• More than one path can be selected (with the same cost)
Cost 500
Cost 250Cost 250
FCFCFC
FCFC
290290290© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
ISL Oversubscription
• Oversubscription occurs when more nodes can contend for the use of one ISL
• Oversubscription ratio is the number of different ports that contend for the use of one ISL
• This a 3:1 over subscription
Multiple Nodes
Switch
Switch
ISL
1G 1G
1G
1G
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
291© 2004 Cisco Systems, Inc. All rights reserved.
FC ERROR MANAGEMENT
OPT-2T019899_06_2004_X
292292292© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Timers
• Four different timers usedReceiver-transmitter time-out (R_T_TOV)Error detect time-out (E_D_TOV)
Resource allocation time-out (R_A_TOV)
Connection request time-out (C_R_TOV)Used in Class 1 You will never see class one
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
293293293© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Timers: R_T_TOV Receiver-Transmitter Time-out
• Used to time events at the link levelLoss of synchronizationTimes Responses for link reset protocol
• Generally controlled in hardware for all link configurations
Default value in FC Standard is 100ms
294294294© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Timers: E_D_TOVError Detect Time-out
• Timers for events and responses at the sequence level
Missing ACK or R_RDY when buffer credit has reached zero
Class 1 or 2 expects response from data frames
N_Port logout
• Timer value is set at fabric login to accommodate the network environment for better scaling according to delivery time of frames
Default is 10 sec
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
295295295© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Timers – R_A_TOVResource Allocation Time-out
• Time-out value for how long to hold resources associated with a failed operation
Needed to free shared resources for reuse
• Value to determine how long a port needs to keep responding to a link service request before an error is detected
R_A_TOV is 2 times E_D_TOVDefault setting in Pt to Pt is 20 sec and fabric is 120 seconds
296296296© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Timers: CR_TOVConnection Request Time-out
• Determines how long the fabric can hold a class-1 request in the queue during connection establishment
• Allows for separation of the time in a stacked queue from the E_D_TOV; This separates queuing time from frame transit time
• Helps in controlling F_BSY issues
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
297297297© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Recovery: Class 3
• Errors in class 3 sequence can only be detected by the Sequence recipient because there are no ACKsor rejects in class 3
• Class 3 SR will discard single or multiple frames until the exchange is terminated
• The upper level recovery may retransmit the entire Sequence or at least the sequence following the error detection
298298298© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Recovery: Class 3
• Errors a class 3 operation can detect:Out of order delivery and potential missing frame based on SEQ_CNT
If missing frame is not Rx’ed within E_D_TOV
Indication of a new sequence when last frame of previous Sequence has not been received (in-order delivery set)
Relative offset not in order with an order delivery set
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
299299299© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Abort Sequence: ABTS
• ABTS protocolUsed to terminate sequence or exchangeTransmitted by the sequence initiator
Can be requested by the sequence recipient by setting bits within the F_CTL of the ACK frameSame class of service delimiter as the sequence being aborted is used for ABTS frame
300300300© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Timers
• Four different timers usedReceiver-transmitter time-out (R_T_TOV)Error detect time-out (E_D_TOV)
Resource allocation time-out (R_A_TOV)
Connection request time-out (C_R_TOV)Used in Class 1 You will never see class one
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
301301301© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Timers: R_T_TOV Receiver-Transmitter Time-out
• Used to time events at the link levelLoss of synchronizationTimes Responses for link reset protocol
• Generally controlled in hardware for all link configurations
Default value in FC Standard is 100ms
302302302© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Timers: E_D_TOVError Detect Time-out
• Timers for events and responses at the sequence level
Missing ACK or R_RDY when buffer credit has reached zero
Class 1 or 2 expects response from data frames
N_Port logout
• Timer value is set at fabric login to accommodate the network environment for better scaling according to delivery time of frames
Default is 10 sec
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
303303303© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Timers – R_A_TOVResource Allocation Time-out
• Time-out value for how long to hold resources associated with a failed operation
Needed to free shared resources for reuse
• Value to determine how long a port needs to keep responding to a link service request before an error is detected
R_A_TOV is 2 times E_D_TOVDefault setting in Pt to Pt is 20 sec and fabric is 120 seconds
304304304© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Timers: CR_TOVConnection Request Time-out
• Determines how long the fabric can hold a class-1 request in the queue during connection establishment
• Allows for separation of the time in a stacked queue from the E_D_TOV; This separates queuing time from frame transit time
• Helps in controlling F_BSY issues
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
305305305© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Recovery: Class 3
• Errors in class 3 sequence can only be detected by the Sequence recipient because there are no ACKsor rejects in class 3
• Class 3 SR will discard single or multiple frames until the exchange is terminated
• The upper level recovery may retransmit the entire Sequence or at least the sequence following the error detection
306306306© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Recovery: Class 3
• Errors a class 3 operation can detect:Out of order delivery and potential missing frame based on SEQ_CNT
If missing frame is not Rx’ed within E_D_TOV
Indication of a new sequence when last frame of previous Sequence has not been received (in-order delivery set)
Relative offset not in order with an order delivery set
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
307307307© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Abort Sequence: ABTS
• ABTS can be sent under abnormal conditionsEnd-to-end credits not requiredSequence initiative not required
Open sequence not required
Maximum number of concurrent sequences allowedUnidirectional for class 1 connection
The reply to an ABTS is a Basic_Accept
iSCSIRFC 3720
308308308© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
309309309© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Session Modules
• What is iSCSI and what is the big picture?
• iSCSI protocol Introduction
• The iSCSI connection
• Security, data integrity and error recovery
• iSCSI protocol details in-depth
• Simple iSCSI connection flows
• Service location protocol for IP storage
310310310© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
What Is iSCSI?
• A SCSI transport protocol that operates on top of TCP
Encapsulates SCSI-3 CDBs (Control Descriptor Blocks) and Data into TCP/IP byte-streams (defined by SAM-2—SCSI Architecture Model 2)
Allows IP hosts to access IP or Fibre Channel-connected SCSI targets
Allows Fibre Channel hosts to access IP SCSI targets
• Standards statusRFC 3720 (assigned May 2004)
Major industry support (Cisco, IBM, EMC, HP, Microsoft)
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
311311311© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Storage Technology SCSI Domain
• To be functional, a SCSI Domain needs to contain a SCSI device that contains a target and a SCSI device that contains an Initiator
SCSI DeviceInitiator
Service DeliverySubsystem
SCSI DeviceTarget
Port
Port
Device Service RequestDevice Service Response
Task Request
Task Response TaskManager
LogicalUnit 1DeviceServer
ApplicationClient
ApplicationClient
312312312© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
SAN, NAS, iSCSI Comparison
Adopted from IBM Redbook “IP Storage Networking: IBM NAS & iSCSI Solutions”
DAS
SCSI
Computer System
SCSI Bus Adapter
SCSI Device Driver
Volume Manager
File System
Application
SAN
SAN
FC
Fibre Channel HBA
SCSI Device Driver
Volume Manager
File System
Application
iSCSI Appliance
IP
File System
Application
SCSI Device DriveriSCSI DriverTCP/IP stack
NIC
Volume Manager
NICTCP/IP stackiSCSI layerBus Adapter
iSCSI Gateway
IP
FC switch
File System
Application
SCSI Device DriveriSCSI DriverTCP/IP stack
NIC
Volume Manager
NICTCP/IP stackiSCSI layerBus Adapter
NAS
IP
NICTCP/IP stack
I/O Redirector
File System
Application
NFS/CIFS
NICTCP/IP stackFile System
Device driver
File I/O
Block I/O
Block I/O
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
313313313© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
IP Storage Networking
• IP storage networking provides solution to carry storage traffic within IP
• Uses TCP: A reliable transport for delivery• Can be used for local data center and long haul applications• Two primary protocols:
IP TCPTCP FCIPFCIP FCFC SCSI Data
IP TCPTCP iSCSIiSCSI SCSI Data
iSCSI—IP-SCSI—Used to Transport SCSI CDBs and Data within TCP/IP Connections
FCIP—IP-SCSI—Used to Transport SCSI CDBs and Data within TCP/IP Connections
314314314© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Initiator and Target Model for iSCSI
• Initiator—SCSI device which is capable of originating SCSI commands and task management requests
• Target—SCSI device which is capable of executing SCSI commands and task management requests
iSCSIGateway
iSCSIInitiator
iSCSITarget
FCTarget
FC Initiator
iSCSIGateway
FCInitiator
FCTarget
iSCSITarget
iSCSI Initiator
iSCSI Target Mode iSCSI Initiator Mode
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
315315315© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
iSCSI Components
• iSCSI is an end-to-end protocol
• iSCSI has human readable SCSI device (node) naming
• iSCSI includes the following base components: IPSEC connectivity securityAuthentication for access configuration
Discovery of iSCSI nodesProcess for remote boot
iSCSI MIB standards
316316316© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
iSCSI: Internet SCSI PDU
• The iSCSI layer encapsulates the SCSI CDB into a iSCSI Protocol Data Unit (PDU) and forwards it to the Transmission Control Protocol (TCP) layer
• It also extracts the CDB from an iSCSI PDU received from the TCP layer, and forwards the CDB to the SCSI layer
• iSCSI mapping provides the SCSI-3 command layer with a reliable transport
• The communications between the Initiator and target will occur over one or more TCP connections
• The TCP connections form a session and will carry the iSCSI PDU’s; the sessions are given an ID called a connection ID (CID); there are two parts of the ID, Initiator Session ID (ISID) and Target ID (TSID) and together make up an “I_T nexus”
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
317317317© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Data Server HostInitiator iSCSI Target
FC Storage DeviceFC Target
ApplicationClient
requests data from
LUN 1
ApplicationClient
requests data from
LUN 1
Port
Port
Port Device Service Request
Device Service Response
LogicalUnit 1
LogicalUnit 1
DeviceServerDeviceServer
LogicalUnit 2
LogicalUnit 2
DeviceServer
ge2
ge2
fc1
fc1
Target Mapping
LUN 1 = LUN 2LUN 1 = LUN 2
iSCSI Model
SCSI CDB’s Carried in iSCSI PDU’s
SCSI CDB’s carried by Fibre Channel Exchange and Sequences
318318318© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
IPIP
iSCSI Stack
SCSI Applications (File Systems, Databases)SCSI Applications (File Systems, Databases)
SCSI Block CommandsSCSI Block Commands SCSI Stream CommandsSCSI Stream Commands Other SCSI CommandsOther SCSI Commands
iSCSISCSI Over TCP/IP
iSCSISCSI Over TCP/IP
TCPTCP
EthernetEthernetFibre ChannelFibre ChannelParallel SCSI InterfacesParallel SCSI Interfaces
FCP SCSI Over FCFCP SCSI Over FCParallel SCSI TransportParallel SCSI Transport
SCSI Commands, Data, and StatusSCSI Commands, Data, and Status
SCSI Device-Type Commands
SCSI Generic Commands
SCSI Transport Protocols
Layer 3 Network Transport
Layer 2 Network
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
319319319© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Sourced Port Destination PortDestination Port
Sequence Number
Acknowledgment Number
Window
Checksum
Offset Reserved U A P R S F
Options and padding
Urgent Pointer
DestinationAddress
SourceAddress Type IP TCP DataData
46–1500 bytes
8 6 6 2
FCS
4 Octet
Preamble
TCP Header
iSCSI encapsulated
iSCSI encapsulated
OpcodeOpcode Opcode Specific FieldsOpcode Specific Fields
Length of Data (after 40Byte header)Length of Data (after 40Byte header)
LUN or Opcode-specific fieldsLUN or Opcode-specific fields
Initiator Task TagInitiator Task Tag
Opcode Specific FieldsOpcode Specific Fields
Data Field …Data Field …
Well-known Ports:21 FTP23 Telnet25 SMTP80 http
3260 iSCSI3260 iSCSI
iSCSIiSCSI Packet
320320320© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
iSCSI Naming and Discovery RFC 3721
• Initiator and target require iSCSI namesName is location independentiSCSI node name = SCSI device name of iSCSI device
Associated with iSCSI nodes, not adapters
Up to 255 byte displayable/human readable string (UTF-8 encoding)
Use SLP, or iSNS, or query target for names (sendtargets)
• Two iSCSI name types:iqn—iSCSI qualified name
eui—Extended Unique Identifier (IEEE EUI-64)
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
321321321© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
iSCSI Name Structure
iSCSI Name Structure
Type . Unique String
iqn.1987-05.com.cisco.1234abcdef987601267da232.bettyiqn.2001-04.com.acme.storage.tape.sys1.xyz
Type Date OrganizationNaming Authority
Subgroup Naming Authority or String Defined by Organization Naming Authority.. .iqn
eui
Date = yyyy-mm when Domain Acquired Reversed Domain Name
Type EUI-64 Identifier (ASCII Encoded Decimal).eui.02004567a425678d
Host Name
322322322© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
iSCSI Naming and Addressing Terms
• iSCSI host nameName of computer
• iSCSI initiator name (iSCSI Node)Name created at iSCSI driver load time on host system
• Initiator—Target Session ID (SSID)One or more TCP connections between Initiator and target; This session ID is derived from iSCSI host name, iSCSI target name and TSID, ISID
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
323323323© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
iSCSI Naming and Addressing Terms
• iSCSI initiator addressIP address on Initiator interface; Initiator can have multiple addresses
• Initiator port—Also known as network portalIP address on initiator no port number assigned, again Initiator can have several network portals
• Target port—Also known as network portalIP address + TCP port number on target interface
There can be more then one target interface
324324324© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
iSCSI Naming and Addressing Terms
• iSCSI target nameUsed to identify multiple SCSI targets behind a single IP address+port; This name is globally unique
• Initiator session IDThis is an initiator-defined session identifier; It will be the same for all connections within a session; An iSCSI initiator port is uniquely identified by the value pair (iSCSI Initiator Name, ISID)
• Target session IDTarget assigned tag for a session with a specific named initiator that, together with the ISID uniquely identifies a session with that initiator
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
325325325© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
iSCSI Naming and Addressing Terms
• iSCSI network entity—Client
• It is a combination of the following:
iSCSI initiator
iSCSI hostiSCSI initiator address
Initiator port (network portal)
• iSCSI network entity—Server
• Is a combination of the following:
iSCSI target name
Target port (network portal)
Initiator—target session (SSID)
326326326© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
iSCSI Naming and Addressing Terms
• iSCSI NodeiSCSI Initiator or iSCSI Target; There can be one or more iSCSI nodes in a network entity
iSCSI node will equal
iSCSI initiator nameiSCSI target name
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
327327327© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
iSCSI Naming and Addressing Terms
• Portal GroupGroups multiple TCP connections across the same session that is is sent across multiple portals
The portal groups are identified by a portal group tag (1-65535)One or more portal groups can provide a path to the same iSCSI node (target node or initiator node)SendTargets requires portal group tag
328328328© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
iSCSI Discovery Methods
• Small networksStatic configuration, initiators and targets‘SendTargets’ command makes configuration easier
• Medium-sized networksService Location Protocol (SLP multicast discovery)
• Large-sized networksiSNS (internet storage name service)
Includes soft zone domains
Includes database for ongoing management
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
329329329© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
iSCSI Architecture
Network Portal10.1.30.1
Network Portal10.1.40.1
iSCSI Node(Initiator)
Network Portal10.1.30.2
Network Portal10.1.40.2
iSCSI Node(Target)
iSCSI Node(Target)
Network Entity (iSCSI Client)
Network Entity (iSCSI Server)
330330330© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
iSCSI Architecture
Network Portal10.1.30.1
Network Portal10.1.40.1
iSCSI Session (Target Side)iSCSI Name + TSID=2
iSCSI Target NodeNode(within Network Entity)
Portal Group 1
Network Portal10.1.50.1
Portal Group 2
iSCSI Session (Target Side)iSCSI Name + TSID=1
IP Network
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
331331331© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
iSCSI Session Model
• An iSCSI session exists between a single iSCSI initiator (host) and a single iSCSI target (iSCSI router)
• An iSCSI session consists of one or more iSCSI (TCP) connections• Login phase begins each connection• Deliver SCSI commands in order
iSCSI (TCP) Connection
iSCSI Session
iSCSIStorage Router
iSCSI Routing Instance
TCP/3260TCP/3260TCP/3260
332332332© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
iSCSI Session Images
• Across all connections within a session, an initiator sees one “target image”
• The target image would represent all identifying elements such as LUN’s
• A target also sees one “initiator image” across all connections within a session
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
333333333© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
3.3.3.3 4.4.4.4 5.5.5.5
Target-1 Target-2 Target-3
Target-5Target-4
iSCSI Target name configured on iSCSI
Device
iSCSI Node
s
Target Port These Network Portals listens for iSCSI connections
on WKP 3260
iSCSI Network Entity-Server
iSCSI Initiator address
Initiator Port
iSCSI Host NameiSCSI Initiator Name
IP TCP Connection
TSID
ISID
iSCSI Driver, Storage NIC
Storage Systems
Configuration will make the
connections between storage
and iSCSI Initiator
SSID
1.1.1.1 2.2.2.2
mike.cisco.com
disk.cisco.com.stor.123
Put It All Together for iSCSI
334334334© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
iSCSI Connections and SCSI Phases
• A SCSI command and its associated data and status phase exchanges must traverse the same TCP connection
• Linked SCSI commands can traverse separate TCP connections for scalability
iSCSI (TCP) Connection 1iSCSI (TCP) Connection 1
iSCSI Session
iSCSI Storage Router
iSCSI RoutingInstance
SCSI Command (1) (Read)SCSI Data (1)
SCSI Status (1)
iSCSI (TCP) Connection 2iSCSI (TCP) Connection 2
Linked SCSI Commands
SCSI Command (1) (Write)SCSI Data (1)
SCSI Status (1)
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
335335335© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
iSCSI Connection Session
• iSCSI Session has four phasesInitial login phaseSecurity authentication phase
Operational negotiation phase
Full featured phase
Session Can Process SCSI Commands and Data after Login Is Complete
336336336© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
iSCSI Session Establishment
Login Begins with the First Connection• Initial login phase
Initiator sends login with text strings for InitiatorName, TargetName, and authentication options (which are then selected by the target)
• Security authentication phaseAuthentication text exchanges (ID, password, certificates, etc)
• Operational negotiation phaseEach side (initiator and target) negotiate the supported options using Keyword=value, or Keyword=value,value,value
Amount of unsolicited bufferTypes of data deliverySolicited, unsolicited, immediate, etc…
• Full featured phaseCan carry SCSI CDBs/data, task management, and responses
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
337337337© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
iSCSI Session Key Points
Sessions:• iSCSI Session = a group of TCP connections linking
an initiator with a target (i.e., can be one or more connections)
• NOTE: A TCP connection that is part of an iSCSI session will only be used to carry iSCSI traffic
• The iSCSI initiator and target use the session to communicating iSCSI commands, control messages, parameters, and data to each other
• TCP connections can be added and removed from a session using the iSCSI Login/Logout commands
338338338© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
iSCSI Sessions
• During session establishment, the target identifies the SCSI initiator port (the “I” in the “I_T nexus”) through the value pair (InitiatorName, ISID)
• Any persistent state (e.g., persistent reservations) on the target associated with a SCSI initiator port is identified based on this value pair
• Any state associated with the SCSI target port (the “T” in the “I_T nexus”) is identified externally by the TargetName and portal group tag and internally in an implementation dependent way
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
339339339© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
iSCSI Connection Allegiance
• For SCSI commands that require data transfer, the data phase and status phase must be sent over the same TCP connection used by the command phase
• Consecutive commands that are part of a SCSI task may use different connections within the session (linked commands)
• Connection allegiance is strictly per-command and not per task
• Multiple connections allow the iSCSI session to be scaled across multiple links/devices
340340340© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
iSCSI Connection Termination
• Session may end with logout or I/O error causing dropped connection
• TCP connections are closed through normal methods i.e. TCP FINs
• Graceful shutdowns can only occur when no outstanding tasks are on the connection and not in full-feature phase
• Termination of connection abnormally may require a recovery method by logout request for all connections; This prevents stale iSCSI PDU’s being received after going down
• Logout can also be issued by the target through asynchronous message PDU
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
341341341© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
iSCSI Security
• Two types of security IPSec secures TCP/IP nodes; setup at TCP/IP startup—before iSCSI login
Session authentication via IKE (Internet Key Exchange)
Packet by packet authentication (also provides Integrity)Privacy via encryption (also provides Integrity)
See SEC-IPS iSCSI techniques (done/setup during iSCSI Login)
Authentication (ensures nodes are authorized to use the iSCSI target node) may use SRP, Chap, or Kerberos
342342342© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Challenge Handshake Authentication Protocol
• In-band initiator-target authentication
• IP-SEC is not assumed
• No clear text password accepted
• Compliant iSCSI initiators and targets MUST implement the CHAP (RFC1994)
• Implementations MUST support use of up to 128 bit random CHAP secrets
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
343343343© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
iSCSI Security
• Various levels of security can fit different topologies
Examples:Secure main floor—No security
Campus LAN—iSCSI authentication and CRC32c (digests)Remote private WAN—IPSec with session/packet authenticationRemote internet WAN—IPSec with privacy encryption
344344344© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
iSCSI Data Integrity
• Basic level of end-to-end data integrity can be reasonably handled by TCP using the standard checksum
• iSCSI CRC32c digest checks for Integrity beyond TCP/IP XOR checksum
a) Header digest
b) Data payload digest
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
345345345© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Digests (Checksums)
• Optional header and data digests protect the integrity of the header and data, respectively; The digests, if present, are located, respectively, after the header and PDU-specific data, and cover the proper data and the padding bytes
• The existence and type of digests are negotiated during the login phase
• The separation of the header and data digests is useful in iSCSI routing applications, in which only the header changes when a message is forwarded; In this case, only the header digest should be recalculated
• Digests are not included in data or header length fields• A zero-length data segment also implies a zero-length
data-digest
346346346© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Error Recovery
Two Considerations for Errors• An iSCSI PDU may fail the digest check and be
dropped, despite being received by the TCP layer; The iSCSI layer must optionally be allowed to recover such dropped PDUs
• A TCP connection may fail at any time during the data transfer; All the active tasks must optionally be allowed to be continued on a different TCP connection within the same session
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
347347347© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Error Recover: iSCSI Initiator
A. NOP-OUT to probe sequence numbers of the target
B. Command retry
C. Recovery R2T support
D. Requesting retransmission of status/data/R2T using the SNACK facility
E. Acknowledging the receipt of the data
F. Reassigning the connection allegiance of a task to a different TCP connection
G. Terminating the entire iSCSI session to start fresh
348348348© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Error Recover: iSCSI Target
A. NOP-IN to probe sequence numbers of the initiator
B. Requesting retransmission of data using the recovery R2T feature
C. SNACK support
D. Requesting that parts of read data be acknowledged
E. Allegiance reassignment support
F. Terminating the entire iSCSI session to force the initiator to start over
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
349349349© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Error Recover Classes
• Within a command (i.e., without requiring command restart)
• Within a connection (i.e., without requiring the connection to be rebuilt, but perhaps requiring command restart)
• Connection recovery (i.e., perhaps requiring connections to be rebuilt and commands to be reissued)
• Session recovery
350350350© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Error Levels
• Level determined during logon text negotiationError recovery level is proposed by an originator in a text negotiation
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
iSCSI PROTOCOL DETAILS IN-DEPTH
351351351© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
352352352© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
iSCSI Key Points
• Tasks:A linked set of SCSI commandsOne and only one SCSI command at a time can be processed within any given iSCSI task
• Initiator Task Tag (ITT) and Target Transfer Tag (TTT)Initiator tags for all pending commands must be unique initiator-wide
SCSI Data PDUs are matched to their corresponding SCSI commands using tags specified in the protocol
ITT for unsolicited dataTTT for solicited data
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
353353353© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
iSCSI Key Points
Solicited or unsolicited messages:• Initiator to target
User data or command parameters will be sent as either solicited data or unsolicited dataSolicited data is sent in response to ready to transfer (R2T) PDUsUnsolicited data can be part of an iSCSI command PDU (“Immediatedata”) or an iSCSI data PDUThe maximum size of an individual data PDU or the immediate part of the initial unsolicited burst may be negotiated during login
• Target to initiatorReady to transfer (R2T) message to Initiator, requesting data for a write commandCommand responsesAsynchronous messages (SCSI and iSCSI) describing an unusual or error event
354354354© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
iSCSI Numbering
• iSCSI uses command and status numberingCommand numbering—Session wide and is used for ordered command delivery over multiple connections within a session; It can also be used as a mechanism for command flow control over a sessionStatus numbering—per connection and is used to enable recovery in case of connection failure
• Fields in the iSCSI PDUs communicate the reference numbers between the initiator and target
During periods when traffic on a connection is unidirectional, iSCSI NOP PDUs may be issued to synchronize the command and status ordering counters of the initiator and target
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
355355355© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
SCSI Command Numbering and Acks within iSCSI
• Initiator and target device have three sequence number registers per session
CmdSN—Current command sequence number; Sent by initator
ExpCmdSN—Expected command by the target; Sent to the initiator by the target to acknowledge CmdSN; Can be used to ACK several sequences
MaxCmdSN—Maximum number target can receive in its queue; Can be sent to Initiator from target to adjust queue size
356356356© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
SCSI Command Numbering and Acks within iSCSI
• iSCSI supports ordered command delivery within the session
• Command-Sequence-Number (CmdSN) is assigned by initiator and carried in the iSCSI PDU
• CmdSN starts at iSCSI login
• CmdSN not assigned to data-out (DataSN used)
• Immediate delivery does not advance CmdSN
• iSCSI must deliver commands to target in order of CmdSN and will not increment until executed state by target
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
357357357© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
SCSI Status Numbering and Acks within iSCSI
• Status Sequence Number (StatSN) is used to number responses to the Initiator from the target
• ExpStatSN is sent by Initiator to acknowledge status
• Status numbering starts after Login; During login there can be only one outstanding command per connection
358358358© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Initiator iSCSI OPcodes
0x00 NOP (No operation, used as ping to target gateway)
0x01 SCSI command (Indicates encapsulated iSCSI packet has a SCSI CDB for target device)
0x02 SCSI task management command
0x03 iSCSI login
0x04 text command
0x05 SCSI data-out (Write data to target device)
0x06 iSCSI logout
0x10 SNACK (Request retransmission from target)
0x1c-0x1e Vendor specific codes
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
359359359© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Target iSCSI OPcodes
0x20 NOP-In (No operation in, used for ping response from target
0x21 SCSI response (Indicates encapsulated iSCSI packet has status or from target device)
0x22 SCSI task management response
0x23 login response
0x24 text response
0x25 SCSI data-in (Read data from target)
0x26 logout response
0x31 Ready to transfer (Sent to initiator from target to indicate it is ready to receive data)
0x32 async message (Message from target to indicate special conditions)
0x3c-0x3e vendor specific codes
0x3f reject
360360360© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
iSCSI PDU’s
• Several different types of iSCSI PDUs used, each of the different iSCSI Operation Codes (Opcodes) determine what iSCSI PDU to use; Some of the more used PDUs are:
Login and logout PDU
Command and response PDUData-In and data-out PDU
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
361361361© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
iSCSI Login PDU
If Set to 1 = Recovery from Failed Connection
If Set to 1 Indicates Initiator Is Ready to
Transit to Next Stage
Initiatior May Provide Initial Text Parameters
in This Area
Unique ID for This
Connection
Current Stage/Next Stage0 – Security Negotiation
1 – Login Operational Negotiation- 3 – Full Feature Phase
Initiator ID for This
Connection
362362362© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
iSCSI Login
• Login Phase used to:Enable TCP connection (Target listens on well known port)Authentication (CHAP)
Negotiate session parameters
Open security protocolsMark the TCP connection as a iSCSI sessionand assign IDs
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
363363363© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
iSCSI Text Mode
During Login Some Sessions or Connection Parameters May Be Negotiated in a Text Format
list = values sent in order of preference
Example of values can be:MaxConnections=<1-65535> T or ISendtargets=all I only
Targetname=<iSCSI-Name> T or I
SessionType=<Discovery|Normal> I onlyOthers—addressed later in slides (see RFC)
364364364© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
iSCSI Full Feature Phase
• iSCSI PDUs can be sent
• PDUs must flow over same connection as login
• Size of PDU is negotiated during login
A Connection Is in Full Feature Mode after a Completed Login
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
365365365© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Data Sequencing within iSCSI
• The iSCSI PDUs used for data input and output are the 0x05 iSCSI command and the 0x25 iSCSI command, along with R2T (0x31 ready to transfer)
DataSN is a number field and advances by 1 for each input (read) and output (write)Targets will operate in two modes, solicited (R2T) or unsolicited (non-R2T)
Target operating in R2T mode can only receive solicited data from the initiatorR2TSN advances by one for each received R2T during the data transfer
• The DataSN and R2TSN fields are for the initiator to detect missing data
366366366© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Data-Out PDU
Final Bit Say This Is the Last PDU of
a Sequence
Data Segment Length Based on Capabilities
Exchange
LUN Number for Data
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
367367367© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Data-In PDUFinal Bit say this is the last read of a sequence Acknowledge Bit
used when error recovery level is 1
or higher
Status bit tells that there is meaningful data in the
StatSN, Status, and Residual Count fields
Flags valid when S bit is set tells
how to read Residual Count
368368368© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
iSCSI Read Command Example
1. Initiator sends iSCSI command PDU (CDB=Read)2. Target sends iSCSI data-in PDU(s)3. Target sends iSCSI response PDUNotes:• Solicited data via read command PDU (Initiator requests data
from the target)• Target may satisfy the single read command with multiple iSCSI
data read PDUs (PDUs can be out-of-order)• Command is not complete until all data and status is received
by the initiator• Good status can be sent within the last iSCSI data-in PDU• All iSCSI data-in PDUs and the response PDU will be delivered
on the same TCP connection that the command was sent on• All data-in PDUs will carry the same value in the ITT field
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
369369369© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
SCSI Command PDULets Target Know if
More Data Is to Follow along with
Expected Data Transfer Length
R=1 If the Command Is Expected to Input Data
W=1 If the Command Is
Expected to Output Data
Task Attributes See RFC for Detailed
Meaning
16 bytes of SCSI CDB,
Some SCSI Commands Have Additional Data and This Field Is Used for the
Accompanied Data
CRC If Capabilities
Required This
370370370© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
SCSI Response PDU
Ox00 = Command Completed at Target0x01 = Target Failure0x08 – 0xff = Reserved for Vendor Response
SCSI Status per SAM2
CRC Check Sums
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
371371371© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
SCSI Status and Response Fields for iSCSI OpCode 0x21
• The status field of the iSCSI PDU is used to report status of the command back to the initiator
• The specific status codes are documented in the SCSI architectural model for the device
• Response field contains the iSCSI codes that are mapped to the SAM-2 response
372372372© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Ready to Transfer PDU
• When the initiator has sent a SCSI write command to the target the target can specify the blocks be delivered in a convenient order; This information is passed to the initiator in the R2T PDU
• Allowing an initiator to write data to a target without a R2T is agreed upon during login
• The target may send several R2T PDUs and have several data transfers pending if allowed by the initiator
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
373373373© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Task Management
• Functions to provide the initiator a way to control management of the target device
Abort the TASKClear allegiance
Logical reset
Target reset
• Each of these and more are broken down in detail in the iSCSI RFC
374374374© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
SACK, NOP-IN, NOP-OUT
• SACKOptionalUsed to request retransmission of numbered responses, data or R2T PDUs from the target
• NOP-INSent by a target as a response to a NOP-Out, as a“ping” to an initiatorOr a means to carry a changed ExpCmdSN and/or MaxCmdSN if there is no other PDU to carry them for a long time
• NOP-OUTUsed by Initiator as a “ping command”, to verify that a connection/session is still active and all its components are operationalUsed to confirm a changed ExpStatSN if there is no other PDU to carry it for a long time
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
375375375© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Message Synchronization and Steering
• Steering of iSCSI out of order TCP segments into pre-allocated buffers instead of temporary buffers
• To decrease reassembly time
• Not needing to rely on message length information
• Provides a synchronization method using fixed interval markers telling where the start of the next iSCSI PDU is in the buffer
• Optional for iSCSI RFC
376376376© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
List of Negotiated Parameters Prior to Going into Full Feature Mode
Header Digest
Data Digest
Max Connections
Send Targets
Target Name
Initiator Name
Target Alias
Initiator Alias
Target Address
Target Portal Group Tag
Initial Ready 2 Transfer
Immediate Data
Max Rec Data Segment Length
Max Burst Length
First Burst Length
Default Time 2 Wait
Default Time 2 Retain
Max Outstanding R2T
Data PDU In-order
Data Sequence In-order
Error Recovery Level
Session Type
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
377377377© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Standards: Where to Find Details
• http://www.ietf.org/html.charters/ips-charter.html
• T10 Technical committee—www.t10.orgTechnical committee of the National Committee on Information Technology Standards (NCITS), deals with the storage devices
• T11 Technical committee—www.t11.orgTechnical committee of the NCITS, deals with the physical interface and transport level
SIMPLE ISCSI CONNECTION FLOWSEXAMPLE OF DISCOVERY SESSION WITH CHAP
378378378© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
379379379© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Establish Initial TCP Session Phase
0X03 Command—LoginKey Values Are Sent, InitiatorName, InitiatorAlias,
SessionType=Discovery, AuthMethod=CHAP/none, HeaderDigest, DataDigest
TCP Port 3260TCP Port 3260
TargetInitiatorTCP port 1026
(Random) Discovery Session
TCP port 1026 (Random) Discovery Session
iSCSI Driver
This Device
Has Already
Initialized Onto the
Fibre Channel
This Device
Has Already
Initialized Onto the
Fibre Channel
iSCSI Flows
0X23 Login ResponseStatus= Accept Login (0X0000), Keyvalues Are Sent,
AuthMethod=CHAP, HeaderDigest=none, DatDigest=none
0X03 Command—LoginKey Values Sent, InitiatorName, InitiatorAlias,
SessionType=Discovery, CHAP_A=5 (CHAP with MD5)
0X23 Login ResponseStatus=Accept Login, KeyValues CHAP_A, CHAP_I & CHAP_C
380380380© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
0X23 Login ResponseFinal PDU in Sequence, Status= Accept login (0X0000)
0X23 Login ResponseStatus=Accept Login, Negotiate Session Parameters
0X03 Command—LoginKey Values Are Sent, InitiatorName, InitiatorAlias,
SessionType=Discovery, CHAP_R, CHAP_N
iSCSI Flows
0X03 Command—LoginKey Values Sent, InitiatorName, InitiatorAlias,
SessionType=Discovery, Negotiate Session Parameters
End of Authentication Phase Start of Parameter Negotiation Phase
for Discovery Session
TargetInitiatorTCP port 1026
(Random) Discovery Session
TCP port 1026 (Random) Discovery Session
iSCSI Driver
TCP Port 3260TCP Port 3260
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
381381381© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
0X24 Login ResponseFinal PDU in Sequence, KeyValue=TargetName (iqn number
along with target name configured on iSCSI Target)
0X04 Text CommandSendTargets=all
TargetInitiatorTCP port 1026
(Random) Discovery Session
TCP port 1026 (Random) Discovery Session
iSCSI Driver
TCP Port 3260TCP Port 3260
iSCSI Flows
0X03 Command—LoginKey Values sent, InitiatorName, InitiatorAlias,
SessionType=Normal, TargetName, AuthMethod=CHAP,none
0X23 Login ResponseStatus=Accept Login, AuthMethod=CHAP
Establish TCP connection for target
TCP port 1027 (random)
Target Session #1
TCP port 1027 (random)
Target Session #1
Note the Addition of
Another TCP Session
Start of Target Session Authentication and Target Session Parameter Negotiation
382382382© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
0X23 Login ResponseStatus=Accept Login, KeyValues CHAP_A, CHAP_I & CHAP_C
0X03 Command—LoginKey Values are sent, InitiatorName, InitiatorAlias,
SessionType=Normal, CHAP_R, CHAP_N
0X03 Command—LoginKey Values are sent, InitiatorName, InitiatorAlias, SessionType=Normal, TargetName, CHAP_A=5
TargetInitiator
iSCSI Driver
TCP Port 3260TCP Port 3260
iSCSI Flows
0X23 Login ResponseStatus=Accept Login
TCP Port 1027 (Random)
Target Session #1
TCP Port 1027 (Random)
Target Session #1
0X03 Command—LoginKey Values sent, InitiatorName, InitiatorAlias,
SessionType=Normal, TargetName, Negotiate session Parameters
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
383383383© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
TargetInitiator
iSCSI Driver
TCP Port 3260TCP Port 3260
TCP Port 1027 (Random)
Target Session #1
TCP Port 1027 (Random)
Target Session #1
0X01 iSCSI CommandSCSI Inquiry CDB 0X12
iSCSI Flows
0X23 Login ResponseStatus=Accept Login, Negotiate session Parameters
0X25 iSCSI Data-in (read)
384© 2004 Cisco Systems, Inc. All rights reserved.
FCIP CONCEPTS
OPT-2T019899_06_2004_X
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
385385385© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Agenda
• What FCIP Is About
• The StandardsFibre Channel T11 StandardsIETF IPS Working Group Drafts
• Understanding FCIP Protocol
• Relationships to Other SCSI Transport Technologies
386386386© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
FCIP: Fibre Channel over IP
• FCIP provides a standard way of encapsulating FC frames within TCP/IP, allowing islands of FC SANs to be interconnected over an IP-based network
• TCP/IP is used as the underlying transport to provide congestion control and in-order delivery of error-free data
• FC frames are treated the same as datagrams
• It is not iFCP, mFCP, IPFC, iSCSI transports or extended FC fabric
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
387387387© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
FCIP Design
IP NetworkTunnel SessionTunnel Session
IP ServicesAvailable at Aggregated
FC SAN Level
FC TapeLibrary
FC TapeLibraryFC Server
FCServer FC
ServerFCJBOD
FCJBOD
FC Switch
FC SwitchFC Switch
FC SwitchFC SwitchFC Switch
FC SwitchFC Switch
FSPF RoutingBackbone
FSPF RoutingBackbone
FSPF RoutingBackbone
FSPF RoutingBackbone
Fiber Channel
SAN
Fiber Channel
SAN
FCIPTunnel
FCIPTunnel
FC Server
Tunnel SessionTunnel Session
388388388© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Four (4) Specifications Define Basic FCIP
• ANSI: http://www.t11.org/index.htmFC-SW-2 describes the operation and interaction of FibreChannel switches, including E_Port, B_Port and fabric operation
FC-BB-2 is a mapping that pertains to the extension of Fibre channel switched networks across a TCP/IP network backbone and defines reference models that support E_Port and B_Port
• IETF IPS working group:Fibre channel over TCP/IP covers the TCP/IP requirements for transporting Fibre Channel frames over an IP network
FC frame encapsulation defines the common Fibre Channel encapsulation format
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
389389389© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
ANSI: FC-SW-2 Standard
• E_Ports are used at both ends of an Inter Switch Link (ISL)
• E_Ports forward user traffic (storage data) and control information (class F SW_ILS frames containing FSPF, zone exchanges, etc.)
• FC-SW-2 defines fabric merge procedures (Domain_ID assignment, zone transfers, etc.)
• FC-SW-2 also defines FSPF
390390390© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
ANSI: FC-SW-2 Essentials (Recap)
• E_Ports provide switch-to-switch connectivity
• E_Ports negotiate parameters such as:ELP—Exchange Link ParametersESC—Exchange Switch Capabilities
• FSPF is enabled over E_Ports only
• Separate fabrics can be merged over E_Ports
• Zoning information is exchanged over E_Ports
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
391391391© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
IETF FCIP: Fibre Channel Over IP
• Each interconnection is called a FCIP link and can contain one (1) or more TCP connection(s)
• Each end of a FCIP link is associated to a virtual ISL link (VE_Port or B_Access Portal)
• VE_Ports communicate between themselves just like normally interconnected E_Ports by using SW_ILS: ELP, EFP, ESC, LKA, BF, RCF, FSPF, etc.
• B_Access portals communicate between themselves by using SW_ILS: EBP, LKA
• The result (when all goes well… ) is a fully merged Fibre Channel fabric between FC switch SAN’s
392392392© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
IETF FCIP
• IETF draft standard that allows IP connectivity to link Fibre Channel storage area networks across WANs
Two methods can be used
1) Similar to Cisco STUN—Nailed up tunnel2) Similar to DLSW—Dynamic peering method
We will visit the details of each in later slides
• draft-ietf-ips-fcovertcpipDraft 12 is current, will RFC Jan/Feb 2003
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
393393393© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
FCIP Architecture Model
FCIP FCIP
FC-2 FC-2
FC-1
FC-0
FC-1
FC-0
TCP TCP
IP IP
LINK LINK
PHY PHY
TCP/IPNetwork
SAN SAN
FCIP Link
Key:
FC-0—Fibre Channel Physical Media Layer
FC-1—Fibre Channel Encode and Decode Layer
FC-2—Fibre Channel Framing and Flow Control Layer
TCP—Transmission Control Protocol
IP—Internet Protocol
LINK—IP Link Layer
PHY—IP Physical Layer
394394394© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
FCIP
• End-station addressing, address resolution, message routing, and other fundamental elements of the network architecture remain unchanged from the Fibre Channel model, with IP introduced exclusively as a transport protocol for an inter-network bridging function
• IP is unaware of the Fibre Channel payload and the fibre channel fabric is unaware of IP
EthernetHeader
EthernetHeader
//
CRCCRCIPChecksum
IPChecksum
TCP FCIPFCIP SCSI Data … FCPFCP
//
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
395395395© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
FCIP
• FCIP only supports class 2, class 3, class 4, and class F frames
• No FC primitive signals or primitive sequences supported
Physical signal sets used by FC ports to indicate events, i.e. NOS, OLS, LR
• IP transport is transparent to FibreChannel topology
396396396© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Understanding FCIP Terms
• FC end node—A Fibre Channel device that uses the connection services provided by the FC fabric
• FC entity—The Fibre Channel specific functional component that combines with an FCIP entity to form an interface between an FC fabric and an IP network
• FC fabric—An entity that interconnects various Nx_Ports attached to it, and is capable of routing FC frames using only the destination ID information in a FC frame header
• FC fabric entity—A Fibre Channel specific element containing one or more Interconnect_Ports (see FC-SW-2) and one or more FC/FCIP entity pairs
• FC frame—The basic unit of Fibre Channel data transfer• FC frame receiver portal—The access point through which an FC frame and
time stamp enters an FCIP data engine from the FC entity• FC frame transmitter portal—The access point through which a reconstituted
FC frame and time stamp leaves an FCIP data engine to the FC entity
• FC/FCIP entity pair—The combination of one FC entity and one FCIP entity
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
397397397© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Understanding FCIP Terms (Cont.)
• FCIP data engine (FCIP_DE)—The component of an FCIP entity that handles FC frame encapsulation, de-encapsulation, and transmission FCIP frames through a single TCP connection
• FCIP entity—The entity responsible for the FCIP protocol exchanges on the IP network and which encompasses FCIP_LEP(s) and FCIP control and services module
• FCIP frame—An FC frame plus the FC frame encapsulation header, encoded SOF and encoded EOF that contains the FC frame
• FCIP link—One or more TCP connections that connect one FCIP_LEP to another
• FCIP link endpoint (FCIP_LEP)—The component of an FCIP entity that that handles a single FCIP link and contains one or more FCIP_DE’s
• Encapsulated frame receiver portal—The TCP access point through which an FCIP frame is received from the IP network by an FCIP data engine
• Encapsulated frame transmitter portal—The TCP access point through which an FCIP frame is transmitted to the IP network by an FCIP data engine
• FCIP special frame (FSF)—A specially formatted FC frame containing information used by the FCIP protocol
398398398© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
FCIP DiagramFC Entity
TCPPorts WKP = 3225
VE_Port
FCIP_LEP
DE DE
FCIP Entity
TCPPorts WKP = 3225
VE_Port
FCIP_LEP
FCIP Physical Link
TCP Connection FC Frames in TCP/IPClass 3 and Class F Can Beon Separate Ports or Connections
IP Address = 172.16.0.5Ethernet Gigabit/WAN Interface
FCIP Link
Virtual ISL
IP Address = 192.168.1.10Ethernet Gigabit/WAN Interface
FCIP Link End PointFCIP Data Engine(Detail)
FCIP Frame TX RX Portal
FC Frame Receiver Portal
FC/FCIP Entity Pair
TX RX
More than One TCP Connection Is Allowed
Dynamic CONNECTION PORT for FCIP Connections
Non Dynamic Connections
FCIP Data Engine
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
399399399© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
ANSI Meets IETF E-Port
• FC-BB-2• FCIP
400400400© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
ANSI Meets IETF B-Port
• FC-BB-2
• FCIP
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
401401401© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
FCIP Standards Stack Details
This Will Be the ISL Connection Either a Bridged connection or
E_Port; Depending on FCIP Implementation selected by Vendor
402402402© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Additional IETF Drafts
• SLP: Service Location Protocoldraft-ietf-ips-fcip-slp Used for dynamic discovery of FCIP ports
• IPSec for storagedraft-ietf-ips-securityMore details later on this requirement for FCIP
• MIBsdraft-ietf-ips-scsi-mibdraft-ietf-ips-fcmgmt-mibdraft-ietf-ips-fcip-mib
• FC-BBPublished ANSI project being superseded by BB-2
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
403403403© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
ANSI: FC-BB-2 Essentials(FCIP E-Port)
• Defines a slightly complex model;
• FC-BB-2 covers the FC portion of this model (FC entity and some of above)
• Cisco’s FCIP E_Port implementations will closely follow this model
404404404© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
IETF: FCIP Essentials(FCIP E-Port)
• FCIP follows the model proposed in FC-BB-2;
• FCIP covers the lower portion of this model (FCIP entity and below)
• Cisco’s FCIP E_Port implementation will follow this model
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
405405405© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
ANSI: FCIP Essentials(FCIP B-Port)
• Again the FC side of the this model follows SW-BB-2 standards
• With B_Port there is no FC switching element so the B_Port device will not be seen as a switch in the fabric but as a passive device
406406406© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
IETF: FCIP Essentials(FCIP B-Port)
• The FCIP part of the B-Port operation is the same as FCIP for the E_Port
• Note in this diagram that implementations of this standard can be any number of ports from 1 to n
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
407407407© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
About FCIP Links
• The FCIP interface represents both the VE_Port and the FCIP link
• An FCIP link is defined as one or more TCP connections
• FCIP link endpoint (LEP) terminates FCIP links
• FCIP data engine: One per TCP connection
Entity 1
TCPPorts WKP = 3225
VE_Port
FCIP_LEP
DE DE
FCIP Link
Class F Class 3
IP Address = 192.168.1.10TCP/IP Network Interface
408408408© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
About the FC Entity
• FC entity interfaces (internally) with FCIP entity
• FC entity components:Control and ServicesModuleProvides FC frame and timestamp along with synchronization with FCIP entity Correct order delivery of FC framesWorks with FCIP entity for flow controlComputes end-to-end transit timeThrows away expired framesAnswers to authentication of TCP connection request
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
409409409© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
About the FCIP Entity
• FCIP entity interfaces (internally) with FC entity
• FCIP entity components:
Provide FC frame and timestamp to FC entityTells FC entity about discarded bytesTells FC entity about new and lost TCP connections and reason codesMonitors special frame changesMakes request to FC entity for authentication
410410410© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
FCIP Link Endpoint: Details
• FCIP_LEP is the translation point between an FC entity and an IP network
• LEP coordinates between FC and TCP flow control mechanisms
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
411411411© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Error Detection and Recovery
• Data engine uses various methods to detect errors but does not correct errors
• Rather, it inserts EOFa (abort) frame delimiters when possible
• Requests sent up to FC entity to handle recovery
412412412© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
IETF: Fibre Channel Frame Encapsulation Header
• Defines the encapsulation header for Fibre Channel frames
• Not specific to FCIP• Includes timestamp, CRC and provision for special frames
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
413413413© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Initialization of Port
B_Port• Link initialization
• Exchange link parameters
• Link reset
E_Port• Link initialization• Exchange link parameters• Reset link• Exchange switch
capabilities• Exchange fabric
parameters• Assign domain IDs• Establish routes• Merge zones if required
414414414© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Link Initialization Flow
NOS = Not Operational Sequence
OLS = Offline Sequence
LR = Link Reset
LRR =Link Reset Response
AC = Activity State
LR = Link Recovery State
LF = Link Failure State
OL = Offline State
LFLF
LROL
LR
AC
AC
NOS
OLS
LR
Idle
LRR
Idle
E_Port on Switch
B_Port or E_Port on FCIP Device
These Are All Special Ordered Sets of 8B/10B Coding
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
415415415© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Link Capture
E_Port on Switch
B_Port or E_Port on FCIP Device
LR & LRR to Initialize Flow
Control Parameters Per
FC-PH
NOS
LR
IDLE
R-RDYIDLE
IDLE
IDLE
R-RDYIDLE
LRR
At this Point B_Port Device Is Up and E_Port to E_Port Exchange Continues
416416416© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
ISL E_Port If It Is an E_Port FCIP Device or If the B_Port Is Now up the Switch to Switch Exchange Continues
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
417417417© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
ELP Data Bit 15 of flag will be a 1 for
B_Port
RA_TOV is fabric wide timer, ED_TOV
is per Link
Credit value is one to start to allow only one
out standing frame during link start-up
Class 2 & 3 supported
PWWN & WWN,Vendor ID also
418418418© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
E_Port and B_Port Summary
FC SANFC SAN
FC Switch FC Switch
FCIP E-Port FCIP E-Port
IP Network
VE - Port
Exchange Fabric Parameters
Exchange Fabric ParametersExchange Fabric Parameters
Exchange Link Parameters Exchange Link ParametersExchange FCIP-Link Parameters
ESCESCESC
FC SANFC SAN
FC Switch FC Switch
FCIP E-Port FCIP E-Port
IP Network
VB - Port
Exchange Link Parameters Exchange Link ParametersExchange FCIP-Link Parameters
Exchange Fabric Parameters
ESC (Exchange Switch Capabilities) if required
B Port Operation
E Port Operation
FCIP B-Port FCIP B-Port
7200 w/ PA-FC-1G 7200 w/ PA-FC-1G
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
419419419© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
FCIP: ISL Connection
• The E-Port or B-Port FCIP Connection Will Provide:Simple name service across the IP tunnelFC discovery between SAN islands
FSPF routing services between fabric switches
Management server informationBuffer credits
420420420© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Comparisons
B-Port and E-Port Differences
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
421421421© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
FCIP Connection Establishment
• Non-dynamicTCP connection to a specific IP address
• DynamicDiscovery of FCIP entities using SLPv2
• Use of FCIP special frame
• Use of options
422422422© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Non-Dynamic TCP Connections
• The FCIP entity is informed of a TCP connection is needed (Most likely done by configuration parameters in the device)
• IP address and security features are established (Configured)
• Destination WWN is determined (Configured)
• TCP/IP parameters are set (Configured)
• Quality of service is determined (Configured)
• Connection request is made to Port 3225 or configured port
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
423423423© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Dynamic TCP Connections SLPv2
• IP security for SLP determined
• Enter FCIP discovery domain process
• Advertise availability to SLP discovery domain service agent
• Locate FCIP entities in the discovery domain as a SLP user agent
• For each discovered entity follow same process as non-dynamic method to establish connection
424424424© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
FCIP Special FrameTCP Connection Is Established
Sending Side• First frame sent after TCP connection is established
• Sending side waits for FSF echo (90 seconds)
• Echo is match or non-match (Non-match terminates TCP connection)
• Creation of FCIP_LEP and FCIP_DE
• Inform FC Entity of connection and usage flags
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
425425425© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
FCIP Special FrameTCP Connection Is Established
Receiving Side (Listening)• Listen for connections on WKN port 3225
or configured port• Checks database to allow connection• Checks security features• Wait for FSF frame (90 seconds)• Inspect FSF contents and send echo frame
Connection nonceDestination FC fabric entity world wide nameConnection usage flagsConnection usage code
426426426© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
FCIP Special Frame Details
• Used to exchange WWNs, entity pair identifiers, TCP connection identifiers and to except or reject connection
• Identify what kind of traffic (SOFi3, SOFn3, EOF) is intended; Not enforced
• In conjunction with connection usage flags, connection usage code help FCIP entity apply proper QoS parameters for the connection
• Adjustments to FSF with use of change bits can be made when frame is echoed back
• If two entities are trying to send FSF connection frames simultaneously first to Rx echo wins
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
427427427© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
• The first frame transmitted in each direction is a special frame used to identify the peers FCIP entities and to synchronize
Connection Nonce (random number)
proto0x01
version0x01
~proto0xFE
~version0xFE
Reserved0x0000
Reserved0xFFFF
ConnUsage flags
0x00 Connectionusage code
proto0x01
version0x01
~proto0xFE
~version0xFE
Flags0x00
Frame Len 0x12
~Flags0x3F
~Frame Len 0x3ED
CRC (Reserved in FCIP)0x00-00-00-00
Source FC Fabric Entity WWN(identify the fabric)
Source FC/FCIP Entity Identifier
Destination FC Fabric Entity WWN
Reserved0x0000
Reserved0xFFFF
0
1
2
3
4/5
6
7
8/9
10/11
12/13
14
15/16
17
Timestamp integer/fraction
~pFlagspFlags 0x00 0x00
1) Special Frame Sent
3) FCIP Tunnel Setup Complete
2)Special Frame echoed
Fibre Channel
IP WAN
Fibre Channel
FC
FC
I Am WWN1, This Is my FC/FCIP
IdentifierAre You Fabric
WWN2?
Ok WWN1, I Am WWN2
Let’s Setup theConnection
FCIP: Tunnel Setup as Proposed in FCIP Draft
FCIP Device
428428428© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
pFlag Breakdown
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
429429429© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
FCIP Header Format
• FCIP header used after FSF exchange is completed
Ones Compliment for Synchronization and Error Checking
430430430© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Connection Options
• TCP selective acknowledgement (SACK)Per RFC 2883
• TCP window scale option
• Protection from sequence number wrap (PAWS)
• TCP keepalives (KAD)
• Flow control mapping between TCP and Fibre Channel
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
431431431© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
FCIP Security Requirements (Per Draft)
To Support IP Network Security FCIP Entities MUST:• Implement cryptographically protected
authentication and cryptographic data integrity keyed to the authentication process, and Implement data confidentiality security features
• FCIP utilizes the IPSec protocol suite to provide data confidentiality and authentication services, and IKE as the key management protocol
• FCIP Security compliant implementations MUST implement ESP and the IPsec protocol suite based cryptographic authentication and data integrity [11], as well as confidentiality using algorithms and transforms as described in this section
432432432© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
FCIP Security Requirements(Per Draft) (Cont.)
• FCIP implementations MUST meet the secure key management requirements of IPsec protocol suite
• FCIP entities MUST implement replay protection against ESP sequence number wrap
• FCIP entities MUST use the results of IKE phase 1 negotiation for initiating an IKE phase 2 “quick mode” exchange and establish new SAs
Note: An External Device May Be Used in Conjunction with the FCIP Implementation to Meet the “Must Implement ESP” Requirement
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
433433433© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Important FC and FCIP Timers
• Resource Allocation Timeout Value (R_A_TOV)Timeout value that determines how long a FC frame can be in transit on the Fibre Channel network This is a fabric wide value with a default value usually at 120 sec on switch networks
• Error Detect Timeout Value (E_D_TOV)A value that times events and responses at the link level; Errors at the link level will cause delays of these eventsThis value is defaulted to 10 sec and should be lower then R_A_TOV; Again this is a fabric wide setting
• Keep Alive Timer K_A_TOVA value that is applied to TCP connection and is used when no data is present
434434434© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Time Stamps and Synchronization
• Clock synchronization is required if timestamps are used
Synchronized to FC servicesSynchronized to IP NTP
• Transit time through IP network is applied via a timestamp Integer
• If no timestamp value is available zero will be used• Fibre channel time values still apply across the ISL
link and are timed-out via lack of RDY coming back• End system devices such as HBA attached hosts
still require normal responses to timers end-to-end (no spoofing)
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
435435435© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Timestamps
• TS are the responsibility of the FC entity
• This allows transit through the FCIP entity to be included in the measurement
• This transit time should be well below R_A_TOV
436436436© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Buffer Credits
• Fibre channel buffer credit methods do not change
• R_Rdy’s will be used to control flow coming from FC switch on a per link basis
• Buffer credit establishment is determined at FLOGI
• Mechanisms to control flow of R_Rdy’s to FC switch based on TCP/IP congestion is per FCIP solution
• FC switches do not require extended credit methods
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
437437437© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Error Recovery
• Errors on FC side of local B_Port are not forwarded over the IP network; Issues such as loss of sync or a FC encapsulation error will not be set to the FC entity
• Errors on IP side are handled by TCP and frame is dropped if checksum is in error
438438438© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Summary
• FCIP is the standards approach to connect FibreChannel ISLs over TCP/IP LAN/WAN connections
• State of draft wording will most likely stay as it is worded today
• Security, network delay and error recovery will be biggest concerns
• No shipping product today conforms to the proposed FCIP draft
• Cisco will have several platforms supporting FCIP solutions
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
439© 2004 Cisco Systems, Inc. All rights reserved.
INTERNET FIBRE CHANNEL PROTOCOL
OPT-2T019899_06_2004_X
440440440© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
iFCP
FC-1
FC-0
TCP
IP
LINK
Gateway RegionPHY
iFCP
FC-1
FC-0
TCP
IP
LINK
Gateway RegionPHY
IP Network
FC-4
iFCP Protocol Model
• iFCP replaces the transport layer of Fibre Channel (FC-2) with an IP network but keeps the FC-4 mapping the existing Fibre Channel transport services on TCP/IP
• iFCP processes differently FC-4 frame images (applications), FC-2 frame images (link service request), FC broadcast and iFCP control frames
• Topology within the gateway regions are opaque to the IP network and other gateway regions (they appear just like collection of N_Ports)
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
441441441© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
IFCPGatewayGateway Region
IFCPGatewayGateway Region
iSNS
N_port-to-N_port session
N_port-to-N_port session
IP Network
iSNS Queries
iSNS Queries
IFCPGateway Gateway Region
IFCPGateway
Gateway Region
iFCP Network Model: iSNS Role
• An iFCP gateway cannot operate without access to an iSNS server
• Client-Server architecture• iSNS functions:
Device Discovery and fabric managementEmulation of the services provided by the FC name server and RSCNDefinition and management of discovery domainsDefinition and management of “logical fabrics”
442442442© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
iFCP Protocol Description: N_Ports Addresses Allocation
• Two different schemes:Address transparent mode (optional): The N_Port FC_IDs are unique across the whole logical fabric
Address translation mode (mandatory): The N_Port FC_IDs are unique only inside the gateway region the N_Port belongs to
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
443443443© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Address Transparent Mode
• All the gateways belonging to the same “logical fabric” cooperate to assign addresses that are unique across the gateway regions that form the logical fabric
• No need for address translation
• Not scalable (max 239 gateways)
444444444© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Address Translation Mode
• iFCP gateways use aliases to map the local representation of addresses of external gateway regions to the real addresses outside the gateway region (comparable to IP NAT)
Requires a rewrite of the FC_IDs in the FC frame header and in the FC payload for some ELS (i.e. ADISC)
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
445445445© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
TCP/IP IFCPGatewayIFCP
Gateway
FC N
S Req
uest
iSNSGive Me the Remote
Gateway IP Address, N_Port ID,
N_Port WWN
2) iSNS query/
reply
1) The N_Port Issues a NS Query
Remote GW IPDest N_Port ID (y.y.y)
3) The Requesting GW Fills Up the Address Translation Table
Local N_Port alias (z.z.z)Dest N_Port WWN
FC_ID = x.x.x
FC_ID = y.y.y
FC N
S Rep
ly
4) The GatewaySends Back to theN_Port the NS Reply (for FC_ID z.z.z)
iFCP Protocol Description: Address Translation Mechanism
446446446© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
TCP/IP IFCPGatewayIFCP
Gateway
Plogi
I did
z.z.z
sid x.
x.x
1) The N_Port I a
PLOGI toD_ID z.z.z
Remote GW IPDest N_Port ID (y.y.y)
4) The Receiving GW Rewrites the S_ID of the Incoming Request
Local N_Port alias (z.z.z)Dest N_Port WWN
FC_ID = x.x.x
FC_ID = y.y.y
2) The GW Makes a Table Lookup Gets the Remote GW IP Address (to Set Up the iFCP Session) and the Actual Dest N_Port ID ( to Rewrite the D_ID) PLOGI did y.y.y
sid x.x.x
Remote GW IPDest N_Port ID (x.x.x) 3) The receiving GW Fills Up
Its Own Translation TableLocal N_Port alias (w.w.w)Dest N_Port WWN
PLOGI did y.y.ysid w.w.w
iFCP Protocol Description: Address Translation Mechanism (Cont.)
• In case of fabric reconfiguration all the address translation tables need to be recalculated with a consequent loss of every active login session
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
447© 2004 Cisco Systems, Inc. All rights reserved.
ISNS AND SLP DISCOVERY PROTOCOLS FOR THE IP-SAN
OPT-2T019899_06_2004_X
448448448© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Discovery Approach
Deploy and Interoperate in Three Stages:1. Naming and static configuration
Configure both targets and initiators
Use SendTargets to reduce initiator config
2. SLPv2 for multicast and simple discoveryConfigure targets
3. iSNS for centralized managementConfigure central iSNS server
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
449449449© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Service Location Protocol (SLP)
• Based on service location protocol v2 (RFC 2608)
• Allows hosts to search for instances of a network service they are interested in:
Example: printers
450450450© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Basic SLP Discovery Requirements
• Find targets by initiator’s worldwide unique identifier
“Tell me which targets you have that I should see”
• Find targets by target’s worldwide unique identifier“Where is target iscsi.com.acme.foo?”
• Propagate attributes needed before connectingBoot information, authentication information
• Scaling requirementsZero-configuration, no servers in small environmentsReduce or eliminate multicast in medium environmentsInteroperate with LDAP/iSNS in large environments
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
451451451© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Service Location Protocol (SLP)
• SA—Service Agent; Services register with SA• UA—User Agent; Queries SA or DA for registered
services• DA—Directory Agent; Proxies for a set of SAs
SA
DA
UA
registerquery/response
query/response servicesservices
servicesregister
Three Components, Two of Which Run in Our Storage Router
452452452© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Service Location Protocol for IP Storage
• Service Agent (SA) Advertises servicesServices have attributes
• User Agent (UA) Finds servicesZero configuration
• Directory Agent (DA)OptionalPropagate service adverts
• SLP ProtocolUDP or TCPMinimize multicastdevice
host
iSCSITarget
SLPUA
SLPDA
iSCSIInitiator
IP
TCP/IP
Management Code
SLPSA
TCP/IP
Management Code
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
453453453© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Implementing SLP for iSCSI
• Targets implement a service agentAnswer multicast requests or register with DA
• Initiators implement a user agentUse multicast or DA to locate targets
• Devices containing targets register:The canonical target or individual targetsAttributes of targets
• Register target at each of its addresses
454454454© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
SLP Summary
• Serverless discovery of targetsOptional, generic DA to scale services
• Zero-configuration of hostsSLP makes careful use of multicast
• Access list and attribute propagation
• Optional message authentication
• Available open source implementations
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
455455455© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
What Is iSNS
iSNS Facilitates Scalable Configuration and Management of iSCSI, iFCP and Fibre Channel (FCP) Storage Devices in an IP Network, By Providing a Set of Services comparable to that Available in Fibre Channel Networks
http://www.ietf.org/internet-drafts/draft-ietf-ips-isns-22.txt
456456456© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
iSNS Functions
There Are Four Main Functions of the iSNS: 1. A name server providing storage resource
discovery
2. Discovery Domain (DD) and login control service
3. State change notification service
4. Open Mapping of Fibre Channel and iSCSI devices
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
457457457© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Basic: How iSNS Works
IP NetworkFibre Channel SAN
iSCSI & iSNS Clients
iSCSI Clients Register with iSNS Server, Done By Adding iSNS IP Address to iSCSI Application Driver
11 1 1
iSCSI Targets Register with iSNS Server2
2
iSNS Clients Query iSNS Server for Storage Location and Name
3 3
iSCSI Client then Selects and Logs into iSCSI Target Using Information from iSNS Server
4
FCFC
FCFC
458458458© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Internet Storage Name Service (iSNS)
• iSNS server functions:Allows an iSNS client to register/deregister/query with the iSNS server
Provides centralized management for enforcing access control of targets from specific initiatorsProvides a state-change notification mechanism for registered iSNS clients on the change of status of other iSNS clients
• Similar to the functionally provided by the FC name Server, Zone Server and the RSCN mechanism
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
459459459© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
iSNS Components
• iSNS protocol (iSNSP)A flexible and lightweight protocol that specifies how iSNS clients and servers communicate
• Discovery Domain (DD)A grouping of storage devices much like a zone in the FCP; discovery domains help in control and manage logins and services available to the clients in the domain; Based on the FC-GS standard for fiber channel; Items like default domain are used
• Discovery Domain Set (DDS)A group of one or more discovery domains; A method to store sets of domains within the iSNS database; Multiple DDSs can be active at one time, unlike zonesets in FCP where only one can be active at a time
460460460© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
iSNS Components
• iSNS clientThe iSNS client is located within storage system and talk to theiSNS server using the iSNSP within its configured device domain;client can belong to one or more DDs; iSNS client registers its attributes with the iSNS server and receives notices of changes within the domain
• iSNS databaseThe iSNS database is the information repository for the iSNS server; it maintains information about iSNS clients attributes; a directory-enabled implementation of iSNS may store client attributes in an LDAP directory infrastructure
• iSNS serveriSNS servers respond to iSNS protocol queries and requests, and initiate iSNS protocol state change notifications; properly authenticated information submitted by a registration request isstored in an iSNS database; listens on port 3205
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
461461461© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
iSNS SCN (State Change Notifications)
• iSNS clients who wish to receive SCN have to explicitly register with iSNS server the events in order to receive the notifications
• Initiator/target/object with add/remove event or to/from discovery domain are the events that can be registered
• iSNS servers generate SCN when either the state of any target device changes or when the target device itself requests an SCN to be generated using SCN event message; iSNS listens to FCNS to registration/deregistration
462462462© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
SCN Types
• Regular registrationsThis type of SCN is used within a DD; The discovery domain will control where the SCN message will go
• Management registrationsUsed by control nodes and can travel outside the DD from which they came
Can be TCP or UDP messaging(Most implementations only using TCP for now)
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
463463463© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Services Provided by the Discovery Domain
• Login controlAuthorization and control policies for storage targets can be maintained by iSNS servers only allowing authorized devices to access the targets
Control of what target portals are accessible within the discovery domain
• Fibre Channel to iSCSI device mappingiSNS database learns and stores naming and discovery information about FC storage devices discovery on the iSCSI Gateway and iSCSI devices in the IP network; This database can then be available by FC and IP iSNS clients
464464464© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
High Availability of iSNS Servers
• Can use SLP to discovery other iSNS servers
• Database transfers between servers using iSNSP or SNMP
• Heartbeat mechanism used between active and backup iSNS servers
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
465465465© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Internet Storage Name Service (iSNS) for iSCSI
• The iSNS protocol (iSNSP) provides:A mechanism for iSCSI clients to discover other iSCSI targets/initiatorsEnforce access controlNotifications from an iSNS server on changes to the status of a logged in iSCSI device Provide ability to discovery iSCSI target on different IP network
• iSCSI target discovery can happen through:Static configuration of initiatoriSCSI sendTargets commandName server/directory server (via iSNS)
466466466© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
iSNSP Header
iSNSP Version—C the Current Version is 0x0001; All Other Values Are RESERVEDiSNSP Function ID—Defines the Type of iSNS Message and the Operation to Be Executed; iSNSP PDU Length—Specifies the Length of the PDU PAYLOAD Field in bytes; The PDU Payload Contains Attributes for the OperationiSNSP Flags—Indicates Additional Information About the Message and the Type of Network Entity That Generated the Message iSNSP Transaction ID—MUST Be Set to a Unique Value for Each Concurrently Outstanding Request Message; Replies MUST Use the same TRANSACTION ID Value as the Associated iSNS Request MessageiSNSP Sequence ID—The SEQUENCE ID Has a Unique Value for Each PDU Within a Single TransactioniSNSP PDU Payload—The iSNSP PDU PAYLOAD Is Variable Length and Contains Attributes Used for Registration and Query OperationsAuthentication Block—For iSNS Multicast and Broadcast Messages, the iSNSP Provides Authentication Capability; The iSNS Authentication Block Is Identical in Format to the SLP Authentication Block
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
467467467© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
iSNSP Commands for iSCSI
The Following Are iSNSP Commands Messages Used in Support of iSCSI:
468468468© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
iSNSP Responses for iSCSI
The Following Are iSNSP Response messages Used in Support of iSCSI:
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
469469469© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
iSNS Queries for iSCSI
• iSNS clients can perform two types of queries:Device attribute query: iSNS server responds with requested attributes of one or more iSNS clients
The iSNS server converts the received query to a FC name server query in the SAN
FC name server will ensure that the resultant set is filtered based on zones
The iSNS server translates each entry returned by the FC name server to the corresponding iSNS clients
Apply filters based on iSCSI access control by removing all statically configured virtual targets the query initiator is notallowed to access
Device get next query: Allows an iterative query of the iSNS server’s iSNS client database
470470470© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Return Information from iSNS iSCSI Query
iSCSi Name
IP Address of Portal to Log to and Ask for This
target
Name of Port on the IP Gateway
Entity
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
471471471© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
iSNS for iFCP
• Will work much the same manor as iSCSI just will require other related attributes to be registered and queried
• Is required for iFCP
• Functions much like domain name server and domain ID manager
• Needs to be highly available service for FC devices
472472472© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
iSNSP Commands for iFCPThe Following Are iSNSP Commands Messages Used in Support of iFCP:
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
473473473© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
iSNSP Responses for iFCPThe Following Are iSNSP Response Messages Used in Support of iFCP:
474474474© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
SLP and iSNS
• SLP used for target discoveryNo configuration required for the simplest networksSmall footprint; no servers required
Just enough discovery for small-to-medium networks
Device-centric access control model
• iSNS adds storage management capabilitiesActive monitoring of initiators and targets
Event propagationPublic key distribution
Centralized access control model
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
475475475© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Using Both SLP and iSNS
• Initiators can use both SLP and iSNS to discover targets
• Targets should use SLP only if not configured for iSNS
• Gateways or proxies may provide local SLP discovery of remote iSNS devices
476© 2004 Cisco Systems, Inc. All rights reserved.
TECHNICAL TOOLS AND SKILLS
OPT-2T019899_06_2004_X
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
477477477© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Storage Networking Toolbox
• Test tools for Fibre Channel and IP
• Host based tools
• Network component serviceability tools
• Software debug tools
• Knowledge
478478478© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Fibre Channel Analyzers
• Most units are based on dedicated hardware, and might be supplied with software tools for performance base lining
Very expensiveOriented to protocol conformance testingRequires 2 GBICs interfaces to be implemented
• Monitoring units might have a retiming mode, to cleanup some of the timing problems on a link, and to separate them from the real problem at layer 1
Statistical software can run on these type unitsCollecting statistics on the status of the line, or other parameters (number of bits, exchanges…)
• Sharing is still a dream in most cases, it is complex to share in the field, so in most cases the portable versions are the most suitable
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
479479479© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Fibre Channel Analyzers
• Snooping GBICs or fiber taps; allow to monitor without service interruption; very important for Fibre Channel work in the field
• Traffic probes; used to remotely monitor the state of a network without service interruption
• Trace viewers (free from the vendor websites)Each vendor has its own PC viewer and must be used with each capture tool; these can be found at each of their websites
480480480© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
FC Test Vendors
• Leaders in dedicated hardware tools:Finisar (www.finisar.com)Xyratex (www.xyratex.com)
Aglient (www.agilent.com)
I-Tech (www.I-tech.com)Ancot (http://www.ancot.com/)
Spirent/Netcom systems (www.netcomsystems.com)
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
481481481© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
SCSI Host-Based Testing
• I/O meterhttp://developer.intel.com/design/servers/devtools/iometer/
• I/O zonehttp://www.iozone.org/
• SCSI toolshttp://scsitools.com/
• Xyratex disk basherhttp://www.xyratex.com/
• Freeshare or software tools for SCSI and I/O analysis, tools for disk manufacturing
• www.ethereal.com• www.wildpackets.com
482482482© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Windows Tools
• iSCSI Driver debug helpersWindows debug utilities
http://www.osr.com/resources_downloads.shtml
http://www.sysinternals.com/
• Detail uses of O/S disk administrator to verify and check health of target devices
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
483483483© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
IP: GiGE
• GiGE testers $$$AgilentSniffer
Fluke
Finisar/ShomitiiSCSI decodes just becoming available on most tools
• All your IP toolsIP Ping, trace, etc.Fibre Channel ping available at http://www.teracloud.com/utilities.html
484484484© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
iSCSI Decoding
• Software only analyzers like Ethereal (www.ethereal.com)
• Hardware analyzers
• Can use monitor command on Cisco switches to span the iSCSI GiGE port to a 10/100
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
485485485© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Available Certifications
• SNIA (Storage Networking Industry Association)Level 1—Fibre Channel storage networking professionalLevel 2—Fibre Channel storage networking practitioner
• iSCSI training available at many education sourcesInfinity I/O, medusa, solution technology, others
• Other certifications that are vendor specific
486© 2004 Cisco Systems, Inc. All rights reserved.
ARCHITECTURAL DESIGN OFSTORAGE AREA NETWORKS
OPT-2T019899_06_2004_X
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
487487487© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Section Agenda
• Introduction
• Hierarchy
• Modularity
• Architecture Examples
INTRODUCTION
488488© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
489489489© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Hierarchy, Modularity and Limited Failure Domains
Why Do This? (Benefits Summary):• Scalable architecture
• Improved performance
• Manage change
• Improve service
• Improved security
• Simplified management and troubleshooting
• Reduced cost of ownership
490490490© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
What Problem Are We Solving?
• Have consistently high performance
• Are reliable, scaleable, and manageable
• Are secure and cost-efficient
• Are service and solution enabling
• Adapt to changing requirements
Applications Must Be Available and Perform Well
Networks that Deliver on this Requirement:
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
491491491© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Network Design Goals
Architecture Provides:• Performance
• Reliability, availability, and scalability—RAS
• Cost efficiencies
• Security
• A base to enable services and solutions
To Meet Mission-Critical Business Objectives, Applications Need to Be Consistently Up,
Available, and High-Performing
492492492© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Architecture:Hierarchy, Modularity, and Domains
Functionally Divides the ProblemCreate Manageable Building BlocksLimits Scope of Potential Failures
HierarchyModularityDomains
Fundamentally, We Break the Network Design Process into Manageable Blocks so that the
Network will Function within the Performance and Scale Limits of Applications, Protocols
and Network Services
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
493493493© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Focus of ThisDiscussion
What Does This Mean?
We Build Networks that Have Structure:
Distribution
Backbone
WAN Internet PSTN
Application Servers
Building Blocks
Core
EnterpriseStorage
Access
494494494© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Applying Design Principles to Storage
• HierarchyPredictable performanceScaleable design
Fault isolation
• ModularityCost-effective
Repeatable
• DomainReliabilitySecurity
Shared Storage
Unified Storage Mgmt
CoreCore
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
HIERARCHY
495© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
496496496© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Hierarchy: Physical and Logical
• Physical hierarchyPredictable performance
Scaleable design
Fault isolationHigh availability
• Logical hierarchyVirtual SANsZoning
Enhances physical hierarchy
Physical Architecture
Logical Architecture Logical Architecture
Virtual SAN A Virtual SAN B
H1 H2
D1 D2H3
Zone 1 Zone 2
Zone 3H1 H2
D1 D7H7
Zone 1 Zone 2
Zone 4
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
497497497© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Hierarchy: PhysicalConsolidated Storage Network
• Cost-effective solutionBenefits of consolidation
• Limited scalabilitySmall to medium business
Expansion can be disruptive
• Single fault redundancyDouble fault would likely result in isolation
iSCSI iSCSI
498498498© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Hierarchy: Physical Collapsed Core Architecture
• Collapsed coreHigh performanceMultiple unequal paths
• Better scalabilityMedium to large enterprise
ISLs can limit scalability
• RedundantMesh topologyNetwork survives some double faults
iSCSI iSCSI
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
499499499© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Hierarchy: PhysicalCore Edge Architecture
• Core—EdgeHigh performanceLoad balancing
Consistent hop count
• Good scalabilityLarge to very large enterprise
Non-disruptive expansion
• Better fault toleranceImproved fault isolationSingle fault within layer okay
iSCSI iSCSI
500500500© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Hierarchy: PhysicalOversubscription
• To be expected in storage networks
• Typically lower factors than we see in LANs
• Architecture should be flexible to accommodate differing requirements for various hosts and storage subsystems
• Bandwidth can be modified non-disruptively by using port channels between switches
• Take into account any “inherent” over subscription in networking hardware
• Use actual anticipated throughput rather than link speed for calculating bandwidth requirements
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
501501501© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Hierarchy: PhysicalInter-Switch Links
• Inter-Switch Link—ISLPhysical FC link between two fabric switches forming a trunkUtilized for FC services and data traffic
• Port ChannelMultiple FC ISLs combined to form a single aggregated trunkAll links in a Port Channel must be directly connected to the same two switchesIndividual link state changes do not cause ISL trunk state changes
ISLPort
Channel
iSCSI iSCSI
502502502© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Hierarchy: PhysicalScalability
• OversubscriptionHigher OS acceptable for some hostsLower OS for High performance hosts and storage devicesConsider impact of multi-path load balancingDetermine acceptable worst case in various failure scenariosCan be non-disruptively changed by adding/ removing links to port channels
8:1OS
1:1OS
Core
3:1OS
4x2GbISL
8x2GbISL
15:1OS
iSCSI iSCSI
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
503503503© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Hierarchy: LogicalVirtual SANs
• VSANs provide a means to build a logical structure on top of a physical SAN
• Similar to how VLANs are used to scale ethernet networks VSANs help scale Fibre Channel networks
• Topology changes are isolated within the VSAN therefore adds, moves, and changes are not disruptive to other VSANs
• VSANs can be utilized to establish administrative domains
• Zoning provides an additional access control mechanism within each VSAN
504504504© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Hierarchy: LogicalLogical Architecture
• Virtual SANsSimilar to Ethernet VLANs except no inter-VSAN flowsEnhanced ISL provides VSAN trunking (EISL)Complimentary to port channel
• Services scalabilityIndependent Fibre Channel services for each VSANZoning is per VSAN
• Failure domainFaults contained within VSAN
iSCSI iSCSI
EISLPort
Channel
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
505505505© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Hierarchy: Logical Maximizing VSAN Architecture
• Isolate multiple paths into separate VSANs
• Independent FC services per VSAN
• Provides complete traffic isolation between redundant paths
• Each VSAN converges independently for faster recovery and improved fault isolation
iSCSI iSCSI
506506506© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Hierarchy: Combining Physical and Logical
iSCSI iSCSI
• Fabric A provides one set of links and Fibre Channel services A• Fabric B provides an independent set of links and services
B
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
MODULARITY
507© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
508508508© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Modularity: Key Elements
• The ability to scale the network while maintaining consistent performance
• Building block approach breaks network into smaller chunks that are easier to understand, replicate, and deploy
• Changes and additions can be made non-disruptively
• Provides consistent and limited failure domains
• Modularity can also define administrative boundaries
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
509509509© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Modularity: Building Blocks
iSCSI iSCSI
Fiber Channel
CoreFunctional Building Blocks Provide Scalability with Deterministic Performance
StorageModules
ApplicationModules
510510510© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Modularity: Utilizing VSANs
• Adds, moves, and changes contained within a VSAN are non-disruptive to other VSANs
• Using VSANs facilitates application modeling and testing
• Per VSAN statistics
• Per VSAN traffic engineering
• Per VSAN administration (if desired)
• Eliminates costs associated with separate physical fabrics
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
511511511© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Modularity: Benefits of VSANs
• Overlay isolated virtual fabrics on same physical infrastructure
Each VSAN contains zones and separate (replicated) fabric servicesVSAN membership determined by port
• VSANs for availabilityIsolate virtual fabrics from fabric-wide faults/reconfigurations
• SecurityComplete hardware isolation
• ScalabilityReplicated fabric servicesThousands of VSANs per storage network
• ManagementRoll Based Access Control—RBACProvides administrative boundaries
Department/Customer “A”
Shared Storage
Department/Customer “B”
VSAN-EnabledFabric
MgmtVSAN
VSANTrunks
512512512© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Modularity:Storage Intelligence and VSANs
Dept 3VSAN
Dept 2VSANDept 1
VSAN
• VSANs created to provide isolation of fabric-wide services.
• Virtualization allows physical storage to be in its own VSANs, separate from the host VSANs.
VSANs provide• Secure isolation of physical storage• Easier configuration• Dynamic configuration of fabrics• Role-based access control
Data Center VSANs
Virtualization
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
ARCHITECTURE EXAMPLES
513© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
514514514© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Architecture: iSCSI
• ScalabilityLess expensive alternative for host not requiring 2GbpsRecommend separate NIC TCP Offload Engine—TOEConsider actual throughput requirements for scalability
• Host ServicesAppears as normal HBACompatible with host based storage utilities—multi-path, load balance, mapping, etc.
TOE
File SystemFile System
Adapter Driver
Adapter Driver
TCP/IPStack
TCP/IPStack
NICDriverNIC
Driver
Block DeviceBlock Device
SCSI GenericSCSI Generic
iSCSIDriveriSCSIDriver
Applications
Network FileSystem
Network FileSystem
NICDriverNIC
Driver
TCP/IPStack
TCP/IPStack
iSCSIDriveriSCSIDriver
TCP/IPStack
TCP/IPStackNIC
DriverNIC
Driver
SCSI AdapterSCSI Adapter
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
515515515© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Architecture: iSCSI High-Availability
• Redundant connections to hosts or servers• High-availability iSCSI services• Redundant paths to backend FC SAN
Redundant iSCSI to Fibre Channel Connections
and Services
Storage Array with Redundant Controller
Ports
MultipleEthernetSwitches
Host with Multiple(iSCSI) NICs and Multipathing
Software Installed
iSCSI DriverMultipathingApplicationApplication
516516516© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Architecture: iSCSI Authentication
• SCSI routing service passes username and MD5-hashed password from initiators to AAA server
• AAA authentication list used to determine which service(s) to use for authentication
iSCSI Hosts
(Initiators)RADIUS TACACS+
RADIUS Server
Local
User1/pwd1User2/pwd2
… / …
CHAP
iSCSI Services
TACACS+ Server
User1/pwd1User2/pwd2
… / …
User1/pwd1User2/pwd2
… / …
Authentication Services List
AAA Authentication ServicesAAA Authentication Services
IP Network FC Fabric
User1/pwd1User1/pwd1
SCSI Routing Instance
iSCSI Storage(Targets)
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
517517517© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Architecture: iSCSI Topology
iSCSI Best Practices• Isolate IP storage network
behind application hosts with VLANs
• Minimized potential for bandwidth contention
• Map VLANs to VSANs for manageability
• Dedicated ethernet interfaces on host for attachment to storage network
iSCSI-enabledHosts
Storage Pool
iSCSIServices
Ethernet Switches
Clients
Front-Side IP Network
Front-Side IP Network
FC Attached Hosts with
HBAs
iSCSI iSCSI iSCSI iSCSI
IP StorageNetwork
FCFabric
518518518© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Architecture: SAN Extension Technology
IPWAN
Technology Choice Requires Matching Storage Application Requirements with Service Availability, Cost, Throughput, and Latency
FCIP FCIP
CWDM
DWDM
SONET/SDH
FC FC
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
519519519© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
• Both fabrics remain connected if one of the paths fails
• Use of portchannel prevents state change on link failure
Architecture: High Availability for SAN Extension: FC
• Utilize disparate paths and portchannel for high availability
• Utilize VSANs to limit the failure domain in the event of lost connectivity
FabricA
FabricA
FabricB
FabricB
PortChannel
FCFC CWDM
DWDM
SONET/SDH
520520520© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Architecture: High Availability for SAN Extension: FCIP
• Utilize disparate paths and portchannel for high availability
• Utilize VSANs to limit the failure domain in the event of lost connectivity
• Recommend not using etherchannels
FabricA
FabricA
FabricB
FabricB
PortChannel PortChannel
IPWAN
IPWAN
FCIPFCIP
• Both fabrics remain connected if one of the paths fails
• Use of portchannel prevents state change on link failure
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
521521521© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Architecture:Legacy Storage Implementation
• Storage is ‘captive’ behind applications
• Inefficient allocation of storage resources
• Multiple administrative domains
Backbone
ApplicationServers
Captive Storage Blocks
LAN Core
Campus Clients Remote Clients Internet Clients
SANIslands
522522522© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Architecture: Factors for Determining Architecture
• Current size and anticipated growth for both application servers and storage elements
• Baseline performance requirements for servers and storage
• Business continuance requirements—SAN extension
• Administrative domains
• Migration plans
• Interoperability considerations
• Costs
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
523523523© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Architecture: Collapsed Core Architecture
• Servers and storage elements connected to collapsed core
• Some scalability especially with iSCSI
• Redundant paths• Achieves
economical storage consolidation
• VSANs can add scalability and management benefits
ApplicationServers
iSCSI
Shared Storage
Unified Storage Mgmt
524524524© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Architecture: Large Scale Architecture
• Application servers connect to edge switches
• Storage devices connect to edge switches
• Highly scalable
• Highly redundant• Highly modular
• Multiple equal paths• VSANs limit the size
of any one SAN
ApplicationServers
Shared Storage
Unified Storage Mgmt
iSCSI
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
525525525© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Network Design Goals
• Performance Planned hierarchy, managed oversubscription, and modular design
• Reliability, Availability, and Scalability—RASLimited failure domains, leveraged VSANs, and modular design
• Cost efficienciesConsolidated storage, central management, and leveraged resources
• SecurityLimited domains, RBAC management, and consistent architecture
• A base to enable services and solutionsBusiness continuance and disaster recoveryManagement of heterogeneous storage elementsUbiquitous access to storage from anywhereInfrastructure for storage virtualization
Architecture Summary:
526526526© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Architecture: End-to-End SAN Architecture
Intelligent WorkgroupStorage Networks
Highly ScalableStorage Networks
Optical Network
Resilient OpticalTransport Networks
Synchronous Replication—Optical (FCIP/FC)
SONET Network
Multiprotocol/MultiserviceSONET Network
Asynchronous Replication—FCIP over SONET FCIPRemote StorageAccess
FC FC
FC FC FC
FC
FC
FC
FC FC
FC FC FC
FC FC
FC FC FC
iSCSI-EnabledStorageNetwork
EthernetSwitches
FC
iSCSI
FC
iSCSI
FC
iSCSI
FC
iSCSI
FC
iSCSI
FC
iSCSI
FC
iSCSI
FC
iSCSI
FC
iSCSI
FC
iSCSI
FC
iSCSI
FC
iSCSI
FC
iSCSI
FC
iSCSI
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
Q & A
527527527© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
528528528© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Complete Your Online Session Evaluation!
WHAT: Complete an online session evaluation and your name will be entered into a daily drawing
WHY: Win fabulous prizes! Give us your feedback!
WHERE: Go to the Internet stations located throughout the Convention Center
HOW: Winners will be posted on the onsiteNetworkers Website; four winners per day
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
529© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
EXTRAS
530© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
FC LOOP OPERATIONS
531531531© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
532532532© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Single Port ARB
RX TX
Port
RX TX
Port
RX
TX
Port
RX Port
TX
IDLE IDLE
IDLEIDLE
AL_PA 2A
AL_PA EF
AL_PA B2AL_PA 01
1. The Loop is initially filled with IDLES
2. Each port is in the monitoring state
3. Because of no activity CFW = Idle
4. Rx IDLES are replaced with CFW
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
533533533© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Single Port ARB
RX TX
Port
RX TX
Port
RX
TX
Port
RX Port
TX
ARB(01) IDLE
IDLEIDLE
AL_PA 2A
AL_PA EF
AL_PA B2AL_PA 01
1. Port_01 begins to arbitrate for access to the Loop
2. Port_01 changes its CFW from IDLE to ARB(01)
3. Port_01 transmits ARB(01) when a fill word is required
534534534© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Single Port ARB
RX TX
Port
RX TX
Port
RX
TX
Port
RX Port
TX
ARB(01) ARB(01)
IDLEARB(01)
AL_PA 2A
AL_PA EF
AL_PA B2AL_PA 01
1. ARB(01) is Rx by the next port and updates its CFW to ARB(01)
When a Port Discards Rx Fill Words and Transmits the CFW this Allows the Port to Compensate for Clock Differences Between Rx Data Stream and Tx Data Stream
2. Whenever a fill word is required ARB(01) is used; With no other activity on the loop ARB(01) is sent
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
535535535© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Single Port ARB
RX TX
Port
RX TX
Port
RX
TX
Port
RX Port
TX
ARB(F0)
OPN
ARB(01)
AL_PA 2A
AL_PA EF
AL_PA B2AL_PA 01
1. When Port_01 receives its own ARB(01) it wins arbitration
2. Port_01 sends on OPN to open a loop circuit and changes its CFW to ARB(F0)
3. Port_01 discards any Rx’ed ARB(x)
536536536© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Single Port ARB
RX TX
Port
RX TX
Port
RX
TX
Port
RX Port
TX
IDLE ARB(F0)
ARB(F0)ARB(F0)
AL_PA 2A
AL_PA EF
AL_PA B2AL_PA 01
1. As each port Rx’s the ARB(F0) it updates its CFW to ARB(F0)
2. Assuming that no other port is arbitrating, ARB(F0) travel the complete loop
3. When ARB(F0) is Rx’ed by Port_01 the CFW in Port_01 is changed to IDLE
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
537537537© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Single Port ARB
RX TX
Port
RX TX
Port
RX
TX
Port
RX Port
TX
IDLE IDLE
ARB(F0)IDLE
AL_PA 2A
AL_PA EF
AL_PA B2AL_PA 01
1. Each port receives the IDLE and updates its CFW to IDLE
Discarding the Receiving Arb(x) Prevents Any Other Port from Winning Arbitration
2. Assuming the no other port is arbitrating and the IDLES travel the complete loop
3. As long as Port_01 owns the loop it discards any Rx’ed IDLE or ARB(x) and continues to send its CFW when necessary
538538538© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Multiple Port ARB
RX TX
Port
RX TX
Port
RX
TX
Port
RX Port
TX
ARB(01) IDLE
IDLEARB(B2)
AL_PA 2A
AL_PA EF
AL_PA B2AL_PA 01
1. Port_01 begins arbitrating for access to the loop; Done by replacing IDLE and ARB(x) with ARB(01)
2. Port_B2 also begins arbitrating for the loop; It replaces Idle and ARB(x) with ARB(B2)
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
539539539© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Multiple Port ARB
RX TX
Port
RX TX
Port
RX
TX
Port
RX Port
TX
ARB(01) ARB(01)
ARB(B2)ARB(B2)
AL_PA 2A
AL_PA EF
AL_PA B2AL_PA 01
1. The ARB(01) gets to Port_2A which updates its CFW with ARB(01) and transmits this when the CFW is needed
2. The ARB(B2) also travels to Port_EF which updates its CFW with ARB(B2)
540540540© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Multiple Port ARB
RX TX
Port
RX TX
Port
RX
TX
Port
RX Port
TX
ARB(01) ARB(01)
ARB(B2)ARB(01)
AL_PA 2A
AL_PA EF
AL_PA B2AL_PA 01
1. When Port_B2 receives ARB(01) it changes its CFW to ARB(01) because of 01 has higher priority(Lower AL_PA wins)
Because Port_B2’s ARB(B2) Is Replaced with ARB(01) It Will Not Win Arbitration at this Time
2. When Port_01 receives ARB(B2) it is replaced with ARB(01)
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
541541541© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Multiple Port ARB
RX TX
Port
RX TX
Port
RX
TX
Port
RX Port
TX
ARB(F0) and OPN
ARB(01)
ARB(01)ARB(01)
AL_PA 2A
AL_PA EF
AL_PA B2AL_PA 01
1. ARB(01) is Rx by Port_01 and wins arbitration
2. Port_01 then opens the loop circuit and updates it’s CFW with ARB(F0) when a fill word is required
3. Port_B2 is still arbitrating but is lower priority
542542542© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Multiple Port ARB
RX TX
Port
RX TX
Port
RX
TX
Port
RX Port
TX
ARB(F0) ARB(F0)
ARB(01)ARB(B2)
AL_PA 2A
AL_PA EF
AL_PA B2AL_PA 01
1. Port_2A receives ARB(F0) and updates the CFW to F0
2. Port_B2 replaces the lower-priority ARB(F0) and transmits ARB(B2)
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
543543543© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Multiple Port ARB
RX TX
Port
RX TX
Port
RX
TX
Port
RX Port
TX
ARB(F0) ARB(F0)
ARB(B2)ARB(B2)
AL_PA 2A
AL_PA EF
AL_PA B2AL_PA 01
1. Port_EF updates its CFW to ARB(B2) and transmits on to Port_01
2. Port_01 Tx’s ARB(F0)
When Port_01 Relinquishes Control of the Loop It Changes Its CFW to ARB(B2) Allowing Port_B2 to Win
3. Port_B2 continues to replace F0 with B2; Port_01 discards all Rx’ed ARB(x) ordered sets
544544544© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Lower Priority Port ARB
RX TX
Port
RX TX
Port
RX
TX
Port
RX Port
TX
IDLE IDLE
IDLEARB(B2)
AL_PA 2A
AL_PA EF
AL_PA B2AL_PA 01
1. Port_B2 begins to arbitrate for the loop by changing CFW to B2
2. Each Rx’ed IDLE and lower-priority ARB(x) is discarded by Port_B2 and the ARB(B2) is substituted in its place
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
545545545© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Lower Priority Port ARB
RX TX
Port
RX TX
Port
RX
TX
Port
RX Port
TX
IDLE IDLE
ARB(B2)ARB(B2)
AL_PA 2A
AL_PA EF
AL_PA B2AL_PA 01
1. ARB(B2) propagates around the loop to Port_EF
2. Port_EF changes its CFW to ARB(B2) and Tx’s the ARB(B2) whenever a fill word is needed
546546546© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Lower Priority Port ARB
RX TX
Port
RX TX
Port
RX
TX
Port
RX Port
TX
ARB(B2) IDLE
ARB(B2)ARB(B2)
AL_PA 2A
AL_PA EF
AL_PA B2AL_PA 01
1. The ARB(B2) propagates around the loop to Port_01
2. Port_01 changes its CFW to ARB(B2) and Tx’s ARB(B2) whenever a fill word is needed
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
547547547© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Lower Priority Port ARB
RX TX
Port
RX TX
Port
RX
TX
Port
RX Port
TX
ARB(01) ARB(B2)
ARB(B2)ARB(B2)
AL_PA 2A
AL_PA EF
AL_PA B2AL_PA 01
1. Port_01 begins arbitrating after a single ARB(B2) has passed
3. The single ARB(B2) travels around the loop to Port_2A. Port_2A passes the ARB(B2)
2. Port_01 has higher priority than Port_B2 and discards ARB(B2) and replaces it with ARB(01)
4. When ARB(01) is Rx’ed at Port_2A its CFW is changed from B2 to 01
ARB(01)
XX
548548548© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Lower Priority Port ARB
RX TX
Port
RX TX
Port
RX
TX
Port
RX Port
TX
ARB(01)
ARB(B2) ARB(F0) and OPN
AL_PA 2A
AL_PA EF
AL_PA B2AL_PA 01
1. The single ARB(B2) is Rx’ed by Port_B2 which wins arbitration and begins to discard any Rx’d ARB(x)
2. Port_B2 changes its CFW to ARB(F0)
ARB(01)
ARB(B2)
© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
549549549© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X
Lower Priority Port ARB
RX TX
Port
RX TX
Port
RX
TX
Port
RX Port
TX
ARB(01) ARB(01)
ARB(F0)ARB(F0)
AL_PA 2A
AL_PA EF
AL_PA B2AL_PA 01
1. Port_EF changes its CFW to ARB(F0) and sends it on to Port_01
2. Port_01 substitutes ARB(01 for every ARB(F0) it receives
3. Port_B2 discards the ARB(01) and sends ARB(F0) as its fill word
4. When Port_B2 relinquishes the loop, it will change its CFW to ARB(01) and allow Port_01 to win the loop
Recommended