1© 2001, 2002 Cisco Systems, Inc. All rights reserved. Confidential and Proprietary.

Flow Control in Fibre Channel(BB_Credits)

Lincoln Dale

ltd@cisco.com

222© 2001, 2002 Cisco Systems, Inc. All rights reserved. Confidential and Proprietary. Andiamo Confidential

Topology

Hosts

FC

StorageSAN Switches

• Hosts connected to a single switch

• Storage connected to a second switch

• Switches connected to each other

333© 2001, 2002 Cisco Systems, Inc. All rights reserved. Confidential and Proprietary. Andiamo Confidential

Buffer Credits (BB_Credits) at FC Class 3

FC

• BB_Credits are the “admission control” mechanism in FC to ensure that FC switches don’t run out of buffers (FC Switches cannot drop frames)

• For Devices operating at FC Class 3 (most devices), Buffer Credits are negotiated on a per-hop basis at device FLOGI.

• BB_Credits are the only flow-control mechanism for FC Class 3.

availableBB_Credits

available

BB_Credits

12available BB_Credits

Frame towards Disk shelf

Return BB_Credit token

12111615

availableBB_Credits

1615

16

available BB_Credits

444© 2001, 2002 Cisco Systems, Inc. All rights reserved. Confidential and Proprietary. Andiamo Confidential

Buffer Credits: The Good and The Bad

FC

• Since BB_Credits are per-hop for FC Class 3:• Any speed mismatch (1G/2G) between senders & receivers

and/or

• Insufficient numbers of BB_Credits in a device

can result in Congestion due to Head of Line (HoL) Blocking• This Congestion can impact other ports & devices in a SAN

Disk shelfcapable ofsustaining200MB/sec

Tapecapable ofsustaining15MB/sec

555© 2001, 2002 Cisco Systems, Inc. All rights reserved. Confidential and Proprietary. Andiamo Confidential

Buffer Credits (BB_Credits):Utopia

FC

Disk shelfcapable ofsustaining200MB/sec

Tapecapable ofsustaining15MB/secFrame towards Tape

Return BB_Credit token

Frame towards Disk shelf

/

Available BB_Credits

12345678

12345678

Frames across switchA

16/16

Available BB_Credits

Available BB_Credits16/16

13/1612/1614/1615/1610/169/1611/16

13/1612/1614/1615/1611/16

8/167/16

16/1613/1612/1614/1615/1611/16

• In an ideal FC network, there is no blocking in any device connected to the fabric.(all devices can process frames at the same rate and negotiate equal levels of BB_Credits)

666© 2001, 2002 Cisco Systems, Inc. All rights reserved. Confidential and Proprietary. Andiamo Confidential

Buffer Credits (BB_Credits):Real world impact on performance

FC

Disk shelfcapable ofsustaining200MB/sec

Tapecapable ofsustaining15MB/secFrame towards Tape

Return BB_Credit token

Frame towards Disk shelf

/

Available BB_Credits

• In reality, not all devices can process frames at the same rate and different devices negotiate different BB_Credits

• This means that a single device is capable of causing HOL blocking across a FC switch or even across a SAN!

12345678

12345678

Frames across switchA

16/16

Available BB_Credits

Available BB_Credits16/16

13/1612/1614/1615/1610/169/1611/16

13/1614/1615/16

8/167/16

1/2 2/20/2

FC Frame blocks at head of Ingress port dueto 0 available BB_Credits

on egress port

Frames to otherdevices/ports backlog behind blocked Frame

at Head-of-Queue

1 Frame Blocked2 Frames Blocked3 Frames Blocked4 Frames Blocked5 Frames Blocked6 Frames Blocked7 Frames Blocked

6/165/16

8 Frames BlockedUncongested Fabric

While this example isn’t perhaps the mostrealistic case (flooding FC frames at adevice with only 2 BB_Credits), it doesshow the ramifications of Fibre Channelstandards mandating “frames must notbe dropped”.

777© 2001, 2002 Cisco Systems, Inc. All rights reserved. Confidential and Proprietary. Andiamo Confidential

Buffer Credits (BB_Credit) impact on performance

• While Head-of-Line Blocking is a transitory event (until some BB_Credits are returned on the blocked port), performance can be adversely affected across an entire multi-switch FC Fabric by a single blocking port

• The Cisco MDS 9000 Series has multiple features to help alleviate the problem:

• Virtual Output Queueing (VoQ) on all ports

• Deep Buffers –255 BB_Credits per port

• Fibre Channel Congestion-Control (FCC) –Detects congested ports and throttles the port causing the congestion at its origin

888© 2001, 2002 Cisco Systems, Inc. All rights reserved. Confidential and Proprietary. Andiamo Confidential

Virtual Output Queueing (VoQ)

• In a typical FC switch, each port has an input queue for frames arriving on a port

• All frames, regardless of destination, are queued in the order they’re received

• This can potentially cause Head-of-Line blocking should one of the destination ports become congested

• Frames will begin to backup in the queue and the sourcing devices will eventually have to stop transmitting data.

Switch without VoQ

Frame to Port 5

Frame to Port 5

Frame to Port 6

Frame to Port 4

Frame to Port 4

Frame to Port 6

Frame to Port 6

Frame to Port 4

Input Queue at Port 1

Top of Queue

999© 2001, 2002 Cisco Systems, Inc. All rights reserved. Confidential and Proprietary. Andiamo Confidential

Virtual Output Queueing (VoQ)

Virtual Output Queuing Model

Frame to Port 5

Frame to Port 5 Frame to Port 6

Frame to Port 4

Frame to Port 4

Frame to Port 6

Frame to Port 6Frame to Port 4

Input Queue at Port 1

Top of Virtual Queue

Input Queue at Port 1 Input Queue at Port 1

Top of Virtual Queue Top of Virtual Queue

• The MDS 9000 series utilizes Virtual Output Queues (VoQ) to eliminate Head-of-Line blocking.

• Each Destination Port is given its own Virtual Queue on the Input Port.

• Congestion on any Destination Port has no effect on traffic destined for other ports.

• Every Source Port on the MDS 9000 series has four Virtual Output Queues for every Destination Port – to cover 4 QoS levels per-port across a chassis

101010© 2001, 2002 Cisco Systems, Inc. All rights reserved. Confidential and Proprietary. Andiamo Confidential

Deep Buffers – 255 Buffer-to-Buffer Credits (BB_Credits) per port

• Typical rule-of-thumb is:

• 2 BB_Credits are required for every 1km @ 1gbit/s

• 4 BB_Credits are required for every 1km @ 2gbit/s

• Other Fibre Channel switches in the marketplace support up to 60 Buffer-to-Buffer Credits (BB_Credits)

• Performance is limited to 2gbit/s maximum over <15km

• Performance is limited to 1gbit/s maximum over <30km

• The MDS 9000 Series has up to 255 Buffer-to-Buffer Credits (BB_Credits) per port

• Wire-rate (2gbit/s) achievable for 64+km

• 1gbit/s achievable for 127+km

111111© 2001, 2002 Cisco Systems, Inc. All rights reserved. Confidential and Proprietary. Andiamo Confidential

3. Switch A rate-limits incoming traffic

Fibre Channel Congestion Control (FCC)

FC Disk shelfcapable ofsustaining200MB/sec

Tapecapable ofsustaining15MB/sec

• Consider the following topology:

• Host A is issuing write operations to disk @ 100MB/sec

• Host B is issuing writes operations to tape @ 50MB/sec

A

B

Congestion on Switch B’s port connecting the tape can cause a performance degredation for Host A writing to Disk

Congestion

Congestion

FCC mitigates the congestion by throttling the traffic at the originating port

1. Switch B detects congestion on port connecting to Tape

2. Switch B signals Switch A to quench the initiator

Limit HostBTape to 15MB/sec

Switch A Switch B

Port Rate-Limited on VoQ

121212© 2001, 2002 Cisco Systems, Inc. All rights reserved. Confidential and Proprietary. Andiamo Confidential

12© 2001, Cisco Systems, Inc. All rights reserved.