IBM BladeCenter Fundamentals
Ethernet connectivity and options 5.3 Copyright IBM Corporation
20091Welcome to Ethernet Connectivity and Options for the
BladeCenter.Unit objectivesAfter completing this unit, you should
be able to:
List the BladeCenter components required for Ethernet
connectivity
Name the management components supporting Ethernet switch
modules
Select the Ethernet I/O switch module to meet a specific
requirement
List the components required to provide additional Ethernet
connectivity to a blade server
Identify the method used to provide blade server Ethernet
connection redundancy 2 Copyright IBM Corporation 20092After
completing this unit, you will be able to:List the BladeCenter
components required to connect Blade servers through the
BladeCenter chassis to an external network.Name the BladeCenter
management components necessary to manage the Ethernet I/O switch
modulesSelect the Ethernet I/O Switch Module to meet a specific
requirementList the components required to provide additional
Ethernet connectivity to a Blade ServerIdentify the method used to
provide Blade Server Ethernet connection redundancy
BladeCenter chassis I/O overviewBladeCenter chassis I/O
overview
Internal chassis traffic flow
Switch I/O module management connectivity
BNT (Nortel) Ethernet switch I/O modules
Server connectivity module
Pass-thru module
High speed I/O modules overview
Ethernet expansion cards
3 Copyright IBM Corporation 20093In this first topic, we will
discuss the BladeCenter chassis I/O implementation.BladeCenter
chassis overview: Switching path
BladeCenter Chassis Physical Components4 Copyright IBM
Corporation 20094This slide illustrates the major physical
components supporting Blade server to Ethernet Switch I/O Module
BladeCenter chassis connectivity.The topology of the physical
components, Blade server to Switch I/O Module, is common to the
BladeCenter chassis for multiple I/O Switch Module types.
Chassis overview: Gigabit Ethernet routingMidplane (Upper
Section)Midplane (Lower Section)SW Module 11..14SW Module
31..141..14SW Module 21..14SW Module 4Network InterfaceExpansion
CardSERDES EthernetSERDES EthernetLine 1Line 3Line 4Line 2Processor
blade #15 Copyright IBM Corporation 20095BladeCenter E ExampleEach
Blade server has a minimum of 2 and, depending upon the model, a
maximum of of 8 LAN interfaces. In particular, the BladeCenter
processor blade has 2 SERDES-based Gb Ethernet interfaces, one for
each Midplane connector. With a daughter card installed, 2 more
network interfaces can be added. Each switch module receives one
LAN input from each processor blade, for a total of 14 inputs. The
following partial listing illustrates the routing:Processor blade 1
LAN 1 (the on-board NIC) connects to Switch Module 1, input
1Processor blade 1 LAN 2 (the 2nd on-board NIC) connects to Switch
Module 2, input 1Processor blade 1 LAN 3 (the 1st NIC on the
optional GbE Daughter card) connects to Switch Module 3, input
1Processor blade 1 LAN 4 (the 2nd NIC on the optional GbE Daughter
card) connects to Switch Module 4, input 1Processor blade 2 LAN 1
(the on-board NIC) connects to Switch Module 1, input 2Processor
blade 2 LAN 2 (the 2nd on-board NIC) Switch Module 2, input
2Processor blade 2 LAN 3 (the 1st NIC on the optional GbE Daughter
card) connects to Switch Module 3, input 2Processor blade 2 LAN 4
(the 2nd NIC on the optional GbE Daughter card) connects to Switch
Module 4, input 2
On the processor blade, LAN 1 and LAN 2 are the on-board SERDES
Gbit Ethernet interfaces, and are routed to switch module 1 and
switch module 2 respectively. This applies to processor blade. LAN
3 and LAN 4 go to switch modules 3 and 4 respectively, and are only
used when a daughter card is installed. Unless a daughter card is
installed in one or more processor blades, there is no need for
switch modules 3 and 4. Further, the switch modules have to be
compatible with the LAN interface generated by the processor blade.
If a Fibre Channel daughter card is installed in an BladeCenter
processor blade, then switch modules 3 and 4 must also be fibre
channel based.
Chassis overview: Ethernet port assignments (1 of
2)1BS121BS1421BS221BS321BS421BS521BS621BS721BS821BS921BS1021BS1121BS1221BS132
Bottom ESM (Bay 2)MM1MM2Top ESM (Bay 1) 1 2 3 4 5 6 7 8 9 10 11 12
13 14 Mgt 1 Mgt 2 ESMExternal uplinks ESMExternal uplinks MM1Uplink
MM2UplinkBladeCenter ChassisEth1Eth1Eth0Eth0 1 2 3 4 5 6 7 8 9 10
11 12 13 14 Mgt 1 Mgt 26 Copyright IBM Corporation 20096BladeCenter
E IllustrationFor all but the BladeCenter S chassis, ports INT1
through INT14 connect to the blade server slots 1 through 14
respectively. Each switch has preset default values for ports going
to the blade servers. Depending upon which vendor the switch is
from will determine these default values. They are hard-coded to
Auto negotiation, but only support 1000/full duplex to the blade
servers at this time. Ports Mgt 1 and Mgt 2 connect to Management
Modules 1 and 2, respectively. As with the internal ports, there
are default values for these ports which are vendor specific. Their
speed is hard-coded at 100 full and cannot be changed. Ports MGT1
and MGT2 cannot be disabled. This is by design to ensure that the
links to the BladeCenter Management Modules are not inadvertently
or intentionally brought down by the administrator. Only one of
these ports (MGT1 or MGT2) is active at one time (since only one
Management Module is active at any given time). The status for Mgt
2 will only change to up/up when the ESM receives an event
notification that the second Management Module is active. The
external ports also have vendor specific default values for ports
going to external connections. These ports default to shutdown when
in a new BladeCenter. You must use the Management Module Web
interface to set External Ports to Enabled to bring them up the
first time.
Chassis overview: Ethernet port assignments (2 of
2)1BS121BS221BS321BS421BS521BS62MM1Top ESM (Bay 1) 1 2 3 4 5 6 8 9
10 11 12 13 Mgt 1 ESMExternal uplinks MM1UplinkBladeCenter S
ChassisEth1Eth07 Copyright IBM Corporation 20097BladeCenter S
ExampleUnlike the other BladeCenter chassis, the BladeCenter S has
two ports per blade to the Ethernet switch module.Ports 1 through 6
connect to the blade server slots 1 through 6 respectively. Each
switch has preset default values for ports going to the blade
servers. Depending upon which vendor the switch is from will
determine these default values. They are hard-coded to Auto
negotiation, but only support 1000/full duplex to the blade servers
at this time. There is also a management port from the switch
module which connects to the Advanced Management Module. As with
the internal ports, there are default values for these ports which
are vendor specific. Their speed is hard-coded at 100 full and
cannot be changed. The management port cannot be disabled. This is
by design to ensure that the link to the Advanced Management Module
is not inadvertently or intentionally brought down by the
administrator. The external ports also have vendor specific default
values for ports going to external connections. These ports default
to shutdown when in a new BladeCenter. You must use the Advanced
Management Module Web interface to set External Ports to Enabled to
bring them up the first time.
Internal chassis traffic flowBladeCenter chassis Ethernet
overview
Internal chassis traffic flow
Switch I/O module management connectivity
BNT (Nortel) Ethernet switch I/O modules
Server connectivity module
Pass-Thru module
High speed I/O modules overview
Ethernet expansion cards
8 Copyright IBM Corporation 20098The next slide section
illustrates the internal chassis flow for Ethernet Layer 2 traffic
along with examples of NIC teaming (bonding) and trunk
failover.
Ethernet connectivity: Layer 2 traffic flow 14 Internal Blade
portsManagementModule PortsCPUHidden internal filter that blocks
traffic flow between the External ports and Mgmt Mod ports.
Prevents STP loops between ESM uplinks and MM uplinks (as well as
between ESMs)Ethernet Switch Module (ESM)
MM1MM2BladeCenter ChassisUpstream connectionsUpstream
connectionsRed = Ethernet External ports9 Copyright IBM Corporation
20099Layer 2 support means that the Ethernet switch provides basic
connectivity. It is capable of dealing with frame headers that
contain physical addresses such as MAC addresses, for example, to
forward network traffic based on a table of MAC addresses, or to
distribute traffic across several physical links to form one
logical higher-bandwidth link.
The diagram above illustrates the internal Layer 2 traffic flow
in the Ethernet Switch Modules (ESMs). The hard coded filter in the
Nortel ESM blocks all traffic between the external ports and the
Management Module ports. Two ESMs in the same BladeCenter chassis
exchange Layer 2 frames across the Management Module.
Also from the diagram we see the following:Two ESMs in the same
BladeCenter chassis can ping or telnet to each other without
connecting external ports. They cannot pass user data to each other
via this path, which passes traffic through the Management
Module.The internal blade ports are not on the same VLAN as the
Management Module ports. Traffic flow: NIC teaming and trunk
failover1 2 3 4 5 61 2 3 4 5 6ESMESMNIC 1NIC 2XXTeamed
Active/StandbyVLAN XxBlade ServerLogical NIC InterfaceIf failure
anywhere on the link toward the upstream switch, the NIC on blade
server does not know about the failure and maycontinue to send
traffic toward the top switch, which will discard the traffic. The
Trunk Failover feature addresses this issue.
If the switch fails in such a way that the link toward the Blade
server goes down, or NIC fails, Blade server can sense this and
redirect traffic out the other NIC toward the bottom switch. NIC
Teaming can take care of this without the need for trunk
failover.10 Copyright IBM Corporation 200910Network Trunks and
FailoverTrunk failover works by shutting down ports directly
connected to the configured blade servers when the configured
upstream trunks go down. The internal ports are put into disabled
state, and the servers react as though the cable to the network
card on a free-standing server had been unplugged. When the
configured external trunks recover, the internal ports are
re-enabled.NIC Teaming is a function that is provided by Broadcom,
the manufacturer of the NIC chips used on the Blade Servers, in
their software. Broadcom provides the Broadcom Advanced Services
Protocol (BASP) which includes NIC teaming, as well as the Broadcom
Advanced Control Suite (BACS) which is a Windows application which
helps configure NIC teaming. NIC teaming allows two or more
physical NICs to be treated as a single logical network object in
Windows or a single /dev/eth file in Linux. The single object or
file can then be assigned network properties such as an IP address
in the same way as any other NIC.
Traffic flow: Trunk failover example 1If Po1 goes down, trunk
failover takes over and shuts down internal downstream defined
port(s)This alerts NIC Teaming to an upstream failure at which
point Teaming switches to the other path out of the serverThis
example shows a single VLAN to Teamed NICs, It is possible to also
carry multiple VLANs to the Teamed NICs1 2 3 4 5 61 2 3 4 5
6ESMESMNIC 1NIC 2Teamed Active/StandbyVLAN XBlade ServerLogical NIC
InterfaceVLAN XMust be carried between ESMs via an external
pathLink state group1 UpstreamLink state group1 UpstreamLink state
group DownstreamLink state group DownstreamVLAN X Must be carried
on all uplinks from ESMNIC TeamingPo1Po1Po1Po2Po211 Copyright IBM
Corporation 200911Lets look at the first example on trunk failover.
Assume that NIC Teaming is on and blade NIC port connected to ESM 1
is active and other is standby. If something goes wrong with the
internal link to the first ESM then teaming driver will do a
failover. But what will happen if external connections will be lost
(connection from the first ESM to the first Enterprise Switch)? The
answer is that nothing happens because internal link is still on
and teaming driver does not do a failover. So network service
becomes unavailable. To address this issue, the trunk failover
technique is used. Trunk failover can disable all internal ports on
ESM in the case of an upstream links failure. Disabled port means
no link so NIC teaming driver performs a failover. This is a
special feature supported only on IBM BladeCenter ESMs. Thus if
Trunk Failover is enabled and we lose connectivity with enterprise
switch 1 then NIC teaming driver will perform a failover and
service will be available through the second enterprise switch and
the second ESM.Traffic flow: Trunk failover example 2If only one
port channel goes down, spanning tree allows backup path to other
port channel
If both port channels go down, trunk failover takes over and
shuts down internal downstream defined port(s)This alerts NIC
Teaming to an upstream failure at which point it switches to the
other CIGESMThis example shows a single VLAN to Teamed NICs, It is
possible to also carry multiple VLANs to the Teamed NICs
1 2 3 4 5 61 2 3 4 5 6ESMESMNIC 1NIC 2Teamed Active/StandbyVLAN
XBlade ServerLogical NIC InterfaceLink state group1 DownstreamLink
state group1 DownstreamVLAN X Must be carried on all uplinks from
ESMNIC TeamingPo2Po1Po1Po2Po2Link state group1 UpstreamLink state
group1 UpstreamPo1Po2Po3Po312 Copyright IBM Corporation 200912In
the second example, we have a single link state group on each ESM.
If only one port channel goes down, spanning tree allows backup
path to other port channel. If both port channels go down, trunk
failover takes over and shuts down internal downstream defined
port(s). This alerts NIC Teaming to an upstream failure at which
point it switches to the other ESM. This example shows a single
VLAN to Teamed NICs, it is possible to also carry multiple VLANs to
the Teamed NICs.
Switch I/O module management connectivityBladeCenter chassis
Ethernet overview
Internal chassis traffic flow
Switch I/O module management connectivity
BNT (Nortel) Ethernet switch I/O modules
Server connectivity module
Pass-Thru module
High speed I/O modules overview
Ethernet expansion cards
13 Copyright IBM Corporation 200913Management of the Ethernet
Switch I/O module is covered next.
Management connectivity for the ESMManagement ModuleEthernet
Switch ModuleManagementNetworkRouted ProductionNetwork213Web
Interface1B1A2344LegendEthernetI2C pathSerialSerial path5
Management WorkstationBlade ServerI2C interfaceEthernet path14
Copyright IBM Corporation 200914The management path for the
Ethernet Switch Module can be configured for access through the
switch directly via external ports on the switch, or through the
AMM network path. In the above figure:Paths 1 and 2 might be
classified as Out-of-band or In-bandPaths 3 and 4 are classified as
traditional In-band managementThe AMM uplinks can be used to manage
the Ethernet Switch I/O Module (ESM), or ESM uplinks, but not both
at the same time- The deciding factor is the setting of External
management over all ports for the ESM(s) in the AMM.If Disabled
(default), ESM will not respond to ARPs for its address on any
ports except Mgt ports 1 and 2.If Enabled, ESM will respond to ARPs
for its address over all of its portsPath 5 is sometimes classified
as a form of Out-of-band and is always available
Setting the External management over all ports to disabled will
allow management of the ESM only through the chassis management
path provided through the AMM.BNT (Nortel) Ethernet switch I/O
modulesBladeCenter chassis Ethernet overview
Internal chassis traffic flow
Switch I/O module management connectivity
BNT (Nortel) Ethernet switch I/O modules
Server connectivity module
Pass-Thru module
High speed I/O modules overview
Ethernet expansion cards
15 Copyright IBM Corporation 200915In this topic, we will
discuss the Blade Network Technologies (BNT) Ethernet Switch I/O
Modules.BNT GbESM switch I/O modules: Layer 2-3Layer 2/3 copper
GbESM14 internal 1 Gb FDX connections6 external 1 Gb FDX
connections2 10/100 Mb FDX management ports
Layer 2/3 Fibre GbESM14 internal 1 Gb FDX connections6 external
1 Gb FDX fibre (SX) connections2 10/100 FDX management ports
BNT Copper and Fibre Layer 2-3 Switch I/O Modules16 Copyright
IBM Corporation 200916The BNT Layer 2/3 Copper and Fiber Gigabit
Ethernet Switch Modules (GbESM) for standard mode of operation
include the following features and functions:Internal ports - 14
internal full-duplex Gigabit ports, one connected to each of the
blade servers in the BladeCenter unit and two internal full-duplex
10/100 Mbps ports connected to the management moduleExternal ports
- Copper switch: Six 1000BASE-T copper RJ-45 connections for making
10/100/1000 Mbps connections - Fiber switch: Six 1000BASE-SX SFP
transceiver-based LC fiber connections for making 1000 Mbps
connections
An RS-232 serial port that provides an additional means to
install software and configure the switch module.BNT L2/3 GbESM:
Simplified block diagramGigabit Ethernet AcceleratorApplication
Switch connectorManagement ProcessorMemoryControlData6 external
10/100/1000 Mbps Copper Ports(Auto-negotiation enabled)Or6 external
10/100/1000 Mbps Copper Ports(Auto-negotiation enabled)And14
internal 1000 Mbps server links2 internal 100 Mbps management
links17 Copyright IBM Corporation 200917The BNT GbESM has two
Gigabit Ethernet Aggregator (GEAs) for switching. It has 1 MB on
chip cache for packet buffers and supports 20 Gigabit Ethernet
ports (14 internal ports and six external ports).
BNT L2/3 GbESM: Network interfacesBlade 1Blade 2Blade 3Blade
12Blade 13Blade 14Int1 (1)Int2 (2)Int3 (3)Int12 (12)Int13 (13)Int14
(14)Ext1 (17)Ext2 (18)Ext5 (21)Ext6 (22)Mgt1 (15)Mgt2
(16)Management Module 1Management Module 2Customer NetworkDefault
VLANVLAN 1Management VLANVLAN 4095IP Interface 12818 Copyright IBM
Corporation 200918Ports INT1 through INT14 connect to the blade
server slots 1 through 14, respectively. The preset default values
for ports going to the blade servers are:IEEE 802.1Q tagging is
enabledDefault VLAN is VLAN1VLAN 4095 is reserved for Serial over
LANThe default values for ports going to the Management Modules
(ports MGT1 andMGT2) are for both ports to be hard-coded as
untagged VLAN 4095 (internal management VLAN).Ports EXT1 through
EXT6 connect to the external ports 1 through 6, respectively. Their
preset default values are untagged and configured on VLAN 1.
BNT Nortel layer 2-7 GbESMThe Layer 2-7 GbESM features:Support
for up to 300,000 simultaneous Layer 2-7 sessions with full
wire-speed packet forwarding for all connectionsLink aggregation on
the four external portsLayer 4 load balancing for computing
optimizationVirtual router redundancy to facilitate high
application availability in the event of application, operating
system or hardware failures
Layer 2/3Layer 4Layer 5-7
19 Copyright IBM Corporation 200919The four external 1000BASE-T
Ethernet ports support 10/100/1000 Mbps connections to external
Ethernet devices, such as backbones, end stations, and servers.
These ports are respectively identified as EXT1, EXT2, EXT3, and
EXT4 in the switch configuration menus and are labeled 1 through 4
(from top to bottom) on the GbE switch module.
BNT (Nortel) Layer 2-7 GbESM: Additional function
20 Copyright IBM Corporation 200920Layered Network Model and
Benefits of a L2-7 Switch:The OSI 7-layer model is represented by
the gray column on the left, and then some example protocols are
listed here, such as IP being at Layer 3, TCP at layer 4,
(alternatively you could have UDP), SSL providing encryption, and
HTTP for communication at Layer 7. A router has the LAN interface
and directs traffic based upon IP addresses. An Application Switch
sits inside of a Data Center; a Point of Presence, or regional
office, accepting the traffic from the router and making the
intelligent decision on how to direct that traffic. An Application
Switch can operate on information found in Layers 2 through 7,
meaning the switch can filter and act on policies on any of that
information, and of particular interest it uniquely operates on
Layers 4 through 7, with the processing power and memory required
to perform those functions.The Nortel L2-7 can address each of the
corresponding layers of the OSI model. At L2, a switch moves an
Ethernet frame from one interface to another not knowing if theres
a server ready to accept this frame. At L7 for example, the switch
knows where to re-direct the Citrix client and in case of failure
will transparently re-direct the Citrix client to a backup server,
in the blade or outside. All functions are executed in hardware
with no added latency.
BNT 1/10 Gb uplink Ethernet switch module Combined 1 Gb and 10
Gb external ports
Three slots for 10 Gb Ethernet SFP+ modules
Six 1 Gb RJ45 ports
Fourteen internal 1 Gb ports
21 Copyright IBM Corporation 200921The BNT 1/10Gb Uplink
Ethernet Switch Module for IBM BladeCenter is a switch option that
provides full Layer 2 and 3 switching and routing capability with
combined 1 Gb and 10 Gb uplinks in a BladeCenter chassis. The
switch module supports two unique operational modes that are
determined by the firmware edition that is loaded onto the
switch:Standard shipped Layer 2/3 modeThe Standard Layer 2/3 mode
provides full implementation of networking services and
technologies supported by the switch. SmartConnect mode or Virtual
Switch Extension (VSE) modeSmartConnect mode provides a
preconfigured subset of functions to enable basic Layer 2
connectivity only. SmartConnect mode is used for less complex
environments where only basic network connectivity is required and
there is no need for advanced networking services. SmartConnect
mode employs a reduced set of features through a GUI that also
allows configuration of virtualization features.
VMreadyThe switchs VMready feature makes it virtualization
aware. The switch automatically discovers the virtual machines of
hypervisors that are connected to internal ports on the switch. It
monitors the creation and movement of virtual machines, whether
they are moved for maintenance reasons or replicated to enable
application scaling. It takes action to ensure that network
settings are migrated along with the virtual machine to help
prevent the possibility of virtual machines being moved to servers
with the wrong virtual local area network (VLAN) or other network
settings. SmartConnect accepts up to 1024 virtual machines.
Virtual aggregationSwitch resources can be pooled together,
combining their capacity while simplifying their management. This
can be accomplished on a number of levels:- Grouping multiple
internal and external switch ports into a single, logical switching
entity with shared bandwidth capacity. Multiple switches located in
multiple blade server chassis, even across racks, operate as one
large virtual switch. Up to 32 such Virtual Switch Groups (VSGs)
can be configured on the switch or stack.Trunking multiple switch
ports into a single, high-bandwidth link to other networking
devices. Each VSG supports up to two external trunks that can be
used either independently or as a primary and backup.
Physical ConnectivityInternal ports - 14 internal full-duplex
Gigabit ports, one connected to each of the blade servers in the
BladeCenter unit - Two internal full-duplex 10/100 Mbps ports
connected to the management module External ports - Three slots for
10 Gb Ethernet SFP+ modules (support for 10GBASE-SR only at the
time of writing) - SFP+ modules are optional and must be ordered
from IBM - Six 10/100/1000 1000BASE-T Gigabit Ethernet ports with
RJ-45 connectors - One RS-232 serial port that provides an
additional means to install software and configure the switch
moduleServer connectivity moduleBladeCenter chassis Ethernet
overview
Internal chassis traffic flow
Switch I/O module management connectivity
BNT (Nortel) Ethernet switch I/O modules
Server connectivity module
Pass-Thru module
High speed I/O modules overview
Expansion cards
22 Copyright IBM Corporation 200922In this topic, we will
discuss the Server Connectivity Module.Server connectivity module:
Product highlightsLow-cost Layer 2 deviceSimplified user interface
to enable server administrator control of deviceIdeal for SMB
customers and applicationsFunction summaryBasic Layer 2
connectivity within chassisNon-networking, server admin setup and
controlSimple, GUI/CLI interfaceCreate blade groups (simple
VLANs)Create external port groups (auto Etherchannel)Enable trunk
failover (redundancy option)Set external port speed Assign tagged
VLANs to blade groups
23 Copyright IBM Corporation 200923The Server Connectivity
Module (SCM) provides a simple Ethernet interface option for
connecting the IBM BladeCenter system to the network
infrastructure. The administrative effort and network skills
required to connect to the network are minimized. The number and
type of configuration options on the SCM are restricted to reduce
the initial setup complexity and to minimize the impact on upstream
networking devices.
Pass-Thru moduleBladeCenter chassis Ethernet overview
Internal chassis traffic flow
Switch I/O module management connectivity
BNT (Nortel) Ethernet switch I/O modules
Server connectivity module
Pass-Thru module
High speed I/O modules overview
Expansion cards24 Copyright IBM Corporation 200924In this topic,
we will discuss the IBM BladeCenter Pass-Thru Module.
Intelligent copper pass-thru module: Product highlights14
external RJ-45 copper ports for making connections to the external
network.The external ports can operate at 100 or 1000 Mbps.14
internal bi-directional ports, connected to each of the blade
serversOne-to-one mapping of internal Ethernet ports on the blade
to external copper RJ-45GUI based operator interface
25 Copyright IBM Corporation 200925Pass-thru modules are I/O
options that can be used for direct connections of blade ports to
the external infrastructure devices such as network or storage
switches. The pass-thru module is almost like a traditional network
patch-panel -- it just routes internal blade ports to a network
external to the BladeCenter. The Intelligent Copper Pass-thru
Module for IBM BladeCenter (ICPM) provides an unswitched/unblocked
network connection to each Blade server that enables the blade
servers in the BladeCenter unit to connect to an existing network
infrastructure. No configuration of the ICPM is required. The ICPM
provides a single connection per blade for every blades internal
Ethernet port. One RJ-45 female connector per blade is on the ICPM
module itself. There are no special cables required for ICPM,
because it uses standard Ethernet RJ-45 patch cords.High speed I/O
modules overviewBladeCenter chassis Ethernet overview
Internal chassis traffic flow
Switch I/O module management connectivity
BNT (Nortel) Ethernet switch I/O modules
Server connectivity module
Pass-Thru module
High speed I/O modules overview
Expansion cards
26 Copyright IBM Corporation 200926In this topic, we will
discuss the IBM BladeCenter Pass-Thru Module.
High speed I/O modules: Overview BNT 10Gb Ethernet Switch I/O
Module BladeCenter H and HT High Speed Module10 External Ports 14
Internal Ports BNT 10Gb Ethernet Pass-Thru I/O Module BladeCenter H
and HT High Speed Module 14 External Ports 14 Internal Ports
BNT 10Gb Ethernet Switch I/O ModuleBNT 10Gb Ethernet Pass-Thru
I/O Module27 Copyright IBM Corporation 200927As introduced earlier,
the BladeCenter H and BladeCenter HT chassis implement High speed
I/O modules, mounted in the horizontal configuration and accessed
through the FFh, or Form Factor horizontal Expansion cards. This
slide illustrates an overview of the High speed modules available
for Ethernet. These modules will be covered in detail in the
follow-on course for the BladeCenter H class chassis.
BNT 10Gb Ethernet Switch ModuleThe BNT 10-port 10Gb Ethernet
Switch Module for IBM BladeCenter offers the most bandwidth of any
blade switch and represents the perfect migration platform for
clients who are still at 1Gb outside the chassis by seamlessly
integrating in the existing 1Gb infrastructure. This is the first
10Gb switch for IBM BladeCenter which is convergence ready --able
to transmit Converged Enhanced Ethernet (CEE) to a Fibre Channel
over Ethernet (FCoE ) capable top-of-rack switch. This feature will
be available through a future firmware upgrade at no additional
cost. This offering will also work with BladeCenter Open Fabric
Manager, providing all the benefits of I/O Virtualization at 10Gb
speeds.
BNT 10Gb Ethernet Pass-Thru ModuleThe 10Gb Ethernet Pass-Thru
Module for IBM BladeCenter provides end-to-end non-blocking 10Gb
setup within the chassis. This device supports both Ethernet and
Converged Enhanced Ethernet (CEE ) packets, which allows the
BladeCenter Chassis to connect to an FCoE-capable top-of-rack
switch. The 14 10Gb Uplink ports are based on optical SFP+
technology, an industry standard connection media.
Expansion cardsBladeCenter chassis Ethernet overview
Internal chassis traffic flow
Switch I/O module management connectivity
BNT (Nortel) Ethernet switch I/O modules
Server connectivity module
Pass-Thru module
High speed I/O modules overview
Expansion cards
28 Copyright IBM Corporation 200928In the previous topic on
Blade servers, the Expansion cards were introduced in relationship
to the Blade server, Midplane and BladeCenter chassis. In this
topic, we will discuss the IBM Blade server Expansion cards
specific to Ethernet.
Ethernet expansion cards: Combination I/O vertical (CIOv) PCI
Express host interface Broadcom BCM5709S communication module
BladeCenter Open Fabric Manager (BOFM) support Connection to
1000BASE-X environments using BladeCenter Ethernet switches
IBM HS22 Blade Server with Ethernet CIOv card installed
CIOv Ethernet Expansion Card29 Copyright IBM Corporation
200929The Ethernet Expansion Card Combination I/O vertical (CIOv)
is an Ethernet expansion card with two 1 Gb Ethernet ports designed
for BladeCenter servers with CIOv expansion slots. It is based on n
Broadcom 5709S ASIC technology and implements TCP Offload (TOE) and
software-based iSCSI support.
The Ethernet CIOv Expansion card implements the following
features;PCI Express host interface Broadcom BCM5709S communication
module BladeCenter Open Fabric Manager (BOFM) support Connection to
1000BASE-X environments using BladeCenter Ethernet switches
Full-duplex (FDX) capability, enabling simultaneous transmission
and reception of data on the Ethernet local area network (LAN)
Failover support Preboot Execution Environment (PXE) support
Support for direct memory access (DMA) Wake on LAN support for both
ports
The Ethernet Expansion Card (CIOv) requires that a supported I/O
module is installed in bay 3 and bay 4 of the chassis that the
cards and servers are installed in. The supported I/O modules
are:
Any IBM, BNT or Cisco Ethernet Switch Module (standard
form-factor switch module) Optical Pass-thru Module Copper
Pass-thru Module Intelligent Copper Pass-thru Module
Note: The Ethernet CIOv Expansion card is currently supported on
the HS22 Blade server only.Ethernet expansion cards: Combination
form factor vertical (CFFv) Based on Broadcom 5704 chipset Two 1 Gb
ports per blade server
IBM HS22 Blade Server with Ethernet CFFv card installed
Ethernet CFFv Expansion Card30 Copyright IBM Corporation
200930The Ethernet Expansion Card Combination Form Factor vertical
is a blade server expansion card with two 1 Gb Ethernet ports. It
provides a PCI-X 1.0 interface via the Broadcom 5704 chipset. This
expansion card can be combined with a CFFh expansion card on the
same blade server.Ethernet expansion cards: QLogic iSCSI Dual ports
TCP/IP Offload Engine TC/IP stack implemented in firmware Provides
iSCSI diskless boot capabilityQLogic iSCSI StFF Expansion Card
QLogic iSCSI Management Utility31 Copyright IBM Corporation
200931Overview of iSCSI ProtocolThe iSCSI protocol is a transport
for SCSI over TCP/IP. Until recently, standard IP protocol
infrastructure (for example, Ethernet) could not provide the
necessary high bandwidth and low-latency needed for storage access.
Special communications infrastructure, mainly Fibre Channel running
Fibre Channel Protocol ,FCP (SCSI over Fibre Channel), was
developed to allow for Storage Area Networks (SANs). With the
recent advances in Ethernet technology, it is now practical (from a
performance perspective) to access storage devices over an IP
network. 1 Gigabit Ethernet is now widely available and is
competitive with 1 and 2 Gigabit Fibre Channel. 10 Gigabit Ethernet
is readily coming available. Similar to FCP, iSCSI allows storage
to be accessed over a Storage Area Network, allowing shared access
to storage. (1)
The QLogic iSCSI Expansion Card is a hardware initiator that
provides iSCSI communicationfrom the blade server to an iSCSI
storage device (target).The QLogic iSCSI Expansion card provides
the following features:Full hardware-based iSCSI initiator and TOE
for storage traffic only (TCP/IP offload engine) functionBlades
enabled to run diskless in a non-Fibre Channel SAN environment
(iSCSI Boot)Dual port card using the QLogic QMC4052
controllerNEBS-compliant for use in NGN/Telco environments
This card requires a supported Ethernet switch module in bays 3
and 4 of the BladeCenter chassis
Description adapted from the IBM RedBook:IBM BladeCenter iSCSI
SAN Solution, August, 2006Authors:Rufus CredleKhalid AnsariScott
LorditchP Chris MojicaEthernet expansion cards: High speed overview
High Speed Form horizontal (CFFh) expansion cards Interface to
Horizontally mounted I/O modules BladeCenter H and HT chassis PCI-E
bus connections
QLogic 2-Port 10Gb Converged NetworkAdapter CFFh
Broadcom 10Gb 4-Port CFFh card IBM BladeCenter 2/4-Port CFFh
card
NetXen 10Gb 2-Port CFFh card32 Copyright IBM Corporation
200932This slide illustrates Ethernet Expansion cards implementing
the PCI-E high-speed bus and horizontal high-speed I/O modules
available on the BladeCenter H and HT chassis.
QLogicThe QLogic 2-port 10Gb Converged Network Adapter (CFFh)
provides 8Gb Fibre Channel storage connectivity and 10Gb networking
over a single Converged Enhanced Ethernet (CEE) link. The QLogic
Expansion card features the following:Support for 10Gb Converged
Enhanced Ethernet (CEE) Support for Fiber Channel over Converged
Enhanced Ethernet (FCoCEE) Support for BladeCenter Open Fabric
Manager for BIOS, UEFI, and FCode Support for SAN boot over CEE,
PXE boot, and iSCSI boot
IBM BladeCenter 2/4 Port Ethernet Expansion CardThe 2/4 Port
Ethernet Expansion Card is based on the Broadcom 5709 Gigabit
Ethernet chipset. It enables 2 ports in the BladeCenter S Chassis.
It enables 4 ports in theBladeCenter H and HT chassis.The 2/4 Port
Ethernet Expansion Card features:TCP offload engine (TOE)Full
fast-path TCP offloadTCP/IP checksum offloadTCP segmentation
offloadPXE 2.0 remote boot support
BroadcomThe Broadcom 10Gb 4-Port Ethernet Expansion Card
supports up to four High-Speed Switch Modules and allows up to 40
Gbps of Ethernet bandwidth to every server in the chassis. The
Broadcom Ethernet expansion card features the following:TCP offload
engine (TOE) - Full fast-path TCP offload - TCP/IP checksum offload
-TCP/IP segmentation offloadiSCSI boot
NetXenThis is a blade server expansion card with two 10 Gigabit
Ethernet interfaces. It is a high-speed expansion card especially
designed to interface with the Nortel 10 Gb EthernetSwitch Module,
and it uses the NetXen NX2031 network controller chip. This
expansion card requires that a Nortel 10 Gb Ethernet Switch Module
be installed in switch bay 7 and bay 9.
Note: For further details, see the document used as reference
source on this page.ITSO Redbook IBM BladeCenter Products and
TechnologyAuthors:David WattsRandall DavisIlia Kroutov
Key wordsDevice Manager FailoverFibre Ethernet Graphical user
interface (GUI) Local Area Network (LAN)Layer 2 7Media Access
Control (MAC) Address Power On Self Test (POST)Spanning
TreeSerializer/deserializer (SERDES)Server Connectivity Module
(SCM) Open Shortest Path First (OSPF)UplinkVirtual Local Area
Network (VLAN) tagging Virtual Router Redundancy Protocol (VRRP)
Trunking33 Copyright IBM Corporation 200933Review Key Words
Checkpoint (1 of 2)True/False: In order to provide Ethernet
connectivity, each blade server requires an Ethernet expansion
card.
A customer requires policy management capability in the
application through Transport layers of the OSI model. Select the
switch module appropriate for these requirements.BNT 10Gb Pass-thru
ModuleBNT L2/3 Copper GbESMIBM Server Connectivity ModuleBNT L2-7
GbESM
True/False: The Ethernet switch I/O modules can be managed
through the AMM management network, or through the switch I/O
module external ports34 Copyright IBM Corporation 200934Write down
your answers here: 3.
Checkpoint solutions (1 of 2)True/False: In order to provide
Ethernet connectivity, each blade server requires an Ethernet
expansion card. Answer: False
A customer requires policy management capability in the
application through Transport layers of the OSI model. Select the
switch module appropriate for these requirements.BNT 10Gb Pass-thru
ModuleBNT L2/3 Copper GbESMIBM Server Connectivity ModuleBNT L2-7
GbESM Answer: d
True/False: The Ethernet switch I/O modules can be managed
through the AMM management network, or through the switch I/O
module external ports Answer: True35 Copyright IBM Corporation
200935Checkpoint (2 of 2)In order to support 4 Ethernet connections
from a Blade server, which of the following is correct?An expansion
blade is required Add an Ethernet expansion card and switch module
in Bay 2Add an Ethernet expansion card and switch modules in Bays 3
and 4Add an IBM Server connectivity module
True/False: NIC Teaming can be used to provide redundancy,
utilizing the paths from the blade server onboard NICs through the
midplane to Ethernet switch I/O modules in Bays 1 and 2 of the
chassis.36 Copyright IBM Corporation 200936Write down your answers
here: 4.5.Checkpoint solutions (2 of 2)In order to support 4
Ethernet connections from a blade server, which of the following is
correct?An expansion blade is required Add an Ethernet expansion
card and switch module in Bay 2Add an Ethernet expansion card and
switch module in Bays 3 and 4Add an IBM Server connectivity module
Answer: c
True/False: NIC Teaming can be used to provide redundancy,
utilizing the paths from the blade server onboard NICs through the
midplane to Ethernet switch I/O modules in Bays 1 and 2 of the
chassis. Answer: True37 Copyright IBM Corporation 200937Unit
summaryHaving completing this unit, you should be able to:
List the BladeCenter components required for Ethernet
connectivity
Name the management components supporting Ethernet switch
modules
Select the Ethernet I/O switch module to meet a specific
requirement
List the components required to provide additional Ethernet
connectivity to a blade server
Identify the method used to provide blade server Ethernet
connection redundancy 38 Copyright IBM Corporation 200938In this
unit, we presented an overview of the Ethernet implementation
within the IBM BladeCenter.Topics included Ethernet traffic flow
from the Blade servers through the chassis and Ethernet switches,
along with representative Ethernet Switch I/O modules and Blade
server expansion cards.
Having completing this unit, you should be able to:List the
BladeCenter components required to connect Blade servers through
the BladeCenter chassis to an external network.Name the BladeCenter
management components necessary to manage the Ethernet I/O switch
modulesSelect the Ethernet I/O Switch Module to meet a specific
requirementList the components required to provide additional
Ethernet connectivity to a Blade ServerIdentify the method used to
provide Blade Server Ethernet connection redundancy
The next activity will be: Lab 3 - BNT 2-3 Gigabit Ethernet
Switch Module
