5.3 Ethernet connectivity and options. Unit objectives After completing this unit, you should be able to: List the BladeCenter components required for

5.3

Ethernet connectivity and options

2

Unit objectives

After completing this unit, you should be able to:

• List the BladeCenter components required for Ethernet connectivity

• Name the management components supporting Ethernet switch modules

• Select the Ethernet I/O switch module to meet a specific requirement

• List the components required to provide additional Ethernet connectivity to a blade server

• Identify the method used to provide blade server Ethernet connection redundancy

3

BladeCenter chassis I/O overview• BladeCenter chassis I/O overview

• Internal chassis traffic flow

• Switch I/O module management connectivity

• BNT (Nortel) Ethernet switch I/O modules

• Server connectivity module

• Pass-thru module

• High speed I/O modules overview

• Ethernet expansion cards

4

BladeCenter chassis overview: Switching path

BladeCenter Chassis Physical Components

5

Chassis overview: Gigabit Ethernet routing

Midplane (Upper Section)

Midplane (Lower Section)

SW Module 1

1………..14

SW Module 3

1………..14

1………..14

SW Module 2

1………..14

SW Module 4

Network Interface

Expansion Card

SERDES Ethernet

SERDES Ethernet

Line 1

Line 3

Line 4

Line 2

Processor blade #1

6

Chassis overview: Ethernet port assignments (1 of 2)

1

BS

1

2

1

BS14

2

1

BS

2

2

1

BS

3

2

1

BS

4

2

1

BS

5

2

1

BS

6

2

1

BS

7

2

1

BS

8

2

1

BS

9

2

1

BS10

2

1

BS11

2

1

BS12

2

1

BS13

2

Bottom ESM (Bay 2)

MM 1

MM 2

Top ESM (Bay 1)

1 2 3 4 5 6 7 8 9 10 11 12 13 14 Mgt 1 Mgt 2

ESMExternal uplinks

ESMExternal uplinks

MM 1Uplink

MM 2Uplink

BladeCenter Chassis

Eth 1

Eth 1

Eth 0

Eth 0

1 2 3 4 5 6 7 8 9 10 11 12 13 14 Mgt 1 Mgt 2

7

Chassis overview: Ethernet port assignments (2 of 2)

1

BS

1

2

1

BS

2

2

1

BS

3

2

1

BS

4

2

1

BS

5

2

1

BS

6

2

MM 1

Top ESM (Bay 1)

1 2 3 4 5 6 8 9 10 11 12 13 Mgt 1

ESMExternal uplinks

MM 1Uplink

BladeCenter S Chassis

Eth 1

Eth 0

8

Internal chassis traffic flow• BladeCenter chassis Ethernet overview





• Pass-Thru module



9

Ethernet connectivity: Layer 2 traffic flow

14 Internal Blade ports

ManagementModule

PortsCPU

Hidden internal filter that blocks traffic flow between the External ports and Mgmt Mod ports. Prevents STP loops between ESM uplinks and MM uplinks (as well as between ESMs)

Ethernet Switch Module (ESM)

MM1

MM2

BladeCenter Chassis

Upstream connectionsUpstream connections

Red = Ethernet

External ports

10

Traffic flow: NIC teaming and trunk failover

1 2 3 4 5 6 1 2 3 4 5 6

ESM ESM

NIC 1

NIC 2

X X

Teamed Active/Standby

VLAN Xx

Blade Server

Logical NIC Interface

If failure anywhere on the link toward the upstream switch, the NIC on blade server does not know about the failure and maycontinue to send traffic toward the top switch, which will discard the traffic. The Trunk Failover feature addresses this issue.

If the switch fails in such a way that the link toward the Blade server goes down, or NIC fails, Blade server can sense this and redirect traffic out the other NIC toward the bottom switch. NIC Teaming can take care of this without the need for trunk failover.

11

Traffic flow: Trunk failover example 1

• If Po1 goes down, trunk failover takes over and shuts down internal downstream defined port(s)– This alerts NIC Teaming to an upstream

failure at which point Teaming switches to the other path out of the server

– This example shows a single VLAN to Teamed NICs, It is possible to also carry multiple VLANs to the Teamed NICs

1 2 3 4 5 6 1 2 3 4 5 6

ESM ESM

NIC 1

NIC 2


VLAN X

Blade Server


VLAN X

Must be carried between ESMs

via an external path• Link state

group1 Upstream

• Link state group1

Upstream

• Link state group

Downstream

• Link state group

Downstream

VLAN X Must be carried on

all uplinks from ESM

• NIC Teaming

• Po1

• Po1

• Po1

• Po2

• Po2

12

Traffic flow: Trunk failover example 2

• If only one port channel goes down, spanning tree allows backup path to other port channel

• If both port channels go down, trunk failover takes over and shuts down internal downstream defined port(s)

– This alerts NIC Teaming to an upstream failure at which point it switches to the other CIGESM

– This example shows a single VLAN to Teamed NICs, It is possible to also carry multiple VLANs to the Teamed NICs

1 2 3 4 5 6 1 2 3 4 5 6

ESM ESM

NIC 1

NIC 2


VLAN X

Blade Server



Downstream


Downstream

VLAN X Must be carried on

all uplinks from ESM

• NIC Teaming

• Po2

• Po1

• Po1

• Po2

• Po2


Upstream


Upstream• Po1

• Po2

• Po3

• Po3

13

Switch I/O module management connectivity

• BladeCenter chassis Ethernet overview








14

Management connectivity for the ESM

Management Module

Ethernet Switch Module

ManagementNetwork

Routed ProductionNetwork

2

1 3

Web Interface

1B

1A

2

3 4

4

Legend

Ethernet

I2C path

Serial

Serial path

5

Management Workstation

Blade Server

I2C interface

Ethernet path

15

BNT (Nortel) Ethernet switch I/O modules• BladeCenter chassis Ethernet overview








16

BNT GbESM switch I/O modules: Layer 2-3

• Layer 2/3 copper GbESM– 14 internal 1 Gb FDX

connections– 6 external 1 Gb FDX

connections– 2 10/100 Mb FDX

management ports

• Layer 2/3 Fibre GbESM– 14 internal 1 Gb FDX

connections– 6 external 1 Gb FDX fibre (SX)

connections– 2 10/100 FDX management

ports

BNT Copper and Fibre Layer 2-3 Switch I/O Modules

17

BNT L2/3 GbESM: Simplified block diagram

Gigabit Ethernet

Accelerator

Ap

pli

ca

tio

n S

wit

ch

c

on

ne

cto

r

Management Processor

Memory

Control

Data

6 external 10/100/1000 Mbps Copper Ports

(Auto-negotiation enabled)

Or6 external 10/100/1000 Mbps Copper Ports

(Auto-negotiation enabled)And

14 internal 1000 Mbps server links2 internal 100 Mbps management links

18

BNT L2/3 GbESM: Network interfaces

Bla

de

1

Bla

de

2

Bla

de

3

Bla

de

12

Bla

de

13

Bla

de

14

Int1 (1)

Int2 (2)

Int3 (3)

Int12 (12)

Int13 (13)

Int14 (14)

Ext1 (17)

Ext2 (18)

Ext5 (21)

Ext6 (22)

Mgt1 (15)

Mgt2 (16)

Management Module 1

Management Module 2

Cu

sto

me

r N

etw

ork

Default VLAN

VLAN 1

Management VLAN

VLAN 4095

IP Interface 128

19

BNT Nortel layer 2-7 GbESM

• The Layer 2-7 GbESM features:• Support for up to 300,000

simultaneous Layer 2-7 sessions with full wire-speed packet forwarding for all connections

• Link aggregation on the four external ports

• Layer 4 load balancing for computing optimization

• Virtual router redundancy to facilitate high application availability in the event of application, operating system or hardware failures

Layer 2/3Layer 4Layer 5-7

20

BNT (Nortel) Layer 2-7 GbESM: Additional function

21

BNT 1/10 Gb uplink Ethernet switch module

• Combined 1 Gb and 10 Gb external ports

• Three slots for 10 Gb Ethernet SFP+ modules

• Six 1 Gb RJ45 ports

• Fourteen internal 1 Gb ports

22

Server connectivity module








• Expansion cards

23

Server connectivity module: Product highlights

• Low-cost Layer 2 device• Simplified user interface to enable

server administrator control of device

• Ideal for SMB customers and applications

• Function summary– Basic Layer 2 connectivity within

chassis– “Non-networking”, server admin setup

and control– Simple, GUI/CLI interface

• Create blade groups (simple VLANs)• Create external port groups (auto

Etherchannel)• Enable trunk failover (redundancy option)• Set external port speed • Assign tagged VLANs to blade groups

24

Pass-Thru module








• Expansion cards

25

Intelligent copper pass-thru module: Product highlights

• 14 external RJ-45 copper ports for making connections to the external network.– The external ports can operate at 100 or 1000 Mbps.

• 14 internal bi-directional ports, connected to each of the blade servers

• One-to-one mapping of internal Ethernet ports on the blade to external copper RJ-45

• GUI based operator interface

26

High speed I/O modules overview








• Expansion cards

27

High speed I/O modules: Overview

• BNT 10Gb Ethernet Switch I/O Module– BladeCenter H and HT High Speed Module– 10 External Ports – 14 Internal Ports

• BNT 10Gb Ethernet Pass-Thru I/O Module– BladeCenter H and HT High Speed Module– 14 External Ports– 14 Internal Ports

BNT 10Gb Ethernet Switch I/O Module

BNT 10Gb Ethernet Pass-Thru I/O Module

28

Expansion cards








• Expansion cards

29

Ethernet expansion cards: Combination I/O vertical (CIOv)

• PCI Express host interface • Broadcom BCM5709S communication module • BladeCenter Open Fabric Manager (BOFM) support • Connection to 1000BASE-X environments using BladeCenter Ethernet switches

IBM HS22 Blade Server with Ethernet CIOv card installed CIOv Ethernet Expansion Card

30

Ethernet expansion cards: Combination form factor vertical (CFFv)

• Based on Broadcom 5704 chipset• Two 1 Gb ports per blade server

IBM HS22 Blade Server with Ethernet CFFv card installed Ethernet CFFv Expansion Card

31

Ethernet expansion cards: QLogic iSCSI

• Dual ports• TCP/IP Offload Engine

– TC/IP stack implemented in firmware

• Provides iSCSI diskless boot capability

QLogic iSCSI StFF Expansion CardQLogic iSCSI Management Utility

32

Ethernet expansion cards: High speed overview

• High Speed Form horizontal (CFFh) expansion cards– Interface to Horizontally mounted I/O modules– BladeCenter H and HT chassis

• PCI-E bus connections

QLogic 2-Port 10Gb Converged NetworkAdapter CFFh

Broadcom 10Gb 4-Port CFFh card

IBM BladeCenter 2/4-Port CFFh card

NetXen 10Gb 2-Port CFFh card

33

Key words

• Device Manager • Failover• Fibre Ethernet • Graphical user interface (GUI) • Local Area Network (LAN)• Layer 2 – 7• Media Access Control (MAC) Address • Power On Self Test (POST)• Spanning Tree• Serializer/deserializer (SERDES)• Server Connectivity Module (SCM) • Open Shortest Path First (OSPF)• Uplink• Virtual Local Area Network (VLAN) tagging • Virtual Router Redundancy Protocol (VRRP) • Trunking

34

Checkpoint (1 of 2)

1. True/False: In order to provide Ethernet connectivity, each blade server requires an Ethernet expansion card.

2. A customer requires policy management capability in the application through Transport layers of the OSI model. Select the switch module appropriate for these requirements.a. BNT 10Gb Pass-thru Moduleb. BNT L2/3 Copper GbESMc. IBM Server Connectivity Moduled. BNT L2-7 GbESM

3. True/False: The Ethernet switch I/O modules can be managed through the AMM management network, or through the switch I/O module external ports

35

Checkpoint solutions (1 of 2)

1. True/False: In order to provide Ethernet connectivity, each blade server requires an Ethernet expansion card.

Answer: False

2. A customer requires policy management capability in the application through Transport layers of the OSI model. Select the switch module appropriate for these requirements.

a. BNT 10Gb Pass-thru Moduleb. BNT L2/3 Copper GbESMc. IBM Server Connectivity Moduled. BNT L2-7 GbESM

Answer: d

3. True/False: The Ethernet switch I/O modules can be managed through the AMM management network, or through the switch I/O module external ports

Answer: True

36

Checkpoint (2 of 2)

4. In order to support 4 Ethernet connections from a Blade server, which of the following is correct?a. An expansion blade is required b. Add an Ethernet expansion card and switch module in Bay 2c. Add an Ethernet expansion card and switch modules in Bays 3 and 4d. Add an IBM Server connectivity module

5. True/False: NIC Teaming can be used to provide redundancy, utilizing the paths from the blade server onboard NICs through the midplane to Ethernet switch I/O modules in Bays 1 and 2 of the chassis.

37

Checkpoint solutions (2 of 2)

4. In order to support 4 Ethernet connections from a blade server, which of the following is correct?a. An expansion blade is required b. Add an Ethernet expansion card and switch module in Bay 2c. Add an Ethernet expansion card and switch module in Bays 3 and 4d. Add an IBM Server connectivity module

Answer: c

5. True/False: NIC Teaming can be used to provide redundancy, utilizing the paths from the blade server onboard NICs through the midplane to Ethernet switch I/O modules in Bays 1 and 2 of the chassis.

Answer: True

38

Unit summary

Having completing this unit, you should be able to:

• List the BladeCenter components required for Ethernet connectivity

• Name the management components supporting Ethernet switch modules

• Select the Ethernet I/O switch module to meet a specific requirement

• List the components required to provide additional Ethernet connectivity to a blade server

• Identify the method used to provide blade server Ethernet connection redundancy

Documents

5.3 Ethernet connectivity and options. Unit objectives After completing this unit, you should be able to: List the BladeCenter components required for