1

NIC Teaming/Bonding

2

ESM Port Assignments

All ports tasked for a specific roll and can not be reallocated

ESMModule

14 ports1000 MbpsInternal linksto BladeServers

2 ports -100 MbpsInternal links to the Management Modules

4 or 6 ports 10/100/1000 Mbps RJ45

or4 ports GigE SFP Fiber

for external network connections

Service port

ESM Connections

INT1INT2INT3INT4INT5INT6INT7INT8INT9INT10INT11INT12INT13INT14

Mgt15Mgt16

EXT1EXT2EXT3EXT4

BladeServerSlot1BladeServerSlot2BladeServerSlot3BladeServerSlot4BladeServerSlot5BladeServerSlot6BladeServerSlot7BladeServerSlot8BladeServerSlot9

BladeServerSlot10BladeServerSlot11BladeServerSlot12BladeServerSlot13BladeServerSlot14

MM1MM2

Serial

BladeCenter

3

BladeCenter Ethernet Port Assignments

1

BSS1

2

1

BSS14

2

1

BSS2

2

1

BSS3

2

1

BSS4

2

1

BSS5

2

1

BSS6

2

1

BSS7

2

1

BSS8

2

1

BSS9

2

1

BSS10

2

1

BSS11

2

1

BSS12

2

1

BSS13

2

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19

Bottom ESM (Bay 2) 20

MM1

MM2

Top ESM (Bay 1) 17 18 19

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 20

ESMExternal uplinks

ESMExternal uplinks

MM1Uplink

MM2Uplink

BladeCenter Chassis

Eth1

Eth1

Eth0

Eth0

(BSS = Blade Server Slot)

4

Broadcom BASP

Broadcom Advanced Control Suite (BACS) is the GUI tool for BASP. Windows Only

Broadcom offers no support for Linux and is now ending development.

Broadcom now supports the TG3 drivers for Linux http://www.broadcom.com/support/ethernet_nic/faq_drivers.php#tg3

IBM is distributing and supporting BASP???

5

What can Broadcom do?

Create a team using: Smart Load Balancing, IEEE 802.3ad (LACP) or Generic Trunking 802.3ad-draft (Cisco EtherChannel).

Create VLAN using 802.1q (max. 64). Both of the above can be done concurrently (team w/ VLANs) Enable 802.1p (Quality of Service) New adapters are TCP Offload Engine (TOE) Enabled.

iSCSI and RDMA Capable Included on planar board on HS21, LS21, LS41 (5708s, 5706s) Requires Microsoft Scalable Networking Pack

http://www.microsoft.com/technet/itsolutions/network/snp/default.mspx

BASP Teaming can be used with third-party NICs except for VLAN support. http://www.broadcom.com/collateral/wp/570X-WP100-R.pdf

6

Flow Control IEEE 802.3x

Flow Control is a parameter that allows the user to enable or disable the receipt or transmission of PAUSE frames.

PAUSE frames enable the adapter and the switch to control the transmit rate. The side that is receiving the PAUSE frame temporarily stops transmitting.

The Server link is 1000 Mbps, Full-Duplex, Flow Control Rx/Tx. -> You can’t change it!!!

Flow Control is especially important when the uplinks are using 100Mbps, since all servers are at 1000 Mbps.

When the switch buffer memory is getting full, the GbESM issue a PAUSE frame to the server (hold-on for a moment).

To see the link status: /info/link

7

Smart Load Balancing (SLB)

Proprietary solution from Broadcom. Only works in a flat Layer 2 environment (no routers). Team MAC address is one of the physical MACs of the NICs in the team.

It’s used to initially reply to ARP requests. Transmit Load Balancing.

When the server receives an ARP request, it replies using the Team MAC. Based on the TCP/UDP information; the teaming driver will change the Source

MAC address with one of the MACs of the physical NICs.

Receive Load Balancing In order to distribute all incoming traffic among all NICs, BASP periodically issues

a Gratuitous ARP (GARP) directly to the destination MAC address with the Source MAC address of one of the physical NICs.

Router will receive this GARP and keep updating the ARP cache. ARP doesn't have an IP header, so it can’t be routed.

8

Smart Load Balancing (SLB)

Always configure SLB in Active-Standby mode. Make a note of the physical MAC address of each NIC before

creating the Team.

Once a NIC is part of a Team, it can’t be used in another Team.

SLB with auto Fall-Back means that once the primary NIC is back on-line, the Team will start using that NIC again. Keep in mind that when you replace a faulty switch, it doesn't have any

configuration, so it might not be good to automatically use it. The MM also perform a series of Link UP/Down tests during the boot process.

SLB with no failback is preferable for these reasons.

9

Generic Trunking

Equivalent of Cisco EtherChannel or Nortel Static MLT.

Team uses one MAC and IP address. Requires both links to end on the same

device also configured with the same Trunking protocol. Only possible in a BladeCenter using the HS40 or LS41 or the BCE card.

If the NICs are connected to different devices, Active-Standby is the only possible option…

If both NIC’s in the team are in Active-Active, the external GbESM ports need to end on a single switch where all ports use same trunking protocol.

Active Active

Core Network

Generic Trunk

10

Linux Bonding (802.3ad)

Functionally very similar to what BASP provides. The “bond” uses the MAC address of the first physical NIC. Output traffic is allocated to physical NICs based on (src hw addr

XOR dst hw addr). This selects the same NIC for traffic to a specific destination hw address.

SLES Bonding https://secure-support.novell.com/KanisaPlatform/Publishing/133/3929220_f.SAL_Public.html

Red Hat Bonding http://kbase.redhat.com/faq/FAQ_85_6313.shtm

VMWare ESX Bonding http://www.vmware.com/vmtn/resources/432

Link to Linux Networking Stuff.

11

Linux Bonding config in Active-Backup

sys1# insmod bonding.o mode=1 miimon=100 sys1# /sbin/ifconfig bond0 10.0.0.1 up sys1# /sbin/ifenslave bond0 eth0 sys1# /sbin/ifenslave bond0 eth1 sys1# /sbin/ifconfig bond0 Link encap:Ethernet HWaddr 00:A0:CC:D0:DA:10 inet addr:10.0.0.1 Bcast:10.0.255.255 Mask:255.255.0.0 UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1 RX packets:7224794 errors:0 dropped:0 overruns:0 frame:0 TX packets:3286647 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 eth0 Link encap:Ethernet HWaddr 00:A0:CC:D0:DA:10 inet addr:10.00.0.1 Bcast:10.0.255.255 Mask:255.255.0.0 UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1 RX packets:3573025 errors:0 dropped:0 overruns:0 frame:0 TX packets:1643167 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:100 Interrupt:10 Base address:0x1080 eth1 Link encap:Ethernet HWaddr 00:A0:CC:D0:DA:10 inet addr:10.0.0.1 Bcast:10.0.255.255 Mask:255.255.0.0 UP BROADCAST RUNNING SLAVE NOARP MULTICAST MTU:1500 Metric:1 RX packets:3651769 errors:0 dropped:0 overruns:0 frame:0 TX packets:1643480 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:100 Interrupt:9 Base address:0x1400

Bond need to be mode=1 (mode=0 is Active-Active).

The NIC in backup is the one with the “NOARP” statement.

By default, preempt is enable. -> Fall-back to the original NIC once it’s OK.

12

Locating BASP software

Software and documentation is included in driver package (.zip or .iso packages) NetXtreme (xseries servers)

http://www-3.ibm.com/pc/support/site.wss/document.do?lndocid=MIGR-43815

NetXtreme II (newer, system x servers) http://www-3.ibm.com/pc/support/site.wss/document.do?lndocid=MIGR-64505

Default extraction location is C:\Drivers\v94B2 Click on Launch.exe file

13

Installing BASP

This application allows you to update the Windows Drivers (recommended)

Management Applications is the BASP Software Default installation is acceptable

14

BACS Control Panel

Tray icon Unconfigured Control Panel

15

Creating a Team

Expert Mode Wizard Mode Be careful of Team Name, as

this becomes the Interface Name Can cause problems if name

has a space

16

Configuring BASP

Choose the Team type you wish to configure SLB is most common

17

Add adapters to Team

18

Configuring BASP ALWAYS configure the SLB Team with one NIC Active and One NIC Standby. Make sure to properly identify the Active and Standby NIC, it‘s not always the one

we think... Smart Load Balancing (SLB) is supported with or without FallBack. SLB Auto-Fallback means the Team will go automaticaly to the primary NIC when

is comes (back) online.

Note: Before a team is

configured on the server

ipconfig /all will list the NIC

MAC address. On the switch

/i/l2/fdb/dump will show you

the MAC database.

19

LiveLink

Do not configure Livelink Trunk Monitoring performs this function

20

Add VLAN

Choose whether or not to create VLANs for the TEAM VLAN tagging not supported with non-broadcom adapters

21

Give Name to VLAN

Friendly name for VLAN Not the actual VLAN ID

22

To Tag, or Not To Tag

Only one untagged VLAN per team. GbESM will handle these frames in the PVID

23

Assign the VLAN tag value

This is the actual VLAN id. VLAN must be configured on GbESM Must assign IP address for the adapter (static or DHCP)

24

A chance to review or commit configuration

Almost complete

25

Preview

Shows how configuration will look prior to saving.

Make any changes in this view. (don’t click cancel thinking you will return to the wizard)

26

Changes to interfaces in the OS

27

High-Availability at the external Network level

Active Backup

Core Network

NICs on server blades operate in “Teaming” mode.

The “NIC Teaming” option is unaware of any outages on the external network

When a failure occurs on the uplinks, the “NIC Teaming” function is not aware of the failure and the servers continue to send data to a “dead end” path!

All data is dropped by the primary switch. No error message is sent to the administrator

Although two switches are used in the BladeCenter the application sees no resilience to failures

Layer 2 switches

28

Active Backup

Core Network With the GbESM “Trunk Monitoring” option,

the switch constantly monitors the uplinks. When a failure is detected the GbESM

disables the internal ports. The “NIC Teaming” function then becomes

aware of the failure and falls back to the secondary NIC.

A Syslog message and SNMP trap are generated and sent to the Network administrator.

High-Availability at the external Network level

29

Enhanced L2 Trunk Fail-Over

Core Network

GbESM is VLAN aware and will only disable internal ports associated with the same VLAN. No un-necessary Fail-Over.

Triggers can be configured to force a Fail-Over when only some of the links fail, for bandwidth-sensitive applications.

Link Aggregation with Fail-Over can be configured with IEEE 802.3ad (LACP).

Internal ports are only re-activated once Spanning-tree is in “Fowarding“ mode.

Wait until

Spanning-Tree is

Forwarding.

30

Configuring Trunk Fail-Over

/cfg/l2/trunk 1

ena

add ext1

add ext2 /cfg/l2/failovr ON /cfg/l2/failovr/trigger 1

limit 0

ena /cfg/l2/failovr/trigger 1/amon

addtrnk 1

Create a normal Trunk. (i.e.: LACP or

EtherChannel)

Enable the Fail-Over feature.

Create a Trigger Profile.

Configure the number of required active link in the Trunk.

0 means that all links needs to be Down for the trigger to

activate.

Apply the trigger conditions to your Trunk.

31

Team 1BC-H

Team 2BC - H

Team 3BC

Team 4BC-H

MM: 192.168.70.10ESM1: 192.168.70.11ESM2: 192.168.70.12

MM: 192.168.70.20ESM1: 192.168.70.21ESM2: 192.168.70.22

MM: 192.168.70.30MMint: 192.168.70.35ESM1: 192.168.70.31ESM2: 192.168.70.32

MM: 192.168.70.40ESM1: 192.168.70.41ESM2: 192.168.70.42

Team 5BC

MM: 192.168.70.50MMint 192.168.70.55ESM1: 192.168.70.51ESM2: 192.168.70.52

Team 6BC-H

MM: 192.168.70.60ESM1: 192.168.70.61ESM2: 192.168.70.62

Team 7BC

MM: 192.168.70.70MMint 192.168.70.75ESM1: 192.168.70.71ESM2: 192.168.70.72

Team 8BC-H

MM: 192.168.70.80ESM1: 192.168.70.81ESM2: 192.168.70.82

OK

TX/RX

Link

IP

AC

DC

AC

DC

PS

1

4

PS

2

3

7

8

9

10

MM 13

MM 24

1

2

5

6

1

2

OK

TX/RX

Link

IP

TX/RX

Link

TX/RX

Link

TX/RX

Link

TX/RX

Link

1

2

3

4

OK

TX/RX

Link

TX/RX

Link

TX/RX

Link

TX/RX

Link

1

2

3

4

OK

TX/RX

Link

TX/RX

Link

TX/RX

Link

TX/RX

Link

1

2

3

4

OK

TX/RX

Link

TX/RX

Link

TX/RX

Link

TX/RX

Link

1

2

3

4

OK

PS

1

4

PS

2

3

7

8

9

10

MM 13

MM 24

1

2

5

6

1

2

OK

TX/RX

Link

IP

TX/RX

Link

TX/RX

Link

TX/RX

Link

TX/RX

Link

1

2

3

4

OK

TX/RX

Link

TX/RX

Link

TX/RX

Link

TX/RX

Link

1

2

3

4

OK

PS

1

4

PS

2

3

7

8

9

10

MM 13

MM 24

1

2

5

6

1

2

OK

TX/RX

Link

IP

TX/RX

Link

TX/RX

Link

TX/RX

Link

TX/RX

Link

1

2

3

4

OK

TX/RX

Link

TX/RX

Link

TX/RX

Link

TX/RX

Link

1

2

3

4

OK

PS

1

4

PS

2

3

7

8

9

10

MM 13

MM 24

1

2

5

6

1

2

OK

TX/RX

Link

IP

TX/RX

Link

TX/RX

Link

TX/RX

Link

TX/RX

Link

1

2

3

4

OK

TX/RX

Link

TX/RX

Link

TX/RX

Link

TX/RX

Link

1

2

3

4

OK

PS

1

4

PS

2

3

7

8

9

10

MM 13

MM 24

1

2

5

6

1

2

OK

TX/RX

Link

IP

TX/RX

Link

TX/RX

Link

TX/RX

Link

TX/RX

Link

1

2

3

4

OK

TX/RX

Link

TX/RX

Link

TX/RX

Link

TX/RX

Link

1

2

3

4

OK

OK

TX/RX

Link

IP

AC

DC

AC

DC

TX/RX

Link

TX/RX

Link

TX/RX

Link

TX/RX

Link

1

2

3

4

OK

TX/RX

Link

TX/RX

Link

TX/RX

Link

TX/RX

Link

1

2

3

4

OK

OK

TX/RX

Link

IP

AC

DC

AC

DC

TX/RX

Link

TX/RX

Link

TX/RX

Link

TX/RX

Link

1

2

3

4

OK

TX/RX

Link

TX/RX

Link

TX/RX

Link

TX/RX

Link

1

2

3

4

OK

Unavailable

Unavailable

USERID/PASSW0RD Administrator/eserver1