Cisco Systems Intelligent Gigabit Ethernet Switch Module for the IBM eServer BladeCenter Cisco Systems Matt Slavin [email protected] Jan 2005

Cisco Systems Intelligent Gigabit Ethernet Switch Module for the

IBM eServer BladeCenter

Cisco SystemsMatt Slavin

[email protected] 2005

Agenda

• CIGESM Introduction and Features• CIGESM Management• Default Operation and Best Practices • Configuration Example• Serial over LAN• Hints and Tips• Important links

Cisco Systems Intelligent Gigabit Ethernet Switch ModuleCo-Branded IBM and Cisco Product for the IBM eServer BladeCenter

• Developed and Manufactured by Cisco for IBM

• Available from IBM and IBM Resellers• CIGESM GA by IBM on Jun 11th ’04• Layer 2-Plus Gigabit Ethernet switch

• Does more than pure L2 switching • Switching decisions based on L2/3/4• Runs Cisco IOS Software• Advanced Management Features

• 4 External 10/100/1000 RJ45 Copper Uplinks• 14 Internal GigE Ports for BladeServers• 1 Service port (console) sealed with cap• Up to 4 CIGESMs supported per

BladeCenter

Cisco Systems IGESM

Chassis View (Rear)

Serial Port(cap plate removed)

4 RJ45 Copper Data Ports10/100/1000 Mbps

8 MB of Flash Memory 32 MB of DRAM 8K MAC Address Support 64 Instances of Spanning-Tree Mini-Jumbo Support (1530 Bytes) Up to 250 Active VLANs

HW Block Diagram

32MB SDRAM

Supervisor CPU

8MB Flash

Serial

4x 10/100/1000

CopperRJ-45

PCI

14x SerDes GigETo Processor Blades

DC-DC Converters+12V

+1.25V

+2.5V

+3.3V

PwrOn

I2C Logic(Control/

Status Interface)

PwrOn

I2C - A I2C - B

service

12xGigESwitch ASIC

12xGigESwitch ASIC

10 Gig

10/100/1000PHY

2x FastE To Mgmt Modules

10/100/1000PHY

QUADPHY

CIGESM Port Assignments

CiscoSystemsIGESMModule

14 ports1000 MbpsInternal links toBladeServers

2 ports -100 MbpsInternal links to theManagement Modules

4 ports10/100/1000 Mbps

RJ45 links for externalnetwork connections

1 portRJ45 Service portSerial console connectionon faceplate

Cisco Systems IGESM Connections

G0/1G0/2G0/3G0/4G0/5G0/6G0/7G0/8G0/9G0/10G0/11G0/12G0/13G0/14

G0/15G0/16

G0/17G0/18G0/19G0/20

17181920

Blade1Blade2Blade3Blade4Blade5Blade6Blade7Blade8Blade9Blade10Blade11Blade12Blade13Blade14

MM1MM2

Serial

• All ports tasked for a specific roll and can not be reallocated

Internal ESM L2 Traffic Flow

14 Internal Blade ports

2 Mgmt Module Ports

CPU / IOS

4 External Ports

Hard Filter that prevents traffic flow between the External ports and Mgmt Module ports

CIGESM

PrimaryMgmt Module

Redundant Mgmt Module

Ethernet Switch 1

Ethernet Switch 2

Blade Server Modules

……..

Copper gigabit uplinks

ManagementManagement Module

NIC 1 NIC 2 NIC 1 NIC 2P2p connection

I2c Bus Management Traffic Only

Blade Chassis With Ethernet Modules

Management Ethernet ports

Example Architecture

14 Blade

Server Blades

SM 2SM 1

Core or Gateway router

Mgmt M.

Mgmt Network

Standard BladeCenter Redundancy Deployment

• 2 L2+ Switch Modules in the chassis• 2 upstream Switches aggregating the

BladeCenter systems• 2-wire channels to alternate switches• A core router at the top• Separate mgmt network

BladeCenter Chassis

Data Center Core/Distribution• Cisco Catalyst 6500 Switch Core or• Cisco Catalyst 7600 Router Gateway• Layer 4-7 Service Modules (FWSM,

SSL, VPN)• Load Balancing (EWLM)• Network Analysis (NAM-1 & NAM-2) • Intrusion Detection (IDS Module)

DistributionLayer

AccessLayer

CIGESM IOS Feature Set

Integrated SecurityIntegrated Security• SSH• SNMPv3 (Crypto)• ACPs based on Layer 2-4 • Port Security w/aging• ACLs based on DSCP filtering• Unicast MAC filtering• Private VLAN Edge• MAC Address Notification• RADIUS / TACACS+• Spanning Tree Root Guard • Trusted Boundary• Aggressive UDLD• 802.1x authentication

Availability / ResiliencyAvailability / Resiliency• Advanced STP Protocols for faster Spanning tree Convergence (Rapid PVST, PVST+, Uplink Fast, Root Guard, Port Fast, STP Backbone Fast) • 802.1d• 802.1s/w• 802.3ad (LACP) and PAgP • VTP• 802.1q VLAN Trunking• Broadcast control / Storm control

Optimized DeliveryOptimized Delivery• IGMP Snooping (Multicast)• IGMP Filtering (Multicast)• L2-L4 QoS with CoS/DSCP• Shaped Round Robin• Strict Priority Queuing• Rate Limiting• Dynamic VLANs

Enhanced ManageabilityEnhanced Manageability• Cisco IOS CLI • Console Port (IBM Service Port)• Embedded CMS• SNMP MIBs for Enterprise Management Systems• Port Mirroring (SPAN, RSPAN)• Cisco Discovery Protocol (CDP)• Network Time Protocol (NTP)• Show interface capabilities• Extensive Debugging and Trouble Shooting Capabilities• CiscoWorks support

Agenda


Intro to Managing the CIGESM

• Ways to access the switch• Management Module

• Launch telnet java applet• Launch http to CMS

• Telnet• Direct launch• Through Management Module• Through IBM Director

• CMS (Cluster Management Suite)• Direct launch• Through Management Module• Through IBM Director

• Service port (console connection)• Default - 9600, N, 8 1

• Default user name and password (all caps):• USERID• PASSW0RD (0=zero)

Management connection flows

BladeServer

Management Module

Cisco SystemsIGESM

ManagementWorkstation

ManagementNetwork

Routed ProductionNetwork

2

1 3

Web Interface

1B

1A

2

3 4

4

I2C Interface

Legend

Ethernet

Ethernet path

I2C Interface

I2C path

RJ45 Serial

RJ45 Serial path

InternalEthernetInterface

G0/15 or 16

ServicePort

5

External PortsExternalEthernetInterface

• Paths 1 and 2 might be classified as Out-of-band or In-band management paths to the CIGESM

• Paths 3 and 4 are classified as traditional In-band management paths to the CIGESM

• Ether/Or Decision:• Can use 1 and 2, OR 3 and 4, but

not both at the same time• For example, you can not

configure for path 2 and path 3 at the same time!

• Path 5 is sometimes classified as a form of Out-of-band and is always available.

Management Path Consideration - MM

• VLAN selection when using MM path to manage CIGESM

• Can set VLAN different on CIGESM, then that on MM upstream switch and still operate

• Confusing at a minimum

• Recommend to set as same VLAN to prevent confusion

ManagementModule

CIGESM

Bay 1 - 4

MGMT “Interface VLAN Y”G0/15

Simple physicalconnection

ETH 0

ETH 1

Trunked - But CIGESMmanagement packets are always

assigned to the Native VLAN(Untagged)

When using the MM interfaceto manage the CIGESM, these

VLANs may be different andthe connection still works

All packetsuntagged

All CIGESM sourceand destination

packets untagged

You can use different VLANs in this design (“interface VLAN Y” on the CIGESMand “switchport access VLAN X” on the upstream connection from the MM) andstill successfully manage the CIGESM through the MM’s interface. The onlystipulation is that the MM and the CIGESM must be in the same IP subnet.

Even though you -can- use different VLANs, it can become confusing, andthus it is usually recommended to use the same VLAN when managing theCIGESM via the Management Module uplinks.

CIGESMuplinks

Upstream Switch

“switchport access VLAN X”

Sw Bay - ExternalManagementover all ports

Disabled

Management Path Consideration - Uplink

• VLAN selection when using CIGESM path to manage CIGESM

• Even though using CIGESM uplinks, VLAN still carried over to MM

• Confusing at a minimum

• Can result in unexpected interaction

• Need to be aware of this

• End Result:• VLAN to be used should be

unique from all other connections to BladeCenter

ManagementModule

CIGESM

Bay 1 - 4

Flow Name Flow Line VLAN Description CIGESM Traffic Y Shows VLAN Y also flowing to MM

BladeCenter

MGMTInterfaceVLAN Y

VLAN Y

ETH 0

ETH 1

To BladeServers

This diagram shows an important consideration of managing the CIGESM via itsown uplinks. The fact that the CIGESM management VLAN will always be carriedover to the Management Module, via the Native VLAN between the CIGESM andthe Management Module, regardless of whether you are using the ManagementModule to manage the CIGESM or not.

Based on this attribute, certain scenarios can lead to unexpected results. Inthis document, scenario 3 is the primary recommend way to use the CIGESMuplinks for management, with scenario 4 as a possible alternative.


Enabled

Management Path Consideration

• Selection of CIGESM management VLAN when multiple CIGESMs in BladeCenter

• MM connection exists between CIGESMs

• If different management VLANs configured, error messages generated

• Native VLAN Mismatch

• Best Practice:• Use same VLAN for each

CIGESM in a given BladeCenter

MgmtModule

CIGESM in Bay 1

MGMT “Interface VLAN X”G0/15

ETH 0

ETH 1

CIGESM managementpackets untagged on

Native VLAN

Since each CIGESM “knows” what VLAN it is using for management, and they areconnected to each other via the Management Module, CDP will see that twodevices on the same connection are using a different VLAN as their respectiveNative VLANs, which will result in “Native VLAN Mismatch” messages showing upon the console port and the CIGESM logs of each CIGESM.

Even though you -can- use different VLANs, it can become confusing, andthus it is always preferred to use the same VLAN on all CIGESMs in a givenchassis.

CIGESMuplinks

CIGESM in Bay 2

MGMT “Interface VLAN Y”G0/15

CIGESMuplinks

ManagementModule uplink

To switch Bays3 and 4

Flow Name Flow Line VLAN Description Native VLAN X and Y Shows internal CIGESM connection via MM

Scenario 1 – RecommendedManagement Module provides path to the CIGESM

Four Basic Rules

1) Disable management over CIGESM uplinks

2) Should block CIGESM management VLAN from CIGESM uplinks

3) CIGESM VLAN not used by any BladeServers

4) Same IP subnet in use on both MM and CGIESM management interface

ManagementNetwork

ManagementModule

CIGESM

Bay 1 - 4

DataNetwork

Flow Name Flow Line VLAN Description Data Traffic A, B, C... Any VLAN other than X MM and CIGESM Traffic X Any VLAN NOT used for Data Traffic

Recommended Design - Physically Isolated NetworksManagement disabled for CIGESM uplinks

MM management via MM uplinkCIGESM management via MM uplink

MM and CIGESM share the same VLAN on MM uplinkData Traffic uses other VLANs

BladeCenter

MGMTInterfaceVLAN X

VLAN X VLAN A,B, C...

802.1Q Trunk(s) orMode AccessMode Access

ETH 0

ETH 1

To BladeServers

Must be same IPsubnet

VLAN X should beblocked


Disabled

Scenario 2 – Recommended Management Module provides path to the CIGESM

Same rules as Scenario 1 except:

MUST block CIGESM management VLAN from CIGESM uplinks Management

Module

CIGESM

Bay 1 - 4

Common Management/Data Network

Flow Name Flow Line VLAN Description Data Traffic A, B, C... Any VLAN other than X MM and CIGESM Traffic X Any VLAN NOT used for Data Traffic

Recommended Design - Common NetworksManagement disabled for CIGESM uplinks


MM and CIGESM share the same VLAN on MM uplinkData Traffic uses other VLANs

BladeCenter

MGMTInterfaceVLAN X


Mode Access

ETH 0

ETH 1

To BladeServers

802.1Q Trunk(s) orMode Access

VLAN X MUSTbe blocked


Disabled


Scenario 3 – RecommendedCIGESM Uplinks used for management

Five Basic Rules CIGESM management over CIGESM uplinks

1) Enable management over CIGESM uplinks

2) CIGESM mgmt VLAN not used by any BladeServers

3) Management VLAN must be carried on CIGESM uplinks

4) CIGESM mgmt VLAN not to be used as MM upstream VLAN

5) Different IP subnets in use on CIGESM and MM

ManagementModule

CIGESM

Bay 1 - 4


Flow Name Flow Line VLAN Description Data Traffic A, B, C... Any VLAN other than X or Y MM Traffic X Any VLAN NOT used for Data Traffic and NOT Y CIGESM Traffic Y Any VLAN NOT used for Data Traffic and NOT X

Recommended Design - Common NetworkManagement enabled for CIGESM uplinks

MM management via MM uplinkCIGESM management via CIGESM uplink

MM on different VLAN than CIGESMData Traffic uses other VLANs

BladeCenter

MGMTInterfaceVLAN Y

VLAN YVLAN X VLAN A, B, C...

802.1QTrunk(s)Mode Access

ETH 0

ETH 1

To BladeServers

Must bedifferent IP

subnets


Enabled

Scenario 4 – Alternative CIGESM Uplinks used for management

Same rules as Scenario 3

IP subnet on ETH0 and ETH1 of MM must be different than that used by the CIGESM

Possible drawbacks:

Mixes traffic types

Possible IP proxy issue

ManagementModule

CIGESM

Bay 1 - 4


Flow Name Flow Line VLAN Description Data Traffic X Any VLAN other than Y CIGESM and Data Traffic Y Any VLAN other than X

Possible Design - Common NetworkManagement enabled for CIGESM uplinks


MM on different VLAN than CIGESMCIGESM management and Data Traffic share a single VLAN

BladeCenter

VLAN X VLAN Y.

802.1Q Trunk(s)or Mode AccessMode Access

ETH 0

ETH 1

To BladeServers

MGMTInterfaceVLAN Y

Must bedifferent IP

subnets

VLAN Y.


Enabled

Scenario 5 – NOT Recommended

• MM may attempt to proxy ARP for BladeServers

• ARP war between MM and CIGESM on upstream network

• Could result in BladeServers reporting duplicate IP address and not getting on to network

Broken Design CIGESM management over CIGESM uplinks, shared VLANs between CIGESM and MM

ManagementModule

CIGESM

Bay 1 - 4


Flow Name Flow Line VLAN Description Data Traffic A, B, C... Any VLAN other than X CIGESM and MM Traffic X Any VLAN NOT used for Data Traffic

Poor Design - Common NetworkManagement enabled for CIGESM uplinks


MM and CIGESM use the same VLAN on different uplinksData Traffic uses other VLANs

BladeCenter

MGMTInterfaceVLAN X


802.1QTrunk(s)Mode Access

ETH 0

ETH 1

To BladeServers


Enabled

Scenario 6 – NOT Recommended

• MM will proxy ARP for CIGESM

• Will result in CIGESM and MM ARP wars

• MM will proxy ARP for BladeServers

• Will result in BladeServers reporting duplicate IP address and not getting on to network

• Mixes all traffic

Broken Design CIGESM management over CIGESM uplinks, shared VLANs between CIGESM and MM and BladeServers

ManagementModule

CIGESM

Bay 1 - 4


Flow Name Flow Line VLAN Description All Traffic X All traffic on a single VLAN

Poor Design - Common NetworkManagement enabled for CIGESM uplinks


MM and CIGESM use the same VLAN on different uplinksMM, CIGESM and Data Traffic all on a single VLAN

BladeCenter

VLAN X

802.1Q Trunk(s) orMode AccessMode Access

ETH 0

ETH 1

To BladeServers

MGMTInterfaceVLAN X


Enabled

VLAN X

Scenario 7 – Eval environment only

• Allows for a single VLAN design but has caveats so not recommended for production environment

• If BladeServer ports trunked, MUST block CIGESM management VLAN

• Mixes all traffic types

Single VLAN design for EVAL CIGESM management over MM uplinks

ManagementModule

CIGESM

Bay 1 - 4


Flow Name Flow Line VLAN Description All Traffic X All traffic on a single VLAN CIGESM Mgmt traffic X via Y CIGESM Mgmt traffic will flow out MM on VLAN X

Possible CIGESM Evaluation Design - Common NetworkManagement disabled for CIGESM uplinks


MM, CIGESM and Data Traffic all on a single VLANNOT recommended in a production environment

BladeCenter

VLAN X VLAN X

802.1Q Trunk(s)or Mode Access

Mode Access

ETH 0

ETH 1

To BladeServers

MGMTInterfaceVLAN Y

VLAN X


Disabled


If Trunk - MUSTblock VLAN Y

CIGESM Management Tools

• CiscoWorks Integration• Full CiscoWorks support now available!• CiscoView provides IGESM front panel view, switch status, port state, and device configuration

• Cisco Cluster Management Suite (CMS) support

• Express Setup for device initialization• Single IP address to manage switch cluster

Front PanelView

PortStatus

Example of Cisco View

Example of Cluster Manager Suite

Example of Management Module Config

Example of Management Module Config

External Ports MUSTbe enabled from MM

Agenda


Default Operation of the CIGESM

• When CIGEMS is in factory default state:• All Bladeserver facing ports are in 802.1Q trunking mode

• If BladeServer is not trunking, will appear in VLAN 2 of CIGESM by default

• All Up-link ports will default to try to match other side• May become 802.1Q trunk or Access port

• If Trunk, native VLAN will be 2 (other side must match)• If Access, defaults to VLAN 1

• Links to Management Modules are in VLAN 1• Default IP addresses for CIGESM:

• Bay 1 192.168.70.127• Bay 2 192.168.70.128• Bay 3 192.168.70.129• Bay 4 192.168.70.130

Default Operation of the CIGESM

• Uplink ports disabled in default condition• Must enable prior to use

• Must be done initially via MM

• Management over uplinks disabled by default• Must be enabled via MM if in-band management is

desired

• Default port configurations are rarely optimal for production!• May be suitable for early deployment lab testing• Just plugging in to upstream switches without proper

config may cause upstream ports to go into disable state

Production Best Practices - Critical

• Rule Number 1: Pay attention to management path configuration• See management path recommendation slides

• Rule Number 2: Upgrade code on the CIGESM to latest release• Latest release 12.1(14)AY4

• Check readme for fixes• See summary section for link to latest code

Production Best Practices - Suggestions

• Disable uplinks prior to initial configuration• Prevents unexpected STP loops while configuring both sides• Can be done with shutdown or disconnecting cables• Once both sides configured, re-enable interfaces

• Use Management Module uplink to manage the CIGESM when possible• Default management configuration is this path

• Save CIGESM (to text file or other) prior to making changes• important in production environments to provide roll-back if

required

• Enable Root Guard on upstream switches connections to CIGESM’s• CIGESM is not a desired Spanning Tree Root switch

• Would almost certainly result in suboptimal flows if CGIESM became the root of the spanning tree


• Configure uplink facing ports exactly as required:• If using trunking, hard code port to trunk

• If using trunking, limit it to carry only required VLANs • Example: If only VLANs 2 and 3 required on port:

• If port is a Trunk port, Native VLAN MUST be allowed

• Leave BladeServer facing ports as Trunks (default), and if Access port is desired, accomplish by changing the Native VLAN

• Particularly important if using Serial Over LAN (SoL)• If no SoL, using “access” setting is fine

description blade1switchport access vlan 2switchport trunk native vlan 2switchport trunk allowed vlan 2-4094switchport mode trunkspanning-tree portfast trunkspanning-tree bpdufilter enable

switch(config-if)# switchport trunk allowed vlan 2,3

switch(config-if)# switchport mode trunk


• Leave BladeServer NICs speed and duplex set for auto-negotiate • BladeServer ports on CIGESM are hard coded to auto-

negotiate• Some newer BladeServer drivers also hard coded and can

not be changed

• Some confusion exists that the Native VLAN must be the same throughout a common L2 network.

• This assumption is not correct• On any given link, both sides must use the same Native VLAN• But different links in the same L2 network can use a different

Native VLAN


• Unless otherwise instructed by network administrators:• Leave VTP mode set for Transparent• Leave STP enabled

• DO NOT change STP settings on ports 15 and 16Incorrect setting could unexpectedly block uplinks

• Use cross-over cables between CIGESM and upstream switch• If link speed/duplex is set to auto – does not matter• If link speed/duplex is hard coded – may matter

• Makes use of the CIGESM Design Guide

• Make use of CIGESM Deployment Redpaper• If you read nothing else, read Section 7.1, 7.2, 7.4.1 and Appendix A

Agenda


Example Topology Configuration

17 18 19 20

CIGESM1

Data Center 6500-1

Mod 6 - 1 Mod 6 - 2 Mod 2 - 25 26 27 28

Data Center 6500-3

Mod 6 - 1Mod 6 - 2 25 26 27 28 - Mod 2

17 18 19 20

CIGESM2

1 2

BladeServer

1

1 2

BladeServer

2

MM1

MM2

Po1

Po2 Po3 Po2 Po3

1 2

BladeServer

3

ManagementNetwork

ManagementWorkstation

Po2 Po2Po1Po1

To Core routers

AggregationLayer

BladeCenter

Topology 2Links between Management Modules and Cisco Systems IGESMs not shown

TrunkedVLAN 10and 15

TrunkedVLAN 20and 25 Teamed/Access



AccessVLAN 10 Access

VLAN 20

Teamed/TrunkedVLAN 35, 40, 45

and 50

TrunkedVLAN 35,

40, 45and 50

1 2

BladeServer

4

AccessVLAN 30

AccessVLAN 30

Access Access

TrunkedVLAN 35,

40, 45and 50

AccessLayer

• Assumptions• Example uses Topology 2 from the

Redpaper

• Port numbers on upstream switches may vary

• IOS is being used in upstream switches (CatOS is possible but not covered here)

• Trunking, LACP aggregation and access connections to BladeServers and uplinks may vary in your customers environment

• For this topology to function correctly – VLANs carried on uplink ports to Aggregation Layer switches must also be carried between the Aggregation Layer switches


• More assumptions for this example• Assumes using out-of-band management via Management

Module• Assumes use of VLANs 1, 10, 15, 20, 25, 30, 35, 40, 45 and

50 (per Redpaper Topology 2 configuration)• Assumes using dual-uplinks on dual CIGESMs

• Only provided here as one possible solution• Your environment will almost certainly be

different!


• Basic steps for deployment• Investigate requirements and plan the design

• Decide Management path information• Path for management (In-band or Out-of-band)• Agree upon IP addressing and VLAN utilization for management

• Decide Data path information (up-link and BladeServer requirements)• Trunking, access, aggregation, VLANs, IP addressing

• Implement the design• Configure Management Module• Configure up-stream devices• Configure CIGESMs• Configure BladeServers

• Confirm expected operation• Should be more than just a ping test

• Designing the connectivity and networking for the BladeCenter• Should be a concerted effort between the network administrators and

the server administrators

• Management path discussions• IP address, mask and default gateway for the Management Module in

the BladeCenter• Need two IP addresses for the MM (1 external and 1 internal)

• If redundant MM, only need to configure the primary• Internal and external IP addresses must be on the same IP subnet• Default gateway must be on the same subnet as the IP address of the MM

• IP address, mask and default gateway for the CIGESM(s) in the BladeCenter

• Need to agree upon management path (via MM or via CIGESM uplinks)• If using in-band (via the uplinks of the CIGESM) need to agree on VLAN for

management

Investigate and Plan

Investigate and Plan

• Data path discussions• Come up with an agreed upon network diagram/design

• Topology 2 from Redpaper in our example• Needs to show uplink ports on CIGESM and their usage

• Agreed upon IP addressing and VLAN usage to carry data traffic between the BladeServers and the production network

• Port speed and duplex (default uplinks for CIGESM are auto-negotiate)

• Type of aggregation (if any) between CIGESM and upstream switches - LACP in our example

• CIGESM supports LACP, PAgP and static aggregation• Some older Cisco code may not support LACP

• Type of trunking or access ports - 802.1Q in our example• CIGESM only supports 802.1Q trunking - Some older Cisco devices may not support

802.1Q• If trunking, native VLAN to be used (default for CIGESM is VLAN 2)• If trunking, VLANs to be carried

Implementation - Configure the MM CIGESM

Upstream Switch Config

• Some possible areas of concern• Concern: Link aggregation desired but upstream switch does

not support LACP• Some older Cisco code does not support LACP

1) Switch to using PAgP for aggregation2) Upgrade code on upstream switch to support LACP

• Some other vendors do not support LACP or PAgP1) Switch to a vendor that supports desired protocol2) Use static aggregation

• Concern: VLAN trunking desired but upstream switch does not support 802.1Q• Some older Cisco switches support ISL only

1) Upgrade/replace upstream switch to support 802.1Q2) Put a switch between the CIGESM and the upstream switch that

supports 802.1Q and ISL3) Use Access ports rather than trunk ports

Configure The CIGESMs

• Must Perform on ALL CIGESMs in a BladeCenter• Recommend uplink cables be left disconnected (or

ports put in shutdown state) until CIGESMs fully configured• Add and Configure VLANs• Configure Management VLAN

• Defaults to VLAN 1• Defaults to IP address 192.168.70.X

• Configure Upstream connections• Configure aggregation• Configure trunking• Add VLANs to be carried on trunks

• Configure Connections to BladeServers• Configure access or trunking• Add VLAN(s) to be carried on ports

• Save Configuration• Save it or loose it!

Description and comments Actions via IOS CLI for the top CIGESM (1)

Step 1 - Configure desired VLANs for CIGESM1Create VLANs 10, 15, 30, 35, 40, 45 and 50 (only named VLAN 10

and 15 for this demonstration)

config t

vlan 10

name Web

vlan 15

name User

vlan 30,35,40,45,50

Step 2- Configure Link Aggregation toward the 6500s.

This example makes use of LACP to form the aggregation.

Ports g0/17 and g0/18 will be going to 6500-1


int range g0/17 - 18

description To-6500-1

channel-group 1 mode active




Step 3- Configure 802.1Q trunking toward 6500s and add allowed VLANs

Note that on the line allowing specific VLANS, there can not be any spaces between the numbers and the commas.

Also note that VLAN 2 is the Native VLAN on these ports by default.

int port-channel 1

description EtherChannel-To-6500-1

switchport trunk native vlan 2

switchport trunk allowed vlan 2,10,15,30,35,40,45,50

switchport mode trunk

int port-channel 2





Configure Top CIGESM (CIGESM 1)


Step 4- Configure 802.1Q trunking to BladeServer1 and add allowed VLANs

int g0/1

switchport trunk allowed vlan 2,10,15

Step 5- Configure access links to BladeServer2 and set access VLAN

int g0/2

switchport mode access

switchport access vlan 10


int g0/3




int g0/4

switchport trunk allowed vlan 2,35,40,45,50

end

Step 8 - Save Cisco Systems IGESM config to NVRAM

Failure to perform this step will result in all changes to the Cisco Systems IGESM being lost if the BladeCenter is powered off or the Cisco Systems IGESM is otherwise restarted.

copy running-config startup-config

Configure Top CIGESM (CIGESM 1)


Step 3.2.1 - Configure desired VLANs for CIGESM2

Create VLANs 20,25, 30, 35, 40, 45 and 50 (only named VLAN 20, and 25 for this demonstration)

config t

vlan 20

name Web

vlan 25

name User

vlan 30,35,40,45,50

Step 2- Configure Link Aggregation toward the 6500s.

This example makes use of LACP to form the aggregation.









Step 3- Configure 802.1Q trunking toward 6500s and add allowed VLANs

Note that on the line allowing specific VLANS, there can not be any spaces between the numbers and the commas.

Also note that VLAN 2 is the Native VLAN on these ports by default.

int port-channel 1





int port-channel 2





Configure Bottom CIGESM (CIGESM 2)



int g0/1

switchport trunk allowed vlan 2,20,25


int g0/2




int g0/3




int g0/4

switchport trunk allowed vlan 2,35,40,45,50

end

Step 8 - Save Cisco Systems IGESM config to NVRAM

Failure to perform this step will result in all changes to the Cisco Systems IGESM being lost if the BladeCenter is powered off or the Cisco Systems IGESM is otherwise restarted.

copy running-config startup-config

Configure Bottom CIGESM (CIGESM 2)

Configure the BladeServers

• Operating System dependent• Install Broadcom NIC drivers (or Intel if HS40)• For simple “Access” port configuration

• Configure physical interfaces with desired IP information

• For advanced port configuration (redundancy or trunking)• Install BASP (teaming) software (or Intel software for HS40)

• Required if using NIC Teaming• Teaming used for Active/Active or Active/Passive• Teaming used to configure 802.1Q

• With teaming software, create “teams”• Required for trunking (802.1Q) to CIGESM• Required for NIC redundancy (A/A or A/P)

• With teaming software, create VLAN logical interfaces as required• Only required if trunking to CIGESM

• Configure logical interfaces with desired IP information

• See Redpaper for more details on configuring BladeServers for this example

Configure BladeServer 1

Description and comments On BladeServer1 - BASP using VLANs on both Ethernet ports

Step 1 - Launch BASP softwareThis step assumes the desired software is already installed.

Click Start ->Programs -> Broadcom ->Broadcom Advanced Control Suite

Step 2 - Create and name two teams, each containing a single interface

Note that this process may seem as if you are configuring for SLB. This is not the case since we will only have a single NIC in each Team, and we are only building the Teams to assign VLANs (thus turning the interfaces into 802.1Q trunk interfaces)

Click Tools ->Create a Team on the tool bar.

Enter ToCIGESM1 in the name field and click Next

Note: leave the Team Type on the default value (Smart Load Balance and Fail Over)

Select the top NIC on the left side of the window, and click the top right pointing arrow, to add this NIC to the Load Balance Members.

Click Finish

Repeat Step 2 for the second NIC, naming the Team ToCIGESM2.

Step 3a - Create desired VLANs on Team CIGESM1.

Create and name VLANs 10 and 15 on the team going to CIGESM1

Click Tools ->Configure a Team on the tool bar.

Select ToCIGESM1 and click OK

Click Add VLAN button on right side of window.

In the VLAN ID field, enter 10

In the VLAN Name field, enter VLAN10-WEB

Note the names should be descriptive but can be anything you prefer. Also note that you want to leave the box labeled Untagged VLAN as un-checked

Click OK to create this VLAN

Repeat step 3a for the second VLAN on this team. Set the VLAN ID to 15 and name it VLAN15-USER



Step 3b- Create desired VLANs on Team CIGESM2.Create and name VLANs 20 and 25 on the team going to CIGESM2


Select ToCIGESM2 and click OK



In the VLAN Name field, enter VLAN20-APPS



Repeat step 4.1.3b for the second VLAN on this team. Set the VLAN ID to 25 and name it VLAN25-BACKUP

Step 4- Save the changes made to BASPThis step will create 4 new logical interfaces in Windows 2000:

ToCIGESM1/VLAN10-WEB

ToCIGESM1/VLAN15-USER

ToCIGESM2/VLAN20-APPS

ToCIGESM2/VLAN25-BACKUP

Note: Exiting the BASP program without clicking on Apply or Ok will result in loosing your configuration changes

Click Apply at the main BASP window

Click the Yes button when warned about a temporary interruption to the network connections

At this time the BASP software will create the new logical interfaces for use with W2K networking.



Step 5 - Configure desired IP address on each VLAN.

This step assumes that the user knows how to add IP addressing information.

Note that the default gateways used are part of the base HSRP config of the 6500s.

Also note that on production systems, one would normally configure one or more DNS servers. This was not included as part of this environment but should be included in most production networks.

For this step, attempting to apply IP addressing directly onto a physical interface is not supported.

From Windows, click Start ->Settings ->Network and Dial-up Connections

You should now see the original physical network interfaces along with the 4 newly created logical interfaces.

Select ToCIGESM1/VLAN10-WEB interface and configure the IP address as follows:

IP Address: 10.1.10.1

Mask: 255.255.255.0

Default Gateway: 10.1.10.254

Select ToCIGESM1/VLAN15-USER interface and configure the IP address as follows:


Mask: 255.255.255.0

Select ToCIGESM1/VLAN20-APPS interface and configure the IP address as follows:


Mask: 255.255.255.0

Select ToCIGESM1/VLAN25-BACKUP interface and configure the IP address as follows:


Mask: 255.255.255.0


Description and comments On BladeServer2 - No BASP software, using physical access links on both Ethernet ports

Step 1 - Configure IP addresses directly on the desired interfaces




This procedure will be no different then configuring a standalone server with two NICs.

Select the Local Area Connection interface and configure the IP address as follows:


Mask: 255.255.255.0


Configure Local Area Connection 2 interface and configure the IP address as follows:


Mask: 255.255.255.0


Description and comments On BladeServer3 - BASP using VLANs on both Ethernet ports for SLB

Step 1 - Launch BASP softwareThis step assumes the desired software is already installed.


Step 2 - Create the team using both NICs Click Tools ->Create a Team on the tool bar.

Enter ToBoth-VLAN30 in the name field and click Next

Note: leave the Team Type on the default value (Smart Load Balance and Fail Over)

Select the first NIC on the left side of the window, and click the top right pointing arrow, to add this NIC to the Load Balance Members.

Select the second NIC on the left side of the window, and click the top right pointing arrow, to add this NIC to the Load Balance Members.

Click Finish

Step 4.3.4- Save the changes made to BASPThis step will create a single new logical interfaces in

Windows 2000:

ToBoth-VLAN30

Note: Exiting the BASP program without clicking on Apply or Ok will result in loosing your configuration changes



At this time the BASP software will create the new logical interface for use with W2K networking


Description and comments On BladeServer3 - BASP using VLANs on both Ethernet ports for SLB

Step 4.3.5 - Configure desired IP address on each VLAN.


Note that the default gateway used is part of the base HSRP config of the 6500s.




You should now see the original physical network interfaces along with the new logical interface.

Select ToCIGESM1/ToBoth-VLAN30 interface and configure the IP address as follows:


Mask: 255.255.255.0



Description and comments On BladeServer4 - BASP using teaming and VLANs on both Ethernet ports

Step 1 - Launch BASP softwareThis step assumes the desired software is

already installed.


Step 2 - Create and name the team Click Tools ->Create a Team on the tool bar.

Enter ToBoth-Trunked in the name field and click Next

Select the first NIC on the left side of the window, and click the top right pointing arrow, to add this NIC to the Load Balance Members.

Select the second NIC on the left side of the window, and click the top right pointing arrow, to add this NIC to the Load Balance Members.

Click Finish.



Step 3 - Create desired VLANs on Team ToBoth-Trunked.

Create and name VLANs 35, 40, 45 and 50 to the team ToBoth-Trunked


Since there is only a single team, the BASP software takes you straight into the Team Configuration page



In the VLAN Name field, enter VLAN35



Repeat step 4.4.3a for the remaining VLANs on this team. Set the values as follows:

VLAN ID to 40, and name it VLAN40



Step 4- Save the changes made to BASPThis step will create 4 new logical interfaces in Windows

2000:

ToBoth-Trunked/VLAN35




Note: Exiting the BASP program without clicking on Apply or OK will result in loosing your configuration changes



At this time the BASP software will create the new logical interfaces for use with W2K networking.



Step 5 - Configure desired IP address on each VLAN.






You should now see the original physical network interfaces along with the 4 newly created logical interfaces.

Select ToBoth-Trunked/VLAN35 interface and configure the IP address as follows:


Mask: 255.255.255.0




Mask: 255.255.255.0



Mask: 255.255.255.0



Mask: 255.255.255.0

Confirm operation

• Check status of connections on CIGESM1 and 2Switch#show interface status

Port Name Status Vlan Duplex Speed Type

Gi0/1 blade1 connected trunk full 1000 1000Mbps SERDES

Gi0/2 blade2 connected 10 full 1000 1000Mbps SERDES

Gi0/3 blade3 connected 30 full 1000 1000Mbps SERDES

Gi0/4 blade4 connected trunk full 1000 1000Mbps SERDES

Gi0/5 blade5 notconnect 2 full 1000 1000Mbps SERDES

…



Gi0/15 mgmt1 connected trunk full 100 10/100/1000BaseTX

Gi0/16 mgmt2 notconnect 1 full 100 10/100/1000BaseTX

Gi0/17 To-6500-1 connected trunk auto auto 10/100/1000BaseTX




Po1 EtherChannel-To-65 connected trunk a-full a-1000

Po2 EtherChannel-To-65 connected trunk a-full a-1000

Confirm operation

• Check trunking and channeling on CIGESM1 and 2Switch#show interface trunk module 0

Port Mode Encapsulation Status Native vlan

Gi0/1 on 802.1q trunking 2

Gi0/2 off 802.1q not-trunking 2

Gi0/3 off 802.1q not-trunking 2

Gi0/4 on 802.1q trunking 2

.

.

.

Gi0/17 on 802.1q trunk-inbndl 2 (Po1)




Confirm operation

• Check Aggregation on CIGESM1 and 2Switch#show etherchannel summary

Flags: D - down P - in port-channel

I - stand-alone s - suspended

H - Hot-standby (LACP only)

R - Layer3 S - Layer2

u - unsuitable for bundling

U - in use f - failed to allocate aggregator

d - default port

Number of channel-groups in use: 2

Number of aggregators: 2

Group Port-channel Protocol Ports

------+-------------+-----------+-----------------------------------------------

1 Po1(SU) LACP Gi0/17(Pd) Gi0/18(P)

2 Po2(SU) LACP Gi0/19(Pd) Gi0/20(P)

Confirm operation

• Check Aggregation on CIGESM1 and 2Switch#show etherchannel 1 port-channel

Port-channels in the group:

------------

Port-channel: Po1 (Primary Aggregator)

------------

Age of the Port-channel = 00d:03h:36m:48s

Logical slot/port = 1/0 Number of ports = 2

HotStandBy port = null

Port state = Port-channel Ag-Inuse

Protocol = LACP

Ports in the Port-channel:

Index Load Port EC state No of bits

------+------+------+------------------+-----------

0 00 Gi0/17 Active 0

0 00 Gi0/18 Active 0

Confirm operation

• Check connectivity on CIGESM1 and 2

switch#show cdp neighbors

Capability Codes: R - Router, T - Trans Bridge, B - Source Route Bridge

S - Switch, H - Host, I - IGMP, r - Repeater, P - Phone

Device ID Local Intrfce Holdtme Capability Platform Port ID

DC6500-1 Gig 0/18 127 R S I WS-C6506 Gig 2/26

DC6500-1 Gig 0/17 127 R S I WS-C6506 Gig 2/25

DC6500-3 Gig 0/20 177 R S I WS-C6506 Gig 2/26

DC6500-3 Gig 0/19 177 R S I WS-C6506 Gig 2/25

CIGESM2 Gig 0/15 159 S I OS-CIGESM-Gig 0/15

Confirm operation

• Check Spanning Tree Blocked Portsswitch#show spanning-tree blockedports

Name Blocked Interfaces List

-------------------- ------------------------------------

VLAN0002 Po2

VLAN0020 Po2

VLAN0025 Po2

VLAN0030 Po2

VLAN0035 Po2

VLAN0040 Po2

VLAN0045 Po2

VLAN0050 Po2

Number of blocked ports (segments) in the system : 8

Based on the topology in this presentation, all VLANs for Port Channel 2 will be Spanning tree

blocked, and only come up if Port Channel 1 goes down

switch#show spanning-tree root

Root Hello Max Fwd

Vlan Root ID Cost Time Age Dly Root Port

---------------- -------------------- ------ ----- --- --- ----------------

VLAN0001 32769 0011.211b.5c00 0 2 20 15

VLAN0002 8194 000e.3848.cf00 3 2 20 15 Po1

VLAN0010 8202 000e.3848.cf00 3 2 20 15 Po1

VLAN0015 8207 000e.3848.cf00 3 2 20 15 Po1

VLAN0030 8222 000e.3848.cf00 3 2 20 15 Po1

VLAN0035 8227 000e.3848.cf00 3 2 20 15 Po1

VLAN0040 8232 000e.3848.cf00 3 2 20 15 Po1

VLAN0045 8237 000e.3848.cf00 3 2 20 15 Po1

VLAN0050 8242 000e.3848.cf00 3 2 20 15 Po1

Confirm operation

Based on the topology in this presentation, all root ports should be on Port Channel 1 (except VLAN 1,

as it is not carried on the uplinks), and only switch to Port Channel 2 on the failure of Port Channel 1

• Check Spanning Tree Root Ports

Confirm operation

switch#show spanning-tree VLAN 10

VLAN0010

Spanning tree enabled protocol rstp

Root ID Priority 8202

Address 000e.3848.cf00

Cost 3

Port 65 (Port-channel1)

Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Bridge ID Priority 32778 (priority 32768 sys-id-ext 10)

Address 0011.211b.5c00

Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Aging Time 300

Interface Role Sts Cost Prio.Nbr Type

---------------- ---- --- --------- -------- --------------------------------

Gi0/1 Desg FWD 4 128.1 Edge P2p

Po1 Root FWD 3 128.65 P2p

Po2 Altn BLK 12 128.66 P2p

• Check Spanning Tree for individual VLAN

Agenda


• Originally used direct serial connection to server

• Often now used in a terminal server environment• Terminal Server converts serial data to LAN• Provides:

• Remotely manage the server over the LAN

• Remotely load scripts for configuring the server

• Multiple servers can feed into a single terminal server to provide a single point of access to multiple servers

• LAN attached Terminal Server access is a form of Serial Over LAN

Why Server Serial Connections?

Stand-alone Server Environment

A B C D E F G HSELECTED

ON-LINE

IBM Server IBM Server IBM Server

LAN Connection

Serial Ports

Terminal Server

Serial to Ethernet conversion takes place in Terminal Server

Rack-optimized SOL

A B C D E F G HSELECTED

ON-LINE

LAN ConnectionTerminal Server

Remote IT Specialist

Network

Rack of 1U Servers

Serial to Ethernet conversion takes place in Terminal Server

Serial connections in a BladeCenter

• BladeServer has no externally accessibly serial ports• Need some other way to access BladeServer via serial• Provided by sending internal serial port data on a BladeServer

out of the BladeServer over its first Ethernet connection• In most cases, SoL only supported via switch in Bay 1

• Jumper on some blade models let you switch to second Ethernet connection

• HS series supports SoL as an optional way to manage the blade• KVM used for initial OS install

• JS series requires SoL for initial install• JS has no KVM type interface, must use SoL for initial OS

install• Can also use SoL to manage after install

Solution for Serial Over LAN in a Blade

BIOS Serial traffic

H8

Ethernet chip

VLAN 4094

The H8 forms the SOL packetThe Ethernet chip adds the packet to its data streamVLAN 4094 is just an example

Serial to Ethernet conversion takes place on BladeServer

BladeCenter SOL

SWITCH MODULE(1 and 2)

(CPU)

Internal Mgmt Network Connections:2 100Mbps ports (1 per mgmt module)

MGMT MODULE(1 of 2)

Ethernet Switch

SERVER BLADE(1 of 14)

Blade Subsystems

Dual Ethernet "Adapter"

Optional Daughter Card

MIDPLANE

to SW 2

to SW 3

to SW 4

to SW 1

PCI

4 14Ethernet Switch

Hardware

SoL – Hardware and OS Support

• Not all Operating Systems support SoL• W2K Server not supported• Most Linux and W2K3 supported on HS series• AIX and Linux on JS20 is supported

• Requires OS configuration steps on some blades• See IBM SoL support doc for details

http://www-1.ibm.com/support/docview.wss?uid=psg1MIGR-54666

• Some BladeServer models may not be supported• May require firmware upgrades on server blades• JS20 requires Broadcom firmware 2.30 or higher

Rules for SoL VLAN on the CIGESM

• It must be a VLAN ID from 1 to 4094 (inclusive)

• The SoL VLAN ID can not be 1002 through 1005 (reserved by the CIGESM)

• To use VLANs above 1005, must be in VTP Transparent mode (default for CIGESM)

• VLANs 1 and 2 are both default VLANs on the CIGESM, and unless otherwise changed, should not be used for the SoL VLAN

• It can not be a VLAN that is being used to carry user data between the BladeServers and the CIGESM uplink ports

• It can not be the management VLAN used between the CIGESM and the Management Modules• The default VLAN for the CIGESM to Management Module traffic is

VLAN 1, but can be changed by the user

Rules for SoL VLAN on the CIGESM

• It must match the VLAN configured for SoL use on the Management Module

• SoL VLAN must be carried to the BladeServer• If BladeServer will be on a single VLAN

• Do not set port for Access if you need SoL• Set port to trunk and set Native VLAN to desired

access VLAN• Set SoL VLAN to be allowed on trunk to

BladeServer

• Why don’t I have to configure this on other vendors solutions for the BladeServer?• Other vendors use VLAN 4095 – See IEEE 802.1Q specs

SoL CIGESM Config Example – VLAN 4094

Description and comments Actions via IOS CLI for the CIGESM

Step 1- Create the desired VLAN to be used for SoLIn this example we will be using VLAN 4094. Switch needs to be in VTP Transparent mode to use this specific VLANPerform the commands in this table starting from a telnet session to the CIGESM, with the

CIGESM in the Enable mode.The SoL VLAN will only be specific to this CIGESM, and can be used by other switches in the

network, as the SoL VLAN on this switch will be isolated from other switches by the commands in step 2.

Note that naming of the VLAN in this step is optional, and is only for the convenience of future users reviewing the switch configuration.

config tvlan 4094name SoL

Step 2- Remove the SoL VLAN from the external uplinks.Note this must be the same VLAN defined in step 1.This will isolate the SoL VLAN to this BladeCenter only.

int range g0/17 -20switchport trunk allowed vlan remove 4094

Step 3 - Add the SoL VLAN on the links going from the CIGESM to the Management Modules and the BladeServers.

Note this must be the same VLAN defined in step 1.This will allow the SoL traffic on the link between the CIGESM and the Management Module

as well as to the BladeServers.

Ports to BladeServers MUST be in trunk mode (not access). If you want the BladeServer in question to be in Access mode, you can simulate this by making it a trunk link and setting the Native VLAN to the desired Access VLAN.

int range g0/1 -16switchport trunk allowed vlan add 4094

Step 4- Exit config mode and save config to NVRAMThe end commend exists out of config mode.Failure to perform this step will result in all changes to the CIGESM being lost if the

BladeCenter is powered off or the CIGESM is otherwise restarted.

endwrite

SoL Config - Management Module

Make sure SoL shows Enabled

SoL Config - Management Module

SoL must also be enabled here

Select SoL VLAN

For CIGESM – CAN NOT be 4095

Must click on Save after making changes

Also strongly recommended to reboot MM after saving the change

SoL Config

• After making changes to MM, save and restart MM• Failure to restart MM may result in SoL not working

• For some BladeServers, must configure OS for SoL to work• Example: HS20 running W2K3 or Linux• See SoL configuration documentation

• For JS20, no BladeServer side config required• Do need to have latest Broadcom firmware (2.30 or higher)• Should have latest AIX drivers

SoL Operation

• Telnet to IP address of MM and log in• User ID and password same as http login

• Start SoL session to desired blade from MM prompt• console –T system:blade[X]

• Where X = BladeServer slot 1 through 14

• May have to press the enter key several times to get a prompt from the blade

• Log in using desired OS’s user ID and password

• Exit SoL session when done• Log out of OS if so desired

• Failure to log out of OS can leave system open to next SoL user

• Press the Esc key and then the open parenthesis• Esc (

• Should now be returned to MM prompt

SoL Hints and Tips – HS series

• If SoL will not come Ready• Verify MM VLAN set as desired and BladeServers are enabled for SoL• Make sure you restart the MM after making and saving SoL changes• Make sure CIGESM sees ports to HS20 as connected

• On CIGESM - show int status• If ports to BladeServers show not connected when they should be, may

need to reboot BladeServers that are showing not connected• Verify CIGESM config

• Is SoL VLAN created and matches the VLAN set on MM?• show vlan

• Is SoL VLAN being carried on BladeServer ports?• show run int g0/X (X = 1 through 14, slot to be checked)

• Is SoL VLAN being carried on port to MM?• Show run int g0/15

• If SoL status shows Ready or Active, but get no data on SoL session screen• Problem is most likely on BladeServer• Verify SoL is configured on the Operating System

SoL Hints and Tips – JS series

• If SoL will not come Ready• Make sure you restart the MM after making SoL changes• Make sure CIGESM sees ports to JS20 as connected

• On CIGESM - show int status• Older AIX driver will fail to bring up links after a CIGESM

restart, must POR the JS20 to make links show connected

• New version of AIX driver resolves this issue

• If SoL intermittently toggles between Ready and Not Ready• JS20 must have Broadcom firmware 2.30 or higher• To see firmware in AIX: lscfg –vpl ent0

• ROM Level………03210230

Sol – Ready and Active

•SoL session established to blade

•Ready for SoL session

SoL “Not ready”

Blade may be powered off

CIGESM may not be configured

CIGESM may be mis-configured

Port to blade may be down

Possible problem with blade firmware

Agenda


Hints and Tips

• Review Appendix A in Deployment Redpaper• A number of topics discussed, including:

• NIC numbering on BladeServers• Different for different OS’s• Can change depending on driver install sequence

• Duplicate IP addresses• Do not place BladeServers in same VLAN as CIGESM

management VLAN interface• Change IP address via MM

• Native VLAN mismatch messages• Place all CIGESM’s in a given BladeCenter in the same

management VLAN• Or – Turn off CDP on CIGESM ports g0/15-16

Hints and Tips

• XP Hyperterm may not connect to console port• Can see output but can not enter input• Issue not seen with W2K or other OS’s• Workarounds

• Set flow control to something other than hardware (sometimes works)• Upgrade to full version of Hyperterm (free) (always works)• Use a different terminal emulator than Hyperterm (always works)

• Upgrade CIGESM to 12.1(14)AY4 or newer• Many fixes

• See release notes in code download • A number of feature enhancements

• Support for Trunk Failover• More Etherchannel load balance options

Hints and Tips

• Unable to enable external ports from switchswitch(config)#int g0/17

switch(config-if)#no shut

% Shutdown not allowed on this interface.• Must enable External ports in advanced setup in MM

• Default is Disabled

Hints and Tips

• Important switch information • show platform summary

Switch#sh platform summary Platform Summary:

Switch is in Bay 1Switch Slot: 1Current IP Addr: 172.26.147.208, 255.255.254.0, gw: 172.26.146.1 Default IP Addr: 10.10.10.91, 255.255.255.0, gw: 0.0.0.0 IP Fields read from VPD: 172.26.147.208, 255.255.254.0, gw: 172.26.146.1Static IP Fields in VPD: 172.26.147.208 255.255.254.0 172.26.146.1 IP Acquisition Method used: static

Active Mgmt Module in Mgmt Slot: 1Native Vlan for Mgmt Module Ethernet ports: 1External Mgmt over Extern ports Disabled Advanced MM Settings

Hints and Tips

• How do I let the CIGESM set and maintain its own IP addressing information?• Default is that the MM controls CIGESM IP addressing• If you change the IP address directly on the CIGESM, it will

revert to MM derived settings on next CIGESM reboot (or sooner)

• To let CIGESM control its IP addressing:• On MM, Advanced settings, set “Preserve new IP

configuration during all resets” to “Disabled” and click on “Save”.

• Reload CIGESM

• Change CIGESM to desired IP addressing

• CIGESM is now in control

• When using this feature – management should be configured for use over the CIGESM uplinks

Hints and Tips

• Using Trunk Failover Feature• Available in AY4 or above code• Used in conjunction with NIC teaming configurations that

create a logical NIC from the combination of 2 or more physical NIC’s

• Should configure trunk failover on all CIGESMs in the BladeCenter connected to the teamed NIC’s

• Need to have external L2 connectivity between CIGESM’s for proper fail-over

• Global command: • link state track X

• Interface command: • link state group X {upstream | downstream}

• Puts downstream ports in err-disable state (down) when all upstream links in group are down

Hints and Tips

• Steps:1) Configure global command2) Configure upstream port(s) or Etherchannel3) Configure downstream port(s)

Configuring downstream before upstream will result in downstream ports going down until upstream is configured

• Trunk Failover Configuration ExampleSwitch# configure terminalSwitch(config)# link state track 1Switch(config)# interface range gi0/17 -18Switch(config-if)# link state group 1 upstreamSwitch(config-if)# interface range gi0/1 -5Switch(config-if)# link state group 1 downstreamSwitch(config-if)# end

Switch# show link state group [ X ] [ detail ]

Hints and Tips

• Switch#show link state group detail Link State Group: 1 Status: Enabled, Down

Upstream Interfaces : Gi0/17(Dwn) Gi0/18(Dwn)

Downstream Interfaces : Gi0/1(Dis) Gi0/2(Dis) Gi0/3(Dis) Gi0/4(Dis) Gi0/5(Dis)

Link State Group: 2 Status: Disabled, Down

Upstream Interfaces :

Downstream Interfaces :

(Up):Interface up (Dwn):Interface Down (Dis):Interface disabled

Agenda


Relevant URLs

• IBM/Cisco Design Guide for the CIGESMhttp://www-1.ibm.com/services/alliances/cisco/files/cisco-igesm-design-guide.pdf

• IBM CIGESM Deployment Redpaperhttp://www.redbooks.ibm.com/abstracts/redp3869.html

• IBM Serial Over LAN Guidehttp://www-1.ibm.com/support/docview.wss?uid=psg1MIGR-54666

• CIGESM IOS Code Download (AY4)http://www-307.ibm.com/pc/support/site.wss/document.do?lndocid=MIGR-58132

MIBS can also be found at the link for AY4 code

• Link to VLAN best practiceshttp://www.cisco.com/en/US/products/hw/switches/ps708/products_white_paper09186a008013159f.shtml

Relevant URLs

• CiscoView Package DownloadSupport provided at FCS

http://www.cisco.com/cgi-bin/tablebuild.pl/cview50

Download cv50.v1-0.readme and cigesm.cv50.v1-0.zip

• CiscoWorks IDU's to support the CIGESM Minimum code on CIGESM to support this is 12.1(14)AY1

http://www.cisco.com/kobayashi/sw-center/cw2000/lan-planner.shtml

Under "Application-Level Updates" for each module

Extra Material

• Feature Configuration• Software Upgrades

VLAN Configuration

• Normal range VLAN• VLAN ID 1 to 1005• Can only add/remove/change when in VTP Server or Transparent mode• When in VTP Client mode, must add VLANs on other switch (that is in VTP Server mode)• Saved in VLAN database file vlan.dat• VLAN 1, and 1002 to 1005 are specially created and cannot be removed

• Extended range VLAN• VLAN ID 1006 to 4094• Must be in VTP Transparent mode when create, not recognized by VTP• Not saved in VLAN database file vlan.dat• Must use config-vlan mode (new method)• Cannot be included in the pruning eligible range• Cannot be configured by VMPS

• CIGESM VLANs supported• VLAN ID 1 to 4094 (4K)• 250 Active VLANs

• VLAN Introduction• http://www.cisco.com/pcgi-bin/Support/browse/psp_view.pl?p=Technologies:VLANs_and_VTP&viewall=true

VLAN Configuration

• Create a new VLAN• VLAN database mode (old – only supports VLANs 1 - 1005)

• config-vlan mode (new – for all VLANs, 1 - 4094)

• Statically assign a port to the VLAN (common on CIGESM)

• Dynamically assign a port to the VLAN (not common on CIGESM)

switch# vlan database

switch(config)# vlan VLAN_#

switch(config-if)# switchport mode access

switch(config-if)# switchport access vlan VLAN_#

switch(config-if)# switchport mode access

switch(config-if)# switchport access vlan dynamic

Configure Trunk

• External ports Default to dynamic desirable DTP mode• Enables DTP (Cisco’s dynamic trunking protocol)

• Actively negotiates trunking state (access or trunk) and encapsulation (802.1Q)

• To force specific mode (Trunk or Access) set manually

• For non-Cisco switches, must set mode manually

• Other DTP modes:• Access: a user port and cannot be a trunk

• Trunk: a trunk and negotiates trunking with other port

• Dynamic auto: passively waits to be contacted.

Configure Trunk

• Set trunk mode manually

• Set trunk native VLAN

• Configure allowed VLANs

switch(config-if)# switchport mode trunk

switch(config-if)# switchport trunk native vlan VLAN_#

switch(config-if)# switchport trunk allowed vlan add VLAN_#

Troubleshooting Trunk

• Check interface type and link status

• Check the spanning tree state of the port

• Check native VLAN on both sides to match

switch# show interface state

switch# show spanning-tree interface gig0/17

switch# show interface gig0/17 switchport

Configure VTP

• VLAN Trunking Protocol (VTP) • Runs over trunk links• Synchronize the VLAN databases of all switches in the VTP domain• A VTP domain is an administrative group• All switches within the group must have the same VTP domain name

• VTP switch roles• Server: Can create/delete/rename VLANs (Cisco default on non-CIGESM switches)• Client: Cannot make VLAN changes• Transparent: Can create/delete/rename VLANs; local database only; forward VTP advertisements (CIGESM default)

• VTP version• Version 1: • Version 2: Support Token Ring and others (CIGESM default)• Mixed V1 and V2: Not compatible (either all running version 1 or all running version 2)

• VTP Introduction• http://www.cisco.com/en/US/tech/tk389/tk689/technologies_tech_note09186a0080094c52.shtml

Configure VTP

• Set up VTP mode (default is transparent on CIGESM)

• Set up VTP domain name (default is null on CIGESM)

• Set up VTP version (version 2 is default on CIGESM)

switch(config)# vtp mode { server | client | transparent }

switch(config)# vtp domain NAME

switch(config)# vtp version { 1 | 2 }

Configure Etherchannel

• Combines several physical links between switches into one logical connection to aggregate bandwidth

• Treats Etherchannel as one path by STP

• Load balancing on source MAC address or destination MAC address

• Achieves redundancy

• LACP, PAgP and Static Aggregation supported on CIGESM

• Etherchannel Introduction• http://www.cisco.com/pcgi-bin/Support/browse/psp_view.pl?p=Technologies:Etherchannel&viewall=true


• All ports must be the same speed and duplex• All ports in the bundle should be enabled• None of them can be a SPAN destination port• Two channel protocols• IEEE 802.3ad Link Aggregation Control Protocol (LACP)

• active mode: Port actively negotiates channeling

• passive mode: Wait to be contacted

• On mode: Forms channel without negotiating

• Cisco proprietary Port Aggregation Protocol (PAgP)• Desirable mode: Actively negotiates

• Auto mode: Responds, do not initiate

• On mode: Forms channel without negotiating


• Bundle several interfaces

• Configure channel protocol

• Configure load-balancing

• Verify Etherchannel

switch(config)# interface { interface | range interface – interface }

switch(config-if-range)# channel-group NUMBER mode { active | passive | auto | on | desirable }

switch(config-if-range)# channel-protocol { lacp | pagp }

switch# show etherchannel summary

switch# show etherchannel NUMBER port-channel

switch(config# port-channel load-balance { dst-mac | src-mac }

Configure SPAN

• SPAN – Switched Port Analyzer (Port mirroring)• Can specify traffic directions

• Receive (Rx) – ingress of source port

• Transmit (Tx) – egress of source port

• Both

• Source port (monitored port) can be any type, in any VLAN

• Destination port (monitoring port) can be any physical port, but not port-channel, not SVI

Configure SPAN

• Create monitor session with source port

• Specify destination port

• Verify SPAN

switch(config)# monitor session NUMBER source interface interface { both | rx | tx }

switch(config)# monitor session NUMBER destination interface interface

switch# show monitor [ session NUMBER ]

Configure Remote SPAN (RSPAN)

g2/25 Data Center 6500-1g2/26 g2/2

1 2

g0/19 - Reflector-port

g0/1

CIGESM1

1

BladeServer

1BladeCenter

Sniffer/IDS/RMON Probe

RSPAN source: Port g0/1 on CIGESM1RSPAN Reflector-port: Port g0/19 on CIGESM1RSPAN VLAN: VLAN 500RSPAN destination: Port g2/2 on 6500-1

VLAN500

• RSPAN – Permits remote monitoring of CIGESM traffic

Configuring Remote SPAN (RSPAN)

• Rules for RSPAN• Understand RSPAN before implementing it

• See deploying CIGESM Redpaper, Topology 3 example

• Reflector port must be unused• Essentially a loop-back port for RSPAN• Do not attach cable or install BladeServer on port used as reflector

port

• Reflector port can be unused BladeServer port (g0/1 – g0/14) or an unused external port (g0/17 – g0/20)

• Must also configure upstream switch to support RSPAN• Must be using 12.1(14)AY1 or above

• Issue with earlier release of CIGESM IOS code and RSPAN


• On CIGESM

Create VLAN to support RSPAN and configure it to support RSPAN:

switch(config)# vlan VLAN_ID

switch(config-vlan)# remote-span

Establish source and destination for the RSPAN:

switch(config)# monitor session NUMBER source interface interface { both | rx | tx }

switch(config)# monitor session NUMBER destination remote vlan VLAN_ID reflector-port interface


• On upstream switch

• Verify RSPANswitch# show monitor [ session NUMBER ]

Create VLAN to support RSPAN and configure it to support RSPAN:

switch(config)# vlan VLAN_ID

switch(config-vlan)# remote-span

Establish source and destination for the RSPAN:

switch(config)# monitor session NUMBER source remote vlan VLAN_ID

switch(config)# monitor session NUMBER destination interface interface

Configure ACL

• ACL – Access Control List• An ACL contains an ordered list of Access

Control Entries (ACE)• Each ACE specifies permit or deny

conditions the packets must satisfy to match the ACE

• CIGESM supports the following types of ACL• IP ACL filters IP, TCP, and UDP traffic• Ethernet or MAC ACL filters Layer 2 traffic• MAC extended access lists use source and destination MAC

addresses and optional protocol type information for matching operations

• Standard IP access lists use source addresses for matching operations

• Extended IP access lists use source and destination addresses and optional protocol type information for matching operations

IGESM Access Control ListsS2

What It Does:

Allows or denies access based on the source or destination address.

Restricts users to designated areas of the network, blocking unauthorized access to all other applications and information.

Benefits:

Prevents unauthorized access to servers and applications.

Allows designated users to access specified servers.

Port-based ACL:Provides granular control at Layer 3 or 4 for limited access by the Layer 2 access port of the device.

ACL - Protecting against Worms

How It Works:

The ACL provide a mechanism to protect servers, users and applications against worms by determining what traffic streams or users can access what ports.

Network

Using ACLs, the virus or worm is not able to replicate from its hosts.

Port 1434

Configure ACL• Numbered IP standard ACL

• Standard ACLs (1 through 99) act upon IP address only

• Numbered IP extended ACL• Extended ACLs (100 through 199) act upon IP address and TCP/UDP port

• Named MAC extended ACL

switch (config)# access-list 2 deny host 171.69.198.102

switch (config)# access-list 2 permit any

switch# show access-lists

switch(config)# access-list 102 deny tcp 171.69.198.0 0.0.0.255 172.20.52.0 0.0.0.255 eq telnet

switch(config)# access-list 102 permit tcp any any

switch(config)# mac access-list extended my-mac-acl

switch(config-ext-macl)# deny any any decnet-iv

switch(config-ext-macl)# permit any any

switch# show access-list

Configure ACL

• Apply IP ACL at physical interface

• Apply MAC ACL at physical interface

switch(config-if)# ip access-group 2 in

switch# show ip interface interface

switch(config-if)# mac access-group my-mac-acl in

switch# show mac access-group interface interface

Miscellaneous

• Add new user account

• Disable CMS

• Example change management VLAN via CLI

switch(config)# username NAME password 0 PASSWORD

switch(config)# no ip http server

switch(config)# vlan 100Switch(config-vlan)# name new_mgmt_vlanSwitch(config-vlan)# exitSwitch(config)# int vlan 100Switch(config-if)# no shutdown Port Gi0/15 allowed vlan list updated to include vlan 100Port Gi0/16 allowed vlan list updated to include vlan 100

Extra Material

• Feature Configuration• Software Upgrades

Software Upgrade

• Only IOS needs to be upgraded – But can upgrade both IOS and CMS

• Regular IOS image vs. crypto image (no procedural difference)•Regular image: cigesm-i6q4l2-mz.121-0.0.42.AY.bin•Crypto image: cigesm-i6k2l2q4-mz.121-0.0.42.AY.bin

• Binary file vs. TAR file (copy vs. archive)•TAR file includes both IOS binary image and CMS files•cigesm-i6q4l2-tar.121-0.0.42.AY.tar•cigesm-i6k2l2q4-tar.121-0.0.42.AY.tar

• Where to get IOS images for CIGESM .....?

• Upgrade through TFTP

Software Upgrade

• Examples in this section:

• Upgrade Via CLI• Upgrade IOS binary image only (copy command)• Upgrade IOS and CMS with TAR file (archive download-sw command)

• Upgrade Via GUI• Upgrade IOS and CMS with TAR file (CMS Software upgrade menu

option)

• IOS Recovery Via CLI• Recovering from a failed upgrade, via the Console port and X-Modem

Software Upgrade – IOS only (.bin)

• Assumes you have a TFTP server installed and running• Free TFTP server can be downloaded from:

• http://support.solarwinds.net/updates/New-customerFree.cfm

1. Download image (.bin) file to TFTP server

2. Ping from the CGIESM to the TFTP server

3. Make sure you have enough space in FLASH• If insufficient space, TFTP of .bin file will fail

switch# ping ip_address_of_tftp_server

switch# dir flash:Directory of flash:/ 3 -rwx 736 Mar 01 1993 00:00:27 vlan.dat 4 -rwx 16 Sep 10 2003 10:00:27 env_vars 5 -rwx 6631 Mar 01 1993 00:10:36 config.text 10 drwx 192 Mar 04 1993 23:32:49 cigesm-i6q4l2-mz.121-0.0.42.AY

7612416 bytes total (1999872 bytes free)

Software Upgrade – IOS only

4. Copy image from TFTP to switch’s FLASH

switch#copy tftp flash:

Address or name of remote host []? 192.168.10.1

Source filename []? cigesm-i6q4l2-mz.121-0.0.42.AY.bin

Destination filename [cigesm-i6q4l2-mz.121-0.0.42.AY.bin]?

Accessing tftp://192.168.10.1/cigesm-i6q4l2-mz.121-0.0.42.AY.bin...

Software Upgrade – IOS only

5. Change switch boot path variable

6. Save the change and reload switch

7. Verify the change

switch(config)# boot system flash:new_image_name

switch# show boot

switch# copy running-config startup-config

switch# reload

switch# show version

Software Upgrade – IOS and CMS (.tar)


• http://support.solarwinds.net/updates/New-customerFree.cfm

1. Download TAR file to TFTP server

2. Ping from the CGIESM to the TFTP server

3. Make sure you have enough space in FLASH• If insufficient space, will overwrite automatically

switch# ping ip_address_of_tftp_server

switch# dir flash:Directory of flash:/ 3 -rwx 736 Mar 01 1993 00:00:27 vlan.dat 4 -rwx 16 Sep 10 2003 10:00:27 env_vars 5 -rwx 6631 Mar 01 1993 00:10:36 config.text 10 drwx 192 Mar 04 1993 23:32:49 cigesm-i6q4l2-mz.121-0.0.42.AY

7612416 bytes total (1999872 bytes free)

Software Upgrade – IOS and CMS

4. Archive download TAR file from TFTP to switch’s FLASH• TAR file (ends in .tar) contains both IOS (.bin file) and all CMS code

• BIN file (ends in .bin) contains IOS code only

switch# archive download-sw ? /force-reload Unconditionally reload system after successful sw upgrade /imageonly Load only the IOS image /leave-old-sw Leave old sw installed after successful sw upgrade /no-set-boot Don't set BOOT -- leave existing boot config alone /overwrite OK to overwrite an existing image /reload Reload system (if no unsaved config changes) after successful sw upgrade /safe Always load before deleting old version flash: Image file ftp: Image file rcp: Image file tftp: Image file


4. Archive download-sw

switch# arch down tftp://192.168.10.1/systemtest/cigesm-i6q4l2-tar.121-0.0.41.AY.tarexamining image...Loading stiletto/cigesm-i6q4l2-tar.121-0.0.41.AY.tar from 192.168.10.1 (via Vlan1): !extracting info (282 bytes)!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!![OK - 5357568 bytes]

Image info: Version Suffix: i6q4l2-121-0.0.41.AY Image Name: cigesm-i6q4l2-mz.121-0.0.41.AY.bin Version Directory: cigesm-i6q4l2-mz.121-0.0.41.AY Ios Image Size: 3042304 Total Image Size: 5355008 Image Feature: LAYER_2|MIN_DRAM_MEG=32 Image Family: CIGESM Image Minimum DRAM required: 32


4. Archive download-sw. The whole process can be interrupted by Ctrl-shift-6.

switch# arch down tftp://192.168.10.1/systemtest/cigesm-i6q4l2-tar.121-0.0.41.AY.tarexamining image........Not enough free space to download w/o first deleting existing and/or current version(s)...Deleting flash:/cigesm-i6k2l2q4-mz.121-0.0.42.AY...done.Extracting files...Loading systemtest/cigesm-i6q4l2-tar.121-0.0.41.AY.tar from 192.168.10.1 (via Vlan1): !extracting info (282 bytes)cigesm-i6q4l2-mz.121-0.0.41.AY/ (directory)cigesm-i6q4l2-mz.121-0.0.41.AY/html/ (directory)extracting cigesm-i6q4l2-mz.121-0.0.41.AY/html/CMS.sgz (1357883 bytes)!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!! [Interrupted]

Premature end of tar fileERROR: Problem extracting files from archive.Switch#


5. Check the boot path variable

6. Reload the switch

7. Verify the upgrade

switch# show bootBOOT path-list: flash:/cigesm-i6k2l2q4-mz.121-0.0.42.AY/cigesm-i6k2l2q4-mz.121-0.0.42.AY.binConfig file: flash:/config.textPrivate Config file: flash:/private-config.textEnable Break: noManual Boot: noHELPER path-list: NVRAM/Config file buffer size: 32768

Software Upgrade – Web Interface


http://support.solarwinds.net/updates/New-customerFree.cfm

• Assumes image is already on the TFTP sever in the correct location for access by the CIGESM

• Assumes CIGESM has connectivity to the TFTP server• Can ping from CIGESM to TFTP IP address

• Can use GUI to load either .bin or .tar file to CIGESM• This example shows .tar being installed

• Assumes you are connected to CMS and are logged in with the necessary privileges



Software Upgrade – Recovery

• What to do if upgrade fails and IOS is not loaded• No way to use TFTP for new image as IOS not running

• CIGESM will stop at “switch:” prompt on next boot• Must be attached to console port to see switch: prompt• Ethernet ports will be unusable

• CIGESM left in unusable state

• Must use Console port and X-Modem to recover• Connect laptop/PC to console port

• Required cable ships with most Cisco routers and switches• Pinouts for DB9-RJ45: http://www.cisco.com/warp/public/473/9.html

• Start terminal emulation program (i.e. hyperterm)• Set serial port to 9600, N, 8, 1• Verify CIGESM is at “switch:” prompt and start the flash file system:

• Check for sufficient space on flash (switch: dir flash:) and delete corrupted “.bin” file if necessary (switch: del flash:/dir/xxx.bin)

• Proceed to X-Modem recovery procedure

switch: flash_init


• To perform X-Modem recovery• Once connected via console port and flash_init complete, start the process• Default speed of 9600 can take hours to complete• Can increase speed for faster upload (BAUD must be in caps):

• Remember to change back to 9600 default when complete• To begin X-Modem transfer (of .bin file, not .tar file – bin can be extracted from

tar using winzip or other such product)

• When the XMODEM request appears, use the appropriate command on the terminal-emulation software to start the transfer to copy the software image to Flash memory

switch: copy xmodem: flash:image_filename.bin

switch: set BAUD 115200

Will lose connection until you reset your terminal emulation to match this new speed


• Once complete, reset speed to 9600 and reboot CIGESM

• Switch should now boot all the way up and allow full operation with newly loaded code

• Note: Depending on how the original IOS upgrade failed, future upgrades using the “archive” command may not succeed (possibly fail with an error referencing the “update” directory.• If you encounter this type of error on future updates, you may

need to delete the “update” directory and all of its files and sub-directories.

• This can be done with a combination of “delete” and “rmdir” commands (may need to cd down into the update directory and manually delete files and directories)

switch: unset BAUD

Will lose connection until you reset your terminal emulation to match this new speed

switch: boot

Documents

Cisco Systems Intelligent Gigabit Ethernet Switch Module for the IBM eServer BladeCenter Cisco Systems Matt Slavin [email protected] Jan 2005