Lessons Learned: Troubleshooting UCS from a TAC … · cs.co/ciscolivebot#BRKINI-2011 ... f241-03-08-5596-a# show mac address-table vlan 211 Legend: * - primary entry, G - Gateway

Lessons Learned: Troubleshooting UCS from a TAC Engineer’s Perspective

Aaron Sampliner, TAC Engineer

BRKINI-2011

© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public

Cisco Spark

Questions? Use Cisco Spark to communicate with the speaker after the session

1. Find this session in the Cisco Live Mobile App

2. Click “Join the Discussion”

3. Install Spark or go directly to the space

4. Enter messages/questions in the space

How

cs.co/ciscolivebot#BRKINI-2011

• The Network is Down!!?

• The Story with Storage

• Serving Up Success with Best Practices

Agenda

The Network is Down!!?Isolating Networking Issues

UCS Networking Basics


Ethernet Switching Modes

• UCS had two types of switching mode

• The mode effects how Layer 2 forwarding concepts are applied

• End Host Mode

• Appears like a hypervisor host to upstream network

• Default and recommended best practice

• Switch Mode

• FI works like a normal layer 2 switch with spanning-tree

7BRKINI-2011


Physical and Logical Ports in UCS

Uplink Ports

Server Ports

Network Interface (NIF) Ports

Host Interface (HIF) Ports

Adapter Port

vNIC / vEthernet / Virtual Interface (VIF)

vNIC

Fabric Interconnect

IOM / Fex

MezzAdapter

BRKINI-2011 8

© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 9BRKINI-2011

Network Troubleshooting for UCS

• Simplify the issue

• UCS only deals with Layer 2

• Are we learning the MAC address on the FI?

• Is the issue fabric specific?

• If multiple servers affected, pick one to work with

TAC Methodology…


Shrinking the Failure Domain

The Network

Is Down

UCS Domain

Is Offline

UCS Server

Not WorkingDB-VM-01

Unreachable

Common Problem:My VM is unreachable on the network


Lab Topology

Port-Channel

UCS 6248UP UCS 6248UP

Nexus 5548 Nexus 5548

vPC

Tracing the Path in UCS


Start from the most southbound endpoint

• Hypervisor Host (blade)

• VM MAC Address

• vSwitch Port Group

• IP Address

Gather information from the hypervisor about the VM…


Understand Virtual Network PathVerifying virtual switch is configured as expected…

Both vmnics are active

Both vmnics connected to vSwitch

VLAN 211 being tagged on vSwitch


Understand Virtual Network PathUsing “esxtop” command with the ‘n’ option…

[root@localhost:~] esxtop

4:50:33pm up 2 days 23:29, 696 worlds, 3 VMs, 6 vCPUs; CPU load average: 0.00, 0.00, 0.00

PORT-ID USED-BY TEAM-PNIC DNAME PKTTX/s MbTX/s PSZTX PKTRX/s MbRX/s PSZRX %DRPTX %DRPRX

33554433 Management n/a vSwitch0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

33554434 vmnic0 - vSwitch0 0.00 0.00 0.00 12.25 0.01 82.00 0.00 0.00

33554435 Shadow of vmnic0 n/a vSwitch0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

33554439 vmk0 vmnic1 vSwitch0 10.11 0.01 181.00 8.16 0.00 73.00 0.00 0.00

33554440 vmnic1 - vSwitch0 10.11 0.01 181.00 20.02 0.01 79.00 0.00 0.00


33554442 512108:rhel7-1 vmnic1 vSwitch0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

33554443 512185:jlil-central vmnic1 vSwitch0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

33554444 510584:Win7-Tool vmnic0 vSwitch0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

50331649 Management n/a vSwitch1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

50331650 vmnic2 - vSwitch1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00


50331652 vmnic4 - vSwitch1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00





Understand Virtual Network Path

• VM specifics

• MAC Address

• IP Address

• Virtual machine is actively using vmnic0 to send traffic northbound in the UCS

• Next, we need to understand which fabric this traffic should be traversing

What we know so far…


Determining Fabric Path in UCS

We use the vmnic MAC Address and match it with the vNIC in UCS

vmnic to vNIC mapping…

[root@localhost:~] esxcfg-nics -l

Name PCI Driver Link Speed Duplex MAC Address MTU Description

vmnic0 0000:06:00.0 enic Up 10000Mbps Full 00:25:b5:a1:a1:a0 1500 Cisco Systems Inc Cisco VIC Ethernet NIC

vmnic1 0000:07:00.0 enic Up 10000Mbps Full 00:25:b5:b1:b1:b1 1500 Cisco Systems Inc Cisco VIC Ethernet NIC

vmnic2 0000:08:00.0 enic Up 10000Mbps Full 00:25:b5:a1:a1:b0 1500 Cisco Systems Inc Cisco VIC Ethernet NIC

vmnic3 0000:85:00.0 enic Up 10000Mbps Full 00:25:b5:b1:b1:b0 1500 Cisco Systems Inc Cisco VIC Ethernet NIC

vmnic4 0000:86:00.0 enic Up 10000Mbps Full 00:25:b5:b1:b1:c0 1500 Cisco Systems Inc Cisco VIC Ethernet NIC

vmnic5 0000:87:00.0 enic Up 10000Mbps Full 00:25:b5:a1:a1:a1 1500 Cisco Systems Inc Cisco VIC Ethernet NIC

Common mistake – assuming vNIC# and vmnic# are the same without verifying


Determining Fabric Path in UCS Match the MAC of the vmnic to the vNIC on the service profile…


Are we learning the MAC address on the FI?

• Based on what we have found we expect the following:

• VM traffic should be traversing Fabric Interconnect A

• VLAN ID should be 211 based on vSwitch config

• VM MAC Address – 00:50:56:8d:29:15

Send traffic from the VM and see what is working…

CiscoLive-2017-A# connect nxos a

CiscoLive-2017-A(nxos)# show mac address-table vlan 211

Legend:

* - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC

age - seconds since last seen,+ - primary entry using vPC Peer-Link

VLAN MAC Address Type age Secure NTFY Ports/SWID.SSID.LID

---------+-----------------+--------+---------+------+----+------------------

* 211 0050.568d.2915 dynamic 10 F F Veth4173


Viewing all virtual interfaces for a serverVerifying that traffic is flowing as expected…

CiscoLive-2017-A# show service-profile circuit server 1/1

Service Profile: JLILL/JLILL-ESX-01

Server: 1/1

Fabric ID: A

Path ID: 1

VIF vNIC Link State Oper State Prot State Prot Role Admin Pin Oper Pin Transport

---------- --------------- ----------- ---------- ------------- ----------- ---------- ---------- ---------

12371 Up Active No Protection Unprotected 0/0/0 0/0/0 Ether

4173 vNIC2 Up Active No Protection Unprotected 0/0/0 0/0/1 Ether


4179 fc0 Up Active No Protection Unprotected 0/0/0 2/0/15 Fc

Fabric ID: B

Path ID: 2


---------- --------------- ----------- ---------- ------------- ----------- ---------- ---------- ---------




4180 fc1 Up Active No Protection Unprotected 0/0/0 0/0/44 Fc

Uplink Pinning


Which uplink is being used?Understanding how pinning works…

Basic rules to define which interface to pin to:

1. Which uplink interfaces are active?

2. Which uplink interfaces carry ALL of the vNIC’s configured VLANs?

3. Which uplink has the least amount of vifs pinned to it currently?

Severity: Major

Code: F0283

Last Transition Time: 2014-02-18T23:08:51.270

ID: 1157440

Status: None

Description: ether VIF 1369 on server 6 / 4 of switch B down, reason: ENM source pinning failed

Affected Object: sys/chassis-6/blade-4/fabric-B/path-1/vc-1369

Name: Dcx Vc Down

Cause: Link Down

Type: Network

Acknowledged: No

Occurrences: 7

Creation Time: 2014-02-11T12:57:11.768

Original Severity: Major

Previous Severity: Cleared

Highest Severity: Major


Which uplink is being used?Uplink pinning commands… CiscoLive-2017-A(nxos)# show pinning border-interfaces active

--------------------+---------+----------------------------------------

Border Interface Status SIFs

--------------------+---------+----------------------------------------

Po1 Active sup-eth2 Veth4137 Veth4145 Veth4173

Veth4175 Veth4178 Veth4183 Veth4195

Veth4197 Veth4200 Veth4208 Veth4210

Veth4212 Veth4214 Veth4216

Eth1/18 Active

CiscoLive-2017-A(nxos)# show pinning server-interfaces

---------------+-----------------+------------------------+----------------

-

SIF Interface Sticky Pinned Border Interface Pinned Duration

---------------+-----------------+------------------------+----------------

-

Eth1/1 No - -

Eth1/2 No - -

Eth1/3 No - -

Eth1/4 No - -

Eth1/11 No - -

Eth1/12 No - -

Veth4137 No Po1 1d 58:3:23

Veth4145 No Po1 1d 57:47:47

Veth4173 No Po1 1d 57:54:31

Two common ways to view


UCS forwarding seems to be working as expectedWhat’s upstream?

CiscoLive-2017-A(nxos)# show port-channel summary

Flags: D - Down P - Up in port-channel (members)

I - Individual H - Hot-standby (LACP only)

s - Suspended r - Module-removed

S - Switched R - Routed

U - Up (port-channel)

M - Not in use. Min-links not met

--------------------------------------------------------------------------------

Group Port- Type Protocol Member Ports

Channel

--------------------------------------------------------------------------------

1 Po1(SU) Eth LACP Eth1/31(P) Eth1/32(P)

• Use show cdp neighbors to determine upstream device and ports

• If not available, trace cables


Reviewing the upstream switchesAre we learning the MAC address?

MAC address not learned for our VM on either upstream switch

f241-03-08-5596-a# show mac address-table vlan 211

Legend:




---------+-----------------+--------+---------+------+----+------------------

* 211 002a.6a35.4a41 static 0 F F sup-eth2

* 211 002a.6a39.2a41 static 0 F F Po3

* 211 547f.ee2f.3381 dynamic 60 F F Po33

f241-03-08-5596-b# show mac address-table vlan 211

Legend:




---------+-----------------+--------+---------+------+----+------------------

* 211 002a.6a35.4a41 static 0 F F Po3

* 211 002a.6a39.2a41 static 0 F F sup-eth2

* 211 547f.ee2f.3381 dynamic 300 F F Po33


Reviewing the upstream switchesConfiguration correct?

Looking at the port-channel configuration, we can see that VLAN 211 is not allowed

f241-03-08-5596-a# show run interface ethernet 1/8

interface Ethernet1/8

switchport mode trunk

switchport trunk allowed vlan 1-210,212-4094

channel-group 11 mode active

f241-03-08-5596-a# show run interface port-channel 11

interface port-channel11

description jlill-ucs-pod


switchport trunk allowed vlan 1-210,212-4094

spanning-tree port type edge trunk

speed 10000

vpc 11

Upstream switchport is Ethernet 1/8 on Nexus 5K

What if we don’t learn the MAC address on the FI?


Did we have issues traversing IOM and VIC?

• Three components left to investigate:

• OS/Driver issues – Did the OS actually send the frame northbound?

• VIC Adapter

• IOM (NIF and HIF ports)

© 2018 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKINI-2011

Cisco VIC AdapterConnecting and identifying logical interfaces…

CiscoLive-2017-A# connect adapter 1/1/1

adapter 1/1/1 # connect

adapter 1/1/1 (top):2# attach-mcp

adapter 1/1/1 (mcp):36# vnic -m

vnic id : internal id of vnic, use for other vnic cmds

vnic name/mac : ucsm provisioned name (-n) or mac address (-m)

vnic type : enet=ethernet, enet_pt=dynamic ethernet, fc=fcoe

vnic host : host

vnic state : state of vnic

lif : internal logical if id, use for other lif/vif cmds

lif state : state of lif

vif uif : bound uplink 0 or 1, =:primary, -:secondary, >:current

vif ucsm : ucsm id for this vif

vif idx : switch id for this vif

vif vlan : default vlan for traffic

vif state : state of vif

-------------------------------------- --------- --------------------------

v n i c l i f v i f

id mac type host state lif state uif ucsm idx vlan state

---- -------------- ------- ---- ----- --- ----- --- ----- ----- ---- -----

14 0025:b5a1:a1a0 enet 0 UP 4 UP =>1 4173 30 1 UP

15 0025:b5b1:b1b1 enet 0 UP 5 UP =>0 4174 29 1 UP

16 0025:b5a1:a1b0 enet 0 UP 6 UP =>1 4175 31 1 UP

17 aa25:b5a1:a1a0 fc 0 UP 7 UP =>1 4179 21 1000 UP

31


Cisco VIC AdapterViewing counters for drops and errors…adapter 1/1/1 (mcp):28# lifstats -a 4

DELTA TOTAL DESCRIPTION

0 0 Tx unicast frames without error

0 0 Tx multicast frames without error

0 0 Tx broadcast frames without error

0 0 Tx unicast bytes without error

0 0 Tx multicast bytes without error

0 0 Tx broadcast bytes without error

0 0 Tx frames dropped

0 0 Tx frames with error

0 0 Tx TSO frames

0 0 Rx unicast frames without error

216 479103 Rx multicast frames without error

5321 10558692 Rx broadcast frames without error

0 0 Rx unicast bytes without error

19077 44142182 Rx multicast bytes without error

386336 778358713 Rx broadcast bytes without error

0 0 Rx frames dropped

0 0 Rx rq drop pkts (no bufs or rq disabled)

0 0 Rx rq drop bytes (no bufs or rq disabled)

0 0 Rx frames with error

0 0 Rx good frames with RSS

0 0 Rx frames with Ethernet FCS error

24 42055 Rx frames len == 64

5403 10715806 Rx frames 64 < len <= 127

27 81854 Rx frames 128 <= len <= 255

83 198061 Rx frames 256 <= len <= 511

0 19 Rx frames 512 <= len <= 1023

0 0 Rx frames 1024 <= len <= 1518

0 0 Rx frames len > 1518

• Tx would mean we sent frames with errors to the IOM

• Rx would mean OS sent bad frame to the adapter

• Small change the connection between adapter and blade is faulty

32

We don’t see any issues on the adapter…


IOM Troubleshooting – 2nd Gen

Uplink #: 1 2 3 4 5 6 7 8

Link status: | | | |

+-+--+--+--+--+--+--+--+-+

SFP: [$][$][$][$][ ][ ][ ][ ]

+-+--+--+--+--+--+--+--+-+

| N N N N N N N N |

| I I I I I I I I |

| 0 1 2 3 4 5 6 7 |

| |

| NI (0-7) |

+------------+-----------+

|

+-------------------------+-------------+-------------+---------------------------+

| | | |

+------------+-----------+ +-----------+------------+ +------------+-----------+ +-------------+----------+

| HI (0-7) | | HI (8-15) | | HI (16-23) | | HI (24-31) |

| | | | | | | |

| H H H H H H H H | | H H H H H H H H | | H H H H H H H H | | H H H H H H H H |

| I I I I I I I I | | I I I I I I I I | | I I I I I I I I | | I I I I I I I I |

| 0 1 2 3 4 5 6 7 | | 8 9 1 1 1 1 1 1 | | 1 1 1 1 2 2 2 2 | | 2 2 2 2 2 2 3 3 |

| | | 0 1 2 3 4 5 | | 6 7 8 9 0 1 2 3 | | 4 5 6 7 8 9 0 1 |

+-+--+--+--+--+--+--+--+-+ +-+--+--+--+--+--+--+--+-+ +-+--+--+--+--+--+--+--+-+ +-+--+--+--+--+--+--+--+-+

[ ][ ][ ][ ][ ][ ][ ][ ] [ ][ ][ ][ ][ ][ ][ ][ ] [ ][ ][ ][ ][ ][ ][ ][ ] [ ][ ][ ][ ][ ][ ][ ][ ]

+-+--+--+--+--+--+--+--+-+ +-+--+--+--+--+--+--+--+-+ +-+--+--+--+--+--+--+--+-+ +-+--+--+--+--+--+--+--+-+

- - | | - | - | | | | | - | | |

1 1 1 1 1 1 1 9 8 7 6 5 4 3 2 1

6 5 4 3 2 1 0

\__\__/__/ \__\__/__/ \__\__/__/ \__\__/__/ \__\__/__/ \__\__/__/ \__\__/__/ \__\__/__/

blade8 blade7 blade6 blade5 blade4 blade3 blade2 blade1

CiscoLive-2017-A# connect iom 1

fex-1# show platform software woodside sts

Board Status Overview:

legend:

' '= no-connect

X = Failed

- = Disabled

: = Dn

| = Up

[$] = SFP present

[ ] = SFP not present

[X] = SFP validation failed

------------------------------


IOM Troubleshooting – 3rd Genfex-1# show platform software tiburon sts


legend:

' '= no-connect

X = Failed

- = Disabled

: = Dn

| = Up

. = SFP waiting for t_start_up timer expiration

i = SFP waiting for prom info read

c = SFP waiting for t_start_up_cooled timer expiration

* = SFP present


------------------------------

* * * * <- QSFPs

+-----------------------------------------------------------------------------------------------+

- - - - - - - - : : : : : : : : : : : : | : : :

+-----------------------------------------------------------------------------------------------+

|0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23|

|I |

|N |

| Tiburon |

| Asic: 0 |

|H |

|I |

|0 1 2 3 4 5 6 7 8 9 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 3 3 3 3 3 3 3 3 3 3 4 4 4 4 4 4 4 4|

| 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7|

+-----------------------------------------------------------------------------------------------+

- - - - - - - - - - - - - - - - - - - - : : : : - - - - - | - | - - - - - - - - - - - - - - - -

3 3 3 2 2 2 2 2 2 2 2 2 2 1 1 1 1 1 1 1 1 1 1 9 8 7 6 5 4 3 2 1

2 1 0 9 8 7 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0

\_\_/_/ \_\_/_/ \_\_/_/ \_\_/_/ \_\_/_/ \_\_/_/ \_\_/_/ \_\_/_/ (r = retimer lane configured)


36


IOM Troubleshootingfex-1# show platform software {tiburon/woodside} rmon 0 hi31

+----------------------+----------------------+-----------------+----------------------+----------------------+-----------------+

| TX | Current | Diff | RX | Current | Diff |

+----------------------+----------------------+-----------------+----------------------+----------------------+-----------------+

| TX_PKT_LT64 | 0| 0| RX_PKT_LT64 | 0| 0|

| TX_PKT_64 | 0| 0| RX_PKT_64 | 386| 15|

| TX_PKT_65 | 379| 15| RX_PKT_65 | 13| 0|

| TX_PKT_128 | 8| 0| RX_PKT_128 | 754| 75|

| TX_PKT_256 | 717| 51| RX_PKT_256 | 0| 0|

| TX_PKT_512 | 12| 0| RX_PKT_512 | 22| 4|

| TX_PKT_1024 | 0| 0| RX_PKT_1024 | 0| 0|

| TX_PKT_1519 | 24| 0| RX_PKT_1519 | 0| 0|

| TX_PKT_2048 | 0| 0| RX_PKT_2048 | 0| 0|

| TX_PKT_4096 | 0| 0| RX_PKT_4096 | 0| 0|

| TX_PKT_8192 | 0| 0| RX_PKT_8192 | 0| 0|

| TX_PKT_GT9216 | 0| 0| RX_PKT_GT9216 | 0| 0|

| TX_PKTTOTAL | 1140| 66| RX_PKTTOTAL | 1175| 94|

| TX_OCTETS | 341435| 20207| RX_OCTETS | 163687| 15984|

| TX_PKTOK | 1140| 66| RX_PKTOK | 1175| 94|

| TX_UCAST | 384| 15| RX_UCAST | 588| 55|

| TX_MCAST | 756| 51| RX_MCAST | 543| 38|

| TX_BCAST | 0| 0| RX_BCAST | 44| 1|

| TX_VLAN | 0| 0| RX_VLAN | 0| 0|

| TX_PAUSE | 0| 0| RX_PAUSE | 0| 0|

| TX_USER_PAUSE | 0| 0| RX_USER_PAUSE | 0| 0|

| TX_FRM_ERROR | 0| 0| | | |

| | | | RX_OVERSIZE | 0| 0|

| | | | RX_TOOLONG | 0| 0|

| | | | RX_DISCARD | 0| 0|

| | | | RX_UNDERSIZE | 0| 0|

| | | | RX_FRAGMENT | 0| 0|

| | | | RX_CRC_NOT_STOMPED | 0| 0|

| | | | RX_CRC_STOMPED | 0| 0|

| TX_OCTETSOK | 341435| 20207| RX_OCTETSOK | 163687| 15984|

+----------------------+----------------------+-----------------+----------------------+----------------------+-----------------+

37


IOM TroubleshootingDo we have errors on the NIF or HIF ports…

fex-1# show platform software woodside loss

+-------+-------------------------------------+------------+-+-----------------------------------+---------------------------------------+

| | | | | | |

| | | | | | frm_to |

| | |Port Extra | | +---------------------------------------|

| | RMON | Drop |S| SS Loss Counters | COS | XOFF |

| +------------+-----------+------------+------------|S|-----------+-----------+-----------+---------------------------------------|

| Port | Tx Pause | Rx Pause | Errors | Counters |x| RX SS | Tx SS | SS Total |0 |1 |2 |3 |4 |5 |6 |7 |0 |1 |

+-------+------------+-----------+------------+------------+-+-----------+-----------+-----------+---+---+---+---+---+---+---+---+---+---+

| 0- NI3| 0| 59896| 0| 7|0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0|

| | | | | |1| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0|

| | | | | |2| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0|

| | | | | |3| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0|

| | | | | |4| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0|

| | | | | |5| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0|

| | | | | |6| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0|

| | | | | |7| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0|

+-------+-------------------------------------+------------+-+-----------+-----------+-----------+---+---+---+---+---+---+---+---+---+---+

| 0-HI27| 770| 0| 0| 0|0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0|

| | | | | |1| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0|

| | | | | |2| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0|

| | | | | |3| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0|

| | | | | |4| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0|

| | | | | |5| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0|

| | | | | |6| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0|

| | | | | |7| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0|

+-------+-------------------------------------+------------+-+-----------+-----------+-----------+---+---+---+---+---+---+---+---+---+---+

38


IOM TroubleshootingTip: Check the IOM ports from NX-OS…

CiscoLive-2017-A(nxos)# show interface counters errors

--------------------------------------------------------------------------------

Port Align-Err FCS-Err Xmit-Err Rcv-Err UnderSize OutDiscards

--------------------------------------------------------------------------------

Eth1/1/1 0 0 0 0 0 0

Eth1/1/2 0 0 0 0 0 0

Eth1/1/3 0 1170 0 1170 0 0

Eth1/1/4 0 0 0 0 0 0

Eth1/1/5 0 0 0 0 0 0

Eth1/1/6 0 0 0 0 0 0

Eth1/1/7 0 0 0 0 0 0

Eth1/1/8 0 0 0 0 0 0

Eth1/1/9 0 0 0 0 0 0

Eth1/1/10 0 0 0 0 0 0

Eth1/1/11 0 0 0 0 0 0

Eth1/1/12 0 0 0 0 0 0

Eth1/1/13 0 0 0 0 0 0

Eth1/1/14 0 0 0 0 0 0

Eth1/1/15 0 0 0 0 0 0

Eth1/1/16 0 0 0 0 0 0

Eth1/1/17 0 0 0 0 0 0

• Ethernet ports x/y/z correlate to HIF’s on the IOM

• x = chassis

• y = module on IOM

• z = port

39


IOM TroubleshootingExample where errors on IOM are indicating issues downstream…

Uplink interfaces rcvd bad frames

Uplink port-channel counters

Adapter-IOM port-channel

CiscoLive-2017-A(nxos)# show interface counters errors

--------------------------------------------------------------------------------

Port Align-Err FCS-Err Xmit-Err Rcv-Err UnderSize OutDiscards

--------------------------------------------------------------------------------

Eth1/6 0 103 0 103 0 0

Eth1/21 0 103 0 103 0 0

Po1027 0 206 0 206 0 0

Po1351 0 207 0 207 0 0

Eth3/1/1 0 0 0 0 0 0

Eth3/1/2 0 0 0 0 0 0

Eth3/1/3 0 0 0 0 0 0

Eth3/1/4 0 207 0 207 0 0

HIF ports on IOM

40

Disjoint Layer 2


What is Disjoint L2?Two different L2 Domains…

• When UCS needs to connect to two different L2 Domains

• Non-adjacent network

• Typical Use Cases

• DMZ

• Backup Solution

• Legacy

Prod vNICBackup

vNIC

Prod Backup

BRKINI-2011 42


Configuration done half-way...

What about these VLANs?


Uplink port configuration in this scenario…

CiscoLive-2017-B(nxos)# show running-config interface ethernet 1/17


description U: Uplink

pinning border

pinning server nf-exporter


switchport trunk allowed vlan 1,104,111,204,211,304,311,900

udld disable

no shutdown

CiscoLive-2017-B(nxos)# show running-config interface port-channel 2




switchport trunk allowed vlan 1,104,111,204,211,304,311

pinning border

speed 10000

44


Understanding the Designated Receiver

• Absence of STP means we rely on other mechanisms to avoid loops

• An uplink will be selected as the broadcast and multicast receiver

• This is referred to as the Designated Receiver

• Done on a per VLAN basis

45BRKINI-2011

Prod

vNIC

Backup

vNIC

Prod Backup

Po2Eth 1/17


Who is the Designated Receiver?

CiscoLive-2017-B(nxos)# show platform software enm internal info vlandb all

vlan_id 1

-------------

Designated receiver: Po2

Membership:

Eth1/17 Po2

vlan_id 104

-------------

Designated receiver: Eth1/17

Membership:

Eth1/17 Po2

vlan_id 111

-------------

Designated receiver: Po2

Membership:

Eth1/17 Po2

vlan_id 900

-------------

Designated receiver: Eth1/17

Membership:

Eth1/17

46

Prod

vNIC

Backup

vNIC

Prod Backup

Po2Eth 1/17

VLAN 900 Only


Disjoint Layer 2 Configured in full…

You must explicitly define which interfaces the VLAN should traverse for ALL VLANs in DJL2


Correct configuration from CLI

CiscoLive-2017-B(nxos)# show running-config interface ethernet 1/17



pinning border

pinning server nf-exporter


switchport trunk allowed vlan 1,900

udld disable

no shutdown

CiscoLive-2017-B(nxos)# show running-config interface port-channel 2




switchport trunk allowed vlan 1,104,111,204,211,304,311

pinning border

speed 10000

48


Networking Troubleshooting Summary

• Keep it simple – UCS Networking is all Layer 2

• Are we learning the MAC on FI?

• Disjoint Layer 2 – Be explicit on VLAN assignment for ALL VLANs.

Quick recap…

The Story with StorageFC and FCoE with UCS


UCS Supported Storage Technologies

Fibre Channel (FC)

FCoE

iSCSI / NFS

51BRKINI-2011

• Connect to existing SAN

• Direct Attach

• Single/Multihop Support

• Direct Attach

• iSCSI Boot Support

• Direct attach supported

through Appliance ports


FC Switching Modes

• End Host Mode

• Default Mode for FIs (NPV)

• Requires NPIV enabled device upstream

• Switch Mode

• Most common use - Direct Attached Storage

Different from Ethernet modes!

SLOT

1

SLOT

5

SLOT

3

SLOT

7

SLOT

2

SLOT

6

SLOT

4

SLOT

8

!

UCS 5108

OK FAIL OK FAIL OK FAIL OK FAIL

How do we talk

to each other?


Fibre Channel Port Types• ‘N’ port: Node ports used to connect devices to switched fabric or point to point configurations.

• ‘F’ port: Fabric ports residing on switches connecting ‘N’ port devices

• ‘E’ port: Expansion ports are essentially trunk ports used to connect two Fibre Channel switches

N N

N F

E E

BRKINI-2011 53


What Is NPIV?• N-Port ID Virtualization (NPIV) provides a means to assign multiple FCIDs to a

single N_Port

• Limitation exists in FC where only a single FCID can be handed out per F-port. Therefore an F-Port can only accept a single FLOGI

• Allows multiple applications to share the same Fiber Channel adapter port

• Main use case is Virtualization

Application Server FC NPIV Core Switch

Email

Web

File Services

Email I/ON_Port_ID 1

Web I/ON_Port_ID 2

File Services I/ON_Port_ID 3

F_Port

F_Port

N_Port

BRKINI-2011 54


What Is NPV? • N-Port Virtualizer (NPV) utilizes NPIV functionality to allow a “switch” to act like

a server performing multiple logins through a single physical link

• Physical servers connected to the NPV switch login to the upstream NPIV core switch

• No local switching is done on an FC switch in NPV mode

• FC edge switch in NPV mode does not take up a domain ID

• Helps to alleviate domain ID exhaustion in large fabrics

FC NPIV Core Switch

Eth1/1

Eth1/2

Eth1/3

Server1N_Port_ID 1

Server2N_Port_ID 2

Server3N_Port_ID 3

F_Port

N-Port

F-Port

F-PortNP-Port

Application Server NPV Switch

BRKINI-2011 55


SAN “End Host” NPV Mode

• Fabric Interconnects in NPV (N Port Virtualization)Mode

• Fabric Interconnect operates in N_Port Proxy (NP) mode

• SAN switch sees Fabric Interconnect as an FC End Host with many N_Ports and many FC IDs assigned

• Server facing ports function as F-proxy ports

• Server vHBA pinned to an FC uplink in the same VSAN. Round Robin selection.

• Provides multiple FC end nodes to one F_Port off an FC Switch

SAN BSAN A

Server 1VSAN 1

vFC

1

vFC

1

N_Proxy (NP) N_Proxy (NP)

F_Prox

yF_Prox

y

N_PortN_Port

6100-

A6100-

B

F_Port

vFC

2

vFC

2

Server 2VSAN 1

vHB

A 1

vHB

A 0

vHB

A 1

vHB

A 0

F_Port

NPIV NPIVFLOGI

FDISC

VSAN

1

VSAN

1

BRKINI-2011 56


Common UCS SAN DeploymentsTraditional deployment of UCS in End Host Mode

FC Storage Array MDS or N5K UCS

SLOT

1

SLOT

5

SLOT

3

SLOT

7

SLOT

2

SLOT

6

SLOT

4

SLOT

8

!

UCS 5108


! ResetConsole

UCS B200 M4

! ResetConsole

UCS B200 M4

CISCO UCS-FI-6332

ENV

LS

STS

BCN

1

2

3

4

L1 L2

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 31 3229 3027 28


SAN FC Switch Mode

• UCS Fabric Interconnect behaves like a FC fabric switch

• Direct Attach FC & FCoE Storage to UCS

• Storage ports can be FC or FCoE

• Light subset of FC Switching features

• Set VSAN on Storage ports

Server 1VSAN 1

vFC

1

vFC

1

F_Port

N_Port

6100-A FC SwitchvFC

2

vFC

2

Server 2VSAN 2

vHB

A 1

vHB

A 0

vHB

A 1

vHB

A 0

FC FCoE

6100-B FC Switch

F_Port

VSAN 1VSAN 2

N_Port

BRKINI-2011 58

FC Boot Configuration


FC Boot - TopologyTraditional deployment of UCS in End Host Mode

FC Storage Array MDS or N5K UCS

SLOT

1

SLOT

5

SLOT

3

SLOT

7

SLOT

2

SLOT

6

SLOT

4

SLOT

8

!

UCS 5108


! ResetConsole

UCS B200 M4

! ResetConsole

UCS B200 M4

CISCO UCS-FI-6332

ENV

LS

STS

BCN

1

2

3

4

L1 L2

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 31 3229 3027 28


FC Boot - UCS ConfigurationBoot from SAN requirements

vHBA


FC Boot - UCS Configuration

Important Settings

• vHBA Name

• WWPN of Target

• Boot LUN ID

Boot from SAN requirements

vHBA

Boot Policy


FC Boot - UCS Verification

Ensure vHBA is

FLOGI’d into FI

CiscoLive-2017-A(nxos)# show npv flogi-table

--------------------------------------------------------------------------------

SERVER EXTERNAL

INTERFACE VSAN FCID PORT NAME NODE NAME INTERFACE

--------------------------------------------------------------------------------

vfc4238 1000 0x6e0051 20:00:00:25:d5:00:00:2f 20:00:00:25:d5:00:00:0f fc2/16

CiscoLive-2017-A# show service-profile circuit server 1/7

Service Profile: asamplin/liveTest

Server: 1/7

Fabric ID: A

Path ID: 1


---------- --------------- ----------- ---------- ------------- ----------- ---------- ---------- ---------


4228 eth0 Offline Unknown No Protection Unprotected 0/0/0 0/0/0 Ether

4238 vhba1 Up Active No Protection Unprotected 0/0/0 2/0/16 Fc


FC Boot - Adapter ProgrammingOption ROM Programmed Correctly!

WWPN from storage

array seen before

server boots


FC Boot - We can see the LUN!Verify that OS installer can see the boot LUN

Troubleshooting FC Boot


FC Boot Troubleshooting – Lunlist CommandCommands entered are bolded



No entry for terminal type "dumb";

using dumb terminal settings.

adapter 1/7/1 (top):1# attach-fls



adapter 1/7/1 (fls):1# lunlist

vnic : 15 lifid: 5

- FLOGI State : flogi est (fc_id 0x6e0051)

- PLOGI Sessions

- WWNN 50:0a:09:83:87:49:80:24 WWPN 50:0a:09:83:87:49:80:24 fc_id 0x450000

- LUN's configured (SCSI Type, Version, Vendor, Serial No.)

LUN ID : 0x0000000000000000 (0x0, 0x4, NETAPP , Hn/ZR40PU7K9)

- REPORT LUNs Query Response

LUN ID : 0x0000000000000000

- Nameserver Query Response

- WWPN : 50:0a:09:83:87:49:80:24

- WWPN : 50:0a:09:81:87:49:80:24

Can only be performed while

server in BIOS/Boot Menu


Lunlist – Step 1: Is the vHBA FLOGI present?









vnic : 15 lifid: 5


Let’s break it down!

FLOGI Must be Established

CiscoLive-2017-A(nxos)# show npv flogi-table

--------------------------------------------------------------------------------

SERVER EXTERNAL

INTERFACE VSAN FCID PORT NAME NODE NAME INTERFACE

--------------------------------------------------------------------------------

vfc4238 1000 0x6e0051 20:00:00:25:d5:00:00:2f 20:00:00:25:d5:00:00:0f fc2/16


Lunlist – Step 2: Is zoning correctly configured?


vnic : 15 lifid: 5


- PLOGI Sessions





LUN ID : 0x0000000000000000


- WWPN : 50:0a:09:83:87:49:80:24

- WWPN : 50:0a:09:81:87:49:80:24

WWPNs returned from

upstream switch

UCS Boot Target

If boot target does not match

returned zone, server will not

boot


Lunlist – Verify Zoning on Upstream Switch

The zone configured on the upstream switch must have the following WWPNs:

• vHBA of Server (Initiator)

• vHBA of Storage (Target)

Similar to ACLs for Ethernet, default behavior is Deny

f241-03-08-5596-a# show zoneset name netapp1-1000 vsan 1000

zoneset name netapp1-1000 vsan 1000

zone name asamplin-live vsan 1000

* fcid 0x6e0051 [pwwn 20:00:00:25:d5:00:00:2f]

* fcid 0x450000 [pwwn 50:0a:09:83:87:49:80:24] [netapp1-1-0a]

* fcid 0x450040 [pwwn 50:0a:09:81:87:49:80:24] [netapp1-1-0c]

Lunlist Output:vnic : 15 lifid: 5


- PLOGI Sessions

- WWNN 50:0a:09:83:87:49:80:24 WWPN 50:0a:09:83:87:49:80:24


Lunlist – Step 3: What is my boot LUN ID?


vnic : 15 lifid: 5


- PLOGI Sessions





LUN ID : 0x0000000000000000


- WWPN : 50:0a:09:83:87:49:80:24

- WWPN : 50:0a:09:81:87:49:80:24

Setting from UCS Boot Policy

Returned from Storage Array

All of these

settings are

defined in boot

policy!


FC Boot Troubleshooting – Lunlist CommandNonworking output









vnic : 15 lifid: 5


- PLOGI Sessions



LUN ID : 0x0000000000000000 access failure



Zoning is not correct on upstream switch

iSCSI Boot Configuration


iSCSI Boot - Topology

iSCSI Storage Array N5K UCS

SLOT

1

SLOT

5

SLOT

3

SLOT

7

SLOT

2

SLOT

6

SLOT

4

SLOT

8

!

UCS 5108


! ResetConsole

UCS B200 M4

! ResetConsole

UCS B200 M4

CISCO UCS-FI-6332

ENV

LS

STS

BCN

1

2

3

4

L1 L2

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 31 3229 3027 28


iSCSI Boot - ConfigurationvNIC Configuration

vNIC


iSCSI Boot - ConfigurationiSCSI vNIC Configuration

vNIC

iSCSI vNIC


iSCSI Boot - ConfigurationiSCSI Boot Policy Configuration

vNIC

iSCSI vNIC

Boot Policy


iSCSI Boot - ConfigurationiSCSI Boot Parameter Configuration

vNIC

iSCSI vNIC

Boot Policy

1. Assign an initiator name to iSCSI vNIC

2. Assign IP address to iSCSI vNIC

1 2


iSCSI Boot - ConfigurationiSCSI Boot Parameter Configuration

vNIC

iSCSI vNIC

Boot Policy

1. Assign an initiator name to iSCSI vNIC

2. Assign IP address to iSCSI vNIC

3. Assign Target IQN to Boot Policy

3


iSCSI Boot - Adapter ProgrammingOption ROM Programmed Correctly!

MAC from storage

array seen before

server boots


iSCSI Boot - We can see the LUN!Verify that OS installer can see the boot LUN

iSCSI BootTroubleshooting


iSCSI Boot Troubleshooting - VerificationF241-11-10-UCS01-B# connect adapter 1/1/1



adapter 1/1/1 (mcp):2# iscsi_get_config

vnic iSCSI Configuration:

----------------------------

vnic_id: 12

host_id: 0

link_state: Up

Initiator Cfg:

initiator_state: ISCSI_INITIATOR_READY

initiator_error_code: ISCSI_BOOT_NIC_NO_ERROR

vlan: 0

dhcp status: false

IQN: iqn.2017-04.allcark.cisco.com:ucs-11-10-1:7

IP Addr: 14.39.4.1

Subnet Mask: 255.255.255.0

Gateway: 14.39.4.254

Target Cfg:

Target Idx: 0

State: ISCSI_TARGET_READY

Prev State: ISCSI_TARGET_DISABLED

Target Error: ISCSI_TARGET_NO_ERROR

IQN: iqn.2008-07.com.whiptail:WT-ACC6-PROD-ACCELA-FCH1820V1VZ

IP Addr: 14.39.4.200

Port: 3260

Boot Lun: 0

Ping Stats: Success (20.589ms)


iscsi_get_config - Breakdown Initiator Configuration

F241-11-10-UCS01-B# connect adapter 1/1/1




vnic iSCSI Configuration:

----------------------------

vnic_id: 12

host_id: 0

link_state: Up

Initiator Cfg:

initiator_state: ISCSI_INITIATOR_READY

initiator_error_code: ISCSI_BOOT_NIC_NO_ERROR

vlan: 0

dhcp status: false

IQN: iqn.2017-04.allcark.cisco.com:ucs-11-10-1:7

IP Addr: 14.39.4.1

Subnet Mask: 255.255.255.0

Gateway: 14.39.4.254

No error is working output

IQN and IP address of iSCSI vNIC configured in UCS


iscsi_get_config - BreakdownTarget Configuration





Target Cfg:

Target Idx: 0

State: ISCSI_TARGET_READY

Prev State: ISCSI_TARGET_DISABLED

Target Error: ISCSI_TARGET_NO_ERROR


IP Addr: 14.39.4.200

Port: 3260

Boot Lun: 0


No error is working output

IQN and IP address of iSCSI Target in Boot Policy

iSCSI vNIC able to ping Target IP address


iscsi_pingQuick Ping Test




adapter 1/1/1 (mcp):2# iscsi_ping

id name tgt address port tcp ping status

--- -------------- --- --------------- ----- ---------------------------------

12 vnic_1 0 14.39.4.200 3260 Success (20.491ms)

13 vnic_2 0 14.39.5.200 3260 Success (19.386ms)

Can only be performed while

server in BIOS/Boot Menu


iscsi_get_config - NonworkingIncorrect LUN in Boot Policy





Target Cfg:

Target Idx: 0

State: INVALID

Prev State: ISCSI_TARGET_GET_LUN_INFO

Target Error: ISCSI_TARGET_GET_HBT_ERROR


IP Addr: 14.39.4.200

Port: 3260

Boot Lun: 5


Error reaching Target

Boot Policy has LUN 5, need to verify with storage admin which LUN we are booting to

iSCSI vNIC able to ping Target IP address

FC User Story


I added links but no extra bandwidth!TAC Case Example

• Hosts Reporting high storage latency

• Customer added 2 additional FC uplinks (tripling bandwidth)

• Customer reported to TAC that nothing changed!

Fabric Interconnect

FC Switch

Newly added

uplinks in black


FC Uplink BehaviorIndividual FC Uplinks

90

fcid1

fcid2

fcid3

fcid4

fcid5

fcid6

Fabric Interconnect

FC SwitchServers pin to individual uplinks

No load balancing

• Customer would need to reboot their servers to send FDISC and utilize new uplinks

• Any new servers would be able to utilize the additional bandwidth

FDISC

FDISC

Yellow Link indicates original uplink

BRKINI-2011


FC Port ChannelsThe power of the bundle

91

Fabric Interconnect

FC Switch

Bundled Port Channel

• Frames are sent round robin per link

BRKINI-2011


Note on FC Port Channels

• FC Port Channels ensure that load balancing happens across FC links

• SAN Uplinks normally carry 1 VSAN at a time

• Port Channel allows dynamic modification of members

• Multiple VSANs needs F-port-channel-trunking enabled

• Requires Cisco FC device upstream

Multiple VSANs Optional


FC Port Channel Verification

• Which VSANs are active on my port channel?

• 201 and 1001 are up because there are vHBAs with active FLOGI into FI

CiscoLive-2017-B(nxos)# show int san 44

san-port-channel 44 is trunking

Hardware is Fibre Channel

Port WWN is 24:2c:54:7f:ee:c5:6c:c0

Admin port mode is NP, trunk mode is on

snmp link state traps are enabled

Port mode is TNP

Port vsan is 1001

Speed is 16 Gbps

Trunk vsans (admin allowed and active) (1,10,200-203,888,1000-1001)

Trunk vsans (up) (201,1001)

Trunk vsans (isolated) (10,200,202,888,1000)

Trunk vsans (initializing) (1,203)

1 minute input rate 13560 bits/sec, 1695 bytes/sec, 4 frames/sec

1 minute output rate 7480 bits/sec, 935 bytes/sec, 4 frames/sec

1940486 frames input, 2478434904 bytes

83 discards, 0 errors

0 CRC, 0 unknown class

0 too long, 0 too short

736055 frames output, 89418044 bytes

0 discards, 0 errors

4 input OLS, 4 LRR, 3 NOS, 0 loop inits

10 output OLS, 2 LRR, 0 NOS, 0 loop inits

last clearing of "show interface" counters never

Member[1] : fc2/15

Member[2] : fc2/16

Interface last changed at Sun May 14 20:03:28 2017

Troubleshooting FC Congestion in UCS


How to check for FC Congestion within UCSSince UCS is normally in End Host mode, congestion is hard to find

• Storage traffic within UCS is FCoE if blades are using FC or FCoE

• Isolate which blades have the issue

• Track down the path within UCS via IOM


Scenario - FC Aborts Seen in OS Logs

• OS log showing issues hitting storage

• Investigate adapter logs on UCS

• Command displayed searches the log for aborts

Where do we begin troubleshooting?



adapter 1/7/1 (top):1# show-log

160309-20:25:43.456386 ecom.ecom_main ecom(8:2): abort called for exch 68f1,

status 3 rx_id 8517 s_stat 0x1 xmit_recvd 0x400 burst_offset 0x400 burst_len 0x0

sgl_err 0x0 last_param 0x0 last_seq_cnt 0x0 tot_bytes_exp 0x400 h_seq_cnt 0x0

exch_type 0x1 s_id 0x450020 d_id 0x450060 host_tag 0x58

160309-20:25:45.526540 ecom.ecom_main ecom(8:2): abort called for exch 69db,

status 3 rx_id 87f0 s_stat 0x1 xmit_recvd 0x2000 burst_offset 0x2000 burst_len 0x0

sgl_err 0x0 last_param 0x0 last_seq_cnt 0x3 tot_bytes_exp 0x2000 h_seq_cnt 0x0

exch_type 0x1 s_id 0x450060 d_id 0x6e0051 host_tag 0xf

Aborts are normal behavior, however a significant

number of aborts can indicate an issue!

Numbers are relative!


Scenario - FC Aborts Seen in OS Logs

• Check show fcns database upstream

• Source FCID and DestFCID should be present

• Conversation was between array and itself, and array to UCS

Verify Source and Destination FCIDs

Abort 1: s_id 0x450020 d_id 0x450060

Abort 2: s_id 0x450060 d_id 0x6e0051

f241-03-08-5596-a# show fcns database vsan 1000

VSAN 1000:

--------------------------------------------------------------------------

FCID TYPE PWWN (VENDOR) FC4-TYPE:FEATURE

--------------------------------------------------------------------------

0x450000 N 50:0a:09:83:87:49:80:24 (NetApp) scsi-fcp:target

[netapp1-1-0a]


[netapp1-2-0a]


[netapp1-1-0c]


[netapp1-2-0c]

0x6e0051 N 20:00:00:25:d5:00:00:2f scsi-fcp:init fc-gs


IOM TroubleshootingWhich server ports do we want to investigate?

Uplink #: 1 2 3 4 5 6 7 8

Link status: | | | |

+-+--+--+--+--+--+--+--+-+

SFP: [$][$][$][$][ ][ ][ ][ ]

+-+--+--+--+--+--+--+--+-+

| N N N N N N N N |

| I I I I I I I I |

| 0 1 2 3 4 5 6 7 |

| |

| NI (0-7) |

+------------+-----------+

|

+-------------------------+-------------+-------------+---------------------------+

| | | |

+------------+-----------+ +-----------+------------+ +------------+-----------+ +-------------+----------+

| HI (0-7) | | HI (8-15) | | HI (16-23) | | HI (24-31) |

| | | | | | | |

| H H H H H H H H | | H H H H H H H H | | H H H H H H H H | | H H H H H H H H |

| I I I I I I I I | | I I I I I I I I | | I I I I I I I I | | I I I I I I I I |

| 0 1 2 3 4 5 6 7 | | 8 9 1 1 1 1 1 1 | | 1 1 1 1 2 2 2 2 | | 2 2 2 2 2 2 3 3 |

| | | 0 1 2 3 4 5 | | 6 7 8 9 0 1 2 3 | | 4 5 6 7 8 9 0 1 |

+-+--+--+--+--+--+--+--+-+ +-+--+--+--+--+--+--+--+-+ +-+--+--+--+--+--+--+--+-+ +-+--+--+--+--+--+--+--+-+

[ ][ ][ ][ ][ ][ ][ ][ ] [ ][ ][ ][ ][ ][ ][ ][ ] [ ][ ][ ][ ][ ][ ][ ][ ] [ ][ ][ ][ ][ ][ ][ ][ ]

+-+--+--+--+--+--+--+--+-+ +-+--+--+--+--+--+--+--+-+ +-+--+--+--+--+--+--+--+-+ +-+--+--+--+--+--+--+--+-+

- - | | - | - | | | | | - | | |

1 1 1 1 1 1 1 9 8 7 6 5 4 3 2 1

6 5 4 3 2 1 0

\__\__/__/ \__\__/__/ \__\__/__/ \__\__/__/ \__\__/__/ \__\__/__/ \__\__/__/ \__\__/__/


fex-1# show platform software woodside sts


legend:

' '= no-connect

X = Failed

- = Disabled

: = Dn

| = Up

[$] = SFP present

[ ] = SFP not present


------------------------------


IOM TroubleshootingChecking the network port between FI and IOM

fex-1# show platform software woodside rmon 0 ni0 | in PAUSE

+----------------------+----------------------+-----------------+----------------------+----------------------+-----------------+

| TX | Current | Diff | RX | Current | Diff |

+----------------------+----------------------+-----------------+----------------------+----------------------+-----------------+

| PORT CNTRS NI0 |

| TX_PAUSE | 0| 0| RX_PAUSE | 0| 0|

| TX_USER_PAUSE | 1956| 78| RX_USER_PAUSE | 87512| 3564|

Pause frames

are normal

behavior.

Number is

relative!

Significantly more user pause on RX

8

7

6

5

UCS 2208XP

4

3

2

1

RX from FI

(coming from upstream)

Remember that

TX and RX are

from the IOM’s

perspective

TX to FI

(going to upstream)

2208XP IOM


Storage Troubleshooting Summary

• Lunlist/iscsi_get_config only available before boot

• Virtually no drawback from a SAN port channel

• Aborts and pause frame numbers are relative, need to look at the aggregate

Quick recap…

Serving Up Success with Best Practices


UCS Manager Health Check

• Connect local (a/b) via SSH

• Show cluster extended-state

• Check for L1/L2 up, is HA ready

CiscoLive-2017-A(local-mgmt)# show cluster extended-state

Cluster Id: 0x2c092182748311e2-0x8ed9547feec569c4

Start time: Wed Apr 19 14:41:20 2017

Last election time: Wed Apr 19 14:42:58 2017

A: UP, PRIMARY

B: UP, SUBORDINATE

A: memb state UP, lead state PRIMARY, mgmt services state: UP

B: memb state UP, lead state SUBORDINATE, mgmt services state: UP

heartbeat state PRIMARY_OK

INTERNAL NETWORK INTERFACES:

eth1, UP

eth2, UP

HA READY

Detailed state of the device selected for HA storage:

Chassis 1, serial: FOX1330GDH1, state: active


UCS Manager Health Check

• Connect local (a/b) via SSH

• Show pmon state local to each FI

• Every process should show running with no cores and no retries

• If there are issues, please contact TAC before upgrading

CiscoLive-2017-A(local-mgmt)# show pmon state

SERVICE NAME STATE RETRY(MAX) EXITCODE SIGNAL CORE

------------ ----- ---------- -------- ------ ----

svc_sam_controller running 0(4) 0 0 no

svc_sam_dme running 0(4) 0 0 no

svc_sam_dcosAG running 0(4) 0 0 no

svc_sam_bladeAG running 0(4) 0 0 no

svc_sam_portAG running 0(4) 0 0 no

svc_sam_statsAG running 0(4) 0 0 no

svc_sam_hostagentAG running 0(4) 0 0 no

svc_sam_nicAG running 0(4) 0 0 no

svc_sam_licenseAG running 0(4) 0 0 no

svc_sam_extvmmAG running 0(4) 0 0 no

httpd.sh running 0(4) 0 0 no

httpd_cimc.sh running 0(4) 0 0 no

svc_sam_sessionmgrAG running 0(4) 0 0 no

svc_sam_pamProxy running 0(4) 0 0 no

dhcpd running 0(4) 0 0 no

sam_core_mon running 0(4) 0 0 no

svc_sam_rsdAG running 0(4) 0 0 no

svc_sam_svcmonAG running 0(4) 0 0 no

Healthy FI

Upgrade Troubleshooting

© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 105

UCS Firmware UpgradesTreat them like elective surgery…

• Pre-op check-up

• The operation

• Recovery Room

• Released from surgical center

Pre-op check-up Review documentation

The operation The upgrade

Recovery room Verify functionality

Patient discharged Resume production

BRKINI-2011


Upgrading UCS – Order of Operations

• Upgrade UCS Manager

• Upgrade IOMs on SUB side

• Upgrade Subordinate FI

• Perform Cluster Lead Change

• Upgrade new Subordinate FI

Upgrade properly to avoid issues


UCS Upgrade – FI Gets ‘Stuck’

---- Basic System Configuration Dialog ----

This setup utility will guide you through the basic configuration of

the system. Only minimal configuration including IP connectivity to

the Fabric interconnect and its clustering mode is performed through these steps.

Type Ctrl-C at any time to abort configuration and reboot system.

To back track or make modifications to already entered values,

complete input till end of section and answer no when prompted

to apply configuration.

Enter the configuration method. (console/gui) ?

Don’t Panic!


UCS Upgrade – FI Gets ‘Stuck’

Enter the configuration method. (console/gui) ? console

Installer has detected the presence of a peer Fabric interconnect. This Fabric interconnect

will be added to the cluster. Continue (y/n) ? y

Enter the admin password of the peer Fabric interconnect:

Connecting to peer Fabric interconnect... done

Retrieving config from peer Fabric interconnect... done

Peer Fabric interconnect Mgmt0 IP Address: 10.1.1.1

Peer Fabric interconnect Mgmt0 IP Netmask: 255.255.255.0

Cluster IP address : 10.1.1.254

Physical Switch Mgmt0 IPv4 address : 10.1.1.2

Apply and save the configuration (select 'no' if you want to re-enter)? (yes/no): yes

Applying configuration. Please wait.

Configuration file - Ok

Always make sure there is console access to your FIs


What if my auto install fails? – TAC Trick!

Set the version to not set and upgrade now to clear out a ‘stuck’ auto install


Don’t forget the drivers!

• Critical step often missed

• Make it part of the server upgrade workflow

• Will require a reboot, couple this with the server fw upgrade

• Consult UCS Hardware and Software Compatibility Matrix


Don’t forget the drivers!


93

7

Customer Response

Yes No

20

80

TAC Perception

Yes No

112BRKINI-2011

Do you read the release notes?Unofficial survey…


TAC Favorites in Documentation

• New features!

• Deprecated Hardware

• Which version of code can I run?

• Open and Resolved Caveats are huge!

Questions?


Cisco Spark

Questions? Use Cisco Spark to communicate with the speaker after the session

1. Find this session in the Cisco Live Mobile App

2. Click “Join the Discussion”

3. Install Spark or go directly to the space

4. Enter messages/questions in the space

How

cs.co/ciscolivebot#BRKINI-2011


• Please complete your Online Session Evaluations after each session

• Complete 4 Session Evaluations & the Overall Conference Evaluation (available from Thursday) to receive your Cisco Live T-shirt

• All surveys can be completed via the Cisco Live Mobile App or the Communication Stations

Don’t forget: Cisco Live sessions will be available for viewing on-demand after the event at www.ciscolive.com/global/on-demand-library/.

Complete Your Online Session Evaluation

http://ciscolive.com/Online

http://www.ciscolive.com/global/on-demand-library/


Continue Your Education

• Demos in the Cisco campus

• Walk-in Self-Paced Labs

• Tech Circle

• Meet the Engineer 1:1 meetings

• Related sessions

117BRKINI-2011

Thank you

Problems with Profiles, Pools, and Policies

UCS Manager Troubleshooting


Is

unavailable

bad?

I have a

fault, what is

the actual

problem?

121BRKINI-2011

Intro to UCS Triage

How do I

quickly find

issues?


Anatomy of a Fault


What does this fault tell us?


UCS Fault Generation – Named Policy Unresolvable

Let’s break UCS!

• VLAN 444 created on vNIC eth1

• What happens when VLAN 444 is deleted?


Named Policy Unresolvable - DeletionTime to delete VLAN 444!


Named Policy Unresolvable Fault

Fault is generated when a template refers to an object that has been deleted.

To resolve this, offending item needs to be removed from UCSM.

The Power of Templates


How Many Clicks to Add a VLAN to a vNIC?

1. Select Service Profile under Servers

2. Select Network Tab

3. Select vNIC

4. Select Modify

5. Add VLAN

Why use a template?

• 5 clicks per vNIC

• Imagine there are 20 servers to update

• 5 clicks x 20 servers = 100 clicks!

• Now imagine we need to change 4 vNICs per server

• 5 clicks x 4 vNICs x 20 servers = 400 clicks!

BRKINI-2011 128


We love templates!

Different template types:

Many different types of templates available in UCSM

Service

ProfilevNIC

vHBALAN/SAN

con policy


Template Types

• Initial vs Updating

• Make large changes at the drop of a hat


How to troubleshoot with TemplatesUnbinding a server is an excellent troubleshooting tool!

UCSM TroubleshootingTips


Unable to Associate – Insufficient ResourcesWhat do we do?

Why are there not

enough resources?

User Questions:

What resources are

we talking about?

How can I add more

resources?


Unable to Associate – Insufficient Resources

• UCSM is looking for hardware that is not available

• Insufficient resources can mean multiple issues

What’s on the profile?

• Our adapters are present, what could be the issue?


Unable to Associate – Insufficient ResourcesWhat else is missing?

• All the hardware is present

• Is there a physical server available to associate to?



• No servers in the server pool selected

Where is the physical server?



• No servers in the server pool selected

• Assign to an available server

• UCSM now has hardware to fulfill the profile requirements

Assign a physical server

Success!


Board Controller Activation Fails

Very commonly seen critical fault on UCS, simple trick to resolve!

CiscoLive-2017-A# scope server 1/7

CiscoLive-2017-A /chassis/server # scope boardcontroller

CiscoLive-2017-A /chassis/server/boardcontroller # show version

BoardController:

Running-Vers: 12.0

Package-Vers: 3.1(2f)B

Activate-Status: Failed

CiscoLive-2017-A /chassis/server/boardcontroller # activate firmware 12.0 force

Warning: When committed this command will reset the end-point

CiscoLive-2017-A /chassis/server/boardcontroller* # commit-buffer

CiscoLive-2017-A /chassis/server/boardcontroller #

Other Common Issue Callouts...


Firmware Auto Sync Policy

• Introduced in 2.2(1b)

• Uses the default host firmware package

• Discovery issues can arise if firmware is too low for environment.

• Recommendation – Set to User Acknowledge

• 2.2(6c) removed Auto Acknowledge option and set No Action to default setting

Things to keep in mind…


Chassis Discovery Policy and Active Links

• Adding links between IOM and FI requires chassis re-acknowledgement

• Same applies for configuring IOM port-channel

• Prior to 2.2(4b), this required an outage for the entire chassis

• Per Fabric Interconnect chassis re-acknowledgment

• If you see Configured as the Port State, you need to re-acknowledge

Things to keep in mind…

CiscoLive-2017-A(nxos)# show interface fex-fabric

Fabric Fabric Fex FEX

Fex Port Port State Uplink Model Serial

---------------------------------------------------------------

1 Eth1/1 Active 1 N20-C6508 FCH1708J6P9

1 Eth1/2 Active 2 N20-C6508 FCH1708J6P9

1 Eth1/3 Configured 3 N20-C6508 FCH1708J6P9

1 Eth1/4 Configured 4 N20-C6508 FCH1708J6P9


Fabric Evacuation

• Force all traffic to desired fabric

• Perfect for redundancy testing

• Use during UCS upgrades

• Auto-install support for 3.1(3)

• C-Series Direct Attach not supported

• FEX connectivity is supported

A couple of use cases to highlight…


IOM TroubleshootingChecking the backplane port facing our server

fex-1# show platform software woodside loss

+-------+-------------------------------------+------------+-+-----------------------------------+---------------------------------------+

| | | | | | |

| | | | | | frm_to |

| | |Port Extra | | +---------------------------------------|

| | RMON | Drop |S| SS Loss Counters | COS | XOFF |

| +------------+-----------+------------+------------|S|-----------+-----------+-----------+---------------------------------------|

| Port | Tx Pause | Rx Pause | Errors | Counters |x| RX SS | Tx SS | SS Total |0 |1 |2 |3 |4 |5 |6 |7 |0 |1 |

+-------+-------------------------------------+------------+-+-----------+-----------+-----------+---+---+---+---+---+---+---+---+---+---+

| 0- HI7| 162938| 0| 222| 0|0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0|

| | | | | |1| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0|

| | | | | |2| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0|

| | | | | |3| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0|

| | | | | |4| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0|

| | | | | |5| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0|

| | | | | |6| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0|

| | | | | |7| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0|

+-------+-------------------------------------+------------+-+-----------+-----------+-----------+---+---+---+---+---+---+---+---+---+---+

This command is a diff, so output will be

relative to the last time it was collected


Blank Host Firmware Package – TAC Trick!

A blank host firmware

package can make

upgrading blades

simple and easy

Documents

Lessons Learned: Troubleshooting UCS from a TAC … · cs.co/ciscolivebot#BRKINI-2011 ... f241-03-08-5596-a# show mac address-table vlan 211 Legend: * - primary entry, G - Gateway