1

6/26/2013 Cisco Live 2013

2

6/26/2013 Cisco Live 2013

On average, how do your network administrators and other network IT professionals spend their time on your access (edge) switches?

3

4

6/26/2013 Cisco Live 2013

5

6

7

8

9

10

Port connectivity issues arise from NIC, cable, or switch port faults. These problems may be hardware or software related. There are two different traffic related issues: Not passing any traffic and Passing too much. If no traffic is passing verify configuration settings on both devices of the link. Move one of the devices to a known working partner and see if the problem moves with it or not. Too much traffic could indicate a number of issues: broadcast storm, unicast flood, or oversubscription. A Sniffer capture is usually necessary to identify the types of packets overtaking the interface. Speed/Duplex issues are typically seen at the access layer where user PCs move and change frequently. It is recommended that, where possible, that you hard code speed/duplex setting – particularly for connections that should not change, eg: servers, routers, etc.

11

12

13

%LINK-4-ERROR: FastEthernet0/1 is experiencing errors The above Link error message pops up because of the following potential problems: Excessive Alignment errors -> 1 per 100 ms Excessive FCS errors -> 1 per 100 ms Excessive TX collisions -> 1 per 100ms Late collisions -> 1 per 100 ms

14

Algorithms are used to determine path (platform specific). Either mac address or IP address used for path determination. All packets take same path for a given source to destination.

15

fixing speed and duplex should be done on both sides See CSCtj21335 and workaround: https://supportforums.cisco.com/docs/DOC-23267

17

TDR= Time domain Reflectometry; available for copper interfaces up to 1GE speed Interfaces will be brought down and up when run on active ports

© 2012, Cisco Systems, Inc. All rights reserved. 18

19

Packets with IP options, Packets with expired TTL, Glean packets, ARP, Snooping, Software ACLs, SNMP, etc.

20

Capturing process utilization at the “right” moment is key for identifying the cause Processes : For ex “show tech” causes the virtual exec process to use some CPU resources Traffic Forwarding :

Data traffic not forwarded by ASIC

Excessive Control Plane / Management traffic:

DoS attacks (TTL=1)

SVI ping test

Requires inspecting CPU queues and ASIC

*Note: show tech causes the virtual exec process to use

some CPU resources

CPU util sustained below 50% will not cause problems.

Example of Syslog msg for high CPU

002182: *Jul 20 04:23:36: %SYS-1-CPURISINGTHRESHOLD: Threshold:

Process CPU Utilization(Total/Intr): 9%/0%, Top 3 processes(Pid/Util):

214/3%, 153/0%, 159/0%

Sorting the output is better than filtering the output with “exclude 0.00%”

because that will exclude processes that you want to see.

2960-S will have a higher CPU util that is considered normal.

The “normal” cpu usage depends on number of members in the stack, routing protocols, spanning tree instances, …

High CPU Utilization?

Looking for lost packets?

Use EEM to take the process usage snapshot at the right time: event manager applet High_CPU event snmp oid 1.3.6.1.4.1.9.9.109.1.1.1.1.6 get-type next entry-op gt entry-val 90 poll-interval 10 action 1.1 syslog msg "High CPU DETECTED. Please wait - logging Information to flash:show_proc_cpu.txt" action 1.2 cli command "enable" action 1.3 cli command "end" action 1.4 cli command "term exec prompt timestamp" action 1.5 cli command "show process cpu sorted | redirect flash:show_proc_cpu.txt" action 2.1 syslog msg "Capturing IP Traffics - logging Information to flash:show_ip_traffic.txt" action 2.2 cli command "show ip traffic | redirect flash:show_ip_traffic.txt" action 3.1 syslog msg "Capturing show tech. Please wait - logging Information to flash:show_tech.txt" action 3.2 cli command "show tech | redirect flash:show_tech.txt" action 3.3 syslog msg "Self-removing applet from configuration..." action 3.4 cli command "configure terminal” action 3.5 cli command "no event manager applet High_CPU" action 3.5 cli command "end"

20

21

Each queue has its own tuned buffers and scheduling. These are not configurable, but were highly tested and tweaked before the Catalyst 3750 was released.

rpc: Internal messaging queue STP: STP messages ipc: L3 internal messaging queue Routing protocols: L3 protocols like OSPF L2 protocol: CDP etc other than STP remote console: Stacking slave console Software forwarding: Fast switching, CPU forwarding Host to Host functions: Ping, Telnet etc. Broadcast: L2 broadcast packets cbt-to-spt: Used by Multicast IGMP snooping: Used by Multicast ICMP: Used by IP Logging: ACL logging and Smart Logging RPF-fail: Multicast RPF fail Queue14: Unused CPU heartbeat: Internal wellness check

For example there are more buffers for STP and handle more STP packets since one doesn’t want to drop them. Logging in comparison is a low priority activity. The thresholds are different can vary from 1 to 100s of packets. They are dynamic and can grow. The thresholds numbers may be changed, if required, depending on new feature requirements etc. The stack ring has capability of reserving bandwidth for priority traffic. We use it to ensure the stack messages can work even under heavy user

load. This is guaranteed bandwidth by hardware even under heavy user overloading of the stack. All stack message are handled in the rpc queue which is tuned with larger buffers and scheduling. Flows eligible for CPU forwarding are:

Control plane traffic Management Traffic TCAM overflow traffic

ACL overflow MAC entry overflow Routing Table Overflow

Special protocol flows, these are typically low volume and unofficially supported.

Depth of CPU Qs cannot be modified

The HW (i.e. the port ASIC) will drop on queue congestion

Overload on one CPU Queue should not affect other Queues STP has its own queue – Queue 1 Queue 4 for the other L2 Protocols Values are cumulative Use “clear controllers cpu” or repeat the command multiple times

21

22

Use the debug platform cpu-queues privileged EXEC command to enable debugging of platform central processing unit (CPU) receive queues. software-fwd-q Debug packets received by the software forwarding queue. When running the debug:

Command Purpose

Step 1 configure terminal Enter global configuration mode.

Step 2 no logging console Disable logging to the console terminal.

Step 3 logging buffered 128000 Enable system message logging to a local buffer, and set the buffer size to

12800 bytes.

Step 4 service timestamps debug datetime msecs localtime

Configure the system to apply a timestamp to debugging messages or

system logging messages.

Step 5 exit Return to privileged EXEC mode.

BGL-3700-3#sh cd ne

Capability Codes: R - Router, T - Trans Bridge, B - Source Route Bridge

S - Switch, H - Host, I - IGMP, r - Repeater, P - Phone,

D - Remote, C - CVTA, M - Two-port Mac Relay

Device ID Local Intrfce Holdtme Capability Platform Port ID

BGL14-TACLAB-ASW-J08

Gig 1/0/2 158 S I WS-C3550- Fas 0/16

BGL14-TACLAB-ASW-J08

Gig 2/0/2 131 S I WS-C3550- Fas 0/40

BGL-3700-3#sh arp

Protocol Address Age (min) Hardware Addr Type Interface

Internet 14.160.38.130 - c471.fe1e.f0c0 ARPA Vlan1

Internet 14.160.38.1 1 0007.7d75.88c0 ARPA Vlan1

BGL-3700-3#

Ping with options from 14.160.38.1

*Mar 1 10:37:33.205 AEDT: SW-FWD-Q:IP packet: Local Port Fwding L3If:Vlan1

22

L2If:GigabitEthernet2/0/2 DI:0x2F, LT:7, Vlan:1 SrcGPN:56, SrcGID:56, ACLLogIdx:0x0, MacDA:c471.fe1e.f0c0, MacSA: 0007.7d75.88c0 IP_SA:14.160.38.1 IP_DA:14.160.38.130 IP_Proto:1 IP Opts

TPFFD:D8C00038_00010001_00A00076-0000002F_E2C50000_00000000

*Mar 1 10:37:33.205 AEDT: SW-FWD-Q:Consumed by SW-Bridging: Local Port Fwding L3If:Vlan1 L2If:GigabitEthernet2/0/2 DI:0x2F, LT:7, Vlan:1 SrcGPN:56, SrcGID:56, ACLLogIdx:0x0, MacDA:c471.fe1e.f0c0, MacSA: 0007.7d75.88c0 IP_SA:14.160.38.1 IP_DA:14.160.38.130 IP_Proto:1 IP Opts

TPFFD:D8C00038_00010001_00A00076-0000002F_E2C50000_00000000

*Mar 1 10:37:53.765 AEDT: SW-FWD-Q:IP packet: Local Port Fwding L3If:Vlan1 L2If:GigabitEthernet2/0/2 DI:0x2F, LT:7, Vlan:1 SrcGPN:56, SrcGID:56, ACLLogIdx:0x0, MacDA:c471.fe1e.f0c0, MacSA: 0007.7d75.88c0 IP_SA:14.160.38.1 IP_DA:14.160.38.130 IP_Proto:1 IP Opts

TPFFD:D8C00038_00010001_00A00076-0000002F_E2C50000_00000000

A good practice to protect and to monitor the CPU utilization is to confifure the process cpu threshold and to configure the SW to control the broadcast, multicast and unicast traffic per interface.

22

23

Debug traffic received by CPU. In case below “routing-protocol-q” is shown Packet ingress intf, Dest MAC, SrcMAC, Dest IP, Src IP are shown

24

When free buffers reaches below watermark(32), throttling might occur, resulting in packet drops slow responsiveness to network management

25

Receives a copy of the traffic for which an ICMP packet needs to be generated. Hardware forwarding of the packet still occurs

26

(due to throttling mechanism it won’t reach 99%) of which

27

Add featureset influence over CPU --3750X - 22%- 50% (depending on number of switches)

28

Configuring Traffic Storm Control to avoid packets flood the LAN, creating excessive traffic and degrading network performance.

29

30

I/O memory is not used for normal packet switching

31

Note: lowest free level since boot up

32

Memory Allocation Failure Memory Allocation failure is the condition where the system has used all available memory (temporarily or permanently), or the memory has fragmented into such small pieces that the switch cannot find a usable available block. Memory Leak A memory leak occurs when a process requests or allocates memory and then forgets to free (de-allocate) the memory when it is finished with that task. As a result, the memory block is reserved until the system is reloaded. Over time, more and more memory blocks are allocated by that process until there is no free memory available.

33

34

Use caution while running the command. Might cause cpu spikes Run multiple times to benchmark

In the above output , Total represent the total number of buffers in the pool, which include used and unused buffers. Permanent identifies the permanent number of allocated buffers in the pool. These buffers are always in the pool and can not be trimmed. In free list identifies the number of buffers currently in the pool that are available for use. Min identifies the minimum number of buffers that the system should attempt to keep in the free list. If the number of buffers in free list falls below the min value, system attempts to create more buffers for that pool. Max-allowed identifies the maximum number of buffers that are allowed in the free list Hits identifies the number of buffers that have been requested from the pool. The hits counter provides a mechanism to determine which pool must meet the highest demand for buffers. Misses identifies the number of times that a buffer has been requested and the system detected in which pool additional buffers were required. The misses counter represents the number of times the system has been forced to create additional buffers. Trims identifies the number of buffers that the system has trimmed from the pool, when the number of buffers in the free list exceeded the number of max-allowed buffers.

35

Created identifies the number of buffers that have been created in the pool. Failures identifies when IOS fails to get a Small buffer, it does not drop the packet. It increments the failed counter and falls through to the next level buffer, which is the Middle buffer and requests a buffer there. If it fails to get a middle buffer, it requests the next level buffer, which is a Big buffer. This process continues until it hits the Huge buffer pool. If it fails to get a Huge buffer, then it drops the packet. No memory identifies the number of failures caused by insufficient memory to create additional buffers. Buffer Misses do not necessarily mean a bad thing, as long as the system is able to create additional buffers . The fields to look for in the 'show buffers' output are Failures and No Memory. If there a lot of Failures and No Memory (constantly incrementing) for any Buffer Pool, try to narrow down the source of the buffer failure.You can use the 'show memory debug leaks' to detect I/O memory leaks as well. However remember - it is mandatory that memory leak detector be invoked multiple times and that only leaks that consistently appear in all reports be interpreted as leaks. This is especially true for packet buffer leaks.

35

36

37

38

39

40

41

42

View Asic stats for Ingress Queue (enqueue’d and dropped) & supervisor Queue- - -Output is different for C3750X than C3750G

C2960S does not have ingress Queues

43

44

45

Why have Ingress QoS?

C2960-S Smallest configurable policing rate is 16Kbps, and 8Kbps for everything else.

2960-X and 2960-XR will follow a similar model

© 2009, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr 48

49

Assume there’s a policer defined on the interface via a service policy

Transition to slower speed link – packets take longer to egress than ingress

Eg: Gigabit interfaces for Data Center Servers and old IP Phones

Over Subscription : Many interfaces transmitting to one egress interface

50

A small burst from the 10Gig (faster) interface causes congestion on 100Mbps (slower) interface

51

Total Passengers: 538 First Class: 9 Business Class: 80 Economy Upper Deck: 106 Economy Lower Deck: 343

© 2009, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr 53

Buffers: relative allocation among queues Reserved: minimal amount percentage of buffer reserved by each queue; extra amount is released to common buffer T1, T2, MAX: flexible threshold, expressed as a percentage of nominal queue buffer, which can be used Each traffic class maps to a specific queue number and threshold number (T1, T2 or T3=MAX) For example, the orange class maps to Q2-T1 and the violet class to Q2-T3 (Q2-MAX), allowing more violet packets to queue as they can use common pool buffer space

© 2009, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr 54

Using “maps”, traffic classes mapped to Queue and threshold

55

56

57

58

59

Note: egress interface speed change at top.

60

Queue numbering is 0 based in this slide, rather than 1 based on previous slide (with DSCP mapping) Old IOS version do not have detailed Queue output. Starts in 12.2(46)SE

61

Besides, using “show mls qos interface <intf> statistics” command, use the platform command to get per interface per queue statistics for drop (and for successfully egress)

62

Fixing drops from previous example on Q4 and T1

63

Modify the queue-set from the previous slide to prevent packet drops Threshold maximum is 3200%

64

65

66

See Chapter “Configuring SDM Templates” in the Catalyst Switch Configuration Guide for more information

67

68

Content Addressable Memory (CAM) • Very high speed lookup in large tables • Binary operation—matches based on 0 or 1 values • Exact match returns “hit” • Useful for lookups where lookup key must exactly match a table entry (VLAN + MAC in bridge table) TCAM Tables • Ternary Content Addressable Memory (TCAM) • Very high-speed, fixed latency lookups with wildcarding • Ternary operation—matches based on 0, 1 or X (don’t care) • Longest match returns “hit” • Memory structure broken into groups of “patterns” and associated “masks” • Masks used to “wildcard” some bits in the patterns • Useful for lookups where not all fields of lookup key (CEF, ACL lookups)

69

70

71

72

73

Power discovery allows switches and PoE capable devices to convey power information. LLDP-MED provides information related to how the device is powered (from the line, from a backup source, from external power source, etc.), power priority (how important is it that this device has power?), and how much power the device needs. NOTE: LLDP-MED just advertises device consumption and is not a negotiation protocol. A third party IEEE device to be able to use PoE+ power ( > 15.4W) needs IEEE 802.3at LLDP Power-via- MDI protocol. Cisco Discovery Protocol includes separate TLVs for power requested and power available, allowing the switch and the PoE capable device to negotiate the power used. Some Cisco IP phones can operate at multiple power settings, lowering their consumption when less power is available. Using CDP the PD requests the worst-case power (including the link loss) required. LLDP PD requests only the power required, the PSE has to add the link loss values. When a powered device connected to a PoE+ port restarts and sends a CDP or LLDP packet with a power TLV, the switch locks to the power-negotiation protocol of that first packet and does not respond to power requests from the other protocol. For example, if the switch is locked to CDP, it does not provide power to devices that send LLDP requests. If CDP is disabled after the switch has locked on it, the switch does not respond to LLDP power requests and can no longer power on any accessories. In this

74

case, you should restart the powered device.

74

In the PoE controllers, there are three separate current thresholds that are used for different purposes. These are I(cut), I(limit), and I(short). I(cut) is used as the threshold at which power is removed from the port if the PD draws more power than allocated. (ex. 15.4W) I(limit) is used as the threshold at which the PoE controller will start to reduce the port voltage in order to control the current, but is not used to remove power from the port completely. I(short) is used as the threshold at which the port sees a very fast current spike that must be dealt with immediately and bypasses all the timers that are used to remove power from the port and shuts down immediately. Imax error is reported by PoE controller of the switch, when a PoE PD device misbehaves and draws more power ( Port Current ) beyond theirs specified limit. Imax error is reported after the device is Powered up and it1s an operating fault. When Iport > Icut for a time period of Tovld (50-75 milliseconds).( Iport - port current, Icut - port cut-off current, Tovld - time duration for the overload condition to be reported). The typical values of Icut in a switch varies by their PoE controller components, but they are always within IEEE range. Tstart error is reported when the device violates Tinsrush (50 milliseconds), what that means, while powering up the device, the device draws a port inrush current which is greater than Iinrush (450 mA) for at least Tinrush (50 milliseconds). Tstart is a start up fault before even the Device reported Power Good. ( T-inrush a Port start up current monitor time, I-inrush a Start up current)

75

6/26/2013 Cisco Live 2013

The workaround is present in the following platforms only : 3750-E,3560-E, 3750-X, 3560-X, 3560-C, 2960S, 2960,2960C The other platforms do not support 2X power mode and the workaround would be to use a longer cable. See DDTS CSCsw18530

75

show platform frontend-controller subordinate <number> Displays the statistics of errors received as reported by a subordinate. In this command output check the state of the Subordinate and look for I2c errors. If the I2C errors are non-zero check if they are incrementing, if yes reload. If issue still exists it could be a bad hardware.

76

Debug commands should always be run with care. Specific debug conditions can be used where available A debug condition can help to only keep the debug condition x/x

77

6/26/2013 Cisco Live 2013

78

79

Check Major version

80

81

82

show switch stack-ring activity was introduced in 12.2(20)SE A new, out-of-the-box switch (one that has not joined a switch stack or has not been manually assigned a stack member number) ships with a default stack member number of 1. When it joins a switch stack, its default stack member number changes to the lowest available member number in the stack. Stack members in the same switch stack cannot have the same stack member number. Every stack member, including a standalone switch, retains its member number until you manually change the number or unless the number is already being used by another member in the stack. If you manually change the stack member number by using the “switch current-stack-member-number renumber new-stack-member-number” global configuration command, the new number goes into effect after that stack member resets (or after you use the “reload slot stack-member-number” privileged EXEC command) and only if that number is not already assigned to any other members in the stack. “show platform sf-asic stat ?” gives more detailed stack statistics for the 3750E

LED on the port with the corresponding switch number will illuminate

For ex, if the switch is # 4 in the stack, port 4’s LED will light up

83

to stop stack port flap Switch <> stack port <> en/disable show switch stack-ports summary was introduced in 12.2(50)SE “# Changes to LinkOK” = number of times stack port went into Link OK “cable length” in CentiMetres

84

85

86

87

88

Note: this mac matches slide with “show platfrom forward ...” command example

89

90

91

92

Use this command to view the egress interface for Layer2 forwarding. In this case egress is Gi1/0/4

93

----- Meeting Notes (4/11/13 15:58) ----- example 1 and 2

94

95

96

97

Notes: ios view of how things should be.

98

The switch does not make fwding decisions based on icmp values. But, the command requires them. Just put in anything in range 0-255

99

100

101

102

Some failure scenarios

103

104

105

Note: change date / time to use datetime, and not uptime. For consistency

106

107

109

6/26/2013 Cisco Live 2013

110

111

6/26/2013 Cisco Live 2013

112

113

114

The Catalyst 2960/3650/3750 supports four egress queues, which can be configured on a per-interface basis to operate in either 4Q3T or 1P3Q3T modes. Additionally, the Catalyst 2960/3650/3750 supports two queue-sets, allowing certain interfaces to be configured in one manner and others to be configured in a different manner. The Catalyst 2960/3650/3750 has Queue 1 (not Queue 4) as the optional priority queue; in a converged campus environment it is recommended to enable the priority queue via the priority-queue out interface command. The three remaining egress queues on the Catalyst 2960/3650/3750 are scheduled by a Shaped Round-Robin (SRR) algorithm, which can be configured to operate in shaped mode or in shared mode. In shaped mode, assigned bandwidth is limited to the defined amount; in shared mode, any unused bandwidth is shared among other classes (as needed). To make the queuing structure consistent with the previously discussed best-practice queuing principles:

Queues 2 through 4 should be set to operate in shared mode (which is the default mode of operation on Queues 2 through 4). The ratio of the shared weights determines the relative bandwidth allocations (the absolute values are meaningless). Since the PQ of the Catalyst 2960/3650/3750 is Q1 (not Q4 as in the Catalyst 3550), the entire queuing model can be flipped upside down, with Q2 representing the Critical Data queue, Q3 representing the Best Effort queue, and Q1 and Q4 representing the Scavenger queue. Therefore, shared weights of 70, 25, and 5 can be assigned to Queues 2, 3, and 4, respectively.

115

116

6/26/2013 Cisco Live 2013

117

Note: add X and S. C3560E(config)#diagnostic monitor test ? <1-6> Test ID Number ID Test Name [On-Demand Test Attributes] --- ------------------------------------------- 1 TestPortASICStackPortLoopback [B*N****] 2 TestPortASICLoopback [B*D*R**] 3 TestPortASICCam [B*D*R**] 4 TestPortASICRingLoopback [B*D*R**] 5 TestMicRingLoopback [B*D*R**] 6 TestPortASICMem [B*D*R**] --- ------------------------------------------- WORD Test ID list (e.g. 1,3-6) or Test Name all Select all test ID C3560E(config)# Scheduled Example:

switch(config)# diagnostic schedule switch 1 test 1 on jan 3 2003

23:32

switch(config)# diagnostic schedule switch 1 test 1 daily 14:45

switch(config)# diagnostic schedule switch 1 test all weekly Monday

3:33

switch(config)# diagnostic schedule switch 5 test 1,3-6 daily 23:55

Router# show run

Building configuration...

Current configuration : 4618 bytes

diagnostic schedule switch 1 test 1 on January 3 2003 23:32 cardindex

1

diagnostic schedule switch 1 test 2 daily 14:45 cardindex 1

diagnostic schedule switch 1 test all weekly Monday 3:33 cardindex 1

diagnostic schedule switch 5 test 1,3-6 daily 23:55 cardindex 1

117

118

119

The bottom two boxes are referencing diag test 2.

120

Here is an example of failure Overall diagnostic result: MAJOR ERROR Test results: (. = Pass, F = Fail, U = Untested) 1) TestPortAsicStackPortLoopback ---> . 2) TestPortAsicLoopback ------------> F 3) TestPortAsicCam -----------------> . 4) TestPortAsicRingLoopback --------> . 5) TestMicRingLoopback -------------> . 6) TestPortAsicMem ----------------->

121

Note: updated switch support for X and S To disable OBFL: no hw-module module [switch-number] logging onboard To clear all the OBFL data in he flash memory except for uptime and CLI commands information clear logging onboard -”show logging onboard status” to see if it is running and for which features -“summary” option contains historical data (compressed) -”continuous” option contains current data (more detailed) -”detail” option contains summary and continuous output -”copy logging onboard module flash:” creates a tar file in flash -”archive tar /xtract flash:” to extract tar file contents as ascii text

122

123

124

125

126

Assumes that Mcast routing or IGMP snooping is setup

127

128

6/26/2013 Cisco Live 2013