Cisco IOS XR Troubleshooting Guide for the Cisco …rabdoul.free.fr/SPADVROUTE12/MCAST/XR Troubleshooting...Contents v Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation

Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router Cisco IOS XR Software, Release 4.0 April, 2011

Americas HeadquartersCisco Systems, Inc.170 West Tasman DriveSan Jose, CA 95134-1706 USAhttp://www.cisco.comTel: 408 526-4000

800 553-NETS (6387)Fax: 408 527-0883

Text Part Number: OL-23591-02

http://www.cisco.com

THE SPECIFICATIONS AND INFORMATION REGARDING THE PRODUCTS IN THIS MANUAL ARE SUBJECT TO CHANGE WITHOUT NOTICE. ALL STATEMENTS, INFORMATION, AND RECOMMENDATIONS IN THIS MANUAL ARE BELIEVED TO BE ACCURATE BUT ARE PRESENTED WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED. USERS MUST TAKE FULL RESPONSIBILITY FOR THEIR APPLICATION OF ANY PRODUCTS.

THE SOFTWARE LICENSE AND LIMITED WARRANTY FOR THE ACCOMPANYING PRODUCT ARE SET FORTH IN THE INFORMATION PACKET THAT SHIPPED WITH THE PRODUCT AND ARE INCORPORATED HEREIN BY THIS REFERENCE. IF YOU ARE UNABLE TO LOCATE THE SOFTWARE LICENSE OR LIMITED WARRANTY, CONTACT YOUR CISCO REPRESENTATIVE FOR A COPY.

The Cisco implementation of TCP header compression is an adaptation of a program developed by the University of California, Berkeley (UCB) as part of UCB’s public domain version of the UNIX operating system. All rights reserved. Copyright © 1981, Regents of the University of California.

NOTWITHSTANDING ANY OTHER WARRANTY HEREIN, ALL DOCUMENT FILES AND SOFTWARE OF THESE SUPPLIERS ARE PROVIDED “AS IS” WITH ALL FAULTS. CISCO AND THE ABOVE-NAMED SUPPLIERS DISCLAIM ALL WARRANTIES, EXPRESSED OR IMPLIED, INCLUDING, WITHOUT LIMITATION, THOSE OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT OR ARISING FROM A COURSE OF DEALING, USAGE, OR TRADE PRACTICE.

IN NO EVENT SHALL CISCO OR ITS SUPPLIERS BE LIABLE FOR ANY INDIRECT, SPECIAL, CONSEQUENTIAL, OR INCIDENTAL DAMAGES, INCLUDING, WITHOUT LIMITATION, LOST PROFITS OR LOSS OR DAMAGE TO DATA ARISING OUT OF THE USE OR INABILITY TO USE THIS MANUAL, EVEN IF CISCO OR ITS SUPPLIERS HAVE BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.

Cisco and the Cisco Logo are trademarks of Cisco Systems, Inc. and/or its affiliates in the U.S. and other countries. A listing of Cisco's trademarks can be found at www.cisco.com/go/trademarks. Third party trademarks mentioned are the property of their respective owners. The use of the word partner does not imply a partnership relationship between Cisco and any other company. (1005R)

Any Internet Protocol (IP) addresses and phone numbers used in this document are not intended to be actual addresses and phone numbers. Any examples, command display output, network topology diagrams, and other figures included in the document are shown for illustrative purposes only. Any use of actual IP addresses or phone numbers in illustrative content is unintentional and coincidental.

Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router © 2010 Cisco Systems, Inc. All rights reserved.

http://www.cisco.com/go/trademarks

Cisco IOSOL-23591-02

C O N T E N T S

Preface xi

Changes to This Document xi

Obtaining Documentation and Submitting a Service Request xii

C H A P T E R 1 General Troubleshooting Procedures 1-1

Prerequisite Documentation for Troubleshooting 1-1

Verifying and Troubleshooting CLI Access 1-2

General CLI Access Information 1-2

User Access Privileges 1-2

Cisco-support Task ID 1-3

CLI Access Through a Console Port 1-3

CLI Access Through a Terminal Server 1-3

CLI Access Through the Management Ethernet Interface 1-4

Validating and Troubleshooting Installation of the Cisco IOS XR Software Package 1-7

Verifying the Software Version 1-8

Validating the Installation 1-10

Validating and Troubleshooting Cisco IOS XR Software Configuration 1-16

Local and Global Configurations 1-16

Collecting Configuration Information 1-19

Verifying the Running Configuration 1-20

Using the show configuration failed Command 1-24

Verifying the System 1-26

Troubleshooting the Backplane Ethernet Control System 1-41

Basic Cisco IOS XR Verification and Troubleshooting Commands 1-46

man Command 1-46

describe Command 1-49

show platform Command 1-49

top Command 1-50

show context Command 1-50

show users Command 1-52

show history Command 1-52

show configuration Command 1-53

Displaying ASIC Errors 1-54

Using Trace Commands 1-56

iii XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router

Contents

MIB Location 1-57

Gathering Information Before You Call Cisco TAC 1-58

Gathering Information about Crashes and Core Dumps 1-58

Capturing Logs 1-58

Using Debug Commands 1-59

Using Diagnostic Commands 1-59

Commands Used to Display Process and Thread Details 1-59

C H A P T E R 2 Verifying and Troubleshooting Interface Status 2-61

Verifying and Troubleshooting Gigabit Ethernet Interfaces 2-61

Verifying and Troubleshooting Pluggable Optical Line Card Interfaces 2-68

C H A P T E R 3 Troubleshooting Interface Connectivity 3-75

Troubleshooting Ping and ARP Connectivity 3-75

Troubleshooting Bidirectional Forwarding Detection 3-81

Using show and debug Commands 3-82

BFD Sessions in Down State 3-83

BFD Sessions Flap 3-83

BFD Sessions Down on Neighboring Router 3-85

BFD Sessions Are Not Created on the LC 3-85

Troubleshooting Ethernet CFM 3-85


MEPs Are Not Created 3-88

MIPs Are Not Created 3-88

No CCMs are Received at the MEP or Peer MEPs Are Not Seen 3-89

Peer MEP Defects and Mismatches Are Seen 3-90

Remote Defect Indication Received 3-91

Peer MEP Times Out But No Alarm Or Action Occurs 3-92

No Debugs or Counters for Higher-Level Packets at a MEP or MIP 3-92

Dropped CFM PDUs 3-92

CFM ping Or traceroute Returns a “not found” Error 3-93

AIS Messages Are Not Sent 3-93

C H A P T E R 4 Troubleshooting Packet Forwarding 4-95

Understanding IPv4 CEF 4-95

Troubleshooting IPv4 CEF 4-96

Troubleshooting Adjacency Information 4-101

Troubleshooting Transient Traffic Drop 4-106

ivCisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router

OL-23591-02

Contents

Troubleshooting Packet Drop in the Fabric 4-109

Troubleshooting Control Plane Information 4-109

C H A P T E R 5 Troubleshooting Bundles and Load Balancing 5-115

Troubleshooting Routing and CEF Issues Related to Bundles and Load Balancing 5-115

Verifying Routing Table Entries for Parallel Links 5-115

Verifying the CEF Database and Measuring Flows 5-117

Troubleshooting Problems with Link Bundles 5-118

Bundle Does Not Come Up 5-118

Bundle Member Not Distributing 5-119

Bundle Not Using MAC-Address From Backplane 5-119

Layer 3 Data Traffic Not Flowing 5-120

Ping Failed over Bundle 5-120

Layer 3 Packets Not Synching Over Bundle 5-121

Layer 2 Traffic Not Flowing 5-121

Bundle Statistics 5-122

Troubleshooting Layer 2 Bundles and Load Balancing 5-122

Verifying the Bundle Status, IGP Route, and CEF Database 5-122

Viewing the Expected Paths and Measuring the Flows 5-123

Troubleshooting Layer 3 Bundles and Load Balancing 5-124

C H A P T E R 6 Troubleshooting Layer 3 Connectivity 6-125


Traffic Loss 6-128

Packets Are Punted and Switched in Software 6-129

Traceroute Fails 6-130

Adding Routes Fails 6-131

Continuous Tracebacks 6-133

fib_mgr Does Not Come Up During LC Reload or After Multiple Process Restarts 6-134

CEF Entries Out of Sync 6-135

fib_mgr Crashes 6-136

Tracebacks Appearing 6-136

Traffic Loss Because of Changing encap on a Subinterface 6-137

Traffic Loss during RSP Failover 6-138

Troubleshooting Virtual Router Redundancy Protocol 6-138


VRRP Fails to Reach Active State 6-140

vCisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router

OL-23591-02

Contents

Tracked Interface Failing, Router State Not Changed 6-140

VRRP State Flapping 6-140

More Than One VRRP Router Active 6-141

VRRP Active Router Not Forwarding Traffic 6-141

Traffic Loss or Unexpected VRRP State After Interface shut/no shut 6-142

Additional Information On Routing Configuration Commands 6-142

C H A P T E R 7 Troubleshooting Router Switch Fabric and Data Path 7-143

Understanding Switch Fabric Architecture 7-143

Getting Started with Fabric Troubleshooting 7-145

Troubleshooting Packet Drops 7-146

Displaying Traffic Status in Line Cards and RSP Cards 7-147

Locating Packet Drops by Examining Counters 7-148

Locating Drops of Punted Packets 7-155

Packet Drop from LC to LC 7-157

Packet Drop Between RSP and LC 7-158

Packet Drop After Certain Actions 7-160

Packet Drop After a Redundancy Switchover 7-161

Packet Drop with Unknown Reason 7-163

Troubleshooting RSP and LC Crashes 7-165

Active RSP Is Crashing 7-165

Standby RSP Is Crashing 7-166

LC Is Crashing 7-167

Troubleshooting Complete Loss of Traffic 7-168

No Traffic from LC to LC 7-169

No Traffic Between RSP and LC 7-170

Gathering Fabric Information Before Calling TAC 7-172

C H A P T E R 8 Troubleshooting MPLS Services 8-173

Verifying MPLS PIE Activation and MPLS Configuration 8-173

Troubleshooting Connectivity Over MPLS 8-174


IP Packets Not Forwarded to LSP 8-175

IP Packets Not Forwarded to MPLS TE Tunnel 8-176

MPLS Packets Not Forwarded to MPLS TE Tunnel 8-176

MPLS TE Tunnels Do Not Come Up 8-176

FRR-Protected Tunnel Goes Down After Triggering FRR 8-177

MPLS TE FRR Database Not Built 8-178

viCisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router

OL-23591-02

Contents

MPLS FRR Switch Time Debugging 8-178

C H A P T E R 9 Troubleshooting L2VPN and Ethernet Services 9-181

Troubleshooting VLAN Traffic and L2 TCAM Classification 9-181

Understanding Problems with VLAN Traffic and L2 TCAM Classification 9-182

Verifying the Configuration Is Correct 9-182

Verifying Interfaces, Subinterfaces, and Packet Forwarding 9-183

Troubleshooting Multipoint Layer 2 Services 9-190

Basic Bridging: Example 9-190

Verifying MAC Address Updates 9-192

Troubleshooting Multipoint Layer 2 Bridging Services (VPLS) 9-195

Troubleshooting Bridge Domains That Use BGP-AD 9-201

Troubleshooting Point-to-Point Layer 2 Services 9-206

Example of Point-To-Point Layer 2 Deployment 9-206


AC Is Down 9-211

Pseudowire Is Down 9-212

VPWS Not Forwarding Traffic from AC to Pseudowire 9-212

Pseudowire Up but Ping Fails 9-213

Traffic Loss 9-213

Traffic Loss During RSP Fail Over 9-213

Preferred Path Not Working 9-214

Troubleshooting Specific Outage Scenarios In Layer 2 Services 9-214


L2VPN Discovery Not Working 9-217

AC Is Down 9-218

Pseudowire Is Down 9-219

VPLS Not Forwarding Flooding Traffic 9-220

VPLS Not Forwarding Flooding Traffic from AC to Pseudowire 9-224

VPLS Not Forwarding Flooding Traffic from Pseudowire to AC 9-224

VPLS Not Forwarding Unicast Traffic from AC to AC 9-225

VPLS Not Forwarding Unicast Traffic from AC to Pseudowire 9-225

VPLS Not Forwarding Flooding Traffic from Pseudowire to AC 9-225

Pseudowire Up but Ping Fails 9-226

Traffic Loss 9-226

Pseudowire Flap Causing Traffic Loss 9-226

Traffic Loss During RSP Fail Over 9-227

Preferred Path Not Working 9-227

Troubleshooting Dynamic Host Configuration Protocol Snooping 9-227

viiCisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router

OL-23591-02

Contents

Show Commands 9-228

Trace Commands 9-228

Syslog Commands 9-228

Tech-support Commands 9-229

Action Commands 9-229

L2VPN Commands 9-229

L2Snoop Commands 9-229

Interface Controller Commands 9-230

Troubleshooting Multiple Spanning Tree 9-230


MSTP Incorrectly or Inconsistently Formed 9-230

MSTP Correctly Formed, but Traffic Flooding 9-231

Packet Forwarding Does Not Match MSTP State 9-231

MSTAG Access Network Does Not Recognize MSTAG Node as Root 9-231

Traffic Not Switching Through MSTAG Node(s) 9-232

Additional References—Command Reference and Configuration Guides 9-232

C H A P T E R 10 Troubleshooting Quality of Service and Access Control Lists 10-233


Service-Policy Configuration Is Rejected 10-235

Packets are Incorrectly Classified 10-235

Packets in Wrong Queue 10-236

Packets Incorrectly Marked 10-236

Packets Incorrectly Policed 10-237

Shaping Incorrect 10-237

Weighted Random Early Detection Incorrect 10-237

Bandwidth Not Guaranteed 10-238

Bandwidth Ratio Not Working 10-238

Non-zero Queue(conform) and Queue(exceed) Counters In show policy-map Commands 10-239

Unable to Modify or Delete policy-map or class-map 10-240

Unable to Modify or Delete class-map ACL 10-240

Unable to Delete service-policy 10-240

After QoS EA Restarts, show policy-map interface Fails 10-240

After QoS EA Restarts, service-policy config Fails 10-241

show policy-map interface Output Error 10-241

Bundle Members Not Configured with service-policy 10-241

Troubleshooting Access Control Lists 10-241

viiiCisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router

OL-23591-02

Contents


ACL Messages Not Appearing 10-243

Fragmented Packets Being Accepted 10-243

Egress Counter Incorrect or Not Working 10-244

ACL Interface Bind Rejected 10-244

Single ACE Using Many TCAMs 10-244

ACL Using Varying TCAM Space 10-245

ACL Logs Not Working for Ethernet Services 10-245

Ethernet Services ACL Bind on Interface Rejected 10-245

Changing ACL Exhausts TCAM 10-245

Cannot Delete ACL 10-246

DF Bit Not Supported 10-246

Max ACL Limit Reached 10-246

Unsupported Combinations in ACL 10-246

No Statistics Counters 10-246

TCAMs Out of Resources 10-246

C H A P T E R 11 Troubleshooting Multicast Services 11-247

Troubleshooting IGMP Snooping (Layer 2 Multicast) 11-247

Using show Commands 11-247

Using the debug, trace, and show tech-support Commands 11-249

Troubleshooting Missing Routes and Forwarding Errors 11-250

Troubleshooting Native Multicast Routing (Layer 3) 11-256


Multicast PIE Installation Fails 11-262

Multicast CLI Unavailable Although PIE Is Installed 11-263

“This command not authorized” Error Message 11-263

Dynamic IGMP Failure 11-263

Traffic Fails on Some Interfaces 11-267

Traffic Fails on Some Interfaces—MGID 11-268

Throughput Loss at Receiver Interfaces 11-268

Reverse Path Forwarding IP Address Problems 11-268

I N D E X

ixCisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router

OL-23591-02

Contents

xCisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router

OL-23591-02

Preface

This guide describes how to troubleshooting a router using the Cisco IOS XR software.

This preface contains the following sections:

• Changes to This Document, page xi

• Obtaining Documentation and Submitting a Service Request, page xii

Changes to This DocumentTable 1 lists the technical changes made to this document since it was first printed.

Table 1 Changes to This Document

Revision Date Change Summary

OL-23591-02 April, 2011 Added details for the following topics:

• Chapter 1, “General Troubleshooting Procedures”—Added information on prerequisite documentation for troubleshooting, Cisco-support task ID, show tech-support command, displaying ASIC errors, gathering logs and system information. Modified the information on diagnostics.

• Divided interface troubleshooting into two separate chapters—Chapter 2, “Verifying and Troubleshooting Interface Status”and Chapter 3, “Troubleshooting Interface Connectivity.”

• Chapter 3, “Troubleshooting Interface Connectivity”—Added information on connectivity fault management (CFM).

• Chapter 6, “Troubleshooting Layer 3 Connectivity”—Added information on CEF and interface accounting.

• Chapter 7, “Troubleshooting Router Switch Fabric and Data Path”—Added information on NP counters.

• Chapter 8, “Troubleshooting MPLS Services”—Corrected syntax of several commands.

continued ...

xiCisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router

OL-23591-02

Preface

Obtaining Documentation and Submitting a Service RequestFor information on obtaining documentation, submitting a service request, and gathering additional information, see the monthly What’s New in Cisco Product Documentation, which also lists all new and revised Cisco technical documentation, at:

http://www.cisco.com/en/US/docs/general/whatsnew/whatsnew.html

Subscribe to the What’s New in Cisco Product Documentation as a Really Simple Syndication (RSS) feed and set content to be delivered directly to your desktop using a reader application. The RSS feeds are a free service and Cisco currently supports RSS version 2.0.

OL-23591-02 April, 2011 ... continued

• Chapter 9, “Troubleshooting L2VPN and Ethernet Services”—Enhanced information about VLAN verification, included sample VLAN and PW configurations, added a section on Verifying MAC Address Updates, enhanced information about multipoint Layer 2 services (VPLS), moved information on DHCP snooping to this chapter, enhanced information on MST access gateways (MSTAGs). Reorganized the chapter for ease of use.

• Chapter 10, “Troubleshooting Quality of Service and Access Control Lists”—Added information on queue conform and queue exceed counters displayed by the policy-map command.

• Chapter 11, “Troubleshooting Multicast Services”—Reorganized this chapter to highlight IGMP snooping (Layer 2 MC) and native MC (Layer 3), and added information to each of these sections.

OL-23591-01 November 2010

(Initial release of this document as a multichapter book.) The content was reorganized for usability and updated to reflect Release 3.9 and 4.0 features.

OL-20794-01 December 2009

Initial release of this document as a single module.

Table 1 Changes to This Document (continued)

Revision Date Change Summary

xiiCisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router

OL-23591-02

http://www.cisco.com/en/US/docs/general/whatsnew/whatsnew.html

Cisco IOS XR Troubleshooting Guide fOL-23591-02

C H A P T E R 1
General Troubleshooting Procedures
This chapter describes general troubleshooting techniques you can use to troubleshoot the Cisco ASR 9000 Aggregation Series Router. It includes the following sections:

• Prerequisite Documentation for Troubleshooting, page 1-1

• Verifying and Troubleshooting CLI Access, page 1-2

• Validating and Troubleshooting Installation of the Cisco IOS XR Software Package, page 1-7

• Validating and Troubleshooting Cisco IOS XR Software Configuration, page 1-16

• Verifying the System, page 1-26

• Troubleshooting the Backplane Ethernet Control System, page 1-41

• Basic Cisco IOS XR Verification and Troubleshooting Commands, page 1-46

• Displaying ASIC Errors, page 1-54

• Using Trace Commands, page 1-56

• MIB Location, page 1-57

• Gathering Information Before You Call Cisco TAC, page 1-58

Prerequisite Documentation for TroubleshootingAs a starting point for troubleshooting, we strongly recommend that you have a system of maintaining and accessing detailed information about your network and ASR 9000 router. This should include:

• Current documentation about the system, including chassis numbers, serial numbers, installed cards, and location of chassis details.

• Diagrams illustrating the connectivity of the router control plane Ethernet network.

• Detailed documentation about the network, including the following:

– Up-to-date internetwork map that outlines the physical location of all the devices on the network and how they are connected, as well as a logical map of interfaces, network addresses, network numbers, subnetworks, and so on

– List of all network protocols implemented in your network; and for each of the protocols implemented, a list of the network numbers, subnetworks, zones, areas, and so on that are associated with them

– All points of contact to external networks

– Routing protocol for each external network connection

1-1or the Cisco ASR 9000 Aggregation Services Router

Chapter 1 General Troubleshooting Procedures Verifying and Troubleshooting CLI Access

– Established baseline for your network, that is, the normal network behavior and performance at different times of the day so that you can compare any problems with a baseline

– Name of the device that is the spanning tree root bridge for the system control plane Ethernet network

• Captured output of all commands

Verifying and Troubleshooting CLI AccessEnsure that the system has been booted. If the system has not booted, see Cisco IOS XR Getting Started Guide for the Cisco ASR 9000 Aggregation Services Router for information on booting a router running Cisco IOS XR software. The following CLI access troubleshooting information is provided:

• General CLI Access Information, page 1-2

• User Access Privileges, page 1-2

• Cisco-support Task ID, page 1-3

• CLI Access Through a Console Port, page 1-3

• CLI Access Through a Terminal Server, page 1-3

• CLI Access Through the Management Ethernet Interface, page 1-4

General CLI Access InformationThe following CLI access information applies to a console port, terminal server, and management Ethernet interface connections.

Once the terminal emulation software is started and you press Enter, a router prompt should appear. If no prompt appears, verify the physical connection to the console port and press Enter again. If the prompt still does not appear, contact Cisco Technical Support. See the “Obtaining Documentation and Submitting a Service Request” section on page xii for Cisco Technical Support contact information.

If a prompt appears, indicating that the CLI is accessible, but your login username and password are invalid, you are prevented from accessing the router. Verify that you have the correct username and password. If you have the correct username and password, but are locked out of the router, you may need to perform password recovery to access the system again. See Cisco ASR 9000 Aggregation Services Router ROM Monitor Guide for password recovery procedures.

User Access PrivilegesWhen you log on to the router, use a username that is associated with a valid user group that has the authorization to execute the required commands. If you suspect user group assignment is preventing you from using a command, contact your AAA administrator for assistance.

See Cisco ASR 9000 Aggregation Services Router System Security Command Reference and Cisco ASR 9000 Aggregation Services Router System Security Configuration Guide for information on users, usernames, and user groups.

1-2Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router

OL-23591-02


Cisco-support Task IDMany of the troubleshooting commands can be performed only by users who are assigned to a user group that includes the cisco-support task ID. Users without the cisco-support task ID receive a “This command is not authorized” response if they attempt to use those commands. The cisco-support commands are normally reserved for use by Cisco Technical Support personnel, because there is some risk that they may cause performance or other issues.

Caution These Cisco support commands are normally reserved for use by Cisco Technical Support personnel only. There is some risk that they may cause performance or other issues that impact products without proper usage, and we highly recommend that you contact Cisco Technical Support prior to using any of these commands. See the “Obtaining Documentation and Submitting a Service Request” section on page xii for information on contacting Cisco TAC.

CLI Access Through a Console PortThe first time a router is started, you must use a direct connection to the console port to connect to the router and enter the initial configuration. See Cisco ASR 9000 Aggregation Services Router Router Getting Started Guide for information on connecting to the router through a console port. When you use a direct connection to the Console port, CLI commands are entered at a terminal or at a computer running terminal emulation software. A direct Console port connection is useful for entering initial configurations and performing some debugging tasks.

CLI Access Through a Terminal ServerA terminal server connection provides a way to access the Console port from a remote location. A terminal server connection is used when you need to perform tasks that require Console port access from a remote location.

Connecting to a router through a terminal server is similar to directly connecting through the Console port. For both connection types, the physical connection takes place through the Console port. The difference is that the terminal server connects directly to the Console port, and you must use a Telnet session to establish communications through the terminal server to the router.

If you are unable to access the CLI through a terminal server, perform the following procedure.

Step 1 Disable flow control (XON/XOFF) on the Terminal Server.

Step 2 Disable local echo mode on the Terminal Server.

Step 3 Verify the router name configured using the hostname command.

Step 4 Check whether the port address is configured correctly.

Step 5 Verify whether the address (interface) used for the reverse Telnet is up/up. The output of the show interfaces brief command provides this information. Cisco recommends you to use loopbacks because they are always up.

Step 6 Ensure that you have the correct type of cabling. For example, you must not use a crossover cable to extend the length.

Step 7 Establish a Telnet connection to the IP address port to test direct connectivity. You must Telnet from both an external device and the terminal server. For example, telnet 172.21.1.1 2003.


OL-23591-02


Step 8 Ensure that you have the transport input telnet command under the line for the target device. The target device is the device that is connected to the terminal server.

Step 9 Use a PC/dumb terminal to connect directly to the console of the target router. The target router is the device connected to the terminal server. This step helps you identify the presence of a port issue.

Step 10 If you are disconnected, check timeouts. You can remove or adjust timeouts.

Note If you encounter authentication failures, remember that the terminal server performs the first authentication (if configured), while the device to which you try to connect performs the second authentication (if configured). Verify whether AAA is configured correctly on both the terminal server and the connecting device.

Step 11 Contact Cisco Technical Support. See the “Obtaining Documentation and Submitting a Service Request” section on page xii for Cisco Technical Support contact information.

CLI Access Through the Management Ethernet InterfaceThe Management Ethernet interface allows you to manage the router using a network connection. Before you can use the Management Ethernet interface, the interface must be configured. See Cisco ASR 9000 Aggregation Services Router Router Getting Started Guide for information on configuring the interface.

Once configured, the network connection takes place between client software on a workstation computer and a server process within the router. The type of client software you use depends on the server process you use. See Cisco ASR 9000 Aggregation Services Router Router Getting Started Guide for information on the client and server services supported by the Cisco IOS XR software.

If you are unable to access the CLI through a management Ethernet interface, perform the following procedure.

SUMMARY STEPS

1. show interface MgmtEth interface-instance

2. show arp MgmtEth interface-instance

3. show ipv4 interface type instance

4. ping

5. Contact Cisco Technical Support if the problem is not resolved


OL-23591-02


DETAILED STEPS

Command or Action Purpose

Step 1 show interfaces MgmtEth interface-instance

Example:RP/0/RSP0/CPU0:router# show interfaces MgmtEth 0/RSP0/CPU0/0

Displays statistics for all interfaces configured on the router.

Check the following:

• MgmtEth interface is up

• Line protocol (state of the Layer 2 line protocol) is up

• Number of input and output errors

If an interface is administratively down, use the no shutdown command to enable the interface.

If an interface is down (operationally down), input or output errors are not within an acceptable range, the management Ethernet interface is not enabled when the no shutdown command is used, or the line protocol is down, see Chapter 2, “Verifying and Troubleshooting Interface Status,” for detailed information on troubleshooting interfaces.

If the interface is up and the input and output errors are within an acceptable range, proceed to Step 2.

Step 2 show arp MgmtEth interface-instance

Example:RP/0/RSP0/CPU0:router# show arp MgmtEth 0/RSP0/CPU0/0

Displays the Address Resolution Protocol (ARP) table for the management Ethernet interface.

Ensure that the expected ARP entries exist for the management Ethernet interface.

If the expected ARP entries exist, proceed to Step 3.

If the expected ARP entries do not exist, verify the physical layer Ethernet interface connectivity. Use the show arp trace command to display the ARP entries in the buffer. See the Chapter 2, “Verifying and Troubleshooting Interface Status,” for more information on troubleshooting interfaces.

Step 3 show ipv4 interface type instance

Example:RP/0/RSP0/CPU0:router# show ipv4 interface MgmtEth 0/RSP0/CPU0/0

Displays the usability status of interfaces configured for IPv4.

If the interface is in the expected state, proceed to Step 4.

If the status of the interface is not as expected, see Chapter 2, “Verifying and Troubleshooting Interface Status,” for more information on troubleshooting interfaces.


OL-23591-02


Examples

The output from the show interfaces MgmtEth command displays the status of the management Ethernet interface. In the following example, the management Ethernet interface is up, and there are 20 input errors and 8 output errors.

RP/0/RSP0/CPU0:router# show interface MgmtEth 0/RSP0/CPU0/0 Tue Sep 14 14:21:07.496 DSTMgmtEth0/RSP0/CPU0/0 is up, line protocol is up Interface state transitions: 1 Hardware is Management Ethernet, address is 001b.53ff.4a62 (bia 001b.53ff.4a62) Description: Connected to Lab LAN Internet address is 172.29.52.137/24 MTU 1514 bytes, BW 100000 Kbit (Max: 100000 Kbit) reliability 73/255, txload 0/255, rxload 0/255 Encapsulation ARPA, Half-duplex, 100Mb/s, THD, link type is autonegotiation output flow control is off, input flow control is off loopback not set, ARP type ARPA, ARP timeout 04:00:00 Last input 00:00:00, output 00:00:00 Last clearing of "show interface" counters never 5 minute input rate 2000 bits/sec, 3 packets/sec 5 minute output rate 0 bits/sec, 0 packets/sec 373082 packets input, 51028824 bytes, 239105 total input drops 62028 drops for unrecognized upper-level protocol Received 2601 broadcast packets, 194653 multicast packets 10 runts, 0 giants, 0 throttles, 0 parity 20 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort 45232 packets output, 3042775 bytes, 0 total output drops Output 24 broadcast packets, 0 multicast packets 8 output errors, 0 underruns, 0 applique, 0 resets 0 output buffer failures, 0 output buffers swapped out 1 carrier transitions

The output from the show arp MgmtEth 0/RSP0/CPU0/0 command displays the ARP table for the management Ethernet interface. Use the output from this command to verify that there are dynamic ARP addresses in the table and that ARP is functioning over the interface. The output shows that ARP is functioning over the management Ethernet interface 0/RSP0/CPU0/0.

RP/0/RSP0/CPU0:router# show arp MgmtEth 0/RSP0/CPU0/0 Tue Sep 14 14:24:03.962 DST

-------------------------------------------------------------------------------0/RSP0/CPU0-------------------------------------------------------------------------------Address Age Hardware Addr State Type Interface

Step 4 ping

Example:RP/0/RSP0/CPU0:router# ping

Checks host reachability and network connectivity on the IP network.

Note Enter a specific IP address or follow the prompts to send the ping message to the target address.

If no problems are detected, proceed to Step 5.

Step 5 Contact Cisco Technical Support. If the problem is not resolved, contact Cisco Technical Support. For Cisco Technical Support contact information, see the “Obtaining Documentation and Submitting a Service Request” section on page xii.



OL-23591-02

Chapter 1 General Troubleshooting Procedures Validating and Troubleshooting Installation of the Cisco IOS XR Software Package

172.29.52.1 01:44:00 0000.0c07.ac01 Dynamic ARPA MgmtEth0/RSP0/CPU0/0172.29.52.13 01:16:59 0010.79e9.6038 Dynamic ARPA MgmtEth0/RSP0/CPU0/0172.29.52.21 01:40:25 0022.0d5a.a6c4 Dynamic ARPA MgmtEth0/RSP0/CPU0/0172.29.52.27 02:18:16 0012.7fd6.ba08 Dynamic ARPA MgmtEth0/RSP0/CPU0/0172.29.52.28 02:05:29 0012.7fd6.ba09 Dynamic ARPA MgmtEth0/RSP0/CPU0/0172.29.52.32 01:42:16 0022.0d26.3bc5 Dynamic ARPA MgmtEth0/RSP0/CPU0/0172.29.52.36 02:39:34 0026.527c.5341 Dynamic ARPA MgmtEth0/RSP0/CPU0/0172.29.52.46 01:36:50 0012.7fd6.b9aa Dynamic ARPA MgmtEth0/RSP0/CPU0/0172.29.52.47 01:36:39 0012.7fd6.b9ab Dynamic ARPA MgmtEth0/RSP0/CPU0/0172.29.52.60 01:35:20 0003.a099.8000 Dynamic ARPA MgmtEth0/RSP0/CPU0/0172.29.52.69 00:00:00 001b.7852.4bd1 Dynamic ARPA MgmtEth0/RSP0/CPU0/0172.29.52.70 01:23:38 0011.93ef.e8e6 Dynamic ARPA MgmtEth0/RSP0/CPU0/0172.29.52.71 02:00:47 0011.93ef.e8fe Dynamic ARPA MgmtEth0/RSP0/CPU0/0172.29.52.75 01:44:59 5a59.0000.0202 Dynamic ARPA MgmtEth0/RSP0/CPU0/0172.29.52.76 01:41:10 0011.93ef.e8ea Dynamic ARPA MgmtEth0/RSP0/CPU0/0172.29.52.81 00:15:35 001a.6c40.d89c Dynamic ARPA MgmtEth0/RSP0/CPU0/0172.29.52.83 00:21:05 001a.6c40.d89c Dynamic ARPA MgmtEth0/RSP0/CPU0/0172.29.52.127 01:43:38 0013.c4cb.a200 Dynamic ARPA MgmtEth0/RSP0/CPU0/0172.29.52.134 01:15:53 001f.6c26.7fc0 Dynamic ARPA MgmtEth0/RSP0/CPU0/0172.29.52.135 01:01:46 001f.6c25.c480 Dynamic ARPA MgmtEth0/RSP0/CPU0/0172.29.52.136 00:43:39 0022.5560.8840 Dynamic ARPA MgmtEth0/RSP0/CPU0/0172.29.52.137 - 001b.53ff.4a62 Interface ARPA MgmtEth0/RSP0/CPU0/0172.29.52.138 - 001b.53ff.4a62 Interface ARPA MgmtEth0/RSP0/CPU0/0172.29.52.161 01:32:12 0019.aaa3.3d48 Dynamic ARPA MgmtEth0/RSP0/CPU0/0172.29.52.171 00:16:12 001c.5838.5b28 Dynamic ARPA MgmtEth0/RSP0/CPU0/0172.29.52.172 00:17:47 001c.5838.5b29 Dynamic ARPA MgmtEth0/RSP0/CPU0/0172.29.52.173 01:57:04 0015.c75f.09f8 Dynamic ARPA MgmtEth0/RSP0/CPU0/0172.29.52.180 01:26:50 0015.c75f.0800 Dynamic ARPA MgmtEth0/RSP0/CPU0/0172.29.52.217 01:16:21 0019.aaa3.b5ff Dynamic ARPA MgmtEth0/RSP0/CPU0/0172.29.52.226 01:28:32 0010.f60e.8400 Dynamic ARPA MgmtEth0/RSP0/CPU0/0172.29.52.243 01:48:25 001e.79c1.e0c1 Dynamic ARPA MgmtEth0/RSP0/CPU0/0

The ping command checks to see if the neighbor is reachable.

RP/0/RSP0/CPU0:router# ping 172.16.52.28 count 10 Tue Sep 14 14:36:52.441 DSTType escape sequence to abort.Sending 10, 100-byte ICMP Echos to 172.16.52.28, timeout is 2 seconds:!!!!!!!!!!Success rate is 100 percent (10/10), round-trip min/avg/max = 1/1/2 ms

Validating and Troubleshooting Installation of the Cisco IOS XR Software Package

The Cisco IOS XR software is divided into software packages allowing you to select which features run on your router. Each package contains the components to perform a specific set of router functions, such as routing, security, or Modular Services Card (MSC) support. Bundles are groups of packages that can be downloaded as a set. For example, the Unicast Routing Core Bundle provides six packages for use on every router.

This section provides information on how to validate and troubleshoot the Cisco IOS XR software package installation. The following sections are provided:

• Verifying the Software Version, page 1-8

• Validating the Installation, page 1-10


OL-23591-02


Verifying the Software VersionTo verify the Cisco IOS XR software version, perform the following procedure.

SUMMARY STEPS

1. show version

2. show install

DETAILED STEPS

The following example shows that the Cisco IOS XR software and active packages are version 4.0.0.

RP/0/RSP0/CPU0:router# show version

Cisco IOS XR Software, Version 4.0.0Copyright (c) 2010 by cisco Systems, Inc.

ROM: System Bootstrap, Version 1.04(20100216:021454) [ASR9K ROMMON],

router uptime is 1 day, 18 hours, 34 minutesSystem image file is "bootflash:disk0/asr9k-os-mbi-4.0.0/mbiasr9k-rp.vm"

cisco ASR9K Series (MPC8641D) processor with 4194304K bytes of memory.MPC8641D processor at 1333MHz, Revision 2.2

2 Management Ethernet


Step 1 show version

Example:RP/0/RSP0/CPU0:router# show version

Displays a variety of system information, including hardware and software version, router uptime, boot settings (configuration register), and active software.

Determine if all expected packages are installed and the current software versions are the expected versions.

If the expected packages are not installed or are not the expected version, install the correct package. See Cisco ASR 9000 Aggregation Series Router Getting Started Guide for information on installing and upgrading Cisco IOS XR software packages.

Step 2 show install

Example:RP/0/RSP0/CPU0:router# show install

Displays a list of all installed and active packages on each node.

Determine if the expected packages are installed on each node.

If the software or active package versions are not as expected for a node, the package is not compatible with the node for which it is being activated, or the package being activated is not compatible with the current active software set, install the correct software or package on the node. See Cisco ASR 9000 Aggregation Series Router Getting Started Guide for information on installing and upgrading Cisco IOS XR software packages.


OL-23591-02


12 DWDM controller(s)12 TenGigE40 GigabitEthernet2 SONET/SDH2 Packet over SONET/SDH219k bytes of non-volatile configuration memory.975M bytes of compact flash card.33994M bytes of hard disk.1605616k bytes of disk0: (Sector size 512 bytes).1605616k bytes of disk1: (Sector size 512 bytes).

Configuration register on node 0/RSP0/CPU0 is 0x0Boot device on node 0/RSP0/CPU0 is disk0:Package active on node 0/RSP0/CPU0:asr9k-optics-supp, V 4.0.0[DT_IMAGE], Cisco Systems, at disk0:asr9k-optics-supp-4.0.0 Built on Wed Sep 8 16:17:30 DST 2010 By sjc5-gf-021 in /auto/ioxbuild8/production/4.0.0.DT_IMAGE/asr9k/workspace for pie

asr9k-fwding, V 4.0.0[DT_IMAGE], Cisco Systems, at disk0:asr9k-fwding-4.0.0 Built on Wed Sep 8 16:12:40 DST 2010 By sjc5-gf-021 in /auto/ioxbuild8/production/4.0.0.DT_IMAGE/asr9k/workspace for pie

asr9k-cpp, V 4.0.0[DT_IMAGE], Cisco Systems, at disk0:asr9k-cpp-4.0.0 Built on Wed Sep 8 16:13:28 DST 2010 By sjc5-gf-021 in /auto/ioxbuild8/production/4.0.0.DT_IMAGE/asr9k/workspace for pie

asr9K-doc-supp, V 4.0.0[DT_IMAGE], Cisco Systems, at disk0:asr9K-doc-supp-4.0.0 Built on Wed Sep 8 16:16:57 DST 2010 By sjc5-gf-021 in /auto/ioxbuild8/production/4.0.0.DT_IMAGE/asr9k/workspace for pie

asr9k-scfclient, V 4.0.0[DT_IMAGE], Cisco Systems, at disk0:asr9k-scfclient-4.0.0 Built on Wed Sep 8 16:13:26 DST 2010 --More-- ...

The following example shows that the Cisco IOS XR software and active packages are version 4.0.0. If there is an expected package missing or an active package is not an expected package, install and activate the missing package or upgrade the unexpected package to the appropriate package. See Cisco ASR 9000 Aggregation Services Router Router Getting Started Guide for details on installing, activating, and upgrading software packages.

RP/0/RSP0/CPU0:router# show install

Node 0/RSP0/CPU0 [RP] [SDR: Owner] Boot Device: disk0: Boot Image: /disk0/asr9k-os-mbi-4.0.0/mbiasr9k-rp.vm Active Packages: disk0:asr9k-mini-p-4.0.0 disk0:asr9k-optic-4.0.0 disk0:asr9k-doc-p-4.0.0 disk0:asr9k-k9sec-p-4.0.0 disk0:asr9k-video-p-4.0.0 disk0:asr9k-mpls-p-4.0.0 disk0:asr9k-mgbl-p-4.0.0 disk0:asr9k-mcast-p-4.0.0

Node 0/1/CPU0 [LC] [SDR: Owner] Boot Device: mem: Boot Image: /disk0/asr9k-os-mbi-4.0.0/lc/mbiasr9k-lc.vm Active Packages: disk0:asr9k-mini-p-4.0.0


OL-23591-02


disk0:asr9k-optic-4.0.0 disk0:asr9k-video-p-4.0.0 disk0:asr9k-mpls-p-4.0.0 disk0:asr9k-mcast-p-4.0.0

Node 0/2/CPU0 [LC] [SDR: Owner] Boot Device: mem: Boot Image: /disk0/asr9k-os-mbi-4.0.0/lc/mbiasr9k-lc.vm Active Packages: disk0:asr9k-mini-p-4.0.0 disk0:asr9k-optic-4.0.0 disk0:asr9k-video-p-4.0.0 disk0:asr9k-mpls-p-4.0.0 disk0:asr9k-mcast-p-4.0.0



Validating the InstallationValidate the Cisco IOS XR software package installation to ensure the packages were installed correctly. The following commands are used to validate the currently installed software packages:

• install verify Command, page 1-10

• show install active Command, page 1-12

• show install committed Command, page 1-14

install verify Command

Use the install verify command to verify the consistency of a previously installed software set with the package file from which it originated.

This command can be used as a debugging tool to verify the validity of the files that constitute the packages to determine if there are any corrupted files. The command is also used to check that the install infrastructure is up and running and to determine if all files are expected. If there are corrupted files, see Cisco ASR 9000 Aggregation Services Router Router Getting Started Guide for information on deactivating and removing software packages and adding and activating software packages.


OL-23591-02


Note The install verify command can take up to two minutes per package to process.

Note The install verify command ignores secure domain router (SDR) boundaries and performs the operation in global scope.

The following example shows the output of the install verify command. The output is used to verify the consistency of a previously installed software set with the package file from which it originated.

RP/0/RSP0/CPU0:router(admin)# install verify

Sat Sep 25 08:18:14.077 DSTInstall operation 3 '(admin) install verify packages' started by user_A'dwolman-r' via CLI at 08:18:14 DST Sat Sep 25 2010.The install operation will continue asynchronously.RP/0/RSP0/CPU0:router(admin)#Info: This operation can take up to 2 minutes per package being verified.Info: Please be patient.Info: 0/0/CPU0 [LC] [SDR: Owner]Info: meta-data: [SUCCESS] Verification Successful.Info: /install/asr9k-optics-supp-4.0.0: [SUCCESS] VerificationInfo: Successful.Info: /install/asr9k-fwding-4.0.0: [SUCCESS] VerificationInfo: Successful.Info: /install/asr9k-cpp-4.0.0: [SUCCESS] Verification Successful.Info: /install/asr9k-scfclient-4.0.0: [SUCCESS] VerificationInfo: Successful.Info: /install/iosxr-video-adv-4.0.0: [SUCCESS] VerificationInfo: Successful.Info: /install/iosxr-mpls-4.0.0: [SUCCESS] Verification Successful.Info: /install/iosxr-mcast-4.0.0: [SUCCESS] VerificationInfo: Successful.Info: /install/iosxr-routing-4.0.0: [SUCCESS] VerificationInfo: Successful.Info: /install/iosxr-infra-4.0.0: [SUCCESS] VerificationInfo: Successful.Info: /install/iosxr-fwding-4.0.0: [SUCCESS] VerificationInfo: Successful.Info: /install/iosxr-diags-4.0.0: [SUCCESS] VerificationInfo: Successful.Info: /install/asr9k-adv-video-supp-4.0.0: [SUCCESS] VerificationInfo: Successful.Info: /install/asr9k-diags-supp-4.0.0: [SUCCESS] VerificationInfo: Successful.Info: /install/asr9k-mcast-supp-4.0.0: [SUCCESS] VerificationInfo: Successful.Info: /install/asr9k-base-4.0.0: [SUCCESS] Verification Successful.Info: 0/6/CPU0 [LC] [SDR: Owner]Info: meta-data: [SUCCESS] Verification Successful.Info: /install/asr9k-optics-supp-4.0.0: [SUCCESS] VerificationInfo: Successful.Info: /install/asr9k-fwding-4.0.0: [SUCCESS] VerificationInfo: Successful.Info: /install/asr9k-cpp-4.0.0: [SUCCESS] Verification Successful.Info: /install/asr9k-scfclient-4.0.0: [SUCCESS] VerificationInfo: Successful.Info: /install/iosxr-video-adv-4.0.0: [SUCCESS] VerificationInfo: Successful.Info: /install/iosxr-mpls-4.0.0: [SUCCESS] Verification Successful.Info: /install/iosxr-mcast-4.0.0: [SUCCESS] Verification


OL-23591-02


Info: Successful.Info: /install/iosxr-routing-4.0.0: [SUCCESS] VerificationInfo: Successful.Info: /install/iosxr-infra-4.0.0: [SUCCESS] VerificationInfo: Successful.Info: /install/iosxr-fwding-4.0.0: [SUCCESS] VerificationInfo: Successful.Info: /install/iosxr-diags-4.0.0: [SUCCESS] VerificationInfo: Successful.Info: /install/asr9k-adv-video-supp-4.0.0: [SUCCESS] VerificationInfo: Successful.Info: /install/asr9k-diags-supp-4.0.0: [SUCCESS] VerificationInfo: Successful.Info: /install/asr9k-mcast-supp-4.0.0: [SUCCESS] VerificationInfo: Successful.Info: /install/asr9k-base-4.0.0: [SUCCESS] Verification Successful.Info: 0/5/CPU0 [LC] [SDR: Owner]...Info: Verification Summary:Info: 0/0/CPU0: SUCCESSFUL. No anomalies found.Info: 0/6/CPU0: SUCCESSFUL. No anomalies found.Info: 0/5/CPU0: SUCCESSFUL. No anomalies found.Info: 0/7/CPU0: SUCCESSFUL. No anomalies found.Info: 0/1/CPU0: SUCCESSFUL. No anomalies found.Info: 0/4/CPU0: SUCCESSFUL. No anomalies found.Info: 0/2/CPU0: SUCCESSFUL. No anomalies found.Info: 0/RSP0/CPU0: SUCCESSFUL. No anomalies found.Info: The system needs no repair.Install operation 3 completed successfully at 08:19:48 DST Sat Sep 25 2010.

show install active Command

Use the show install active command to display active software packages. Verify that the command output matches the output of the show install committed command. If the output does not match, when you reload the router, the software displayed in the show install committed command output is the software that will be loaded. For example, the following output shows two different software package versions, one is the active version and the other is the committed version, so when the router reloads, the 3.9.1 version will be loaded even though 4.0.0 is the currently active version on 0/RSP0/CPU0.

RP/0/RSP0/CPU0:router(admin)# show install active location 0/RSP0/cpu0

Node 0/RSP0/CPU0 [RP] [SDR: Owner] Boot Device: disk0: Boot Image: /disk0/asr9k-os-mbi-4.0.0/mbiasr9k-rp.vm << 4.0.0 is active, not committed Active Packages: disk0:asr9k-mini-p-4.0.0 disk0:asr9k-optic-4.0.0 disk0:asr9k-doc-p-4.0.0 disk0:asr9k-k9sec-p-4.0.0 disk0:asr9k-video-p-4.0.0 disk0:asr9k-mpls-p-4.0.0 disk0:asr9k-mgbl-p-4.0.0 disk0:asr9k-mcast-p-4.0.0

RP/0/RSP0/CPU0:router(admin)# show install committed location 0/RSP0/cpu0

Node 0/RSP0/CPU0 [RP] [SDR: Owner] Boot Device: disk0: Boot Image: /disk0/asr9k-os-mbi-3.9.1/mbiasr9k-rp.vm<< 3.9.1 is committed


OL-23591-02


Committed Packages: disk0:asr9k-mini-p-3.9.1 disk0:asr9k-optic-3.9.1 disk0:asr9k-doc-p-3.9.1 disk0:asr9k-k9sec-p-3.9.1 disk0:asr9k-video-p-3.9.1 disk0:asr9k-mpls-p-3.9.1 disk0:asr9k-mgbl-p-3.9.1 disk0:asr9k-mcast-p-3.9.1

If the expected active software packages are not displayed, install the packages (if required) and activate the packages. See Cisco ASR 9000 Aggregation Services Router Router Getting Started Guide for information on installing and activating Cisco IOS XR software packages. The following example output shows the active packages for all cards in a router. The output displays the disk on which each package is located.

RP/0/RSP0/CPU0:router# show install active

Node 0/RSP0/CPU0 [RP] [SDR: Owner] Boot Device: disk0: Boot Image: /disk0/asr9k-os-mbi-4.0.0/mbiasr9k-rp.vm Active Packages: disk0:asr9k-mini-p-4.0.0 disk0:asr9k-optic-4.0.0 disk0:asr9k-doc-p-4.0.0 disk0:asr9k-k9sec-p-4.0.0 disk0:asr9k-video-p-4.0.0 disk0:asr9k-mpls-p-4.0.0 disk0:asr9k-mgbl-p-4.0.0 disk0:asr9k-mcast-p-4.0.0

Node 0/0/CPU0 [LC] [SDR: Owner] Boot Device: mem: Boot Image: /disk0/asr9k-os-mbi-4.0.0/lc/mbiasr9k-lc.vm Active Packages: disk0:asr9k-mini-p-4.0.0 disk0:asr9k-optic-4.0.0 disk0:asr9k-video-p-4.0.0 disk0:asr9k-mpls-p-4.0.0 disk0:asr9k-mcast-p-4.0.0 Node 0/1/CPU0 [LC] [SDR: Owner] Boot Device: mem: Boot Image: /disk0/asr9k-os-mbi-4.0.0/lc/mbiasr9k-lc.vm Active Packages: disk0:asr9k-mini-p-4.0.0 disk0:asr9k-optic-4.0.0 disk0:asr9k-video-p-4.0.0 disk0:asr9k-mpls-p-4.0.0 disk0:asr9k-mcast-p-4.0.0


Node 0/4/CPU0 [LC] [SDR: Owner] Boot Device: mem:


OL-23591-02


Boot Image: /disk0/asr9k-os-mbi-4.0.0/lc/mbiasr9k-lc.vm Active Packages: disk0:asr9k-mini-p-4.0.0 disk0:asr9k-optic-4.0.0 disk0:asr9k-video-p-4.0.0 disk0:asr9k-mpls-p-4.0.0 disk0:asr9k-mcast-p-4.0.0



Node 0/7/CPU0 [LC] [SDR: Owner] Boot Device: mem: Boot Image: /disk0/asr9k-os-mbi-4.0.0/lc/mbiasr9k-lc.vm Active Packages: disk0:asr9k-mini-p-4.0.0 disk0:asr9k-optic-4.0.0 disk0:asr9k-video-p-4.0.0 disk0:asr9k-mpls-p-4.0.0

The output shows the name of the disk on which the packages are located. In the above example, the active packages for each node are on disk0, and for all nodes, the composite package asr9k-os-mbi-4.0.0 is active. Additional packages shown are optional packages that have been activated after the initial loading of the Cisco ASR 9000 Aggregation Series Router Unicast Routing Core Bundle.

show install committed Command

Use the show install committed command to display committed software packages. The committed software packages are the software packages that will be booted on a router reload.

Committed packages are the packages that are persistent across router reloads. If you install and activate a package, it remains active until the next router reload. If you commit a package set, all packages in that set remain active across router reloads until the package set is replaced with another committed package set. The show install committed command is useful to ensure software is installed and committed after a router reload. If the expected software is not installed and committed, see Cisco ASR 9000 Aggregation Services Router Getting Started Guide for information on installing and committing Cisco IOS XR software packages.

The following command output shows the committed software packages on all cards in the router. The output displays the disk on which each package is located.

RP/0/RSP0/CPU0:router# show install committed

Node 0/RSP0/CPU0 [RP] [SDR: Owner]


OL-23591-02


Boot Device: disk0: Boot Image: /disk0/asr9k-os-mbi-4.0.0/mbiasr9k-rp.vm Committed Packages: disk0:asr9k-mini-p-4.0.0 disk0:asr9k-optic-4.0.0 disk0:asr9k-doc-p-4.0.0 disk0:asr9k-k9sec-p-4.0.0 disk0:asr9k-video-p-4.0.0 disk0:asr9k-mpls-p-4.0.0 disk0:asr9k-mgbl-p-4.0.0 disk0:asr9k-mcast-p-4.0.0

Node 0/0/CPU0 [LC] [SDR: Owner] Boot Device: mem: Boot Image: /disk0/asr9k-os-mbi-4.0.0/lc/mbiasr9k-lc.vm Committed Packages: disk0:asr9k-mini-p-4.0.0 disk0:asr9k-optic-4.0.0 disk0:asr9k-video-p-4.0.0 disk0:asr9k-mpls-p-4.0.0 disk0:asr9k-mcast-p-4.0.0





Node 0/6/CPU0 [LC] [SDR: Owner] Boot Device: mem:


OL-23591-02

Chapter 1 General Troubleshooting Procedures Validating and Troubleshooting Cisco IOS XR Software Configuration

Boot Image: /disk0/asr9k-os-mbi-4.0.0/lc/mbiasr9k-lc.vm Committed Packages: disk0:asr9k-mini-p-4.0.0 disk0:asr9k-optic-4.0.0 disk0:asr9k-video-p-4.0.0 disk0:asr9k-mpls-p-4.0.0 disk0:asr9k-mcast-p-4.0.0 Node 0/7/CPU0 [LC] [SDR: Owner] Boot Device: mem: Boot Image: /disk0/asr9k-os-mbi-4.0.0/lc/mbiasr9k-lc.vm Committed Packages: disk0:asr9k-mini-p-4.0.0 disk0:asr9k-optic-4.0.0 disk0:asr9k-video-p-4.0.0 disk0:asr9k-mpls-p-4.0.0 disk0:asr9k-mcast-p-4.0.0

The output shows the name of the disk on which the packages are located. In the above example, the committed packages for each node are on disk0, and for all nodes, the composite package asr9k-os-mbi-4.0.0 is committed. Additional packages shown are optional packages that have been committed after the initial loading of the Cisco ASR 9000 Aggregation Series Router Unicast Routing Core Bundle.

Validating and Troubleshooting Cisco IOS XR Software Configuration

Validating the Cisco IOS XR software configuration includes collecting configuration information on the router to determine configuration changes and verifying the current running configuration. When a configuration fails during a commit, the failed configuration can be viewed to help determine why the configuration was not committed.

The following sections are provided:

• Local and Global Configurations, page 1-16

• Collecting Configuration Information, page 1-19

• Verifying the Running Configuration, page 1-20

• Using the show configuration failed Command, page 1-24

Local and Global ConfigurationsTo troubleshoot configurations, you need to determine whether the problem is in the local configuration or the shared (global) configuration.

• The local configuration is specific to the individual LC or RP to which it belongs. Every LC and RP has a data store containing the local data for that node, including configuration and operational data for the local interfaces. An example of a local configuration is the port designations on a particular LC.

• The shared (global) configuration applies to the entire router, and is shared with all of the LCs and RPs. An example of a shared configuration is the routing protocol parameters.

To view the local configuration, use the show running-config interface * command. The output displays all the configured interfaces on the node.


OL-23591-02


RP/0/RSP0/CPU0:router# show running-config interface *

interface Bundle-Ether16 description Connect to router-S Port-Ch 16 mtu 9216 bundle maximum-active links 1!interface Bundle-Ether16.160 l2transport description Connect to router-S Port-Ch 16 Service Instance 160 encapsulation dot1q 160!...!interface Loopback0ipv4 address 10.144.144.144 255.255.255.255!interface tunnel-ip10!interface tunnel-te44190 description Primary GE Tunnel from router-S to router-T ipv4 unnumbered Loopback0 priority 0 0 autoroute announce signalled-bandwidth 100000 destination 10.19.19.19 fast-reroute record-route path-option 1 explicit name Primary_GE_Path_to_router-T ospf 100 area 0!...interface MgmtEth0/RSP0/CPU0/0 description Connected to LAN ipv4 address 172.29.52.137 255.255.255.0!interface MgmtEth0/RSP0/CPU0/1 shutdown! interface GigabitEthernet0/0/0/0 shutdown!interface GigabitEthernet0/0/0/1 shutdown!interface GigabitEthernet0/0/0/2 shutdown!...interface TenGigE0/7/0/1 shutdown!interface TenGigE0/7/0/2 shutdown!interface TenGigE0/7/0/3 shutdown!interface POS0/2/0/0 description Connected to PE_router-2 POS 0/2/0/0


OL-23591-02


!interface POS0/2/0/1 description Connected to PE_router-3 POS 0/2/0/1!controller SONET0/2/0/0 clock source internal!controller SONET0/2/0/1 clock source internal!

Use the show sysdb trace commands to display the contents of the system database after a configuration change. The trace information includes a history of any changes to the running configuration. You can specify either a local node or the shared plane.

The following example output shows the contents of the local database, that is, for a specific location (node):

RP/0/RSP0/CPU0:router# show sysdb trace verification location 0/5/cpu0 reverse

.

.

.Timestamp nid reqid jid tid reg_hndl connid action path432 wrapping entries (8192 possible, 158 filtered, 590 total)Sep 23 04:35:39.969 0/RSP0/CPU0 8168 354 1 94 4483 apply reply '--'Sep 23 04:35:39.960 0/RSP0/CPU0 8168 354 1 94 4505 Apply called 'cfg/if/act/tunnel-ip10/v'Sep 23 04:35:39.960 0/RSP0/CPU0 8168 354 1 94 4505 verify reply: accept '--'Sep 23 04:35:39.685 0/RSP0/CPU0 8168 354 1 94 4505 Verify called 'cfg/if/act/tunnel-ip10/v'Sep 23 04:35:39.678 0/RSP0/CPU0 0 354 1 94 4505 register 'cfg/if/act/tunnel-ip[0-9]*/mtu/tunnel-ip'Sep 23 04:35:39.678 0/RSP0/CPU0 0 354 1 94 4505 register 'cfg/if/act/tunnel-ip[0-9]*/im/bw'Sep 23 04:35:39.678 0/RSP0/CPU0 0 354 1 94 4505 register 'cfg/if/act/tunnel-ip[0-9]*/tunl_gre/keepalive'Sep 23 04:35:39.678 0/RSP0/CPU0 0 354 1 94 4505 register 'cfg/if/act/tunnel-ip[0-9]*/tunl_gre/dfbit_disable'Sep 23 04:35:39.678 0/RSP0/CPU0 0 354 1 94 4505 register 'cfg/if/act/tunnel-ip[0-9]*/tunl_gre/ttl'Sep 23 04:35:39.678 0/RSP0/CPU0 0 354 1 94 4505 register 'cfg/if/act/tunnel-ip[0-9]*/tunl_gre/tos'Sep 23 04:35:39.678 0/RSP0/CPU0 0 354 1 94 4505 register 'cfg/if/act/tunnel-ip[0-9]*/tunl_gre/mode'...

The following example output shows the contents of the shared database, that is, the configuration data that is shared with all LC and RP in the router:

RP/0/RSP0/CPU0:router# show sysdb trace verification shared-plane reverse

Config Shared Server====================Timestamp nid reqid jid tid reg_hndl connid action path2259 wrapping entries (4096 possible, 0 filtered, 2259 total)


OL-23591-02


Sep 23 19:34:40.202 0/3/CPU0 0 241 8 384 1430 unregister 'from-'Sep 23 19:34:40.197 0/3/CPU0 0 241 15 385 1434 unregister 'from-'Sep 23 19:34:40.196 0/3/CPU0 0 163 1 386 1440 unregister 'from-'Sep 23 19:14:45.076 0/3/CPU0 0 163 1 386 1440 register 'cfg/gl/ipv4/cef/hardware/forwarding/update/synchronous'Sep 23 19:14:41.679 0/3/CPU0 0 241 15 385 1434 register 'cfg/gl/dbgtrace/node/831/'Sep 23 19:14:41.593 0/3/CPU0 0 241 8 384 1430 register 'cfg/gl/dbgtrace/node/831/'Sep 23 19:12:36.472 0/3/CPU0 0 241 8 381 1375 unregister 'from-'Sep 23 19:12:36.471 0/3/CPU0 0 241 15 382 1378 unregister 'from-'Sep 23 19:12:36.470 0/3/CPU0 0 163 1 383 1383 unregister 'from-'Sep 23 19:07:56.914 0/3/CPU0 0 163 1 383 1383 register 'cfg/gl/ipv4/cef/hardware/forwarding/update/synchronous'...

The show processes location node-id | include sysdb command displays all active database processes for a specified node.

RP/0/RSP0/CPU0:router# show process location 0/1/CPU0 | include sysdb Thu Nov 4 14:06:30.191 DST279 1 0 56K 10 Sigwaitinfo 739:28:22:0145 0:00:00:0057 sysdb_svr_local279 2 1 56K 10 Receive 0:00:00:0779 0:00:02:0459 sysdb_svr_local279 3 1 56K 10 Receive 0:03:34:0474 0:00:03:0285 sysdb_svr_local279 4 1 56K 10 Receive 0:05:03:0006 0:00:02:0368 sysdb_svr_local277 1 0 64K 10 Sigwaitinfo 739:28:21:0305 0:00:00:0046 sysdb_mc277 2 0 64K 10 Receive 739:28:21:0274 0:00:00:0003 sysdb_mc277 3 1 64K 10 Receive 166:59:14:0698 0:00:00:0038 sysdb_mc277 4 1 64K 10 Receive 0:01:49:0941 0:00:00:0106 sysdb_mc277 6 1 64K 10 Receive 739:15:22:0734 0:00:00:0058 sysdb_mc

See Chapter 1, “General Troubleshooting Procedures” for additional information on troubleshooting processes.

Collecting Configuration InformationCollecting configuration information allows you to determine if changes to the system have occurred. It also allows you to determine if these changes could impact the system. The following commands allow you to determine if there was an unknown commit, if there was a commit that overwrote a previous configuration, or there are configuration changes that should be removed from the running configuration.

• show configuration commit changes {[since] commit-id | last number-of-commits} [diff]—the command output displays changes made to the running configuration by previous configuration commits.

RP/0/RSP0/CPU0:router# show configuration commit changes since 1000000319

Wed May 17 09:30:27.877 UTC Building configuration...no logging consoleno domain ipv4 host ce1no domain ipv4 host ce2domain ipv4 host ce6 172.29.52.73domain ipv4 host ce7 172.29.52.78


OL-23591-02


no domain ipv4 host pe1no domain ipv4 host pe2domain ipv4 host pe6 172.29.52.128domain ipv4 host pe7 172.29.52.182interface GigabitEthernet0/1/5/1 no negotiation!end

• show configuration commit list [number-of-commits] [detail]—the command output displays a list of the commit IDs (up to 100) available for rollback.

RP/0/RSP0/CPU0:router# show configuration commit list

Wed May 17 09:31:21.727 UTC SNo. Label/ID User Line Client Time Stamp~~~~ ~~~~~~~~ ~~~~ ~~~~ ~~~~~~ ~~~~~~~~~~1 1000000324 userA vty0 CLI 16:50:33 UTC Wed May 10 20062 1000000323 userA vty0 CLI 16:49:51 UTC Wed May 10 20063 1000000322 userB vty0 CLI 16:48:05 UTC Wed May 10 20064 1000000321 userC vty2 CLI 19:11:26 UTC Wed May 03 20065 1000000320 userA vty2 CLI 19:10:45 UTC Wed May 03 20066 1000000319 userB vty2 CLI 18:03:01 UTC Wed May 03 20067 1000000318 userB vty2 CLI 18:02:43 UTC Wed May 03 20068 1000000317 userB vty2 CLI 18:02:38 UTC Wed May 03 20069 1000000316 userC vty2 CLI 17:59:16 UTC Wed May 03 200610 1000000315 userC vty2 CLI 17:46:38 UTC Wed May 03 200611 1000000314 userA vty2 CLI 15:40:04 UTC Wed May 03 200612 1000000313 userA vty2 CLI 13:05:09 UTC Wed May 03 200613 1000000312 userD con0_RSP0_C CLI 13:49:31 UTC Mon May 01 2006

• commit confirmed minutes (executed from config mode)—This command commits the configuration on a trial basis for a minimum of 30 seconds and a maximum of 300 seconds (5 minutes). During the trial configuration period, enter commit to confirm the configuration. If commit is not entered, then the system will revert to the previous configuration when the trial time period expires.

Verifying the Running ConfigurationTo verify the running configuration, perform the following procedure.

SUMMARY STEPS

1. configure

2. show running-config

3. describe hostname hostname

4. end

5. show sysdb trace verification shared-plane | include path

6. show sysdb trace verification location node-id

7. show cfgmgr trace

8. show configuration history commit

9. show configuration commit changes {last | since | commit-id}


OL-23591-02


10. show config failed startup

11. cfs check

DETAILED STEPS


Step 1 configure

Example:RP/0/RSP0/CPU0:router# configure

Enters global configuration mode.

Step 2 show running-config

Example:RP/0/RSP0/CPU0:router(config)# show running-config

Displays the contents of the running configuration.

Verify that the running configuration is as expected.

Step 3 describe hostname hostname

Example:RP/0/RSP0/CPU0:router(config)# describe hostname router_A

Determines the path.

Step 4 end

Example:RP/0/RSP0/CPU0:router(config)# end

Saves configuration changes.

• When you issue the end command, the system prompts you to commit changes:

Uncommitted changes found, commit them before exiting(yes/no/cancel)? [cancel]:

– Entering yes saves configuration changes to the running configuration file, exits the configuration session, and returns the router to EXEC mode.

– Entering no exits the configuration session and returns the router to EXEC mode without committing the configuration changes.

– Entering cancel leaves the router in the current configuration session without exiting or committing the configuration changes.


OL-23591-02


Examples

The following example shows the output of the show running-config command:

RP/0/RSP0/CPU0:router# show running-config

Building configuration...!! IOS XR Configuration 4.0.1.10I!! Last configuration change at Thu Sep 23 04:35:38 2010 by user_A!hostname router.

Step 5 show sysdb trace verification shared-plane | include path

Example:RP/0/RSP0/CPU0:router# show sysdb trace verification shared-plane | include gl/a/hostname

Displays details of recent verification sysDB transactions and changes on the shared plane allowing you to verify whether the configuration was verified correctly.

Specifying the path filters the data to display only the sysDB path for the router.

Verify that changes to the SysDB were verified and accepted.

Step 6 show sysdb trace verification location node-id

Example:RP/0/RSP0/CPU0:router# show sysdb trace verification location 0/3/CPU0

Displays details of recent verification sysDB transactions and changes on local plane configurations.

Verify that changes to the SysDB were verified and accepted.

Step 7 show cfgmgr trace

Example:RP/0/RSP0/CPU0:router# show cfgmgr trace

Displays cfgmgr trace information.

Step 8 show configuration history commit

Example:RP/0/RSP0/CPU0:router# show configuration history commit

Displays a list of historical changes to the configuration.

Verify that the timeline of changes is as expected.

Step 9 show configuration commit changes {last | since | commit-id}

Example:RP/0/RSP0/CPU0:router# show configuration commit changes last 15

Displays detailed committed configuration history information.

Verify that the history information is as expected.

Step 10 show configuration failed startup

Example:RP/0/RSP0/CPU0:router# show configuration failed startup

Displays information on any configurations that failed during startup.

Step 11 cfs check

Example:RP/0/RSP0/CPU0:router# cfs check

Checks the current configuration to see if there are any missing configurations.



OL-23591-02


.

.logging suppress duplicatestelnet vrf default ipv4 server max-servers 100domain name cisco.comdomain lookup disabletaskgroup default!...interface preconfigure GigabitEthernet0/3/0/7 shutdown!interface preconfigure GigabitEthernet0/3/0/8 shutdown!interface preconfigure GigabitEthernet0/3/0/9 shutdown!interface preconfigure GigabitEthernet0/3/0/10 shutdown!...

The output is used to determine if the configuration is as expected.

In the following example, the path to SysDB where the configuration is stored in the database is displayed.

RP/0/RSP0/CPU0:router(config)# describe hostname router

Package: iosxr-infra iosxr-infra V4.0.0 IOS-XR Infra Package Definition Vendor : Cisco Systems Desc : IOS-XR Infra Package Definition Build : Built on Wed Sep 8 16:07:48 DST 2010 Source : By sc-g-01 in /auto/ioxbuild8/production/4.0.0/asr9k/workspace for pie Card(s): RP, NP24-4x10GE, NP24-40x1GE, NP40-40x1GE, NP40-4x10GE, NP40-8x10GE, NP40-2_20_COMBO, NP80-8x10GE, NP80-16x10GE, A9K-SIP-700, A9K-SIP-500 Restart information: Default: parallel impacted processes restart Size Compressed/Uncompressed: 38MB/85MB (44%)

Component: shellutil V[ci-401/7] Common shell utility applications

User needs ALL of the following taskids:

host-services (READ) or root-lr (READ WRITE)

It will take the following actions: Create/Set the configuration item: Path: gl/a/hostname Value: router


OL-23591-02


Using the show configuration failed CommandUse the show configuration failed command to browse a failed configuration. The configuration can be classified as failed during startup or during a configuration commit.

• Startup Failed Configuration, page 1-24

• Commit Configuration Failed, page 1-25

Startup Failed Configuration

A configuration can be classified as failed during startup for three reasons:

• Syntax errors

Syntax errors are generated by the parser and usually indicate that there is an incompatibility with the CLI commands. Correct the syntax errors and reapply the configuration. A syntax error can be an invalid CLI entry or a CLI syntax change. See the “Obtaining Documentation and Submitting a Service Request” section on page xii in the Preface for information on obtaining Cisco IOS XR software CLI documentation.

• Semantic errors

Semantic errors are generated by the backend components when the configuration is being restored by the configuration manager during startup of the router. Semantic errors include logical problems (invalid logic).

• Apply errors

Apply errors are generated when a configuration has been successfully verified and accepted as part of running configuration but the backend component is not able to update its operational state. The configuration shows both as the running configuration (since it was correctly verified) and as a failed configuration because of the backend operational error. To find the component apply owner, use the describe on the CLI that failed to be applied.

Note You may browse startup failed configurations for up to the previous four router reloads.

Use the show configuration failed startup command and the load configuration failed startup command to browse and reapply any failed configuration. The load configuration failed startup command can be used in configuration mode to load the failed startup configuration into the target configuration session, then the configuration can be modified and committed. See Cisco ASR 9000 Aggregation Services Router Router Getting Started Guide for information on committing a configuration.

RP/0/RSP0/CPU0:router# show configuration failed startup

!! CONFIGURATION FAILED DUE TO SYNTAX/AUTHORIZATION ERRORS telnet vrf default ipv4 server max-servers 5 interface POS0/7/0/3 router static address-family ipv4 unicast 0.0.0.0/0 172.18.189.1

!! CONFIGURATION FAILED DUE TO SEMANTIC ERRORS router bgp 217 !!% Process did not respond to sysmgr ! RP/0/RSP0/CPU0:router#

RP/0/RSP0/CPU0:router# config


OL-23591-02


RP/0/RSP0/CPU0:router(config)# load config failed startup noerror

Loading. 263 bytes parsed in 1 sec (259)bytes/sec RP/0/RSP0/CPU0:mike3(config-bgp)#show configuration Building configuration... telnet vrf default ipv4 server max-servers 5 router static address-family ipv4 unicast 0.0.0.0/0 172.18.189.1 ! !router bgp 217 ! end

The failed configuration is loaded into the target configuration, minus the errors that caused the startup configuration to fail.

RP/0/RSP0/CPU0:router(config-bgp)# commit

Use the show configuration failed command to display failed items in the last configuration commit, including reasons for the error.

In any mode, the configuration failures from the most recent commit operation are displayed.

The show configuration failed command can be used in EXEC mode and configuration mode. The command is used in EXEC mode when the configuration does not load during startup. The command is used in configuration mode to display information when a commit fails.

The following example shows the show configuration failed command.

RP/0/RSP0/CPU0:router(config)# interface pos 0/6/0/4 RP/0/RSP0/CPU0:router(config-if)# no vrf RP/0/RSP0/CPU0:router(config-if)# commit

% Failed to commit one or more configuration items during an atomic operation, no changes have been made. Please use 'show configuration failed' to view the errors

RP/0/RSP0/CPU0:router(config-if)# exitRP/0/RSP0/CPU0:router(config)# show configuration failed

Wed May 2 13:14:08.426 EST EDT !! CONFIGURATION FAILED DUE TO SEMANTIC ERRORS interface POS0/6/0/4 no vrf !!% The interface's numbered and unnumbered IPv4/IPv6 addresses must be removed prior to changing or deleting the VRF !

Note The show configuration failed command in configuration mode only exists as long as the configuration session is active. Once you exit configuration mode, the command cannot be used to display the failed configuration.

Commit Configuration Failed

The following example shows an invalid task ID configuration that fails to commit. The show configuration failed command provides information on why the configuration failed.

RP/0/RSP0/CPU0:router(config)# taskgroup isis RP/0/RSP0/CPU0:router(config-tg)# commit

% Failed to commit one or more configuration items during an atomic operation, s

RP/0/RSP0/CPU0:router(config-tg)# show configuration failed


OL-23591-02

Chapter 1 General Troubleshooting Procedures Verifying the System

!! CONFIGURATION FAILED DUE TO SEMANTIC ERRORStaskgroup isis!!% Usergroup/Taskgroup names cannot be taskid names!

If a configuration commit fails, do not exit configuration mode (return to EXEC mode) as you will not be able to view the failed configuration.

RP/0/RSP0/CPU0:router# configureRP/0/RSP0/CPU0:router(config)# taskgroup bgp RP/0/RSP0/CPU0:router(config-tg)# endUncommitted changes found, commit them before exiting(yes/no/cancel)? [cancel]:y

% Failed to commit one or more configuration items during an atomic operation, s

RP/0/RSP0/CPU0:router(config)# exitUncommitted changes found, commit them before exiting(yes/no/cancel)? [cancel]:nRP/0/RSP0/CPU0:router# show configuration failed RP/0/RSP0/CPU0:router#

Verifying the SystemTo verify the general status and state of a router using Cisco IOS XR software, perform the following procedure.

SUMMARY STEPS

1. admin

2. show platform [node-id]

3. show version


5. show logging

6. show environment

7. show context

8. exit

9. show context

10. show memory summary detail location all

11. show memory heap summary {job-id | all}

12. top processes


14. show system verify start show system verify report

15. show {ipv4 | ipv6} interface brief


OL-23591-02


DETAILED STEPS


Step 1 admin

Example:RP/0/RSP0/CPU0:router# admin

Enters administration mode.

Step 2 show platform [node-id]

Example:RP/0/RSP0/CPU0:router(admin)# show platform

Displays information about the status of cards and modules installed in the router.

• Some cards support a CPU module and service processor (SP) module. Other cards support only a single module.

• A card module is also called a node. When all nodes are working properly, the status of each node displayed in the State column is IOS-XR RUN.

• If you run the command without a node-id (show platform as shown in the example), the output will include all nodes in the system.

• Type the show platform node-id command to display information for a specific node. Replace node-id with a node name from the show platform command Node column.

Step 3 show version

Example:RP/0/RSP0/CPU0:router(admin)# show version

Displays information about the router, including image names, uptime, and other system information.

Verify that the expected software version and images are installed.


Example:RP/0/RSP0/CPU0:router(admin)# show running-config

Displays all of the nondefault commands currently running, including hardware module power status, secure domain router (SDR) configuration, and fabric configuration. The output also displays the users defined in administration mode with root-system access.

Verify that the serial numbers for the nodes in the current running configuration are what you expected. The expected rack numbers and serial numbers should be listed in the current system documentation. See the “Prerequisite Documentation for Troubleshooting” section on page 1-1.

Also verify that the hardware module power status is as expected and the SDR and fabric configurations are as expected.

Step 5 show logging

Example:RP/0/RSP0/CPU0:router(admin)# show logging

Displays all syslog messages stored in the buffer. The command output displays the device operation history from a system perspective.

Analyze the logged events and their order of happening. Check for anything out of the ordinary such as errors, tracebacks, or crashes. Also check for any Severity 1 or Severity 2 errors.


OL-23591-02


Step 6 show environment

Example:RP/0/RSP0/CPU0:router(admin)# show environment

Displays environmental monitor parameters for the system.

Verify that the parameters are as expected.

Step 7 show context

Example:RP/0/RSP0/CPU0:router(admin)# show context

Displays core dump context information on fabric cards, alarm modules, fan controllers, and service processors (system-owned cards). See the “show context Command” section on page 1-50 for more information on the show context command output.

Step 8 exit

Example:RP/0/RSP0/CPU0:router(admin)# exit

Exits administration mode.

Step 9 show context

Example:RP/0/RSP0/CPU0:router# show context

Displays core dump context information on CPUs responsible for routing and Cisco Express Forwarding (CEF). See the “show context Command” section on page 1-50 for more information on the show context command output.

Step 10 show memory summary detail location all

Example:RP/0/RSP0/CPU0:router# show memory summary detail location all

Displays information about the memory available on the router after the system image decompresses and loads.

Verify that the expected memory is available or installed. Ensure that all memory regions have adequate free space available.

Step 11 show memory heap summary {job-id | all}

Example:RP/0/RSP0/CPU0:router# show memory heap summary all

Displays a summary of the information about the heap space. The output displays each process and the amount of memory allocated for each process.

Note The job-id is the output of the show processes command.

Verify if there are any processes using a large amount of memory.

Step 12 top processes

Example:RP/0/RSP0/CPU0:router# top processes

Press q to exit the command.

Provides a live update of process resource consumption.

Press ‘M’ to sort by memory usage.

Verify that the resource consumption is as expected.


Example:RP/0/RSP0/CPU0:router# show running-config

Displays the contents of the currently running configuration.

Verify that the contents of the current running configuration are what you expected.



OL-23591-02


Examples

The output from the show platform command indicates that all expected nodes are in the run state. If all nodes in the system are active, the cards should be in the IOS XR RUN and the SPAs should be in the OK state. The example output shows that all expected nodes are in the run state.

RP/0/RSP0/CPU0:router(admin)# show platform

Node Type State Config State-----------------------------------------------------------------------------0/RSP0/CPU0 A9K-RSP-4G(Active) IOS XR RUN PWR,NSHUT,MON0/FT0/SP FAN TRAY READY 0/FT1/SP FAN TRAY READY 0/1/CPU0 A9K-40GE-B IOS XR RUN PWR,NSHUT,MON0/2/CPU0 A9K-SIP-700 IOS XR RUN PWR,NSHUT,MON0/2/0 SPA-2XOC48POS/RPR OK PWR,NSHUT,MON0/3/CPU0 A9K-2T20GE-B IN-RESET PWR,NSHUT,MON0/4/CPU0 A9K-8T/4-B IOS XR RUN PWR,NSHUT,MON0/6/CPU0 A9K-4T-B IOS XR RUN PWR,NSHUT,MON0/PM0/SP A9K-3KW-AC READY PWR,NSHUT,MON0/PM1/SP A9K-3KW-AC READY PWR,NSHUT,MON0/PM2/SP A9K-3KW-AC READY PWR,NSHUT,MON

The output from the show version command indicates the version of software being run on the nodes and from which location (disk or network). Check that the expected software version and images are installed. The example output shows that the Cisco IOS XR software version is 4.0.0 and that the installed pie versions are also 4.0.0.

RP/0/RSP0/CPU0:router(admin)# show version

Cisco IOS XR Software, Version 4.0.0[Default]Copyright (c) 2010 by Cisco Systems, Inc.

ROM: System Bootstrap, Version 1.04(20100216:021454) [ASR9K ROMMON],

Step 14 show system verify startshow system verify report

Example:RP/0/RSP0/CPU0:router# show system verify startRP/0/RSP0/CPU0:router# show system verify report

A two-step command that produces system reports.

• show system verify start—Starts the system verify process (creates the initial baseline file)

• show system verify report—Generates a report for the system verification process (report of the current status)

The output of the show system verify report command provides a comparison of the system at the time of the show system verify start snapshot and the show system verify report snapshot. The output provides a sanity check of the system provided the show system verify start system snapshot was taken when the system was healthy or before an event.

Verify that the system parameters are as expected.

Step 15 show (ipv4 | ipv6} interface brief

Example:RP/0/RSP0/CPU0:router# show ipv4 interface brief

Displays the usability status of interfaces.

Verify that all expected interfaces are listed, that they have the correct assigned address, and that they are in the expected states.



OL-23591-02


router uptime is 1 day, 15 hours, 53 minutesSystem image file is "bootflash:disk0/asr9k-os-mbi-4.0.0/mbiasr9k-rp.vm"

cisco ASR9K Series (MPC8641D) processor with 4194304K bytes of memory.MPC8641D processor at 1333MHz, Revision 2.2

2 Management Ethernet12 DWDM controller(s)12 TenGigE40 GigabitEthernet2 SONET/SDH2 Packet over SONET/SDH219k bytes of non-volatile configuration memory.975M bytes of compact flash card.33994M bytes of hard disk.1605616k bytes of disk0: (Sector size 512 bytes).1605616k bytes of disk1: (Sector size 512 bytes).

Configuration register on node 0/RSP0/CPU0 is 0x102Boot device on node 0/RSP0/CPU0 is disk0:Package active on node 0/RSP0/CPU0:asr9k-optics-supp, V 4.0.0[DT_IMAGE], Cisco Systems, at disk0:asr9k-optics-supp-4.0.0 Built on Wed Sep 8 16:17:30 DST 2010 By sjc5-gf-021 in /auto/ioxbuild8/production/4.0.0.DT_IMAGE/asr9k/workspace for pie

asr9k-fwding, V 4.0.0[DT_IMAGE], Cisco Systems, at disk0:asr9k-fwding-4.0.0 Built on Wed Sep 8 16:12:40 DST 2010 By sjc5-gf-021 in /auto/ioxbuild8/production/4.0.0.DT_IMAGE/asr9k/workspace for pie

asr9k-cpp, V 4.0.0[DT_IMAGE], Cisco Systems, at disk0:asr9k-cpp-4.0.0 Built on Wed Sep 8 16:13:28 DST 2010 By sjc5-gf-021 in /auto/ioxbuild8/production/4.0.0.DT_IMAGE/asr9k/workspace for pie

asr9K-doc-supp, V 4.0.0[DT_IMAGE], Cisco Systems, at disk0:asr9K-doc-supp-4.0.0 Built on Wed Sep 8 16:16:57 DST 2010 By sjc5-gf-021 in /auto/ioxbuild8/production/4.0.0.DT_IMAGE/asr9k/workspace for pie

asr9k-scfclient, V 4.0.0[DT_IMAGE], Cisco Systems, at disk0:asr9k-scfclient-4.0.0 Built on Wed Sep 8 16:13:26 DST 2010 By sjc5-gf-021 in /auto/ioxbuild8/production/4.0.0.DT_IMAGE/asr9k/workspace for pie

iosxr-security, V 4.0.0[DT_IMAGE], Cisco Systems, at disk0:iosxr-security-4.0.0 Built on Wed Sep 8 16:16:48 DST 2010 By sjc5-gf-021 in /auto/ioxbuild8/production/4.0.0.DT_IMAGE/asr9k/workspace for pie --More--

The output from the show running-config command displays the current running configuration, that is, all of the nondefault commands currently active. Verify that the contents of the current running configuration are as expected.

Tip The output of this command in exec mode is different from the output in admin mode. You should run the command from each of these modes to locate all of the configuration information.


Building configuration...!! Last configuration change at 18:56:31 UTC Tue Feb 28 2006 by user_A!hostname routerclock timezone PST 8.


OL-23591-02


logging console informationaltelnet vrf default ipv4 server max-servers 100domain name cisco.comdomain lookup disabletaskgroup default!taskgroup igpadmin task write rib task write isis task write ospf!taskgroup routeadmin task write bgp inherit taskgroup igpadmin!usergroup default taskgroup default!usergroup igp_admin taskgroup igpadmin!usergroup route_admin taskgroup routeadmin!tacacs-server host 172.29.52.69 port 49 key 7 060506324F41

aaa group server tacacs+ DOC_LAB_TACACS+ server 172.29.52.69 server 172.29.52.68!aaa authorization exec LAB_AAA group DOC_LAB_TACACS+ localaaa authorization exec CONSOLE_AAA group DOC_LAB_TACACS+ noneaaa authorization commands LAB_AAA group DOC_LAB_TACACS+ noneaaa authorization commands CONSOLE_AAA group DOC_LAB_TACACS+ noneaaa authentication login default group DOC_LAB_TACACS+ localaaa authentication login CONSOLE_AAA group DOC_LAB_TACACS+ localaaa default-taskgroup defaultexplicit-path name Primary_GE_Path_to_P19 index 1 next-address strict ipv4 unicast 10.114.4.44 index 2 next-address strict ipv4 unicast 10.114.4.11 index 3 next-address strict ipv4 unicast 10.119.4.11 index 4 next-address strict ipv4 unicast 10.119.4.19 index 5 next-address strict ipv4 unicast 10.19.19.19!line console accounting exec CONSOLE_AAA accounting commands CONSOLE_AAA authorization exec CONSOLE_AAA authorization commands CONSOLE_AAA login authentication CONSOLE_AAA exec-timeout 600 0 session-timeout 600!line default exec-timeout 600 0 session-timeout 600!...interface preconfigure GigabitEthernet0/3/0/18 shutdown


OL-23591-02


!interface preconfigure GigabitEthernet0/3/0/19 shutdown!interface preconfigure TenGigE0/3/0/0 shutdown!interface preconfigure TenGigE0/3/0/1 shutdown!router static address-family ipv4 unicast 0.0.0.0/0 172.29.52.1 200 !!router isis 100 is-type level-2-only net 49.0001.0000.0000.0044.00 nsf cisco address-family ipv4 unicast metric-style wide mpls traffic-eng level-2-only mpls traffic-eng router-id Loopback0 ! interface Loopback0 passive address-family ipv4 unicast ! ! interface TenGigE0/4/0/0 bfd minimum-interval 50 bfd multiplier 3 bfd fast-detect ipv4 address-family ipv4 unicast ! !!router ospf 100 nsr router-id 10.144.144.144 bfd minimum-interval 50 bfd multiplier 3 mpls ldp sync nsf cisco area 0 mpls ldp sync-igp-shortcuts mpls traffic-eng interface Loopback0 passive enable !...http serverssh server vrf defaultigmp snooping profile default system-ip-address 10.144.144.144 minimum-version 2 internal-querier tcn query solicit ttl-check disable router-alert-check disable!igmp snooping profile mrouter


OL-23591-02


router-guard mrouter!router pim address-family ipv4 mofrr mofrr-acl rp-address 10.11.11.11 rp-address 10.144.144.144 bidir-acl bidir spt-threshold infinity !!end

RP/0/RSP0/CPU0:router# adminWed Oct 27 14:52:07.000 DST

RP/0/RSP0/CPU0:router(admin)# show running-config Wed Oct 27 14:52:12.766 DSTBuilding configuration...!! IOS XR Admin Configuration 4.0.0username doclabuser-c group root-system group cisco-support secret 5 $1$RJVQ$6w7saUHgk16v5HXRWEp6m/!username doclabuser-r group root-system secret 5 $1$.uOF$O9N0aRRk.V1qe250IavLw1!alias cr copy run disk0a:/usr/base_config_admin alias sa show alias alias sc show config commit list alias sd show diag alias si show install req alias sl show led alias sp show platform alias sr show run alias sv show version alias nda no debug allend

The output from the show logging command displays the contents of the logging buffer. The output displays details on syslog historical events. Analyze the logged events and the order in which they happened. Check for anything out of the ordinary such as errors, tracebacks, or crashes. Also check for any Severity 1 or Severity 2 errors.

RP/0/RSP0/CPU0:router(admin)# show logging

Syslog logging: enabled (0 messages dropped, 0 flushes, 0 overruns) Console logging: level informational, 693 messages logged Monitor logging: level debugging, 0 messages logged Trap logging: level informational, 0 messages logged Buffer logging: level debugging, 4467 messages logged

Log Buffer (307200 bytes):

LC/0/3/CPU0:Sep 13 23:58:03.272 : pfm_node_lc[230]: %PLATFORM-NP-0-NP_INIT_FAILURE : Set|prm_server[110670]|Network Processor Unit(0x1007000)|Persistent Initialization Failure.


OL-23591-02


LC/0/3/CPU0:Sep 13 23:58:03.276 : pfm_node_lc[230]: %PLATFORM-PFM-0-CARD_RESET_REQ : pfm_dev_sm_perform_recovery_action, Card reset requested by: Process ID: 110670 (prm_server), Fault Sev: 0, Target node: 0/3/CPU0, CompId: 0x1f, Device Handle: 0x1007000, CondID: 1027, Fault Reason: Persistent Initialization Failure. LC/0/3/CPU0:Sep 13 23:58:03.276 : syslog_dev[85]: pfm_node_lc[230]: Request Graceful Reboot via Sysmgr: Reason: pfm_dev_sm_perform_recovery_action, Card reset requested by: Process ID: 110670 (prm_server), Fault Sev: 0, Target node: 0/3/CPU0, CompId: 0x1f, Device Handle: 0x1007000, CondID: 1027, Fault Reason: Persistent Initialization Failure. LC/0/3/CPU0:Sep 13 23:58:03.277 : sysmgr[87]: %OS-SYSMGR-2-REBOOT : reboot required, process (pfm_node_lc) reason (pfm_dev_sm_perform_recovery_action, Card reset requested by: Process ID: 110670 (prm_server), Fault Sev: 0, Target node: 0/3/CPU0, CompId: 0x1f, Device Handle: 0x1007000, CondID: 1027, Fault Reason: Persistent Initialization Failure. ) LC/0/3/CPU0:Sep 13 23:58:03.467 : sysmgr[87]: %OS-SYSMGR-3-ERROR : sysmgr_shutdown_cleanup_handler: shutdown script execution timed-out! Node will reset LC/0/3/CPU0:Sep 13 23:58:03.467 : sysmgr[87]: %OS-SYSMGR-7-DEBUG : sysmgr_shutdown_cleanup_handler: shutdown script execution timed-out! Node will reset LC/0/3/CPU0:Sep 13 23:58:03.468 : sysmgr[87]: %OS-SYSMGR-3-ERROR : sysmgr_shutdown_cleanup_handler: shutdown triggered by (pfm_node_lc) did not complete in 45 seconds, shutting down RP/0/RSP0/CPU0:Sep 13 23:58:16.859 : shelfmgr[299]: %PLATFORM-SHELFMGR-0-MAX_RESET_BRINGDOWN : Can not boot node 0/3/CPU0 A9K-2T20GE-B due to multiple resets, putting it IN_RESET state. The probable cause is an unexpected event on the node or a failure in communication with the node. Please refer to the Cisco ASR 9000 System Error Message Reference Guide for further information if needed.

--More--

The output from the show environment command displays environmental monitor parameters for the system. Verify that the environment parameters are as expected. Environment parameter anomalies are logged in the syslog, so if an environment parameter displayed in the show environment command output is not as expected, check the syslog using the show logging command. The syslog provides details on any logged problems.

RP/0/RSP0/CPU0:router(admin)# show environment

Wed Sep 15 09:48:27.178 DST

Temperature Information---------------------------------------------

R/S/I Modules Sensor (deg C)

0/1/* host Inlet0 36.4 host Hotspot0 46.7 0/2/* spa0 InletTemp 35.5 spa0 Hotspot 35.5 host Inlet0 34.5 host Hotspot0 61.0 0/3/* host Inlet0 31.1 host Hotspot0 32.5 0/RSP0/* host Inlet0 31.3 host Hotspot0 42.0

0/4/*


OL-23591-02


host Inlet0 34.9 host Hotspot0 45.2

0/6/* host Inlet0 38.0 host Hotspot0 48.9

0/FT0/* host Inlet0 31.9 host Hotspot0 33.5 0/FT1/* host Inlet0 32.3 host Hotspot0 35.3

Voltage Information---------------------------------------------

R/S/I Modules Sensor (mV) Margin

0/1/* host IBV 10592 n/a host 5.0V 4925 n/a host VP3P3_CAN 3289 n/a host 3.3V 3302 n/a host 2.5V 2516 n/a host 1.8VB 1812 n/a host 1.2VB 1193 n/a host 1.8VA 1806 n/a host 0.9VB 886 n/a host 1.2V_LDO_BRG0 1193 n/a host 1.2V_LDO_BRG1 1195 n/a host 1.8VC 1811 n/a host 1.5VB 1505 n/a host 1.5VA 1503 n/a host 1.1V(1.05V_CPU) 1053 n/a host 0.75VA 752 n/a host 0.75VB_0.75VC 754 n/a host 1.1VB 1103 n/a host 1.2V_TCAM0 1003 n/a host 1.2V_TCAM1 1000 n/a host 1.0V_Bridge_LDO 999 n/a host 1.0VB 1042 n/a host 0.75VD_and_0.75VE 752 n/a host 1.2V_TCAM2 1006 n/a host 1.2V_TCAM3 1002 n/a host 1.5VC 1504 n/a host 1.8VD 1804 n/a host 1.1VC 1100 n/a host ZARLINK_3.3V 3272 n/a host ZARLINK_1.8V 1807 n/a host 1.2V_DB 1195 n/a host 3.3V_DB 3318 n/a host 2.5V_DB 2535 n/a host 1.5V_DB 1509 n/a...LED Information


OL-23591-02


---------------------------------------------

R/S/I Modules LED Status0/RSP0/* host Critical-Alarm Off host Major-Alarm Off host Minor-Alarm Off host ACO Off host Fail Off

Fan Information---------------------------------------------

Fan speed (rpm): FAN0 FAN1 FAN2 FAN3 FAN4 FAN5 FAN6 FAN7 FAN8 FAN9 FAN10 FAN11

0/FT0/* 3510 3480 3510 3570 3540 3510 3510 3480 3540 3540 3480 34800/FT1/* 3510 3510 3510 3540 3480 3510 3390 3510 3510 3540 3450 3480Power Supply Information---------------------------------------------

R/S/I Modules Capacity Status (W)0/PM0/* host PM 3000 Ok

0/PM1/* host PM 3000 Ok

0/PM2/* host PM 3000 Ok

R/S/I Power Draw Voltage Current (W) (V) (A)0/PM0/* 622.9 53.7 11.60/PM1/* 774.7 53.8 14.40/PM2/* 0.0 54.1 0.0--------------Total: 1397.6

Power Shelves Type: AC

Total Power Capacity: 9000WUsable Power Capacity: 9000WSupply Failure Protected Capacity: 6000WFeed Failure Protected Capacity: 3000WWorst Case Power Used: 3170W

Slot Max Watts ---- --------- 0/1/CPU0 350 0/2/CPU0 450 0/RSP0/CPU0 235 0/RSP1/CPU0 235 (default) 0/4/CPU0 350 0/6/CPU0 350 0/FT0/SP 600


OL-23591-02


0/FT1/SP 600

Worst Case Power Available: 5830WSupply Protected Capacity Available: 2830WFeed Protected Capacity Available: Not Protected

The output from the show context command displays core dump context information. See the “show context Command” section on page 1-50 for more information on the show context command output.

RP/0/RSP0/CPU0:router# show context

node: node0_1_CPU0------------------------------------------------------------------

Crashed pid = 61524 (pkg/bin/tcam_mgr)Crashed tid = 1Crash time: Wed Apr 05, 2006: 18:27:26Core for process at harddisk:/dumper/first.tcam_mgr.abort.node0_1_CPU0.ppc.Z

Stack Trace#0 0xfc1d3fa0#1 0xfc1c6340#2 0xfc1c5364#3 0xfc1c542c#4 0x48210930#5 0x482110b8#6 0x48212ba4#7 0x48203dd8#8 0x4820c61c#9 0xfc1557ec#10 0xfc15573c#11 0xfc152fb8#12 0x4820d140 Registers info r0 r1 r2 r3 R0 00000000 481ff7b0 4824a55c 00000000 r4 r5 r6 r7 R4 0000f054 00000001 00000006 00000000 r8 r9 r10 r11 R8 00000000 fc220000 481fffc0 00000000 r12 r13 r14 r15 R12 4823be90 4824a4a0 48230000 00000000 r16 r17 r18 r19 R16 00000048 00000001 00000019 48256520 r20 r21 r22 r23 R20 00000000 00000000 00000003 00000045 r24 r25 r26 r27 R24 00000003 00000000 00000003 4825dc34 r28 r29 r30 r31 R28 00000006 0000f054 48254064 481ff810 cnt lr msr pc R32 00000000 fc1c6340 0000d932 fc1d3fa0 cnd xer R36 28004024 00000008

DLL Info DLL path Text addr. Text size Data addr. Data size Version/hfr-os-3.3.90/lib/libinfra.dll 0xfc142000 0x00034200 0xfc1343b8 0x00000bbc 0/lib/libc.dll 0xfc1a8000 0x00079dd8 0xfc222000 0x00002000 0

Crash Package InfomationPackage: hfr-mgbl, Source: By edde-bld1 in /vws/aga/production/3.3.90.1I/hfr/workspace for c2.95.3-p8


OL-23591-02


Package: hfr-mcast, Source: By edde-bld1 in /vws/aga/production/3.3.90.1I/hfr/workspace for c2.95.3-p8Package: hfr-mpls, Source: By edde-bld1 in /vws/aga/production/3.3.90.1I/hfr/workspace for c2.95.3-p8Package: hfr-rout, Source: By edde-bld1 in /vws/aga/production/3.3.90.1I/hfr/workspace for c2.95.3-p8Package: hfr-k9sec, Source: By edde-bld1 in /vws/aga/production/3.3.90.1I/hfr/workspace for c2.95.3-p8Package: hfr-lc, Source: By edde-bld1 in /vws/aga/production/3.3.90.1I/hfr/workspace for c2.95.3-p8Package: hfr-fwdg, Source: By edde-bld1 in /vws/aga/production/3.3.90.1I/hfr/workspace for c2.95.3-p8Package: hfr-admin, Source: By edde-bld1 in /vws/aga/production/3.3.90.1I/hfr/workspace for c2.95.3-p8Package: hfr-base, Source: By edde-bld1 in /vws/aga/production/3.3.90.1I/hfr/workspace for c2.95.3-p8Package: hfr-os-mbi, Source: By edde-bld1 in /vws/aga/production/3.3.90.1I/hfr/workspace for c2.95.3-p8

node: node0_6_CPU0------------------------------------------------------------------

node: node0_RSP0_CPU0------------------------------------------------------------------

node: node0_RSP1_CPU0------------------------------------------------------------------

The example output shows that the pkg/bin/tcam_mgr process crashed.

The output from the show memory command displays information about the memory available on the router after the system image decompresses and loads. Verify that the expected memory is available or installed. Ensure that all memory regions have adequate free space available. The example output shows that there is 2.003 gigabits of application memory available.

RP/0/RSP0/CPU0:router# show memory summary detail location all

Physical Memory: 4.000G total (2.003G available) Application Memory : 3.826G (2.003G available) Image: 48.725M (bootram: 48.725M) Reserved: 128.000M, IOMem: 1.980G, flashfsys: 0 Shared window mfwdv6: 449.910K Shared window mfwd_info: 701.910K Shared window soasync-app: 242.402K Shared window soasync: 242.402K Shared window li: 3.359K Shared window ipv4_fib: 1.003M Shared window l2fib: 2.425M Shared window statsd_db: 67.386K Shared window mgid: 587.390K Shared window ifc-protomax: 1.290M Shared window ifc-mpls: 7.981M Shared window ifc-ipv6: 7.212M Shared window ifc-ipv4: 11.286M Shared window infra_statsd: 3.402K Shared window im_rd: 1.104M Shared window im_db: 1.204M Shared window infra_ital: 67.316K Shared window netio_fwd: 292 Shared window vkg_bmp_adj: 211.371K Shared window aib: 623.375K Shared window rspp_ma: 3.351K Shared window im_rules: 293.308K Shared window aaa: 67.382K


OL-23591-02


Shared window pfm_node: 131.304K Shared window atc_cache: 35.359K Shared window spp: 619.312K Shared window qad: 134.707K Shared window pcie-server: 43.246K Total shared window: 37.931M Allocated Memory: 378.742M Program Text: 46.175M Program Data: 37.796M Program Stack: 16.539M

The show system verify start command starts the system verification process and the show system verify report generates the output from the system verification process. The output allows you to verify that the system parameters are as expected.

RP/0/RSP0/CPU0:router# show system verify start

Storing initial router status ... done.

The example output compares the system from the time the show system verify start command took the first snapshot to the snapshot taken of the system when the show system verify report command took the second snapshot and generated the comparison. If there are no changes, [OK] is displayed. If there are changes between the first and second snapshot, the specific change is noted and marked with [WARNING] or [FAIL].

RP/0/RSP0/CPU0:router# show system verify report

Getting current router status ... System Verification Report==========================- Verifying Memory Usage- Verified Memory Usage : [OK] - Verifying CPU Usage- Verified CPU Usage : [OK]

- Verifying Blocked Processes- Verified Blocked Processes : [OK] - Verifying Aborted Processes- Verified Aborted Processes : [OK] - Verifying Crashed Processes- Verified Crashed Processes : [OK]

- Verifying LC Status- Verified LC Status : [OK] - Verifying QNET StatusUnable to get current LC status info- Verified QNET Status : [FAIL]

- Verifying GSP Fabric Status- Verified GSP Fabric Status : [OK] - Verifying GSP Ethernet Status- Verified GSP Ethernet Status : [OK]

- Verifying POS interface Status- Verified POS interface Status : [OK] - Verifying TenGigE interface Status- Verified TenGigE interface Status : [OK]

- Verifying TCP statistics- Verified TCP statistics : [OK] - Verifying UDP statistics tcp_udp_raw WARNING messages for router UDP Packets sent has not increased during this period.


OL-23591-02


- Verified UDP statistics : [WARNING]- Verifying RAW statistics- Verified RAW statistics : [OK]

- Verifying RIB Status- Verified RIB Status : [OK] - Verifying CEF Status- Verified CEF Status : [OK] - Verifying CEF Consistency Status- Verified CEF Consistency Status : [OK] - Verifying BGP Status- Verified BGP Status : [OK] - Verifying ISIS Status- Verified ISIS Status : [OK] - Verifying OSPF Status- Verified OSPF Status : [OK]

- Verifying Syslog Messages- Verified Syslog Messages : [OK]

System may not be stable. Please look into WARNING messages.

The show interface brief command displays the usability status of the configured interfaces. Verify that all expected interfaces are listed. For an interface to be usable, both the interface hardware (Status) and line protocol must be up. The protocol is Up if the interface can provide two-way communication. The example output displays IP addresses, status, and protocol status for each interface. The output shows that all assigned interfaces (interfaces that are configured with IP addresses) have an interface hardware status and line protocol status of Up.

RP/0/RSP0/CPU0:router# show ipv4 interface brief

Interface IP-Address Status ProtocolBundle-Ether16 unassigned Up Up Bundle-Ether16.160 unassigned Up Up Bundle-Ether16.161 unassigned Up Up Bundle-Ether16.162 10.194.8.44 Up Up Bundle-Ether16.163 10.194.12.44 Up Up Loopback0 10.144.144.144 Up Up tunnel-te44190 10.144.144.144 Up Up tunnel-te44192 10.144.144.144 Up Up tunnel-te44194 10.144.144.144 Up Up tunnel-te44196 10.144.144.144 Up Up MgmtEth0/RSP0/CPU0/0 172.29.52.137 Up Up MgmtEth0/RSP0/CPU0/1 unassigned Shutdown Down GigabitEthernet0/1/0/0 unassigned Shutdown Down GigabitEthernet0/1/0/1 unassigned Shutdown Down GigabitEthernet0/1/0/2 10.147.4.44 Up Up GigabitEthernet0/1/0/3 unassigned Up Up GigabitEthernet0/1/0/3.160 unassigned Up Up GigabitEthernet0/1/0/3.161 unassigned Up Up GigabitEthernet0/1/0/3.185 unassigned Up Up GigabitEthernet0/1/0/3.189 unassigned Up Up GigabitEthernet0/1/0/3.215 unassigned Up Up GigabitEthernet0/1/0/4 unassigned Shutdown Down GigabitEthernet0/1/0/5 unassigned Shutdown Down GigabitEthernet0/1/0/6 unassigned Shutdown Down GigabitEthernet0/1/0/7 unassigned Up Up GigabitEthernet0/1/0/7.185 unassigned Up Up GigabitEthernet0/1/0/7.187 unassigned Up Up GigabitEthernet0/1/0/7.189 unassigned Up Up GigabitEthernet0/1/0/7.210 unassigned Up Up GigabitEthernet0/1/0/7.211 unassigned Up Up GigabitEthernet0/1/0/7.215 unassigned Up Up


OL-23591-02

Chapter 1 General Troubleshooting Procedures Troubleshooting the Backplane Ethernet Control System

GigabitEthernet0/1/0/8 10.146.4.44 Up Up GigabitEthernet0/1/0/9 unassigned Shutdown Down GigabitEthernet0/1/0/10 unassigned Shutdown Down GigabitEthernet0/1/0/11 unassigned Shutdown Down GigabitEthernet0/1/0/12 10.194.16.44 Up Up GigabitEthernet0/1/0/13 unassigned Shutdown Down GigabitEthernet0/1/0/14 unassigned Shutdown Down GigabitEthernet0/1/0/15 unassigned Shutdown Down GigabitEthernet0/1/0/16 unassigned Up Up GigabitEthernet0/1/0/17 unassigned Up Up GigabitEthernet0/1/0/18 10.194.4.44 Up Up GigabitEthernet0/1/0/19 unassigned Up Up GigabitEthernet0/1/0/19.2127 unassigned Up Up GigabitEthernet0/1/0/19.2130 unassigned Up Up GigabitEthernet0/1/0/20 unassigned Up Up GigabitEthernet0/1/0/20.2125 unassigned Up Up GigabitEthernet0/1/0/21 unassigned Shutdown Down GigabitEthernet0/1/0/22 unassigned Shutdown Down GigabitEthernet0/1/0/23 10.114.4.44 Up Up GigabitEthernet0/1/0/24 unassigned Shutdown Down GigabitEthernet0/1/0/25 unassigned Shutdown Down GigabitEthernet0/1/0/26 unassigned Shutdown Down GigabitEthernet0/1/0/27 10.145.4.44 Up Up GigabitEthernet0/1/0/28 unassigned Shutdown Down GigabitEthernet0/1/0/29 unassigned Shutdown Down GigabitEthernet0/1/0/30 unassigned Up Up GigabitEthernet0/1/0/30.215 unassigned Up Up GigabitEthernet0/1/0/31 unassigned Up Up GigabitEthernet0/1/0/32 unassigned Shutdown Down GigabitEthernet0/1/0/33 unassigned Shutdown Down GigabitEthernet0/1/0/34 unassigned Shutdown Down GigabitEthernet0/1/0/35 unassigned Shutdown Down GigabitEthernet0/1/0/36 unassigned Shutdown Down GigabitEthernet0/1/0/37 unassigned Shutdown Down GigabitEthernet0/1/0/38 unassigned Shutdown Down GigabitEthernet0/1/0/39 unassigned Shutdown Down POS0/2/0/0 unassigned Up Up POS0/2/0/1 unassigned Up Up TenGigE0/4/0/0 10.114.8.44 Up Up TenGigE0/4/0/1 unassigned Shutdown Down TenGigE0/4/0/2 unassigned Shutdown Down TenGigE0/4/0/3 unassigned Shutdown Down TenGigE0/4/0/4 unassigned Shutdown Down TenGigE0/4/0/5 unassigned Shutdown Down TenGigE0/4/0/6 unassigned Shutdown Down TenGigE0/4/0/7 unassigned Shutdown Down TenGigE0/6/0/0 unassigned Shutdown Down TenGigE0/6/0/1 unassigned Shutdown Down TenGigE0/6/0/2 unassigned Shutdown Down TenGigE0/6/0/3 unassigned Shutdown Down

Troubleshooting the Backplane Ethernet Control SystemThis section describes techniques that you can use to troubleshoot the control plane Ethernet network on routers using Cisco IOS XR software. The system control plane Ethernet network is used for processes on different devices to communicate for functions such as system device discovery, image transfers, heartbeat messages, alarms, and configuration management.


OL-23591-02


All devices in a system using Cisco IOS XR software connect to the system control plane Ethernet network, also called the Ethernet over backplane channel (EOBC). The control plane is provided using Gigabit Ethernet (GE) links between nodes. The GE links are internal to the chassis and cannot be removed.


OL-23591-02


Figure 1-1 shows the control plane Ethernet network (the dotted line in the drawing).

Figure 1-1 Cisco ASR 9000 Aggregation Services Router Control Ethernet Topology

RSP 0

FabricChip

FabricInterface

Chip

SystemTiming

GESwitch

CPU

VOQScheduler

40x1GELine Card

FPGA

FabricInterface

Chip

10 xSFP

10 xSFP

FPGA

10 xSFP

NPU NPU NPU NPU

10 xSFP

CPU

8x10GE 2:1Oversubscribed

Line Card

FPGA

FabricInterface

Chip

FPGA

CPU

4x10GELine Card

FPGA

FabricInterface

Chip

FPGA

GE PHYGE PHYGE PHY

CPU

RSP 1

Backplane

Backplane

FabricChip

FabricInterface

Chip

SystemTiming

GESwitch

CPU

VOQScheduler

Data Plane

Control Plane

10GE

XFP

10GE

XFP

10GE

XFP

10GE

XFP

10GE

XFP

10GE

XFP

10GE

XFP

10GE

XFP

10GE

XFP

10GE

XFP

10GE

XFP

10GE

XFP

2472

72

8x10GE 80GLine Rate Card

GEPHY

FabricInterface

Chip

FPGA

NPU

10GEXFP

NPU

10GEXFP

NPU

10GEXFP

NPU

10GEXFP

FabricInterface

Chip

FPGA

NPU

10GEXFP

NPU

10GEXFP

NPU

10GEXFP

NPU

10GEXFP

CPU

GESW

ToFPGAs

ToNPUs

2x10GE + 20x1GECombo Line Card

FPGAFPGA

NPU

10x

SFP

NPU

10x

SFP

NPU

10GEXFP

NPU

10GEXFP

GEPHY

CPU

GESW

ToFPGAs

ToNPUs

FabricInterface

Chip

NPU NPU NPU NPU NPU NPU NPU NPU


OL-23591-02


To verify and troubleshoot booting of the system control plane Ethernet network, perform the following procedure.

SUMMARY STEPS

1. show platform

2. show controllers backplane ethernet clients all location node-id

3. show controllers backplane ethernet clients 18 statistics location node-id

4. Contact Cisco Technical Support if the problem is not resolved.

DETAILED STEPS


Step 1 show platform

Example:

RP/0/RSP0/CPU0:router# show platform

Displays information about the status of cards and modules installed in the router.

Verify that the expected nodes display IOS XR RUN under the State column of the command output.

Step 2 show controllers backplane ethernet clients all location node-id

Example:RP/0/RSP0/CPU0:router# show controllers backplane ethernet clients all location 0/RSP0/CPU0

Displays information about all local client applications. Each row contains the client Ethernet server ID and the client process ID (PID).

The system allows client processes to send and receive packets over the control Ethernet. It uses client IDs to demultiplex packets that arrive at the node.

Two client IDs in the output are important for troubleshooting boot problems:

• Client Ethernet server ID 18—used for boot requests

• Client Ethernet server ID 22—used for heartbeats

Step 3 show controllers backplane ethernet clients 18 statistics location node-id

Example:RP/0/RSP0/CPU0:router# show controllers backplane ethernet clients 18 statistics location 0/RSP0/CPU0

Displays a list of client statistics for the specified client ID.

Check the values for:

• Packets input

• Packets delivered

If they contain values other than 0, boot requests have been received and replies have been sent (packets output).

If they contain values of 0, check the system control plane Ethernet network physical connectivity.

If there are no problems with the physical connectivity, contact Cisco Technical Support. For Cisco Technical Support contact information, see the “Obtaining Documentation and Submitting a Service Request” section on page xii.

Step 4 Contact Cisco Technical Support if the problem is not resolved.

If the problem is not resolved, contact Cisco Technical Support. For Cisco Technical Support contact information, see the “Obtaining Documentation and Submitting a Service Request” section on page xii.


OL-23591-02


Examples

The output from the show platform command indicates that all expected nodes are in the run state. If all nodes in the system are active, the cards should be in the IOS XR RUN and the SPAs should be in the OK state. The example output shows that all expected nodes are in the run state.


Node Type State Config State-----------------------------------------------------------------------------0/RSP0/CPU0 A9K-RSP-4G(Active) IOS XR RUN PWR,NSHUT,MON0/1/CPU0 A9K-40GE-B IOS XR RUN PWR,NSHUT,MON0/2/CPU0 A9K-SIP-700 IOS XR RUN PWR,NSHUT,MON0/2/0 SPA-2XOC48POS/RPR OK PWR,NSHUT,MON0/3/CPU0 A9K-2T20GE-B IN-RESET PWR,NSHUT,MON0/4/CPU0 A9K-8T/4-B IOS XR RUN PWR,NSHUT,MON0/6/CPU0 A9K-4T-B IOS XR RUN PWR,NSHUT,MON

The following example shows the current state of each Ethernet server client.

RP/0/RSP0/CPU0:router# show controllers backplane ethernet clients all location 0/RSP0/CPU0

Intf Client ethernet Client Description Name server id Process Id------------------------------------------------------------------------------GE0_RSP0_CPU0 1 110639 QNX network manager 2 221272 Group services 3 0 Reserved for Attach 4 221274 Plugin controller 5 0 Designated SC 6 0 ASR9K H/W diags 7 221279 IP packet handler 8 217149 Redundancy controller 9 0 ASR9K Virtual console 10 110638 ASR9K Virtual terminal 11 49196 Control ethernet echo 12 0 Control eth echo reply 13 221274 Card Configuration Protocol 14 0 Reserved for Attach 15 0 Chassis controller 16 0 Forwarding driver 17 0 MBI hello 18 110640 MBI Boot Server Source 19 0 HSR ES client 20 0 Packets for ethernet server 21 0 For Diag application 22 233589 heartbeat request 23 0 heartbeat reply 24 221275 Async IPC client 25 0 Test application 1 26 0 Test application 2 27 0 Test client out-of-band

The following example shows that there are 18 nodes in the run state, which means that 12 boot requests have been received by eth_server and 12 replies have been sent:

RP/0/RSP0/CPU0:router# show controllers backplane ethernet clients 18 statistics location 0/RSP1/CPU0

Client ShelfMgr, ES Client Id 18, PID 53338 running on FastEthernet0_RSP0_0 12 packets input, 8676 bytes 12 packets delivered, 8676 bytes 0 packets discarded (0 bytes) in garbage collection


OL-23591-02

Chapter 1 General Troubleshooting Procedures Basic Cisco IOS XR Verification and Troubleshooting Commands

0 (0 bytes) unicast packets filtered 0 (0 bytes) multicast packets filtered 0 (0 bytes) buffer mgmt policy discards 0 (0 bytes) locking error discards 12 packets output, 8676 bytes, 0 could not be transmitted

Basic Cisco IOS XR Verification and Troubleshooting Commands

The following commands are used to collect information to aid in verifying the system and troubleshooting problems:

• man Command, page 1-46

• describe Command, page 1-49

• show platform Command, page 1-49

• top Command, page 1-50

• show context Command, page 1-50

• show users Command, page 1-52

• show history Command, page 1-52

• show configuration Command, page 1-53

man CommandThe man command provides online help for standard Cisco IOS XR CLI commands using manual (man) pages. The command is used to display the manual pages for a specific command based on the command name, a feature, or a keyword. Each man page contains the command name, syntax, command mode, usage, examples, and related commands.

Note To run the man command, you must have the Cisco IOS XR Documentation Package, “asr9k-doc.pie-4.0.0, .man pages for Cisco IOS XR software on the Cisco ASR 9000 Series Router chassis,” loaded. If you are running a release later than 4.0.0, the package installation envelope (PIE) name might be different. For the appropriate PIE name and an explanation of PIE installation, see the “Upgrading Cisco IOS XR Software” section of the Release Notes document for the IOS XR version you are running

The following example shows the output from the man command show users command.

RP/0/RSP0/CPU0:router# man command show usersTue Sep 14 14:39:16.409 DSTBuilding index table...Total Number of Command Entries:2726 [OK]

DESCRIPTION

Displays information about the active lines on the router.

To display information about the active lines on the router, use the show users


OL-23591-02

http://www.cisco.com/en/US/products/ps9853/prod_release_notes_list.html


command in EXEC mode.

show users

SYNTAX DESCRIPTION

This command has no arguments or keywords.

COMMAND DEFAULT

No default behavior or values

COMMAND MODES

EXEC

COMMAND HISTORY

Release

Modification

Release 3.7.2

This command was introduced.

USAGE GUIDELINES

To use this command, you must be in a user group associated with a task group that includes the proper task IDs. If you suspect user group assignment is preventing you from using a command, contact your AAA administrator for assistance.

Use the show users command to display the line number, connection name, idle time, hosts, and terminal location. An asterisk (*) indicates the current terminal session.

===================================================================================

Note:

To display all user groups and task IDs associated with the currently logged-in user, use the show user command in EXEC mode. See the Authentication, Authorization, and Accounting Commands on Cisco IOS XR Software module in Cisco^B^`ASR^B^`9000 Series Aggregation Services Router System Security Command Reference.

===================================================================================

TASK ID

Task ID

Operations

tty-access

read

EXAMPLES

The following example shows sample output identifying an active vty terminal session:


OL-23591-02


* * * * * * * * * * * * * * START OF LISTING * * * * * * * * * * * * * * RP/0/RSP0/CPU0:router# show users Line User Service Conns Idle Location con0_RSP0_CPU0 cisco hardware 0 18:33:48 vty0 cisco telnet 0 00:30:36 10.33.54.132 * vty1 cisco telnet 0 00:00:00 10.33.54.132 * * * * * * * * * * * * * * END OF LISTING * * * * * * * * * * * * * * *

Table 1 describes the significant fields shown in the display.

show users Field Descriptions

Field

Description

Line

All current connections. An asterisk (*) indicates the active connection.

User Username of the user logged into the line.

Service

Physical or remote login service used.

Conns

Number of outgoing connections.

Idle

Interval (in hours:minutes:seconds) since last keystroke.

Location

IP address of remote login host. For local (physical) terminal connections, this field is blank.

RELATED COMMANDS

Command

Description

show line Displays the parameters of terminal lines.

Displays the parameters of a terminal line.

show user

Displays all user groups and task IDs associated with the currently logged-in user.


OL-23591-02


describe CommandThe describe command provides a preview of a command without actually implementing it. This command lists information about the package, component, and task ID for a specific command. You must be in the appropriate configuration mode for the specific command. For example, to display the package, component, and task ID information for the router bgp 1 command, you must be in global configuration mode.

The following example shows the output from the describe router bgp 1 command.

RP/0/RSP0/CPU0:router(config)#describe router bgp 1

Package: iosxr-routing iosxr-routing V4.0.0[DT_IMAGE] IOS-XR Routing Package Definition Vendor : Cisco Systems Desc : IOS-XR Routing Package Definition Build : Built on Wed Sep 8 16:10:14 DST 2010 Source : By router-021 in /files/4.0.0.DT_IMAGE/asr9k/workspace fo8 Card(s): RP, NP24-4x10GE, NP24-40x1GE, NP40-40x1GE, NP40-4x10GE, NP40-8x10GE, NP40-2_20_COMBO, NP80-8x10GE, NP80-16x10GE, A9K-SIP-700, A9K-SIP-500 Restart information: Default: parallel impacted processes restart Size Compressed/Uncompressed: 8556KB/22MB (37%)

Component: ipv4-bgp V[ci-401/18] IPv4 Border Gateway Protocol (BGP)

User needs ALL of the following taskids:

bgp (READ WRITE)

show platform CommandThe show platform command displays a high level overview of the entire physical system. Use the show platform command in administration mode to display a summary of the nodes in the system, including node type and status.

The following example shows the output from the show platform command in administration mode.

RP/0/RSP0/CPU0:router(admin)#show platformTue Sep 14 14:52:52.558 DSTNode Type State Config State-----------------------------------------------------------------------------0/RSP0/CPU0 A9K-RSP-4G(Active) IOS XR RUN PWR,NSHUT,MON0/FT0/SP FAN TRAY READY 0/FT1/SP FAN TRAY READY 0/1/CPU0 A9K-40GE-B IOS XR RUN PWR,NSHUT,MON0/2/CPU0 A9K-SIP-700 IOS XR RUN PWR,NSHUT,MON0/2/0 SPA-2XOC48POS/RPR OK PWR,NSHUT,MON0/3/CPU0 A9K-2T20GE-B IN-RESET PWR,NSHUT,MON0/4/CPU0 A9K-8T/4-B IOS XR RUN PWR,NSHUT,MON0/6/CPU0 A9K-4T-B IOS XR RUN PWR,NSHUT,MON0/PM0/SP A9K-3KW-AC READY PWR,NSHUT,MON0/PM1/SP A9K-3KW-AC READY PWR,NSHUT,MON0/PM2/SP A9K-3KW-AC READY PWR,NSHUT,MON


OL-23591-02


top CommandThe top command is used to monitor CPU usage on the system through interactive process statistics.

The following example show the output from the top command.

RP/0/RSP0/CPU0:router# top

Computing times...224 processes; 803 threads;CPU states: 93.6% idle, 3.6% user, 2.7% kernelMemory: 4096M total, 3504M avail, page size 4K

JID TID LAST_CPU PRI STATE HH:MM:SS CPU COMMAND 91 1 0 10 Rcv 0:16:29 2.25% spp 256 10 0 10 Rcv 0:19:43 2.16% netio 340 10 0 10 Rcv 0:03:21 0.13% udp 294 1 1 10 Rcv 0:03:24 0.09% sc 1 12 1 10 Rcv 0:00:03 0.09% procnto-600-smp-instr 65816 1 0 10 Rply 0:00:00 0.07% top 1 11 1 10 Run 0:00:19 0.05% procnto-600-smp-instr 60 5 1 10 Rcv 0:00:43 0.02% eth_server 256 11 1 10 Rcv 0:00:15 0.02% netio 340 14 1 10 Rcv 0:01:06 0.01% udp

Press q to exit the command.

show context CommandThe show context command displays core dump context information for the last ten core dumps. The command output is used for post-analysis in the debugging of processes (determine if any process crashes have occurred).

If there are no crashed processes, the show context command displays no output for each node. The following example shows the output of the show context command with no crashed processes.


node: node0_1_CPU0------------------------------------------------------------------

node: node0_6_CPU0------------------------------------------------------------------

node: node0_RSP0_CPU0------------------------------------------------------------------

node: node0_RSP1_CPU0------------------------------------------------------------------

The following example shows the output from the show context command where there is a crashed process.


node: node0_1_CPU0------------------------------------------------------------------

Crashed pid = 61524 (pkg/bin/tcam_mgr)Crashed tid = 1Crash time: Wed Apr 05, 2006: 18:27:26


OL-23591-02


Core for process at harddisk:/dumper/first.tcam_mgr.abort.node0_1_CPU0.ppc.Z

Stack Trace#0 0xfc1d3fa0#1 0xfc1c6340#2 0xfc1c5364#3 0xfc1c542c#4 0x48210930#5 0x482110b8#6 0x48212ba4#7 0x48203dd8#8 0x4820c61c#9 0xfc1557ec#10 0xfc15573c#11 0xfc152fb8#12 0x4820d140 Registers info r0 r1 r2 r3 R0 00000000 481ff7b0 4824a55c 00000000 r4 r5 r6 r7 R4 0000f054 00000001 00000006 00000000 r8 r9 r10 r11 R8 00000000 fc220000 481fffc0 00000000 r12 r13 r14 r15 R12 4823be90 4824a4a0 48230000 00000000 r16 r17 r18 r19 R16 00000048 00000001 00000019 48256520 r20 r21 r22 r23 R20 00000000 00000000 00000003 00000045 r24 r25 r26 r27 R24 00000003 00000000 00000003 4825dc34 r28 r29 r30 r31 R28 00000006 0000f054 48254064 481ff810 cnt lr msr pc R32 00000000 fc1c6340 0000d932 fc1d3fa0 cnd xer R36 28004024 00000008

DLL Info DLL path Text addr. Text size Data addr. Data size Version/hfr-os-3.3.90/lib/libinfra.dll 0xfc142000 0x00034200 0xfc1343b8 0x00000bbc 0/lib/libc.dll 0xfc1a8000 0x00079dd8 0xfc222000 0x00002000 0

Crash Package InfomationPackage: hfr-mgbl, Source: By edde-bld1 in /vws/aga/production/3.3.90.1I/hfr/workspace for c2.95.3-p8Package: hfr-mcast, Source: By edde-bld1 in /vws/aga/production/3.3.90.1I/hfr/workspace for c2.95.3-p8Package: hfr-mpls, Source: By edde-bld1 in /vws/aga/production/3.3.90.1I/hfr/workspace for c2.95.3-p8Package: hfr-rout, Source: By edde-bld1 in /vws/aga/production/3.3.90.1I/hfr/workspace for c2.95.3-p8Package: hfr-k9sec, Source: By edde-bld1 in /vws/aga/production/3.3.90.1I/hfr/workspace for c2.95.3-p8Package: hfr-lc, Source: By edde-bld1 in /vws/aga/production/3.3.90.1I/hfr/workspace for c2.95.3-p8Package: hfr-fwdg, Source: By edde-bld1 in /vws/aga/production/3.3.90.1I/hfr/workspace for c2.95.3-p8Package: hfr-admin, Source: By edde-bld1 in /vws/aga/production/3.3.90.1I/hfr/workspace for c2.95.3-p8Package: hfr-base, Source: By edde-bld1 in /vws/aga/production/3.3.90.1I/hfr/workspace for c2.95.3-p8Package: hfr-os-mbi, Source: By edde-bld1 in /vws/aga/production/3.3.90.1I/hfr/w


OL-23591-02


orkspace for c2.95.3-p8

node: node0_6_CPU0------------------------------------------------------------------

node: node0_RSP0_CPU0------------------------------------------------------------------

node: node0_RSP1_CPU0------------------------------------------------------------------

Use the show context command to locate the core dump file path. For example, the core dump file path shown in the command output is: harddisk:/dumper/first.tcam_mgr.abort.node0_1_CPU0.ppc.Z. The command output shows a crash on a node. The process is pkg/bin/tcam_mgr.

Collect the following information and send it to Cisco Technical Support. For Cisco Technical Support contact information, see the “Obtaining Documentation and Submitting a Service Request” section on page xii.

• ppc.Z file—This file contains the binary core dump information. Use the path listed in the command output to copy the contents of the ppc.Z file. The path shown in the command output is: harddisk:/dumper/first.tcam_mgr.abort.node0_1_CPU0.ppc.Z

• ppc.txt file—This file contains content on the core dump similar to the show context command output. Use the path listed in the command output to copy the contents of the ppc.txt file. The path shown in the command output is: harddisk:/dumper/first.tcam_mgr.abort.node0_1_CPU0.ppc.txt

• Collect the show version or show install active command output.

show users CommandThe show users command displays information on active lines on the router including the line number, user, service, number of connections, idle time, and remote terminal location. An asterisk (*) indicates the current terminal session.

The following example shows the output from the show users command.

RP/0/RSP0/CPU0:router# show users

Line User Service Conns Idle Location* vty0 User_A telnet 0 00:00:00 161.44.1925 vty1 User-B telnet 0 00:00:03 161.44.1929

show history CommandThe show history command displays a history of the command entered for the current command mode. You can enter the show history command to display a history of commands entered in EXEC, ADMIN, or CONFIG mode.

Examples

RP/0/RSP0/CPU0:router# show historyRP/0/RSP0/CPU0:router(admin)# show historyRP/0/RSP0/CPU0:router(config)# show history

The following example shows the output from the show history command in EXEC mode:


OL-23591-02


RP/0/RSP0/CPU0:router# show history

RP/0/RSP0/CPU0:router# show history Thu Oct 28 14:20:50.328 DST show vrrp interface gigabitEthernet 0/1/0/0 show vrrp interface brief show vrrp brief show vrrp detail show vrrp interface gigabitEthernet 0/1/0/1 show vrrp interface gigabitEthernet 0/1/0/1 statistics all

The detailed history provides a timestamp also:

RP/0/RSP0/CPU0:router# show history detail Thu Oct 28 14:26:06.199 DST 1 Thu Oct 28 14:02:25.310 show vrrp interface gigabitEthernet 0/1/0/0 2 Thu Oct 28 14:03:34.854 show vrrp interface brief 3 Thu Oct 28 14:04:02.042 show vrrp brief 4 Thu Oct 28 14:04:08.167 show vrrp detail 5 Thu Oct 28 14:08:25.180 show vrrp interface gigabitEthernet 0/1/0/1 6 Thu Oct 28 14:09:03.402 show vrrp interface gigabitEthernet 0/1/0/1 statistics all

show configuration CommandThe show configuration command displays details on uncommitted configuration changes, that is, the commands you are about to commit. You can enter the show configuration command to display the changes in EXEC, ADMIN, or CONFIG mode.

Use the show configuration command with the running keyword to display the running (active) configuration.

Prior to committing the target configuration, use the show configuration command with the merge keyword from any configuration mode to display the result of merging the target configuration with the running configuration.

Examples

RP/0/RSP0/CPU0:router# show configuration RP/0/RSP0/CPU0:router# show configuration running-configRP/0/RSP0/CPU0:router(admin)# show configuration runningRP/0/RSP0/CPU0:router(config)# show configuration RP/0/RSP0/CPU0:router(config)# show configuration runningRP/0/RSP0/CPU0:router(config)# show configuration merge

In this example, the show configuration command displays uncommitted changes made during a configuration session:

RP/0/RSP0/CPU0:router# configure RP/0/RSP0/CPU0:router(config)# interface tengige0/3/0/3 RP/0/RSP0/CPU0:router(config-if)# description faq RP/0/RSP0/CPU0:router(config-if)# ipv4 address 10.10.11.20 255.0.0.0 RP/0/RSP0/CPU0:router(config-if)# show configuration Building configuration... interface TenGigE0/3/0/3 description faq ipv4 address 10.10.11.20 255.0.0.0 end


OL-23591-02

Chapter 1 General Troubleshooting Procedures Displaying ASIC Errors

The following example shows sample output from the show configuration command with the optional merge keyword. The command is entered during a configuration session. The output displays the result of merging the target and running configuration, without committing the changes.

RP/0/RSP0/CPU0:router# configure RP/0/RSP0/CPU0:router(config)# interface tengige0/3/0/3RP/0/RSP0/CPU0:router(config-if)# description faq RP/0/RSP0/CPU0:router(config-if)# ipv4 address 10.10.11.20 255.0.0.0 RP/0/RSP0/CPU0:router(config-if)# show configuration merge Building configuration... hostname router interface TenGigE0/0/0/0 ipv4 address 10.2.3.4 255.0.0.0 exitinterface TenGigE0/3/0/3 description faq ipv4 address 10.1.1.1 255.0.0.0 shutdown end

Displaying ASIC ErrorsThe following example shows how to display ASIC errors for each ASIC in a SIP-700 optical LC. If an error is displayed, dump the individual ASIC instance number to obtain details on the ASIC error.

RP/0/RSP0/CPU0:router# show asic-errors all location 0/6/CPU0Thu Oct 21 19:00:54.178 DST************************************************************* Fia ASIC Error Summary *************************************************************Instance : 0Number of nodes : 0SBE error count : 0MBE error count : 0Parity error count : 0CRC error count : 0Generic error count : 0Reset error count : 0--------------------

************************************************************* Mace ASIC Error Summary *************************************************************Instance : 0Number of nodes : 0SBE error count : 0MBE error count : 0Parity error count : 0CRC error count : 0Generic error count : 0Reset error count : 0--------------------

************************************************************* Prm_np ASIC Error Summary *************************************************************Instance : 0


OL-23591-02

Chapter 1 General Troubleshooting Procedures Displaying ASIC Errors

Number of nodes : 3SBE error count : 0MBE error count : 0Parity error count : 0CRC error count : 0Generic error count : 0Reset error count : 0--------------------Instance : 1Number of nodes : 3SBE error count : 0MBE error count : 0Parity error count : 0CRC error count : 0Generic error count : 0Reset error count : 0--------------------Instance : 2Number of nodes : 3SBE error count : 0MBE error count : 0Parity error count : 0CRC error count : 0Generic error count : 0Reset error count : 0--------------------Instance : 3Number of nodes : 3SBE error count : 0MBE error count : 0Parity error count : 0CRC error count : 0Generic error count : 0Reset error count : 0--------------------

The following ASIC error types are supported:

• FIA (Fabric Interface ASIC)

• Mace ASIC

• Prm_np ASIC

The following ASIC error classifications are supported:

• Single Bit Errors (SBE)—Correctable ECC protected single bit errors in external or internal memory.

Not reported to PM on each occurrence and reported to the platform manager (PM) as Minor when software threshold rate is exceeded. Report alarm using Alarm Logging, and Debugging Event Management System (ALDEMS).

Error data:

– Address—Address that encountered the SBE

– Syndrome—Syndrome if available

• Multiple Bit Errors—Uncorrectable multiple bit error in memory.

Reported to PM as Major and ALDEMS for each occurrence.

Error data:

– Address—Address that encountered the SBE.


OL-23591-02

Chapter 1 General Troubleshooting Procedures Using Trace Commands

– Data—Actual error data.

• PARITY Errors—Parity error in all applicable memory.

Reported to PM as Major.

• Cyclic redundancy check (CRC) Errors—CRC errors in EIO other links.

Not reported for each occurrence. When the threshold is reached it is reported as Major to the PM.

• GENERIC Errors—Errors that do not fall under any of the other classifications.

Threshold and alarm reporting is done.

• RESET Errors—Logged for each reset instance of the ASIC.

Reported to PM when threshold is exceeded.

Error data:

– Interrupt status—Interrupt status bits due to ASIC reset.

– Halt status—Halt status bits.

– Reset node key—Key for the error node that causes the reset.

– Time—Reset time.

The following ASIC error fault severities are supported:

• Critical—Affected component is unusable or card is reset if no redundant card exists.

• Major—Partially service affecting fault, causing the card to run in degraded mode. For redundant cards, consider performing a switchover.

• Minor—Non-service affecting fault.

• OK—No fault.

Using Trace CommandsTrace commands provide an ‘always on’ debug feature. Many major functions in Cisco IOS XR software have “trace” functionality to show the last actions it conducted allowing you to analyze function events. Use the show trace commands to display the trace data for a specific feature or process. Use the ? in the CLI to determine if a command has the trace keyword. The following example shows that the show arp command has the trace keyword.

RP/0/RSP0/CPU0:router# show arp ? A.B.C.D IP address or hostname of ARP entry BVI Bridge-Group Virtual Interface Bundle-Ether Aggregated Ethernet interface(s) GigabitEthernet GigabitEthernet/IEEE 802.3 interface(s) H.H.H 48-bit hardware address of ARP entry MgmtEth Ethernet/IEEE 802.3 interface(s) TenGigE TenGigabitEthernet/IEEE 802.3 interface(s) api-stats Show ARP API statistics data client ARP Client show commands dagr Show Direct Attached Gateway Redundancy group information idb Show the internal ARP interface data block location specify a node name resolution Show the ARP resolution history trace Show trace data for the ARP component traffic ARP traffic statistics vrf Specify a VRF | Output Modifiers <cr>


OL-23591-02

Chapter 1 General Troubleshooting Procedures MIB Location

The following example shows the last 20 events in the address resolution protocol (ARP) table.

RP/0/RSP0/CPU0:router# show arp trace tailf last 20

1349 wrapping entries (2048 possible, 0 filtered, 1349 total)Apr 19 09:52:29.857 ipv4_arp/arp 0/RSP0/CPU0 t1 ARP-TABLE: creating incomplete entry for address: 172.18.105.255Apr 19 09:52:34.501 ipv4_arp/arp 0/RSP0/CPU0 t1 ARP-TABLE: address resolution failed for 172.18.105.255Apr 19 09:52:41.856 ipv4_arp/arp 0/RSP0/CPU0 t1 ARP-TABLE: received address resolution request for 172.18.105.255Apr 19 09:52:46.324 ipv4_arp/arp 0/RSP0/CPU0 t1 ARP-TABLE: address resolution failed for 172.18.105.255Apr 19 09:52:59.979 ipv4_arp/arp 0/RSP0/CPU0 t1 ARP-TABLE: entry 172.18.105.255: deleted from tableApr 19 09:59:37.463 ipv4_arp/arp 0/RSP0/CPU0 t1 ARP-TABLE: received address resolution request for 172.18.105.255Apr 19 09:59:37.463 ipv4_arp/arp 0/RSP0/CPU0 t1 ARP-TABLE: creating incomplete entry for address: 172.18.105.255Apr 19 09:59:39.515 ipv4_arp/arp 0/RSP0/CPU0 t1 ARP-TABLE: received address resolution request for 172.18.105.255Apr 19 09:59:42.082 ipv4_arp/arp 0/RSP0/CPU0 t1 ARP-TABLE: address resolution failed for 172.18.105.255Apr 19 09:59:45.007 ipv4_arp/arp 0/RSP0/CPU0 t1 ARP-TABLE: entry 172.18.105.255: deleted from tableApr 19 09:59:50.101 ipv4_arp/arp 0/RSP0/CPU0 t1 ARP-TABLE: received address resolution request for 172.18.105.255Apr 19 09:59:50.101 ipv4_arp/arp 0/RSP0/CPU0 t1 ARP-TABLE: creating incomplete entry for address: 172.18.105.255Apr 19 09:59:54.820 ipv4_arp/arp 0/RSP0/CPU0 t1 ARP-TABLE: address resolution failed for 172.18.105.255Apr 19 10:00:00.008 ipv4_arp/arp 0/RSP0/CPU0 t1 ARP-TABLE: entry 172.18.105.255: deleted from tableApr 19 10:04:11.675 ipv4_arp/arp 0/RSP0/CPU0 t1 ARP-TABLE: received address resolution request for 172.18.105.255Apr 19 10:04:11.675 ipv4_arp/arp 0/RSP0/CPU0 t1 ARP-TABLE: creating incomplete entry for address: 172.18.105.255Apr 19 10:04:16.272 ipv4_arp/arp 0/RSP0/CPU0 t1 ARP-TABLE: address resolution failed for 172.18.105.255Apr 19 10:04:30.028 ipv4_arp/arp 0/RSP0/CPU0 t1 ARP-TABLE: entry 172.18.105.255: deleted from tableApr 19 10:04:44.097 ipv4_arp/arp 0/RSP0/CPU0 t1 ARP-TABLE: received address resolution request for 172.18.105.255Apr 19 10:04:44.097 ipv4_arp/arp 0/RSP0/CPU0 t1 ARP-TABLE: creating incomplete entry for address: 172.18.105.255Apr 19 10:04:48.810 ipv4_arp/arp 0/RSP0/CPU0 t1 ARP-TABLE: address resolution failed for 172.18.105.255

MIB LocationTo locate and download MIBs, use the Cisco MIB Locator found at the following URL and choose a platform under the Cisco Access Products menu: http://cisco.com/public/sw-center/netmgmt/cmtk/mibs.shtml.


OL-23591-02

Chapter 1 General Troubleshooting Procedures Gathering Information Before You Call Cisco TAC

Gathering Information Before You Call Cisco TACWe recommend that you have a system of maintaining and accessing detailed information about your network and ASR 9000 router, including system hardware and software, network diagrams, and captured output from commands. For additional details, see the “Prerequisite Documentation for Troubleshooting” section on page 1-1.

Before calling the Cisco Technical Assistance Center (TAC), you should gather the information described in the following sections, if possible. This information will be helpful for troubleshooting.

Caution We strongy recommend that, if possible, you gather the information described in this section before you reset any cards. If you reset cards before you gather information, the system erases the information and it will be more difficult to diagnose and repair the problem.

• Gathering Information about Crashes and Core Dumps, page 1-58

• Capturing Logs, page 1-58

• Using Debug Commands, page 1-59

• Using Diagnostic Commands, page 1-59

• Commands Used to Display Process and Thread Details, page 1-59

Timesaver Before contacting Cisco Technical Support, review the information provided at the following URL: http://www.cisco.com/web/services/ts/access/index.html.

For information on contacting Cisco Technical Support, see the “Obtaining Documentation and Submitting a Service Request” section on page xii.

Gathering Information about Crashes and Core DumpsGather system information with the following commands:

• show install active summary

• show version

• show run

• show context

• show log

• show inventory

• show diagnostics

Upload any core dumps that were written to disk0, disk1, or harddisk directories.

Capturing LogsSee the “Prerequisite Documentation for Troubleshooting” section on page 1-1 in Chapter 1, “General Troubleshooting Procedures,” for information on collecting current system information.

Collect system information using the following commands:


OL-23591-02

http://www.cisco.com/web/services/ts/access/index.html


• show tech-support—Displays system information for Cisco Technical Support and includes a traditional dump of the configuration and show command outputs.

Note Some tech-support commands require the user to be assigned the cisco-support task ID. For a mapping of commands to task IDs and allowed operations, see Cisco IOS XR Task ID Reference Guide.

• show logging—Displays the contents of the logging buffers

• show system verify—Displays system verification information

Using Debug CommandsFor details on using debug commands, see Cisco IOS XR Using Debug Guide.

Using Diagnostic CommandsThe Cisco ASR 9000 Aggregation Series Router diagnostic tests verify control Ethernet and fabric data paths. If a diagnostic tests fails, it indicates a bad data path. The integrity of the covered data paths is verified when the diagnostic tests pass.

The diagnostic tests generally test data paths between multiple nodes, therefore error reports need to be analyzed to narrow down the possible points of failure in a system.

All diagnostic tests run within the 1 second to 1 minute range.

Note On the Cisco ASR 9000 Aggregation Services Router, only online diagnostics are supported.

To run a specified on-demand diagnostic test or series of tests, use the diagnostic start location command.

Example:

RP/0/RSP0/CPU0:router(admin)# diagnostic start location 0/RSP1/CPU0 test 1

RP/0/RSP0/CPU0:router(admin)# diagnostic stop location 0/RSP1/CPU0

For details on the diagnostic commands, see Cisco ASR 9000 Series Aggregation Services Router Interface and Hardware Component Command Reference.

Commands Used to Display Process and Thread DetailsFor details on processes and threads, see the “Understanding Processes and Threads” section in Cisco ASR 9000 Aggregation Services Router Router Getting Started Guide.


OL-23591-02



OL-23591-02


C H A P T E R 2
Verifying and Troubleshooting Interface Status
This chapter describes how to verify that interfaces are up (operational), and how to troubleshoot problems on interfaces. It includes the following sections:

• Verifying and Troubleshooting Gigabit Ethernet Interfaces, page 2-61

• Verifying and Troubleshooting Pluggable Optical Line Card Interfaces, page 2-68

Verifying and Troubleshooting Gigabit Ethernet InterfacesTo troubleshoot gigabit Ethernet interfaces, perform the following procedure.

SUMMARY STEPS

1. show interfaces type instance

2. show controllers interface-type interface-instance stats

3. show netio idb interface-type interface-instance

4. show hw-module subslot address counters or show hw-module subslot counters framer

5. If your system has pluggable optical line cards, perform additional troubleshooting steps applicable to these cards.



Chapter 2 Verifying and Troubleshooting Interface Status Verifying and Troubleshooting Gigabit Ethernet Interfaces

DETAILED STEPS

The following example shows POS 0/0/1/0 with no input drop counters.

RP/0/RSP0/CPU0:router# show interfaces pos 0/0/1/0

POS0/0/1/0 is up, line protocol is up Hardware is Packet over SONET/SDH Internet address is 172.18.140.1/24 MTU 4474 bytes, BW 155520 Kbit reliability 255/255, txload 1/255, rxload 1/255


Step 1 show interfaces type instance

Example:RP/0/RSP0/CPU0:router# show interfaces gigabitEthernet 0/0/0/0

Displays statistics for all interfaces configured on the specified node. Check for interface errors and conflicting configurations, such as IP addresses defined incorrectly on interfaces.

Step 2 show controllers interface-type interface-instance stats

Example:RP/0/RSP0/CPU0:router# show controllers gigabitEthernet 0/1/0/23 stats

Displays interface controller status and configuration statistics for the specified node. Check for input drops.

If input drops are found, contact Cisco Technical Support. For Cisco Technical Support contact information, see the “Obtaining Documentation and Submitting a Service Request” section on page xii in the Preface.

Step 3 show netio idb interface-type interface-instance

Example:RP/0/RSP0/CPU0:router# show netio idb gigabitEthernet 0/0/0/0

Displays network input and output information for a specified node.

Check the software counters for the interface. Under Chains and Protocol chains in the output check if any drops occurred at a particular point in the Encap or Decap of the packet. The drops are displayed in the column on the right, showing drop packets and bytes for each step in the chain.

Step 4 show hw-module subslot address counters

or

show hw-module subslot counters framer

Example:RP/0/RSP0/CPU0:router# show hw-module subslot 0/0/0 counters

or

RP/0/RSP0/CPU0:router# show hw-module subslot counters framer

Use these commands to check for status and interface drops on shared port adapters (SPAs).

Step 5 If your system has pluggable optical line cards, perform the additional troubleshooting steps applicable to these cards. See “Verifying and Troubleshooting Pluggable Optical Line Card Interfaces” section on page 2-68.

Check for interface drops on the pluggable optical line cards.

Step 6 Contact Cisco Technical Support. For Cisco Technical Support contact information, see the “Obtaining Documentation and Submitting a Service Request” section on page xii in the Preface.


OL-23591-02


Encapsulation HDLC, crc 32, controller loopback not set, keepalive set (10 sec) Last clearing of "show interface" counters never 5 minute input rate 3000 bits/sec, 0 packets/sec 5 minute output rate 3000 bits/sec, 0 packets/sec 199794 packets input, 222359750 bytes, 0 total input drops 0 drops for unrecognized upper-level protocol 0 runts, 0 giants, 0 throttles, 0 parity 0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort 89911 packets output, 213413210 bytes, 0 total output drops 0 output errors, 0 underruns, 0 applique, 0 resets 0 output buffer failures, 0 output buffers swapped out 0 carrier transitions

RP/0/RSP0/CPU0:router# show controllers gigabitEthernet 0/1/0/2 stats Wed Nov 3 12:41:54.201 DSTStatistics for interface GigabitEthernet0/1/0/2 (cached values):

Ingress: Input total bytes = 0 Input good bytes = 0

Input total packets = 0 Input 802.1Q frames = 0 Input pause frames = 0 Input pkts 64 bytes = 0 Input pkts 65-127 bytes = 0 Input pkts 128-255 bytes = 0 Input pkts 256-511 bytes = 0 Input pkts 512-1023 bytes = 0 Input pkts 1024-1518 bytes = 0 Input pkts 1519-Max bytes = 0

Input good pkts = 0 Input unicast pkts = 0 Input multicast pkts = 0 Input broadcast pkts = 0

Input drop overrun = 0 Input drop abort = 0 Input drop invalid VLAN = 0 Input drop invalid DMAC = 0 Input drop invalid encap = 0 Input drop other = 0

Input error giant = 0 Input error runt = 0 Input error jabbers = 0 Input error fragments = 0 Input error CRC = 0 Input error collisions = 0 Input error symbol = 0 Input error other = 0

Input MIB giant = 0 Input MIB jabber = 0 Input MIB CRC = 0

Egress: Output total bytes = 0 Output good bytes = 0

Output total packets = 0 Output 802.1Q frames = 0


OL-23591-02


Output pause frames = 0 Output pkts 64 bytes = 0 Output pkts 65-127 bytes = 0 Output pkts 128-255 bytes = 0 Output pkts 256-511 bytes = 0 Output pkts 512-1023 bytes = 0 Output pkts 1024-1518 bytes = 0 Output pkts 1519-Max bytes = 0

Output good pkts = 0 Output unicast pkts = 0 Output multicast pkts = 0 Output broadcast pkts = 0

Output drop underrun = 0 Output drop abort = 0 Output drop other = 0

Output error other = 0

RP/0/RSP0/CPU0:router# show netio idb pos 0/0/1/0

POS0/0/1/0 (handle: 0x010800a0, nodeid:0x1) netio idb:---------------------------------name: POS0_0_1_0interface handle: 0x010800a0interface global index: 2physical media type: 14dchain ptr: <0x482dd660>echain ptr: <0x48247d58>fchain ptr: <0x482dd774>driver cookie: <0x4824cd68>driver func: <0x4824cd54>number of subinterfaces: 0subblock array size: 0DSNCNF: 0x00000000interface stats info: IN unknown proto pkts: 0 IN unknown proto bytes: 0 IN multicast pkts: 0 OUT multicast pkts: 0 IN broadcast pkts: 0 OUT broadcast pkts: 0 IN drop pkts: 0 OUT drop pkts: 0 IN errors pkts: 0 OUT errors pkts: 0

Chains--------------------Base decap chain: hdlc <14> <0xfd6a0a74, 0x00000000> < 0, 0>

Protocol chains:---------------<Protocol number> (name) Stats Type Chain_node <caps num> <function, context> <drop pkts, drop bytes><9> (chdlc) Stats IN: 48466 pkts, 3559516 bytes; OUT: 41378 pkts, 910312 bytes Encap: l2_adj_rewrite <86> <0xfceada88, 0x482c390c> < 0, 0> queue_fifo <56> <0xfcedea68, 0x482ddc30> < 0, 0>


OL-23591-02


txm_nopull <60> <0xfce8fa5c, 0x482ddda4> < 0, 0> Decap: queue_fifo <56> <0xfcedea4c, 0x482ddc30> < 0, 0> chdlc <13> <0xfd6a252c, 0x00000000> < 0, 0> Fixup: l2_adj_rewrite <86> <0xfcead45c, 0x00000000> < 0, 0> queue_fifo <56> <0xfcedea68, 0x482ddc30> < 0, 0> txm_nopull <60> <0xfce8fa5c, 0x482ddda4> < 0, 0><10> (clns) Stats IN: 0 pkts, 0 bytes; OUT: 0 pkts, 0 bytes Encap: clns <15> <0xfcfaa030, 0x00000000> < 0, 0> hdlc <14> <0xfd6a0678, 0x00000000> < 0, 0> l2_adj_rewrite <86> <0xfceada88, 0x48305a90> < 0, 0> queue_fifo <56> <0xfcedea68, 0x482ddc30> < 0, 0> txm_nopull <60> <0xfce8fa5c, 0x482ddda4> < 0, 0> Decap: queue_fifo <56> <0xfcedea4c, 0x482ddc30> < 0, 0> clns <15> <0xfcfa9508, 0x00000000> < 0, 0> Fixup: l2_adj_rewrite <86> <0xfcead45c, 0x00000000> < 0, 0> queue_fifo <56> <0xfcedea68, 0x482ddc30> < 0, 0> txm_nopull <60> <0xfce8fa5c, 0x482ddda4> < 0, 0><12> (ipv4) Stats IN: 0 pkts, 0 bytes; OUT: 0 pkts, 0 bytes Encap: ipv4 <26> <0xfd0f341c, 0x482dd460> < 0, 0> hdlc <14> <0xfd6a0678, 0x00000000> < 0, 0> l2_adj_rewrite <86> <0xfceada88, 0x48349b54> < 0, 0> queue_fifo <56> <0xfcedea68, 0x482ddc30> < 0, 0> txm_nopull <60> <0xfce8fa5c, 0x482ddda4> < 0, 0> Decap: queue_fifo <56> <0xfcedea4c, 0x482ddc30> < 0, 0> ipv4 <26> <0xfd0f3474, 0x00000000> < 0, 0> Fixup: l2_adj_rewrite <86> <0xfcead45c, 0x00000000> < 0, 0> queue_fifo <56> <0xfcedea68, 0x482ddc30> < 0, 0> txm_nopull <60> <0xfce8fa5c, 0x482ddda4> < 0, 0>

Protocol SAFI counts:--------------------

Protocol SAFI Pkts In Bytes In Pkts Out Bytes Out--------------- ---------- ---------- ---------- ---------- ---------- ipv4 Unicast 0 0 0 0 ipv4 Multicast 0 0 0 0 ipv4 Broadcast 0 0 0 0 ipv6 Unicast 0 0 0 0 ipv6 Multicast 0 0 0 0

The following example shows counters implemented for Multiprotocol Label Switching (MPLS) packets. The following output under Protocol Chains in the show netio idb command shows the MPLS packets incrementing:

mpls <25> <0xfcc7b2b8, 0x00000000> < 152, 17328>

RP/0/RSP0/CPU0:router# show netio idb gigabitEthernet 0/2/0/1

GigabitEthernet0/2/0/1 (handle: 0x01280040, nodeid:0x21) netio idb:---------------------------------name: GigabitEthernet0_2_0_1interface handle: 0x01280040interface global index: 3physical media type: 30dchain ptr: <0x482e0700>


OL-23591-02


echain ptr: <0x482e1024>fchain ptr: <0x482e13ec>driver cookie: <0x4829fc6c>driver func: <0x4829f040>number of subinterfaces: 4096subblock array size: 7DSNCNF: 0x00000000interface stats info: IN unknown proto pkts: 0 IN unknown proto bytes: 0 IN multicast pkts: 0 OUT multicast pkts: 0 IN broadcast pkts: 0 OUT broadcast pkts: 0 IN drop pkts: 0 OUT drop pkts: 0 IN errors pkts: 0 OUT errors pkts: 0

Chains--------------------Base decap chain: ether <30> <0xfd018cd8, 0x482c736c> < 0, 0>

Protocol chains:---------------<Protocol number> (name) Stats Type Chain_node <caps num> <function, context> <drop pkts, drop bytes><7> (arp) Stats IN: 0 pkts, 0 bytes; OUT: 0 pkts, 0 bytes Encap: l2_adj_rewrite <86> <0xfcaa997c, 0x4831a33c> < 0, 0> pcn_output <54> <0xfd054bfc, 0x48319f04> < 0, 0> q_fq <43> <0xfd05f4b8, 0x48320fec> < 0, 0> txm_nopull <60> <0xfcadba38, 0x4824c0fc> < 0, 0> Decap: pcn_input <55> <0xfd054bfc, 0x4830ba8c> < 0, 0> q_fq_input <96> <0xfd05f330, 0x48312c7c> < 0, 0> arp <24> <0xfcbfc2cc, 0x00000000> < 0, 0> Fixup: l2_adj_rewrite <86> <0xfcaa945c, 0x00000000> < 0, 0> pcn_output <54> <0xfd054bfc, 0x48319f04> < 0, 0> q_fq <43> <0xfd05f4b8, 0x48320fec> < 0, 0> txm_nopull <60> <0xfcadba38, 0x4824c0fc> < 0, 0><10> (clns) Stats IN: 0 pkts, 0 bytes; OUT: 1861623 pkts, 2062483853 bytes Encap: clns <15> <0xfcbe2c80, 0x00000000> < 0, 0> ether <30> <0xfd0189b4, 0x482c736c> < 0, 0> l2_adj_rewrite <86> <0xfcaa997c, 0x482d8660> < 0, 0> pcn_output <54> <0xfd054bfc, 0x48319f04> < 0, 0> q_fq <43> <0xfd05f4b8, 0x48320fec> < 0, 0> txm_nopull <60> <0xfcadba38, 0x4824c0fc> < 0, 0> Decap: pcn_input <55> <0xfd054bfc, 0x4830ba8c> < 0, 0> q_fq_input <96> <0xfd05f330, 0x48312c7c> < 0, 0> clns <15> <0xfcbe2444, 0x00000000> < 0, 0> Fixup: l2_adj_rewrite <86> <0xfcaa945c, 0x00000000> < 0, 0> pcn_output <54> <0xfd054bfc, 0x48319f04> < 0, 0> q_fq <43> <0xfd05f4b8, 0x48320fec> < 0, 0> txm_nopull <60> <0xfcadba38, 0x4824c0fc> < 0, 0><12> (ipv4) Stats IN: 0 pkts, 0 bytes; OUT: 759095 pkts, 57691220 bytes Encap: ipv4 <26> <0xfcc03dfc, 0x482e0414> < 0, 0> ether <30> <0xfd0189b4, 0x482c736c> < 0, 0>


OL-23591-02


l2_adj_rewrite <86> <0xfcaa997c, 0x4831a294> < 0, 0> pcn_output <54> <0xfd054c48, 0x48319f04> < 0, 0> q_fq <43> <0xfd05f4b8, 0x48320fec> < 0, 0> txm_nopull <60> <0xfcadba38, 0x4824c0fc> < 0, 0> Decap: pcn_input <55> <0xfd054c48, 0x4830ba8c> < 0, 0> q_fq_input <96> <0xfd05f330, 0x48312c7c> < 0, 0> ipv4 <26> <0xfcc03e80, 0x00000000> < 0, 0> Fixup: l2_adj_rewrite <86> <0xfcaa945c, 0x00000000> < 0, 0> pcn_output <54> <0xfd054c48, 0x48319f04> < 0, 0> q_fq <43> <0xfd05f4b8, 0x48320fec> < 0, 0> txm_nopull <60> <0xfcadba38, 0x4824c0fc> < 0, 0><13> (mpls) Stats IN: 204 pkts, 23256 bytes; OUT: 0 pkts, 0 bytes Encap: mpls <25> <0xfcc7ddbc, 0x00000000> < 0, 0> ether <30> <0xfd0189b4, 0x482c736c> < 0, 0> l2_adj_rewrite <86> <0xfcaa997c, 0x4831a2e8> < 0, 0> pcn_output <54> <0xfd0561f0, 0x48319f04> < 0, 0> q_fq <43> <0xfd05f4b8, 0x48320fec> < 0, 0> txm_nopull <60> <0xfcadba38, 0x4824c0fc> < 0, 0> Decap: pcn_input <55> <0xfd0561f0, 0x4830ba8c> < 0, 0> q_fq_input <96> <0xfd05f330, 0x48312c7c> < 0, 0> mpls <25> <0xfcc7b2b8, 0x00000000> < 152, 17328> Fixup: l2_adj_rewrite <86> <0xfcaa945c, 0x00000000> < 0, 0> pcn_output <54> <0xfd0561f0, 0x48319f04> < 0, 0> q_fq <43> <0xfd05f4b8, 0x48320fec> < 0, 0> txm_nopull <60> <0xfcadba38, 0x4824c0fc> < 0, 0><22> (ether_sock) Stats IN: 0 pkts, 0 bytes; OUT: 0 pkts, 0 bytes Encap: ether_sock <98> <0xfd01a774, 0x482c736c> < 0, 0> l2_adj_rewrite <86> <0xfcaa997c, 0x482d85f0> < 0, 0> pcn_output <54> <0xfd054bfc, 0x48319f04> < 0, 0> q_fq <43> <0xfd05f4b8, 0x48320fec> < 0, 0> txm_nopull <60> <0xfcadba38, 0x4824c0fc> < 0, 0> Decap: pcn_input <55> <0xfd054bfc, 0x4830ba8c> < 0, 0> q_fq_input <96> <0xfd05f330, 0x48312c7c> < 0, 0> ether_sock <98> <0xfd01a91c, 0x482c736c> < 0, 0> Fixup: l2_adj_rewrite <86> <0xfcaa945c, 0x00000000> < 0, 0> pcn_output <54> <0xfd054bfc, 0x48319f04> < 0, 0> q_fq <43> <0xfd05f4b8, 0x48320fec> < 0, 0> txm_nopull <60> <0xfcadba38, 0x4824c0fc> < 0, 0>



The following example shows the SPA status.

RP/0/RSP0/CPU0:router# show hw-module subslot 0/4/1 counters

Wed Nov 3 09:00:34.258 EDT


OL-23591-02

Chapter 2 Verifying and Troubleshooting Interface Status Verifying and Troubleshooting Pluggable Optical Line Card Interfaces

Subslot 0/4/1 counts info:------------------------SPA inserted: YESSPA type: 1xCHOC48 SPASPA operational state: READYSPA insertion time: Mon Nov 1 08:28:12 2010SPA last time ready: Mon Nov 1 08:30:04 2010SPA uptime [HH:MM:SS]: 48:30:30

The following Packet-over-SONET/SDH (POS) port example shows the SPA counters. The output displays any drop counters or error counters incrementing for the interface.

RP/0/RSP0/CPU0:router# show hw-module subslot counters framer

SPA 0/2/0 device framer 0/0 info:

ABC (port 0) framer registers:

DEF Framer counters:STREAM 0 Rx Bytes (48-bit) (#0x60e7d078-0x883c): 567674 Rx Good Bytes (48-bit) (#0x60e7d080-0x8840): 479474 Rx Good Packets (48-bit) (#0x60e7d040-0x8820): 21795 Tx Byte Cnt Reg (48-bit) (#0x60e81070-0xa838): 567709 Tx Good Bytes Cnt Reg (48-bit) (#0x60e81068-0xa834): 479496 Tx Transmitted Packet Cnt Reg (48-bit) (#0x60e81040-0xa820): 21796

SPA 0/2/0 device framer 1/0 info:

ABC (port 1) framer registers:

DEF Framer counters:STREAM 0 Rx Bytes (48-bit) (#0x60dfd078-0x883c): 567628 Rx Good Bytes (48-bit) (#0x60dfd080-0x8840): 479438 Rx Good Packets (48-bit) (#0x60dfd040-0x8820): 21793 Tx Byte Cnt Reg (48-bit) (#0x60e01070-0xa838): 567690 Tx Good Bytes Cnt Reg (48-bit) (#0x60e01068-0xa834): 479478 Tx Transmitted Packet Cnt Reg (48-bit) (#0x60e01040-0xa820): 21795

Verifying and Troubleshooting Pluggable Optical Line Card Interfaces

Troubleshooting the pluggable optical line card interfaces includes verifying that you have an optical card installed, enabled, and functioning properly. To troubleshoot the configured pluggable optical line card interfaces, perform the following procedure.

SUMMARY STEPS

1. show controllers plim asic SPAQFPBridgeCtrl counters instance <0 – 3> all location node-id

2. show hw-module subslot brief pluggable-optics

3. show hw-module subslot address status pluggable-optics


OL-23591-02


4. show hw-module subslot address errors pluggable-optics

5. show hw-module subslot address registers pluggable-optics


DETAILED STEPS

Examples

The following example displays the drop counters and error countersincrementing on the SPA in the line card.

RP/0/RSP0/CPU0:router# show controllers plim asic SPAQFPBridgeCtrl counters instance 0 all location 0/4/CPU0Wed Nov 3 08:58:07.062 EDT


Step 1 show controllers plim asic SPAQFPBridgeCtrl counters instance <0 – 3> all location node-id

Example:RP/0/RSP0/CPU0:router# show controllers plim asic SPAQFPBridgeCtrl counters instance 0 all location 0/4/CPU0

Displays the drop counters and error counters incrementing on the interface.

Step 1 show hw-module subslot brief pluggable-optics

Example:RP/0/RSP0/CPU0:router# show hw-module subslot brief pluggable-optics

Displays a brief summary of the pluggable optics status for all line card nodes, including optics type, vendor, and state. Check that the state is enabled for the node that you are troubleshooting.

Step 2 show hw-module subslot address status pluggable-optics

Example:RP/0/RSP0/CPU0:router# show hw-module subslot 0/2/0 status pluggable-optics

Displays the status of the pluggable optics for the specified line card node, including faults and environmental data. Check that the state and the transceiver are enabled. Check for any warnings or alarms.

Step 3 show hw-module subslot address errors pluggable-optics

Example:RP/0/RSP0/CPU0:router# show hw-module subslot 0/2/0 errors pluggable-optics

Displays any errors that are present on the node. Note if there are any errors.

Verify that Phased Initialization displays Phase Reached: 4. Verify that Socket Verification displays “passed” for both Compatibility and Security.

Step 4 show hw-module subslot address registers pluggable-optics

Example:RP/0/RSP0/CPU0:router# show hw-module subslot 0/2/0 registers pluggable-optics

Displays all available information on the optics including the IDPROM contents.

Step 5 Contact Cisco Technical Support. For Cisco Technical Support contact information, see the “Obtaining Documentation and Submitting a Service Request” section on page xii in the Preface.


OL-23591-02


SPAQFPBridgeCtrl 0:=======================================

------------ SPAQFPBridgeCtrl 0 SPA1 Counters --------RX Bytes: 0RX Packets: 0TX Bytes: 0TX Packets: 0--------------- SPAQFPBridgeCtrl 0 HT Counters ----------HT RX Bytes: 0HT RX Packets: 0HT TX Bytes: 0HT TX Packets: 0-------------- SPAQFPBridgeCtrl 0 NP Counters ---------RXNP eSPI Bytes: 0RXNP eSPI Packets: 0TXNP eSPI Bytes: 0TXNP eSPI Packets: 0-------------- SPAQFPBridgeCtrl 0 Diag Counters ---------RXNP Online Diag Bytes: 0RXNP Online Diag Packets: 0TXNP Online Diag Bytes: 0TXNP Online Diag Packets: 0-------------- SPAQFPBridgeCtrl 0 Error Counters ---------ERP Read out of sync Error: 0CPUIF ERP Parity Error: 0CPUIF ERP Protocol Error: 0RXRLDC Mem Low Byte ECC: 0RXRLDC Mem High Byte ECC: 0TXRLDC Mem Low Byte ECC: 0TXRLDC Mem High Byte ECC: 0-------------- SPAQFPBridgeCtrl 0 Drop Counters ---------TXNP Errored Low Priority Packets: 0TXNP Errored High Priority Packets: 0TXNP Errored Loopback Packets: 0TXNP Errored HT Packets: 0TXNP Errored Low Priority Bytes: 0TXNP Errored High Priority Bytes: 0TXNP Errored Loopback Bytes: 0TXNP Errored HT Bytes: 0--------- SPAQFPBridgeCtrl 0 RXDCM Drops ----------------- SPAQFPBridgeCtrl 0 TXDCM Drops --------

The following example shows typical outputs for these commands.

RP/0/RSP0/CPU0:router# show hw-module subslot brief pluggable-optics

SPA 0/2/0 device pluggable-optics 0/0 info:

POS0/2/0/0: ID: SFP Extended ID: 4 Xcvr Type: OC48 SR/STM16 I-16 (44) Connector: LC Vendor name: CISCO-FINISAR Vendor part number: FTRJ1321P1BTL-C4 State: Enabled


POS0/2/0/1: ID: SFP


OL-23591-02


Extended ID: 4 Xcvr Type: OC48 SR/STM16 I-16 (44) Connector: LC Vendor name: CISCO-FINISAR Vendor part number: FTRJ1321P1BTL-C4 State: Enabled

RP/0/RSP0/CPU0:router# show hw-module subslot 0/2/0 status pluggable-optics


POS0/2/0/0: State: Enabled Environmental Information - raw values Temperature: 48.136 C Supply voltage: 32970 in units of 100uVolt Tx bias: 12678 in units of 2uAmp Tx power: -4 dBm (2848 in units of 0.1 uW) Rx power: -7 dBm (1554 in units of 0.1 uW)

Transceiver: Enabled SW TX Fault: None SW LOS: Active No active alarms No active warnings Version Identifier (VID): V01 Product Identifier (PID): SFP-OC48-SR Part Number (PN): 10-1961-01 CLEI: WM1T2TTAAA


POS0/2/0/1: State: Enabled Environmental Information - raw values Temperature: 46.172 C Supply voltage: 32914 in units of 100uVolt Tx bias: 11166 in units of 2uAmp Tx power: -5 dBm (2282 in units of 0.1 uW) Rx power: -5 dBm (2411 in units of 0.1 uW)

Transceiver: Enabled SW TX Fault: unavailable SW LOS: unavailable No active alarms No active warnings Version Identifier (VID): V01 Product Identifier (PID): SFP-OC48-SR Part Number (PN): 10-1961-01 CLEI: WM1T2TTAAA

RP/0/RSP0/CPU0:router# show hw-module subslot 0/2/0 errors pluggable-optics


POS0/2/0/0: Phased Initialization Phase Reached: 4 Phase Exit Code: Success 0 Phase Read Offset: 256

Socket Verification


OL-23591-02


Compatibility: Compatibility passed Security: Security passed


POS0/2/0/1: Phased Initialization Phase Reached: 4 Phase Exit Code: Success 0 Phase Read Offset: 256

Socket Verification Compatibility: Compatibility passed Security: Security passed

RP/0/RSP0/CPU0:router# show hw-module subslot 0/2/0 registers pluggable-optics


POS0/2/0/0: ID: SFP Extended ID: 4 Xcvr Type: OC48 SR/STM16 I-16 (44) Connector: LC Encoding: reserved Bit Rate: 2500 Mbps Single mode fiber supported length: 2 km Single mode fiber supported length: 20 m Upper bit rate limit: not specified Lower bit rate limit: not specified Date code (yy/mm/dd): 05/12/13 Vendor name: CISCO Vendor OUI: 36965 Vendor Part Number (PN): FTRJ1321P1BTL-C4 Vendor Rev: B Vendor SN (SN): FNS0951J0VN Options implemented: LOS Signal TX Fault Signal TX Disable Signal Enhanced options implemented: Alarm/Warning Flags Diagnostic monitoring implemented: Exernally Calibrated Digital Diagnostic Monitoring Idprom contents (hex): 0x00: 03 04 07 00 01 00 00 12 00 01 05 05 19 00 02 14 0x10: 00 00 00 00 43 49 53 43 4F 2D 46 49 4E 49 53 41 0x20: 52 20 20 20 00 00 90 65 46 54 52 4A 31 33 32 31 0x30: 50 31 42 54 4C 2D 43 34 42 20 20 20 05 1E 00 23 0x40: 00 1A 00 00 46 4E 53 30 39 35 31 4A 30 56 4E 20 0x50: 20 20 20 20 30 35 31 32 31 33 20 20 58 80 01 D3 0x60: 00 00 02 4C D2 86 D7 04 F8 8D 92 6D 3C 8B D2 2D 0x70: 60 F3 BD 00 00 00 00 00 00 00 00 00 62 56 5D CD 0x80: FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF 0x90: FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF Status/Control Register: 10f8 Alarm Status: 0000 Warning Status: 0000

THRESHOLDS high alarm high warning low warning low alarm Temperature C +110.000 +093.000 -30.000 -40.000


OL-23591-02


Voltage V 003.9000 003.7000 002.9000 002.7000 Bias Current mA 080.0000 070.0000 004.0000 002.0000 Transmit power mW 002.7234 001.7251 000.1704 000.1143 Receive power mW 005.0328 003.3461 000.0373 000.0120 Diagnostics contents (hex): 0x00: 6E 00 D8 00 5D 00 E2 00 98 58 69 78 90 88 71 48 0x10: 9C 40 03 E8 88 B8 07 D0 6A 62 04 77 43 63 06 A8 0x20: C4 98 00 78 82 B5 01 75 00 00 00 00 00 00 00 00 0x30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x40: 00 00 00 00 3E 72 B9 A5 41 37 B7 FE 01 00 00 00 0x50: 00 77 FF AB 01 00 00 00 01 00 00 00 10 00 00 90 0x60: 21 EC 81 18 01 0E 00 BA 19 DA 00 00 00 00 10 F8 0x70: 00 00 00 80 00 00 00 00 00 00 00 00 00 00 00 01 0x80: 57 4D 31 54 32 54 54 41 41 41 31 30 2D 31 39 36 0x90: 31 2D 30 31 56 30 31 20 8A FB 55 00 00 00 00 64 0xA0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0xB0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 AA AA 0xC0: 53 46 50 2D 4F 43 34 38 2D 53 52 20 20 20 20 20 0xD0: 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 66 0xE0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0xF0: 00 00 00 00 00 00 00 00 00 40 00 40 00 00 00 00


POS0/2/0/1: ID: SFP Extended ID: 4 Xcvr Type: OC48 SR/STM16 I-16 (44) Connector: LC Encoding: reserved Bit Rate: 2500 Mbps Single mode fiber supported length: 2 km Single mode fiber supported length: 20 m Upper bit rate limit: not specified Lower bit rate limit: not specified Date code (yy/mm/dd): 06/04/25 Vendor name: CISCO Vendor OUI: 36965 Vendor Part Number (PN): FTRJ1321P1BTL-C4 Vendor Rev: B Vendor SN (SN): FNS1017R0BV Options implemented: LOS Signal TX Fault Signal TX Disable Signal Enhanced options implemented: Alarm/Warning Flags Diagnostic monitoring implemented: Exernally Calibrated Digital Diagnostic Monitoring Idprom contents (hex): 0x00: 03 04 07 00 01 00 00 12 00 01 05 05 19 00 02 14 0x10: 00 00 00 00 43 49 53 43 4F 2D 46 49 4E 49 53 41 0x20: 52 20 20 20 00 00 90 65 46 54 52 4A 31 33 32 31 0x30: 50 31 42 54 4C 2D 43 34 42 20 20 20 05 1E 00 23 0x40: 00 1A 00 00 46 4E 53 31 30 31 37 52 30 42 56 20 0x50: 20 20 20 20 30 36 30 34 32 35 20 20 58 80 01 CE 0x60: 00 00 02 91 44 1B BF 32 33 64 55 09 CB 9B FD A3 0x70: 39 11 A4 00 00 00 00 00 00 00 00 00 88 17 A0 7B 0x80: FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF 0x90: FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF Status/Control Register: 10f8 Alarm Status: 0000 Warning Status: 0000


OL-23591-02


THRESHOLDS high alarm high warning low warning low alarm Temperature C +110.000 +093.000 -30.000 -40.000 Voltage V 003.9000 003.7000 002.9000 002.7000 Bias Current mA 080.0000 070.0000 004.0000 002.0000 Transmit power mW 003.5276 002.2272 000.2022 000.1291 Receive power mW 005.0147 003.3353 000.0407 000.0155 Diagnostics contents (hex): 0x00: 6E 00 D8 00 5D 00 E2 00 98 58 69 78 90 88 71 48 0x10: 9C 40 03 E8 88 B8 07 D0 89 CC 05 0B 57 00 07 E6 0x20: C3 E3 00 9B 82 49 01 97 00 00 00 00 00 00 00 00 0x30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x40: 00 00 00 00 3E 73 C5 30 40 46 23 7C 01 00 00 00 0x50: 00 5B FF F2 01 00 00 00 01 00 00 00 10 00 00 7C 0x60: 21 D8 80 B8 00 54 00 22 2A 02 00 00 00 00 10 F8 0x70: 00 00 00 80 00 00 00 00 00 00 00 00 00 00 00 01 0x80: 57 4D 31 54 32 54 54 41 41 41 31 30 2D 31 39 36 0x90: 31 2D 30 31 56 30 31 20 8A FB 55 00 00 00 00 64 0xA0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0xB0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 AA AA 0xC0: 53 46 50 2D 4F 43 34 38 2D 53 52 20 20 20 20 20 0xD0: 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 66 0xE0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0xF0: 00 00 00 00 00 00 00 00 00 40 00 40 00 00 00 00


OL-23591-02


C H A P T E R 3
Troubleshooting Interface Connectivity
This chapter explains how to troubleshoot problems with connectivity between interfaces on the ASR 9000 and interfaces on remote devices. It includes the following topics:

• Troubleshooting Ping and ARP Connectivity, page 3-75

• Troubleshooting Bidirectional Forwarding Detection, page 3-81

• Troubleshooting Ethernet CFM, page 3-85

Troubleshooting Ping and ARP ConnectivityFollow the steps in this section to troubleshoot ping andAddress Resolution Protocol (ARP) connectivity problems. The overall approach is to verify that the routing protocol is up, the network topology is properly configured, and neighbors are up and reachable.

This procedure sends ping messages to the remote end and analyzes the resulting responses. For Ethernet interfaces, Address Resolution Protocol (ARP) connectivity is a prerequisite for ping connectivity—ARP must work first before ICMP echo can work. Therefore, check ARP and ensure it is working so that ICMP echo and ping can work. (Optical interfaces do not involve ARP.)

You should trace the path of the ping packets to see if they are getting dropped at any point along the path. Typical steps to locate these drops are:

• Run the command show inject stats —See if the packets went from the CPU to the NP.

• Run the command show interface stats—Look at the Tx counters to see if the packet was sent.

• Check the remote interface statistics to see if they received the packets.

• Check the remote punt and ARP statistics.

• Run commands in this list (above) on the return patch to check for drops.

Figure 3-1 shows the general approach to troubleshooting ping and ARP connectivity issues. Throubleshooting is required because a ping attempt has failed. In this example, the IGP protocol refers to the protocol currently configured on the network you are troubleshooting—OSPF, EIGRP, IS-IS, or RIP.


Chapter 3 Troubleshooting Interface Connectivity Troubleshooting Ping and ARP Connectivity

Figure 3-1 Example of Troubleshooting Ping and ARP Connectivity Issues

Follow these steps to troubleshoot ping and ARP connectivity issues. See Figure 3-1 to help you locate the steps that apply to your network scenario.

Step 1 Ping the remote end and check for a response. If there is no response, continue with this procedure to determine why the ping was unsuccessful and to connect successfully to the remote end.

Tip Use a systematic process to isolate the location of the failure. Ping the local interface first. If that is successful, ping the directly connected neighbor (single hop). If that is successful, ping the next hop, and so forth.

Step 2 Verify that the interface is configured as Layer 3.

No

Is thisan Ethernetinterface?

Yes No

Is theIGP protocol

running?

Is theIGP protocol

up?

Was anARP entrycreated?

Was pingsuccessful?

Yes

Tryping

Tryping

No

No

Yes

No

Yes

DebugARP

2799

71

Exit

Yes

Debugping packets


OL-23591-02


Step 3 Check the routing table to make sure the IP address you are trying to ping has a route in the Routing Information Base (RIB) table.

RP/0/RSP0/CPU0:router# show route

Step 4 Verify that the IGP protocol is running and the connection to the neighbor is up. The following example assumes OSPF as the IGP.

RP/0/RSP0/CPU0:router# show protocols ospf

RP/0/RSP0/CPU0:router# show ospf neighbor

Step 5 Verify that packets are coming in and going out on the Ethernet interface.

Be aware of the following ARP behaviors when you are reviewing the display from the show commands in this step:

• A normal ping will send an ARP packet out followed by the actual ICMP echo packets. ARP must work before ICMP echo can work. If the system is receiving zero packets back, then there was no ARP reply. Even a single packet back means there was an ARP reply. (The system sends the ARP packet only if there is no ARP entry. Otherwise, it skips the ARP and proceeds with the ICMP echo.)

• By default, the system attempts to ping the remote router five times. If the remote router was recently connected to the network, the first ping will fail because the system needs time to resolve the ARP packet.

RP/0/RSP0/CPU0:router# show interfaces location node-id

or

RP/0/RSP0/CPU0:router# show interfaces type

Step 6 Display the ARP information. The IP address that you attempted to ping should be in the output.

RP/0/RSP0/CPU0:router# show arp

Step 7 If this port was previously attached to another device, or some othe major change has taken place on the remote end, use the clear arp-cache command to build a new entry. Verify that the MAC address in the ARP table is correct (see the MAC address in the Hardware Addr column in the example in Step 6).

RP/0/RSP0/CPU0:router# clear arp-cache

RP/0/RSP0/CPU0:router# show arp

Step 8 Determine whether an ARP entry exists for the destination IP.

RP/0/RSP0/CPU0:router# show arp location node-id

a. If an ARP entry does not exist or is incomplete, add a static ARP entry. Ensure that the Tx adjacency points to ‘COMPLETE’.

RP/0/RSP0/CPU0:router# show cef {ipv4} prefix hardware egress detail location node-id

Caution After you finish using the static ARP entry for troubleshooting purposes, you must remove it. If you do not remove the static ARP entry, it will cause traffic to be misdirected.

b. If the ARP entry points to ‘COMPLETE’, it means that the ARP entry is not being updated. Troubleshooting should now focus on why the ARP entry is not getting added (this includes steps such as show arp, show arp idb, show adjacency gig node-id detail location node-id, and show arp trace).


OL-23591-02


c. If the Tx adjacency still points to ‘PUNT’, it means ARP is adding the entry in the database, but fib_mgr fails to mark the adjacency as ‘COMPLETE’.

d. This could be a fib_mgr, ARP, or AIB problem. Delete and reconfigure the static ARP entry with AIB and CEF debugs on. The debugs show if ARP is adding the entry inside the AIB and if the AIB is informing fib_mgr.

Step 9 Send a burst of traffic to help troubleshoot whether traffic is getting through. This can help in scenarios such as a disconnected cable, intermittent drops at unknown locations, and so forth.

a. Configure a static ARP entry, then send a large number of ping packets (for example, 100 or 1,000 packets) with a zero timeout. This sends out a burst of traffic from the router. When the ping fails, the system displays a dot instead of an exclamation point. If the ping is not possible, the system displays a ‘U’ instead of a dot.

Example:

RP/0/RSP0/CPU0:router# ping 192.0.2.55 count 100 timeout 0Wed Sep 29 15:09:29.809 EDTType escape sequence to abort.Sending 100, 100-byte ICMP Echos to 192.0.2.55, timeout is 0 seconds:....................................................................................................Success rate is 0 percent (0/100)

b. There are parameters to the ping command that can be used to change the time delay for the reply. Try lowering the delay, changing parameters to send 100 ping requests, and so forth.

c. There is a mode that allows you to suppress the ARP request and send out only the ICMP echo packets. See if the pings are failing intermittently or all the time.

Step 10 If ARP connectivity fails, perform the following steps to find out why.

a. Remove the static ARP entry.

b. Local ping—Ping your own interface (the interface your router uses to send out the pings).

c. Determine whether the local ping was successful. If the local ping failed (no response), pings out of that interface will also fail.

d. If you have an ARP entry, verify that there are outgoing/incoming ICMP echo/reply packets.

Note A ping is represented by a dot, exclam point, or capital U. The RSP allows a specific number of seconds for the ping to complete. The ARP is hidden inside this event.

Step 11 If you have to dig deeper into the issue, use the following commands to dump ping packets.

Example

RP/0/RSP0/CPU0:router# show routeC 172.21.116.0/24 is directly connected, 2d19h, MgmtEth0/RSP0/CPU0/0is directly connected, 2d19h, MgmtEth0/RSP1/CPU0/0L 172.21.116.10/32 is directly connected, 2d20h, MgmtEth0/RSP0/CPU0/0L 172.21.116.11/32 is directly connected, 2d19h, MgmtEth0/RSP1/CPU0/0L 172.21.116.12/32 [0/0] via 172.21.116.12, 2d19h, MgmtEth0/RSP0/CPU0/0O 192.168.12.0/24 [110/2] via 192.168.111.11, 2d19h, GigabitEthernet0/2/0/1[110/2] via 192.168.121.12, 2d19h, GigabitEthernet0/2/0/2O 192.168.21.0/24 [110/2] via 192.168.111.11, 2d19h, GigabitEthernet0/2/0/1[110/2] via 192.168.121.12, 2d19h, GigabitEthernet0/2/0/2C 192.168.111.0/24 is directly connected, 2d20h, GigabitEthernet0/2/0/1L 192.168.111.1/32 is directly connected, 2d20h, GigabitEthernet0/2/0/1


OL-23591-02


O 192.168.112.0/24 [110/2] via 192.168.111.11, 2d19h, GigabitEthernet0/2/0/1O 192.168.113.0/24 [110/2] via 192.168.111.11, 2d19h, GigabitEthernet0/2/0/1

RP/0/RSP0/CPU0:router# show protocols ospf

Routing Protocol OSPF 100 Router Id: 10.144.144.144 Distance: 110 Non-Stop Forwarding: Enabled Redistribution: None Area 0 MPLS/TE enabled Loopback0 GigabitEthernet0/1/0/2 GigabitEthernet0/1/0/8 GigabitEthernet0/1/0/18 GigabitEthernet0/1/0/23 TenGigE0/4/0/0* Indicates MADJ interface

RP/0/RSP0/CPU0:router# show ospf neighborNeighbors for OSPF 100

Neighbor ID Pri State Dead Time Address Interface10.164.164.164 1 FULL/DR 00:00:36 10.147.4.64 GigabitEthernet0/1/0/2 Neighbor is up for 2d17h10.166.166.166 1 FULL/DR 00:00:39 10.146.4.66 GigabitEthernet0/1/0/8 Neighbor is up for 2d17h10.19.19.19 1 FULL/BDR 00:00:33 10.194.4.19 GigabitEthernet0/1/0/18 Neighbor is up for 2d18h10.11.11.11 1 FULL/BDR 00:00:34 10.114.4.11 GigabitEthernet0/1/0/23 Neighbor is up for 2d18h10.11.11.11 1 FULL/BDR 00:00:39 10.114.8.11 TenGigE0/4/0/0 Neighbor is up for 2d18h

Total neighbor count: 5

The following example shows the packet counts for a line card. Note that there are packets being input and output.

RP/0/RSP0/CPU0:router# show interfaces location 0/4/CPU0 Wed Sep 1 09:22:03.427 DSTTenGigE0/4/0/0 is up, line protocol is up Interface state transitions: 1 Hardware is TenGigE, address is 001b.53ff.a780 (bia 001b.53ff.a780) Layer 1 Transport Mode is LAN Description: Connected to P11_CRS-4 10GE 0/2/5/0 Internet address is 10.114.8.44/24 MTU 9100 bytes, BW 10000000 Kbit (Max: 10000000 Kbit) reliability 255/255, txload 0/255, rxload 0/255 Encapsulation ARPA, Full-duplex, 10000Mb/s, LR, link type is force-up output flow control is off, input flow control is off loopback not set, ARP type ARPA, ARP timeout 04:00:00 Last input 00:00:00, output 00:00:00 Last clearing of "show interface" counters never 5 minute input rate 28000 bits/sec, 39 packets/sec 5 minute output rate 45000 bits/sec, 39 packets/sec 2356786692 packets input, 151622450429 bytes, 26 total input drops 0 drops for unrecognized upper-level protocol Received 2 broadcast packets, 2327063140 multicast packets


OL-23591-02


0 runts, 0 giants, 0 throttles, 0 parity 0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort 30320436 packets output, 4277187228 bytes, 18 total output drops Output 1 broadcast packets, 495705 multicast packets 0 output errors, 0 underruns, 0 applique, 0 resets 0 output buffer failures, 0 output buffers swapped out 1 carrier transitions

RP/0/RSP0/CPU0:router# show arp Wed Sep 1 10:16:22.472 DST

-------------------------------------------------------------------------------0/4/CPU0-------------------------------------------------------------------------------Address Age Hardware Addr State Type Interface10.114.8.11 02:40:44 001b.0c63.67ff Dynamic ARPA TenGigE0/4/0/010.114.8.44 - 001b.53ff.a780 Interface ARPA TenGigE0/4/0/0

-------------------------------------------------------------------------------0/1/CPU0-------------------------------------------------------------------------------Address Age Hardware Addr State Type Interface10.114.4.11 00:15:22 001b.0c63.67e7 Dynamic ARPA GigabitEthernet0/1/0/2310.114.4.44 - 001b.53ff.87f7 Interface ARPA GigabitEthernet0/1/0/2310.145.4.38 01:43:50 001e.f77d.5219 Dynamic ARPA GigabitEthernet0/1/0/2710.145.4.44 - 001b.53ff.87fb Interface ARPA GigabitEthernet0/1/0/2710.146.4.44 - 001b.53ff.87e8 Interface ARPA GigabitEthernet0/1/0/810.146.4.66 02:56:39 0022.0d26.3bc4 Dynamic ARPA GigabitEthernet0/1/0/810.147.4.44 - 001b.53ff.87e2 Interface ARPA GigabitEthernet0/1/0/210.147.4.64 00:33:21 0022.0d26.36c4 Dynamic ARPA GigabitEthernet0/1/0/210.194.4.19 03:16:59 001a.3029.d400 Dynamic ARPA GigabitEthernet0/1/0/1810.194.4.44 - 001b.53ff.87f2 Interface ARPA GigabitEthernet0/1/0/1810.194.8.44 - 001b.53ff.87f0 Interface ARPA Bundle-Ether16.16210.194.12.44 - 001b.53ff.87f0 Interface ARPA Bundle-Ether16.16310.194.16.44 - 001b.53ff.87ec Interface ARPA GigabitEthernet0/1/0/12

-------------------------------------------------------------------------------0/RSP0/CPU0-------------------------------------------------------------------------------Address Age Hardware Addr State Type Interface172.29.52.1 01:51:49 001e.f77d.2a19 Dynamic ARPA MgmtEth0/RSP0/CPU0/0172.29.52.13 03:29:42 0010.79e9.6038 Dynamic ARPA MgmtEth0/RSP0/CPU0/0172.29.52.21 00:50:04 0022.0d5a.a6c4 Dynamic ARPA MgmtEth0/RSP0/CPU0/0172.29.52.22 02:44:58 0001.6443.1678 Dynamic ARPA MgmtEth0/RSP0/CPU0/0172.29.52.27 02:36:46 0012.7fd6.ba08 Dynamic ARPA MgmtEth0/RSP0/CPU0/0172.29.52.28 03:04:37 0012.7fd6.ba09 Dynamic ARPA MgmtEth0/RSP0/CPU0/0

RP/0/RSP0/CPU0:router# show cef 192.168.1.1/32 hardware egress detail location 0/4/CPU0192.168.6.73/32, version 0, internal 0x40040001 (ptr 0x9f613944) [1], 0x0 (0x9eb9bdfc), 0x4500 (0xa0156184) Updated Sep 22 20:20:54.369 remote adjacency to GigabitEthernet0/1/0/23 Prefix Len 32, traffic index 0, precedence routine (0) gateway array (0x9e935bbc) reference count 249, flags 0xd00, source lsd (2), [84 type 5 flags 0x101001 (0x9fb06898) ext 0x0 (0x0)] LW-LDI[type=5, refc=3, ptr=0x9eb9bdfc, sh-ldi=0x9fb06898] via 10.114.4.11, GigabitEthernet0/1/0/23, 10 dependencies, weight 0, class 0 [flags 0x0] path-idx 0 next hop 10.114.4.11 remote adjacency local label 16021 labels imposed {16031} via 10.114.8.11, TenGigE0/4/0/0, 13 dependencies, weight 0, class 0 [flags 0x0]


OL-23591-02

Chapter 3 Troubleshooting Interface Connectivity Troubleshooting Bidirectional Forwarding Detection

path-idx 1 next hop 10.114.8.11 local adjacency local label 16021 labels imposed {16031}... TX Adjacency Raw data for tx adj struct: Raw result1: 0x03000100 0x01000000 0x7e23001b 0x0c6367ff Raw result2: 0x95650300 0x00000000 0x00000000 0x00000000 --------------------------------------------- Search Ctrl Flags: ------------------ match : 1 valid : 1 gre_adj : 0 null_route : 0 tx_punt : 0 tx_drop : 0 next_hop_down : 0 adj_complete : 0 punt_ifib : 0 nhop_down : 0 stop : 0 match_all_bit: 0 default_action: 1 uidb_index : 0x0001 l3_mtu : 9086 dest mac : 001b.0c63.67ff prefix_adj_cnt_index: 0x95650300

RX Adjacency Raw data for rx adj struct: Raw result1: 0x13000100 0x00001300 0x0c000280 0x00000000 --------------------------------------------- Search Ctrl Flags: ------------------ rx_punt : 0 rx_drop : 0 rx_adj_SFP : 1 rp_destined : 0 rp_drop : 0 match : 1 valid : 1 rx_LAG_adj : 0 match_all_bit : 0 pri_adj_down : 0 default_action: 1 rx_adj_field : 0x0013 egress_ifh : 0xc000280

Load distribution: 0 1 (refcount 84)

Hash OK Interface Address 0 Y GigabitEthernet0/1/0/23 remote 1 Y TenGigE0/4/0/0 10.114.8.11

Troubleshooting Bidirectional Forwarding DetectionBidirectional Forwarding Detection (BFD) is a detection protocol designed to provide fast forwarding path failure detection times for all media types, encapsulations, topologies, and routing protocols. This section contains the following subsections:

• Using show and debug Commands, page 3-82

• BFD Sessions in Down State, page 3-83

• BFD Sessions Flap, page 3-83


OL-23591-02


• BFD Sessions Down on Neighboring Router, page 3-85

• BFD Sessions Are Not Created on the LC, page 3-85

Using show and debug Commands

SUMMARY STEPS

1. show bfd [ipv4 | all] [location node-id]

2. show bfd client [detail]

3. show bfd [ipv4 | all] session [detail | [interface ifname] | [location node-id] ] [detail]

4. show bfd counters packet [ interface ifname] location node-id

5. show bfd trace {adjacency | error | fsm | packet} [interface ifname] [location node-id]

6. show tech-support routing bfd {file | location | rack}

DETAILED STEPS


Step 1 show bfd [ipv4 | all] [location node-id]

Example:RP/0/RSP0/CPU0:router# show bfd location 0/4/CPU0

View general BFD information on the Route Switch Processor (RSP), such as the number of sessions. Use the location keyword to display information for a specific LC. If not specified, information for all locations displays.

Step 2 show bfd client [detail]

Example:RP/0/RSP0/CPU0:router# show bfd client detail

View BFD clients. Use the detail keyword to display more information.

Step 3 show bfd [ipv4 | all] session [detail | [interface ifname] | [location node-id] ] [detail]

Example:RP/0/RSP0/CPU0:router# show bfd session interface Gig2/1/0/0 detail

View BFD session information. Filter results using the following parameters and keywords:

• location—BFD sessions hosted on this location.

• interface—BFD sessions on the specified interface (no wildcards).

• detail—Detailed session information: statistics, number of state transitions.

Step 4 show bfd counters packet [interface ifname] location node-id

Example:RP/0/RSP0/CPU0:router# show bfd counters packet interface POS 0/3/0/0 location 0/3/cpu0

View packet counters information. Filter results using the following parameters and keywords:

• location—Packet counters for BFD sessions hosted on this location.

• interface—Packet counters for BFD sessions on the specified interface (no wildcards).

• invalid—Invalid packet counter information.


OL-23591-02


BFD Sessions in Down StateTo troubleshoot BFD sessions in the down state, perform the following steps.

Step 1 Verify IP connectivity. Verify there is no IP packet loss. RP/0/RSP0/CPU0:router# ping local-remote-address

Step 2 Ensure that the router and remote device are configured with the following parameters:

• Number of BFD sessions they can support

• Timers to support the police rates

BFD Sessions FlapTo check various BFD parameters, perform the following steps. Also see:

• BFD Sessions Down on Neighboring Router, page 3-85

• BFD Sessions Are Not Created on the LC, page 3-85

Step 1 Verify IP connectivity. RP/0/RSP0/CPU0:router# ping local-IP-address

Step 5 show bfd trace {adjacency | error | fsm | packet} [interface ifname] [location node-id]

Example:RP/0/RSP0/CPU0:router# show bfd trace fsm location 0/4/CPU0

View tracing information from the RSP. Filter results using the following parameters and keywords:

• adjacency—Traces generated when BFD receives an adjacency update from the Adjacency Information Base (AIB) Finite State Machine (FSM) display.

• error—Traces generated when an error is detected.

• fsm—Traces generated when there is a state change in a session.

• packet—Traces generated when there is a change in a Tx or Rx packet.

• location—Traces for BFD traces on the specified interface.

Note Log the trace to a file to save the results.

Step 6 show tech-support routing bfd {file | location | rack}

Example:RP/0/RSP0/CPU0:router# show tech-support routing bfd location 0/1/CPU0

View BFD debugging information.



OL-23591-02


Step 2 View input and output counters. RP/0/RSP0/CPU0:router# show interface

Step 3 View session detail information. RP/0/RSP0/CPU0:router# show bfd session detail

Step 4 View session packet counters. RP/0/RSP0/CPU0:router# show bfd counter

Step 5 View SPP counters. RP/0/RSP0/CPU0:router# show spp node

Note SPP means software packet processing, but is more commonly referred to as vector path processing (VPP).

Step 6 View resource usage. RP/0/RSP0/CPU0:router# monitor process

Step 7 View IP connectivity. Verify there is no IP packet loss. RP/0/RSP0/CPU0:router# ping local remote address If the following message appears, the BFD flap is a result of the application flap.

bfd_agent[104]: %BFD-6-SESSION_REMOVED : BFD session to neighbor 192.168.1.1 on interface Gi0/5/0/0 has been remove

Step 8 Verify that the SPP is not losing packets. RP/0/RSP0/CPU0:router# show spp node location

Step 9 Check LC CPU and memory usage. RP/0/RSP0/CPU0:router# monitor processes location

Step 10 Check the local interface counters. RP/0/RSP0/CPU0:router# show interfaces type interface-name

Step 11 Check any QoS policies applied to the interface. RP/0/RSP0/CPU0:router# show policy-map interface

Step 12 Repeat Step 1 through Step 11on the remote end.

BFD Sessions Flap Because of Local Echo Failure

BFD sessions flap may be locally triggered because the router detects echo failure.


OL-23591-02

Chapter 3 Troubleshooting Interface Connectivity Troubleshooting Ethernet CFM

Examine LC CPU utilization: RP/0/RSP0/CPU0:router# monitor process location

Examine the SPP process on the LC CPU to determine the delay encountered by BFD echo packets: RP/0/RSP0/CPU0:router# show bfd trace performance reverse location

Rule out BFD echo packet loss: show bfd counters packet location

BFD Sessions Flap Because of SPP Process Restart

If BFD failure detection is configured to be within 1 second, the BFD session would flap if the SPP process is restarted on the LC.

BFD Sessions Down on Neighboring RouterThe neighbor router sends this message to indicate its BFD is going down:

LC/0/6/CPU0:Aug 8 16:42:56.821: bfd_agent[104]: %L2-BFD-6-SESSION_STATE_DOWN: BFD session to neighbor 192.168.1.1 on interface Gi0/5/0/0 has gone down. Reason: Nbor signalled down

BFD Sessions Are Not Created on the LCUp to 1024 BFD sessions are allowed per LC. Configuring more than 1024 BFD sessions may result in random BFD sessions not being created.

Troubleshooting Ethernet CFMEthernet Connectivity Fault Management (CFM) monitors, detects, and diagnoses remote network faults end-to-end across the network. It does this using keepalives and MAC-based ping and traceroutes. Unlike most other Ethernet protocols which are restricted to a single physical link, CFM frames can transmit across the entire end-to-end Ethernet network.

This section describes how to troubleshoot problems with CFM on the local ASR 9000 router. For more information on how to use CFM to troubleshoot problems across the network, see the “Ethernet CFM” section in Cisco ASR 9000 Series Aggregation Services Router Interface and Hardware Component Configuration Guide.

Figure 3-2 shows an example of maintenance domains across a network.


OL-23591-02


Figure 3-2 CFM Maintenance Domains Across a Network

This section contains the following topics:


• MEPs Are Not Created, page 3-88

• MIPs Are Not Created, page 3-88

• No CCMs are Received at the MEP or Peer MEPs Are Not Seen, page 3-89

• Peer MEP Defects and Mismatches Are Seen, page 3-90

• Remote Defect Indication Received, page 3-91

• Peer MEP Times Out But No Alarm Or Action Occurs, page 3-92

• No Debugs or Counters for Higher-Level Packets at a MEP or MIP, page 3-92

• Dropped CFM PDUs, page 3-92

• CFM ping Or traceroute Returns a “not found” Error, page 3-93

• AIS Messages Are Not Sent, page 3-93

Tip For an extensive discussion of CFM usage and CFM command examples, see Cisco ASR 9000 Series Aggregation Services Router Interface and Hardware Component Configuration Guide.

2075

81

Operator Domains

Service Provider Domain Level 6

Operator 1PE 1CE 1 CE 2PE 2 PE 3 PE 4

Operator 2

Customer Domain Level 7

MEP

Level 6 Level 6

Level 4

Level 3

MEP MEP

MIP

MIP MIP

MEP MEPMIP MIP

MIP MEP


OL-23591-02


Using show and debug CommandsThe show and debug commands in this section are useful for troubleshooting CFM. Further details on these commands can be found in Cisco ASR 9000 Series Aggregation Services Router Interface and Hardware Component Command Reference.

These commands are useful for checking the validity of the configuration commands:

• show run ethernet cfm—View all CFM global configuration.

• show ethernet cfm configuration-errors—Displays any problems that have been detected in the CFM configuration.

These commands are useful for verifying CFM functions:

• show ethernet cfm local maintenance-points—Displays a summary of the MEPs and MIPs that have been created.

• show ethernet cfm local meps—Displays information about local MEPs, including continuity check messages (CCMs), details about the types of packets being sent and received, and counters for each packet type.

• show ethernet cfm peer meps—Displays information about peer MEPs, including details about any peer MEP defects that have been detected.

• show ethernet cfm traceroute-cache—Displays the contents of the traceroute cache, that is, the result of recent traceroute operations.

• show ethernet cfm interfaces ais—Displays a summary of interfaces where AIS messages are being sent or received.

• show ethernet cfm interfaces statistics—Displays counters for CFM PDUs that are dropped per interface.

• show ethernet cfm ccm-learning-database—Displays the contents of the CCM learning database, which the system uses when it responds to received traceroute (linktrace) messages.

• debug ethernet cfm packets—Enables debugging of sent and received CFM PDUs.

• debug ethernet cfm protocol-state—Enables debugging of major CFM protocol state-machine operations.

If you need to collect information to provide to Cisco, the following commands are also useful, in addition to those listed above. Note that many of these commands require the cisco-support task ID.

• show tech-support ethernet

• show ethernet cfm trace

• show ethernet cfm interfaces status

• show ethernet cfm services

• debug ethernet cfm platform—Displays platform-specific debugging information for CFM.

• debug ethernet oam platform—Displays platform-specific debugging information for OAM.

• show spp node—Displays SPP counters.

• show spp sid stats—Check the SPP stream ID (SID) statistics to see that CFM traffic is injected and punted.

• show spp client—Displays information from the RSP about traffic on the SPP. Check to see if there are any SPP drops.


OL-23591-02


Note To clear the spp counters, run the command clear spp {client | interface | node-counters} location node-id. This command clears client statistics, interface statistics, and per-node counters, depending on the keyword you use.

MEPs Are Not CreatedIf MEPs have been configured but have not been created, follow the troubleshooting steps in this section.

Step 1 Display information about errors that might be preventing configured CFM operations from becoming active, as well as any warnings that have occurred.

show ethernet cfm configuration-errors

Step 2 Display a list of local maintenance points that have been created. Verify that the list contains the expected nodes.

show ethernet cfm local maintenance-points

Step 3 Display operational states of local MEPs. Verify that the states are as expected.

show ethernet cfm local meps

MIPs Are Not CreatedThis section explains what to do if MIP creation has been configured but MIPs have not been created as expected. Understand the factors that can impact MIP creation, then troubleshoot the specific MIP creation issues.

Understanding the Factors that Impact MIP Creation

Configuring MIP creation for a service does not guarantee that MIPs will actually be created for all interfaces in that service. MIPs are only created on interfaces that are correctly configured for Layer 2 switching, that is, interfaces that:

• Are configured as Layer 2 interfaces

• Have an appropriate encapsulation configured

• Have been added to a bridge domain or point-to-point xconnect.

The CFM standard (IEEE 802.1ag-2007) specifies an algorithm that is used to determine whether a MIP should be created, and at what level. For details of this implementation, see the mip auto-create command in Cisco ASR 9000 Series Aggregation Services Router Interface and Hardware Component Command Reference.

Troubleshooting MIP Creation Issues

Follow these steps to troubleshoot MIP creation issues.


OL-23591-02


Step 1 Display information about errors that might be preventing configured CFM operations from becoming active, as well as any warnings that have occurred.

show ethernet cfm configuration-errors

Step 2 Display a list of local maintenance points that have been created. Verify that the list contains the expected nodes. Check for MEPs configured on the interface, and for MIPs enabled on a service at a lower level.

show ethernet cfm local maintenance-points

Step 3 If MIP creation is not functioning, verify that the bridge domain or xconnect is configured correctly. To verify or troubleshoot these bridge domain and xconnect configurations, see Chapter 9, “Troubleshooting L2VPN and Ethernet Services.”

No CCMs are Received at the MEP or Peer MEPs Are Not SeenThis section explains how to troubleshoot the following conditions:

• Continuity check messages (CCMs) are not seen at one or more maintenance end points (MEPs).

• Peer MEPs are not seen.

CFM MEPs exchange CCMs periodically according to parameters configured on the system. These CCMs are multicast to all other MEPs in the service at the same level. When the local MEP receives a CCM, it creates an entry in the peer MEP table. If CCMs are not being exchanged correctly, perform the following steps.

Step 1 Verify that CCM is enabled and there is a supported encapsulation on the interface.


Step 2 View configured MEPs and maintenance intermediate points (MIPs).

RP/0/RSP0/CPU0:router# show ethernet cfm local maintenance-points

Step 3 Display operational states of local and peer MEPs. Verify that CCM is enabled and the states are as expected.

RP/0/RSP0/CPU0:router# show ethernet cfm local meps RP/0/RSP0/CPU0:router# show ethernet cfm local meps verboseRP/0/RSP0/CPU0:router# show ethernet cfm peer meps RP/0/RSP0/CPU0:router# show ethernet cfm peer meps detail

Step 4 View packets seen by the CFM PI. Enable all of the options. The output shows if packets are dropped, forwarded, or processed.

RP/0/RSP0/CPU0:router# debug ethernet cfm packets packet-type ccm

Step 5 View remote MEPs shown by the specific LC CFM instance. If CCMs are not received, the peer does not display.

Step 6 View CFM SID statistics seen by the SPP. This displays any CFM traffic that is injected and punted.

RP/0/RSP0/CPU0:router# show spp sid stats

Step 7 View SPP drops.

RP/0/RSP0/CPU0:router# show spp client location location


OL-23591-02


Step 8 Show bi-state alarms, to check for invalid encapsulation.

Step 9 Check for dropped PDUs as described in the “Dropped CFM PDUs” section on page 3-92.

Peer MEP Defects and Mismatches Are SeenThis section explains what to do when MEPs are exchanging CCMs, but there are defects or mismatches in the received CCMs. Perform the following steps to locate the specific problem. Then take appropriate corrective action.

Step 1 Run the following commands to obtain the output you will need for detailed troubleshooting.

RP/0/RSP0/CPU0:router# show ethernet cfm local mepsRP/0/RSP0/CPU0:router# show ethernet cfm local meps verboseRP/0/RSP0/CPU0:router# show ethernet cfm peer mepsRP/0/RSP0/CPU0:router# show ethernet cfm peer meps detail

Step 2 Wrong level—Check the output of the commands in Step 1 to see if CCMs are being received at a lower level than the level of the local MEP. This indicates a misconfiguration, for example:

• The domain level is configured incorrectly on the local device or the peer device.

• An intended MEP at the lower level has not been configured, and as a result the CCMs it would consume are reaching the local MEP.

• The forwarding path within the network has been misconfigured, such that CCM packets are being received from an unintended source.

Step 3 Cross-connect (wrong MAID)—Check the output of the commands in Step 1 to see if CCMs are being received with an maintenance association identifier (MAID) that does not match the MAID configured locally for the service. The MAID is formed from the maintenance domain identifier (MDID) and the short maintenance association name (SMAN). By default, the MDID is set to the name of the domain and the SMAN is set to the name of the service. A crossconnect error indicates a misconfiguration, for example:

• The domain name or ID is configured incorrectly on the local device or on the peer device.

• The service name or ID is configured incorrectly on the local device or on the peer device.

• The forwarding path within the network has been misconfigured, such that CCM packets are being received from an unintended source.

Step 4 Wrong interval—Check the output of the commands in Step 1 to see if CCMs are being received with a CCM interval that does not match the locally configured CCM interval. This indicates that the interval is configured incorrectly on either the local device or the peer device. For a given service, the same CCM interval must be configured on all devices.

Step 5 Loop (local MAC address received)—Check the output of the commands in Step 1 to see if CCMs are being received with the source MAC equal to the MAC address of the interface for the local MEP. This indicates that there is a loop in the network such that the local device is receiving its own packets, or that two devices in the network are configured with the same MAC address.

Step 6 Configuration (local MEP ID received)—Check the output of the commands in Step 1 to see if CCMs are being received from a peer MEP with the same MEP ID as the local MEP. This defect indicates that two MEPs are configured with the same MEP ID. Across the entire network, each MEP in the service must be configured with a different MEP ID.


OL-23591-02


Step 7 Peer interface down—Check the output of the commands in Step 1 to see if CCMs are being received that indicate the interface on the peer MEP is down, or that the interface on every peer MEP is STP blocked. This indicates a problem with the operational state of the network.

Step 8 Missing (crosscheck)—Check the output of the commands in Step 1. If crosscheck is configured specifying this peer MEP, but no CCMs are being received, the peer MEP is missing. This might indicate a failure in the network.

Step 9 Unexpected (crosscheck)—Check the output of the commands in Step 1. If crosscheck is configured and CCMs are being received from a peer MEP that is not specified, these CCMs are unexpected. This may indicate a misconfiguration or that CCMs are being received from an unintended source.

Step 10 Remote defect received—Check the output of the commands in Step 1. If received CCMs indicate that the peer MEP has detected a defect, take the action recommended in the “Remote Defect Indication Received” section on page 3-91.

Remote Defect Indication ReceivedThis section explains what to do if the local router receives a remote defect indication (RDI) from a peer router.

Understanding How RDIs are Exchanged

When a MEP detects a defect (as described in the “Peer MEP Defects and Mismatches Are Seen” section on page 3-90), it sets the remote defect indication (RDI) in the CCMs it is sending. When another MEP receives the RDI, it recognizes that the peer MEP sending the CCMs has detected a defect.

• In a point-to-point service, most defects will be detected by both MEPs; therefore both MEPs will send the RDI and both will receive the RDI. However, a unidirectional failure in the network could cause one of the MEPs to detect a crosscheck missing defect, while the other MEP does not detect any defect. In this case, the RDI sent by the first MEP serves to notify the second MEP of the problem.

• In a multipoint service, if there is a defect on any MEP or pair of MEPs, all other MEPs in the service receive the RDIs from the MEP or MEPS that detected the defect.

Locating the Source of RDIs and Resolving Defects

Step 1 Run the following commands to obtain the output you will need for troubleshooting RDIs.

RP/0/RSP0/CPU0:router# show ethernet cfm peer mepsRP/0/RSP0/CPU0:router# show ethernet cfm peer meps detail

Step 2 Determine the peer MEP from which the RDI is being received.

Step 3 Log into the peer device and follow the steps in the “Peer MEP Defects and Mismatches Are Seen” section on page 3-90.


OL-23591-02


Peer MEP Times Out But No Alarm Or Action OccursThis section explains what to do if a peer MEP times out without generating an alarm or automatic corrective action. You need to verify that crosscheck is configured and functioning correctly.

If the local MEP stops receiving CCMs from a peer MEP, it times out by default after 3.5 times the CCM interval. This is refered to as a loss of continuity. By default, a loss of continuity does not trigger any other actions, such as log messages, SNMP traps, AIS, or Ethernet fault detection (EFD). These actions are only triggered if crosscheck is configured for the service and the peer MEP is specified. In this case, the loss of continuity causes a crosscheck missing defect, and this in turn triggers the other actions.

Step 1 Check that MEP crosscheck is configured for the service.


Step 2 Check for the crosscheck missing defect.

RP/0/RSP0/CPU0:router# show ethernet cfm peer meps

No Debugs or Counters for Higher-Level Packets at a MEP or MIPCFM packets at a higher level than the highest MEP or MIP configured on the interface are forwarded by the network processor and are not handled by the software. Therefore, it is not possible to display debugs or counters for these packets. In addition, certain packets at the same level as a MIP are forwarded by the network processor. Again it is not possible to display debugs or counters for these.

Dropped CFM PDUsIf CFM PDUs are not reaching the expected destination or are not being processed as expected, it is possible that they are being dropped. The PDU drops could be caused by any of these reasons:

• Dropped by the network processor

• Dropped when being passed to software due to exceeding the supported CFM packet rate of 16,000 packets per second per line card

Note Note that the CFM packet rate limit (16,000 CFM packets per second per line card) includes all CFM packet types, including linktrace (traceroute) and loopback (ping) packets, as well as CCMs and Ethernet SLA probes. Normally, the number of linktrace or loopback packets is low; however, the use of “continuity-check auto-traceroute” can cause a high number of linktrace packets to be sent, if a number of peer MEPs time out in quick succession.

• Dropped because the interface or the forwarding node is down

• Dropped because the PDU is invalid or not formed properly

• Dropped because a higher level MEP was reached

• Dropped due to an unknown PDU type

• Dropped because the configured maximum MEPs limit (default 100) has been reached for the service


OL-23591-02


To display information for troubleshooting dropped CFM PDUs, perform the following steps. Take corrective actions based on the outputs of the commands in these steps.

Step 1 Enable packet debugging to determine whether forwarded packets are being received at the MIP.

RP/0/RSP0/CPU0:router# debug ethernet cfm packets [received dropped interface gigabitEthernet node-id]

Step 2 Display the statistics of the CFM PDUs per interface. Look for any drops ted to packets that are improperly formed, invalid, wrong level, or unknown type.

RP/0/RSP0/CPU0:router# show ethernet cfm interfaces statistics

Step 3 Display the local MEPs and look for discarded CCMs. Discarded CCMs might indicate the the configured maximum MEPs limit (default 100 MEPs per service) is reached.

RP/0/RSP0/CPU0:router# show ethernet cfm local meps verbose

Step 4 View peer MEPs seen by every local MEP.


Step 5 Check the STP status on the interfaces with MEPs or MIPs. CFM PDUs originating at MEPs on a STP block port get forwarded, however, PDUs forwarded on a MIP are subject to the STP port state. This means that if MIP is on a port that is STP blocked, then CFM PDUs will be dropped at the MIP.

Step 6 View STP state and CFM peer MEP status.

RP/0/RSP0/CPU0:router# show spanning-tree mst mstp

CFM ping Or traceroute Returns a “not found” ErrorThis section explains what to do if you perform a CFM ping or traceroute and receive a “not found” error.

For the ping or traceroute commands, the target is specified by means of a MAC address or a MEP ID. If the target is specified as a MAC address, the MAC address is copied directly into the message. However, if a MEP ID is specified, the system looks in the peer MEP table to find the MAC address for the corresponding peer MEP. If there is no peer MEP for the service with the specified MEP ID, or if there is more than one peer MEP for the service with the specified MEP ID, this lookup fails and the system returns a “not found” error.

View the peer MEPs and check that there is an entry for the MEP ID that was being used as the target MEP ID in the ping or traceroute command.


AIS Messages Are Not SentAlarm indication signal (AIS) can be enabled in configuration, either for MEPs or explicitly on an interface. The system sends AIS messages when it detects a peer MEP defect, when it receives AIS or LCK messages, or when the interface is down. AIS messages are sent in one of two ways:


OL-23591-02


• If there is another MEP on the interface at a higher level, and in the same direction, the AIS messages are sent internally from the lower level MEP to the next highest level MEP. In this case, no actual PDUs are transmitted.

• Otherwise, if there is a MIP on the interface then AIS PDUs are transmitted at the level of the MIP. If there is no MIP on the interface, no AIS messages are transmitted.

Use the following steps for troubleshooting.

Step 1 Verify that AIS is enabled in the configuration.


Step 2 Verify that there is a MIP.

RP/0/RSP0/CPU0:router# show ethernet cfm local maintenance-points

Step 3 Display the information published in the interface AIS table, including a record of the AIS transmissions. Determine whether AIS messages are actually being sent.

RP/0/RSP0/CPU0:router# show ethernet cfm interfaces ais

Step 4 Determine whether the system should be sending AIS messages.

RP/0/RSP0/CPU0:router# show ethernet cfm local meps detail


OL-23591-02


C H A P T E R 4
Troubleshooting Packet Forwarding
This chapter explains how to troubleshoot router forwarding.

Cisco Express Forwarding (CEF) is the mechanism that enables packet forwarding. CEF information is examined when data forwarding is not occurring as expected. Troubleshooting CEF involves comparing the Routing Information Base (RIB) information to the software Forwarding Information Base (FIB), verifying that the hardware is programmed correctly, verifying that the adjacencies are built correctly, verifying the control plane is built correctly, and gathering any necessary trace information.

The only prerequisite for CEF is a valid route in the RIB.

This chapter includes the following sections:

• Understanding IPv4 CEF, page 4-95

• Troubleshooting IPv4 CEF, page 4-96

• Troubleshooting Adjacency Information, page 4-101

• Troubleshooting Transient Traffic Drop, page 4-106

• Troubleshooting Packet Drop in the Fabric, page 4-109

• Troubleshooting Control Plane Information, page 4-109

Understanding IPv4 CEFCEF is an advanced, Layer 3 IP switching technology that optimizes network performance. It also improves the scalability for networks with large and dynamic traffic patterns, such as the Internet and networks characterized by intensive Web-based applications.

Information conventionally stored in a route cache is stored in several data structures for CEF switching. The data structures provide optimized lookup for efficient packet forwarding. The two main components of CEF operation are forwarding information base (FIB) and adjacency tables:

• CEF uses a FIB to make IP destination prefix-based switching decisions. FIB maintains a mirror image of the forwarding information contained in the IP routing table. When routing or topology changes occur in the network, the IP routing table is updated, and those changes are reflected in the FIB. The FIB maintains next hop address information based on the information in the IP routing table. There is a one-to-one correlation between FIB entries and routing table entries, therefore FIB contains all known routes and eliminates the need for route cache maintenance that is associated with switching paths such as fast switching and optimum switching.

• Nodes in the network are said to be adjacent if they can reach each other with a single hop across a link layer. In addition to the FIB, CEF uses adjacency tables to prepend Layer 2 addressing information. The adjacency table maintains Layer 2 next-hop addresses for all FIB entries.


Chapter 4 Troubleshooting Packet Forwarding Troubleshooting IPv4 CEF

Figure 4-1 shows the components that contribute information to the CEF process, including autosynchronization of the RIB with the FIB.

Note In this document, the FIB is also referred to as the CEF table.

Figure 4-1 CEF Process

Troubleshooting IPv4 CEFTo troubleshoot IPv4 CEF information, perform the following procedure.

This procedure checks that neighbors are recognized, packets are flowing along the expected path, and packets are not being dropped between neighbor interfaces.

SUMMARY STEPS

1. show route ipv4 prefix

2. show cef ipv4 prefix mask detail

3. show cef ipv4 prefix mask detail location node-id (on ingress line card)

4. show cef ipv4 prefix mask detail location node-id (on egress line card)

5. show cef ipv4 prefix mask hardware ingress detail location node-id

6. show cef ipv4 prefix mask hardware egress detail location node-id

7. show cef ipv4 interface type instance location node-id

8. show cef ipv4 summary location node-id

9. show cef ipv4 trace location node-id

10. show cef platform trace ipv4 all location node-id

11. show controllers pse qfp feature forward client ltrace unicast error location node-id


BGP

ISIS

HWLC-CPURP

GSPRIB

LSD

BCDL

LDP

RSVP

FIBProcess

Netio

IFMGRAIB

OSPF

STATICROUTES

2088

08

SwFIB

HwFIB


OL-23591-02


DETAILED STEPS


Step 1 show route ipv4 prefix

Example:RP/0/RSP0/CPU0:router# show route 192.168.2.0

Displays the current routes in the Routing Information Base (RIB).

• Check the prefix and mask, as well as the next hop and outgoing interface, to ensure that they are what is expected.

• Note the timer value that shows how long the route has been in the routing table. If the timer value is low the route may be flapping.

A lower timer value is present when a route is installed in the RIB for a short period of time. A low timer value may indicate flapping. For example, if a BGP route was being installed and removed from the RIB table every sixty seconds, then the route is flapping.

Look for routes that have not been installed in the routing table for very long. The route will either be stable or flapping. If the route is flapping, contact contact Cisco Technical Support. For Cisco Technical Support contact information, see the “Obtaining Documentation and Submitting a Service Request” section on page xii in the Preface.

• Check that route is learned via the routing protocol you are expecting, and that the metric is what you expect.

Step 2 show cef ipv4 prefix mask detail

Example:RP/0/RSP0/CPU0:router# show route ipv4 192.168.2.0 255.255.255.0 detail

Displays the IPv4 Cisco Express Forwarding (CEF) table detailed entry information.

• Compare the prefix, mask, next hop ip, and outgoing interface information with the information in the RIB. The information in the RIB is displayed using the show route ipv4 prefix mask command as in Step 1.

• Check that the adjacency is valid or the expected type of adjacency. For example, if it is a remote adjacency, then the adjacency information exists on another node.

• Check that the expected hash (load balance) and egress interfaces are listed.


OL-23591-02


Step 3 show cef ipv4 prefix mask detail location node-id

Example:RP/0/RSP0/CPU0:router# show cef ipv4 192.168.2.0 255.255.255.0 detail location 0/14/cpu0

Displays the IPv4 CEF table for the designated ingress node.

• Compare the prefix, mask, next hop ip, and outgoing interface information with the information in the RIB. The information in the RIB is displayed using the show route ipv4 prefix command as in Step 1.


Check that the expected hash (load balance) and egress interfaces are listed.

Step 4 show cef ipv4 prefix mask detail location node-id

Example:RP/0/RSP0/CPU0:router# show cef ipv4 192.168.2.0 255.255.255.0 detail location 0/13/cpu0

Displays the IPv4 CEF table for the designated egress node.

• Compare the prefix, mask, next hop ip, and outgoing interface information with the information in the RIB. The information in the RIB is displayed using the show route ipv4 prefix mask command.


• Check that the expected hash (load balance) and egress interfaces are listed.

Step 5 show cef ipv4 prefix mask hardware ingress detail location node-id

Example:

RP/0/RSP0/CPU0:router# show cef ipv4 192.168.2.0 255.255.255.0 hardware ingress detail location 0/14/cpu0

Displays the IPv4 CEF table and corresponding forwarding chain for the designated ingress node.

• Check that the prefix and mask are valid.

• Check the nexthop IP address is as expected

• Check that the entry type is set to forward.

• Check the adjacency packet counter and byte counter.

Step 6 show cef ipv4 prefix mask hardware egress detail location node-id

Example:

RP/0/RSP0/CPU0:router# show cef ipv4 192.168.2.0 255.255.255.0 hardware detail egress location 0/13/cpu0

Displays the IPv4 CEF table and corresponding forwarding chain for the designated egress node.


• Check the nexthop IP address is as expected

• Check that the entry type is set to forward.

• Check the adjacency packet counter and byte counter.

Step 7 show cef ipv4 interface type instance location node-id

Example:

RP/0/RSP0/CPU0:router# show cef ipv4 interface

tengige 1/3/0/7 location 1/3/cpu0

Displays IPv4 CEF-related information for an interface.

Verify the interface handle ‘interface is marked’ is as expected. The command output also shows how many references there are to the interface in CEF table and the IPv4 MTU.

Use this command for the ingress and egress interfaces.



OL-23591-02


Examples

The following examples show routes to two networks, one that is directly connected and one that is learned. In the first example, the route was installed about 19 days ago, which might be as expected. However, in the second example, the route was installed only 54 seconds ago, so it appears to be flapping:

RP/0/RSP0/CPU0:router# show route ipv4 10.114.4.11 Tue Jul 13 09:25:47.754 DSTRouting entry for 10.114.4.0/24 Known via "connected", distance 0, metric 0 (connected) Installed Jul 12 14:18:06.668 for 19:07:41 <<< This route appears to be stable Routing Descriptor Blocks directly connected, via GigabitEthernet0/1/0/23 Route metric is 0 Redist Advertisers:

Step 8 show cef ipv4 summary location node-id

Example:RP/0/RSP0/CPU0:router# show cef ipv4 summary location 0/3/cpu0

Displays a summary of the IPv4 CEF table. Check the VPN routing and forwarding (VRF) names associated with the node, the route update drops, and that there are the expected number of incomplete adjacencies.

Note the number of routes CEF has entries for, the number of load sharing elements, and the number of references to this node.

Use this command for the ingress and egress line cards and route processor (RP).

Step 9 show cef ipv4 trace location node-id

Example:

RP/0/RSP0/CPU0:router# show cef ipv4 trace location 0/3/cpu0

Displays IPv4 CEF trace table information.

Check if there is any flap on the prefix.

Use this command for the RP, and ingress and egress interfaces for the local line card.

Step 10 show cef platform trace ipv4 all location node-id

Example:

RP/0/RSP0/CPU0:router# show cef platform trace ipv4 all location 0/3/cpu0

Displays CEF IPv4 hardware status and configuration trace table information.

Use this command for the ingress and egress interfaces for the local line card.

Step 11 show controllers pse qfp feature forward client ltrace unicast error location node-id

Example:RP/0/RSP0/CPU0:router# show contro pse qfp feature forward client ltrace unicast error location node-id

(For SIP-700 line cards only) Displays trace files that contain information on any engine error (if any) that occurred in the unicast hardware structure programming.

Step 12 Contact Cisco Technical Support. If the problem is not resolved, contact Cisco Technical Support. For Cisco Technical Support contact information, see the “Obtaining Documentation and Submitting a Service Request” section on page xii in the Preface.



OL-23591-02


ospf 100

RP/0/RSP0/CPU0:router# show route ipv4 10.119.4.19 Tue Jul 13 09:28:38.407 DSTRouting entry for 10.119.4.0/24 Known via "ospf 100", distance 110, metric 2, type intra area Installed Jul 12 15:00:10.327 for 00:00:54 <<< This route appears to be flapping Routing Descriptor Blocks 10.114.4.11, from 10.19.19.19, via GigabitEthernet0/1/0/23 Route metric is 2 10.114.8.11, from 10.19.19.19, via TenGigE0/4/0/0 Route metric is 2 No advertising protos.

The following examples show interface details.

RP/0/RSP0/CPU0:router# show cef ipv4 interface TenGigE 0/6/0/1 location 0/4/CPU0 Tue Jul 13 11:39:13.693 DSTUNKNOWN intf 0x00000001 is unknown if_handle 0x00000001 if_type 0x0 idb info 0xa4d610d8 flags 0x301 ext 0xa5fe50cc Vrf Local Info (0x0) Interface last modified Jul 12, 2010 14:17:49, modify Interface is marked as point to point interface Reference count 1 Next-Hop Count 8 Protocol Reference count 1 Protocol ipv4 not configured or enabled on this card Primary IPV4 local address NOT PRESENT

RP/0/RSP0/CPU0:router# show cef ipv4 interface TenGigE 0/6/0/1 location 0/6/CPU0 Tue Jul 13 11:39:39.969 DSTTenGigE0/6/0/1 is down if_handle 0x100000c0 if_type 0x1e idb info 0xa4d61298 flags 0x1 ext 0x0 Vrf Local Info (0x0) Interface last modified Jul 12, 2010 14:17:48, create Reference count 1 Next-Hop Count 0 Protocol Reference count 0 Protocol ipv4 not configured or enabled on this card Primary IPV4 local address NOT PRESENT

The following example shows the CEF summary. Use this display to check the VRF names, route update drops, and adjacencies:

RP/0/RSP0/CPU0:router# show cef ipv4 summary location 0/1/CPU0 Tue Jul 13 12:50:48.259 DSTRouter ID is 10.144.144.144IP CEF with switching (Table Version 552) for node0_1_CPU0 Load balancing: L4 Tableid 0xe0000000 (0xa4a6ddb0), Vrfid 0x60000000, Vrid 0x20000000, Flags 0x301 Vrfname default, Refcount 251 163 routes, 0 reresolve, 0 unresolved (0 old, 0 new), 13040 bytes 60 load sharing elements, 129968 bytes, 342 references 8 shared load sharing elements, 8564 bytes 52 exclusive load sharing elements, 121404 bytes 0 CEF route update drops, 0 CEF rcc update drops 176 revisions of existing leaves Resolution Timer: 15s 0 prefixes modified in place 0 deleted stale prefixes 99 prefixes with label imposition, 111 prefixes with label information 23 next hops 0 incomplete next hops0 PD backwalks on LDIs with backup path


OL-23591-02

Chapter 4 Troubleshooting Packet Forwarding Troubleshooting Adjacency Information

Troubleshooting Adjacency InformationTo troubleshoot adjacency information on Cisco IOS XR software, perform the following procedure.

SUMMARY STEPS

1. show arp location node-id

2. show arp traffic location node-id

3. show adjacency interface-type interface-instance remote detail location node-id

4. show adjacency interface-type interface-instance remote detail hardware location node-id

5. show adjacency ipv4 nexthop ipv4-address detail location node-id

6. show adjacency interface-type interface-instance detail location node-id

7. show adjacency ipv4 nexthop ipv4-address detail hardware location node-id

8. show adjacency interface-type interface-instance detail hardware location node-id

9. show adjacency trace location node-id

10. show adjacency trace client aib-client location node-id

11. show adjacency hardware trace location node-id

12. show cef adjacency tunnel-te tunnel-id hardware {egress | ingress} location node-id


DETAILED STEPS


Step 1 show arp location node-id

Example:RP/0/RSP0/CPU0:router# show arp location 0/12/cpu0

Displays the Address Resolution Protocol (ARP) for an egress line card with a broadcast interface.

Ensure that you can find the IP address and that correct MAC address of the neighbor is learned.

Step 2 show arp traffic location node-id

Example:RP/0/RSP0/CPU0:router# show arp traffic location 0/12/cpu0

Displays ARP traffic statistics for an egress line card with a broadcast interface.

Check for any errors or IP packet drops.

Step 3 show adjacency interface-type interface-instance remote detail location node-id

Example:RP/0/RSP0/CPU0:router# show adjacency pos 0/13/0/2 remote detail location 0/14/cpu0

Displays detailed CEF adjacency table information for a remote ingress line card.

Ensure that the output shows IPv4 adjacency information and that an adjacency exists.


OL-23591-02


Step 4 show adjacency interface-type interface-instance remote detail hardware location node-id

Example:RP/0/RSP0/CPU0:router# show adjacency pos 0/13/0/2 remote detail hardware location 0/14/cpu0

Displays adjacency information for a remote ingress line card.


• Check that the table look-up (TLU) pointers match the TLU pointers in the show cef ipv4 prefix mask hardware ingress detail location node-id command. For example:

Step 5 show adjacency ipv4 nexthop ipv4-address detail location node-id

Example:RP/0/RSP0/CPU0:router# show adjacency ipv4 nexthop 192.168.2.0 detail location 0/12/cpu0

Displays adjacencies on an egress line card with a broadcast interface that are destined to the specified IPv4 next hop.

When an egress interface is broadcast, use the show adjacency ipv4 nexthop command to display the adjacency information.

Compare the mac layer rewrite information that shows the destination L2 address in the first part followed by the source L2 address, and the Ethernet value with the output from the show arp location node-id command.

Step 6 show adjacency interface-type interface-instance detail location node-id

Example:RP/0/RSP0/CPU0:router# show adjacency pos 0/13/0/2 detail location 0/13/cpu0

Displays CEF adjacency table information for an egress line card with a point to point interface.

There should be two IPv4 entries in the command output. Ensure both entries exist.

• The SRC MAC only entry is used for multicast switching

• The point to point entry is used for unicast switching.

On broadcast interfaces you will have a SRC MAC only and one for each nexthop IP address. Please note the MTU is for the IPv4 minus the Layer 2 header. Use the show im chains command to display MTU details.

Step 7 show adjacency ipv4 nexthop ipv4-address detail hardware location node-id

Example:RP/0/RSP0/CPU0:router# show adjacency ipv4 nexthop 192.168.2.0 detail hardware location 0/12/cpu0

Displays the hardware programming associated with the adjacency. Verify that the packets are being switched in the hardware.

Step 8 show adjacency interface-type interface-instance detail hardware location node-id

Example:RP/0/RSP0/CPU0:router# show adjacency pos 0/13/0/2 detail hardware location 0/13/cpu0

Displays the hardware programming information for a point-to-point interface such as the Packet-over-SONET/SDH (POS) interface. The rewrite information is slightly different because there is no MAC rewrite string as there is in Ethernet.

Verify that the rewrite is appropriate for the encapsulation on the interface. Compare the CEF hardware output and verify that the pointer matches the egress adjacency.



OL-23591-02


Examples

RP/0/RSP0/CPU0:router# show adjacency pos 0/2/0/1 remote detail hardware location 0/0/CPU0 Wed Nov 3 13:16:32.119 DSTInterface Address Version Refcount ProtocolPO0/2/0/1 (remote) 15 1( 0) fint_n2n 040001c0 flags 1 0 2 0 packets, 0 bytes

RP/0/RSP0/CPU0:router# show cef 10.3.3.3 hardware ingress location 0/2/CPU0 Wed Nov 3 13:19:23.263 DST10.3.3.3/32, version 0, internal 0x40040001 (ptr 0xa667ad70) [1], 0x0 (0xa5728bc4), 0x4500 (0xa754df28) Updated Oct 12 18:26:50.344 remote adjacency to GigabitEthernet0/1/0/23 Prefix Len 32, traffic index 0, precedence routine (0) via 10.114.4.11, GigabitEthernet0/1/0/23, 10 dependencies, weight 0, class 0 [flags 0x0] path-idx 0 next hop 10.114.4.11 remote adjacency local label 16018 labels imposed {16012} via 10.114.8.11, TenGigE0/4/0/0, 12 dependencies, weight 0, class 0 [flags 0x0]

Step 9 show adjacency trace location node-id

Example:RP/0/RSP0/CPU0:router# show adjacency trace location 0/1/cpu0

Displays CEF adjacency trace table information.

Use this command for the egress interfaces for the local line card.

Step 10 show adjacency trace client aib-client location node-id

Example:RP/0/RSP0/CPU0:router# show adjacency trace client ipv4_fib_mgr location 0/13/cpu0

Displays CEF adjacency trace table information for a specified adjacency information base (AIB) client.


Step 11 show adjacency hardware trace location node-id

Example:RP/0/RSP0/CPU0:router# show adjacency hardware trace location 0/13/cpu0

Displays CEF adjacency hardware trace table information.


Step 12 show cef adjacency tunnel-te tunnel-id hardware {egress | ingress} location node-id

Example:show cef adjacency tunnel-te 1 hardware egress location 0/13/CPU

Displays the IPv4 tunnel engineering (TE) tunnel adjacencies. Verify the tunnel adjacencies are as expected.




OL-23591-02


path-idx 1 next hop 10.114.8.11 remote adjacency local label 16018 labels imposed {16012}

TBM Node Data:Node (0x00000002):0 0x8952700d 0x00000004 0x00000000 0xf7ff0000 Node (0x89527010):1 0x8944c2dd 0x88f92d50 0x08888888 0x88888888 Node (0x8944c2f0):2 0x88f9453d 0x00000000 0x10000000 0x00000000 Node (0x88f94530):3 0x88f9454d 0x00000000 0x80000000 0x00000000 Node (0x88f94540):4 0x88f94555 0x00000000 0x01000000 0x00000000 Node (0x88f94550):5 0x00002020 0x00000000 0x88fccc60 0x88f96320

Hardware Leaf Data (0x88f94550):0x00002020 0x00000000 0x88fccc60 0x88f96320

IP Leaf Data: as:0 prefix_len:32 for_us:0x0 dft_route:0x0 real_intf:0x1 free1: 0x0 hw_use_only: 0x0 lspa_ptr: 0x0 oce_chain_p: 0x88f96320 extre_fib_data_ptr: 0x88fccc60

Hardware Extended Leaf Data: fib_leaf_extension_length: 0 interface_receive: 0x0 traffic_index_valid: 0x0 qos_prec_valid: 0x0 qos_group_valid: 0x0 valid_source: 0x0 traffic_index: 0x0 nat_addr: 0x0 reserved: 0x0 qos_precedence: 0x0 qos_group: 0x0 peer_as_number: 0 path_list_ptr: 0x0 connected_intf_id: 0x0 ipsub_session_uidb: 0xffffffff Path_list: urpf loose flag: 0x0 List of interfaces:

OCE Loadbalance Data for ptr 0x88f96320: num_entries:2 level:0x1 pad_1:0x0 l3_lbe_ptr:0x8942d140

LBE Array for 0x8942d140 Entry 0: oce_chain_p 0x88f975b0 Entry 0: bgp_ipv4_next_hop_addr: 0x0 Entry 1: oce_chain_p 0x88f96e40 Entry 1: bgp_ipv4_next_hop_addr: 0x0

OCE Label Object Data for ptr 0x88f975b0: flags: 0x0 number of labels: 1 protocol: 0 number bk labels: 0 out labels: 0x3e92 next_hw_oce_ptr: 0x88f97850 counter_ptr: 0x893e9720 Stats for ptr 0x893e9720: byte count: 0 packet count: 0

OCE RX Adj Data for 0x88f97850: base: 37(CPP HW RX ADJ MPLS) adj_flags: 0x0 pd_16: 0x1005 pd_32: 0x2f output_uidb: 0x1fea counters_ptr: 0x893dc8a0 byte count: 0 packet count: 0

OCE Label Object Data for ptr 0x88f96e40: flags: 0x0 number of labels: 1 protocol: 0 number bk labels: 0 out labels: 0x3e92


OL-23591-02


next_hw_oce_ptr: 0x88f97840 counter_ptr: 0x893e9750 Stats for ptr 0x893e9750: byte count: 0 packet count: 0

OCE RX Adj Data for 0x88f97840: base: 37(CPP HW RX ADJ MPLS) adj_flags: 0x0 pd_16: 0x6013 pd_32: 0x1 output_uidb: 0x1fd0 counters_ptr: 0x893dc8b0 byte count: 0 packet count: 0

The following example shows that the address information matches. The addresses are indicated in bold.

RP/0/RSP0/CPU0:router# show arp location 0/1/cpu0

Address Age Hardware Addr State Type Interface10.27.50.157 02:08:34 0016.c761.f509 Dynamic ARPA TenGigE0/1/0/2 RP/0/RSP0/CPU0:router# show adjacency ipv4 nexthop 212.27.50.157 detail loccation 0/1/cpu0

Interface Address Version Refcount ProtocolTenGigE0/1/0/2 10.27.50.157 41 2 ipv4 0016c761f5090015fa9959890800 mtu: 1500, flags 0 0 0 2894 packets, 156876 bytes 0xffffffff

RP/0/RSP0/CPU0:router# show adjacency gigabitEthernet 0/1/0/1 remote detail hardware location allWed Nov 3 13:10:23.519 DST-------------------------------------------------------------------------------0/1/CPU0-------------------------------------------------------------------------------Interface Address Version Refcount ProtocolGi0/1/0/1 (remote) 6 1( 0) fint_n2n 020000c0 flags 1 0 2 0 packets, 0 bytes-------------------------------------------------------------------------------0/RSP1/CPU0-------------------------------------------------------------------------------Interface Address Version Refcount Protocol-------------------------------------------------------------------------------0/RSP0/CPU0-------------------------------------------------------------------------------Interface Address Version Refcount ProtocolGi0/1/0/1 (remote) 7 1( 0) fint_n2n 020000c0 flags 1 0 2

RP/0/RSP0/CPU0:router# show cef adjacency tunnel-te 1 hardware egress location 0/3/CPU0 Wed Nov 3 13:37:17.935 DSTInterface not found (tunnel-te1)

Display protocol is ipv4Interface Address Type Refcount

BE16.162 special 2 Interface: BE16.162 Type: glean Interface Type: 0x19, Base Flags: 0x4400 (0x9e4e9bb0) Nhinfo PT: 0x9e4e9bb0, Idb PT: 0x9e3591d8, If Handle: 0x80001a0 Dependent adj type: remote (0x9f8af79c) Dependent adj intf: BE16.162


OL-23591-02

Chapter 4 Troubleshooting Packet Forwarding Troubleshooting Transient Traffic Drop

Ancestor If Handle: 0x0

BE16.163 special 2 Interface: BE16.163 Type: glean Interface Type: 0x19, Base Flags: 0x4400 (0x9e4e9d1c) Nhinfo PT: 0x9e4e9d1c, Idb PT: 0x9e359218, If Handle: 0x80001e0 Dependent adj type: remote (0x9f8b033c) Dependent adj intf: BE16.163 Ancestor If Handle: 0x0

tt44190 Prefix: 0.0.0.0/32 local 3 no next-hop adj Interface: NULLIFHNDL --More--

Troubleshooting Transient Traffic DropPerform this procedure to troubleshoot transient drops in packet forwarding. The approach to troubleshooting transient drops is as follows:

1. Determine the interface drops.

2. Determine the line card type. This is necessary because the next steps depend on whether you are troubleshooting an Ethernet or SIP-700 line card (LC).

3. (For Ethernet LC) Determine which NP contains the counters for the interface your are troubleshooting.

4. (For Ethernet LC) View the counters on the appropriate NP.

5. For SIP-700 LC, display the drop statistics on the LC.

SUMMARY STEPS

1. show interface interface-type node-id

2. show platform

3. show controllers np ports all location node-id (for Ethernet)

4. show controllers np count np-id location node-id (for Ethernet)

5. show controllers pse qfp stat drop location node-id (for SIP-700)

DETAILED STEPS


Step 1 show interface interface-type node-id

Example:show interface gigabitEthernet 0/0/0/0

Displays the interface drops.

Step 2 show platform Determines the line card type. This is necessary because the next steps depend on whether you are troubleshooting an Ethernet or SIP-700 line card (LC).


OL-23591-02


Example

RP/0/RSP0/CPU0:router# show interface gigabitEthernet 0/0/0/0Tue Oct 26 21:04:12.805 UTCGigabitEthernet0/0/0/0 is up, line protocol is up Interface state transitions: 5 Hardware is GigabitEthernet, address is 001b.53ff.a018 (bia 001b.53ff.a018) Internet address is 45.1.1.1/24 MTU 2014 bytes, BW 1000000 Kbit reliability 255/255, txload 0/255, rxload 0/255 Encapsulation ARPA, Full-duplex, 1000Mb/s, SXFD, link type is force-up output flow control is off, input flow control is off loopback not set, ARP type ARPA, ARP timeout 04:00:00 Last input 00:00:00, output 00:00:00 Last clearing of "show interface" counters 1w4d 5 minute input rate 4000 bits/sec, 0 packets/sec 5 minute output rate 11000 bits/sec, 0 packets/sec 1590651 packets input, 551036131 bytes, 0 total input drops <<< drops by framer or HW 97206 drops for unrecognized upper-level protocol <<< drops Received 0 broadcast packets, 332301 multicast packets 0 runts, 0 giants, 0 throttles, 0 parity 0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort <<< drops 1536152 packets output, 1427163508 bytes, 0 total output drops <<< sum of all output drops, including drops from buffer, qos, or HW. Output 0 broadcast packets, 339069 multicast packets 0 output errors, 0 underruns, 0 applique, 0 resets 0 output buffer failures, 0 output buffers swapped out 0 carrier transitions


Step 3 show controllers np ports all location node-id

Example:

(For Ethernet LC) Displays the port mapping between the interface and the NP. View the output and determine which NP contains the counters for the interface your are troubleshooting.

Step 4 show controllers np count {np-id | all} ocation node-id

show controllers np count {np-id all} location node-id | i DROP

Example:show controllers np count all location 0/0/CPU0

show controllers np count all location 00/0/CPU0 | i DROP

(For Ethernet LC) View the counters on the appropriate NP. The first command displays all counters, whether related to drops or not. The second command limits the display to only those counters that include the string DROP.

For additional information on interpreting NP counters, see the “Displaying Traffic Status in Line Cards and RSP Cards” section on page 7-147.

Step 5 show controllers pse qfp stat drop location node-id

Example:show controllers pse qfp stat drop location 0/6/CPU0

(For SIP-700 LC) Display the drop statistics on the LC.



OL-23591-02


Tue Oct 26 20:58:49.575 UTCNode Type State Config State-----------------------------------------------------------------------------0/RSP0/CPU0 A9K-RSP-4G(Active) IOS XR RUN PWR,NSHUT,MON0/0/CPU0 A9K-40GE-L IOS XR RUN PWR,NSHUT,NMON <<< Ethernet0/3/CPU0 A9K-8T/4-E IOS XR RUN PWR,NSHUT,MON <<< Ethernet0/4/CPU0 A9K-8T-E IOS XR RUN PWR,NSHUT,MON <<< Ethernet0/6/CPU0 A9K-SIP-700 IOS XR RUN PWR,NSHUT,NMON <<< SIP-7000/6/0 SPA-2XCHOC12/DS0 OK PWR,NSHUT,MON <<< SPA0/6/1 SPA-5X1GE-V2 OK PWR,NSHUT,MON <<< SPA

RP/0/RSP0/CPU0:router# show controllers np ports all loc 0/0/CPU0Tue Oct 26 20:57:11.468 UTC

Node: 0/0/CPU0:----------------------------------------------------------------

NP Bridge Fia Ports-- ------ --- ---------------------------------------------------0 0 0 GigabitEthernet0/0/0/30 - GigabitEthernet0/0/0/391 0 0 GigabitEthernet0/0/0/20 - GigabitEthernet0/0/0/292 1 0 GigabitEthernet0/0/0/10 - GigabitEthernet0/0/0/193 1 0 GigabitEthernet0/0/0/0 - GigabitEthernet0/0/0/9

RP/0/RSP0/CPU0:router# show controllers np counters all location 0/0/CPU0Tue Oct 26 20:54:53.095 UTC

Node: 0/0/CPU0:----------------------------------------------------------------

Show global stats counters for NP0, revision v3

Read 23 non-zero NP counters:Offset Counter FrameValue Rate (pps)------------------------------------------------------------------------------- 22 PARSE_ENET_RECEIVE_CNT 74772482296 60925 23 PARSE_FABRIC_RECEIVE_CNT 80571 0 26 MODIFY_FABRIC_TRANSMIT_CNT 36431746029 29685 28 PARSE_INGRESS_DROP_CNT 18816500 0

RP/0/RSP0/CPU0:router# show controllers np count all location 0/0/CPU0 | i DROP Tue Oct 26 20:56:10.714 UTC 28 PARSE_INGRESS_DROP_CNT 38183944221 0 30 RESOLVE_INGRESS_DROP_CNT 157639443 0 31 RESOLVE_EGRESS_DROP_CNT 2559 0 291 DROP_IPV4_NOT_ENABLED 38174791832 0 438 RESOLVE_MAC_NOTIFY_CTRL_DROP_CNT 2559 0 28 PARSE_INGRESS_DROP_CNT 18816500 0

Note For a description of how to interpret NP counter information, see the “Displaying Traffic Status in Line Cards and RSP Cards” section on page 7-147.

RP/0/RSP0/CPU0:router# show controllers pse qfp stat drop location 0/6/CPU0Tue Oct 26 20:57:49.864 UTC

Global Drop Statistics for QFP 0----------------------------------------------------------------Global Drop Stats Packets Octets----------------------------------------------------------------


OL-23591-02

Chapter 4 Troubleshooting Packet Forwarding Troubleshooting Packet Drop in the Fabric

AttnInvalidSpid 0 0 BadAdj 0 0 BadBhdr 0 0

Troubleshooting Packet Drop in the FabricTo check whether packets are being dropped in the fabric, use the following commands.

• show controllers fabric fia bridge stats location node-id

• show controllers fabric fia drops ingress location node-id

• show controllers fabric fia drops egress location node-id

• show controllers fabric fia stats location node-id

For detailed fabric troubleshooting procedures, see Chapter 7, “Troubleshooting Router Switch Fabric and Data Path.”

Troubleshooting Control Plane InformationTo troubleshoot control plane information on Cisco IOS XR software, perform the following procedure.

SUMMARY STEPS

1. show netio idb interface-type interface-instance location node-id

2. show uidb index

3. show uidb data location node-id interface-type interface-instance {ingress | egress}

4. show imds interface brief



OL-23591-02

Chapter 4 Troubleshooting Packet Forwarding Troubleshooting Control Plane Information

DETAILED STEPS


Step 1 show netio idb interface-type interface-instance location node-id

Example:RP/0/RSP0/CPU0:router# show netio idb tengige0/0/0/0 location 0/0/cpu0

Displays control plane information for the software switching path. The output provides useful statistics for determining software forwarding issues.

• Verify the encap and decap paths

• Ensure that all the appropriate steps in the chain are shown for all the features that may be enabled on the interface.

Note Fixup is a direct pointer to a routine in the output path after a CEF rewrite. this is an optimized path if a CEF rewrite exists and is usable.

• Verify that the ifhandle and global uidb value is correct.


Step 2 show uidb index

Example:RP/0/RSP0/CPU0:router# show uidb index

Displays the micro-interface descriptor block (IDB) index assigned by the software.

Check that the interface and the universal interface descriptor block (UIDB) value are what is expected.

Compare the IDB index to the uidb index value in the show adjacency ipv4 interface-type interface-instance detail hardware location node-id command output.

Step 3 show uidb data location node-id interface-type interface-instance {ingress | egress}

Example:show uidb data location 0/6/CPU0 gigabitEthernet 0/0/0/2 ingress

Displays, from a software perspective, features that are enabled on a selected interface.

• Check the UIDB value.

• Check what flags are enabled for the UIDB.

• Check the ifhandle in the UIDB to make sure it is correct.

Compare the output to the configuration of the interface and expected features.


Step 4 show imds interface brief

Example:RP/0/RSP0/CPU0:router# show imds interface brief

Displays interface manager distribution server (IMDS) interface information.

Note This is just a partial output not full output.

Check the state, MTU, encapsulation being used, and the ifhandle for each interface.



OL-23591-02


Examples

The following example displays the control plane information for the software switching path. Check for any errors or drops.

RP/0/RSP0/CPU0:router# show netio idb tenGigE 0/1/1/0 location 0/1/cpu0

TenGigE0/1/1/0 (handle: 0x01180020, nodeid:0x11) netio idb:---------------------------------name: TenGigE0_1_1_0interface handle: 0x01180020interface global index: 2physical media type: 30dchain ptr: <0x482ae8e0>echain ptr: <0x482d791c>fchain ptr: <0x482d79b8>driver cookie: <0x4824ad58>driver func: <0x4824ad44>number of subinterfaces: 4096subblock array size: 3DSNCNF: 0x00000000interface stats info: IN unknown proto pkts: 0 IN unknown proto bytes: 0 IN multicast pkts: 0 OUT multicast pkts: 0 IN broadcast pkts: 0 OUT broadcast pkts: 0 IN drop pkts: 0 OUT drop pkts: 0 IN errors pkts: 0 OUT errors pkts: 0

Chains--------------------Base decap chain: ether <30> <0xfd7aef88, 0x48302824> < 0, 0>

Protocol chains:---------------<Protocol number> (name) Stats Type Chain_node <caps num> <function, context> <drop pkts, drop bytes><7> (arp) Stats IN: 0 pkts, 0 bytes; OUT: 0 pkts, 0 bytes Encap: l2_adj_rewrite <86> <0xfcec7a88, 0x4834efec> < 0, 0> queue_fifo <56> <0xfcedda68, 0x482dbee4> < 0, 0> txm_nopull <60> <0xfcea2a5c, 0x482dc11c> < 0, 0> Decap: queue_fifo <56> <0xfcedda4c, 0x482dbee4> < 0, 0> arp <24> <0xfd1082cc, 0x00000000> < 0, 0> Fixup: l2_adj_rewrite <86> <0xfcec745c, 0x00000000> < 0, 0> queue_fifo <56> <0xfcedda68, 0x482dbee4> < 0, 0> txm_nopull <60> <0xfcea2a5c, 0x482dc11c> < 0, 0><12> (ipv4) Stats IN: 0 pkts, 0 bytes; OUT: 0 pkts, 0 bytes Encap: ipv4 <26> <0xfd10f41c, 0x482d7724> < 0, 0> ether <30> <0xfd7aeb44, 0x48302824> < 0, 0> l2_adj_rewrite <86> <0xfcec7a88, 0x4834f104> < 0, 0> queue_fifo <56> <0xfcedda68, 0x482dbee4> < 0, 0> txm_nopull <60> <0xfcea2a5c, 0x482dc11c> < 0, 0> Decap: queue_fifo <56> <0xfcedda4c, 0x482dbee4> < 0, 0>


OL-23591-02


ipv4 <26> <0xfd10f474, 0x00000000> < 0, 0> Fixup: l2_adj_rewrite <86> <0xfcec745c, 0x00000000> < 0, 0> queue_fifo <56> <0xfcedda68, 0x482dbee4> < 0, 0> txm_nopull <60> <0xfcea2a5c, 0x482dc11c> < 0, 0><22> (ether_sock) Stats IN: 0 pkts, 0 bytes; OUT: 0 pkts, 0 bytes Encap: ether_sock <98> <0xfd7b1630, 0x48302824> < 0, 0> l2_adj_rewrite <86> <0xfcec7a88, 0x48304c1c> < 0, 0> queue_fifo <56> <0xfcedda68, 0x482dbee4> < 0, 0> txm_nopull <60> <0xfcea2a5c, 0x482dc11c> < 0, 0> Decap: queue_fifo <56> <0xfcedda4c, 0x482dbee4> < 0, 0> ether_sock <98> <0xfd7b1874, 0x48302824> < 0, 0> Fixup: l2_adj_rewrite <86> <0xfcec745c, 0x00000000> < 0, 0> queue_fifo <56> <0xfcedda68, 0x482dbee4> < 0, 0> txm_nopull <60> <0xfcea2a5c, 0x482dc11c> < 0, 0>



The following example shows that the micro-idb index value is 12.

RP/0/RSP0/CPU0:router# show uidb index tengige1/3/0/6.30 location 1/3/cpu0

------------------------------------------------------------------------------ Location Interface-name Interface-Type Ingress-index Egress-index--------------------------------------------------------------------------- 1/3/CPU0 TenGigE1_3_0_6.30 Sub-interface 20 12

Comparing the IDB index value of 12 in the show uidb index command to the uidb index value in the following command output shows that the values are the same.

RP/0/RSP0/CPU0:router# show adjacency ipv4 tengige1/3/0/6.30 detail hardware location 1/3/cpu0

Interface Address Version Refcount ProtocolTenGigE1/3/0/6.30 (src mac only) 90 1 ipv4 000000000000001243602d8b8100001e0800 mtu: 1500, flags 1 0 1 453 packets, 42582 bytes 453 hw-only-packets, 42582 hw-only-bytes ether egress adjacency TLU1 : 0x4407 [HW: 0x00401862 0xc4170800 0x8100001e 0x01060700] num. entries : 1 uidb index : 12 counter msb : 0x2 counter lsb : 0xc417 vlan e or len : 0x800 ether len : 0x8100 (33024) vlan info : 30 next ptr : 0x10607


OL-23591-02


The following example displays, from a software perspective, features that are enabled on a selected interface. Compare the output to the configuration of the interface and expected features. Verify that the configured features are correctly enabled.

RP/0/RSP0/CPU0:router# show uidb data location 0/6/cpu0

-------------------------------------------------------------------------- Location = 0/6/CPU0 Index = 0 Pse direction = INGRESS

Global general 16 bytes: ------------------------ ROUTER_ID: 45.104.151.108 MINIMUM MASK DESTINATION: 0 / 0 MINIMUM MASK SOURCE: 0 / 0 BYTES OF SNIFF PACKET: 0 SUPPRESS PUNT ACL: 0 MPLS PROPAGATE TTL FLAG: 1 PARITY: 0 FABRIC QOS ENABLE FLAG: 0-------------------------------------------------------------------------- Location = 0/6/CPU0 Index = 0 Pse direction = EGRESS

Global general 16 bytes: ------------------------ ROUTER_ID: 45.104.151.108 MINIMUM MASK DESTINATION: 0 / 0 MINIMUM MASK SOURCE: 0 / 0 BYTES OF SNIFF PACKET: 0 SUPPRESS PUNT ACL: 0 MPLS PROPAGATE TTL FLAG: 1 PARITY: 0 IPV4 PREFIX ACCNTG: 0-------------------------------------------------------------------------- Location = 0/6/CPU0 Ifname/Ifhandle = GigabitEthernet0_6_5_0 Index = 1 Pse direction = INGRESS

General 16 bytes: ----------------- IFHANDLE: 0x168002 STATUS: 0 IPV4 ENABLE: 0 IPV6 ENABLE: 0 MPLS ENABLE: 0 STATS POINTER: 0x2c400 SPRAYER QUEUE: 32 IPV4 MULTICAST: 0 IPV6 MULTICAST: 0 USE TABLE ID IPV4: 0 USE TABLE ID IPV6: 0 USE TABLE ID MPLS: 0 TABLE ID: 0 QOS ENABLE: 0 QOS ID: 0 NETFLOW SAMPLING PERIOD: 0 L2 PKT DROP: 0 L2 QOS ENABLE: 0 SRC FWDING: 0 *BUNDLE IFHANDLE: 0


OL-23591-02


*TUNNEL IFHANDLE: 0 *L2 ENCAP: 3

* Not programmed in hardware...


OL-23591-02


C H A P T E R 5
Troubleshooting Bundles and Load Balancing
This chapter explains the procedures for troubleshooting link bundles and load balancing on the Cisco ASR 9000 Aggregation Services Router.

A link bundle is a group of ports that are bundled together and act as a single link. The advantages of link bundles are:

• Multiple links can span several LCs to form a single interface; thus, the failure of a single link does not cause a loss of connectivity.

• Bundled interfaces increase bandwidth availability, because traffic is forwarded over all available members of the bundle. Therefore, traffic can move onto another link if one of the links within a bundle fails. This allows you to add or remove bandwidth without interrupting packet flow.

This chapter contains the following sections:

• Troubleshooting Routing and CEF Issues Related to Bundles and Load Balancing, page 5-115

• Troubleshooting Problems with Link Bundles, page 5-118

• Troubleshooting Layer 2 Bundles and Load Balancing, page 5-122

• Troubleshooting Layer 3 Bundles and Load Balancing, page 5-124

Troubleshooting Routing and CEF Issues Related to Bundles and Load Balancing

Cisco Express Forwarding (CEF) uses the path information in the IP routing table to balance traffic over link bundles. For this reason, verifying correct load balancing with CEF begins with confirming the contents of the IP routing table and CEF database.

• Verifying Routing Table Entries for Parallel Links, page 5-115

• Verifying the CEF Database and Measuring Flows, page 5-117

Verifying Routing Table Entries for Parallel LinksPerform this procedure to verify the contents of the IP routing table.

SUMMARY STEPS

1. show route destination-address


Chapter 5 Troubleshooting Bundles and Load Balancing Troubleshooting Routing and CEF Issues Related to Bundles and Load Balancing

2. configure

3. router ospf process

4. maximum paths number

5. end


7. show ospf process interface brief

8. show running-config router ospf process


Step 1 show route destination-address


Displays the routes to a destination address. Use a destination address on another host that is reachable through the parallel links.

Verify that number of routes in the routing table equals the number of parallel links. If you have fewer routes than expected, continue with this procedure.

Step 2 configure

Example:RP/0/RSP0/CPU0:router# configure

Enters global configuration mode.

Step 3 router ospf process

Example:RP/0/RSP0/CPU0:router# router ospf 200

Enters configuration mode for the OSPF process.

Step 4 maximum paths number

Example:RP/0/RSP0/CPU0:router# maximum paths 3

Configures the maximum number of paths over which to load balance. By default, OSPF balances up to 4 equal-cost paths.

Step 5 end

Example:RP/0/RSP0/CPU0:router# end

Ends the configuration process. Enter yes at the prompt to commit the changes.



Displays the routes to a destination address.

Verify that number of routes in the routing table equals the number of parallel links. If you have fewer routes than expected, continue with this procedure.


OL-23591-02

Chapter 5 Troubleshooting Bundles and Load Balancing Troubleshooting Routing and CEF Issues Related to Bundles and Load Balancing

Verifying the CEF Database and Measuring FlowsPerform this procedure to verify the contents of the CEF database.

SUMMARY STEPS

1. show cef ipv4 [prefix [mask]] | interface-type interface-path-id] [detail] [location node-id]

2. show cef [ipv4 | ipv6] exact-route source-address destination address [protocol type] [source-port source-port] [destination-port destination-port] [ingress-interface type interface-path-id] [policy-class value] [detail | location node-id]

3. show interfaces [type interface-path-id | all | local | location node-id] [accounting | brief | detail | summary]

DETAILED STEPS

Step 7 show ospf process interface brief

Example:RP/0/RSP0/CPU0:router# show ospf 200 interface brief

Shows interface information for all routes to the destination address, which displays the cost metric. OSPF balances loads over equal-cost routes only, so verify that the interfaces have equal costs. To load balance over unequal paths, use Enhanced Interior Gateway Routing Protocol or Interior Gateway Routing Protocol (EIGRP/IGRP) as the IGP instead.

Step 8 show running-config router ospf process

Example:RP/0/RSP0/CPU0:router# show running-configuration router ospf process

Displays the running configuration for the OSPF process. This is another way to determine if the interfaces have different costs.



Step 1 show cef ipv4 [prefix [mask]] | interface-type interface-path-id] [detail] [location node-id]

Example:RP/0/RSP0/CPU0:router# show cef ipv4 10.1.2.1 detail

Displays the CEF forwarding table. Verify that it contains the same interfaces that the routing table has for this destination.


OL-23591-02

Chapter 5 Troubleshooting Bundles and Load Balancing Troubleshooting Problems with Link Bundles

Troubleshooting Problems with Link BundlesThis section explains how to troubleshoot problems with link bundles. It contains the following subsections:

• Bundle Does Not Come Up, page 5-118

• Bundle Member Not Distributing, page 5-119

• Bundle Not Using MAC-Address From Backplane, page 5-119

• Layer 3 Data Traffic Not Flowing, page 5-120

• Ping Failed over Bundle, page 5-120

• Layer 3 Packets Not Synching Over Bundle, page 5-121

• Layer 2 Traffic Not Flowing, page 5-121

• Bundle Statistics, page 5-122

Bundle Does Not Come Up

Step 1 Ensure that the member port is not “shutdown”. Ensure that the MAC burned-in address (BIA) of the port is valid.

RP/0/RSP0/CPU0:router# show interface

Step 2 If running Link Aggregation Control Protocol (LACP), ensure that LACP packets are able to send and receive accordingly. If LACP packets are not able to send and receive accordingly, check interface counters to identify at what stage packets are dropped.

RP/0/RSP0/CPU0:router# show lacp counters

Step 2 show cef [ipv4 | ipv6] exact-route source-address destination address [protocol type] [source-port source-port] [destination-port destination-port] [ingress-interface type interface-path-id] [policy-class value] [detail | location node-id]

Example:RP/0/RSP0/CPU0:router# show cef exact-route 192.168.254.1 10.1.2.1 protocol ospf source-port 5500 destination-port 80 ingress-interface gi0/6/5/4

Displays the exact route that a specific flow would take, including the egress interface for a specific source and destination IP. Use this command for several flows to verify that they are distributed equally over the parallel interfaces.

Step 3 show interfaces [type interface-path-id | all | local | location node-id] [accounting | brief | detail | summary]

Example:RP/0/RSP0/CPU0:router# show interfaces accounting rates

Displays the traffic rates by interface. Use this command to verify that the simulated traffic takes the expected egress interface.



OL-23591-02


Step 3 View LACP statistics.

RP/0/RSP0/CPU0:router# show lacp

Step 4 Ensure that the other side of the link is up (bundle and members).

RP/0/RSP0/CPU0:router# show bundle

Bundle Member Not Distributing

Step 1 Ensure that the member is up. RP/0/RSP0/CPU0:router# show interface node-id

Step 2 Ensure that the remote side is up.

Step 3 Ensure that the LACP parameters are the same on both sides.

a. If LACP is enabled, check its status.

RP/0/RSP0/CPU0:router# show lacp bundle

b. Ensure that bundle members have the same characteristics.

RP/0/RSP0/CPU0:router# show running interface interface-name

c. If the bundle members have different characteristics, make them all the same.

d. Ensure that LACP packets are transmitted and received.

RP/0/RSP0/CPU0:router# debug bundlemgr local packets port node-id

Workaround

If the bundle with LACP cannot come up, use one side of the bundle in passive mode and the other in active mode. At least one side must be active.

Bundle Not Using MAC-Address From Backplane

Step 1 Ensure that the backplane MAC is programmed. Note that this command has to be run in the admin mode.

RP/0/RSP0/CPU0:router(admin)# show diag chassis eeprom-info

Step 2 Display the backplane information.

RP/0/RSP0/CPU0:router# show controllers backplane bpe-trace


OL-23591-02


Layer 3 Data Traffic Not Flowing

Regular Interface (No Subinterfaces)

Step 1 View the Address Resolution Protocol (ARP). RP/0/RSP0/CPU0:router# show arp

Step 2 Verify that the lag table is programmed properly in the hardware. RP/0/RSP0/CPU0:router# show interface bundle-ether bundle-id

Step 3 View the running configuration information. RP/0/RSP0/CPU0:router# show running-config

Step 4 View information about packets forwarded by CEF. RP/0/RSP0/CPU0:router# show cef

Step 5 RP/0/RSP0/CPU0:router# show cef hardware ingress location node-id

Step 6 RP/0/RSP0/CPU0:router# show cef hardware egress location node-id

Subinterface

Step 1 Troubleshoot Layer 3 IPv4 traffic.

Step 2 Ensure that VLAN traffic coming in matches that on the incoming interface.

Ping Failed over Bundle

Step 1 View the ARP. RP/0/RSP0/CPU0:router# show arp

Step 2 View the ARP information on the particular LC or RSP. RP/0/RSP0/CPU0:router# show arp location node-id

Step 3 RP/0/RSP0/CPU0:router# show cef hardware detail location node-id ingress

Step 4 RP/0/RSP0/CPU0:router# show interface

Step 5 Use the hash calculator to determine which bundle member (interface) to test.

Step 6 Remove the interface from the bundle.

Step 7 Assign an IP address to the interface.

Step 8 Ping the interface.

Step 9 Ensure that the ARP is resolved between the router and the node being pinged.


OL-23591-02


Step 10 Ensure that the MAC address in the ARP table of the other side corresponds to that on the router.

Step 11 Ensure that the MAC address of the bundle is valid.

Step 12 Ensure that the routing and hardware routing table has an entry to the next hop.

Step 13 Check the interface counters to see if ping packets are transmitted and being received on the router member port of the bundle.

Step 14 Check the ucode counters to see where packets are dropped on the incoming or outgoing member of the bundle.

Step 15 Make sure that the table lookup (TLU) entries are allocated and bundle adjacency information is properly programmed.

show cef adajacency bundle-type bundle-number hardware egress detail location location-id

show cef adajacency bundle-type bundle-number hardware ingress remote detail location location-id

Workaround

Try a different port.

Layer 3 Packets Not Synching Over Bundle

Step 1 View the interface information: RP/0/RSP0/CPU0:router# show interface

Step 2 Turn on the debug of that protocol or look at the protocol counters to see if the protocol packets are being sent and received.

Step 3 If the protocol packets are not being sent or received, check the interface counters to see if interface indicates packets in and out.

Step 4 If the interface level indicates that packets are coming in and out but not reaching protocol, check the ucode counters to see if there are any drops.

Layer 2 Traffic Not Flowing

VPLS

Step 1 Verify that the AC is up. RP/0/RSP0/CPU0:router# show l2vpn bridge-domain

Step 2 Verify that the bridge domain is up. RP/0/RSP0/CPU0:router# show l2vpn bridge-domain


OL-23591-02

Chapter 5 Troubleshooting Bundles and Load Balancing Troubleshooting Layer 2 Bundles and Load Balancing

Step 3 Look for MTU mismatches. RP/0/RSP0/CPU0:router# show l2vpn bridge-domain detail

VPWS

Step 1 View brief information on configured cross-connects. RP/0/RSP0/CPU0:router# show l2vpn xconnect summary

Step 2 RP/0/RSP0/CPU0:router# show l2vpn xconnect state

Step 3 RP/0/RSP0/CPU0:router# show controllers bundle bundle-ether bundle-id location node-id

Bundle StatisticsLayer 2 statistics are not supported in the show interface accounting command for bundle interfaces in the current release.

Troubleshooting Layer 2 Bundles and Load BalancingThis section describes how to troubleshoot Layer 2 bundles and load balancing. It includes the following topics:

• Verifying the Bundle Status, IGP Route, and CEF Database, page 5-122

• Viewing the Expected Paths and Measuring the Flows, page 5-123

Verifying the Bundle Status, IGP Route, and CEF DatabaseCEF uses the path information in the IP routing table to balance traffic over multiple links. For this reason, confirming proper CEF load balancing begins with confirming the contents of the IP routing table. When troubleshooting a bundle, verify that the bundle is up and that the IGP route to the desired destination includes the bundle interface.

SUMMARY STEPS


2. show bundle {Bundle-Ether | Bundle-POS} interface-path-id

3. show interface {Bundle-Ether | Bundle-POS} bundle-id

4. show arm router-id

5. show controllers bundle {Bundle-Ether | Bundle-POS} bundle-id location node-id


7. show cef ipv4 prefix


OL-23591-02


DETAILED STEPS

Viewing the Expected Paths and Measuring the FlowsCisco IOS XR provides a bundle utility that predicts how Layer 2 loads are balanced across member links. This is an interactive tool prompts for the information that the load balancing algorithm uses to allocate flows to member links.



Example:RP/0/RSP0/CPU0:router# show running-config

Displays the running configuration.

Verify that the configuration related to the bundles are correct.

Step 2 show bundle {Bundle-Ether | Bundle-POS} interface-path-id

Example:RP/0/RSP0/CPU0:router# show bundle bundle-ether 12

Displays the bundle status.

Verify that the bundle has the expected number of links. If not, troubleshoot the bundle first.

Step 3 show interface {Bundle-Ether | Bundle-POS} bundle-id

Example:RP/0/RSP0/CPU0:router# show interface bundle-ether 12

Displays the interface status.

Verify that the interface is assigned to the bundle.

Step 4 show arm router-id

Example:RP/0/RSP0/CPU0:router# show arm router-id

—

Step 5 show controllers bundle {Bundle-Ether | Bundle-POS} bundle-id location node-id

Example:RP/0/RSP0/CPU0:router# show controllers bundle bundle-ether 12 location 0/4/CPU0

—



Displays the routes to a destination address. Use a destination address on another host that is reachable through the bundle.

Verify that the route to the desalination address includes the bundle interface. If not, make sure that the bundle interface is included in the IGP process configuration.

Step 7 show cef ipv4 prefix

Example:RP/0/RSP0/CPU0:router# show cef ipv4 10.1.2.1

Displays the CEF forwarding table. Verify that it contains the same bundle interface that the routing table has for this subnet prefix.


OL-23591-02


SUMMARY STEPS

1. bundle-hash {Bundle-Ether | Bundle-Pos} interface-path-id

2. show interfaces [type interface-path-id | all | local | location node-id] [accounting | brief | detail | summary]

DETAILED STEPS

Troubleshooting Layer 3 Bundles and Load BalancingThis section provides commands for troubleshooting Layer 3 bundles and load balancing.

Step 1 RP/0/RSP0/CPU0:router# show arm router-id

Step 2 RP/0/RSP0/CPU0:router# show bundle

Step 3 RP/0/RSP0/CPU0:router# show interface bundle-ether bundle-id

Step 4 RP/0/RSP0/CPU0:router# show running-config

Step 5 RP/0/RSP0/CPU0:router# show arm router-ids

Step 6 Find out which member is carrying the traffic out. RP/0/RSP0/CPU0:router# bundle-hash bundle-ether bundle-id

Step 7 View each member of the bundle to see which member is actually carrying the traffic out. RP/0/RSP0/CPU0:router# show interface

Step 8 Display the exact route, including the egress interface for a specific source and destination IP. Use this command for several flows to verify that they are distributed equally over the parallel interfaces.

show cef [ipv4 | ipv6] exact-route source-address destination address [protocol type] [source-port source-port] [destination-port destination-port] [ingress-interface type interface-path-id] [policy-class value] [detail | location node-id]


Step 1 bundle-hash {Bundle-Ether | Bundle-Pos} interface-path-id

Example:RP/0/RSP0/CPU0:router# bundle-hash bundle-ether 12

Launches the bundle-hash utility. This is an interactive utility that prompts for the necessary information.

Step 2 show interfaces [type interface-path-id | all | local | location node-id] [accounting | brief | detail | summary]

Example:RP/0/RSP0/CPU0:router# show interfaces tenGigE 0/5/0/1

Displays interface information, which includes the traffic rates. Use this command for each link in the bundle to verify that the simulated traffic takes the expected link. Use clear counters to make it easier to view the traffic allocation.


OL-23591-02


C H A P T E R 6
Troubleshooting Layer 3 Connectivity
This section explains how to troubleshoot Layer 3 routing problems. If a ping to a remote site fails, the cause could be in an interface or in the Layer 3 routing. The overall approach for troubleshooting a failed ping should be to troubleshoot the interface failures and interface connectivity first, then proceed to troubleshooting Layer 3 routing if necessary.

Note For interface troubleshooting, perform the procedures listed in Chapter 2, “Verifying and Troubleshooting Interface Status” and Chapter 3, “Troubleshooting Interface Connectivity.”

This chapter contains the following topics:


• Traffic Loss, page 6-128

• Packets Are Punted and Switched in Software, page 6-129

• Traceroute Fails, page 6-130

• Adding Routes Fails, page 6-131

• Continuous Tracebacks, page 6-133

• fib_mgr Does Not Come Up During LC Reload or After Multiple Process Restarts, page 6-134

• CEF Entries Out of Sync, page 6-135

• fib_mgr Crashes, page 6-136

• Tracebacks Appearing, page 6-136

• Traffic Loss Because of Changing encap on a Subinterface, page 6-137

• Traffic Loss during RSP Failover, page 6-138

• Troubleshooting Virtual Router Redundancy Protocol, page 6-138

• Additional Information On Routing Configuration Commands, page 6-142

Using show and debug CommandsSUMMARY STEPS

1. show cef location node-id

2. show cef ipv4 {prefix/mask} location node-id


Chapter 6 Troubleshooting Layer 3 Connectivity Using show and debug Commands

3. show bgp summary

4. show bgp [{ipv4 | all} {unicast | multicast | all}] dampened-paths

5. show bgp flap-statistics [ip-address[/mask]]

6. show arp [vrf vrf-name] [ip-address [location node-id] | hardware-address [location node-id] | traffic [location node-id | interface-name]

7. show interface accounting [location]

8. show cef ipv4 [prefix/mask] hardware [ingress | egress] location node-id

9. show cef platform trace common [all | errors | events | info] [location node-id]

10. show cef vrf [vrfname] [prefix]

DETAILED STEPS


Step 1 show cef location node-id

Example:RP/0/RSP0/CPU0:router# show cef location 0/2/CPU0

View all IPv4 routes of Cisco Express Forwarding (CEF) on an LC).

Note Use this when there are only a few routes.

Step 2 show cef ipv4 {prefix/mask} location node-id

Example:RP/0/RSP0/CPU0:router# show cef ipv4 192.168.1.1/32 location 0/2/CPU0

View a prefix’s route on an LC.

Step 3 show bgp summary View Border Gateway Protocol (BGP) neighbors without an inbound and outbound policy for each active address family.

Note Use this when there are many routes.

Step 4 show bgp [{ipv4 | all} {unicast | multicast | all}] dampened-paths

Example:RP/0/RSP0/CPU0:router# show bgp dampened-paths

View which routes have dampening enabled.

Step 5 show bgp flap-statistics [ip-address[/mask]]

Example:RP/0/RSP0/CPU0:router# show bgp flap-statistics

View BGP flap statistics.

Note Use this for routes that have had dampening enabled.

If you do not specify arguments or keywords, all routes for the address family are displayed.

If you enter an IP address without mask or prefix length, the longest matching prefix is displayed.


OL-23591-02

Chapter 6 Troubleshooting Layer 3 Connectivity Using show and debug Commands

Example

RP/0/RSP0/CPU0:router# show cef vrf vrf1 192.168.1.2 hardware egress location 0/1/CPU0192.168.1.2/32, version 0, internal 0x40800001 (ptr 0xaac1c468) [1], 0x0 (0xaab8c7b0), 0x0 (0x0) Updated Oct 1 21:29:37.684 local adjacency 130.130.1.2 Prefix Len 32, traffic index 0, Adjacency-prefix, precedence routine (0) via 130.130.1.2, GigabitEthernet0/1/0/0, 3 dependencies, weight 0, class 0 [flags 0x0] path-idx 0 next hop 130.130.1.2 local adjacency

TBM Node Data:Node (0x00000100):0 0x8d40047d 0x00000000 0xffffffff 0xffffffff

Step 6 show arp [vrf vrf-name] [ip-address [location node-id] | hardware-address [location node-id] | traffic] [location node-id | interface-name]

Example:RP/0/RSP0/CPU0:router# show arp

View Address Resolution Protocol (ARP) records.

For bundle and VLAN-on-Bundle interfaces, enter location node-id. This tells the system which cache entries to show.

Note If vrf is entered, it must appear immediately after show arp, and you must enter a vrf-name.

Step 7 show interface accounting [location]

Example:RP/0/RSP0/CPU0:router# show interface accounting location 0/4/CPU0

View packet accounting on an interface per protocol.

Step 8 show cef ipv4 {prefix/mask} hardware {ingress | egress} location node-id

Example:RP/0/RSP0/CPU0:router# show cef ipv4 38.1.1.2/32 hardware egress location 0/4/CPU0

View IPv4 prefix/route in the hardware of an LC.

This information helps determine if the destination IP or prefix action is COMPLETE, PUNT or DROP.

Step 9 show cef platform trace common [all | errors | events | info] [location node-id]

Example:RP/0/RSP0/CPU0:router# show cef platform trace common all errors location 0/4/CPU0

View common Dynamic Link Library (DLL) code traces.

Step 10 show cef vrf [vrfname] [prefix]

Example:RP/0/RSP0/CPU0:router# show cef vrf 0xx

RP/0/RSP0/CPU0:router# show cef vrf vrf1 192.168.1.2 hardware egress location 0/1/CPU0

Verify that the L3 MTU value, encapsulation string value, byte count, and packet count are as expected. (See the example below.)



OL-23591-02

Chapter 6 Troubleshooting Layer 3 Connectivity Traffic Loss

Node (0x8d400470):1 0x8cb928bd 0x00000005 0x00008083 0x80038003 Node (0x8cb92900):2 0x8d81511d 0x89885f70 0x00000000 0x00800000 Node (0x8d815110):3 0x8d8151ed 0x00000000 0x00800000 0x00000000 Node (0x8d8151e0):4 0x8d814ead 0x00000000 0x40000000 0x00000000 Node (0x8d814ea0):5 0x8d8cd40d 0x00000004 0x80000000 0x00000001 Node (0x8d8cd410):6 0x8d8dc0a5 0x8d814d70 0x0e000000 0x00000000 Node (0x8d8dc0c0):7 0x00002000 0x00000000 0x8d8dc0d0 0x8d8274d0

Hardware Leaf Data (0x8d8dc0c0):0x00002000 0x00000000 0x8d8dc0d0 0x8d8274d0

IP Leaf Data: as:0 prefix_len:32 for_us:0x0 dft_route:0x0 real_intf:0x0 free1: 0x0 hw_use_only: 0x0 lspa_ptr: 0x0 oce_chain_p: 0x8d8274d0 extre_fib_data_ptr: 0x8d8dc0d0

Hardware Extended Leaf Data: fib_leaf_extension_length: 0 interface_receive: 0x0 traffic_index_valid: 0x0 qos_prec_valid: 0x0 qos_group_valid: 0x0 valid_source: 0x0 traffic_index: 0x0 nat_addr: 0x0 reserved: 0x0 qos_precedence: 0x0 qos_group: 0x0 peer_as_number: 0 path_list_ptr: 0x0 connected_intf_id: 0x0 ipsub_session_uidb: 0xffffffff Path_list: urpf loose flag: 0x0 List of interfaces:

OCE Loadbalance Data for ptr 0x8d8274d0: num_entries:1 level:0x1 pad_1:0x0 l3_lbe_ptr:0x8d8274e0

LBE Array for 0x8d8274e0 Entry 0: oce_chain_p 0x8d8274c0 Entry 0: bgp_ipv4_next_hop_addr: 0x0

OCE Adj Data for 0x8d8274c0:adj:0x50717cc0 base: 6 (CPP HW IPv4 Adjacency Object) encap_length: 14l3_mtu: 1500 adj_flags: 0000fixup_flags: 0000output_uidb: 0x1fa0 adj2:0x50588f00encap: 00008282010200211bfcc2400800nh_addr: 0x00 0x00 0x00 0x00 oce_chain_p: 0x00000000counters: 0x893d46f0byte count: 4447644 packet count: 71732

Traffic LossThis section provides steps for troubleshooting traffic loss.

Step 1 Check for packet loss by examining transmitted packets on the local router and the receive packets on the destination router.


OL-23591-02

Chapter 6 Troubleshooting Layer 3 Connectivity Packets Are Punted and Switched in Software

RP/0/RSP0/CPU0:router# show interface accounting Wed Dec 8 13:12:47.627 PSTNo accounting statistics available for Bundle-Ether16.10No accounting statistics available for GigabitEthernet0/1/0/7.210MgmtEth0/RSP0/CPU0/0 Protocol Pkts In Chars In Pkts Out Chars Out IPV4_UNICAST 2225064 168207595 67521 3479370 ARP 29433 1765984 5855 245910

Step 2 View the hardware data structures involved with the prefix (destination-ip)/(mask). Verify that the RIB table is consistent with the information that the IGP learned from neighbors. that the CEF tables are consistent with the RIB. For routes that are learned (not directly connected), the CEF table in the RSP should be the same as the CEF table in the LC.

show cef {ipv4} [destination ip | destination-ip/mask] hardware egress detail location node-id

Step 3 View the ARP information on the particular LC or RSP.

show arp location node-id

Step 4 View any PI code ltrace errors recorded.

RP/0/RSP0/CPU0:router# debug controllers pse qfp forward drop {0 | 1} location 0/2/CPU0

Packets Are Punted and Switched in Software

Step 1 Verify that the hardware chains for the destination IP address are pointing to either of the following:

• COMPLETE adjacency—Valid outgoing path exists.

• PUNT adjacency—Hardware does not know how to send the packet out, it just punts (diverts) the packet to be switched in software. If the transmit adjacency is PUNT, this could be because ARP is not resolved yet.

Step 2 To show if an ARP entry exists for the destination IP, use the show arp location command: RP/0/RSP0/CPU0:router# show arp location node-id

a. If an ARP entry does not exist or is incomplete, add a static ARP entry. Ensure that the Tx adjacency points to ‘COMPLETE’. RP/0/RSP0/CPU0:router# show cef {ipv4} 192.168.1.1/32 hardware egress detail location

0/4/CPU0

b. If so, then it means the issue is that of ARP entry not getting updated. Troubleshooting should now focus on why the ARP entry is not getting added (this includes steps like show arp, show arp idb, show adjacency gig node-id detail location node-id, show arp trace, and so forth).

c. If the Tx adjacency still points to ‘PUNT’, it means ARP is adding the entry in its database, but fib_mgr fails to mark the adjacency as ‘COMPLETE’.

d. This could be a fib_mgr, ARP, or AIB problem. Delete and reconfigure the static ARP entry with AIB and CEF debugs on. The debugs show if ARP is adding the entry inside the AIB and if the AIB is informing fib_mgr.


OL-23591-02

Chapter 6 Troubleshooting Layer 3 Connectivity Traceroute Fails

Step 3 Packets could be dropped in the fabric. To verify this, view the fabric counters.

Workaround

Step 1 Use the shut command (followed by commit) and the no shut command (followed by commit) on the outgoing interface.

Step 2 Add a static ARP entry for the destination IP.

Traceroute FailsUse traceroute to verify the connectivity to a destination. When traceroute fails to a destination, use the following commands:

• show cef {ipv4} {destination_ip}/(mask} hardware egress detail location node-id—View the hardware data structures involved with the prefix.

• show interface location {outgoing_interface} accounting—View input and output packets from the outgoing interface.

Step 1 Check if the destination IP address has the proper transmit adjacency. See the ‘Tx Adjacency’ state (it should be ‘COMPLETE’). RP/0/RSP0/CPU0:router# show cef {ipv4} prefix hardware egress detail location node-id

Step 2 If the transmit adjacency is not complete, there is an issue. If it is pointing to ‘PUNT’, that means probably the mac-address corresponding to the destination IP has not been learned. Try adding a ‘static arp’ entry and see if transmit adjacency moves to ‘COMPLETE’. If the destination IP is advertised by a routing protocol such as OSPF, then the transmit adjacency should never show as ‘PUNT’. If the transmit adjacency is shown as ‘DROP’, that means there is a static route to the destination IP explicitly pointing the route to a DROP. If the transmit adjacency is shown as ‘COMPLETE’, it means there is no problem in the hardware chains that are set up. You should see the counters.

Step 3 See if the output packets are equal to the traceroute packets sent. RP/0/RSP0/CPU0:router# show interface location outgoing_interface accounting

Workaround

Step 1 Use the shut command (followed by commit) and the no shut command (followed by commit) on the outgoing interface.

Step 2 Add a static ARP entry for the destination IP.


OL-23591-02

Chapter 6 Troubleshooting Layer 3 Connectivity Adding Routes Fails

Adding Routes FailsPerform the steps in this section to troubleshoot failures in adding routes. During Out Of Resource (OOR), the router does not accept additional routes until existing routes are deleted.

Note The sample commands in this section are applicable to Ethernet LCs, not SIP-700 LCs.

Step 1 Determine if any resources are experiencing problems. View the state of various data structures. Ideally the state should be GREEN. If it is either YELLOW or RED, it indicates an OOR condition.

RP/0/RSP0/CPU0:router# show cef resource location node-id

RP/0/RSP0/CPU0:ASR-9010#show cef resource location 0/0/CPU0Thu Oct 28 09:07:52.405 DSTCEF resource availability summary state: GREENCEF will work normally ipv4 shared memory resource: GREEN ipv6 shared memory resource: GREEN mpls shared memory resource: GREEN common shared memory resource: GREEN DATA_TYPE_TABLE_SET hardware resource: GREEN DATA_TYPE_TABLE hardware resource: GREEN DATA_TYPE_IDB hardware resource: GREEN DATA_TYPE_IDB_EXT hardware resource: GREEN DATA_TYPE_LEAF hardware resource: GREEN DATA_TYPE_LOADINFO hardware resource: GREEN DATA_TYPE_PATH_LIST hardware resource: GREEN DATA_TYPE_NHINFO hardware resource: GREEN DATA_TYPE_LABEL_INFO hardware resource: GREEN DATA_TYPE_FRR_NHINFO hardware resource: GREEN DATA_TYPE_ECD hardware resource: GREEN DATA_TYPE_RECURSIVE_NH hardware resource: GREEN DATA_TYPE_TUNNEL_ENDPOINT hardware resource: GREEN DATA_TYPE_LOCAL_TUNNEL_INTF hardware resource: GREEN DATA_TYPE_ECD_TRACKER hardware resource: GREEN DATA_TYPE_ECD_V2 hardware resource: GREEN DATA_TYPE_ATTRIBUTE hardware resource: GREEN DATA_TYPE_LSPA hardware resource: GREEN DATA_TYPE_LDI_LW hardware resource: GREEN DATA_TYPE_LDSH_ARRAY hardware resource: GREEN DATA_TYPE_TE_TUN_INFO hardware resource: GREEN DATA_TYPE_DUMMY hardware resource: GREEN DATA_TYPE_IDB_VRF_LCL_CEF hardware resource: GREEN DATA_TYPE_TABLE_UNRESOLVED hardware resource: GREEN DATA_TYPE_MOL hardware resource: GREEN DATA_TYPE_MPI hardware resource: GREEN DATA_TYPE_SUBS_INFO hardware resource: GREEN DATA_TYPE_GRE_TUNNEL_INFO hardware resource: GREEN

Step 2 Determine which hardware table is OOR. Compare ‘max entries’ and ‘used entries’ too see which of the data structures is using the entries close to the max limit.

RP/0/RSP0/CPU0:router# show cef platform resource location node-id

RP/0/RSP0/CPU0:router# show cef platform resource loc 0/0/CPU0 Thu Oct 28 15:41:47.725 PST Node: 0/0/CPU0----------------------------------------------------------------IPV4_LEAF_P usage is same on all NPs


OL-23591-02

Chapter 6 Troubleshooting Layer 3 Connectivity Adding Routes Fails

NP: 0 struct 23: IPV4_LEAF_P (maps to ucode stru = 54 in TopSearch1)Used Entries: 298 Max Entries: 524288 -------------------------------------------------------------IPV6_LEAF_P usage is same on all NPsNP: 0 struct 24: IPV6_LEAF_P (maps to ucode stru = 55 in TopSearch1)Used Entries: 4 Max Entries: 131072 -------------------------------------------------------------R_LDI usage is same on all NPsNP: 0 struct 6: R_LDI (maps to ucode stru = 11 in TopSearch1)Used Entries: 8 Max Entries: 65536 -------------------------------------------------------------NR_LDI usage is same on all NPsNP: 0 struct 7: NR_LDI (maps to ucode stru = 12 in TopSearch1)Used Entries: 31 Max Entries: 524288 -------------------------------------------------------------RPF_STRICT usage is same on all NPsNP: 0 struct 9: RPF_STRICT (maps to ucode stru = 15 in TopSearch1)Used Entries: 0 Max Entries: 65536 -------------------------------------------------------------NP: 0 struct 12: TX_ADJ (maps to ucode stru = 18 in TopSearch1)Used Entries: 21 Max Entries: 131072 -------------------------------------------------------------NP: 1 struct 12: TX_ADJ (maps to ucode stru = 18 in TopSearch1)Used Entries: 21 Max Entries: 131072 -------------------------------------------------------------NP: 2 struct 12: TX_ADJ (maps to ucode stru = 18 in TopSearch1)Used Entries: 18 Max Entries: 131072 -------------------------------------------------------------NP: 3 struct 12: TX_ADJ (maps to ucode stru = 18 in TopSearch1)Used Entries: 18 Max Entries: 131072 -------------------------------------------------------------RX_ADJ usage is same on all NPsNP: 0 struct 13: RX_ADJ (maps to ucode stru = 19 in TopSearch1)Used Entries: 28 Max Entries: 32768 -------------------------------------------------------------TE_NH_ADJ usage is same on all NPsNP: 0 struct 14: TE_NH_ADJ (maps to ucode stru = 20 in TopSearch1)Used Entries: 6 Max Entries: 32768 -------------------------------------------------------------L2VPN_LDI usage is same on all NPsNP: 0 struct 16: L2VPN_LDI (maps to ucode stru = 13 in TopSearch1)Used Entries: 0 Max Entries: 32768 -------------------------------------------------------------LABEL_UFIB usage is same on all NPsNP: 0 struct 28: LABEL_UFIB (maps to ucode stru = 1 in TopParse)Used Entries: 4 Max Entries: 290000 -------------------------------------------------------------

Step 3 After determining which data structure is OOR, verify if it is expected or unexpected. Usually, for each LEAF (either IPv4), it requires four entries of NR_LDI structure. So if you find the NR_LDI structure going OOR, see if you have appropriate number of IP LEAFs to take this NR_LDI number to such a limit.

Step 4 If show cef resource location node-id shows the state in GREEN, it means that the problem is not caused by an OOR condition. The reason for not being able to add further routes is some thing else. Enable the following debugs to observe what is happening:

• RP/0/RSP0/CPU0:router# debug cef errors location node-id

• RP/0/RSP0/CPU0:router# debug cef {ipv4} error location node-id

• If you observe any tracebacks, decode the tracebacks by using SBT tool.


OL-23591-02

Chapter 6 Troubleshooting Layer 3 Connectivity Continuous Tracebacks

Step 5 View platform ltrace errors for protocols IPv4—show cef platform trace {ipv4} error reverse

location node-id.

Step 6 View platform ltrace common errors for all protocols—show cef platform trace common error

reverse location node-id.

Workaround

If it is an OOR condition and expected, delete some existing routes.

Continuous Tracebacks When tracebacks appear continuously on the console (typically every 15 seconds), programming of the entry inside the hardware is not successful. This causes the software to try repeatedly after every 15 seconds. It is possible that the layer just above the hardware or the hardware itself is not up and running.

Step 1 View all platform ltrace common messages. Verify that both CPPs are in ACTIVE_SOLO state.

Note Step 1 is applicable to SIP-700 line cards only.

show controllers pse qfp system state location node-id

Example

show controllers pse qfp system state location 0/1/CPU0

CPP HA client processes registered (5 of 5) cpp_sp : Initialized cpp_cdm : Initialized cpp_driver1 : Initialized cpp_driver0 : Initialized cpp_cp : Initialized------------------------------------------CPP 0: dir=INGRESS Role: curr=ACTIVE_SOLO next=ACTIVE_SOLO <<< CPP 0 in ACTIVE_SOLO stateClient State: ENABLEImage: /pkg/ucode/cpp/cpp-thor-ucodeImage desc: Ucode dir: /nobackup/eruan/thor2/cpp/dp/obj/thor/thor-ingress-hw Image: thor_ingress HW: CPP10 Built by: eruan Host: sjc-lds-447 Time: Tue Sep 28 15:04:57 2010 Component: cpp/dp asr41-9k-cgn/2 Load Cnt: 1 Last load: Oct 01, 2010 21:27:36.488431Active Threads: 0-159Stuck Threads: <NONE>Fault Manager Flags: ignore_fault: FALSE ignore_stuck_thread: FALSE crashdump_in_progress: FALSE------------------------------------------


OL-23591-02

Chapter 6 Troubleshooting Layer 3 Connectivity fib_mgr Does Not Come Up During LC Reload or After Multiple Process Restarts

CPP 1: dir=EGRESS Role: curr=ACTIVE_SOLO next=ACTIVE_SOLO <<< CPP 1 in ACTIVE_SOLO stateClient State: ENABLEImage: /pkg/ucode/cpp/cpp-thor-ucodeImage desc: Ucode dir: /nobackup/eruan/thor2/cpp/dp/obj/thor/thor-egress-hw Image: thor_egress HW: CPP10 Built by: eruan Host: sjc-lds-447 Time: Tue Sep 28 14:55:59 2010 Component: cpp/dp asr41-9k-cgn/2 Load Cnt: 1 Last load: Oct 01, 2010 21:27:36.500431Active Threads: 0-159Stuck Threads: <NONE>Fault Manager Flags: ignore_fault: FALSE ignore_stuck_thread: FALSEcrashdump_in_progress: FALSE

Step 2 View all platform ltrace protocol messages for IPv4 or IPv6. show cef platform trace {ipv4 | ipv6} all reverse location node-id

Step 3 Check that the NP provisioning layer (or PRM) is up. PRM is a layer just above hardware. If PRM is down, no entry is programmed in hardware, indicating that NP may have had a problem during initialization. show controllers NP summary

Step 4 View the NP driver logs to find out if there have been NP initialization errors. If there are NP initialization errors, it is likely an NP problem. show controllers NP drvlog location node-id

Step 5 Use the SBT tool to decode the tracebacks. From root of the workspace, use ./util/bin/sbt -p (process_name) -f (log_file).

Workaround

Step 1 Restart prm_server process.

Step 2 Reboot LC.

fib_mgr Does Not Come Up During LC Reload or After Multiple Process Restarts

Fib_mgr depends on underlying hardware. If the underlying process or hardware does not come up, it is likely that fib_mgr will not come up.

• show controllers NP summary location node-id—Check that the NP provisioning layer (or PRM) is up.

• show controllers NP drvlog location node-id—View the NP driver logs.


OL-23591-02

Chapter 6 Troubleshooting Layer 3 Connectivity CEF Entries Out of Sync

• show cef platform trace common all reverse location node-id—View platform ltrace common messages.

• show cef platform trace common event reverse location node-id—View platform ltrace common events.

• show cef platform trace {ipv4 | ipv6 | mpls} error reverse location node-id—View platform ltrace error messages recorded for protocols IPv4, IPv6, or MPLS.

• show cef trace all reverse location node-id—View all CEF ltrace messages.

Step 1 Use the show controllers NP summary location and show controllers NP drvlog location commands to determine if either the PRM or the underlying NP has a problem. If so, the fib_mgr will not come up. Troubleshoot at the PRM layer or NP layer.

Step 2 If both CPPs are in ACTIVE_SOLO state, the problem is likely a software bug. In this case, collect the core file and decode the tracebacks using the SBT tool. From root of the workspace, use ./util/bin/sbt -p (process_name) -f (log_file).

Workaround

Step 1 Restart the prm_server process.

Step 2 Reboot the LC.

CEF Entries Out of SyncThe cef entry on RSP may be pointing to the management interface and as a result the traffic originating from the router may go out on the management interface instead of through the LC interface.

• show controllers np drvlog location node-id—Shows the PRM view of the Direct Table on the NP.

• show tech-support cef—Collects relevant platform independent traces.

• show cef trace events reverse location node-id—View platform independent cef ltrace events.

• show cef trace errors reverse location node-id—View platform independent cef ltrace errors.

• show cef platform trace common event reverse location node-id—View CEF platform common event traces.

• show cef platform trace common error reverse location node-id—View CEF platform common error traces.

Step 1 Look for a default route 0.0.0.0/0 configured to go out through the management interface.

Step 2 Look for a static ARP configured for the prefix in question. It is possible that ARP is installing two entries through both the management interface and also through the LC interface (because the prefix is reachable by both routes).


OL-23591-02

Chapter 6 Troubleshooting Layer 3 Connectivity fib_mgr Crashes

Step 3 If the above is not the case, use the show arp command to see if an ARP entry is advertising through the management interface. If this is the case, clear the ARP and verify the cef entries again.

Workaround

• Use the shut command (followed by commit) and the no shut command (followed by commit) on the management interface.

• Use the clear arp-cache command.

• Reboot the LC.

fib_mgr Crashes • show cef platform trace common all reverse location node-id—View CEF platform common

traces.

• show cef platform trace common event reverse location node-id—View CEF platform common event traces.

• show cef platform trace common error reverse location node-id—View CEF platform common error traces.

• show cef platform trace {ipv4 | ipv6 | mpls} error reverse location node-id—View CEF platform protocol traces for IPv4 or MPLS.

Step 1 If the trigger is a prm restart or crash, this is expected.

Step 2 If the underlying process (prm_server) is down or crashed, it is likely fib_mgr will not come up.

Step 3 Save the core file.

Step 4 Use the SBT to decode the tracebacks. From root of the workspace, use ./util/bin/sbt -p (process_name) -f (log_file).

Step 5 Save the console logs.

Workaround

Restart fib_mgr or reboot the LC.

Tracebacks AppearingIn this scenario, a few error tracebacks appear on the console because of some trigger (such as interface shut/no shut, or any other similar trigger).

• show cef trace event location node-id—View CEF traces for major events.

• show cef trace errors location node-id—View CEF traces for major errors.

• show cef platform trace common errors location node-id—View CEF platform traces for common errors across all protocols.


OL-23591-02

Chapter 6 Troubleshooting Layer 3 Connectivity Traffic Loss Because of Changing encap on a Subinterface

• show cef platform trace {ipv4 | ipv6 | mpls} errors location node-id—View CEF platform traces for errors in protocols IPv4 or MPLS.

• show logging

Step 1 Decode the tracebacks using the SBT tool. From root of the workspace, use ./util/bin/sbt -p (process_name_ -f (log_file).

Step 2 Save core files.

Workaround

If the tracebacks are impacting service, do the following:

Step 1 Restart the fib_mgr process and check whether that reduces the tracebacks.

Step 2 If the tracebacks continue, reboot the LC.

Traffic Loss Because of Changing encap on a SubinterfaceWhen traffic is being forwarded through a Layer 3 subinterface and if the encapsulation is changed on that subinterface, it is sometimes observed that the traffic does not resume until after 15 seconds.

• show cef trace event reverse location node-id—View CEF trace messages for major events.

• show cef trace error reverse location node-id—View CEF trace messages for major errors.

• show cef platform trace common error location node-id—View CEF platform traces for common errors across all protocols.

• show cef platform trace {ipv4 | ipv6 | mpls} event location node-id—View CEF platform traces for major events in protocols IPv4, IPv6, or MPLS.

• show cef platform trace {ipv4 | ipv6 | mpls} error location node-id—View CEF platform traces for major errors in protocols IPv4, IPv6, or MPLS.

• show arp trace location node-id—Shows arp related traces.

• show arp-gmp trace location node-id—Shows arp-gmp related traces.

• show arp location node-id—View ARP-related information.

This type of traffic loss could happen typically when there is a static arp entry for the prefix which is experiencing traffic loss. For example, consider the following configuration:

interface GigabitEthernet0/4/0/39.2 ipv4 address 209.165.201.1 255.0.0.0 dot1q vlan 300

When encapsulation changes from dot1q vlan 300 to dot1q vlan 200 on the subinterface, fib_mgr deletes all prefixes corresponding to this interface and creates them again. It takes 15 seconds to add all prefixes; traffic does not get forwarded for that time. For example, there is an interface with address 192.0.2.0/8. There is a static ARP entry for 192.0.2.5.

RP/0/RSP0/CPU0:router# show run | inc arp


OL-23591-02

Chapter 6 Troubleshooting Layer 3 Connectivity Traffic Loss during RSP Failover

The delay is less likely to happen with regular adjacency (not the static ARP).

When VLAN color changes, the following occurs:

• Adjacency is deleted, and the adjacency route 192.0.2.5 is deleted.

• Connected route is deleted.

• Adjacency is added before the connected route is added. The FIB treats adding an adjacency without a covering connected route as an error, so the route 192.0.2.5 is placed in retry.

• Connected route 192.0.2.0/8 is added.

• Because the FIB retry timer is 15 seconds, the adjacency route 192.0.2.5 is added after 15 seconds.

Workaround

Remove the static ARP entry.

Traffic Loss during RSP FailoverSometimes RSP switchover (keyword failover in CLI) causes traffic loss. This may mean the IGP over which the prefixes are learned is going down. The following assumes OSPF as the IGP.

• show process failover—Shows process details during failover.

• debug ospf ha—Enables OSPF HA related debugs.

• debug ospf instance nsf—View before failover and collect the debug log.

• show process failover—Shows process details after failover.

• show redundancy—Provides status of the standby node after failover.

Check if the next hop router had a failover.

• If so, the OSPF will go down.

• If not, verify that nsf cisco is configured under OSPF.

– If nsf cisco is configured, see if the next hop is reachable during failover.

– If the next hop is not reachable, a link may be going down or having negotiation problems.

– If the next hop is reachable, the problem is likely a software bug.

Workaround

Reload the router.

Troubleshooting Virtual Router Redundancy ProtocolVirtual Router Redundancy Protocol (VRRP) enables a group of routers to form a single virtual router. This section contains the following subsections:


• VRRP Fails to Reach Active State, page 6-140

• Tracked Interface Failing, Router State Not Changed, page 6-140

• VRRP State Flapping, page 6-140


OL-23591-02

Chapter 6 Troubleshooting Layer 3 Connectivity Troubleshooting Virtual Router Redundancy Protocol

• More Than One VRRP Router Active, page 6-141

• VRRP Active Router Not Forwarding Traffic, page 6-141

• Traffic Loss or Unexpected VRRP State After Interface shut/no shut, page 6-142


SUMMARY STEPS

1. show vrrp [interface type interface-id] [brief]

2. show vrrp interface type interface-id detail

3. show vrrp [interface {type interface-id}] statistics [all]

4. show controllers type interface-id

5. debug vrrp [ all | edm | events | packets ]

DETAILED STEPS


Step 1 show vrrp [interface type interface-id] [brief]

Example:RP/0/0/CPU0:# show vrrp brief

View all VRRP groups status.

Step 2 show vrrp interface type interface-id detail

Example:RP/0/0/CPU0:# show vrrp gigabitEthernet 0/1/0/1 detail

View detailed information of VRRP groups.

Step 3 show vrrp [interface {type interface-id}] statistics [all]

Example:RP/0/0/CPU0:# show vrrp statistics

View VRRP statistics.

Step 4 show controllers type interface-id

Example:RP/0/0/CPU0:# show controllers gigabitEthernet 0/3/0/9

View the VRRP group MAC addresses as part of unicast filter list.

Step 5 debug vrrp [ all | edm | events | packets | packets ]

Example:RP/0/0/CPU0:# debug vrrp packets tengige 0/3/0/9

Debug the VRRP.


OL-23591-02


VRRP Fails to Reach Active StateRun the following command on both routers:

RP/0/RSP0/CPU0:router# show vrrp detail

Misconfiguration

Step 1 Ensure that the interface with VRRP configured is up.

Step 2 Ensure that an IP address is configured, on the same subnet as the interface, and delay is configured. RP/0/RSP0/CPU0:router# show vrrp detail

Higher Priority Router Already Active

Examine the output of the show vrrp command:

• If the Master address for VRRP shows an IP address instead of local, the router with that IP address is Active.

• If preemption is enabled, but the other router has higher priority, then it will remain in the Active state.

Operational priority may not match the configured priority. If interfaces are down, this negatively impacts operational priority.

Preemption is Disabled and Another Router Already Active

Examine the output of the show vrrp command. If preemption is disabled, and the router has higher priority, it will not take over unless preemption is enabled.

Tracked Interface Failing, Router State Not ChangedOn both routers:

RP/0/RSP0/CPU0:router# show vrrp detail If preemption is enabled and this router has higher operational priority than the other router, this router remains in the Active state. Configured priority or the decrement for tracked interfaces needs to be configured appropriately such that the state transition takes place. If the IP address is the same as the interface IP address, the router does not change to the Standby state.

VRRP State FlappingOn both routers:

Step 1 RP/0/RSP0/CPU0:router# show vrrp detail


OL-23591-02


Step 2 RP/0/RSP0/CPU0:router# debug vrrp packets Check timestamps to determine whether there is a delay in sending or receiving packets. Check the CPU usage to see if some process is hogging the system resources.

Step 3 RP/0/RSP0/CPU0:router# show spp node-counters location interface-running-vrrp

More Than One VRRP Router Active

Step 1 Verify that the same IP is configured on both ends. RP/0/RSP0/CPU0:router# show vrrp detail

Step 2 Check timestamps to determine whether there is a delay in sending or receiving packets. Check the CPU usage to see if a process is overusing resources.

Step 3 Enter the debug command for VRRP packets on the peer. RP/0/RSP0/CPU0:router# debug vrrp packets Check for lines similar to: RP/0/RSP0/CPU0:Sep 8 14:16:39.217 : vrrp[357]: Gi0/5/0/0: VR1: Pkt: ADVER: IN: pri 100 src 192.0.2.11. This means advertisement packets are being received by VRRP. If these are absent, no packets are being received and VRRP becomes active. Look for lines similar to: RP/0/RSP0/CPU0:Sep 8 14:18:47.876 : vrrp[357]: Gi0/5/0/0: VR1: Pkt: ADVER: Out: pri 100 src 192.0.2.11. This means the peer is sending VRRP packets.

Step 4 Check the output of the show spp node-counters location interface-running-vrrp on both routers, and look for packet drops. RP/0/RSP0/CPU0:router# show spp node-counters location interface-running-vrrp

VRRP Active Router Not Forwarding TrafficOn both routers:

Step 1 Find the virtual MAC address for the group. RP/0/RSP0/CPU0:router# show vrrp detail

Step 2 RP/0/RSP0/CPU0:router# show ether-ctrl trace

Step 3 Ensure that the virtual MAC address is in the unicast address filter list and verify the router is receiving traffic. RP/0/RSP0/CPU0:router# show controllers type interface-running-vrrp


OL-23591-02

Chapter 6 Troubleshooting Layer 3 Connectivity Additional Information On Routing Configuration Commands

Traffic Loss or Unexpected VRRP State After Interface shut/no shutIn case of shut/no shut on a VRRP-enabled interface, the following has been observed:

• If preemption is enabled, recovery times are higher than switchover times. This means higher traffic loss has occurred when the interface is no shut.

• If preemption is disabled, some VRRP groups are preempted after no shut of an interface.

If you observe either of the above conditions after an interface no shut, perform the following steps on both routers.

Step 1 RP/0/RSP0/CPU0:router# show vrrp detail

Step 2 RP/0/RSP0/CPU0:router# show ether-ctrl trace

Step 3 RP/0/RSP0/CPU0:router# show controllers type interface-running-n

Step 4 RP/0/RSP0/CPU0:router# debug vrrp packets interface—For the interface on which no shut is being performed.

Step 5 Enter the no shut command.

Step 6 Observe the console logs and look for lines similar to: RP/0/RSP0/CPU0:Sep 8 14:16:39.217 : vrrp[357]: Gi0/5/0/0: VR1: Pkt: ADVER: IN: pri 100

src 192.0.2.11. Note the time lag between the no shut and the first such message seen. For that amount of time, there is traffic loss between two routers.

Step 7 If there is no traffic flowing between two routers after a no shut event, check the STP configuration on the Cisco ASR 9000 Series Router. Lowering the fwd delay timer might help in reducing the traffic loss.

Step 8 For preemption disabled case, if the groups still preempt after reducing the fwd delay timer, repeat Step 1 through Step 4, and find the time period of traffic loss between the two routers. The preemption can be avoided by configuring the minimum delay to be higher than the time period of traffic loss. Minimum delay can be configured as follows:

RP/0/RSP0/CPU0:router(config)# router vrrp interface gigabitEthernet 0/2/0/10 vrrp delay minimum 10 reload 5

Additional Information On Routing Configuration CommandsUse the following guides if you need to review routing configuration commands

• Cisco ASR 9000 Series Aggregation Services Router Routing Configuration Guide, Release 4.0

• Cisco ASR 9000 Series Aggregation Services Router Routing Command Reference, Release 4.0


OL-23591-02

http://www.cisco.com/en/US/docs/routers/asr9000/software/asr9k_r4.0/routing/configuration/guide/b_rc40asr9k.html

http://www.cisco.com/en/US/docs/routers/asr9000/software/asr9k_r4.0/routing/command/reference/b_rr40asr9kbook.html


C H A P T E R 7
Troubleshooting Router Switch Fabric and Data Path
This chapter describes techniques to troubleshoot router switch fabric and data path. It includes the following sections:

• Understanding Switch Fabric Architecture, page 7-143

• Getting Started with Fabric Troubleshooting, page 7-145

• Troubleshooting Packet Drops, page 7-146

• Troubleshooting RSP and LC Crashes, page 7-165

• Troubleshooting Complete Loss of Traffic, page 7-168

• Gathering Fabric Information Before Calling TAC, page 7-172

Understanding Switch Fabric ArchitectureFigure 7-1 provides an overview of the switch fabric architecture.


Chapter 7 Troubleshooting Router Switch Fabric and Data Path Understanding Switch Fabric Architecture

Figure 7-1 Switch Fabric Architecture

As shown in Figure 7-1, there are two fabric interface ASIC on each RSP. Each fabric interface ASIC provides 40 GB of throughput. If one RSP is lost, the shelf can still operate at full capacity without loss of bandwidth.

Each line card (LC) has four 23 GB fabric channels on which to send traffic to the fabric ASICs. The switch fabric is in an active/active relationship. All four fabric ASICs are active, even though the RSP cards are in an active/standby relationship. The system performs load balancing on unicast traffic across these four channels.

The arbiters are in an active/standby relationship (the arbiter on the active RSP card is the active arbiter). Both the active and standby arbiters receive requests for switch fabric access from the LCs. If there is a switchover of the active RSP, the standby RSP arbiter has a current copy of switch fabric requests, which helps to speed up the switchover.

Active Fabric

SwitchFabric 0

SwitchFabric 1

Arbiter

RSP0

Active RP

Active Fabric

SwitchFabric 0

SwitchFabric 1

Arbiter

RSP1

Standby RP 2813

42

Fabric I/O(LC)

Fabric I/O(LC)

23G fabric channelsFabric requests


OL-23591-02

Chapter 7 Troubleshooting Router Switch Fabric and Data Path Getting Started with Fabric Troubleshooting

Figure 7-2 shows the data path from ingress to egress. (Several types of LCs are shown in this example.)

Figure 7-2 Data Path

As shown in the drawing, the path travelled by each data packet is:

Incoming interface on LC--> NP mapped to incoming interface on LC --> Bridge3 on LC --> FIA on LC --> Crossbar switch on RSP --> FIA on LC ---> Bridge3 on LC ---> NP mapped to outgoing interface ---> Outgoing Interface

Note In this document, the network processor ASICs are referred to either as network processors (NPs) or network processor units (NPUs).

Getting Started with Fabric TroubleshootingTo begin troubleshooting problems with the fabric, perform the following steps.

Step 1 Look for active platform fault manager (PFM) alarms on the LCs and RSPs.

Step 2 Check that you have the appropriate version of the bridge field-programmable gate arrays (FPGAs) in your RSP card.

Step 3 Check that you have the correct software version, board, and FPGA and ASIC versions.

RP/0/RSP0/CPU0:router# show version

2808

88

Backplane

RSP0

Fabricarbiter

CPU

GESwitch

FabricFabric

Fabric I/O

SystemTiming

RSP1

Fabricarbiter

CPU0

GESwitch

FabricFabric

Fabric I/O

SystemTiming

40x1GEFixed LC

10xSFP

10xSFP

10xSFP

10xSFP

CPU

GEPHYFabric I/O

NPU NPU

FPGA

NPU NPU

FPGA

8x10GEFixed LC

10 GE

XF

P

10 GE

XF

P

10 GE

XF

P

10 GE

XF

P

CPU

GEPHYFabric I/O

NPU NPU

FPGA

10 GE

XF

P

10 GE

XF

P

10 GE

XF

P

10 GE

XF

P

NPU NPU

FPGA

4x10GEFixed LC

10 GE

XF

P

10 GE

XF

P

CPU

GEPHYFabric I/O

NPU NPU

FPGA

10 GE

XF

P

10 GE

XF

P

NPU NPU

FPGA


OL-23591-02

Chapter 7 Troubleshooting Router Switch Fabric and Data Path Troubleshooting Packet Drops

RP/0/RSP0/CPU0:router# show inventory raw RP/0/RSP0/CPU0:router# show hw-module fpd location all

Step 4 Check if there are any errors detected by the system diagnostics.

RP/0/RSP0/CPU0:router# show diag

Step 5 Check that you have the appropriate version of the NPs in your RSP cards.

RP/0/RSP0/CPU0:router# show controllers np summary all

Node: 0/1/CPU0:---------------------------------------------------------------- [total 4 NP] Driver - Version 10.26a Build 9 ( Dec 13 2008, 20:47:03 ) NP 0 : Hardware rev v2 A1 : Ucode - Version: 255.255 Build Date: ( Dec 12 2008, 2:13:00 ) NP 1 : Hardware rev v2 A1 : Ucode - Version: 255.255 Build Date: ( Dec 12 2008, 2:13:00 ) NP 2 : Hardware rev v2 A1 : Ucode - Version: 255.255 Build Date: ( Dec 12 2008, 2:13:00 ) NP 3 : Hardware rev v2 A1 : Ucode - Version: 255.255 Build Date: ( Dec 12 2008, 2:13:00 ) Node: 0/2/CPU0: <-- [ LC built with A0 NPU that has known issue ]---------------------------------------------------------------- [total 4 NP] Driver - Version 10.26a Build 9 ( Dec 13 2008, 20:47:03 ) NP 0 : Hardware rev v2 A0 : Ucode - Version: 255.255 Build Date: ( Dec 12 2008, 2:13:00 ) NP 1 : Hardware rev v2 A0 : Ucode - Version: 255.255 Build Date: ( Dec 12 2008, 2:13:00 ) NP 2 : Hardware rev v2 A0 : Ucode - Version: 255.255 Build Date: ( Dec 12 2008, 2:13:00 ) NP 3 : Hardware rev v2 A0 : Ucode - Version: 255.255 Build Date: ( Dec 12 2008, 2:13:00 )

Troubleshooting Packet DropsThis section explains how to track packets through the system from ingress to egress, and how to troubleshoot packet drops. It includes the following sections:

• Displaying Traffic Status in Line Cards and RSP Cards, page 7-147

• Locating Packet Drops by Examining Counters, page 7-148

• Locating Drops of Punted Packets, page 7-155

• Packet Drop from LC to LC, page 7-157

• Packet Drop Between RSP and LC, page 7-158

• Packet Drop After Certain Actions, page 7-160

• Packet Drop After a Redundancy Switchover, page 7-161

• Packet Drop with Unknown Reason, page 7-163


OL-23591-02


Displaying Traffic Status in Line Cards and RSP CardsFigure 7-3 shows the traffic path on the LC and the corresponding CLI commands you use to display the status at each point in the path.

Figure 7-3 LC Traffic Path and Corresponding CLI Commands

2813

43

Fabric I/O

BridgeFPGA 0

BridgeFPGA 1

PHY

PHY

NPU-0

NPU-1

PHY

PHY

NPU-2

NPU-3

Line Card

show controllers np fabric-counters all np3 location 0/6/CPU

show interfaces gigabitEthernet 0/6/0/4

show controllers np counters all

show controllers fabric fia bridge ddr-status location <...>show controllers fabric fia bridge flow-control location <...>show controllers fabric fia bridge stats location <...>show controllers fabric fia bridge sync-status location <...>

show controllers fabric fia link-status location <...>show controllers fabric fia stats location <...>show controllers fabric fia drops <ingress | egress> location <...>show controllers fabric fia errors <ingress | egress> location <...>

To RSPfabric I/O


OL-23591-02


Figure 7-4 shows the traffic path on the RSP and the corresponding CLI commands you use to display information at each point in the path.

Figure 7-4 RSP Traffic Path and Corresponding CLI Commands

Locating Packet Drops by Examining CountersTo locate the source of packet drops, perform the following procedure.

SUMMARY STEPS

1. Clear the interface counters

2. Clear the NP counters

3. Clear the fabric counters

4. Start the traffic pattern that caused the packet drop

5. Display the NP-to-interface mapping.

6. Check the counters at the input interface

7. Check the NP counters

8. Check the NP Bridge3 counters

9. Check the bridge counters

2813

44

Fabricarbiter

FabricXBAR 0

FabricXBAR 1

CPU

FPGA

RSP-0

show controllers fabric fia bridge ddr-status location <...>show controllers fabric fia bridge stats location <...>

show controllers fabric fia link-status location <...>show controllers fabric fia stats location <...>show controllers fabric fia drops <ingress | egress> location <...>show controllers fabric fia errors <ingress | egress> location <...>

Fabric I/O

show controllers fabric crossbar serdes instance <0 or 1> location <...>show controllers fabric crossbar statistics instance <0 or 1> location <...>show controllers fabric Itrace crossbar all location <...>

show controllers fabric arbiter serdes location <...>show controllers fabric arbiter configstatus location <...> <0..4> <0>


OL-23591-02


10. Check the fabric interface ASIC (FIA) counters

11. Check the crossbar counters

Note For the procedure to troubleshoot drops of punted packets, see the Locating Drops of Punted Packets, page 7-155.

DETAILED STEPS

Step 1 Clear the interface counters.

RP/0/RSP0/CPU0:router# clear counters all

Clear "show interface" counters on all interfaces [confirm]

Step 2 Clear the NP counters.

RP/0/RSP0/CPU0:router# clear controller np counters all

Step 3 Clear fabric counters.

a. Clear FIA and bridge counters on the LC and RSP.

RP/0/RSP0/CPU0:router# clear controller fabric fia location

b. Clear fabric crossbar counters.

RP/0/RSP0/CPU0:router# clear controller fabric crossbar-counters location

Step 4 Start the traffic pattern that caused the packet drop.

Step 5 Run the following command to display the NP-to-interface mapping.

RP/0/RSP0/CPU0:router# show controllers np ports all

Step 6 Check the counters at the input interface.

RP/0/RSP0/CPU0:router# show interfaces type location

Step 7 Check the NP counters to verify that traffic is flowing in NP counters along the data path.

RP/0/RSP0/CPU0:router# show controllers np counters {np0|np1|np2|np3|all} location node-id {| include DROP}

RP/0/RSP0/CPU0:router# show controllers np counters np3 location 0/0/CPU0

RP/0/RSP0/CPU0:router# show controllers np counters np3 location 0/0/CPU0 | include DROP

The show controllers np command displays information about counters that helps you troubleshoot drops in the LCs. The names of the internal NP counters have the general format STAGE_DIRECTION_ACTION, for example, PARSE_FABRIC_RECEIVE_CNT, RESOLVE_EGRESS_DROP_CNT, and MODIFY_FRAMES_PADDED_CNT.

The values of stage, directon, and action are as follows:

• There are five stages in the NP:

– Parse

– Search-I

– Modify

– Search-II


OL-23591-02


– Resolve

• Examples of the direction are:

– Ingress

– Egress

– Next_hop

• Examples of the action are:

– Drop_count

– Down

There are additional counters, such as DROP, PUNT, and DIAGS, that provide important information but are not associated with a specific internal NP stage. Drop and punt counters are kept as an aggregate total per stage.

Example

RP/0/RSP0/CPU0:router# show controllers np ports all

Thu Jan 1 02:18:48.264 UTC Node: 0/0/CPU0:----------------------------------------------------------------NP Bridge Fia Ports -- ------ --- ---------------------------------------------------0 1 0 GigabitEthernet0/0/0/30 - GigabitEthernet0/0/0/39 1 1 0 GigabitEthernet0/0/0/20 - GigabitEthernet0/0/0/29 2 0 0 GigabitEthernet0/0/0/10 - GigabitEthernet0/0/0/19 3 0 0 GigabitEthernet0/0/0/0 - GigabitEthernet0/0/0/9

RP/0/RSP0/CPU0:router# show interfaces tenGigE 0/1/0/0

Thu Jan 1 01:10:01.908 UTCTenGigE0/1/0/0 is up, line protocol is up Interface state transitions: 1 Hardware is TenGigE, address is 001e.bdfd.1736 (bia 001e.bdfd.1736) Layer 2 Transport Mode MTU 1514 bytes, BW 10000000 Kbit reliability 255/255, txload 0/255, rxload 0/255 Encapsulation ARPA, Full-duplex, 10000Mb/s, LR, link type is force-up output flow control is off, input flow control is off loopback not set, Maintenance is enabled, ARP type ARPA, ARP timeout 04:00:00 Last clearing of "show interface" counters never 5 minute input rate 0 bits/sec, 0 packets/sec 5 minute output rate 0 bits/sec, 0 packets/sec 0 packets input, 0 bytes, 0 total input drops 0 drops for unrecognized upper-level protocol Received 0 broadcast packets, 0 multicast packets 0 runts, 0 giants, 0 throttles, 0 parity 0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort 0 packets output, 0 bytes, 0 total output drops Output 0 broadcast packets, 0 multicast packets 0 output errors, 0 underruns, 0 applique, 0 resets 0 output buffer failures, 0 output buffers swapped out 1 carrier transitions


OL-23591-02


In the following example, there were some ingress and egress drops in the RESOLVE stage. All of these drops in the ingress (9 drops) and egress (6 drops) were caused by the next hop being unreachable (a total of 15 drops for IPv4 next hop down).

RP/0/RSP0/CPU0:router# show controllers np counters np3 location 0/0/CPU0 | include DROP Mon Nov 15 12:18:35.289 EST

30 RESOLVE_INGRESS_DROP_CNT 9 0 31 RESOLVE_EGRESS_DROP_CNT 6 0 295 DROP_IPV4_NEXT_HOP_DOWN 15 0

The following example shows a typical output from the same command, but without the modifier | include DROP.

RP/0/RSP0/CPU0:router# show controllers np counters np3 Mon Nov 15 12:20:35.289 EST

Node: 0/0/CPU0:----------------------------------------------------------------Show global stats counters for NP3, revision v3

Read 20 non-zero NP counters:Offset Counter FrameValue Rate (pps)------------------------------------------------------------------------------- 23 PARSE_FABRIC_RECEIVE_CNT 417 0 30 RESOLVE_INRESS_DROP_CNT 9 0 31 RESOLVE_EGRESS_DROP_CNT 6 0 53 MODIFY_FRAMES_PADDED_CNT 3230 0 67 PARSE_MOFRR_SWITCH_MSG_RCVD_FROM_FAB 920 0 70 RESOLVE_INGRESS_L2_PUNT_CNT 1081 0 71 RESOLVE_EGRESS_L3_PUNT_CNT 4613 0 74 RESOLVE_LEARN_FROM_NOTIFY_CNT 3484 0 75 RESOLVE_BD_FLUSH_DELETE_CNT 104 0 83 RESOLVE_MOFRR_HASH_UPDATE_CNT 463 0 87 RESOLVE_MOFRR_SWITCH_MSG_INGNORED 407 0 111 DIAGS 536 0 295 DROP_IPV4_NEXT_HOP_DOWN 15 0...

Step 8 Check the NP Bridge3 counters.

RP/0/RSP0/CPU0:router# show controllers np fabric-counters all ?

all All NP instances np0 NP0 instance np1 NP1 instance np2 NP2 instance np3 NP3 instance

RP/0/RSP0/CPU0:router# show controllers np fabric-counters all <np instance or all> location <location>

RP/0/RSP0/CPU0:router# show controllers np fabric-counters all np3 location 0/5/CPU0

Check the NP-bridge rx/tx counters for each NP on the LC. View the packet sent and received counts, bytes transferred, packet counters categorized by packet size, and so forth. The fields of interest are:

xaui_a_t_transmited_packets_cnt: The number of packets sent by the NP to the bridge

xaui_a_r_received_packets_cnt: The number of packets sent by the bridge to the NP

Step 9 Check the bridge counters


OL-23591-02


RP/0/RSP0/CPU0:router# show controllers fabric fia bridge stats location node-id

Examples

RP/0/RSP0/CPU0:router# show controllers fabric fia bridge stats location 0/RSP0/CPU0 Mon Nov 22 14:14:48.010 PSTDevice Rx Interface Packet Error Threshold Count Drops Drops --------------------------------------------------------------------------------Bridge0 From-Fabric(DDR) 492283 0 0 From CPU 492283 0 0

RP/0/RSP0/CPU0:router# show controllers fabric fia bridge stats location 0/1/CPU0Mon Nov 22 14:18:54.834 PST

UC - Unicast , MC - MulticastLP - LowPriority , HP - HighPriority

-------------------------------------------------------------------------------- FIA 0 ******Cast/ Packet Packet Error Threshold Prio Direction Count Drops Drops --------------------------------------------------------------------------------

Unicast Egress Stats********************UC HP Fabric to NP-0 70329 0 0 UC LP Fabric to NP-0 0 0 0 UC HP Fabric to NP-1 70329 0 0 UC LP Fabric to NP-1 0 0 0 UC HP Fabric to NP-2 70329 0 0 UC LP Fabric to NP-2 0 0 0 UC HP Fabric to NP-3 70329 0 0 UC LP Fabric to NP-3 0 0 0 ----------------------------------------------------------------UC Total Egress 281316 0 0

Multicast Egress Stats*********************MC HP Fabric to NP-0 0 0 0 MC LP Fabric to NP-0 0 0 0 MC HP Fabric to NP-1 0 0 0 MC LP Fabric to NP-1 0 0 0 MC HP Fabric to NP-2 0 0 0 MC LP Fabric to NP-2 0 0 0 MC HP Fabric to NP-3 0 0 0 MC LP Fabric to NP-3 0 0 0 ---------------------------------------------------------------MC Total Egress 0 0 0

Cast/ Packet Packet Prio Direction Count --------------------------------------------------Unicast Ingress Stats*********************UC HP NP-0 to Fabric 70329 UC LP NP-0 to Fabric 0 UC HP NP-1 to Fabric 70329 UC LP NP-1 to Fabric 0 UC HP NP-2 to Fabric 70329


OL-23591-02


UC LP NP-2 to Fabric 0 UC HP NP-3 to Fabric 70329 UC LP NP-3 to Fabric 0 --------------------------------------------------UC Total Ingress 281316

Multicast Ingress Stats***********************MC HP NP-0 to Fabric 0 MC LP NP-0 to Fabric 0 MC HP NP-1 to Fabric 0 MC LP NP-1 to Fabric 0 MC HP NP-2 to Fabric 0 MC LP NP-2 to Fabric 0 MC HP NP-3 to Fabric 0 MC LP NP-3 to Fabric 0 --------------------------------------------------MC Total Ingress 0

Ingress Drop Stats (MC & UC combined)**************************************PriorityPacket Error Threshold Direction Drops Drops --------------------------------------------------LP NP-0 to Fabric 0 0 HP NP-0 to Fabric 0 0 LP NP-1 to Fabric 0 0 HP NP-1 to Fabric 0 0 LP NP-2 to Fabric 0 0 HP NP-2 to Fabric 0 0 LP NP-3 to Fabric 0 0 HP NP-3 to Fabric 0 0 -------------------------------------------------- Total IngressDrops 0 0

Step 10 Check the FIA counters

RP/0/RSP0/CPU0:router# show controllers fabric fia stats location locationExamples:RP/0/RSP0/CPU0:router# show controllers fabric fia stats location 0/RSP0/CPU0

Wed Aug 25 12:36:43.151 DST

FIA:0 DDR Packet counters:=========================From Punt 686545 To Punt 582387

FIA:0 SuperFrame counters:=========================To Unicast Xbar[0] 821335 To Unicast Xbar[1] 0 To Unicast Xbar[2] 0 To Unicast Xbar[3] 0 To MultiCast Xbar[0] 7758 To MultiCast Xbar[1] 0 To MultiCast Xbar[2] 15807 To MultiCast Xbar[3] 0

From Unicast Xbar[0] 629854 From Unicast Xbar[1] 0 From Unicast Xbar[2] 1


OL-23591-02


From Unicast Xbar[3] 0 From MultiCast Xbar[0] 2589 From MultiCast Xbar[1] 0 From MultiCast Xbar[2] 2588 From MultiCast Xbar[3] 0

FIA:0 Total Drop counters:=========================Ingress drop: 0 Egress drop: 2 Total drop: 2

RP/0/RSP0/CPU0:router# show controllers fabric fia stats location 0/2/CPU0

FIA:0 DDR Packet counters:=========================From Bridge#[0] 510 To Bridge #[0] 510 From Bridge#[1] 510 To Bridge #[1] 510

FIA:0 SuperFrame counters:=========================To Unicast Xbar[0] 19 To Unicast Xbar[1] 20 To Unicast Xbar[2] 0 To Unicast Xbar[3] 0 To MultiCast Xbar[0] 0 To MultiCast Xbar[1] 0 To MultiCast Xbar[2] 0 To MultiCast Xbar[3] 0

From Unicast Xbar[0] 19 From Unicast Xbar[1] 20 From Unicast Xbar[2] 0 From Unicast Xbar[3] 0 From MultiCast Xbar[0] 0 From MultiCast Xbar[1] 0 From MultiCast Xbar[2] 0 From MultiCast Xbar[3] 0

FIA:0 Total Drop counters:=========================Ingress drop: 0 Egress drop: 0 Total drop: 0

RP/0/RSP0/CPU0:router# show controllers fabric fia q-depth [location location]

Thu Jan 1 02:16:37.227 UTCFIA 0------Total Pkt queue depth count = 0

Step 11 Check the crossbar counters to make sure there are no dropped packets.

RP/0/RSP0/CPU0:router# show controllers fabric crossbar statistics instance [0|1] location location

Example:RP/0/RSP0/CPU0:router# show controllers fabric crossbar statistics instance 0 location 0/RSP0/CPU0

Location: 0/RSP0/CPU0 (physical slot 4)


OL-23591-02


Asic Instance: 0 Fabric info for node 0/RSP0/CPU0 (physical slot: 4)

Dropped packets : mcast unicast +---------------------------------------------------------------+ Input buf bp pkts : 0 0 Output buf bp pkts : 0 0 Xbar timeout buf bp pkts : 0 0 HOL drop pkts : 0 0 Null POE drop pkts : 0 0

Locating Drops of Punted PacketsTo locate drops of punted packets, perform the following procedure.

SUMMARY STEPS

1. Clear all packet counters

2. Start traffic

3. Check traffic counters at each component

4. Check NP counters for NP mapping to interface, and check NP0 for inject packet count

5. Check fabric-related counters

6. Check punt FPGA counters

DETAILED STEPS

Step 1 Clear all packet counters as described in the “Locating Packet Drops by Examining Counters” section on page 7-148.

Step 2 Start traffic.

Step 3 Check traffic counters at each component in the punted packet path. Use a procedure similar to the one described in the “Locating Packet Drops by Examining Counters” section on page 7-148. However, for punted packets, the data path is:

Incoming Interface --> NP --> LC CPU --> NP --> Bridge3 --> LC FIA --> RSP Crossbar--> Punt FPGA on RSP --> RSP CPU --> RSP FIA --> RSP Crossbar --> LC FIA --> LC CPU --> NP0 ---> LC FIA ---> Crossbar ---> RSP FIA ---> RSP CPU

Step 4 Check the NP counters for NP mapping to interface, and check NP0 for the inject packet count. The following fields provide information on the NP counters:

801 PARSE_FABRIC_RECEIVE_CNT

820 PARSE_LC_INJECT_TO_FAB_CNT

872 RESOLVE_INGRESS_L2_PUNT_CNT

970 MODIFY_FABRIC_TRANSMIT_CNT

822 PARSE_FAB_INJECT_IPV4_CNT

Step 5 Check the fabric-related counters for any packet drops.

RP/0/RSP0/CPU0:router# show controllers fabric crossbar statistics instance 0 location 0/RSP0/CPU0


OL-23591-02


RP/0/RSP0/CPU0:router# show controllers fabric fia stats [location location]

Example: RP/0/RSP0/CPU0:router# show controllers fabric fia stats location 0/5/CPU0

RP/0/RSP0/CPU0:router# show controllers fabric fia bridge stats [location location]

Examples:RP/0/RSP0/CPU0:router# show controllers fabric fia bridge stats location 0/RSP0/CPU0

Wed Aug 25 14:12:03.916 DST

Device Rx Interface Packet Error Threshold Count Drops Drops --------------------------------------------------------------------------------Bridge0 From-Fabric(DDR) 603698 0 0 From CPU 711734 0 0

RP/0/RSP0/CPU0:router# show controllers fabric fia bridge stats location 0/5/CPU0

Wed Aug 25 14:12:20.867 DST

UC - Unicast , MC - MulticastLP - LowPriority , HP - HighPriority

-------------------------------------------------------------------------------- FIA 0 ******Cast/ Packet Packet Error Threshold Prio Direction Count Drops Drops --------------------------------------------------------------------------------

Unicast Egress Stats********************UC HP Fabric to NP-0 28 0 0 UC LP Fabric to NP-0 0 0 0 UC HP Fabric to NP-1 28 0 0 UC LP Fabric to NP-1 0 0 0 UC HP Fabric to NP-2 28 0 0 UC LP Fabric to NP-2 0 0 0 UC HP Fabric to NP-3 28 0 0 UC LP Fabric to NP-3 0 0 0 ----------------------------------------------------------------UC Total Egress 112 0 0

Multicast Egress Stats*********************MC HP Fabric to NP-0 205 0 0 MC LP Fabric to NP-0 2 0 0 MC HP Fabric to NP-1 205 0 0 MC LP Fabric to NP-1 2 0 0 MC HP Fabric to NP-2 205 0 0 MC LP Fabric to NP-2 2 0 0 MC HP Fabric to NP-3 205 0 0 MC LP Fabric to NP-3 2 0 0 ---------------------------------------------------------------MC Total Egress 828 0 0

--More--

Step 6 To check for packets punted to and injected from the LC or RP CPU, run the following commands.

RP/0/RSP0/CPU0:router# show spp interface location node-id


OL-23591-02


RP/0/RSP0/CPU0:router# show spp node-counters location node-id

RP/0/RSP0/CPU0:router# show spp node location node-id

RP/0/RSP0/CPU0:router# show spp sid stats location node-id

RP/0/RSP0/CPU0:router# show spp client location node-id

Note To clear the spp counters, run the command clear spp {client | interface | node-counters} location node-id. This command clears client statistics, interface statistics, and per-node counters, depending on the keyword you use.

Step 7 To query the punt switch for the statistics on the LC CPU, run the following command.

RP/0/RSP0/CPU0:router# show controllers punt-switch switch-stats location node-id

Packet Drop from LC to LCIn this scenario, you have configured the system, RSP and LC have come up and are stable, LC to LC traffic is going through, but some packets are dropped.

The possible causes are:

• Traffic dropped at interface

• Traffic dropped at NP3

• Traffic dropped at bridge

• Traffic dropped at the fabric I/O

• Synchronization between the fabric I/O and the fabric NP or fabric arbiter NP has a problem

• Traffic has wrong vqi

• Oversubscribed traffic

• Unknown failures

Locate the Problem and Take Corrective Action

Follow this procedure to locate the problem. After you locate the problem, take corrective action based on your findings. Corrective action might include, for example, configuration updates or hardware/software version upgrades.

Step 1 If not already done, perform the procedures in the “Getting Started with Fabric Troubleshooting” section on page 7-145 to verify that you have the correct versions of the hardware and software.

Step 2 Collect the sync status of fabric on the LC.

show controllers fabric fia link-status location <0/1/CPU0>

show controllers fabric fia bridge ddr-status location <0/1/cpu0>

show controllers fabric fia bridge sync-status location 0/1/cpu0

Step 3 Collect configuration information.

show run


OL-23591-02


Step 4 dump PFM errors on both source and destination LC.

show pfm location <0/1/cpu0>

Step 5 Collect the fabric I/O/Bridge counters on both source and destination card.

show interfaces


show controllers fabric fia stats location 0/1/CPU0

show controllers fabric fia bridge stats location 0/1/CPU0

Step 6 Collect redundancy information.

show redundancy

Where to Go Next

If you have not been able to locate or correct the problem, you might be able to clear it by performing the following steps. However, these steps might delete information that would help you perform additional troubleshooting with Cisco Technical Support. Some of the steps involve stopping or reducing traffic streams, which might not be appropriate on a deployed system. Consult with your network administrator before you perform any of these steps.

Caution Before you follow these next steps, consider contacting Cisco Technical Support. Some of these steps can cause loss of data that would be useful for future analysis and troubleshooting, or could cause loss of traffic.

Step 1 Perform ‘reset –h’ at LC ROMMON and reboot the LC again to see if this clears the problem.

Step 2 Pull out the LC and reinsert it to see if it can boot up.

Step 3 Stop other streams of traffic to see if this failed stream can go through.

Step 4 Reduce the rate of the traffic to see if the drop continues.

Packet Drop Between RSP and LCIn this scenario, you have configured the system, RSP and LC have come up and are stable, but one of the following problems occurred:

• Protocol or ping traffic (punt path traffic) has some drops

• Initially the ping/protocol packets are not going through, but later recover.






OL-23591-02



• Sync between the fabric I/O and the fabric NP or fabric arbiter NP has a problem


• Traffic drop at Punt FPGA

• sn database sync issue





Step 2 Collect the sync status of fabric on the linecard.





show run

Step 4 Dump the PFM errors for the card.


show pfm location <0/rsp0/cpu0>

Step 5 Collect the fabric I/O/bridge counters on both RSP and LC.

show interfaces




show controllers fabric fia stats location 0/rsp0/CPU0

Where to Go Next



OL-23591-02






Step 4 Determine whether the drop is a single burst in the beginning or is continuous.

Step 5 Determine if the drop is associated with particular packet size.

Packet Drop After Certain ActionsIn this scenario, the system is configured, RSP and LC have come up, and traffic is flowing properly for some time. However, after certain action such as configuration change, online insertion and removal (OIR) of LC/RSP, LC reload, or software upgrade, some traffic drop or complete traffic loss is observed.













Step 1 Perform the procedures in the “Getting Started with Fabric Troubleshooting” section on page 7-145 to verify that you have the correct versions of the hardware and software.

Step 2 Collect the sync status of fabric on the linecard.






OL-23591-02


show run




Step 5 Collect the fabric I/O/bridge counters on both the RSP and LC.

show interfaces






show redundancy

Where to Go Next






Step 4 Repeat Step 1 through Step 3 to determine whether the results are reproducible.

Packet Drop After a Redundancy SwitchoverIn this scenario, you have configured the system, RSP and LC have come up, and traffic is flowing properly for some time. However, after a switchover (by a command or OIR), you see some traffic drop or complete traffic loss.




OL-23591-02









• Fabric is stuck





Step 2 Collect the sync status of fabric on the linecard before and after the switchover.





show run





show interfaces






show redundancy


OL-23591-02


Where to Go Next



Step 1 Stop other streams of traffic to see if this failed stream can go through again.

Step 2 Repeat Step 1 several times to determine if the result is reproducible.

Step 3 Perfom a switchover back to the other side to determine whether both directions are having the same traffic problems.

Step 4 After obtaining the necessary approvals from your network and system administrators (because this step will stop all traffic on this unit), reboot the entire system and check to see if it recovers.

Packet Drop with Unknown ReasonIn this scenario, you have configured the system, RSP and LC have come up, and traffic is flowing properly for a significant time (at least several days). However, for an unknown reason, the system experiences traffic drops or complete traffic loss.









• Fabric is stuck






OL-23591-02


Step 2 Collect the sync status of fabric on the linecard before and after the switchover.








show interfaces






show redundancy

Step 6 Check for drops on the the fabric I/O interface (FIA drop counters) on the LC in both the ingress (to fabric) and egress (from fabric) directions.

show controllers fabric fia drops egress location show controllers fabric fia drops ingress location show controllers fabric fia error egress location show controllers fabric fia error ingress location

Step 7 Check for drops on the bridge. Counters are a combination of high priority (HP), low priority (LP), unicast, multicast, DDR, and DDR-threshold packets. They are furthur segregated into critical and informational based on their severity. All Ethernet linecards have 2 bridges. Use the following command to obtain this information.

show controllers fabric fia bridge stats location <linecard location>

Step 8 Check if there are any drops on Punt FPGA on RSP.

show controllers fabric fia bridge stats location 0/RSP0/CPU0

Where to Go Next



OL-23591-02

Chapter 7 Troubleshooting Router Switch Fabric and Data Path Troubleshooting RSP and LC Crashes


Step 1 Stop other streams of traffic to see if this failed stream can go through again.

Step 2 Reboot the LCs one at a time and check if the traffic recovers.

Step 3 After obtaining the necessary approvals from your network and system administrators (because this step will stop all traffic on this unit), reboot the entire system and check to see if it recovers.

Step 4 Reconfigure the system to see if it recovers.

Troubleshooting RSP and LC CrashesThis section explains how to troubleshoot the following problems:

• Active RSP Is Crashing, page 7-165

• Standby RSP Is Crashing, page 7-166

• LC Is Crashing, page 7-167

Active RSP Is CrashingIn this scenario, the active RSP keeps crashing and the RSP console shows that the active fabric manager or fia_rsp (the fabric I/O process) terminates repeatedly.


• Initialization of the fabric I/O fails for some reason

• Fabric self-test fails

• The synchronization between the fabric I/O and the fabric NP or fabric arbiter NP has a problem





Step 2 Collect the sync status of fabric on the RSP card.

show controllers fabric fia link-status location <0/RSP0/CPU0>

show controllers fabric fia bridge sync-status location



OL-23591-02




Step 4 Collect the fabric I/O/Punt counters.

show controllers fabric fia stats location <0/rsp0/CPU0>

Where to Go Next



Step 1 Perform ‘reset –h’ at LC ROMMON and reboot the RSP again to see if this clears the problem.

Step 2 Pull out the RSP and reinsert it to see if it can boot up.

Step 3 Swap the slot (put the RSP card into the other RSP slot) and see if it can boot up properly.

Standby RSP Is CrashingIn this scenario, the active RSP is up and running, but the standby RSP keeps crashing. The RSP console shows that the standby fabric manager or fia_rsp (the fabric I/O process) terminates repeatedly.


• Initialization of the standby fabric I/O fails for some reason

• Fabric self-test on the standby card fails

• The sync between the fabric I/O and the fabric NP or fabric arbiter NP has a problem

• Communication between the active and standby card is not working






OL-23591-02


Step 2 Collect the sync status of fabric on the RSP card.

show controllers fabric fia link-status location <0/RSP0/CPU0>



Step 4 Dump the redundancy status.

show redundancy

Step 5 Collect the fabric I/O/ punt counters.

show controllers fabric fia stats location <0/1/CPU0>

Where to Go Next



Step 1 Perform ‘reset –h’ at the ROMMON and reboot the standby RSP again to see if this clears the problem.

Step 2 Pull out the RSP and reinsert it to see if it can boot up.

Step 3 Swap the slot (put the RSP card into the other RSP slot) and see if it can boot up properly.

LC Is Crashing In this scenario, a LC keeps crashing and the RSP console shows that fia_lc (the fabric I/O process) terminates repeatedly.


• Initialization of the LC fabric I/O fails for some reason

• Fabric self-test on the LC fails

• The synchronization between the fabric I/O and the fabric NP or fabric arbiter NP has a problem

• Communication between the LC and the RSP is not working properly

• There is a sync problem between the fabric I/O and the bridge



OL-23591-02

Chapter 7 Troubleshooting Router Switch Fabric and Data Path Troubleshooting Complete Loss of Traffic




Step 2 Collect the sync status of the fabric on the LC.






Step 4 Collect the fabric I/O/ bridge counters.

show controllers fabric fia stats location <0/1/CPU0>

show controllers fabric fia bridge stats location <0/1/CPU0>

Where to Go Next



Step 1 Perform ‘reset –h’ at the LC ROMMON and reboot the LC again to see if this clears the problem.


Step 3 Swap the slot (pull out the LC and insert it into another LC slot) and see if it can boot up properly.

Step 4 Put a different LC of same type to see if that card can booting up properly.

Troubleshooting Complete Loss of TrafficThis section explains how to troubleshoot scenarios in which the system is active but traffic does not go through. It includes the following topics:

• No Traffic from LC to LC, page 7-169


OL-23591-02


• No Traffic Between RSP and LC, page 7-170

No Traffic from LC to LCIn this scenario, you have configured the system and the RSP and LC have come up and are stable, but no LC-to-LC traffic is going through.


• Traffic dropped at the interface


• Traffic dropped at the bridge













show run




Step 5 Collect the fabric I/O/bridge counters on both the source and destination cards.

show interfaces





show redundancy


OL-23591-02


Where to Go Next




Step 2 Pull out the LC and reinsert it to see if it can boot up and carry traffic.


Step 4 Run online diagnostics to locate errors in the system. For additional information on diagnostics, see the “Using Diagnostic Commands” section on page 1-59.

No Traffic Between RSP and LCIn this scenario, you have configured the system and the RSP and LC have come up and are stable, but no protocol or ping traffic (punt path traffic) is going through.


• Traffic dropped at the interface


• Traffic dropped at the bridge




• Traffic dropped at the punt FPGA

• Traffic dropped at the protocol level





OL-23591-02








show run





show interfaces






show redundancy

Where to Go Next




Step 2 Pull out the LC and reinsert it to see if it can boot up and carry traffic.

Step 3 Pull out the RSP card and reinsert it to see if it can boot up and carry traffic.



OL-23591-02

Chapter 7 Troubleshooting Router Switch Fabric and Data Path Gathering Fabric Information Before Calling TAC

Step 5 Run online diagnostics to locate errors in the system. For additional information on diagnostics, see the “Using Diagnostic Commands” section on page 1-59.

Gathering Fabric Information Before Calling TACIf you need support from Cisco to troubleshoot the fabric, we recommend that you gather the following information if time permits:

• Output of the following commands (this will display software version, and the line card, fabric card, FPGA, and ASIC versions)

show version show inventory raw show diag show hw-module fpd location

• Information on chassis type

(admin) show inventory

• Platform-related information

show platform

• Ingress interface(s), egress interface(s), and expected packet path

• Drop counters

• Logs (capture all logs on the RSP console port)


OL-23591-02


C H A P T E R 8
Troubleshooting MPLS Services
This chapter describes techniques to troubleshoot MultiProtocol Label Switching (MPLS) services. MPLS carries different kinds of traffic, such as IP packets and Ethernet frames. The general flow of packets in a Cisco ASR 9000 Aggregation Series Router is as follows:

Incoming interface => Ingress NP => Switch fabric => Egress NP => Outgoing interface.

This chapter contains the following subsections:

• Verifying MPLS PIE Activation and MPLS Configuration, page 8-173

• Troubleshooting Connectivity Over MPLS, page 8-174


• IP Packets Not Forwarded to LSP, page 8-175

• IP Packets Not Forwarded to MPLS TE Tunnel, page 8-176

• MPLS Packets Not Forwarded to MPLS TE Tunnel, page 8-176

• MPLS TE Tunnels Do Not Come Up, page 8-176

• FRR-Protected Tunnel Goes Down After Triggering FRR, page 8-177

• MPLS TE FRR Database Not Built, page 8-178

• MPLS FRR Switch Time Debugging, page 8-178

Verifying MPLS PIE Activation and MPLS ConfigurationFor operation of MPLS, the MPLS PIE must be active and MPLS must be present in your running configuration:

• Verify that the MPLS PIE is installed, committed, and activated. It is not installed by default.

• Verify that MPLS is configured in your running-config. After you install the MPLS PIE, you must commit it. If you configure MPLS but you have not committed the MPLS PIE, the system deletes all of your MPLS configuration if you reload the router image.

Caution Verify that the MPLS PIE is committed before you configure MPLS. Otherwise all of your MPLS configuration data will be lost if the image is reloaded.


Chapter 8 Troubleshooting MPLS Services Troubleshooting Connectivity Over MPLS

Troubleshooting Connectivity Over MPLSThis section explains how to troubleshoot MPLS connectivity for multipoint Layer 2 services.

Step 1 Ping the opposite interface (on the remote router) on the MPLS interface. Verify that the ping is successful.

Step 2 Verify that the remote interface shows up as an ospf neighbor.

show ospf neigbor

Step 3 Verify that the remote router ID (typically the remote router loopback) is in the routing table.

show route ipv4

Step 4 Ping the IP address of the remote router (the same IP address that was displayed in Step 3). Verify that the ping is successful.

Step 5 Verify that label distribution protocol (LDP) is up between the local and remote routers.

show mpls ldp neighbor

Step 6 Verify that you can find the ID of the remote router in an MPLS command. In the case of a PW, this ID will be theIPv4 address for the PW.

Step 7 Verify that the BGP neighbor is up.

Step 8 If you are using PW in the core, verify that the PWs are properly configured on both PEs.

Step 9 Check that configurations are correct on all peers in the VPLS domain. This includes, for example, loopbacks, IGP (OSPF or ISIS), LDP, BGP, and L2VPN.

Note L2VPN services rely on Layer 3 connectivity from the PE through the core. If you need to reconfigure any routing parameters, use the procedures shown in Cisco ASR 9000 Series Aggregation Services Router Routing Configuration Guide, Release 4.0.


1. debug mpls ldp transport events

2. debug mpls ldp transport connections

3. show mpls forwarding tunnels detail

4. debug mpls ea platform {all | errors | events | info} [ location ]

5. show cef platform trace [adj | all } common | fwdwlk | ipfrr | ipv4 | ipv6 | mpls | rpf | te] location node-id

6. show cef platform resource location node-id

7. show mpls forwarding labels label-id hardware egress location node-id

8. show mpls ldp discovery


OL-23591-02



Chapter 8 Troubleshooting MPLS Services IP Packets Not Forwarded to LSP

9. show mpls ldp neighbor

DETAILED STEPS

IP Packets Not Forwarded to LSPThese commands help to troubleshoot IP packets not being forwarded on the label switched path (LSP).

Step 1 Check the prefix information.

RP/0/RSP0/CPU0:router# show cef prefix/length

Step 2 Check the hardware label FIB.

RP/0/RSP0/CPU0:router# show mpls forwarding labels label-id hardware egress location node-id

Step 3 Find out whether the ARP resolves for the next hop prefix.

RP/0/RSP0/CPU0:router# show arp prefix location node-id

Workaround

If ARP is not resolved, ping the destination and run the show arp command again to see if the ARP resolves. If it does not resolve, it means there is no path to reach the destination, or the destination is not operational.

If ARP is resolved, use the clear arp location node-id command to clear the ARP information on the node. This command clears the current ARP table entries, and the system will refill the ARP entries with the latest ARP information. This might help if there are stale and incorrect entries in the ARP table.


Step 1 debug mpls ldp transport event Displays and logs discovery and connection setup/shutdown events.

Step 2 debug mpls ldp transport connection Displays and logs connection setup/shutdown events.

Step 3 show mpls forwarding tunnels detail Display MPLS tunnel status.

Step 4 debug mpls ea platform {all | errors | events | info} [location]

Displays and logs MPLS setup events and errors.

Step 5 show cef platform trace [adj | all | common | fwdwlk | ipfrr | ipv4 | ipv6 | mpls | rpf | te] location node-id

Display data path setup event and error logs.

Step 6 show cef platform resource location node-id Display line card resource event and error logs.

Step 7 show mpls forwarding label label-id hardware egress location node-id

Display MPLS label status.

Step 8 show mpls ldp discovery Display MPLS LDP status.

Step 9 show mpls ldp neighbor Display MPLS LDP neighbor status.


OL-23591-02

Chapter 8 Troubleshooting MPLS Services IP Packets Not Forwarded to MPLS TE Tunnel

IP Packets Not Forwarded to MPLS TE Tunnel

Step 1 Check the tunnel adjacency of the prefix.

RP/0/RSP0/CPU0:router# show cef prefix hardware ingress location node-id

Step 2 Ensure that the MPLS traffic tunnel is up.

RP/0/RSP0/CPU0:router# show mpls traffic-eng tunnels up

Step 3 Check the hardware TE label FIB by running the following command on the ingress LC for the unicast traffic.

RP/0/RSP0/CPU0:router# show cef adjacency tunnel-te tunnel-id hardware ingress location node-id

Workaround:

Enter the shut command (followed by commit) and the no shut command (followed by commit) on the tunnel interface to reprogram the hardware.

MPLS Packets Not Forwarded to MPLS TE Tunnel

Step 1 Ensure that transmit adjacency is complete. RP/0/RSP0/CPU0:router# show mpls forwarding labels label-id hardware egress location node-id

Step 2 Ensure that hardware tunnel adjacency is complete by running the following command on the ingress LC for the unicast traffic.

RP/0/RSP0/CPU0:router# show cef mpls adjacency tunnel-te te-id hardware egress location node-id

Workaround

Perform the shut command (followed by commit) and the no shut command (followed by commit) of the tunnel interface to reprogram the hardware.

MPLS TE Tunnels Do Not Come Up

Step 1 Ensure that the tunnel egress interface is configured in RSVP.

rsvp interface Bundle-Ether1 bandwidth 100000 ! interface GigabitEthernet0/1/0/2 bandwidth 100000 !


OL-23591-02

Chapter 8 Troubleshooting MPLS Services FRR-Protected Tunnel Goes Down After Triggering FRR

interface GigabitEthernet0/4/0/8 <<---- tunnel egress interface bandwidth 100000 ! interface GigabitEthernet0/4/0/20 bandwidth 100000 !mpls traffic-eng <<---- Ensure that the tunnel egress interface is configured in mpls traffic-engineering config interface Bundle-Ether1 ! interface GigabitEthernet0/1/0/2 ! interface GigabitEthernet0/4/0/8 <<---- tunnel egress interface ! interface GigabitEthernet0/4/0/20 !

Step 2 Verify that the tunnel egress interface is up, for example:

RP/0/RSP0/CPU0:router# show interface GigabitEthernet0/4/0/8

Step 3 Ensure that traffic engineering is configured in OSPF.

router ospf te log adjacency changes detail router-id 192.168.1.30 area 0 mpls traffic-eng

Step 4 Ensure that ping is successful on tunnel destination IP.

FRR-Protected Tunnel Goes Down After Triggering FRRFast ReRoute (FRR) is a mechanism for protecting MPLS Traffic Engineering (TE) label-switched paths (LSPs) from link and node failures by locally repairing the LSPs at the point of failure, allowing data to continue to flow on them while their headend routers attempt to establish new end-to-end LSPs to replace them. FRR locally repairs the protected LSPs by rerouting them over backup tunnels that bypass failed links or nodes.

Step 1 Ping the address in an updated sender template. RP/0/RSP0/CPU0:router# ping PLR_Address

Step 2 Ensure that the MP address is reachable. Check forwarding over the backup tunnel is working. RP/0/RSP0/CPU0:router# ping backup_tunnel_destination

Step 3 Ensure that the backup tunnel is in Up, Up state. RP/0/RSP0/CPU0:router# show mpls traffic-eng tunnels

Step 4 Check RSVP traces to find out why the tunnel went down. RP/0/RSP0/CPU0:router# show mpls traffic-eng trace event


OL-23591-02

Chapter 8 Troubleshooting MPLS Services MPLS TE FRR Database Not Built

MPLS TE FRR Database Not Built

Step 1 Ensure that the protected tunnel is fast reroutable.

a. RP/0/RSP0/CPU0:router# show mpls traffic-eng fast-reroute database

Step 2 Ensure that backup does not pass through a protected interface. RP/0/RSP0/CPU0:router# show mpls traffic-eng tunnel backup protected-interface

Step 3 Ensure that backup has enough backup bandwidth (if configured). RP/0/RSP0/CPU0:router# show mpls traffic-eng tunnels backup

Step 4 Ensure that backup and protected tunnels have a merge point (check hop information). RP/0/RSP0/CPU0:router# show mpls traffic-eng tunnels

Step 5 Ensure that protected and backup tunnels are in Up, Up state. RP/0/RSP0/CPU0:router# show mpls traffic-eng tunnels brief

Note Protected tunnels with 0 signaled bandwidth cannot be protected by limited backup-bw tunnels.

Step 6 Enable debugs, remove, and reapply backup tunnel-te. RP/0/RSP0/CPU0:router# debug mpls traffic-eng frr

Step 7 Shut and no shut the backup tunnel and/or protected tunnel (if possible). This resets backup tunnel assignments.

Step 8 Ensure that the fast-reroute option is not configured on the backup tunnel. RP/0/RSP0/CPU0:router# show running-config interface tunnel-te15

Step 9 Find out if backup is assigned to a protected LSP.

a. RP/0/RSP0/CPU0:router# show mpls traffic-eng fast-reroute database

b. RP/0/RSP0/CPU0:router# show mpls traffic-eng forwarding

c. RP/0/RSP0/CPU0:router# show rsvp fast-reroute

Step 10 Ensure that the pool-type of the protected LSP bandwidth and backup-bw of the backup tunnel matches.

MPLS FRR Switch Time Debugging

Step 1 Ensure that the FRR database is built and in ready state. RP/0/RSP0/CPU0:router# show mpls traffic-eng fast-reroute database

Step 2 Upon FRR triggered, ensure that FRR is in the active state. RP/0/RSP0/CPU0:router# show mpls traffic-eng fast-reroute database


OL-23591-02

Chapter 8 Troubleshooting MPLS Services MPLS FRR Switch Time Debugging

Step 3 Check FRR switch time of LC that the primary tunnel is failed. RP/0/RSP0/CPU0:router# show mpls traffic-eng fast-reroute log location node-id

Step 4 Ensure that both primary and backup tunnels on the LC received the FRR trigger.

RP/0/RSP0/CPU0:router# show cef platform trace te all location node-id


OL-23591-02

Chapter 8 Troubleshooting MPLS Services MPLS FRR Switch Time Debugging


OL-23591-02


C H A P T E R 9
Troubleshooting L2VPN and Ethernet Services
This chapter describes techniques to troubleshoot Layer 2 virtual private network (L2VPN) features. In this document, L2VPN refers to a family of Layer 2 functions and Ethernet services provided by the Cisco ASR 9000 Aggregation Series Router.

If you are experiencing a problem with L2VPN traffic, the source of the problem could be caused by any of the following conditions:

• Interfaces in the customer edge (CE) router down or configured incorrectly.

• Interfaces in the provider edge (PE) router down or configured incorrectly.

• MAC address updates not functioning correctly.

• Bridge domain not configured correctly.

• Routing in the core network down or not configured correctly.

This chapter contains the following sections that explain how to troubleshoot these conditions:

• Troubleshooting VLAN Traffic and L2 TCAM Classification, page 9-181

• Troubleshooting Multipoint Layer 2 Services, page 9-190

• Troubleshooting Point-to-Point Layer 2 Services, page 9-206

• Troubleshooting Specific Outage Scenarios In Layer 2 Services, page 9-214

• Troubleshooting Dynamic Host Configuration Protocol Snooping, page 9-227

• Troubleshooting Multiple Spanning Tree, page 9-230

• Additional References—Command Reference and Configuration Guides, page 9-232

Troubleshooting VLAN Traffic and L2 TCAM ClassificationThis section explains how to troubleshoot VLAN traffic problems related to Layer 2 TCAM classification. (TCAM = ternary content addressable memory.) It contains the following topics:

• Understanding Problems with VLAN Traffic and L2 TCAM Classification, page 9-182

• Verifying the Configuration Is Correct, page 9-182

• Verifying Interfaces, Subinterfaces, and Packet Forwarding, page 9-183


Chapter 9 Troubleshooting L2VPN and Ethernet Services Troubleshooting VLAN Traffic and L2 TCAM Classification

Understanding Problems with VLAN Traffic and L2 TCAM ClassificationIf traffic on a VLAN is not getting through, the traffic might not be reaching the subinterface for which it is intended. The problem could be related to any of the following:

• The main interface (trunk) or subinterface—Problems could be caused by physical issues or configuration errors.

• Incorrect classification (tagging) of the traffic—If traffic has the wrong VLAN tag, it cannot reach the intended subinterface. Furthermore, the main interface cannot route the traffic, because it does not classify or forward tagged traffic.

• A remote peer could be sending messages with an unknown VLAN number or encapsulation type.

Drop counters on the main interface and subinterface indicate where the traffic is being dropped.

• If a packet has an incorrect VLAN tag, the main interface drops the packet and the main interface drop counter increments.

• If the packet has a correct VLAN tag, it reaches the intended subinterface, but if the subinterface drops the packet for any reason, the subinterface drop counter increments.

Verifying the Configuration Is CorrectIn many cases, VLAN traffic failures are caused by configuration problems. Some configuration omissions and errors can go unnoticed, because a bridge domain does not always display a commit failure when an incorrect configuration is committed. You need to verify that your configuration is correct by using the show commands listed in this section.

The system allows you to configure and commit a bridge domain with subinterfaces assigned to the ACs, even if you have not yet created the subinterfaces themselves. However, the ACs will be operationally down until you configure and commit the necessary subinterfaces.

Verify that your configuration is consistent with the following recommendations and requirements:

• We recommend as a best practice that you assign the same VLAN tag to all the ACs in a bridge domain.

• When you create a main interface for the AC (in interface config mode):

– You cannot configure an encapsulation statement

– You must include the l2transport keyword on a separate command line

Example:

interface GigabitEthernet0/1/0/1 l2transport

• When you create a subinterface for the AC (in interface config mode):

– You must include the l2transport keyword on the same command line

– You must configure an encapsulation statement

Example:

interface GigabitEthernet0/2/0/2.2 l2transport encapsulation dot1q 100

• Review your running configuration to verify that it is complete and the necessary interfaces are up. (show running-config).


OL-23591-02


• Ensure that the interfaces and subinterfaces for the ACs are actually up. View the up/down status of the bridge domain, ACs, and PWs (if present) by means of the show l2vpn bridge-domain summary command. Verify that the counts are incrementing, which means that the ACs are up.

• Make sure that bridge ports (for example, ACs and PWs) are assigned to the bridge domains.

• Verify that a unique main or subinterface is assigned to each AC in the bridge domain.

Verifying Interfaces, Subinterfaces, and Packet ForwardingPerform these steps to verify that the interface and subinterface (if applicable) are up, and that Layer 2 virtual private network (L2VPN) packets are being forwarded on the interface and subinterface.

Correct any problems you discover, then rerun the show commands in this section.

Step 1 Display the main interface state and subinterface state. (The main interface is also called the trunk interface, and it is identified as trunk in some of the CLI commands.)

RP/0/RSP0/CPU0:router# show interface

RP/0/RSP0/CPU0:router# show running-config interface

RP/0/RSP0/CPU0:router# show ethernet trunk

• Verify that the interfaces and subinterfaces are up or down as expected.

• Run this command a second time to verify that counters are being incremented.

• Verify that the port settings (for example, MTU, duplex) are as expected.

• Verify that traffic is being directed to the correct subinterfaces. If it is not, the configuration of the classification might be incorrect.

• Verify that there is no traffic running on the main (trunk) interface; traffic that is misclassified might default to run on the main interface.

• Verify that the encapsulations match what you expect on the subinterfaces.

• Use the interface statistics for the subinterface to determine whether packets are being demultiplexed to the correct subinterface. Use the interface statistics on the parent physical/bundle interface to determine whether traffic is being sent/received out of the trunk port. The Layer 2 statistics for the physical/bundle interface sum over all of the child/subinterfaces.

The counters on the main interface count packets as they are sent/received physically on the wire. On the other hand, the subinterface counters are located in the forwarding engine.

• Check the interface packet drop counters to determine if packets are being dropped and if they are, where and why.

Step 2 Display the state of interface as recognized by the L2VPN object. Verify that L2VPN packets are being forwarded on interface and subinterface (if applicable).

RP/0/RSP0/CPU0:router# show l2vpn forwarding interface gigabitEthernet interface-id hardware ingress location node-id

Step 3 Display the Ethernet tags and check for any errors or mismatches. This command gives tag information in a very concise format, if you want to check the encapsulation on multiple subinterfaces.

RP/0/RSP0/CPU0:router# show ethernet tags


OL-23591-02


Step 4 Verify that the subinterface matching order is as expected. The match-order option lists the subinterfaces in the order that they match traffic. If the traffic is being classified to a different interface than you expect, this command can help you determine why.

RP/0/RSP0/CPU0:router# show ethernet tags match-order

Step 5 Display the interface debug counters for each network processor unit. The following example shows the NP counters.

RP/0/RSP0/CPU0:router# show controllers np counters {all | np0 | np1 | np2 | np3}

Step 6 If the output of the command in Step 5 shows that the UIDB_TCAM_MISS_AGG_DROP counter is incrementing, it is possible that the physical port is receiving tagged traffic that does not match the encapsulation statement of any subinterface. The parent/main interface is an untagged Layer 3 interface, and rejects any tagged traffic that fails classification against any of its subinterfaces/children.

RP/0/RSP0/CPU0:router# clear controllers np counters all location node-idRP/0/RSP0/CPU0:router# show controllers np counters {all | np0 | np1 | np2 | np3}

a. Verify that there is incoming tagged traffic that does not match the encapsulation statement of any subinterface, and that this traffic is not needed (that is, you do not intend to configure a subinterface to receive and forward this traffic).

– Encapsulation not matched but the traffic is needed—Create the necessary subinterface or correct the encapsulation statement on the applicable existing subinterface.

– Encapsulation not matched, traffic not needed, and no encapsulation default currently configured—Go to Substep b.

– Encapsulation not matched, traffic not needed, and there is an encapsulation default currently configured—Go to Substep c.

b. Add an encapsulation default subinterface to receive all of the tagged traffic with unwanted encapsulation statements. Check whether the UIDB_TCAM_MISS_AGG_DROP goes to zero, and the default subinterface counters start going up. This process shifts the incrementing of counters away from the main interface and isolates it on the default subinterface.

c. Verify that the Layer 2 encapsulation default subinterface is properly configured.

Note See the example below with the CLI statement encapsulation default.

Example

In this example, the system displays information on the subinterface 0/0/0/0.1.

RP/0/RSP0/CPU0:router# show running-config interfaceinterface GigabitEthernet0/0/0/0.1 l2transport encapsulation dot1q 10!interface GigabitEthernet0/0/0/0.2 l2transport encapsulation dot1q 10 second-dot1q 20...

RP/0/0/CPU0:router# show interfaces GigabitEthernet 0/0/0/0.1GigabitEthernet0/0/0/0.1 is up, line protocol is up <<< This subinterface is up Interface state transitions: 1


OL-23591-02


Hardware is VLAN sub-interface(s), address is 02fe.08cb.26c5 Layer 2 Transport Mode MTU 1518 bytes, BW 1000000 Kbit (Max: 1000000 Kbit) reliability Unknown, txload Unknown, rxload Unknown Encapsulation 802.1Q Virtual LAN, <<< Encapsulation is correct Outer Match: Dot1Q VLAN 10 <<< Encapsulation Ethertype Any, MAC Match src any, dest any<<< Encapsulation loopback not set, ARP type ARPA, ARP timeout 04:00:00 Last input never, output never Last clearing of "show interface" counters never 1400 packets input, 2800 bytes 7000 input drops, 8400 queue drops, 9800 input errors 4200 packets output, 5600 bytes 11200 output drops, 12600 queue drops, 14000 output errors

In this example, Bundle-Ether16 is the main interface (also referred to as the trunk interface or Layer 3 interface), and Bundle-Ether16.160 and Bundle-Ether16.161 are subinterfaces.

RP/0/RSP0/CPU0:router# show interfaces Bundle-Ether16 is up, line protocol is up <<< The main interface is up Interface state transitions: 1 Hardware is Aggregated Ethernet interface(s), address is 001b.53ff.87f0 Description: Connect to P19_C7609-S Port-Ch 16 Internet address is Unknown MTU 9216 bytes, BW 1000000 Kbit (Max: 1000000 Kbit) reliability 255/255, txload 0/255, rxload 0/255 Encapsulation ARPA, loopback not set, ARP type ARPA, ARP timeout 04:00:00 No. of members in this bundle: 2 GigabitEthernet0/1/0/16 Full-duplex 1000Mb/s Active GigabitEthernet0/1/0/17 Full-duplex 1000Mb/s Standby Last input 00:00:00, output 00:00:00 Last clearing of "show interface" counters never 5 minute input rate 0 bits/sec, 0 packets/sec 5 minute output rate 1000 bits/sec, 0 packets/sec 202037 packets input, 18079605 bytes, 1 total input drops <<< Includes the sum of packets on all the subinterfaces in addition to the packets on the main interface. 5964 drops for unrecognized upper-level protocol Received 0 broadcast packets, 202037 multicast packets 0 runts, 0 giants, 0 throttles, 0 parity 0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort 490241 packets output, 53719536 bytes, 0 total output drops Output 3 broadcast packets, 490238 multicast packets 0 output errors, 0 underruns, 0 applique, 0 resets 0 output buffer failures, 0 output buffers swapped out 0 carrier transitions

Bundle-Ether16.160 is up, line protocol is up <<< This subinterface is upInterface state transitions: 1 Hardware is VLAN sub-interface(s), address is 001b.53ff.87f0 Description: Connect to P19_C7609-S Port-Ch 16 Service Instance 160 Layer 2 Transport Mode MTU 9220 bytes, BW 1000000 Kbit (Max: 1000000 Kbit) reliability Unknown, txload Unknown, rxload Unknown Encapsulation 802.1Q, loopback not set, <<< Encapsulation is correct ARP type ARPA, ARP timeout 04:00:00 Last input never, output never Last clearing of "show interface" counters never 5425 packets input, 368952 bytes <<< Traffic is present on this subinterface 1 input drops, 0 queue drops, 0 input errors 161269 packets output, 11611364 bytes 0 output drops, 0 queue drops, 0 output errors


OL-23591-02


Bundle-Ether16.161 is up, line protocol is up <<< This subinterface is upInterface state transitions: 1 Hardware is VLAN sub-interface(s), address is 001b.53ff.87f0 Description: Connect to P19_C7609-S Port-Ch 16 Service Instance 161 Layer 2 Transport Mode --More--

RP/0/RSP0/CPU0:router# show running-config interface

interface Bundle-Ether16 description Connect to C7609-S Port-Ch 16 mtu 9216 bundle maximum-active links 1!interface Bundle-Ether16.160 l2transport description Connect to C7609-S Port-Ch 16 Service Instance 160 encapsulation dot1q 160 <<< Encapsulation is correct!interface Bundle-Ether16.161 l2transport description Connect to C7609-S Port-Ch 16 Service Instance 161 encapsulation dot1q 161!interface Bundle-Ether16.162 description Connect to C7609-S Port-Ch 16.162 ipv4 address 192.0.2.44 255.255.255.0 encapsulation dot1q 162!interface Bundle-Ether16.163 description Connect to C7609-S Port-Ch 16.163 ipv4 address 192.0.2.44 255.255.255.0 encapsulation dot1q 163!interface Loopback0 --More--

RP/0/RSP0/CPU0:router# show ethernet trunk Trunk Sub types Sub statesInterface St Ly MTU Subs L2 L3 Up Down Ad-DownBE16 Up L3 9216 4 2 2 4 0 0Gi0/1/0/3 Up L3 9014 5 5 0 5 0 0Gi0/1/0/7 Up L3 9014 6 6 0 6 0 0Gi0/1/0/19 Up L3 9014 2 2 0 2 0 0Gi0/1/0/20 Up L3 9014 1 1 0 1 0 0Gi0/1/0/30 Up L3 9014 1 1 0 1 0 0

Summary 19 17 2 19 0 0

The following example shows the NP counters. For a description of how to interpret NP counter information, see the “Displaying Traffic Status in Line Cards and RSP Cards” section on page 7-147.

Note If you want to clear counters at any time during this procedure (to make it easier to see which counters are incrementing), use the command clear controllers np counters all location node-id.

RP/0/RSP0/CPU0:router# show controllers np counters all Fri Oct 29 10:49:57.377 DST

Node: 0/0/CPU0:----------------------------------------------------------------


OL-23591-02



Read 17 non-zero NP counters:Offset Counter FrameValue Rate (pps)------------------------------------------------------------------------------- 23 PARSE_FABRIC_RECEIVE_CNT 189232 0 34 RESOLVE_EGRESS_DROP_CNT 165012 0 53 MODIFY_FRAMES_PADDED_CNT 175313 0 67 PARSE_MOFRR_SWITCH_MSG_RCVD_FROM_FAB 4158 0 70 RESOLVE_INGRESS_L2_PUNT_CNT 48244 0 74 RESOLVE_LEARN_FROM_NOTIFY_CNT 160848 0 75 RESOLVE_BD_FLUSH_DELETE_CNT 10804 0 87 RESOLVE_MOFRR_SWITCH_MSG_INGNORED 4158 0 111 DIAGS 24024 0 223 PUNT_STATISTICS 1193133 1 224 PUNT_STATISTICS_EXCD 1 0 225 PUNT_DIAGS_RSP_ACT 24220 0 468 RESOLVE_MAC_NOTIFY_CTRL_DROP_CNT 160854 0 600 PARSE_FAB_MACN_RECEIVE_CNT 160853 0 601 PARSE_FAB_DEST_MACN_RECEIVE_CNT 1 0 --More--

This example shows that L2VPN packets are being forwarded on the interface and subinterface (if applicable).

RP/0/RSP0/CPU0:router# show running-config l2vpnl2vpn bridge group BG bridge-domain BD1 interface TenGigE0/1/0/0.0 ! interface TenGigE0/1/0/3.0 ! interface TenGigE0/1/0/4.0 ! neighbor 10.100.1.1 pw-id 2 ! ! !! RP/0/RSP0/CPU0:router# show l2vpn forwarding interface Te0/1/0/0.0 detail loc 0/1/cpu0Local interface: TenGigE0/1/0/0.0, Xconnect id: 0x440003, Status: up Segment 1 AC, TenGigE0/1/0/0.0, status: Bound Statistics: packets: received 55749484, sent 1 bytes: received 3567966976, sent 42 packets dropped: PLU 0, tail 0 bytes dropped: PLU 0, tail 0 Segment 2 Bridge id: 0, Split horizon group id: 0 Storm control: disabled MAC learning: enabled MAC port down flush: enabled Flooding: Broadcast & Multicast: enabled Unknown unicast: enabled MAC aging time: 300 s, Type: inactivity MAC limit: 4000, Action: none, Notification: none MAC limit reached: no MAC Secure: disabled, Logging: disabled DHCPv4 snooping: profile not known on this node, disabled Dynamic ARP Inspection: disabled, Logging: disabled


OL-23591-02


IP Source Guard: disabled, Logging: disabled IGMP snooping profile: profile not known on this node Router guard disabled...Xconnect id: 0xfffc0001, Status: down Segment 1 MPLS, Destination address: 210.100.1.1, pw-id: 2, status: Not bound Pseudowire label: UNKNOWN Control word disabled Statistics: packets: received 0, sent 0 bytes: received 0, sent 0 packets dropped: PLU 0, tail 0, out of order 0 bytes dropped: PLU 0, tail 0, out of order 0 Segment 2 Bridge id: 0, Split horizon group id: 0 Storm control: disabled MAC learning: enabled MAC port down flush: enabled Flooding: Broadcast & Multicast: enabled Unknown unicast: enabled MAC aging time: 300 s, Type: inactivity MAC limit: 4000, Action: none, Notification: none MAC limit reached: no MAC Secure: disabled, Logging: disabled DHCPv4 snooping: profile not known on this node, disabled Dynamic ARP Inspection: disabled, Logging: disabled IP Source Guard: disabled, Logging: disabled IGMP snooping profile: profile not known on this node Router guard disabled

This example displays detailed tag information for multiple subinterfaces.

RP/0/0/CPU0:router# show ethernet tags St: AD - Administratively Down, Dn - Down, Up - UpLy: L2 - Switched layer 2 service, L3 = Terminated layer 3 service,Xtra C - Match on Cos, E - Match on Ethertype, M - Match on source MAC-,+: Ingress rewrite operation; number of tags to pop and push respectively

Interface St MTU Ly Outer Inner Xtra -,+Gi0/0/0/0.1 Up 1518 L2 .1Q:10 - - 0 0Gi0/0/0/0.2 Up 1522 L2 .1Q:10 .1Q:20 - 0 0

This example shows the configuration and query of the Ethernet tags.

RP/0/RSP0/CPU0:router# show run interface gig0/0/0/0.1Thu Oct 14 08:57:16.831 EDTinterface GigabitEthernet0/0/0/0.1 l2transport encapsulation dot1q 1!

RP/0/RSP0/CPU0:router# show ethernet tags gigabitEthernet 0/0/0/0.1 detail location 0/0/CPU0GigabitEthernet0/0/0/0.1 is up, service is L2 Interface MTU is 1518, switched L2 MTU is 1518 Outer Match: Dot1Q VLAN 1 Local traffic encap: Dot1Q VLAN 1 Pop 0 tags, push none

In this example, 0.2 is listed before 0.1. Any traffic with outer VLAN .1Q 10, and inner tag .1Q 20 would match Gi0/0/0/0.2.


OL-23591-02


RP/0/0/CPU0:router# show ethernet tags match-orderSt: AD - Administratively Down, Dn - Down, Up - UpLy: L2 - Switched layer 2 service, L3 = Terminated layer 3 service,Xtra C - Match on Cos, E - Match on Ethertype, M - Match on source MAC-,+: Ingress rewrite operation; number of tags to pop and push respectively

Interface St MTU Ly Outer Inner Xtra -,+Gi0/0/0/0.2 Up 1522 L2 .1Q:10 .1Q:20 - 0 0Gi0/0/0/0.1 Up 1518 L2 .1Q:10 - - 0 0

This example displays the VFI statistics.

RP/0/RSP0/CPU0:router# show l2vpn bridge-domain detail

Bridge group: 189, bridge-domain: 189, id: 0, state: up, ShgId: 0, MSTi: 0 MAC learning: enabled MAC withdraw: enabled MAC withdraw for Access PW: enabled Flooding: Broadcast & Multicast: enabled Unknown unicast: enabled MAC aging time: 300 s, Type: inactivity MAC limit: 4000, Action: none, Notification: syslog MAC limit reached: no MAC port down flush: enabled MAC Secure: disabled, Logging: disabled Split Horizon Group: none Dynamic ARP Inspection: disabled, Logging: disabled IP Source Guard: disabled, Logging: disabled DHCPv4 snooping: disabled IGMP Snooping profile: none Bridge MTU: 9000 MIB cvplsConfigIndex: 1 Filter MAC addresses: Create time: 22/09/2010 04:16:14 (2w4d ago) No status change since creation ACs: 2 (2 up), VFIs: 0, PWs: 0 (0 up), PBBs: 0 (0 up) List of ACs: AC: GigabitEthernet0/1/0/3.189, state is up...List of VFIs: VFI 190 PW: neighbor 10.19.19.19, PW ID 190, state is up ( established ) PW class Use_Tu-44190, XC ID 0xfffc0003 Encapsulation MPLS, protocol LDP PW type Ethernet, control word disabled, interworking none PW backup disable delay 0 sec Sequencing not set

Preferred path tunnel TE 44190, fallback disabled MPLS Local Remote ------------ ------------------------------ ------------------------- Label 16002 101 Group ID 0x1 0x0 Interface 190 unknown MTU 1998 1998 Control word disabled disabled PW type Ethernet Ethernet VCCV CV type 0x2 0x6 (LSP ping verification) (LSP ping verification) (BFD PW FD only) VCCV CC type 0x6 0x6


OL-23591-02

Chapter 9 Troubleshooting L2VPN and Ethernet Services Troubleshooting Multipoint Layer 2 Services

(router alert label) (router alert label) (TTL expiry) (TTL expiry) ------------ ------------------------------ ------------------------- MIB cpwVcIndex: 4294705155 Create time: 22/09/2010 04:16:14 (2w4d ago) Last time status changed: 22/09/2010 04:21:04 (2w4d ago) MAC withdraw message: send 0 receive 0 Static MAC addresses: Statistics: packets: received 849493, sent 2 bytes: received 54153872, sent 120 DHCPv4 snooping: disabled IGMP Snooping profile: none VFI Statistics: drops: illegal VLAN 0, illegal length 0

This example shows how to set up an encapsulation default subinterface. in this scenario, you expect incoming traffic on gig0/1/0/1 to be all single-tagged dot1q 100. However, you see some occasional traffic with other encapsulations being dropped. These drops could be due to a few stray packets (for example dot1q 200), and they are dropped without being processed on gig0/1/0/1; the UIDB_TCAM_MISS_AGG_DROP counter is incremented. You can configure one default subinterface to catch all the stray packets. Then the drops appear as counters on this isolated default interface, not as UIDB_TCAM_MISS_AGG_DROP on the main interface.

interface gig0/1/0/1 mtu 1500!interface gig0/1/0/1.1 l2transport encapsulation dot1q 100!interface gig0/1/0/1.2 l2transport encapsulation default <=== encapsulation default!

Troubleshooting Multipoint Layer 2 ServicesThis section explains how to troubleshoot multipoint Layer 2 services, and includes these topics:

• Basic Bridging: Example, page 9-190

• Verifying MAC Address Updates, page 9-192

• Troubleshooting Multipoint Layer 2 Bridging Services (VPLS), page 9-195

• Troubleshooting Bridge Domains That Use BGP-AD, page 9-201

Basic Bridging: ExampleFigure 9-1 shows an example of a bridge domain configuration. The configuration commands are listed below the drawing. Make sure that your own configuration is consistent with the applicable CLI structure and syntax shown in this example.


OL-23591-02


Figure 9-1 Example of Bridge Domain Configuration

The configuration for Figure 9-1 is as follows.

interface GigabitEthernet0/1/0/1 l2transport!interface GigabitEthernet0/2/0/2!interface GigabitEthernet0/2/0/2.2 l2transport encapsulation dot1q 100!interface GigabitEthernet0/5/0/8 bundle id 1 mode active!interface GigabitEthernet0/5/0/9 bundle id 1 mode active!interface Bundle-Ether1!interface Bundle-Ether1.1 l2transport encapsulation dot1q 100!l2vpn bridge group bg_example bridge-domain mybd interface GigabitEthernet0/1/0/1 ! interface GigabitEthernet0/2/0/2.2 ! interface Bundle-ether1.1 ! ! !!

Use show commands to display the status of the network.

Step 1 Verify that bundle members Gig0/5/0/8 and Gig0/5/0/9 are both Active, that is, that Link Aggregation Control Protocol (LACP) indicates that they are connected with their adjacent neighbors.

RP/0/RSP0/CPU0:router# show bundle bundle-ether1

2550

23

Bridge domain “mybd”

Routergig0/1/0/1

Bridge port 1gig0/1/0/1

gig0/2/0/2

Bridge port 2gig0/2/0/2.2

EFPs

Bridge port 3bundle-ether1.1

gig0/5/0/8

gig0/5/0/9

EFPs

bundle-ether1


OL-23591-02


Note For additional information on troubleshooting bundles and LACP, see the “Troubleshooting Problems with Link Bundles” section on page 5-118.

Step 2 Follow the steps in the “Troubleshooting VLAN Traffic and L2 TCAM Classification” section on page 9-181 for the ACs—Gig0/1/0/1, Gig0/2/0/2, and Bundle-ether1.1.

Step 3 Display the bridge domain running configuration and ensure that it contains the appropriate commands for your network.

RP/0/RSP0/CPU0:router# show run l2vpn bridge group bg_example

Step 4 Verify that the bridge domain, bridge ports, and ACs are all in Up state.

RP/0/RSP0/CPU0:router# show l2vpn bridge-domain bd-name mybd

Step 5 View additional details of the bridge domain, such as the feature settings and verify they are as expected.

RP/0/RSP0/CPU0:router# show l2vpn bridge-domain bd-name mybd detail

Verifying MAC Address UpdatesThis section explains how to determine whether MAC addresses are being flooded, learned and updated, which are all prerequisites for traffic to be switched properly on the bridge domain. Even if traffic is flowing, you need to verify that the system is continuing to flood, learn, and update MAC address information appropriately.

You can track MAC learning on a specific MAC address for a node that could be several hops away. This information helps you evaluate the health of the network:

• Determine whether a source MAC address been learned on a specific bridge domain.

• Determine the specific bridge port on which the source MAC address was learned (either a PW or an AC), and provide information about the status of that bridge port.

• View the age timer on the learned MAC address, which is a statistic on the traffic stream. The system periodically checks that it is updating learned MAC addresses, and, if it is updating MAC addresses successfully, the system restarts the age timer at the initial value (0). This reset occurs at the half-age time, and the system sends a MAC update notification. If the configured maximum time elapses (default 5 minutes) without an update, the MAC address ages out, which means there is no communication and traffic is not getting through.

To find out whether a MAC address is being learned, monitor the age repeatedly, for example, every 10 seconds for five iterations. If the MAC age continues to increment beyond the half-age time, it means there is no traffic flowing during the time you monitored it.

Step 1 Display the MAC address table for the bridge domain. Verify that MAC addresses are being learned and resynced. Include the specific bridge domain and MAC address of interest, so the output will display the specific bridge-port (AC or PW) on which the specific MAC address was learned.

RP/0/RSP0/CPU0:router# show l2vpn forwarding bridge-domain bridge-group:bridge-domain mac-address mac-address-id location node-id

If the MAC address was learned on a PW, the output shows the IP address of the neighbor. Otherwise it shows the MAC address of the AC.


OL-23591-02


A bridge domain is an entity that exists on multiple LCs. However, the show command singles out one LC. If the MAC address was learned on a bridge-port on a different LC, the display output reports the LC on which it was learned—not the actual bridge-port. To get the bridge-port data, rerun the command on the actual LC on which it was learned.

Step 2 (Optional) As an alternative to the procedure in Step 1, you can run a more general command without specifying a specific bridge domain or MAC address. However, the output could flood your terminal screen.

Caution Before you run this command without specifying a particular bridge domain and MAC address, take steps to limit the amount of data that can be output on your terminal screen. Otherwise the amount of output could be extremely large.

This command displays all the MAC addresses learned on all bridge domains. As a safety mechanism, before you enter this command, set your terminal length, for example:

RP/0/RSP0/CPU0:router# term length 20

If you need the full display, direct the output to a file, for example:

RP/0/RSP0/CPU0:router# loc 0/6/cpu0 | file disk0:bdoutput.txt

RP/0/RSP0/CPU0:router# show l2vpn forwarding bridge-domain mac-address location node-id

Step 3 Display the MAC table for the bridge domain and verify that the MAC address has been learned. Notice the bridge port (the same as the attachment circuit [AC]) from which the MAC address was learned, and whether it was learned through a pseudowire (PW).

Caution Before you run this command without specifying a MAC address ID, take steps to limit the amount of data that can be output on your terminal screen. Otherwise the amount of output could be extremely large.

This command displays all the MAC addresses learned on a bridge domain. As a safety mechanism, before you enter this command, set your terminal length, for example:




One other approach to limit the output is to run the command with a pipe filter and CTRL-C after you see the output you want.

RP/0/RSP0/CPU0:router# show l2vpn forwarding bridge-domain bridge-group:bridge-domain mac-address detail location node-id [ | begin GigabitEthernet interface-id ]

Step 4 Use the following command to display the data for a specific bridge domain and MAC address.

RP/0/RSP0/CPU0:router# show l2vpn forwarding bridge-domain bridge-group:bridge-domain mac-address mac-address detail location node-id

Example



OL-23591-02


RP/0/RSP0/CPU0:router# show l2vpn forwarding bridge-domain mac-address location 0/6/CPU0

Mac Address Type Learned from/Filtered on LC learned Resync Age Mapped to -----------------------------------------------------------------------------------------0000.0001.0101 dynamic Gi0/6/0/1.1 0/6/CPU0 0d 0h 1m 59s N/A 0000.0001.0102 dynamic Gi0/6/0/1.1 0/6/CPU0 0d 0h 1m 59s N/A 0000.0002.0202 dynamic (192.0.2.20, 1:101) 0/6/CPU0 0d 0h 1m 59s N/A 0000.0003.0303 dynamic (192.0.2.40, 1:101) 0/6/CPU0 0d 0h 1m 59s N/A

RP/0/RSP0/CPU0:router# show l2vpn forwarding bridge-domain test:test mac-address 0000.9999.9999 detail location 0/5/CPU0

Bridge-domain name: test:test, id: 0, state: up MAC learning: enabled Flooding: Broadcast & Multicast: enabled Unknown unicast: enabledNumber of bridge ports: 2 Number of MAC addresses: 1

GigabitEthernet0/5/0/17.60, state: oper up Number of MAC: 1Mac Address: 0000.9999.9999, LC learned: 0/5/CPU0 <<< MAC is learned Age: 0d 0h 0m 7s, Flag: local

RP/0/RSP0/CPU0:router# show l2vpn forwarding bridge-domain testgrp:testbr mac-address 0000.8888.8888 detail location 0/5/cpu0

Bridge-domain name: testgrp:testbr, id: 0, state: up MAC learning: enabled Flooding: Broadcast & Multicast: enabled Unknown unicast: enabled MAC aging time: 300 s, Type: inactivity MAC limit: 4000, Action: none, Notification: syslog MAC limit reached: no Security: disabled DHCPv4 snooping: profile not known on this node IGMP snooping: disabled, flooding: disabled Bridge MTU: 1500 bytes Number of bridge ports: 2 Number of MAC addresses: 2 Multi-spanning tree instance: 0

GigabitEthernet0/5/0/17.60, state: oper up Number of MAC: 1 Sent(Packets/Bytes): 8000/800000 Received(Packets/Bytes): 27000/2700000 Storm control drop counters: Broadcast(Packets/Bytes): 0/0 Multicast(Packets/Bytes): 0/0 Unknown unicast(Packets/Bytes): 0/0

Nbor 8.8.8.8 pw-id 98 <<< MAC is learned on a pseudowire Number of MAC: 1 Sent(Packets/Bytes): 27000/2592000 Received(Packets/Bytes): 8000/768000 Storm control drop counters: Broadcast(Packets/Bytes): 0/0 Multicast(Packets/Bytes): 0/0 Unknown unicast(Packets/Bytes): 0/0 Mac Address: 0000.8888.8888, LC learned: 0/5/CPU0 Age: 0d 0h 0m 10s, Flag: local


OL-23591-02


Troubleshooting Multipoint Layer 2 Bridging Services (VPLS)This section provides information on troubleshooting multipoint Layer 2 bridging services, also called virtual private LAN services (VPLS) on the Cisco ASR 9000 Aggregation Services Router. VPLS enables geographically separated local-area network (LAN) segments to be interconnected as a single bridged domain over an MPLS network and provides transparent multipoint Layer 2 connectivity for customers.

This section contains the following topics:

• Understanding VPLS Architecture, page 9-195

• Verifying MPLS PIE Activation, MPLS Configuration, and MPLS Connectivity, page 9-196

• Procedure for Troubleshooting Multipoint Layer 2 Services, page 9-196

• Example of Point-To-Point Layer 2 Deployment, page 9-206

Understanding VPLS Architecture

The VPLS architecture allows end-to-end connection between provider edge (PE) routers, which supports delivery of multipoint Ethernet services. Without VPLS, end-to-end connectivity between PE routers is achieved by creating a full-mesh of real connections between each PE router. With VPLS, as shown in Figure 9-2, the full mesh of real connections is replaced by a full mesh of virtual (pseudowire) connections. In this example, the interconnections between the network provider edge (N-PE) nodes are made by means of pseudowires (PWs) through an IP/MPLS core network. The PWs can be created either through manual configuration or autodiscovery.

Figure 9-2 is a partial implementation of a VPLS architecture. In a full VPLS architercture (not shown here), the full mesh of pseudowires is replaced by a combination of pseudowires and one or more bridge domains in the P core network. Each PE router would have a single PW connecting the router to a P router in the core. This core P router would have a bridge domain, and this bridge domain would terminate all PE router PWs. This would replace the full mesh of Figure 9-2 with a hub-and-spoke, the hub being the bridge domain in the P router.

Figure 9-2 Example of VPLS Architecture with Pseudowires in MPLS Core

The VPLS network requires the creation of a bridge domain (Layer 2 broadcast domain) on each of the PE routers. The VPLS PE device holds all the VPLS forwarding MAC tables and bridge domain information. In addition, it is responsible for all flooding broadcast frames and multicast replications.

N-PE N-PEMPLS Core CECE

Ethernet(VLAN/Port/EFP)

Ethernet(VLAN/Port/EFP)

Attachment circuit Attachment circuit

Full Mesh PWs + LDP

2086

84


OL-23591-02


Verifying MPLS PIE Activation, MPLS Configuration, and MPLS Connectivity

This section is applicable to operation of multipoint Layer 2 services over PWs. For PWs to function, the MPLS PIE must be active and MPLS must be present in your running configuration:

• Verify that the MPLS PIE is installed, committed, and activated. It is not installed by default.

• Verify that MPLS is configured in your running-config. After you install the MPLS PIE, you must commit it. If you configure MPLS but you have not committed the MPLS PIE, the system deletes all of your MPLS configuration if you reload the router image.

Caution Verify that the MPLS PIE is committed before you configure MPLS. Otherwise all of your MPLS configuration data will be lost if the image is reloaded.

PWs operate over the MPLS network, therefore, MPLS connectivity is a prerequite for bringing up a PW. To verify MPLS connectivity, see the “Troubleshooting Connectivity Over MPLS” section on page 8-174.

Procedure for Troubleshooting Multipoint Layer 2 Services

Perform these steps if you are having connectivity problems with Layer 2 multipoint services.

Step 1 Check for the following underlying problems, which can cause failure of the multipoint Layer 2 services.

• The bridge domain uses an attachment circuit (AC) for which the interfaces have not been created.

• The AC interface for the bridge domain is operationally down.

• The AC interface for the bridge domain is administratively down.

• The AC is not configured as Layer 2 (the l2transport keyword is missing from the configuration command).

• The traffic on the AC interface is not classified properly (wrong encapsulation statement).

• There is an MTU mismatch between the local and remote routers.

Step 2 Verify that you can ping the opposite interface (on the remote router) from the MPLS interface.

Step 3 Verify that the remote interface shows up as an ospf neighbor.

show ospf neighbor

Step 4 Verify that the remote router ID, typically the remote router loopback, is in the routing table.

show route ipv4

Step 5 Ping the remote router with the same IP address that is used for the PW (ping x.x.x.x).

Step 6 Verify that you can find the remote router ID in an MPLS command. It should be the ipv4 address for the PW.

Step 7 Verify that the BGP neighbor is up. (This step is necessary only if BGP autodiscovery has been configured.)

show bgp neighbors

Step 8 Verify that the VFI is advertized in both PEs, and that PWs are established.

show l2vpn bridge-domain [brief | detail]


OL-23591-02


Step 9 Check local and remote labels.

show mpls forwarding [labels]

show l2vpn forwarding detail location

Step 10 Verify that PWs are bound in the Layer 2 forwarding information base (L2FIB) with the proper cross-connect ID.

show l2vpn forwarding detail location

Step 11 Verify that NLRIs are received and PWs created.

show l2vpn discovery [summary]

Example

The following example shows that autodiscovery is on, the PW is up, and NLRIs have been received from the peer router. Check the cross-connect ID. Check the local and remote label and compare with the label binding in the MPLS label switching database (LSD) by means of the show mpls forwarding command. In this example, the local MPLS label ID is 16005.

RP/0/RSP0/CPU0:router# show l2vpn bridge-domain detail Bridge group: bg1, bridge-domain: bg1_bd1, id: 0, state: up, ShgId: 0, MSTi: 0 MAC learning: enabled MAC withdraw: enabled Flooding: Broadcast & Multicast: enabled Unknown unicast: enabled MAC aging time: 300 s, Type: inactivity MAC limit: 4000, Action: none, Notification: syslog MAC limit reached: no MAC port down flush: enabled Security: disabled Split Horizon Group: none DHCPv4 snooping: disabled IGMP Snooping profile: none Bridge MTU: 1500ACs: 1 (1 up), VFIs: 1, PWs: 2 (2 up), PBBs: 0 (0 up) List of ACs: AC: GigabitEthernet0/6/0/1.1, state is up Type VLAN; Num Ranges: 1 VLAN ranges: [2, 2] MTU 1504; XC ID 0x2040001; interworking none MAC learning: enabled Flooding: Broadcast & Multicast: enabled Unknown unicast: enabled MAC aging time: 300 s, Type: inactivity MAC limit: 4000, Action: none, Notification: syslog MAC limit reached: no MAC port down flush: enabled Security: disabled Split Horizon Group: none DHCPv4 snooping: disabled IGMP Snooping profile: none Storm Control: disabled Static MAC addresses: Statistics: packets: received 5650000, sent 5650000 bytes: received 429400000, sent 429400000


OL-23591-02


Storm control drop counters: packets: broadcast 0, multicast 0, unknown unicast 0 bytes: broadcast 0, multicast 0, unknown unicast 0 List of Access PWs: List of VFIs: VFI bg1_bd1_vfi VPN-ID: 101, Auto Discovery: <<< BGP is provisioned, service is connected, autodiscovery is on, and IP address is advertized Route Distinguisher: 101:1 Import Route Targets: 101:1 Export Route Targets: 101:1 Signaling protocol: LDPAS Number: 1 VPLS-ID: 1:101 L2VPN Router ID: 10.10.10.10 PW: neighbor 10.20.20.20, PW ID 1:101, state is up ( established ) <<< PW is upPW class not set, XC ID 0xfffc0001 <<< cross-connect ID Encapsulation MPLS, Auto-discovered (BGP), protocol LDP PW type Ethernet, control word disabled, interworking none PW backup disable delay 0 sec Sequencing not set

MPLS Local Remote ------------ ------------------------------ ------------------------- Label 16005 16006 <<< Local and remote labels have been received, which indicates that signaling is up. The local MPLS label is 16005. BGP Peer ID 10.10.10.10 10.20.20.20 <<< Received the NLRI from the BGP peer, which means the PW is established. LDP ID 10.10.10.10 10.20.20.20 AII 10.10.10.10 10.20.20.20 AGI 1:101 1:101 Group ID 0x0 0x0 Interface bg1_bd1_vfi bg1_bd1_vfi MTU 1500 1500 Control word disabled disabled PW type Ethernet Ethernet VCCV CV type 0x2 0x2 (LSP ping verification) (LSP ping verification) VCCV CC type 0x6 0x6 (router alert label) (router alert label) (TTL expiry) (TTL expiry) ------------ ------------------------------ ------------------------- MIB cpwVcIndex: 1 Create time: 14/04/2010 23:10:51 (00:37:19 ago) Last time status changed: 14/04/2010 23:10:56 (00:37:14 ago) MAC withdraw message: send 0 receive 0 Static MAC addresses: Statistics: packets: received 2825000, sent 2825004 bytes: received 214700000, sent 214700304

RP/0/RSP0/CPU0:router# show mpls forwarding Local Outgoing Prefix Outgoing Next Hop Bytes Label Label or ID Interface Switched ------ ----------- ------------------ ------------ --------------- ------------16000 Pop 10.20.20.20/32 Gi0/6/0/21 10.0.0.2 226000292 16001 Pop 10.30.30.30/32 Gi0/6/0/3 10.0.0.2 0 16002 Pop 10.0.0.0/24 Gi0/6/0/3 10.0.0.2 0 16003 16003 10.40.40.40/32 Gi0/6/0/3 10.0.0.2 226000620 16004 Unlabelled 10.0.1.253/32 Mg0/RSP0/CPU0/0 10.2.0.4 0 16005 Pop PW(10.20.20.20:2814754062073957) \ <<< The local MPLS label is 16005.


OL-23591-02


BD=0 point2point 214700000 16006 Pop PW(10.40.40.40:2814754062073957) \ BD=0 point2point 214700000

This example displays the L2VPN fowarding parameters.

RP/0/RSP0/CPU0:router# show running-config l2vpnl2vpn bridge group BG bridge-domain BD1 interface TenGigE0/1/0/0.0 ! interface TenGigE0/1/0/3.0 ! interface TenGigE0/1/0/4.0 ! neighbor 210.100.1.1 pw-id 2 ! ! !!RP/0/RSP0/CPU0:router# show l2vpn forwarding detail location 0/1/CPU0Local interface: TenGigE0/1/0/0.0, Xconnect id: 0x440003, Status: up Segment 1 AC, TenGigE0/1/0/0.0, status: Bound Statistics: packets: received 56564799, sent 1 bytes: received 3620147136, sent 42 packets dropped: PLU 0, tail 0 bytes dropped: PLU 0, tail 0 Segment 2 Bridge id: 0, Split horizon group id: 0 Storm control: disabled MAC learning: enabled MAC port down flush: enabled Flooding: Broadcast & Multicast: enabled Unknown unicast: enabled MAC aging time: 300 s, Type: inactivity MAC limit: 4000, Action: none, Notification: none MAC limit reached: no MAC Secure: disabled, Logging: disabled DHCPv4 snooping: profile not known on this node, disabled Dynamic ARP Inspection: disabled, Logging: disabled IP Source Guard: disabled, Logging: disabled IGMP snooping profile: profile not known on this node Router guard disabledLocal interface: TenGigE0/1/0/3.0, Xconnect id: 0x440004, Status: up Segment 1 AC, TenGigE0/1/0/3.0, status: Bound Statistics: packets: received 0, sent 56573295 bytes: received 0, sent 3620839278 packets dropped: PLU 0, tail 0 bytes dropped: PLU 0, tail 0 Segment 2 Bridge id: 0, Split horizon group id: 0 Storm control: disabled MAC learning: enabled MAC port down flush: enabled Flooding: Broadcast & Multicast: enabled Unknown unicast: enabled MAC aging time: 300 s, Type: inactivity


OL-23591-02


MAC limit: 4000, Action: none, Notification: none MAC limit reached: no MAC Secure: disabled, Logging: disabled DHCPv4 snooping: profile not known on this node, disabled Dynamic ARP Inspection: disabled, Logging: disabled IP Source Guard: disabled, Logging: disabled IGMP snooping profile: profile not known on this node Router guard disabledLocal interface: TenGigE0/1/0/4.0, Xconnect id: 0x440005, Status: up Segment 1 AC, TenGigE0/1/0/4.0, status: Bound Statistics: packets: received 0, sent 56573508 bytes: received 0, sent 3620856636 packets dropped: PLU 0, tail 0 bytes dropped: PLU 0, tail 0 Segment 2 Bridge id: 0, Split horizon group id: 0 Storm control: disabled MAC learning: enabled MAC port down flush: enabled Flooding: Broadcast & Multicast: enabled Unknown unicast: enabled MAC aging time: 300 s, Type: inactivity MAC limit: 4000, Action: none, Notification: none MAC limit reached: no MAC Secure: disabled, Logging: disabled DHCPv4 snooping: profile not known on this node, disabled Dynamic ARP Inspection: disabled, Logging: disabled IP Source Guard: disabled, Logging: disabled IGMP snooping profile: profile not known on this node Router guard disabledXconnect id: 0xfffc0001, Status: down Segment 1 MPLS, Destination address: 210.100.1.1, pw-id: 2, status: Not bound Pseudowire label: UNKNOWN Control word disabled Statistics: packets: received 0, sent 0 bytes: received 0, sent 0 packets dropped: PLU 0, tail 0, out of order 0 bytes dropped: PLU 0, tail 0, out of order 0 Segment 2 Bridge id: 0, Split horizon group id: 0 Storm control: disabled MAC learning: enabled MAC port down flush: enabled Flooding: Broadcast & Multicast: enabled Unknown unicast: enabled MAC aging time: 300 s, Type: inactivity MAC limit: 4000, Action: none, Notification: none MAC limit reached: no MAC Secure: disabled, Logging: disabled DHCPv4 snooping: profile not known on this node, disabled Dynamic ARP Inspection: disabled, Logging: disabled IP Source Guard: disabled, Logging: disabled IGMP snooping profile: profile not known on this node Router guard disabled

The following example shows that BGP is connected and active, and that there are VPNs and NLRIs on the bridge domain.


OL-23591-02


RP/0/RSP0/CPU0:router# show l2vpn discovery summary BGP: connected=yes, active=yes, stdby=yesServices Bridge domain: registered=yes, Num VPNs=1 Num Local Edges=1, Num Remote Edges=2, Num Received NLRIs=2 Xconnect: registered=yes, Num VPNs=0 Num Local Edges=0, Num Remote Edges=0, Num Received NLRIs=0

The following example shows that the local router ID is advertised and that NLRIs are recieved from the remote peers.

RP/0/RSP0/CPU0:router# show l2vpn discoveryService Type: VPLS, Connected List of VPNs (1 VPNs):

Bridge group: bg1, bridge-domain: bg1_bd1, id: 0, signaling protocol: LDP VPLS-ID: 1:101 Local L2 router id: 10.10.10.10 <<< advertised List of Remote NLRI (2 NLRIs): <<< NLRIs received from the remote peer address Local Addr Remote Addr Remote L2 RID Time Created --------------- --------------- --------------- ------------------- 10.10.10.10 10.20.20.20 10.20.20.20 04/14/2010 23:10:51 10.10.10.10 10.40.40.40 10.40.40.40 04/14/2010 23:19:06

Troubleshooting Bridge Domains That Use BGP-ADPerform this procedure to verify that the configuration is correct for the features you are troubleshooting. In addition, run this procedure on all peers in the VPLS domain. (For peers that are not ASR 9000 nodes, run a procedure similar to this one to check the running configurations.)

For detailed configuration procedures, see the Cisco ASR 9000 Series Aggregation Services Routers Configuration Guides.

Step 1 Verify the configuration of BGP autodiscovery with LDP signaling.

a. Configure Loopback and Links with IP addresses.

b. Configure IGP (OSPF or ISIS)

c. Configure LDP

d. Configure BGP

e. Configure L2VPN (VPLS)

Example

####Sample Configuration from WEST:####CONFIGURE LOOPBACKs and LinksInterface loopback0Ipv4 address 10.10.10.10 255.255.255.255!Interface gig0/6/0/1.1 l2transportDescription Attachment Circuit connected to Customer siteEncapsulation dot1q 2!Interface gig0/6/0/21Description Connected to EAST NodeIpv4 address 10.0.0.1 255.255.255.0!


OL-23591-02

http://www.cisco.com/en/US/products/ps9853/products_installation_and_configuration_guides_list.html

http://www.cisco.com/en/US/products/ps9853/products_installation_and_configuration_guides_list.html


Interface gig0/6/0/3Description Connected to CENTRAL NodeIpv4 address 192.0.2.1 255.255.255.0!####CONFIGURE IGPRouter ospf 1 Router-id 10.10.10.10 Nsr Nsf cisco Area 0 interface loopback0 interface gig0/6/0/3 interface gig0/6/0/21

####CONFIGURE MPLS LDPMpls ldp graceful-restart log neighbor interface gig0/6/0/21 interface gig0/6/0/3 router-id 10.10.10.10

####CONFIGURE BGPRouter bgp 1 bgp router-id 10.10.10.10 bgp graceful-restart address-family ipv4 unicast address-family l2vpn vpls-vpws <<< This shows you have configured this family in BGP so it will be able to handle the discovery of the neighbor. ! neighbor 192.0.2.20 remote-as 1 update-source loopback0 address-family ipv4 unicast address-family l2vpn vpls-vpws neighbor 172.30.30.30 remote-as 1 update-source loopback0 address-family ipv4 unicast address-family l2vpn vpls-vpws

####CONFIGURE L2VPNl2vpn bridge group bg1 bridge-domain bg1_bd1 interface gig0/6/0/1.1 ! vfi bg1_bd1_vfi vpn-id 101 autodiscovery bgp rd 101:1 route-target 101:1 signaling-protocol ldp vpls-id 1:101

Step 2 Verify the configuration of L2VPN parameters.

a. show l2vpn atom-db

b. show l2vpn discovery summary

c. show l2vpn discovery

d. show l2vpn bridge-domain


OL-23591-02


e. show l2vpn bridge-domain brief

f. show l2vpn bridge-domain detail

Example

RP/0/RSP0/CPU0:router# show l2vpn atom-db Wed Apr 14 23:28:41.905 EDT

Peer ID VC ID Encap Signaling FEC Discovery ____________________________________________________________________________

192.0.2.20 1:101 MPLS LDP 129 BGP 192.168.40.40 1:101 MPLS LDP 129 BGP

RP/0/RSP0/CPU0:router# show l2vpn discovery summary Wed Apr 14 23:24:46.156 EDTBGP: connected=yes, active=yes, stdby=yesServices Bridge domain: registered=yes, Num VPNs=1 Num Local Edges=1, Num Remote Edges=2, Num Received NLRIs=2 Xconnect: registered=yes, Num VPNs=0 Num Local Edges=0, Num Remote Edges=0, Num Received NLRIs=0

RP/0/RSP0/CPU0:router# show l2vpn discovery Wed Apr 14 23:23:00.513 EDT

Service Type: VPLS, Connected List of VPNs (1 VPNs):

Bridge group: bg1, bridge-domain: bg1_bd1, id: 0, signaling protocol: LDP VPLS-ID: 1:101 Local L2 router id: 10.10.10.10 <<< advertized List of Remote NLRI (2 NLRIs): <<< NLRIs received from those remote peer addresses Local Addr Remote Addr Remote L2 RID Time Created --------------- --------------- --------------- ------------------- 10.10.10.10 192.0.2.20 192.0.2.20 04/14/2010 23:10:51 10.10.10.10 192.168.40.40 192.168.40.40 04/14/2010 23:19:06

RP/0/RSP0/CPU0:router# show l2vpn bridge-domain Wed Apr 14 23:46:37.190 EDTBridge group: bg1, bridge-domain: bg1_bd1, id: 0, state: up, ShgId: 0, MSTi: 0 Aging: 300 s, MAC limit: 4000, Action: none, Notification: syslog Filter MAC addresses: 0 ACs: 1 (1 up), VFIs: 1, PWs: 2 (2 up), PBBs: 0 (0 up) List of ACs: Gi0/6/0/1.1, state: up, Static MAC addresses: 0 List of Access PWs: List of VFIs: VFI bg1_bd1_vfi Neighbor 192.0.2.20 pw-id 1:101, state: up, Static MAC addresses: 0 Neighbor 192.168.40.40 pw-id 1:101, state: up, Static MAC addresses: 0

RP/0/RSP0/CPU0:router# show l2vpn bridge-domain brief Wed Apr 14 23:47:42.003 EDTBridge Group/Bridge-Domain Name ID State Num ACs/up Num PWs/up-------------------------------- ----- ---------- -------------- --------------bg1/bg1_bd1 0 up 1/1 2/2


OL-23591-02


RP/0/RSP0/CPU0:router# show l2vpn bridge-domain detail Wed Apr 14 23:48:11.152 EDTBridge group: bg1, bridge-domain: bg1_bd1, id: 0, state: up, ShgId: 0, MSTi: 0 MAC learning: enabled MAC withdraw: enabled Flooding: Broadcast & Multicast: enabled Unknown unicast: enabled MAC aging time: 300 s, Type: inactivity MAC limit: 4000, Action: none, Notification: syslog MAC limit reached: no MAC port down flush: enabled Security: disabled Split Horizon Group: none DHCPv4 snooping: disabled IGMP Snooping profile: none Bridge MTU: 1500

ACs: 1 (1 up), VFIs: 1, PWs: 2 (2 up), PBBs: 0 (0 up) List of ACs: AC: GigabitEthernet0/6/0/1.1, state is up Type VLAN; Num Ranges: 1 VLAN ranges: [2, 2] MTU 1504; XC ID 0x2040001; interworking none MAC learning: enabled Flooding: Broadcast & Multicast: enabled Unknown unicast: enabled MAC aging time: 300 s, Type: inactivity MAC limit: 4000, Action: none, Notification: syslog MAC limit reached: no MAC port down flush: enabled Security: disabled Split Horizon Group: none DHCPv4 snooping: disabled IGMP Snooping profile: none Storm Control: disabled Static MAC addresses: Statistics: packets: received 5650000, sent 5650000 bytes: received 429400000, sent 429400000 Storm control drop counters: packets: broadcast 0, multicast 0, unknown unicast 0 bytes: broadcast 0, multicast 0, unknown unicast 0 List of Access PWs: List of VFIs: VFI bg1_bd1_vfi VPN-ID: 101, Auto Discovery: BGP, state is Provisioned (Service Connected) <<< It is Advertized Route Distinguisher: 101:1 Import Route Targets: 101:1 Export Route Targets: 101:1 Signaling protocol: LDP

AS Number: 1 VPLS-ID: 1:101 L2VPN Router ID: 10.10.10.10 PW: neighbor 192.0.2.20, PW ID 1:101, state is up ( established ) <<< PW is up PW class not set, XC ID 0xfffc0001 Encapsulation MPLS, Auto-discovered (BGP), protocol LDP PW type Ethernet, control word disabled, interworking none PW backup disable delay 0 sec


OL-23591-02


Sequencing not set

MPLS Local Remote ------------ ------------------------------ ------------------------- Label 16005 16006 <<< local and remote labels have been received, which means the signaling is up. BGP Peer ID 10.10.10.10 192.0.2.20 <<< Received the NLRI, which means the PW is established. LDP ID 10.10.10.10 192.0.2.20 AII 10.10.10.10 192.0.2.20 AGI 1:101 1:101 Group ID 0x0 0x0 Interface bg1_bd1_vfi bg1_bd1_vfi MTU 1500 1500 Control word disabled disabled PW type Ethernet Ethernet VCCV CV type 0x2 0x2 (LSP ping verification) (LSP ping verification) VCCV CC type 0x6 0x6 (router alert label) (router alert label) (TTL expiry) (TTL expiry) ------------ ------------------------------ ------------------------- MIB cpwVcIndex: 1 Create time: 14/04/2010 23:10:51 (00:37:19 ago) Last time status changed: 14/04/2010 23:10:56 (00:37:14 ago) MAC withdraw message: send 0 receive 0 Static MAC addresses: Statistics: packets: received 2825000, sent 2825004 bytes: received 214700000, sent 214700304

Step 3 Verify the configuration of MPLS forwarding and Label Switching Database (LSD) parameters.

a. show mpls forwarding

b. show mpls lsd forwarding

Example

RP/0/RSP0/CPU0:router# show mpls forwarding Wed Apr 14 23:41:49.325 EDTLocal Outgoing Prefix Outgoing Next Hop Bytes Label Label or ID Interface Switched ------ ----------- ------------------ ------------ --------------- ------------16000 Pop 192.0.2.20/32 Gi0/6/0/21 10.0.0.2 226000292 16001 Pop 172.30.30.30/32 Gi0/6/0/3 192.0.2.2 0 16002 Pop 172.16.0/24 Gi0/6/0/3 192.0.2.2 0 16003 16003 192.168.40.40/32 Gi0/6/0/3 192.0.2.2 226000620 16004 Unlabelled 10.0.1.253/32 Mg0/RSP0/CPU0/0 10.2.0.4 0 16005 Pop PW(192.0.2.20:2814754062073957) \ <<< PW has label and traffic is running BD=0 point2point 214700000 16006 Pop PW(192.168.40.40:2814754062073957) \ <<< PW has label and traffic is running BD=0 point2point 214700000

RP/0/RSP0/CPU0:router# show mpls lsd forwarding Wed Apr 14 23:42:12.259 EDTIn_Label, (ID), Path_Info: <Type>16000, (IPv4, 'default':4U, 192.0.2.20/32), 1 Paths 1/1: IPv4, 'default':4U, Gi0/6/0/21, nh=10.0.0.2, lbl=3, tun_id=0 flags=(RETAIN)


OL-23591-02

Chapter 9 Troubleshooting L2VPN and Ethernet Services Troubleshooting Point-to-Point Layer 2 Services

16001, (IPv4, 'default':4U, 172.30.30.30/32), 1 Paths 1/1: IPv4, 'default':4U, Gi0/6/0/3, nh=20.0.0.2, lbl=3, tun_id=0 flags=(RETAIN) 16002, (IPv4, 'default':4U, 172.16.0.0/24), 1 Paths 1/1: IPv4, 'default':4U, Gi0/6/0/3, nh=20.0.0.2, lbl=3, tun_id=0 flags=(RETAIN) 16003, (IPv4, 'default':4U, 192.168.40.40/32), 1 Paths 1/1: IPv4, 'default':4U, Gi0/6/0/3, nh=20.0.0.2, lbl=16003, tun_id=0 flags=(RETAIN) 16004, (IPv4, 'default':4U, 10.0.1.253/32), 1 Paths 1/1: IPv4, 'default':4U, Null, nh=10.2.0.4, lbl=None, tun_id=0 flags=() 16005, (PW, (192.0.2.20:2814754062073957)), 1 Paths 1/1: PW, bridge_id=0, shg_id=1, xc_id=0xfffc0001, f=0x4, lbl=Pop-PW-Ether [Attached]16006, (PW, (192.168.40.40:2814754062073957)), 1 Paths 1/1: PW, bridge_id=0, shg_id=1, xc_id=0xfffc0002, f=0x4, lbl=Pop-PW-Ether [Attached]

Troubleshooting Point-to-Point Layer 2 ServicesThis section provides information on troubleshooting point-to-point Layer 2 services. It contains the following subsections:

• Example of Point-To-Point Layer 2 Deployment, page 9-206


• AC Is Down, page 9-218

• Pseudowire Is Down, page 9-219

• VPWS Not Forwarding Traffic from AC to Pseudowire, page 9-212

• Pseudowire Up but Ping Fails, page 9-213


• Traffic Loss During RSP Fail Over, page 9-213

• Preferred Path Not Working, page 9-214

Example of Point-To-Point Layer 2 DeploymentThis section contains an example of a point-to-point Layer 2 deployment involving a router with a bridge domain on one side of the network and a router with a cross-connect on the other. The two routers are connected by a PW. The PW is a virtual point-to-point connection between the two routers. As shown in Figure 9-3, the traffic for the PW (the virtual connection between Routers 1 and 2) passes through Router3, but Routers 1 and 2 behave as if they are directly connected over the PW.


OL-23591-02


Figure 9-3 Example of Deployment with Bridge Domain and XConnect Joined by Pseudowire

Figure 9-3 NotesRouter1 has a bridge domain (“mybd”) with three bridge ports—Two ACs and one PW:

• (AC/EFP) gig0/1/0/1.5

• (AC/EFP) gig0/1/0/2.6

• (PW) neighbor 10.2.2.2 pw-id 1

Router2 has an xconnect. The xconnect has two members—One AC and one PW. The xconect is represented by the dotted line inside the Router2 box. The dotted line also includes the AC/EFP:

• (AC/EFP) gig0/2/0/1.7

• (PW) neighbor 10.1.1.1 pw-id 1

The PW is represented by the circles (one in Router1 and one in Router2) connected by a dotted line. It is a virtual point-to-point connection from Router1 to Router2. In reality, the traffic for the PW passes through Router3, but Router1 and Router2 behave as if they are directly connected over the PW. The port at the right of Router1 and the port at the left of Router2 are the MPLS connections to Router3.

The configurations for this deployment example are as follows. Make sure that your own configuration is consistent with the applicable CLI structure and syntax shown in this example.

Router1interface GigabitEthernet0/1/0/1!interface GigabitEthernet0/1/0/1.5 l2transport encapsulation dot1q 100!interface GigabitEthernet0/1/0/2.6 l2transport encapsulation dot1q 100!interface GigabitEthernet0/1/0/3 ipv4 address 10.0.13.1 255.255.255.0!interface Loopback0 ipv4 address 10.1.1.1 255.255.255.255!router ospf 1

gig0/1/0/1

gig0/1/0/2

Bridge port #1gig0/1/0/1.5 gig0/2/0/1.7

xconnect

Bridge port #2gig0/1/0/2.6

Bridge domain“mybd”

Router1MPLS/OSPF router ID

(loopback) 10.1.1.1

Router2MPLS/OSPF router ID

(loopback) 10.2.2.2

Pseudowire

gig0/1/0/3 gig0/2/0/2

gig0/2/0/1

gig0/3/0/1 gig0/3/0/2

Router3

2819

22


OL-23591-02


log adjacency changes router-id 10.1.1.1 area 0 interface GigabitEthernet0/1/0/3 ! interface Loopback0 ! !!mpls ldp router-id 10.1.1.1 log neighbor ! interface GigabitEthernet0/1/0/3 !!l2vpn bridge group examples bridge-domain mybd interface GigabitEthernet0/1/0/1.5 ! interface GigabitEthernet0/1/0/2.6 ! neighbor 10.2.2.2 pw-id 1 ! ! !!

Router2interface GigabitEthernet0/2/0/1!interface GigabitEthernet0/2/0/1.7 l2transport encapsulation dot1q 100!interface GigabitEthernet0/2/0/2 ipv4 address 10.0.23.1 255.255.255.0!interface Loopback0 ipv4 address 10.2.2.2 255.255.255.255!router ospf 1 log adjacency changes router-id 10.2.2.2 area 0 interface GigabitEthernet0/2/0/2 ! interface Loopback0 ! !!mpls ldp router-id 10.2.2.2 log neighbor ! interface GigabitEthernet0/2/0/2 !!l2vpn xconnect group examples p2p myxc


OL-23591-02


interface GigabitEthernet0/2/0/1.7 ! neighbor 10.1.1.1 pw-id 1 ! !!

Router 3interface GigabitEthernet0/3/0/1 ipv4 address 10.0.13.2 255.255.255.0!interface GigabitEthernet0/3/0/2 ipv4 address 10.0.23.2 255.255.255.0!interface Loopback0 ipv4 address 10.3.3.3 255.255.255.255!router ospf 1 log adjacency changes router-id 10.3.3.3 area 0 interface GigabitEthernet0/3/0/1 ! interface GigabitEthernet0/3/0/2 ! interface Loopback0 ! !!mpls ldp router-id 10.3.3.3 log neighbor ! interface GigabitEthernet0/3/0/1 ! interface GigabitEthernet0/3/0/2 !!

Use the following procedure to locate any problems with traffic flow in this network. The IP addresses are based on the sample configurations for Routers 1, 2, and 3 (above).

Step 1 Verify ping connectivity over the MPLS links.

• From Router1 gig0/1/0/3 to Router3 gig0/3/0/1—ping 10.0.13.2

• From Router2 gig0/2/0/2 to Router3 gig0/3/0/2—ping 10.0.23.2

Step 2 Verify that OSPF neighbor links are up on the links (the same links listed in Step 1).


Step 3 Verify that the Router1 routing table contains the loopback address of Router2 (10.2.2.2). Also verify that the Router2 routing table contains the loopback address of Router1 (10.1.1.1).

RP/0/RSP0/CPU0:router# show route ipv4

Step 4 Verify that Router1 can ping the Router2 loopback address, and Router2 can ping the Router1 loopback address.

• From Router1—ping 10.2.2.2

• From Router2—ping 10.1.1.1


OL-23591-02


Step 5 Verify that MPLS neighbors are established in the links (the same links listed in Step 1).

RP/0/RSP0/CPU0:router# show mpls ldp neighbor

Step 6 Verify that Router1 has an MPLS label to reach the Router2 loopback address. Also verify that Router2 has an MPLS label to reach the Router1 loopback address.

Note The output of this command contains one additional MPLS label. This additional label represents the pseudowire between Router1 and Router2.

RP/0/RSP0/CPU0:router# show mpls forwarding

Step 7 Verify that the status of the Router1 bridge domain is UP, and that all all ACs are up.

RP/0/RSP0/CPU0:router# show l2vpn bridge-domain mybd

Step 8 Verify that the status of the Router1 PW is UP.

RP/0/RSP0/CPU0:router# show l2vpn bridge-domain mybd

Step 9 Verify that the Router2 xconnect is UP, and all ACs are UP.

RP/0/RSP0/CPU0:router# show l2vpn xconnect group examples

Step 10 Verify that the Router2 PW is UP.

RP/0/RSP0/CPU0:router# show l2vpn xconnect group examples


SUMMARY STEPS

1. show l2vpn xconnect [detail | group | interface | neighbor | state | summary | type | state unresolved]

2. show l2vpn forwarding {detail | hardware | interface | location | message | resource | summary | unresolved} location node-id

3. show mpls forwarding [detail | {label label number} | interface interface-id | labels value | location | prefix [network/mask | length] | summary | tunnels tunnel-id]


OL-23591-02


DETAILED STEPS

AC Is Down

Step 1 View the interface state. RP/0/RSP0/CPU0:router# show interface

Step 2 View the state of the xconnect. RP/0/RSP0/CPU0:router# show l2vpn xconnect detail

Step 3 Ensure that the AC interface has l2transport configured.

Step 4 Ensure that the AC interface is up.

Step 5 Ensure that the MTUs match. RP/0/RSP0/CPU0:router# show l2vpn bridge-domain interface type interface-name detail


Step 1 show l2vpn xconnect [detail | group | interface | neighbor | state | summary | type | state unresolved]

Example:RP/0/RSP0/CPU0:router# show l2vpn xconnect

View brief information on configured cross-connects. Filter results using the following parameters and keywords:

• detail—Detailed information

• group—All cross-connects in a specified group

• interface—Interface and subinterface

• neighbor—Neighbor

• state—Xconnect state types: up, down

• summary—AC information from the AC Manager database

• type—Xconnect types: ac-pw, locally switched

• state unresolved—Unresolved cross-connects

Step 2 show l2vpn forwarding {detail | hardware | interface | location | message | resource | summary | unresolved} location node-id

Example:RP/0/RSP0/CPU0:router# show l2vpn forwarding location 0/2/cpu0

View the matching AC subinterface.

Step 3 show mpls forwarding [detail | {label label number} | interface interface-id | labels value | location | prefix [network/mask | length] | summary | tunnels tunnel-id]

View the MPLS Label Forwarding Information Base (LFIB) entries with a local labels range.


OL-23591-02


Pseudowire Is Down

Step 1 View the pseudowire state.

RP/0/RSP0/CPU0:router# show l2vpn xconnect neighbor

Step 2 On the MPLS-enabled interface that connects to the router at the remote end of the PW, view MPLS LDP neighbor information. Check these conditions:

a. Ensure that, if the MPLS router-id uses a loopback interface (it usually does), the loopback interface is present in the OSPF configuration, so that a route to its address is advertised for the other router to reach.

b. Ensure that an LDP session is established with the PE peer.

RP/0/RSP0/CPU0:router# show mpls ldp neighbor neighbor

Step 3 Ensure that the MPLS infrastructure has allocated a label for the mpls-id IP address on the opposite router, and an additional label for the PW tunnel itself.

RP/0/RSP0/CPU0:router# show mpls forwarding

Step 4 (Perform this step if the MPLS LSP does not come up.) On the MPLS-enabled interface that connects to the router at the remote end of the PW, view OSPF neighbor information. Verify that the IP address of the MPLS router ID is reachable:

a. Ensure that this IP address appears in the routing table.

b. Ping this IP address and verify that it replies successfully.

c. Ensure that the PW ID (keyword "pw-id" in the configuration syntax) is identical on both ends of the PW.


Step 5 Ensure that pseudowires are properly configured on both PEs.

Step 6 Ensure that the MPLS package is installed.

Step 7 Ensure that the core interface is up.

Step 8 Ensure that OSPF is the routing protocol.

Step 9 Ensure that the MTUs match.

RP/0/RSP0/CPU0:router# show l2vpn xconnect neighbor

VPWS Not Forwarding Traffic from AC to PseudowireThis section provides information on troubleshooting forwarding of traffic from the AC to the PW over virtual private wire services (VPWS). VPWS connects to endpoints defined by physical interfaces or subinterfaces by emulating a virtual wire between them using the underlying MPLS technology.

Step 1 View pseudowire hardware information. RP/0/RSP0/CPU0:router# show l2vpn forwarding neighbor 192.168.12.5 pw-id 100 hardware

egress location node-id0


OL-23591-02


Step 2 View the bridge information about Broadcast, Multicast and Unknown Unicast. RP/0/RSP0/CPU0:router# show l2vpn bridge-domain bd-name 1 det

Step 3 Ensure that the MAC limit has not been exceeded. RP/0/RSP0/CPU0:router# show l2vpn forwarding bridge-domain 1:1 detail location

Step 4 Ensure that the pseudowires and AC are up.

Step 5 Verify that the hardware is programmed for both ACs.

Step 6 RP/0/RSP0/CPU0:router# show l2vpn forwarding interface GigabitEtherne0/5/0/2 hardware ingress detail location node-id

Step 7 Verify that the hardware is programmed for pseudowires.

Pseudowire Up but Ping Fails

Step 1 View the bridge domain state. RP/0/RSP0/CPU0:router# show l2vpn bridge-domain bd-name node-id detail

Step 2 Ensure that both CEs are on the same subnet.


Step 4 Ensure that the end-to-end encapsulations match.

Traffic Loss

Step 1 View the bridge domain state. RP/0/RSP0/CPU0:router# show l2vpn bridge-domain bd-name bd-name-id detail

Step 2 View segment counters to see if the packet and byte switched count increased. RP/0/RSP0/CPU0:router# show l2vpn forwarding interface GigabitEthernet node-id detail

location node-id

Step 3 Ensure that the bandwidth rates match between the CEs.

Traffic Loss During RSP Fail Over When RSP fail over is performed, some times it is seen that the traffic loss is experienced. This may be because the IGP over which the prefixes are learned is going down. The following assumes OSPF as the IGP.

• show process failover—View process details during failover


OL-23591-02

Chapter 9 Troubleshooting L2VPN and Ethernet Services Troubleshooting Specific Outage Scenarios In Layer 2 Services

• debug ospf ha—Enables OSPF HA related debugs

• debug ospf instance nsf—Before FO (Fail Over) and collect the debug log

• show process failover—After FO

Step 1 One thing to check immediately is if the next hop router also experienced an FO mechanism (Similar to what is done on this router). If so, the OSPF may go down.

Step 2 If not, verify that ‘nsf cisco’ is configured under the OSPF. If ‘nsf cisco’ is configured, see if the next hop is reachable during FO. If not, there may be a reachability issue like a link going down or negotiation problems.

Preferred Path Not Working

Step 1 View the state of the bridge domain. RP/0/RSP0/CPU0:router# show l2vpn bridge-domain bd-name bd-name-id detail

Step 2 View ingress UIDB. RP/0/RSP0/CPU0:router# show l2vpn forwarding interface interface hardware ingress detail

location node-id

Troubleshooting Specific Outage Scenarios In Layer 2 ServicesThis section contains the following topics, which help you troubleshoot specific Layer 2 outages:


• L2VPN Discovery Not Working, page 9-217

• AC Is Down, page 9-218

• Pseudowire Is Down, page 9-219

• VPLS Not Forwarding Flooding Traffic, page 9-220

• VPLS Not Forwarding Flooding Traffic from AC to Pseudowire, page 9-224

• VPLS Not Forwarding Flooding Traffic from Pseudowire to AC, page 9-224

• VPLS Not Forwarding Unicast Traffic from AC to AC, page 9-225

• VPLS Not Forwarding Unicast Traffic from AC to Pseudowire, page 9-225

• VPLS Not Forwarding Flooding Traffic from Pseudowire to AC, page 9-225

• Pseudowire Up but Ping Fails, page 9-226


• Pseudowire Flap Causing Traffic Loss, page 9-226

• Traffic Loss During RSP Fail Over, page 9-227

• Preferred Path Not Working, page 9-227


OL-23591-02



1. show l2vpn bridge-domain summary

2. show l2vpn bridge-domain [bd-name bridge-domain name | brief | detail | group bridge-domain group name | interface {type interface-id} | neighbor IP address [pw-id value] | summary]

3. show l2vpn discovery summary

4. show l2vpn forwarding bridge-domain [ bridge-domain-name] {detail | hardware {egress | ingress}} {location node-id}

DETAILED STEPS


Step 1 show l2vpn bridge-domain summary

Example:RP/0/RSP0/CPU0:router# show l2vpn bridge-domain summary

View the bridge-domain bridge-ports, which will be identified in the output as attachment circuits (ACs) and/or pseudowires (PWs) as applicable.

Verify that the bridge-domains, ACs, and PWs (as applicble) are up.

Tip Repeat this command periodically. Check that traffic counts are going up over time on the PWs and ACs in the bridge-domain.


OL-23591-02


Step 2 show l2vpn bridge-domain neighbor

show l2vpn bridge-domain group group-id

show l2vpn bridge-domain interface type node-id

Example:

RP/0/RSP0/CPU0:router# show l2vpn bridge-domain neighbor

RP/0/RSP0/CPU0:router# show l2vpn bridge-domain group 12

show l2vpn bridge-domain interface gigabitethernet 0/1/0/5

Look for the status of any bridge-domains that might be experiencing problems.

• bd-name bridge-domain name—(Optional) Displays the bridges by the bridge ID. The bridge-domain name argument is used to name a bridge domain.

• brief—(Optional) Displays brief information about the bridges.

• detail—(Optional) Displays the output for the Layer 2 VPN (L2VPN) to indicate whether or not the MAC withdrawal feature is enabled and the number of MAC withdrawal messages that are sent or received from the pseudowire.

• group bridge-domain group name—(Optional) Displays filter information on the bridge-domain group name. The bridge-domain group name argument is used to name the bridge domain group.

• interface—(Optional) Displays the filter information for the interface on the bridge domain.

• type—Interface type.

• interface-id—Identifies a physical interface or a virtual interface.

• neighbor IP address—(Optional) Displays only the bridge domain that contains the pseudowires to match the filter for the neighbor. The IP address argument is used to configure IP address of the neighbor.

• pw-id value—(Optional) Displays the filter for the pseudowire ID. The range is from 1 to 4294967295.

Step 3 show l2vpn discovery summary

Example:

RP/0/RSP0/CPU0:router# show l2vpn discovery summary

View the BGP autodiscovery status and results. This display shows the network layer reachability information (NLRI) that has been sent by the local router and received from the remote router.

Verify that BGP is active, and that the bridge domain and cross-connect are registered.



OL-23591-02


L2VPN Discovery Not Working

Step 1 Check the configuration is valid (show run l2vpn, show run bgp, show run mpls ldp).

Step 2 Check that the BGP output shows the remote prefix has been received (show bgp).

Step 3 Check L2VPN discovery to verify that the local router received the LDP NLRI update from the remote VPLS router (show l2vpn discovery private).

Example

These examples show the output from the show bgp commands.

RP/0/RSP0/CPU0:router# show bgp l2vpn vplsStatus codes: s suppressed, d damped, h history, * valid, > best i - internal, r RIB-failure, S staleOrigin codes: i - IGP, e - EGP, ? - incomplete Network Next Hop Rcvd Label Local LabelRoute Distinguisher: 101:1 (default for vrf bg1:bg1_bd1)*> 10.10.10.10/32 0.0.0.0 nolabel nolabel *>i192.0.2.20/32 192.0.2.20 nolabel nolabel *>i192.168.40.40/32 192.168.40.40 nolabel nolabel

Processed 3 prefixes, 3 paths

RP/0/RSP0/CPU0:router# show bgp l2vpn vpls rd 101:1 192.168.40.40Thu Apr 15 00:00:21.930 EDTBGP routing table entry for 10280:10280/32, Route Distinguisher: 101:1Versions: Process bRIB/RIB SendTblVer Speaker 6 6

Step 4 show l2vpn forwarding bridge-domain

Example:RP/0/RSP0/CPU0:router# show l2vpn forwarding bridge-domain ABC mac-address interface Gi0/1/2/1.2 detail hardware location 0/4/CPU0 bridge

View forwarding bridge domain information. Filter results using the following parameters and keywords:

• bridge-domain-name—(Optional) Name of a bridge domain.

• detail—Displays all the detailed information on the attachment circuits and pseudowires.

• hardware—Displays the hardware location entry.

• egress—Reads information from the egress PSE.

• ingress—Reads information from the ingress PSE.

• location node-id—Displays the bridge-domain information for the specified location.

Step 5 show l2vpn forwarding bridge-domain detail location

Example:RP/0/RSP0/CPU0:router# show l2vpn forwarding bridge-domain detail location 0/1/CPU0

View the display to see which direction is experiencing a traffic loss. If you have PWs in the core, the PWs should be in the bound state and traffic should be flowing in the bound PWs.



OL-23591-02


Last Modified: Apr 14 23:19:06.805 for 00:41:15Paths: (1 available, best #1) Not advertised to any peer Path #1: Received by speaker 0 Local 192.168.40.40 (metric 3) from 172.30.30.30 (192.168.40.40) Origin IGP, localpref 0, valid, internal, best, import-candidate, imported Extended community: RT:101:1 L2VPN AGI:1:101 Originator: 40.40.40.40, Cluster list: 30.30.30.30

This example shows the output from the show l2vpn discovery command.

RP/0/RSP0/CPU0:router# show l2vpn discovery privateService Type: VPLS, Connected List of VPNs (1 VPNs):

Bridge group: bg1, bridge-domain: bg1_bd1, id: 0, signaling protocol: LDP

AD event trace history [Total events: 3] ----------------------------------------- Time Event Status/PWID Flags/PeerID ==== ===== =============== ============ 04/14/2010 23:09:42 Add edge edge_id/type 10.10.10.10 0 04/14/2010 23:10:51 Rcv LDP nlri upd l2rid/nh 192.0.2.20 192.0.2.20 04/14/2010 23:19:06 Rcv LDP nlri upd l2rid/nh 192.168.40.40 192.168.40.40

VPLS-ID: 1:101 Local L2 router id: 10.10.10.10 List of Remote NLRI (2 NLRIs): Local Addr Remote Addr Remote L2 RID Time Created --------------- --------------- --------------- ------------------- 10.10.10.10 192.0.2.20 192.0.2.20 04/14/2010 23:10:51 10.10.10.10 192.168.40.40 192.168.40.40 04/14/2010 23:19:06

AD event trace history [Total events: 4] ----------------------------------------- Time Event Status/PWID Flags/PeerID ==== ===== =============== ============ 04/14/2010 23:09:42 Snd LDP nlri l2rid 10.10.10.10 0 04/14/2010 23:09:42 Snd refresh 0 0x0 04/14/2010 23:10:51 Upd bmgr ledge_id/nh 10.10.10.10 192.0.2.20 04/14/2010 23:19:06 Upd bmgr ledge_id/nh 10.10.10.10 192.168.40.40

AC Is Down

Step 1 RP/0/RSP0/CPU0:router# show interface

Step 2 RP/0/RSP0/CPU0:router# show l2vpn bridge interface detail

Step 3 Ensure that the AC interface has l2transport configured.

Step 4 Ensure that the AC interface is up.

Step 5 Ensure that the MTUs match. RP/0/RSP0/CPU0:router# show l2vpn bridge-domain interface type interface-name detail


OL-23591-02


Pseudowire Is DownA pseudowire (PW) is both an L2VPN component and an MPLS component. If a PW is down in one node, it could be caused by a problem in the local or remote node. Follow these steps to troubleshoot a problem with a PW in an L2VPN network.

Note For PW troubleshooting in point-to-point networks, see the “Troubleshooting Point-to-Point Layer 2 Services” section on page 9-206.

Step 1 Check the configuration is valid (show run l2vpn, show run bgp, show run mpls ldp).

Step 2 Verify that L2VPN discovery shows the received NLRI (show l2vpn discovery). If the NLRI is not received, follow the procedure in the “L2VPN Discovery Not Working” section on page 9-217.

Step 3 View the local and remote labels in the bridge-domain (show l2vpn bridge-domain detail) and compare these labels with the label binding in LSD (show mpls lsd forwarding labels). See the example below.

Step 4 View OSPF neighbor information.


Step 5 View MPLS LDP neighbor information.

RP/0/RSP0/CPU0:router# show mpls ldp neighbor neighbor

Step 6 View the bridge neighbor state.

RP/0/RSP0/CPU0:router# show l2vpn bridge-domain neighbor

Step 7 If PWs are involved, be sure they are properly configured on both PEs. See the “Troubleshooting Point-to-Point Layer 2 Services” section on page 9-206.



Step 10 Ensure that an IGP (for example OSPF) is up.

Step 11 Ensure that an LDP session is established with the PE peer.



Example

These commands allow you to view the local and remote labels in the bridge-domain and compare them with the label binding in LSD.

RP/0/RSP0/CPU0:router# show l2vpn bridge-domain detailSignaling protocol: LDP AS Number: 1 VPLS-ID: 1:101 L2VPN Router ID: 10.10.10.10 PW: neighbor 192.0.2.20, PW ID 1:101, state is up ( established ) PW class not set, XC ID 0xfffc0001 Encapsulation MPLS, Auto-discovered (BGP), protocol LDP PW type Ethernet, control word disabled, interworking none PW backup disable delay 0 sec


OL-23591-02


Sequencing not set

MPLS Local Remote ------------ ------------------------------ ------------------------- Label 16005 16006 BGP Peer ID 10.10.10.10 192.0.2.20 LDP ID 10.10.10.10 192.0.2.20 AII 10.10.10.10 192.0.2.20 AGI 1:101 1:101 Group ID 0x0 0x0 Interface bg1_bd1_vfi bg1_bd1_vfi MTU 1500 1500 Control word disabled disabled PW type Ethernet Ethernet VCCV CV type 0x2 0x2 (LSP ping verification) (LSP ping verification) VCCV CC type 0x6 0x6 (router alert label) (router alert label) (TTL expiry) (TTL expiry) ------------ ------------------------------ -------------------------

RP/0/RSP0/CPU0:router# show mpls lsd forwarding labels 16005Thu Apr 15 00:07:39.888 EDTIn_Label, (ID), Path_Info: <Type>16005, (PW, (192.0.2.20:2814754062073957)), 1 Paths 1/1: PW, bridge_id=0, shg_id=1, xc_id=0xfffc0001, f=0x4, lbl=Pop-PW-Ether [Attached]

RP/0/RSP0/CPU0:router# show mpls forwarding labels 16005 Thu Apr 15 00:09:10.067 EDTLocal Outgoing Prefix Outgoing Next Hop Bytes Label Label or ID Interface Switched ------ ----------- ------------------ ------------ --------------- ------------16005 Pop PW(192.0.2.20:2814754062073957) \ BD=0 point2point 214700000

VPLS Not Forwarding Flooding Traffic

Step 1 Check the configuration is valid (show run l2vpn, show run bgp, show run mpls ldp, show run interface).

Step 2 Make sure the PW is up and verify the XC ID.

Step 3 View the local and remote label in the bridge-domain (show l2vpn bridge-domain detail) and compare these labels with the abel binding in LSD (show mpls lsd forwarding labels). If the NLRI is not received, follow the procedure in the “L2VPN Discovery Not Working” section on page 9-217. See the example below.

Step 4 View the forwarding bridge-domain parameters (show l2vpn forwarding bridge-domain detail location) to see which direction is experiencing a traffic loss. If you have PWs in the core, the PWs should be in the bound state and traffic should be flowing in the bound PWs. See the example below.

Step 5 Display the MAC table for the bridge domain and verify that the MAC address has been learned. Notice the bridge port (the same as the attachment circuit [AC]) from which the MAC address was learned, and whether it was learned through a pseudowire (PW).


OL-23591-02


Caution Before you run this command without specifying a MAC address ID, take steps to limit the amount of data that can be output on your terminal screen. Otherwise the amount of output could be extremely large.

This command displays all the MAC addresses learned on a bridge domain. As a safety mechanism, before you enter this command, set your terminal length, for example:




One other approach to limit the output is to run the command with a pipe filter and CTRL-C after you see the output you want.

RP/0/RSP0/CPU0:router# show l2vpn forwarding bridge-domain bridge-group:bridge-domain mac-address detail location node-id [ | begin GigabitEthernet interface-id ]

Step 6 View the NP counters. Capture this output for both ingress and egress line cards. For a description of how to interpret NP counter information, see the “Displaying Traffic Status in Line Cards and RSP Cards” section on page 7-147.

RP/0/RSP0/CPU0:router# show controllers np counters all location

Step 7 View OSPF neighbor information. RP/0/RSP0/CPU0:router# show ospf neighbor

Step 8 View MPLS LDP neighbor information. RP/0/RSP0/CPU0:router# show mpls ldp neighbor neighbor

Step 9 If PWs are involved, be sure they are properly configured on both PEs. See the “Troubleshooting Point-to-Point Layer 2 Services” section on page 9-206.



Step 12 Ensure that OSPF is the routing protocol.

Step 13 Ensure that an LDP session is established with the PE peer.

Step 14 Ensure that the MTUs match. RP/0/RSP0/CPU0:router# show l2vpn bridge-domain detail

Example

These commands allow you to view the local and remote labels in the bridge-domain and compare them with the label binding in LSD.

RP/0/RSP0/CPU0:router# show l2vpn bridge-domain detailSignaling protocol: LDP AS Number: 1 VPLS-ID: 1:101 L2VPN Router ID: 10.10.10.10 PW: neighbor 192.0.2.20, PW ID 1:101, state is up ( established )


OL-23591-02


PW class not set, XC ID 0xfffc0001 Encapsulation MPLS, Auto-discovered (BGP), protocol LDP PW type Ethernet, control word disabled, interworking none PW backup disable delay 0 sec Sequencing not set

MPLS Local Remote ------------ ------------------------------ ------------------------- Label 16005 16006 BGP Peer ID 10.10.10.10 192.0.2.20 LDP ID 10.10.10.10 192.0.2.20 AII 10.10.10.10 192.0.2.20 AGI 1:101 1:101 Group ID 0x0 0x0 Interface bg1_bd1_vfi bg1_bd1_vfi MTU 1500 1500 Control word disabled disabled PW type Ethernet Ethernet VCCV CV type 0x2 0x2 (LSP ping verification) (LSP ping verification) VCCV CC type 0x6 0x6 (router alert label) (router alert label) (TTL expiry) (TTL expiry) ------------ ------------------------------ -------------------------

RP/0/RSP0/CPU0:router# show mpls lsd forwarding labels 16005Thu Apr 15 00:07:39.888 EDTIn_Label, (ID), Path_Info: <Type>16005, (PW, (192.0.2.20:2814754062073957)), 1 Paths 1/1: PW, bridge_id=0, shg_id=1, xc_id=0xfffc0001, f=0x4, lbl=Pop-PW-Ether [Attached]

RP/0/RSP0/CPU0:router# show mpls forwarding labels 16005 Thu Apr 15 00:09:10.067 EDTLocal Outgoing Prefix Outgoing Next Hop Bytes Label Label or ID Interface Switched ------ ----------- ------------------ ------------ --------------- ------------16005 Pop PW(192.0.2.20:2814754062073957) \ BD=0 point2point 214700000

RP/0/RSP0/CPU0:router# show l2vpn forwarding bridge-domain detail location 0/6/CPU0Fri Jan 7 13:54:45.740 PST

Bridge-domain name: 189:189, id: 0, state: up MAC learning: enabled MAC port down flush: enabled Flooding: Broadcast & Multicast: enabled Unknown unicast: enabled MAC aging time: 300 s, Type: inactivity MAC limit: 4000, Action: none, Notification: syslog MAC limit reached: no MAC Secure: disabled, Logging: disabled DHCPv4 snooping: profile not known on this node Dynamic ARP Inspection: disabled, Logging: disabled IP Source Guard: disabled, Logging: disabled IGMP snooping: disabled, flooding: enabled Bridge MTU: 9000 bytes Number of bridge ports: 2 Number of MAC addresses: 2 Multi-spanning tree instance: 0

GigabitEthernet0/1/0/3.189, state: oper up Number of MAC: 2


OL-23591-02


Statistics: packets: received 0, sent 0 bytes: received 0, sent 0 Storm control drop counters: packets: broadcast 0, multicast 0, unknown unicast 0 bytes: broadcast 0, multicast 0, unknown unicast 0 Dynamic arp inspection drop counters: packets: 0, bytes: 0 IP source guard drop counters: packets: 0, bytes: 0 ...

RP/0/RSP0/CPU0:router# show controllers np counters all Mon Nov 15 12:20:35.289 EST

Node: 0/0/CPU0:----------------------------------------------------------------Show global stats counters for NP0, revision v3

Read 20 non-zero NP counters:Offset Counter FrameValue Rate (pps)------------------------------------------------------------------------------- 23 PARSE_FABRIC_RECEIVE_CNT 417 0 30 RESOLVE_INRESS_DROP_CNT 9 0 31 RESOLVE_EGRESS_DROP_CNT 6 0 53 MODIFY_FRAMES_PADDED_CNT 3230 0 67 PARSE_MOFRR_SWITCH_MSG_RCVD_FROM_FAB 920 0 70 RESOLVE_INGRESS_L2_PUNT_CNT 1081 0 71 RESOLVE_EGRESS_L3_PUNT_CNT 4613 0 74 RESOLVE_LEARN_FROM_NOTIFY_CNT 3484 0 75 RESOLVE_BD_FLUSH_DELETE_CNT 104 0 83 RESOLVE_MOFRR_HASH_UPDATE_CNT 463 0 87 RESOLVE_MOFRR_SWITCH_MSG_INGNORED 407 0 111 DIAGS 536 0 295 DROP_IPV4_NEXT_HOP_DOWN 15 0...

The following command allows you to view the bridge domain forwarding data.

RP/0/RSP0/CPU0:router# show l2vpn forwarding bridge-domain location 0/6/CPU0 Thu Apr 15 00:15:29.581 EDT Bridge MACBridge-Domain Name ID Ports addr Flooding Learning State -------------------------------- ------ ----- ------ -------- -------- ---------bg1:bg1_bd1 0 3 4 Enabled Enabled UP

The following command allows you to view the bridge domain MAC details. The output from this command can be very large, so you should limit the terminal screen output or send the data to a file.


RP/0/RSP0/CPU0:router# show l2vpn forwarding bridge-domain mac-address location 0/6/CPU0Mac Address Type Learned from/Filtered on LC learned Resync Age Mapped to --------------------------------------------------------------------------------0000.0001.0101 dynamic Gi0/6/0/1.1 0/6/CPU0 0d 0h 1m 59s N/A 0000.0001.0102 dynamic Gi0/6/0/1.1 0/6/CPU0 0d 0h 1m 59s N/A 0000.0002.0202 dynamic (192.0.2.20, 1:101) 0/6/CPU0 0d 0h 1m 59s N/A 0000.0003.0303 dynamic (192.168.40.40, 1:101) 0/6/CPU0 0d 0h 1m 59s N/A


OL-23591-02


VPLS Not Forwarding Flooding Traffic from AC to Pseudowire

Step 1 View ingress UIDB and XID for the segment.

RP/0/RSP0/CPU0:router# show l2vpn forwarding interface hardware ingress detail location

Step 2 If PWs are invloved, view PW hardware information.

RP/0/RSP0/CPU0:router# show l2vpn forwarding neighbor 192.168.12.5 pw-id 100 hardware egress location node-id0

Step 3 View MPLS leaf information. RP/0/RSP0/CPU0:router# show mpls forwarding labels hardware egress detail location

Step 4 View bridge information about Broadcast, Multicast and Unknown Unicast. RP/0/RSP0/CPU0:router# show l2vpn bridge-domain bd-name 1 det


Step 6 View PI event traces. RP/0/RSP0/CPU0:router# show l2vpn trace location

Step 7 Ensure that the pseudowires (as applicable) and AC are up.

Step 8 Verify the hardware is programmed for both ACs.

Step 9 RP/0/RSP0/CPU0:router# show l2vpn forwarding interface GigabitEtherne0/5/0/2 hardware ingress detail location node-id

Step 10 Verify the hardware is programmed for pseudowires.

VPLS Not Forwarding Flooding Traffic from Pseudowire to AC

Step 1 View ingress UIDB and XID for the segment. RP/0/RSP0/CPU0:router# show l2vpn forwarding interface hardware ingress detail location

Step 2 View MPLS leaf information. RP/0/RSP0/CPU0:router# show mpls forwarding labels hardware egress detail location

Step 3 View bridge information about Broadcast, Multicast and Unknown Unicast. RP/0/RSP0/CPU0:router# show l2vpn bridge-domain bd-name 1 det




OL-23591-02


Step 6 Ensure that the pseudowires (as applicable) and AC are up.

Step 7 Verify that the hardware is programmed for both ACs.

RP/0/RSP0/CPU0:router# show l2vpn forwarding interface GigabitEtherne0/5/0/2 hardware ingress detail location node-id

Step 8 Verify that the hardware is programmed for PW (if applicable).

VPLS Not Forwarding Unicast Traffic from AC to AC


Step 2 Ensure that the hardware is programmed for both ACs.

Step 3 Ensure that the destination MAC entry is programmed for the LC’s destination interface. RP/0/RSP0/CPU0:router# show l2vpn forwarding bridge-domain mac-address location node-id

VPLS Not Forwarding Unicast Traffic from AC to Pseudowire

Step 1 View the bridge domain state.

RP/0/RSP0/CPU0:router# show l2vpn bridge-domain bd-name bd-name-id detail

Step 2 Ensure that the hardware is programmed for both AC and PW (as applicable).

Step 3 Ensure that the destination MAC entry is programmed for the LC’s destination interface.

RP/0/RSP0/CPU0:router# show l2vpn forwarding bridge-domain mac-address location node-id

VPLS Not Forwarding Flooding Traffic from Pseudowire to AC


Step 2 Ensure that the hardware is programmed for both AC and PW (as applicable).


OL-23591-02


Pseudowire Up but Ping Fails

Step 1 Determine where the ping packets are being dropped, view the xconnect AC interface counters and the L2VPN counters for the PW. For information on ping procedures, see the “Troubleshooting Ping and ARP Connectivity” section on page 3-75.

Step 2 View the bridge domain state.

RP/0/RSP0/CPU0:router# show l2vpn bridge-domain bd-name bd-name-id detail

Step 3 Ensure that both CEs are on the same subnet.


Step 5 Ensure that the end-to-end encapsulations match.

Traffic Loss

Step 1 Determine where the packets are being dropped, view the xconnect AC interface counters and the L2VPN counters for the PW. For information on ping procedures, see the “Troubleshooting Ping and ARP Connectivity” section on page 3-75.


Step 3 View segment counters to see if the packet and byte switched count increased. RP/0/RSP0/CPU0:router# show l2vpn forwarding interface GigabitEthernet interface-id detail location node-id

Step 4 Ensure that the bandwidth rates match between the CEs.

Pseudowire Flap Causing Traffic Loss


Step 2 View segment counters to see if the packet and byte switched count increased. RP/0/RSP0/CPU0:router# show l2vpn forwarding interface GigabitEthernet interface-id detail location node-id



OL-23591-02

Chapter 9 Troubleshooting L2VPN and Ethernet Services Troubleshooting Dynamic Host Configuration Protocol Snooping

Traffic Loss During RSP Fail Over

Step 1 View the state of the xconnect. RP/0/RSP0/CPU0:router# show l2vpn bridge detail

Step 2 View the counter for the segment. RP/0/RSP0/CPU0:router# show l2vpn forwarding interface GigabitEthernet interface-id detail location node-id



location node-id

Step 5 Check all routers in the MPLS path to ensure the following are configured:

a. MPLS LDP graceful restart

b. OSPF NSF

Step 6 View the segment counters to see if the packet and byte switched count increased. RP/0/RSP0/CPU0:router# show l2vpn forwarding interface GigabitEthernet node-id detail

location node-id

Preferred Path Not Working



location node-id

Troubleshooting Dynamic Host Configuration Protocol SnoopingDynamic Host Configuration Protocol Snooping (DHCP snooping) provides DHCP security by filtering untrusted DHCP messages, and by building and maintaining a DHCP snooping binding table. An untrusted message is a message that is received from outside the network or firewall and that can cause traffic attacks within your network. This section describes the following commands:

• Show Commands, page 9-228


OL-23591-02


• Trace Commands, page 9-228

• Syslog Commands, page 9-228

• Tech-support Commands, page 9-229

• Action Commands, page 9-229

• L2VPN Commands, page 9-229

• L2Snoop Commands, page 9-229

• Interface Controller Commands, page 9-230

Show CommandsThe DHCP application runs on the RSP. It has several EXEC mode CLI show commands that present the application's configuration state, DHCP client state, and DHCP packet statistics.

• show dhcp ipv4 snoop binding—View the state of DHCP clients in a table.

• show dhcp ipv4 snoop binding mac-address macaddress—View detailed state of DHCP Clients with the specified MAC Address.

• show dhcp ipv4 snoop binding summary—View the total number of DHCP Clients.

• show dhcp ipv4 snoop profile—View a list of DHCP snoop profiles.

• show dhcp ipv4 snoop profile name name—View details of a specific DHCP snoop profile.

• show dhcp ipv4 snoop statistics—View aggregate DHCP snoop Rx, Tx, and drop packets for each bridge domain.

• show dhcp ipv4 snoop statistics bridge-domain name—View detailed DHCP snoop Rx, Tx, and drop packets for each message type in a bridge domain.

Trace CommandsThe DHCP application has over 1200 Trace logs. The Trace logs record significant events that occur in the application. Trace logs that are associated with a specific DHCP client will contain the client MAC address.

• show dhcp ipv4 trace errors—View error traces.

• show dhcp ipv4 trace events—View event traces.

• show dhcp ipv4 trace packets—View packet processing traces.

• show dhcp ipv4 trace snoop errors—View error traces for DHCP snoop feature.

• show dhcp ipv4 trace snoop events—View event traces for the DHCP snoop feature.

• show dhcp ipv4 trace snoop internal—View internal debug traces for the DHCP snoop feature.

Syslog CommandsThe DHCP application has over 1600 syslog logs. These logs record events that occur in the application.

• debug dhcp ipv4 errors—View error logs.

• debug dhcp ipv4 events—View event logs.


OL-23591-02


• debug dhcp ipv4 packet—View packet processing logs.

• debug dhcp ipv4 snoop errors—View error logs for DHCP snoop feature.

• debug dhcp ipv4 snoop events—View event logs for the DHCP snoop feature.

• debug dhcp ipv4 snoop internal—View internal debug logs for the DHCP snoop feature.

Tech-support CommandsThe DHCP application has four tech-support commands that call groups of DHCP CLI commands. Use tech-support commands for information about the DHCP application for debugging.

• show tech-support dhcp ipv4 snoop file filename

• show tech-support dhcp ipv4 snoop bridge-domain-name bridge-domain-id file filename—View information for the specified bridge domain.

• show tech-support dhcp ipv4 snoop profile-name profilename file filename—View information for the specified profile.

Action CommandsUse the following CLI commands to clear DHCP snoop binding states:

• clear dhcp ipv4 snoop binding—Clears all DHCP snoop client bindings.

• clear dhcp ipv4 snoop binding bridge-domain bridge-domain-name—Clears all DHCP snoop client bindings in the specified bridge domain.

• clear dhcp ipv4 snoop binding mac-address macaddress—Clears the DHCP snoop client bindings with the specified MAC address.

L2VPN Commands DHCP snoop is enabled on L2VPN ACs by attaching a DHCP snoop profile to a bridge domain or AC. The DHCP snoop trusted attribute is configured on an AC according to the value of the trusted attribute in the DHCP snoop profile. L2VPN CLI commands are used to display the status of DHCP snoop attributes on L2VPN bridge domains and ACs.

• show l2vpn bridge-domain bd-name bridgename detail—View the L2VPN DHCP snoop configuration for the specified bridge domain.

• show l2vpn forwarding interface interface detail location location—View the L2VPN DHCP snoop configuration for a specific interface.

L2Snoop CommandsL2Snoop receives and transmits DHCP snoop packets between NETIO and the DHCP snoop application on the RSP.

show l2snoop statistics pcb all—View the L2SNOOP DHCP packet Rx/Tx statistics to and from the DHCP snoop application on the RSP.


OL-23591-02

Chapter 9 Troubleshooting L2VPN and Ethernet Services Troubleshooting Multiple Spanning Tree

Interface Controller CommandsInterface controllers receive and send DHCP snoop packets between the wire and the network processors.

show controllers interface stats—View the interface controller statistics that include DHCP packets that are sent and received from the wire.

Troubleshooting Multiple Spanning TreeMultiple Spanning Tree (MST) is an IEEE standard based on the Cisco proprietary Multiple Instances Spanning Tree Protocol (MISTP) implementation. This section explains how to troubleshoot MST and contains the following subsections:


• MSTP Incorrectly or Inconsistently Formed, page 9-230

• MSTP Correctly Formed, but Traffic Flooding, page 9-231

• Packet Forwarding Does Not Match MSTP State, page 9-231

• MSTAG Access Network Does Not Recognize MSTAG Node as Root, page 9-231

• Traffic Not Switching Through MSTAG Node(s), page 9-232

Using show and debug CommandsFor a complete list of MST show and debug commands, see the “Multiple Spanning Tree Protocol Commands” chapter in the “Cisco ASR 9000 Series Aggregation Services Router L2VPN and Ethernet Services Command Reference” module.

MSTP Incorrectly or Inconsistently FormedWhen the spanning tree is misformed, it is often because of misconfiguration or BPDU loss. This generally manifests as more than one node showing itself as ROOT, but can also result in disagreement on which nodes are ROOT.

MSTP Incorrectly or Inconsistently Formed—Misconfiguration

Ensure that the following match on the nodes:

• Configuration name

• Bridge revision

• Provider-bridge mode

• Instance to VLAN mapping

Run the following command to check that the configuration is consistent across multiple devices.

RP/0/RSP0/CPU0:router# show spanning-tree mst protocol-instance-id configuration


OL-23591-02

Chapter 9 Troubleshooting L2VPN and Ethernet Services Troubleshooting Multiple Spanning Tree

MSTP Incorrectly or Inconsistently Formed—BPDU Loss

Determine if node A is sending BPDUs to node B. The output of the following command includes a count of the number of BDPUs being sent and received. Run the command several times for each interface connecting the nodes.

RP/0/RSP0/CPU0:router# show spanning-tree mst protocol-instance-id interface interface-name

Only designated ports will send periodic BPDUs, but non-designated ports send updates on topology changes and startup. Ensure that BPDUs sent and received are going up as appropriate.

MSTP Correctly Formed, but Traffic FloodingIntermittent BPDU loss may mean the spanning tree will not show up incorrectly in the show commands, but will send out topology change notifications. These notifications cause a MAC flush, forcing traffic to flood until the MAC addresses are re-learned.

Run the following commend to check whether there have been any flushes.

RP/0/RSP0/CPU0:router# show spanning-tree mst <protocol-instance> topology-change flushes

Look for topology change notifications. Run the following command and look for TC 1:

Note This option is verbose.

Packet Forwarding Does Not Match MSTP State

Step 1 Shut down redundant links, remove MSTP configuration, and ensure that basic bridging works.

RP/0/RSP0/CPU0:router# show spanning-tree mst name

RP/0/RSP0/CPU0:router# show interface interface-name

Step 2 Check the state of each port as calculated by MSTP, and compare it with packet transmit and receive counts on ports and Ethernet flow points (EFPs) that are controlled by MSTP. Normal data packets should be sent/received only on ports that are in forwarding (FWD) state. In steady state operation, BPDUs are sent if there is at least one MSTI that is in Designated role.

Step 3 Ensure that BPDUs are flowing and that root bridge selection is correct. Check those related scenarios first.

RP/0/RSP0/CPU0:router# show l2vpn bridge-domain [detail]

This command will show the status of members of the bridge domain. Ensure that the relevant bridge domain members are up.

Step 4 Check forwarding state as programmed in hardware.

MSTAG Access Network Does Not Recognize MSTAG Node as RootIf multiple spanning tree access gateway (MSTAG) is not recognized as root, check that the two MSTAG devices are configured with the same root-id and root-priority for every MST instance (MSTI), and that the root-priority is lower than any of the access devices (preferably 0).

Also check (on the access devices) for any disputes; disputes are an indication of a misconfiguration.


OL-23591-02

Chapter 9 Troubleshooting L2VPN and Ethernet Services Additional References—Command Reference and Configuration Guides

Step 1 To view the BPDUs being sent by MSTAG, run the following command.

RP/0/RSP0/CPU0:router# show spanning-tree mstag protocol-instance-id bpdu interface interface-name

There are two ways of configuring MSTAG:

• Advertise as though both nodes are separate—requires each node have a unique bridge id and the configurations complement each other.

• Advertise as though each node is a different port on the same node—configuration is identical except for the port id.

Commands for MSTAG must target the untagged EFP instead of the base interface. Perform the following steps to verify your configuration and debug MSTAG.

Step 2 Verify the running configuration.

RP/0/RSP0/CPU0:router# show running-config spanning-tree {mst | mstag | repag} name

Traffic Not Switching Through MSTAG Node(s)

Step 1 Collect L2VPN and UIDB data to verify the data path is healthy. RP/0/RSP0/CPU0:router# show l2vpn bridge-domain [detail]

Step 2 Ensure that the forwarding state is set as it was programmed in the hardware.

Additional References—Command Reference and Configuration Guides

The following documents provide information on the commands and configuration procedures for L2VPN and Ethernet Services:

• Cisco ASR 9000 Series Aggregation Services Router L2VPN and Ethernet Services Command Reference

• Cisco ASR 9000 Series Aggregation Services Router L2VPN and Ethernet Services Configuration Guide

Use the following guide when you configure routing. L2VPN services rely on Layer 3 connectivity from the provider edge (PE) through the core:

Cisco ASR 9000 Series Aggregation Services Router Routing Configuration Guide, Release 4.0


OL-23591-02



C H A P T E R 10
Troubleshooting Quality of Service and Access Control Lists
This chapter describes techniques for troubleshooting quality of service (QoS) and access control list (ACL) features.

The system supports the following QoS features:

• Multilevel priority scheduling for voice and video applications with minimal jitter, latency and packet loss.

• Priority propagation to ensure service integrity for voice and video throughout all hierarchy layers, even at peak hours with high traffic load.

• Differentiated Service Code Point (DSCP), MPLS experimental bit (EXP) and IEEE 802.1p IP Precedence bit classification with marking, policing and scheduling, ingress and egress.

This chapter includes the following sections:


• Service-Policy Configuration Is Rejected, page 10-235

• Packets are Incorrectly Classified, page 10-235

• Packets in Wrong Queue, page 10-236

• Packets Incorrectly Marked, page 10-236

• Packets Incorrectly Policed, page 10-237

• Shaping Incorrect, page 10-237

• Weighted Random Early Detection Incorrect, page 10-237

• Bandwidth Not Guaranteed, page 10-238

• Bandwidth Ratio Not Working, page 10-238

• Non-zero Queue(conform) and Queue(exceed) Counters In show policy-map Commands, page 10-239

• Unable to Modify or Delete policy-map or class-map, page 10-240

• Unable to Modify or Delete class-map ACL, page 10-240

• Unable to Delete service-policy, page 10-240

• After QoS EA Restarts, show policy-map interface Fails, page 10-240

• After QoS EA Restarts, service-policy config Fails, page 10-241

• show policy-map interface Output Error, page 10-241


Chapter 10 Troubleshooting Quality of Service and Access Control Lists Using show and debug Commands

• Bundle Members Not Configured with service-policy, page 10-241

• Troubleshooting Access Control Lists, page 10-241


1. show run policy-map

2. show run classmap

3. show run interface

4. show policy-map interface type interface-name [output | input]

5. show qos interface type interface-name [output | input]

6. show qos-ea interface type interface-name [output | input]

7. show qos-ea km

8. debug qos-ea ?

DETAILED STEPS


Step 1 show run policy-map

Example:RP/0/RSP0/CPU0:router# show run policy-map l1-all

View policy-map with name.

Step 2 show run classmap

Example:RP/0/RSP0/CPU0:router# show run class-map c2

View class-map configuration with name.

Step 3 show run interface

Example:RP/0/RSP0/CPU0:router# show run interface g0/2/0/0

View the service-policy binding for a given port/subinterface.

Step 4 show policy-map interface type interface-name [output|input]

Example:RP/0/RSP0/CPU0:router# show policy-map interface g0/2/0/0

View all the statistics, queue IDs and class information.


OL-23591-02

Chapter 10 Troubleshooting Quality of Service and Access Control Lists Service-Policy Configuration Is Rejected

Service-Policy Configuration Is Rejected

Step 1 If the service-policy configuration is rejected or failed to commit, check the error message with the show configuration failed command.

Step 2 If resource usage is more than what is configured, verify how many are checkpointed.

RP/0/RSP0/CPU0:router# show qos-ea ha chkpt all info location node-id

Step 3 Check the OOR.

Step 4 Verify resources used.

Step 5 Verify summary information.

RP/0/RSP0/CPU0:router# show qos summary {queue | police | policy}

Packets are Incorrectly Classified

Step 1 Verify packets are arriving on the correct interface.

Step 2 Verify the packet fields are as expected.

Step 3 Note the packet type.

Step 5 show qos interface type interface-name [output|input]

Example:RP/0/RSP0/CPU0:router# show qos int g0/2/0/0 out

View all the configuration of each class in hardware.

Step 6 show qos-ea interface type interface-name [output|input]

Example:RP/0/RSP0/CPU0:router# show qos-ea int g0/2/0/0 out

View all the class information structures.

Step 7 show qos-ea km

Example:RP/0/RSP0/CPU0:router# show qos-ea km policy l2-all vmr interface g0/2/0/0 sw

View the key manager (TCAM key manager) related fields associated to a policy-map/interface binding.

Step 8 debug qos-ea ?

Example:RP/0/RSP0/CPU0:router# debug qos-ea ?

—



OL-23591-02

Chapter 10 Troubleshooting Quality of Service and Access Control Lists Packets in Wrong Queue

Step 4 Verify the KM policy information matches UIDB configuration. RP/0/RSP0/CPU0:router# show qos-ea km policy policy info location filename RP/0/RSP0/CPU0:router# show qos-ea km policy policy vmr interface filename hw detail

Step 5 Verify VMR entries for each class. RP/0/RSP0/CPU0:router# show qos-ea km policy policy vmr interface filename hw detail

Step 6 Verify which class the packets are actually matching. If packet fields should match different class, then NP Microcode needs to debug this further. RP/0/RSP0/CPU0:router# show policy-map interface filename {output | input} [member filename]

Step 7 Verify if in ingress QoS lookup occurs before Layer 2 ingress rewrite and that in egress Layer 2 rewrite occurs before QoS lookup. RP/0/RSP0/CPU0:router# show qos type interface {input | output} location node-id

Step 8 RP/0/RSP0/CPU0:router# show run interface type node-id

Step 9 RP/0/RSP0/CPU0:router# show run policy-map policy

Step 10 RP/0/RSP0/CPU0:router# show run class-map classmap

Packets in Wrong Queue

Step 1 RP/0/RSP0/CPU0:router# show qos-ea type interface {input | output} location node-id

Step 2 Verify the packets are correctly classified.

Step 3 Verify hash structure. RP/0/RSP0/CPU0:router# show qos-ea type interface {input | output} location node-id

Step 4 Verify the hash key for the class and hash result of the class has correct Queue ID.

Packets Incorrectly Marked

Step 1 Verify packets are classified correctly. RP/0/RSP0/CPU0:router# show qos type interface {input | output} location node-id

Step 2 RP/0/RSP0/CPU0:router# show qos-ea km policy policy vmr interface filename hw

Step 3 RP/0/RSP0/CPU0:router# show qos-ea km policy policy vmr interface filename sw

Step 4 RP/0/RSP0/CPU0:router# show policy-map interface filename {input | output} [member filename]


OL-23591-02

Chapter 10 Troubleshooting Quality of Service and Access Control Lists Packets Incorrectly Policed

Step 5 Verify marking value. RP/0/RSP0/CPU0:router# show qos type interface {input | output} location node-id

Packets Incorrectly Policed

Step 1 Ensure that packets are correctly classified.

Step 2 Verify whether policer CIR/CBS/PIR/PBS are set correctly as per configured service-policy. Also verify the rate at which traffic is coming to match against policed rate. RP/0/RSP0/CPU0:router# show qos type interface {input | output} location node-id

Step 3 Get the token bucket and police node index of the class. RP/0/RSP0/CPU0:router# show qos-ea type interface {input | output} location node-id


Shaping Incorrect

Step 1 Ensure that packets are correctly classified. RP/0/RSP0/CPU0:router# show qos type interface {input | output} location node-id

Step 2 Verify whether shaper CIR/CBS/PIR/PBS are set correctly as per configured service-policy. Get the shape profile ID and entity handle information (np, tm, level, index, offset). RP/0/RSP0/CPU0:router# show qos-ea type interface {input | output} location node-id

Step 3 Verify the shaper profiles in hardware if they are correctly configured.


Weighted Random Early Detection Incorrect


Step 2 Verify whether the weighted random early detection (WRED) curves are correctly configured with minimum and maximum thresholds of each curve are as per the configured service policy. RP/0/RSP0/CPU0:router# show qos type interface {input | output} location node-id


OL-23591-02

Chapter 10 Troubleshooting Quality of Service and Access Control Lists Bandwidth Not Guaranteed

Step 3 Get the WRED profile ID and entity handle information (np, tm, level, index, offset). RP/0/RSP0/CPU0:router# show qos-ea type interface {input | output} location node-id


Bandwidth Not Guaranteed


Step 2 Verify whether the weights of each class are configured correctly as per the bandwidth ratio among classes. RP/0/RSP0/CPU0:router# show qos type interface {input | output} location node-id


Step 4 If its correctly configured, then get the WFQ profile ID and entity handle information (np, tm, level, index, offset) of the class. RP/0/RSP0/CPU0:router# show qos-ea type interface {input | output} location node-id


Bandwidth Ratio Not Working



Step 3 Verify whether the commit weights of each class is configured correctly as per the bandwidth ratio among classes. Also verify that excess weights are configured as per the bandwidth remaining ratio configuration. RP/0/RSP0/CPU0:router# show qos type interface {input | output} location node-id

Step 4 RP/0/RSP0/CPU0:router# show qos-ea type interface {input | output} location node-id


Step 6 Get the WFQ profile ID and entity handle information (np, tm, level, index, offset) of the class.

RP/0/RSP0/CPU0:router# show qos-ea type interface {input | output} location node-id

Step 7 If commit and excess weights are correct:

a. Check queue size of each class.

b. Increase the queue size.


OL-23591-02

Chapter 10 Troubleshooting Quality of Service and Access Control Lists Non-zero Queue(conform) and Queue(exceed) Counters In show policy-map Commands

Non-zero Queue(conform) and Queue(exceed) Counters In show policy-map Commands

This section explains what to do if the show policy-map command displays non-zero values for Queue(conform) and Queue(exceed) counters.

On the ASR 9000, every hardware queue has a configured committed information rate (CIR) and peak information rate (PIR) value. CIR corresponds to the guaranteed bandwidth for the queue and PIR corresponds to the maximum bandwidth (also known as the shape rate) for the queue. To configure the CIR and PIR, use the police rate command. The syntax is:

RP/0/RSP0/CPU0:router# police rate {value [units] | percent percentage} [burst burst-size [burst-units]] [peak-rate value [units]] [peak-burst peak-burst [burst-units]]

In this command, CIR is the police rate value and PIR is the police peak-rate value.

Example

RP/0/RSP0/CPU0:router# show policy-map location 0/7/0/30

GigabitEthernet0/7/0/30.1001 output: STS-1

Class class-default Classification statistics (packets/bytes) (rate - kbps) Matched : 4167078/4179237900 79024 Transmitted : 2224974/2229365484 42017 Total Dropped : 1942095/1949863380 36801 Policy SHAPE-OUT Class BFD-OUT Classification statistics (packets/bytes) (rate - kbps) Matched : 4786/296732 5 Transmitted : 4786/296732 5 Total Dropped : 0/0 0 Policing statistics (packets/bytes) (rate - kbps) Policed(conform) : 4786/296732 5 Policed(exceed) : 0/0 0 Policed(violate) : 0/0 0 Policed and dropped : 0/0 Policed and dropped(parent policer) : Un-determined Queueing statistics Queue ID : 8 High watermark (Unknown) Inst-queue-len (packets) : 0 Avg-queue-len (Unknown) Taildropped(packets/bytes) : 0/0 Queue(conform) : 0/0 0 Queue(exceed) : 4786/296732 5 RED random drops(packets/bytes) : 0/0

A non-zero value displayed for Queue(exceed) does not mean that there is a packet drop, but rather the number of packets above the configured (or system selected) CIR rate on that queue. Although you could change the Queue(exceed) behavior by explicitly configuring a bandwidth and/or a shape rate on each queue, it is not necessary to do so. You can treat these counters as informational or simply ignore them.

In the police rate command, if you do not explicitly configure a value for the police rate (the CIR), the system automatically assigns one. The Queue(conform) counter in the show policy-map command is the number of packets/bytes that were transmitted within this CIR value, and the Queue(exceed) value is the number of packets/bytes that were transmitted within the PIR value. The Queue(exceed) counter is based on whether the parent bandwidth is exceed or conform. If there is no parent bandwidth, all traffic is counted as excess.


OL-23591-02

Chapter 10 Troubleshooting Quality of Service and Access Control Lists Unable to Modify or Delete policy-map or class-map

Unable to Modify or Delete policy-map or class-map

Step 1 Verify the policy is applied on an interface. RP/0/RSP0/CPU0:router# show running-config

Step 2 Remove service-policy on the interfaces.

Step 3 Modify the policy-map.

Unable to Modify or Delete class-map ACL • show config failed

• show running-config

Step 1 Verify the ACL is part of a match statement in a class-map.

Step 2 Verify the class-map is part of any policy-map that is applied on an interface.

Step 3 If the policy-map is applied on interface, ACL modification/deletion is not allowed.

Step 4 Remove all the service-policy configuration of this policy-map and modify ACLs.

Unable to Delete service-policyStep 1 RP/0/RSP0/CPU0:router# show config failed

Step 2 Restart the qos_ma_ea process.

After QoS EA Restarts, show policy-map interface Fails • show running-config

• show qos-ea ha chkpt all info location node-id

• show qos-ea ha chkpt if-qos all location node-id

Step 1 Verify if the state of QoS EA is in in_sync (state = 2). RP/0/RSP0/CPU0:router# show qos-ea ha state location node-id

Step 2 If there is no error, do the following:

a. RP/0/RSP0/CPU0:router# debug generic


OL-23591-02

Chapter 10 Troubleshooting Quality of Service and Access Control Lists After QoS EA Restarts, service-policy config Fails

b. Collect debugs by performing the failing command.

After QoS EA Restarts, service-policy config Fails

Step 1 Verify the state of QoS EA is in in_sync (state = 2). RP/0/RSP0/CPU0:router# show qos-ea ha state location node-id

Step 2 If there is no error, do the following:

a. RP/0/RSP0/CPU0:router# debug generic

b. Collect the debugs by performing the failing command.

show policy-map interface Output ErrorFor bundles, specify member interface. Policy information for bundle-interface is not available in the current release.

• show policy-map interface {output | input} member

• show {qos | qos-ea} interface {output | input} location node-id

Bundle Members Not Configured with service-policyFor bundles, specify member interface. Policy information for bundle-interface is not available in the current release.

• show policy-map interface {output | input} member

• show {qos | qos-ea} interface {output | input} location node-id

Troubleshooting Access Control ListsThis section explains how to troubleshooting problems with access control lists (ACLs). ACLs are used for packet filtering and selecting traffic types to be analyzed, forwarded, or influenced in some way. Access control entries (ACEs) are individual permit or deny statement within an ACL. Each ACE includes an action element (“permit” or “deny”) and a filter element based upon criteria such as source address, destination address, protocol, protocol-specific parameters, and so on. This section contains the following topics:


• ACL Messages Not Appearing, page 10-243

• Fragmented Packets Being Accepted, page 10-243

• Egress Counter Incorrect or Not Working, page 10-244


OL-23591-02

Chapter 10 Troubleshooting Quality of Service and Access Control Lists Troubleshooting Access Control Lists

• ACL Interface Bind Rejected, page 10-244

• Single ACE Using Many TCAMs, page 10-244

• ACL Using Varying TCAM Space, page 10-245

• ACL Logs Not Working for Ethernet Services, page 10-245

• Ethernet Services ACL Bind on Interface Rejected, page 10-245

• Changing ACL Exhausts TCAM, page 10-245

• Cannot Delete ACL, page 10-246

• DF Bit Not Supported, page 10-246

• Max ACL Limit Reached, page 10-246

• Unsupported Combinations in ACL, page 10-246

• No Statistics Counters, page 10-246

• TCAMs Out of Resources, page 10-246


SUMMARY STEPS

1. show access-lists ipv4 [rp-access [hardware {ingress | egress} {sequence-number | location node-id | summary [rp-access] | maximum [detail] [usage {pfilter location node-id}]

2. debug feature-ea-dll {all | error | info | resmgr | vmr}

DETAILED STEPS


Step 1 show access-lists ipv4 [rp-access [hardware {ingress | egress} {sequence-number | location node-id | summary [rp-access] | maximum [detail] [usage {pfilter location node-id}]

Example:RP/0/RSP0/CPU0:router# show access-lists ipv4 dtho 10 ipv4 access-list dtho 10 permit ipv4 any any

View all IPv4 ACL contents. Filter results using the following parameters and keywords:

• access-list-name-—IPv4 ACL name.

• hardware—Ingress specifies an inbound interface, egress specifies an outbound interface.

• sequence-number—ACL number, 1 to 2147483646.

• location node-id—Rack/slot/module notation of ACL.

• summary—Summary of all current IPv4 ACLs.

• maximum—Maximum configurable IPv4 ACLs and ACEs.

• detail—Out-of-resource (OOR) details, OOR limits the number of ACLs and ACEs configured.

• usage—View the usage of the ACL on a given line card (LC).

• pfilter—Packet filtering for the LC.

Step 2 debug feature-ea-dll {all | error | info | resmgr | vmr}

View error messages at various levels.


OL-23591-02


ACL Messages Not Appearing

Step 1 View ACEs in the ACLs. RP/0/RSP0/CPU0:router# show access-lists ipv4

Step 2 View TCAM entries in the ACLs.. RP/0/RSP0/CPU0:router# show access-lists ipv4 hardware {ingress | egress} detail ...

Step 3 Configure the logs in the ACL. RP/0/RSP0/CPU0:router# ipv4 access-lists log-update threshold

Workaround

If an entry with the fragment flag is not present, remove the access-list from all the interfaces and reapply it.

Note Fragmented packets are not matched against the deny ACE without fragment keyword. Add the explicit fragment keyword in the ACE to deny the fragment packet. See the workaround commands in the “Fragmented Packets Being Accepted” section on page 10-243.

Fragmented Packets Being Accepted

Step 1 View ACEs in the ACLs. RP/0/RSP0/CPU0:router# show access-list ipv4

Step 2 View IPv4 counters, for example, fragment. RP/0/RSP0/CPU0:router# show ipv4 traffic

Step 3 View TCAM entries in the ACLs.

Step 4 Ensure that the fragment keyword is in the ACE. RP/0/RSP0/CPU0:router# deny ipv4 any any fragments

Step 5 Check the fragment packet count received by the device.

Step 6 View TCAM entries.

Workaround

Fragmented packets are not matched against the deny ACE without the fragment keyword. If there is not an entry with the fragment flag, perform the following procedure.

Step 1 Remove the ACL from all interfaces.


OL-23591-02


Step 2 Add the explicit fragment keyword in the ACE to deny the fragment packet.

Step 3 Reapply the ACL to all interfaces.

Egress Counter Incorrect or Not Working

Step 1 View known routes. RP/0/RSP0/CPU0:router# show route ipv4

Step 2 View ARP table entries. Look for the next hop. RP/0/RSP0/CPU0:router# show arp

Step 3 View TCAM entries for the ACL. RP/0/RSP0/CPU0:router# show access-list ipv4 hardware

Workaround

Step 1 If the route is missing or the ARP is incomplete, use the no shut command to recover.

Step 2 If the UIDB table or TCAM entry is incorrect, remove the ACL from all of the interfaces and reapply it.

ACL Interface Bind RejectedView errors encountered when the configuration was applied. RP/0/RSP0/CPU0:router# show configuration failed

Workaround

If the error is related to TCAM space, remove the ACEs from the ACL. There is a limit of 64 TCAM entries per ACL.

Single ACE Using Many TCAMs


Step 2 Check the number of ranges in the ACE.


OL-23591-02


ACL Using Varying TCAM SpaceView Pre-Internal Forwarding Information Base (Pre-IFIB) hardware statistic entries. RP/0/RSP0/CPU0:router# show lpts pifib brief

ACL Logs Not Working for Ethernet ServicesEthernet services logging is not supported.

Ethernet Services ACL Bind on Interface Rejected

Step 1 View any errors encountered when the configuration was applied. RP/0/RSP0/CPU0:router# show configuration failed


Step 3 View trace log for pfilter_ea.

Workaround

Step 1 If a field in the ACL is not supported, remove it from the ACE.

Step 2 If the TCAM is out of space, reduce the ACEs in the ACL.

Step 3 Reduce the ranges in the ACL.

Changing ACL Exhausts TCAMView ACEs configured for the ACL. RP/0/RSP0/CPU0:router# show access-list {ethernet-service/ipv4}

Workaround

Remove the old ACL before applying the new one.


OL-23591-02


Cannot Delete ACL

Step 1 View any errors encountered when the configuration was applied. RP/0/RSP0/CPU0:router# show configuration error

Step 2 View interfaces using the ACL. RP/0/RSP0/CPU0:router# show access-list {ethernet-services | ipv4} usage pfilter

Step 3 View IPv4 trace information. RP/0/RSP0/CPU0:router# show access-list ipv4 trace

Step 4 View the Ethernet services trace. RP/0/RSP0/CPU0:router# show access-list ethernet-services trace

DF Bit Not SupportedThe Do Not Fragment (DF) bit is not supported as match criteria in the current release.

Max ACL Limit ReachedThe maximum number of ACL IDs per network processor (NP) is 2048. Interfaces share TCAM entries for the ACL name and direction.

Unsupported Combinations in ACLThere may be unsupported field combinations in the access-list. Verify that the combinations in the access-list are currently supported. The current release supports the following combinations:

• VLAN OUT + L2 PROTO + MAC SA + MAC DA

• VLAN OUT + VLAN IN + MAC SA + MAC DA

• VLAN OUT + VLAN IN + L2 PROTO + MAC DA

No Statistics CountersStatistics counters are not supported in the current release.

TCAMs Out of ResourcesThe TCAMs Out of Resources message means you have attempted to provision more than the available number of TCAM entries.


OL-23591-02


C H A P T E R 11
Troubleshooting Multicast Services
This chapter describes techniques that you can use to troubleshoot multicast services. It includes the following sections:

• Troubleshooting IGMP Snooping (Layer 2 Multicast), page 11-247

• Troubleshooting Native Multicast Routing (Layer 3), page 11-256

Troubleshooting IGMP Snooping (Layer 2 Multicast)This section explains how to troubleshoot problems with Internet Group Management Protocol (IGMP) snooping. IGMP snooping restricts multicast flows at Layer 2 to only those segments with at least one interested receiver.

A prerequisite for implementing IGMP snooping is that the network must be configured with a Layer 2 VPN (L2VPN). IGMP snooping is supported only under L2VPN bridge domains.

This section covers the following topics:

• Using show Commands, page 11-247

• Using the debug, trace, and show tech-support Commands, page 11-249

• Troubleshooting Missing Routes and Forwarding Errors, page 11-250

Using show Commands

SUMMARY STEPS

1. Confirm correct topology and configuration

a. show l2vpn bridge-domain summary

b. show igmp snooping bridge-domain

c. show l2vpn bridge-domain bd-name bd-name

d. show l2vpn bridge-domain bd-name bd-name detail

e. show igmp snooping bridge-domain bd-name detail

f. show igmp snooping port bridge-domain bd-name

g. show igmp snooping profile

2. Confirm that IGMP Snooping is sending and receiving control traffic


Chapter 11 Troubleshooting Multicast Services Troubleshooting IGMP Snooping (Layer 2 Multicast)

a. show igmp snooping summary statistics

b. show igmp snooping bridge-domain bd-name detail statistics

c. show igmp snooping port [if-type if-name] detail statistics

3. Confirm that IGMP Snooping is creating group state as expected

a. show igmp snooping group

b. show igmp snooping group port

c. show igmp snooping group source

d. show igmp snooping group bridge-domain bd-name

e. show igmp snooping port if-type if-name group [detail]

4. Confirm that the forwarding state matches the IGMP Snooping state

a. show l2vpn forwarding bridge-domain [bridge group name:bd-name] mroute ipv4 location lc-name

b. show l2vpn forwarding bridge-domain [bridge group name:bd-name] mroute ipv4 hardware [ingress | egress] location lc-name

5. View the number of packets sent, received, and failed for IGMP from and to l2snoop; determine whether packets are being dropped. Use the command show l2snoop statistics summary.

DETAILED STEPS


Step 1 a. show l2vpn bridge-domain summary

b. show igmp snooping bridge-domain

c. show l2vpn bridge-domain bd-name bd-name

d. show l2vpn bridge-domain bd-name bd-name detail

e. show igmp snooping bridge-domain bd-name detail

f. show igmp snooping port bridge-domain bd-name

g. show igmp snooping profile

Confirm correct topology and configuration:

a. List all L2VPN bridge domains

b. View IGMP Snooping state in all bridge domains.

c. View information about the specified bridge domain, including the list of interfaces and VFIs.

d. View detailed information about the specified bridge domain, including the IGMP Snooping profile. Verify that the L2VPN and IGMP configurations are consistent. The profile, bridge-domain, and bridge group specified in the IGMP configuration must be consistent with that specified in the L2VPN configuration.

e. View detailed IGMP Snooping information in the specified bridge domain.

f. See IGMP Snooping view of the ports and port state in the specified bridge domain.

g. View the snooping profile information.

Step 2 a. show igmp snooping summary statistics

b. show igmp snooping bridge-domain bd-name detail statistics

c. show igmp snooping port [if-type if-name] detail statistics

Confirm that IGMP Snooping is sending and receiving control traffic:

a. View global traffic statistics.

b. View traffic at the bridge domain level.

c. View traffic at the port level.


OL-23591-02


Using the debug, trace, and show tech-support CommandsThe following commands are useful for debugging IGMP snooping.

Note These commands might cause a large amount of information to be displayed on your output terminal.

This command gathers information about the IGMP control packets in the system, for example, whether JOIN and QUERY packets are being received and transmitted.

debug igmp snooping {all | error | event | init | management | packet | packet-error | proto | topo}

• all—All debugging

• error—Error debugging

• event—Event debugging

• init—Init debugging

• management—Management debugging

• packet—Packet debugging

• packet-error—Packet error debugging

• proto—Proto debugging

• topo—Topology debugging

If IGMP control packets are not being received and transmitted as expected, use this command to help locate the cause of the problem.

debug l2snoop {call | error | events | init | packet}

• call—L2snoop function call related debugging

Step 3 a. show igmp snooping group

b. show igmp snooping group port

c. show igmp snooping group source

d. show igmp snooping group bridge-domain bd-name

e. show igmp snooping port if-type if-name group [detail]

Confirm that IGMP Snooping is creating group state as expected:

a. View group state in all bridge domains.

b. View the groups created for a given port.

c. View the group created for a given source.

d. View group state in the specified bridge domain.

e. View group state on the specified interface.

Step 4 a. show l2vpn forwarding bridge-domain [bridge group name:bd-name] mroute ipv4 location lc-name

b. show l2vpn forwarding bridge-domain [bridge group name:bd-name] mroute ipv4 hardware [ingress | egress] location lc-name

Confirm that the forwarding state matches the IGMP Snooping state:

a. View forwarding state in the L2FIB on the specified line card.

b. View forwarding state installed in the hardware on the specified line card.

Step 5 a. show l2snoop statistics summary View the number of packets sent, received, and failed for IGMP from and to l2snoop; determine whether packets are being dropped.



OL-23591-02


• error—L2snoop error debugging

• events—L2snoop events debugging

• init—L2snoop init debugging

• packet—Decode L2snoop packet sends and receives

The following commands provide additional information for advanced troubleshooting.

show igmp snooping trace

• all—Show IGMP snoop trace all

• error—Show IGMP snooping trace error

• file—Specific file

• hexdump—Display traces in hexadecimal

• last—Display last <n> entries

• location—Card location

• packet-error—Show IGMP snooping trace packet error

• reverse—Display latest traces first

• stats—Display statistics

• tailf—Display new traces as they are added

• unique—Unique entries with counts

• verbose—Display internal debugging information

• wrapping—Wrapping entries

show tech-support igmp snooping {file | terminal}

• file—Specify a valid file name (for example, disk0:tmp.log)

• terminal—Send output to terminal

Troubleshooting Missing Routes and Forwarding ErrorsThis section explains what to do if packets are not being directed to the expected routes or are not being received by the RSP card.

Step 1 Check the L2VPN configuration to verify that the IGMP snooping profile is configured on the bridge domain and optionally on one or more bridge ports. The snooping profile must be present in the bridge domain for IGMP snooping to be enabled.

RP/0/RSP0/CPU0:router# show run l2vpn

Step 2 Verify that IGMP snooping is enabled, that is, attached to the bridge-domain and optionally to one or more bridge ports.


Step 3 Verify that Layer 2 multicast routes (mroutes) are present in the bridge-domain.

RP/0/RSP0/CPU0:router# show l2vpn forwarding bridge-domain [bridge-domain-id] mroute ipv4 summary location node-id

Step 4 Verify the status of flood forwarding. Confirm that the forwarding state matches the snooping state.


OL-23591-02


RP/0/RSP0/CPU0:router# show l2vpn forwarding bridge-domain [bridge-domain-id] detail location node-id

Step 5 Verify that the querier is enabled in the snooped domain. Without a querier, the system drops all IGMP reports with reason No Querier.

RP/0/RSP0/CPU0:router# show igmp snooping summary statistics include-zeroes | include Reports NoThu Jan 6 12:03:10.715 MEZ Reports No Querier: 0 V3 Reports No Querier: 0 V3 Reports No Sources: 0

Tip Use variations of this command (for example, omitting the include modifier) to show other types of errors, such as time to live errors (TTL not 1), which are useful in troubleshooting.

Step 6 Verify that the bridge domain, number of mrouters, ports, and IP addresses are as expected. Run the snooping statistics several times to notice any trends and the corresponding locations.

RP/0/RSP0/CPU0:router# show igmp snooping bridge-domain [bridge-domain-id] detail statistics

Step 7 Verify that packets are being received and transmitted as expected, and that there are no failed packets. Clear the counters and rerun them several times to notice any trends and the corresponding locations.

RP/0/RSP0/CPU0:router# show l2snoop statistics pcb all location active-RSP-location

The output of this command is an aggregate of all Layer 2 snoop counters, therefore, further investigation is needed to determine if any observed failures are related to IGMP snooping. (In this command, PCB = protocol control block.)

Step 8 Check whether the IGMP_SNOOP packet counters are incrementing on the network processors (NPs). If so, this indicates that the packets are being punted. Clear the counters and rerun them several times to notice any trends and the corresponding locations.

RP/0/RSP0/CPU0:router# show controller np counters <np number or all> location <LC location where punts are expected>

Examples

1. This example shows that the IGMP snooping profile is configured in the bridge domain.

RP/0/RSP0/CPU0:router# show run l2vpnTue Jan 4 09:59:49.849 PSTl2vpn router-id 10.144.144.144 pw-class CW_enable encapsulation mpls control-word ! !xconnect group g1 p2p p1 neighbor 10.1.1.1 pw-id 100 pw-class c1 ! ! !


OL-23591-02


bridge group 215 bridge-domain 215 mtu 9000 igmp snooping profile default interface GigabitEthernet0/1/0/3.215 storm-control multicast pps 500 storm-control broadcast pps 4500 ! interface GigabitEthernet0/1/0/7.215 ! interface GigabitEthernet0/1/0/30.215 ! vfi 215 neighbor 10.19.19.19 pw-id 215 ! ! ! !...

2. This example shows that IGMP snooping profile is present in the bridge domain, therefore IGMP snooping is enabled.

RP/0/RSP0/CPU0:router# show l2vpn bridge-domain detail...Bridge group: 215, bridge-domain: 215, id: 4, state: up, ShgId: 0, MSTi: 0 MAC learning: enabled MAC withdraw: enabled MAC withdraw for Access PW: enabled Flooding: Broadcast & Multicast: enabled Unknown unicast: enabled MAC aging time: 300 s, Type: inactivity MAC limit: 4000, Action: none, Notification: syslog MAC limit reached: no MAC port down flush: enabled MAC Secure: disabled, Logging: disabled Split Horizon Group: none Dynamic ARP Inspection: disabled, Logging: disabled IP Source Guard: disabled, Logging: disabled DHCPv4 snooping: disabled IGMP Snooping profile: default Bridge MTU: 9000 MIB cvplsConfigIndex: 5 Filter MAC addresses: Create time: 24/11/2010 15:47:48 (5w5d ago) No status change since creation ACs: 3 (3 up), VFIs: 1, PWs: 1 (1 up), PBBs: 0 (0 up) List of ACs:...

3. This example shows that Layer 2 multicast routes (mroutes) are present in the bridge-domain.

RP/0/RSP0/CPU0: router# show l2vpn forwarding bridge-domain 215:215 mroute ipv4 summary location 0/1/CPU0 Tue Jan 4 10:20:57.264 PSTGlobal Mroute Stats:-----------------------------------------------------------------------------------


OL-23591-02


Message Count Info1 Info2 Time ======= ===== ===== ===== ==== mcast route count 7 0x4 0x0 Jun 28 01:48:14.324 mcast route update dropped 0 0x0 0x0 - mcast route delete dropped 0 0x0 0x0 - mcast route del all drop 0 0x0 0x0 - mcast route add 27 0x4 0x0 Jun 28 01:48:14.324 mcast route delete 20 0x4 0x0 Jan 9 14:49:08.761 mcast route delete all 0 0x0 0x0 - mcast xid add 124 0x4 0x0 Jun 28 01:48:14.324 mcast xid delete 15 0x4 0x0 Jan 17 10:41:06.123 mcast stale xid delete 0 0x0 0x0 - mcast stale delete 0 0x0 0x0 - mcast bulk messages 48 0x0 0x0 Jun 28 01:48:14.324

Per Bridge Mroute Stats: Bridge-domain: 215:215 ,id: 4 L2 Multicast Route entries: 7 mroute add: 27, delete: 20 xid add: 124, delete: 15 mroute delete all: 0 mroute update dropped: 0 mroute delete dropped: 0 mroute delete all dropped: 0 Deleted stale mroute entries: 0 Deleted stale xid entries: 0

4. This example shows that IGMP snooping is enabled and flooding is disabled.

RP/0/RSP0/CPU0:router# show l2vpn forwarding bridge-domain 215:215 detail location 0/1/CPU0 Mon Jan 3 14:49:50.332 PST

Bridge-domain name: 215:215, id: 4, state: up MAC learning: enabled MAC port down flush: enabled Flooding: Broadcast & Multicast: enabled Unknown unicast: enabled MAC aging time: 300 s, Type: inactivity MAC limit: 4000, Action: none, Notification: syslog MAC limit reached: no MAC Secure: disabled, Logging: disabled DHCPv4 snooping: profile not known on this node Dynamic ARP Inspection: disabled, Logging: disabled IP Source Guard: disabled, Logging: disabled IGMP snooping: enabled, flooding: disabled Bridge MTU: 9000 bytes Number of bridge ports: 4 Number of MAC addresses: 0 Multi-spanning tree instance: 0...

5. This example shows the bridge domain, number of mrouters, ports, and IP addresses.

RP/0/RSP0/CPU0:router# show igmp snooping bridge-domain 215:215 detail statisticsMon Jan 3 10:56:17.534 PST

Bridge Domain Profile Act Ver #Ports #Mrtrs #Grps #SGs------------- ------- --- --- ------ ------ ----- ----215:215 default Y v2 4 3 6 0

Profile Configured Attributes:


OL-23591-02


System IP Address: 10.144.144.144 Minimum Version: 2 Report Suppression: Enabled Unsolicited Report Interval: 1000 (milliseconds) TCN Query Solicit: Enabled TCN Membership Sync: Disabled TCN Flood: Enabled TCN Flood Query Count: 2 Router Alert Check: Disabled TTL Check: Disabled Internal Querier Support: Enabled Internal Querier Version: 3 Internal Querier Timeout: 0 (seconds) Internal Querier Interval: 60 (seconds) Internal Querier Max Response Time: 10.0 (seconds) Internal Querier Robustness: 2 Internal Querier TCN Query Interval: 10 (seconds) Internal Querier TCN Query Count: 2 Internal Querier TCN Query MRT: 0 (seconds) Querier Query Interval: 60 (seconds) Querier LMQ Interval: 1000 (milliseconds) Querier LMQ Count: 2 Querier Robustness: 2 Startup Query Interval: 15 seconds Startup Query Count: 2 Startup Query Max Response Time: 10.0 seconds Mrouter Forwarding: Enabled Querier: IP Address: 10.161.161.161 Port: Neighbor 10.19.19.19 pw-id 215 Version: v2 Query Interval: 60 seconds Robustness: 2 Max Resp Time: 10.0 seconds Time since last G-Query: 30 seconds Internal Querier Statistics (elapsed time since last cleared 5w4d): Rx General Queries: 142074 Rx General Queries When Disabled: 142072 Rx General Queries As Querier: 1 Rx General Queries As Non Querier: 0 Rx General Queries As Winner: 0 Rx General Queries As Loser: 0 Rx Global Leaves: 1590 Rx Global Leaves When Disabled: 1590 Rx Global Leaves As Non Querier: 0 Rx Global Leaves Ignored: 0 Rx Pim Enabled Notifications: 0 Rx Pim Disabled Notifications: 0 Rx Local Query Solicitations: 0 Tx General Queries: 0 Mrouter Ports: 3 Dynamic: Neighbor 10.19.19.19 pw-id 215 Dynamic: GigabitEthernet0/1/0/30.215 Dynamic: GigabitEthernet0/1/0/7.215 STP Forwarding Ports: 0 ICCP Group Ports: 0 Groups: 6 Member Ports: 6 V3 Source Groups: 0 Static/Include/Exclude: 0/0/0 Member Ports (Include/Exclude): 0/0 Traffic Statistics (elapsed time since last cleared 5w4d): Received Reinjected Generated Messages: 827226 411703 1599


OL-23591-02


IGMP General Queries: 142074 124872 4 IGMP Group Specific Queries: 12 0 0 IGMP G&S Specific Queries: 0 0 0 IGMP V2 Reports: 452290 286719 0 IGMP V3 Reports: 0 0 3 IGMP V2 Leaves: 159 112 4 IGMP Global Leaves: 2 - 1588 PIM Hellos: 232689 0 - Rx Packet Treatment: Packets Flooded: 124874 Packets Forwarded To Members: 0 Packets Forwarded To Mrouters: 286841 Packets Consumed: 398308 Reports Suppressed: 165570 Rx Errors: Packets Missing Router Alert: 452268 Leaves Non-Member: 109 Rx Other: None Tx Errors: No Querier in BD: 5 Startup Query Sync Statistics: None

6. This example shows that packets are being received and transmitted, and that there are no failed packets.

RP/0/RSP0/CPU0:router# show l2snoop statistics pcb all location 0/RSP0/CPU0Tue Jan 4 10:48:17.423 PST

Statistics for PCB 0x50026960 Send: 0 packets received from application 0 xipc pulse received from application 0 packets sent to network (NetIO) 0 packets failed getting queued to network (NetIO)Rcvd: 238504 packets received from network 238504 packets queued to application 0 packets failed queued to application

Statistics for PCB 0x50024fe0 Send: 424768 packets received from application 423062 xipc pulse received from application 424768 packets sent to network (NetIO) 0 packets failed getting queued to network (NetIO)Rcvd: 611715 packets received from network 611715 packets queued to application 0 packets failed queued to application

7. This example shows the IGMP_SNOOP control packet counters on the network processors (NPs).

RP/0/RSP0/CPU0:router# show controllers np counters np2 location 0/1/CPU0Mon Jan 3 12:10:40.215 PST

Node: 0/1/CPU0:----------------------------------------------------------------


Read 56 non-zero NP counters:Offset Counter FrameValue Rate (pps)------------------------------------------------------------------------------- 22 PARSE_ENET_RECEIVE_CNT 163222344 43 23 PARSE_FABRIC_RECEIVE_CNT 77489373 22 24 PARSE_LOOPBACK_RECEIVE_CNT 114662 0


OL-23591-02

Chapter 11 Troubleshooting Multicast Services Troubleshooting Native Multicast Routing (Layer 3)

29 MODIFY_FABRIC_TRANSMIT_CNT 75222687 22 30 MODIFY_ENET_TRANSMIT_CNT 146294912 42 31 PARSE_INGRESS_DROP_CNT 17654881 1 33 RESOLVE_INGRESS_DROP_CNT 44 0 34 RESOLVE_EGRESS_DROP_CNT 407520 0 112 DIAGS 56853 0 134 IGMP_SNOOP 3 0 <<< snoop 148 IPV4MC_DO_ALL 5860 0 149 IPV4MC_DO_ALL_EXCD 13590 0 170 PUNT_IFIB 69828282 20 172 PUNT_ADJ 2 0 224 PUNT_STATISTICS 5667050 2 225 PUNT_STATISTICS_EXCD 1 0 226 PUNT_DIAGS_RSP_ACT 57337 0 230 NETIO_RP_TO_LC_CPU_PUNT 1011 0 313 BFD_NOT_ENABLED 24 0...

Troubleshooting Native Multicast Routing (Layer 3)This section explains how to troubleshoot native multicast. Native multicast routing (also called IP multicast) is a bandwidth-conserving technology that reduces traffic by simultaneously delivering a single stream of information to thousands of corporate recipients and homes.

This section contains the following subsections:


• Multicast PIE Installation Fails, page 11-262

• Multicast CLI Unavailable Although PIE Is Installed, page 11-263

• “This command not authorized” Error Message, page 11-263

• Dynamic IGMP Failure, page 11-263

• Traffic Fails on Some Interfaces, page 11-267

• Traffic Fails on Some Interfaces—MGID, page 11-268

• Throughput Loss at Receiver Interfaces, page 11-268

• Reverse Path Forwarding IP Address Problems, page 11-268

Using show and debug CommandsThis section explains how to use the show and debug commands.

Figure 11-1 illustrates the flow of information in the native multicast process, along with some of the important show commands.

• On the RSP card, the IGMP and PIM send control packet information, including joins, queries, reports, leaves and join-prune, to the MRIB. The MRIB populates the global routing tables and allocates multicast group IDs (MGIDs).

• On the LC, the MFIB receives routing information from the MRIB and programs the necessary tables and structures in the hardware (network processors or NP) and sets up the multicast routes and groups.


OL-23591-02


Figure 11-1 Native Multicast Process and Corresponding CLI Commands

SUMMARY STEPS

1. show igmp {global-interface | groups | | interface | nsf | old-output | snooping | ssm | summary | traffic | vrf name}

2. show pim [vrf vrf-name] {bgp-safi | bsr | context | df | global | group-map | interface | ipv4 | ipv6 | join-prune | ma | mdt | mstatic | multicast | neighbor | nsf | old-output | range-list | rpf | safi-all | summary | table-context | topology | traffic | tunnel | unicast}

3. show mrib [vrf vrf-name] {client | ipv4 | ipv6 | label-table-info | mdt-interface | mpls | nsf | old-output | platform | route | route-collapse | table-info |tlc }

4. show mfib [vrf vrf-name] {bundle-hash | connections | counter | encap-info | hardware | interface | ipv4 | ipv6 | lsm | mdt | nsf | route | svd | table-info}

5. show mfib hardware {adjacency | connection | interface | ltrace | resource-counters | route | table} location node-id

6. show mfib hardware route { accept-bitmap | internal | mofrr | olist | statistics | summary } {* | A.B.C.D | A.B.C.D/length | detail | hex-dump} location node-id

7. show mfib hardware route summary location node-id

8. debug mrib errors

9. debug mrib events

10. debug mfib warning

11. debug mfib errors

RSP-0

Line-Card

IGMP PIM

show pim group-mapshow pim topology route-countshow pim neighborshow pim summary

show controller np counters allshow controller np struct <id> allshow controller np summary all

show mfib routeshow mfib connectionsshow mfib countershow mfib nsfshow mfib hardware route olist loc <>show mfib hardware connection loc <>show mfib hardware ltrace loc <>show mfib hardware interface loc <>

show mrib client filtershow mrib route summaryshow mrib nsf

show igmp trafficshow igmp group summaryshow igmp interface

NP

MRIB

MFIB

2822

76


OL-23591-02


12. debug mlib errors

13. debug mlib warning

DETAILED STEPS


Step 1 show igmp {global-interface | groups | interface | nsf | old-output | ranges | snooping | ssm | summary | traffic | vrf name}

Example:RP/0/RSP0/CPU0:router# show igmp groups

View all Internet Group Management Protocol (IGMP)-related information in the control plane. IGMP is a protocol used by IPv4 systems to report IP multicast memberships to neighboring multicast routers.

Filter results using the following parameters and keywords:

• global-interface—IGMP global Interface Descriptor Block (IDB) data structures, IDBs have information like IP addresses, interface states, and packet statistics. There is one IDB for each interface and one for each subinterface.

• groups—IGMP group memberships.

• interface—IGMP interface information.

• nsf—Current multicast NSF state for IGMP, either normal or activated for NSF. The latter state indicates that recovery is in progress due to an IGMP failure. The total NSF timeout and time remaining are displayed until NSF expiration.

• old-output—Provides backward compatibility.

• ranges—IGMP group-map ranges

• snooping—IGMP snooping parameters.

• ssm—Source Specific Multicast (SSM)-related information.

• summary—IGMP summary.

• traffic—IGMP traffic counters.

• vrf name—Specify a Virtual Private Network (VPN) routing and forwarding (VRF).


OL-23591-02


Step 2 show pim [vrf vrf-name]{bgp-safi | bsr | context | df | global | group-map | interface | ipv4 | ipv6 | join-prune | ma | mdt | mstatic | multicast | neighbor | nsf | old-output | range-list | rpf | safi-all | summary | table-context | topology | traffic | tunnel | unicast}

Example:RP/0/RSP0/CPU0:router# show pim neighbor

View Protocol Independent Multicast (PIM)-related information in the control plane. Filter results using the following parameters and keywords:

• bgp-safi—Border Gateway Protocol (BGP) secondary address family (SAFI) database.

• bsr—PIM Bootstrap Router (BSR) information.

• context—PIM VRF Contexts.

• df—Bidirectional Designated Forwarder (DF).

• global—PIM global summary

• group-map—PIM group-to-protocol mapping information.

• interface—PIM interface information.

• ipv4—IPv4 Address Family.


• join-prune—PIM Join/Prune information.

• ma—PIM Management Agent information

• mdt—Data MDT information.

• mstatic— Multicast Static Route information.

• multicast—SAFI Multicast.

• neighbor—PIM neighbor information.

• nsf—Non-stop forwarding.

• old-output—Provides backward compatibility.

• range-list—PIM range-list information.

• rpf—RPF information.

• safi-all—SAFI wildcard.

• summary—PIM summary information.

• table-context—PIM Table context.

• topology—PIM topology table information.

• traffic—PIM traffic counters.

• tunnel—Tunnel interfaces.

• unicast—SAFI Unicast.

• vrf—VRF. If you include vrf, you must include it immediately after show pim, and also specify a vrf-name.



OL-23591-02


Step 3 show mrib [vrf vrf-name] {client | ipv4 | ipv6 | label-table-info | mdt-interface | mpls | nsf | old-output | platform | route | route-collapse | table-info |tlc}

Example:RP/0/RSP0/CPU0:router# show mrib client

View Multicast Routing Information Base (MRIB) information. Filter results using the following parameters and keywords:

• client—MRIB client connections.

• ipv4—IPv4 address family

• ipv6—IPv6 address family

• label-table-info—MRIB label table information

• mdt-interface—MDT interface handle DB

• mpls—MRIB MPLS related information

• nsf—Non-stop forwarding

• old-output—Display the old show output

• platform—Platform-specific data

• route—Routing database.

• route-collapse—MRIB route collapse database

• table-info—MRIB VRF table information.

• tlc—MRIB table-linecard database

• vrf—VRF. If you include vrf, you must include it immediately after show mrib, and also specify a vrf-name.



OL-23591-02


Step 4 show mfib [vrf vrf-name] {bundle-hash | connections | counter | encap-info | hardware | interface | ipv4 | ipv6 | lsm | mdt | nsf | route | svd | table-info}

Example:RP/0/RSP0/CPU0:router# show mfib nsf

View Multicast Forwarding Information Base (MFIB) information in the control plane. Filter results using the following parameters and keywords:

• bundle-hash—Bundle hash for given Interface,S,G tuple

• connections—Status of MFIB connections to servers.

• counter—MFIB global counters.

• encap-info—Multicast Virtual Private Network (MVPN) Encap information.

• hardware—Cisco ASR 9000 Series Router hardware.

• interface—MFIB interface specific information.



• lsm—Label Switched Multicast.

• mdt—MDT tunnel information.

• nsf—Multicast NSF status.

• route—Routing database.

• svd—Singular Value Decomposition (SVD) events.

• table-info—Table information.

• vrf—VRF. If you include vrf, you must include it immediately after show mfib, and also specify a vrf-name or all.

Step 5 show mfib hardware {interface | ltrace | resource-counters | route} location node-id

Example:RP/0/RSP0/CPU0:router# show mfib hardware route olist location 0/4/CPU0

View all hardware data in the Multicast PD. Filter results using the following parameters and keywords:

• interface—Cisco ASR 9000 Series Router hardware.

• ltrace—IP Multicast platform specific trace information.

• resource-counters—Allocated and freed hardware resources.

• route—Platform-specific information for the routing database.

• location—Specify the MFIB location.

Note The output of these commands can be large when there are a large number of routes and output interface lists (olists) configured.



OL-23591-02


Multicast PIE Installation Fails

Step 1 View detailed information for the specified install id. RP/0/RSP0/CPU0:router# show install log [1-4294967295] detail

Step 6 show mfib hardware route {accept-bitmap | internal | mofrr | olist | statistics | summary} {* | A.B.C.D | A.B.C.D/length | detail | hex-dump} location node-id

Example:RP/0/RSP0/CPU0:router# show mfib hardware route olist location 0/4/CPU0

View all hardware ROUTE data in the multicast PD. Filter results using the following parameters and keywords:

• accept-bitmap—Accepting interface list for bidir routes.

• internal—Display route internal structures

• mofrr—Display per-route MoFRR information

• olist—Output interface list (olist) stored in the hardware.

• statistics—Per route packets and bytes counters.

• summary—Summary of routes.

• *—Shared tree entries.

• A.B.C.D—Source/group IP address.

• A.B.C.D/length—Group IP address/prefix length.

• detail—Details of each route (requires 140 columns).

• hex-dump—Hex dump of the PLU and TLU.

• location—Specify the MFIB location.

Note The output of these commands can be large when there are a large number of routes and olists configured.

Step 7 show mfib hardware route summary location node-id

Example:RP/0/RSP0/CPU0:router# show mfib hardware route summary location 0/4/CPU0

View all hardware ROUTE data in the multicast PD. Filter results using the following parameters and keywords:

• summary—Summary of routes.

• location—MFIB location.

Step 8 debug mrib errors

Example:RP/0/RSP0/CPU0:router# debug mrib errors

To monitor Multicast Routing Information Base (MRIB) internal errors, use the debug mrib errors command in EXEC mode. To disable debugging output, use the no form of this command.

Step 9 debug mrib events Use these debug commands to obtain additional information.Step 10 debug mfib warning

Step 11 debug mfib errors

Step 12 debug mlib errors

Step 13 debug mlib warning



OL-23591-02


Step 2 Ensure that the PIE file name is correct and reissue the command.

Step 3 Ensure that the location of the PIE file is correct and reissue the command

Step 4 Ensure that the PIE file has proper permissions (755) and reissue the command.

Step 5 If you are loading from the TFTP directory, ensure that the following are true:

a. Router has network connectivity.

b. TFTP address is properly configured.

c. TFTP server has connectivity. RP/0/RSP0/CPU0:router# ping tftp-server-addr

d. If loading locally from a router, ensure that the PIE file is stored on the router.

Step 6 Verify all nodes are in the “IOS XR RUN State” RP/0/RSP0/CPU0:router# show platform

Multicast CLI Unavailable Although PIE Is Installedshow install active—View active package information

Ensure that the following are correct:

• PIE file name.

• PIE file location.

• PIE file installation on ALL nodes.

“This command not authorized” Error MessageWhile issuing certain commands in config or EXEC mode, the “This command not authorized” error message appears, disabling further access. This means the user does not have the appropriate privileges. Check to see that you have “Cisco-Support” and ‘root” privileges to use the desired command.

show config run—(from admin mode) View current operating admin configuration of the system.

Dynamic IGMP FailureA dynamic IGMP failure occurs when the dynamic source and group states (*,G) are timing out. There are two scenarios that suggest you need to troubleshoot this problem:

• IGMP join messages sent from a host does not resul in creation of a new route or addition of an OLIST member; similarly, IGMP leave messages sent from a host do not result in deletion of an existing route or removal of an existing OLIST member.

• The groups and routes are configured and set up correctly, but when traffic is sent to the Cisco ASR 9000 Aggregation Series Router from the tester, it is not received at the Rx tester port.

Step 1 Verify that IGMP packets are being received by the IGMP process on the RSP.


OL-23591-02


RP/0/RSP0/CPU0:router# show ipv4 traffic Thu Jan 6 10:28:24.622 PST

IP statistics: Rcvd: 294930104 total, 3052259 local destination 0 format errors, 0 bad hop count 593296 unknown protocol, 0 not a gateway 0 security failures, 0 bad source, 369473 bad header 1453418 with options, 0 bad, 0 unknown Opts: 0 end, 0 nop, 0 basic security, 0 extended security 0 strict source rt, 0 loose source rt, 0 record rt 0 stream ID, 0 timestamp, 1453418 alert, 0 cipso Frags: 0 reassembled, 0 timeouts, 0 couldn't reassemble, 0 fragments received 0 fragmented, 0 fragment count, 0 fragment max drop Bcast: 0 sent, 0 received Mcast: 9878922 sent, 291744628 received Drop: 0 encapsulation failed, 3 no route, 0 too big Sent: 13116263 total...

RP/0/RSP0/CPU0:router# show igmp traffic Thu Jan 6 10:30:27.821 PST

IGMP Traffic CountersElapsed time since counters cleared: 6w0d

Received SentValid IGMP Packets 491447 777437Queries 368364 246191Reports 123083 531246Leaves 0 0Mtrace packets 0 0DVMRP packets 0 0PIM packets 0 0

Errors:Malformed Packets 0Bad Checksums 0Socket Errors 0Bad Scope Errors 0Auxiliary Data Len Errors 0Packets dropped due to invalid socket 0Packets which couldn't be accessed 0Packet allocation failure 0Other packets drops 0

Step 2 Verify that IGMP has the specified group/source on the interface.

RP/0/RSP0/CPU0:router# show igmp interface gigabitEthernet 0/1/0/2 Thu Jan 6 10:34:30.664 PST

GigabitEthernet0/1/0/2 is up, line protocol is up Internet address is 10.147.4.44/24 IGMP is enabled on interface Current IGMP version is 3 IGMP query interval is 60 seconds IGMP querier timeout is 125 seconds IGMP max query response time is 10 seconds Last member query response interval is 1 seconds IGMP activity: 6 joins, 0 leaves IGMP querying router is 10.147.4.44 (this system)


OL-23591-02


Step 3 Verify that the multicast routing information base (MRIB) has the expected route and outgoing interface list (OLIST).

RP/0/RSP0/CPU0:router# show mrib route Thu Jan 6 10:38:42.800 PST

IP Multicast Routing Information BaseEntry flags: L - Domain-Local Source, E - External Source to the Domain, C - Directly-Connected Check, S - Signal, IA - Inherit Accept, IF - Inherit From, D - Drop, MA - MDT Address, ME - MDT Encap, MD - MDT Decap, MT - MDT Threshold Crossed, MH - MDT interface handle CD - Conditional Decap, MPLS - MPLS Decap, MF - MPLS Encap, EX - Extranet MoFE - MoFRR Enabled, MoFS - MoFRR StateInterface flags: F - Forward, A - Accept, IC - Internal Copy, NS - Negate Signal, DP - Don't Preserve, SP - Signal Present, II - Internal Interest, ID - Internal Disinterest, LI - Local Interest, LD - Local Disinterest, DI - Decapsulation Interface EI - Encapsulation Interface, MI - MDT Interface, LVIF - MPLS Encap, EX - Extranet, A2 - Secondary Accept

(*,10.66.66.66) RPF nbr: 10.114.8.11 Flags: C Up: 6w0d Incoming Interface List TenGigE0/4/0/0 Flags: A, Up: 6w0d Outgoing Interface List GigabitEthernet0/1/0/8 Flags: F NS, Up: 6w0d

(10.191.4.1,232.166.166.166) RPF nbr: 10.114.8.11 Flags: MoFE MoFS Up: 3w5d MOFRR State: Active Sequence No 2856 Incoming Interface List TenGigE0/4/0/0 Flags: A, Up: 3w5d GigabitEthernet0/1/0/23 Flags: A2, Up: 3w5d Outgoing Interface List GigabitEthernet0/1/0/8 Flags: F NS, Up: 3w5d

(*,10.60.0.0/16) RPF nbr: 10.144.144.144 Flags: IF Up: 6w0d Incoming Interface List Loopback0 Flags: F A, Up: 6w0d TenGigE0/4/0/0 Flags: A, Up: 6w0d GigabitEthernet0/1/0/18 Flags: A, Up: 3w5d GigabitEthernet0/1/0/23 Flags: A, Up: 6w0d GigabitEthernet0/1/0/27 Flags: A, Up: 2w5d Outgoing Interface List Loopback0 Flags: F A, Up: 6w0d

Step 4 Verify that the multicast forwarding information base (MFIB) on the LC has the interface as an OLIST member.

RP/0/RSP0/CPU0:router# show mfib route location 0/1/CPU0 Thu Jan 6 10:47:06.989 PST

IP Multicast Forwarding Information BaseEntry flags: C - Directly-Connected Check, S - Signal, D - Drop, IA - Inherit Accept, IF - Inherit From, MA - MDT Address, ME - MDT Encap, MD - MDT Decap, MT - MDT Threshold Crossed, MH - MDT interface handle, CD - Conditional Decap, DT - MDT Decap True, EX - Extranet MoFE - MoFRR Enabled, MoFS - MoFRR StateInterface flags: F - Forward, A - Accept, IC - Internal Copy, NS - Negate Signal, DP - Don't Preserve, SP - Signal Present, EG - Egress, EI - Encapsulation Interface, MI - MDT Interface, EX - Extranet, A2 - Secondary Accept


OL-23591-02


Forwarding/Replication Counts: Packets in/Packets out/Bytes outFailure Counts: RPF / TTL / Empty Olist / Encap RL / Other

(*,10.0.0.0/4), Flags: C Up: 6w0d Last Used: never SW Forwarding Counts: 0/0/0 SW Replication Counts: 0/0/0 SW Failure Counts: 0/0/0/0/0

(*,10.66.66.66), Flags: C Up: 6w0d Last Used: never SW Forwarding Counts: 0/0/0 SW Replication Counts: 0/0/0 SW Failure Counts: 0/0/0/0/0 TenGigE0/4/0/0 Flags: A, Up:6w0d GigabitEthernet0/1/0/8 Flags: NS EG, Up:6w0d

Step 5 Verify that the MFIB in the hardware has the interface as an OLIST member.

RP/0/RSP0/CPU0:router# show mfib hardware interface gigabitEthernet 0/1/0/27 location 0/1/CPU0 Thu Jan 6 10:55:27.723 PSTLC Type: A9K-40GE-L--------------------------------------------------------------------Interface Handle RefCnt TTL Routes uIDB Enbld Comment--------------------------------------------------------------------Gi0/1/0/27 0x2000740 8 0 3 55 True success--------------------------------------------------------------------ROUTE INFORMATION:Legend: S: Source, G: Group, P: Prefix length, PI: Packets cn, PO: packets out, RF: RPF failures, TF: TTL failures, OF: OLIST failures, F: Other failuresRoute flags - (Ingress) C: Chip ID, IC: BACL check, IP: Punt this packet to LC CPU, ID: Directly connected, IS: RPF interface signal, IU: Punt copy to RP, IF: Punt to LC CPU if forwarded, IM: Result match, IV: Valid entry, IR: RPF IF, IA: Fabric slotmask, IG: Mulicast group IDRoute flags - (Egress) ET: Table ID to be used for OLIST lookup, EO: OLIST count bit, ER: Route MGID to be used for OLIST/NRPF lookup, EM: Result match, EV: Valid entry, EC: Count of OLIST members on this chip, BS: Base of the statistics pointer

Interface: Gi0/1/0/27

S:0.0.0.0 G:10.60.0.0 P:16 PI:1784 PO:0 RF:670 TF:0 OF:0 F:0 --------------------------------------------------------------------------------- C IC IP ID IS IU IF IM IV IR IA IG ET EO ER EM EV EC BS --------------------------------------------------------------------------------- 0 T F F F F F T T 0x2000740 0x0 0x4208 0 F 4 T T 0 0x36c6c 1 T F F F F F T T 0x2000740 0x0 0x4208 0 F 4 T T 0 0x36b04 2 T F F F F F T T 0x2000740 0x0 0x4208 0 F 4 T T 0 0x36b1c 3 T F F F F F T T 0x2000740 0x0 0x4208 0 F 4 T T 0 0x36d44 ---------------------------------------------------------------------------------

S:0.0.0.0 G:10.60.66.66 P:32 PI:113837 PO:999688227 RF:198 TF:0 OF:0 F:0 --------------------------------------------------------------------------------- C IC IP ID IS IU IF IM IV IR IA IG ET EO ER EM EV EC BS --------------------------------------------------------------------------------- 0 T F F F F F T T 0x0 0x2 0x420b 0 F 7 T T 0 0x36c7b 1 T F F F F F T T 0x0 0x2 0x420b 0 T 7 T T 1 0x36b13 2 T F F F F F T T 0x0 0x2 0x420b 0 F 7 T T 0 0x36b2b


OL-23591-02


3 T F F F F F T T 0x0 0x2 0x420b 0 F 7 T T 0 0x36d53 ---------------------------------------------------------------------------------

S:0.0.0.0 G:10.60.64.64 P:32 PI:84205 PO:973945464 RF:200 TF:0 OF:0 F:0 --------------------------------------------------------------------------------- C IC IP ID IS IU IF IM IV IR IA IG ET EO ER EM EV EC BS --------------------------------------------------------------------------------- 0 T F F F F F T T 0x0 0x2 0x420a 0 F 6 T T 0 0x36c76 1 T F F F F F T T 0x0 0x2 0x420a 0 T 6 T T 1 0x36b0e 2 T F F F F F T T 0x0 0x2 0x420a 0 F 6 T T 0 0x36b26 3 T F F F F F T T 0x0 0x2 0x420a 0 F 6 T T 0 0x36d4e ---------------------------------------------------------------------------------

Step 6 One possible cause could be that the IGMP group is timing out. One way to check is to create a static route for the (*,G) and see if traffic is now received. If it is, it means that the groups are timing out.

Step 7 To confirm the result from Step 6, remove the static route and decrease the query interval (resulting in more queries per minute) to make it clearer: RP/0/RSP0/CPU0:router# conf RP/0/RSP0/CPU0:router(config)# router igmp RP/0/RSP0/CPU0:router(config-igmp)# query-interval 1 RP/0/RSP0/CPU0:router(config-igmp)# commit

Step 8 Ensure that packets are going out of interface at the interval set.

Step 9 Check that the tester responds with an IGMP membership report. If the packets are received at the tester, the result from Step 6 is confirmed. Use the workaround.

Workaround

Configure static groups as a temporary workaround.

Traffic Fails on Some InterfacesTraffic is failing on some interfaces or channels. You determine that

• Groups and routes are configured and set up correctly.

• Traffic is sent to the ASR 9000 from the tester.

• Traffic is received correctly on some interfaces but not on others or some video channels are received correctly on an interface while others are not.

Possible causes could be:

• OLIST may not be properly configured.

• UIDB values not correctly set in hardware.

• MGID not correctly set up.

Step 1 Ensure that packets are going from the ingress network processor (NP) through the fabric to the egress NP. RP/0/RSP0/CPU0:router# show mfib hardware route statistics location {ingress node-id |

egress node-id }


OL-23591-02


Step 2 View olist interfaces for the route. RP/0/RSP0/CPU0:router# show mfib hardware route olist location {ingress node-id | egress

node-id }

Step 3 View the statistics for the specific route and source. RP/0/RSP0/CPU0:router# show mfib hardware route stat [src ip addr] location {ingress

node-id | egress node-id }

Traffic Fails on Some Interfaces—MGID

Step 1 Ensure that packets are going from the ingress NP through the fabric to the egress NP. RP/0/RSP0/CPU0:router# show mfib hardware route statistics location {ingress node-id |

egress node-id }

Step 2 View olist interfaces for the route. RP/0/RSP0/CPU0:router# show mfib hardware route olist location {ingress node-id | egress

node-id }

Step 3 Ensure that packets are transmitted out of ingress NP to fabric and received by egress NP from the fabric.

RP/0/RSP0/CPU0:router# show mfib hardware route statistics location {ingress node-id | egress node-id }

Step 4 View the MGID for the route.

Throughput Loss at Receiver InterfacesTraffic is sent and received on routes but there is a loss of throughput at the receiver.

Step 1 Ensure that packets are going from the ingress NP through the fabric to the egress NP.

RP/0/RSP0/CPU0:router# show mfib hardware route statistics location {ingress node-id | egress node-id}

Step 2 The preceding command checks if if packets are punted to the RP. If so, check if the source of that channel is setting some IP options or not.

Reverse Path Forwarding IP Address ProblemsReverse Path Forwarding (RPF) ensures loop-free forwarding of multicast packets in multicast routing. This section contains the following subsections:


• Packets from Wrong IP Address—Loose RPF, page 11-269


OL-23591-02


• Packets Forwarded with Wrong IP Address—Strict RPF, page 11-269


show cef ipv4 interface—View IPv4 Cisco Express Forwarding (CEF)-related information for an interface.

Packets from Wrong IP Address—Loose RPF

In loose RPF, the packets incoming on that particular interface are checked to determine if the source IP of the packet is reachable through some interface on the box. If not, the packet is dropped.

RP/0/RSP0/CPU0:router# show cef {ipv4} prefix hardware egress detail location node-id

Workaround

Unconfigure and configure loose RPF on the interface.

Packets Forwarded with Wrong IP Address—Strict RPF

In strict RPF, the packets incoming on that particular interface are checked to determine if the source IP of the packet is reachable through the same interface on the box on which the packet came in. If not, the packet is dropped.

Verify non-null rpf_ptr and uidb list with the show cef command.

RP/0/RSP0/CPU0:router# show cef {ipv4 | ipv6} prefix hardware egress detail location node-id

Workaround

Unconfigure and configure strict RPF on the interface.


OL-23591-02



OL-23591-02


I N D E X

A

access control list (ACL) 10-241

access privileges 1-2

adjacency 4-101

alarm indication signal (AIS) 3-93

ARP 3-75

ASIC errors 1-54

B

bidirectional forwarding detection 3-81

C

CCM 3-89

CEF troubleshooting 4-95

CFM 3-85

cfs check command 1-22

cisco-support task ID 1-3

Cisco Technical Support 1-58

CLI access 1-2

commit confirmed command 1-20

connectivity fault management (CFM) 3-85

Continuity check messages 3-89

control plane 4-109

control plane Ethernet network

overview 1-41

counters, NP 7-148

crashes, RSP and LC 7-165

crosscheck for MEPs 3-92

D

describe command 1-49

describe hostname command 1-21

diagnostic commands 1-59

documentation

prerequisite for troubleshooting 1-1

dynamic IGMP failure 11-263

E

Ethernet CFM 3-85

Ethernet Connectivity Fault Management 3-85

F

fabric 4-109, 7-143

forwarding information base (FIB) 4-95

G

gathering information 1-58

I

IGMP reports dropped 11-251

IGMP snooping 11-247

IGMP snooping querier 11-251

installation, software 1-7, 1-10

install verify command 1-10

interfaces 2-61

interfaces, connectivity 3-75

interfaces, optical 2-68

IN-1or the Cisco ASR 9000 Aggregation Services Router

Index

IP multicast 11-256

L

L2VPN 9-181

Layer 3 routing problems 6-125

M

MAC address updates 9-192

maintenance domains 3-85

Management Ethernet interface 1-4

man command 1-46

memory 1-38

MEP crosscheck 3-92

MEP defects 3-90

MEPs and MIPs 3-88

MIB download 1-57

MPLS 8-173

multicast 11-247

Multiple Spanning Tree (MST) 9-230

multipoint Layer 2 bridging services (VPLS) 9-195

N

native multicast routing 11-256

network

documenting 1-1

NP counters 7-148

O

optical line card 2-68

P

packet drops 7-158

ping 3-75

IN-2Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggreg

point-to-point Layer 2 deployment 9-206

prerequisite documentation 1-1

prompt, router 1-2

punted packets 6-129, 7-155

Q

QoS features 10-233

R

RDI 3-91

remote defect indication 3-91

router prompt 1-2

RSP switchover 6-138

S

show adjacency command 4-102

show adjacency detail hardware command 4-102

show adjacency hardware trace location command 4-103

show adjacency ipv4 nexthop command 4-102

show adjacency ipv4 nexthop detail hardware command 4-102

show adjacency remote detail command 4-101

show adjacency remote detail hardware command 4-102

show adjacency trace client command 4-103

show adjacency trace command 4-103

show arp command 1-5, 4-101

show arp traffic location command 4-101

show asic-errors command 1-54

show cef adjacency tunnel-te command 4-103

show cef ipv4 detail command 4-97

show cef ipv4 detail location command 4-98

show cef ipv4 hardware egress command 4-98

show cef ipv4 hardware ingress command 4-98

show cef ipv4 interface command 4-98

show cef ipv4 summary command 4-99

show cef ipv4 trace command 4-99

ation Services Router OL-23591-02

Index

show cef platform trace ipv4 all command 4-99

show cfgmgr trace command 1-22

show configuration commit changes command 1-19, 1-22

show configuration commit list command 1-20

show configuration failed command 1-24

show configuration failed startup command 1-22

show configuration history commit command 1-22

show context command 1-28, 1-50

show controllers backplane ethernet clients all command 1-44

show controllers backplane ethernet local clients statistics command 1-44

show controllers stats command 2-62

show environment command 1-28

show ethernet cfm configuration-errors command 3-87

show history command 1-52, 1-53

show hw-module subslot address status pluggable-optics 2-68

show hw-module subslot command 2-62

show imds interface brief 4-110

show install active command 1-12

show install command 1-8

show install committed command 1-12, 1-14

show interface brief command 1-29

show interface command 2-62

show interfaces brief command 1-3

show interfaces command 1-5

show ipv4 interface command 1-5

show logging command 1-27, 1-59

show memory 1-38

show memory heap command 1-28

show memory summary command 1-28

show netio idb command 2-62, 4-110

show platform command 1-27, 1-44, 1-49

show route ipv4 command 4-97

show running-config 1-21

show running-config command 1-16, 1-27, 1-28

show sysdb trace command 1-18

show sysdb trace verification command 1-22

show system verify command 1-29, 1-59

Cisco IOS XR TroubleshootinOL-23591-02

show tech-support command 1-59

show uidb data command 4-110

show uidb index command 4-110

show users command 1-52

show version command 1-8, 1-27

software installation 1-7, 1-10

software version 1-8

sohw hw-module subslot counters framer command 2-62

startup failed 1-24

T

TAC

gathering information 1-58

TCAM classification 9-182

top command 1-50

top processes command 1-28

trace 1-56

tracebacks 6-133, 6-136

trace commands 1-56

traceroute 6-130

traffic drop, transient 4-106

traffic engineering tunnel 8-176

traffic loss 7-168

transport input telnet command 1-4

tunnel, TE 8-176

U

user access 1-2

V

version 1-8

VLAN traffic 9-182

VPLS 9-195

VPWS deployment 9-206

VRRP 6-138

IN-3g Guide for the Cisco ASR 9000 Aggregation Services Router

Index

W

WRED weighted random early detection 10-237

IN-4Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggreg
ation Services Router
OL-23591-02

Documents

Cisco IOS XR Troubleshooting Guide for the Cisco …rabdoul.free.fr/SPADVROUTE12/MCAST/XR Troubleshooting...Contents v Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation