n7kts

This article provides guidance for troubleshooting issues that may appear when using Cisco Nexus 7000 Series. This articleintroduces tools and methodologies to recognize a problem, determine its cause, and find possible solutions. However, thesedocumentation helps only in basic troubleshooting.

We encourage users to review the Cisco Live Presentation for detailed troubleshooting for Nexus 7000.

Sections of this presentation covers, both platform independent, and platform specific step by step troubleshooting for mostcommon issues. Access to this presentation is available FREE. Follow the below instructions to access the presentation.

1. Visit https://www.ciscolivevirtual.com/

2. Register for free.

3. Click on "Cisco Live Virtual" link.

4. Click on the ?Sessions? Tab on top, and select ?2011 Sessions Catalog?

5. In the search box, type ?BRKCRS-3144? and Submit search.

6. Select the session. You can either View the Session (or) download the pdf

Welcome to Cisco DocWiki. We encourage registered Cisco.com users to contribute to this wiki to improve Cisco product documentation. Note that youcannot log in to DocWiki with Cisco.com "guest" account credentials.

See Terms of Use and About DocWiki for more information about Cisco DocWiki.Select the "edit" tab to edit an article or select the "discussion" tab to submit questions or comments about the article.

Click here to return to the Cisco Nexus 7000 Series documentation on www.cisco.com.

Contents1 Audience and Generating a PDF of ThisGuide

2 Organization

Audience and Generating a PDF of This GuideThis article is for experienced network administrators who configure and maintain NX-OS devices.

Cisco Nexus 7000 Series NX-OS Troubleshooting Guide -- Book PDF

OrganizationThis article is organized into the following sections:

Troubleshooting Overview

Troubleshooting Installs, Upgrades, and Reboots

Troubleshooting Licensing

Cisco Nexus 7000 Series NX-OS Troubleshooting Guide

Contents 1

Troubleshooting VDCs

Troubleshooting CFS

Troubleshooting Ports

Troubleshooting vPCs

Troubleshooting VLANs

Troubleshooting STP

Troubleshooting Routing

Troubleshooting Unicast Traffic

Troubleshooting Multicast Traffic

Troubleshooting WCCP

Troubleshooting Memory

Troubleshooting Packet Flow Issues

Before Contacting Technical Support

Troubleshooting Tools and Methodology

This article introduces the basic concepts, methodology, and general troubleshooting guidelines for problems that may occur whenconfiguring and using Cisco NX-OS.

We encourage users to review the Cisco Live Presentation for detailed troubleshooting for Nexus 7000.

Sections of this presentation covers, both platform independent, and platform specific step by step troubleshooting for mostcommon issues. Access to this presentation is available FREE. Follow the below instructions to access the presentation.

1. Visit https://www.ciscolivevirtual.com/

2. Register for free.

3. Click on "Cisco Live Virtual" link.

4. Click on the ?Sessions? Tab on top, and select ?2011 Sessions Catalog?

5. In the search box, type ?BRKCRS-3144? and Submit search.

Guide ContentsTroubleshooting Overview (this section)Troubleshooting Installs, Upgrades, and RebootsTroubleshooting Licensing


Organization 2

Troubleshooting VDCsTroubleshooting CFSTroubleshooting PortsTroubleshooting vPCsTroubleshooting VLANsTroubleshooting STPTroubleshooting RoutingTroubleshooting Unicast TrafficTroubleshooting WCCPTroubleshooting MemoryTroubleshooting Packet Flow IssuesTroubleshooting FCoEBefore Contacting Technical SupportTroubleshooting Tools and Methodology

Contents1 Overview of the TroubleshootingProcess

1.1 Gathering Information 1.2 Verifying Ports 1.3 Verifying Layer 2Connectivity

1.4 Verifying Layer 3Connectivity

2 Overview of Symptoms 3 System Messages

3.1 System Message Text 3.2 syslog ServerImplementation

4 Troubleshooting with Logs 5 Troubleshooting Modules 6 Viewing NVRAM logs 7 Contacting Customer Support 8 See Also 9 Further Reading 10 External Links

Overview of the Troubleshooting ProcessTo troubleshoot your network, follow these general steps:

Maintain a consistent Cisco NX-OS release across all your devices.1. See the Cisco NX-OS release notes for your Cisco NX-OS release for the latest features, limitations, and caveats.2. Enable system message logging. See System Messages.3. Troubleshoot any new configuration changes after implementing the change.4. Gather information that defines the specific symptoms. See Gathering Information.5. Verify the physical connectivity between your device and end devices. See Verifying Ports.6. Verify the Layer 2 connectivity. See Verifying Layer 2 Connectivity.7. Verify the end-to-end connectivity and the routing configuration. See Verifying Layer 3 Connectivity.8. After you have determined that your troubleshooting attempts have not resolved the problem, contact Cisco TAC or yourtechnical support representative.

9.


Contents 3

Note: View the Cisco Nexus 7000 instructional videos for an overview of Cisco NX-OS.

Gathering Information

This section describes the tools that are commonly used to troubleshoot problems within your network. Specific troubleshootingarticles may include additional tools and commands specific to the symptoms and possible problems covered in that article.

Note: You should have an accurate topology of your network to isolate problem areas. Contact your network architect for thisinformation.

Use the following commands to gather general information on your device:

show module show version show running-config show logging log show interfaces brief show vlan show spanning-tree show {ip | ipv6} routing show processes | include ER show accounting log

Verifying Ports

Answer the following questions to verify that your ports are connected correctly and are operational:

Are you using the correct media (copper, optical, fiber type)? Is the media broken or damaged? Is the port LED green on the module? Is the interface in the correct VDC?

Use the show vdc membership command to check which VDC that the interface is a member of. You must loginto the device with the network-admin role to use this command.

Is the interface operational?

Use the show interface brief command. The status should be up.

See Troubleshooting Ports for more troubleshooting tips for ports.

Verifying Layer 2 Connectivity

Use the following commands to verify Layer 2 connectivity:

Use the show vlan all-ports command to verify that all the necessary interfaces are in the same VLAN. The status shouldbe active for the VLAN.

Use the show port-channel compatibility-parameters command to verify that all the ports in a port channel areconfigured the same for the speed, the duplex, and the trunk mode.


Overview of the Troubleshooting Process 4

Use the show running-config spanning-tree command to verify that the Spanning Tree Protocol (STP) is configured thesame on all devices in the network.

Use the show processes | include ER command to verify that no essential Layer 2 processes are in the error state. Use the show spanning-tree blockedports command to display the ports that are blocked by STP. Use the show mac address-table dynamic vlan command to determine if learning or aging is occurring at each node.

See Troubleshooting VLANs and Troubleshooting STP for more information on troubleshooting Layer 2 issues.

Verifying Layer 3 Connectivity

Answer the following questions to verify Layer 3 connectivity:

Have you configured a default gateway? Have you configured the same dynamic routing protocol parameters throughout your routing domain or configured staticroutes?

Are any IP access lists, filters, or route maps blocking route updates?

Use the following commands to verify your routing configuration:

show arp show ip routing show platform forwarding

See Ping and Traceroute to verify Layer 3 connectivity. See Troubleshooting Routing for more information on troubleshootingLayer 3 issues.

Overview of SymptomsThis article uses a symptom-based troubleshooting approach that allows you to diagnose and resolve your Cisco NX-OS problemsby comparing the symptoms that you observed in your network with the symptoms listed in each chapter.

By comparing the symptoms in this publication to the symptoms that you observe in your own network, you should be able todiagnose and correct software configuration issues and inoperable hardware components so that the problems are resolved withminimal disruption to the network. Those problems and corrective actions include the following:

Identify key Cisco NX-OS troubleshooting tools. Obtain and analyze protocol traces using SPAN and RSPAN or Ethanalyzer on the CLI. Identify or rule out physical port issues. Identify or rule out switch module issues. Diagnose and correct Layer 2 issues. Diagnose and correct Layer 3 issues. Recover from switch upgrade failures. Obtain core dumps and other diagnostic data for use by Cisco TAC or your customer support representative.


Verifying Layer 2 Connectivity 5

System MessagesThe system software sends syslog (system) messages to the console (and, optionally, to a logging server on another device). Notall messages indicate a problem with your device. Some messages are purely informational, while others might help diagnoseproblems with links, internal hardware, or the device software.

System Message Text

Message text is a text string that describes the condition. This portion of the message might contain detailed information about theevent, including terminal port numbers, network addresses, or addresses that correspond to locations in the system memoryaddress space. Because the information in these variable fields changes from message to message, it is represented here by shortstrings enclosed in square brackets ([ ]). A decimal number, for example, is represented as [dec ].

PORT-3-IF_UNSUPPORTED_TRANSCEIVER: Transceiver for interface [chars] is not supported.

Use this string to find the matching system message in the NX-OS System Messages Reference.

Each system message is followed by an explanation and recommended action. The action may be as simple as "No action isrequired." It may involve a fix or a recommendation to contact technical support as shown in the following example:

Error Message PORT-3-IF_UNSUPPORTED_TRANSCEIVER: Transceiver for interface [chars] is notsupported.

Explanation Transceiver (SFP) is not from an authorized vendor.

Recommended Action Enter the show interface transceiver CLI command or similar DCNM command todetermine the transceiver being used. Please contact your customer support representative for a list of authorizedtransceiver vendors.

syslog Server Implementation

The syslog facility allows the Cisco NX-OS device to send a copy of the message log to a host for more permanent storage. Thisfeature allows you to examine the logs over a long period of time or if the Cisco NX-OS device is not accessible.

This example shows how to configure a Cisco NX-OS device to use the syslog facility on a Solaris platform. Although a Solarishost is being used, the syslog configuration on all UNIX and Linux systems is very similar.

syslog uses the facility to determine how to handle a message on the syslog server (the Solaris system in this example) and themessage severity. Different message severities are handled differently by the syslog server. They could be logged to different filesor e-mailed to a particular user. Specifying a severity level on the syslog server determines that all messages of that level andgreater severity (lower number) will be acted upon as you configure the syslog server.

Note: You should configure the syslog server so that the Cisco NX-OS messages are logged to a different file from thestandard syslog file so that they cannot be confused with other non-Cisco syslog messages. Do not locate the logfile on


System Messages 6

the / file system. You do not want log messages to fill up the / file system. This example uses the following values:

syslog client: switch1 syslog server: 172.22.36.211 (Solaris) syslog facility: local1 syslog severity: notifications (level 5, the default) File to log Cisco NX-OS messages to: /var/adm/nxos_logs

To configure the syslog feature on Cisco NX-OS, follow these steps:

switch1# config terminal1. switch1(config)# logging server 192.0.2.1 6 facility local12.

Use the show logging server command to verify the syslog configuration.

switch1# show logging server

Logging server: enabled{172.22.36.211} server severity: notifications server facility: local1 server VRF: management

To configure a syslog server, follow these steps:

1. Modify /etc/syslog.conf to handle local1 messages. For Solaris, you must allow at least one tab between the facility.severity andthe action (/var/adm/nxos_logs).

local1.notice /var/adm/nxos_logs

2. Create the log file.

touch /var/adm/nxos_logs

3. Restart the syslog process.

/etc/init.d/syslog stop/etc/init.d/syslog start

syslog service starting.

4. Verify that the syslog process has started.

ps -ef |grep syslogdroot 23508 1 0 11:01:41 ? 0:00 /usr/sbin/syslogd

Test the syslog server by creating an event in Cisco NX-OS. In this case, port e1/2 was shut down and reenabled and the followingwas listed on the syslog server. The IP address of the switch is listed in brackets.

tail -f /var/adm/MDS_logs


syslog Server Implementation 7

Sep 17 11:07:41 [172.22.36.142.2.2] : 2004 Sep 17 11:17:29 pacific: PORT-5-IF_DOWN_INITIALIZING: %$VLAN 1%$ Interface e 1/2 is down (Initializing)

Sep 17 11:07:49 [172.22.36.142.2.2] : 2004 Sep 17 11:17:36 pacific: %PORT-5-IF_UP: %$VLAN 1%$ Interface e 1/2 is up in mode access

Sep 17 11:07:51 [172.22.36.142.2.2] : 2004 Sep 17 11:17:39 pacific: %VSHD-5-VSHD_SYSLOG_CONFIG_I: Configuring console from pts/0 (dhcp-171-71-49-125.cisco.com

Troubleshooting with LogsCisco NX-OS generates many types of system messages on the device and sends them to a syslog server. You can view thesemessages to determine what events may have led up to the current problem condition that you are facing.

Use the following commands to access and view logs in Cisco NX-OS:

switch# show logging ?

console Show console logging configurationinfo Show logging configurationinternal syslog syslog internal informationip IP configurationlast Show last few lines of logfilelevel Show facility logging configurationlogfile Show contents of logfileloopback Show logging loopback configurationmodule Show module logging configurationmonitor Show monitor logging configurationnvram Show NVRAM logonboard show logging onboardpending server address pending configurationpending-diff server address pending configuration diffserver Show server logging configurationsession Show logging session statusstatus Show logging statustimestamp Show logging timestamp configuration

This example shows the output of the show logging server command:

switch# show logging server

Logging server: enabled{172.28.254.254} server severity: notifications server facility: local7 server VRF: management

Troubleshooting ModulesYou can directly connect to a module console port to troubleshoot module bootup issues. Use the attach console modulecommand to connect to the module console port.

Viewing NVRAM logsSystem messages that are priority 0, 1, or 2 are logged into NVRAM on the supervisor module. After a switch reboots, you candisplay these syslog messages in NVRAM by using the show logging nvram command.


Troubleshooting with Logs 8

switch# show logging nvram

2008 Sep 10 15:51:58 switch %$ VDC-1 %$ %SYSMGR-2-NON_VOLATILE_DB_FULL: System non-volatile storage usage is unexpectedly high at 99%.2008 Sep 10 15:52:13 switch %$ VDC-1 %$ %PLATFORM-2-PFM_SYSTEM_RESET: Manual system restart from Command Line Interface2008 Sep 10 15:57:49 switch %$ VDC-1 %$ %KERN-2-SYSTEM_MSG: Starting kernel... - kernel2008 Sep 10 15:58:00 switch %$ VDC-1 %$ %CARDCLIENT-2-REG: Sent2008 Sep 10 15:58:01 switch %$ VDC-1 %$ %USER-1-SYSTEM_MSG: R2D2: P1 SUP NO GMTL FOR P1 SUP - r2d22008 Sep 10 15:58:01 switch %$ VDC-1 %$ %USER-1-SYSTEM_MSG: R2D2: P1 SUP NO GMTL FOR P1 SUP - r2d22008 Sep 10 15:58:05 switch %$ VDC-1 %$ %USER-1-SYSTEM_MSG: R2D2: P1 SUP: ResetTx/Rx during QOS INIT - r2d22008 Sep 10 15:58:16 switch %$ VDC-1 %$ %USER-2-SYSTEM_MSG: can't dlsym ssnmgr_is_session_command: please link this binary with ssnmgr.so! - svi2008 Sep 10 15:58:16 switch %$ VDC-1 %$ %CARDCLIENT-2-SSE: LC_READY sent2008 Sep 10 15:58:17 switch %$ VDC-1 %$ snmpd: load_mib_module :Error, while loading the mib module /isan/lib/libpmsnmp_common.so (/isan/lib/libpmsnmp_common.so: undefined symbol: sme_mib_get_if_info)2008 Sep 10 15:58:17 switch %$ VDC-1 %$ %CARDCLIENT-2-SSE: MOD:6 SUP ONLINE2008 Sep 10 15:58:17 switch %$ VDC-1 %$ %VDC_MGR-2-VDC_LIC_WARN: Service using grace period will be shutdown in 9 day(s)

Contacting Customer SupportIf you are unable to solve a problem after using the troubleshooting suggestions in these articles, contact a customer servicerepresentative for assistance and further instructions. Before you call, have the following information ready to help your serviceprovider assist you as quickly as possible:

Date that you received the switch Chassis serial number (located on a label on the right side of the rear panel of the chassis) Type of software and release number Maintenance agreement or warranty information Brief description of the problem Brief explanation of the steps that you have already taken to isolate and resolve the problem

For more information on steps to take before calling Technical Support, see Before Contacting Technical Support.

See AlsoBefore Contacting Technical Support

Further ReadingThe following links contain further information on this topic from Cisco.com:

Cisco NX-OS System Messages Reference


Viewing NVRAM logs 9

External LinksExternal links contain content developed by external authors. Cisco does not review this content for accuracy.

Nexus: Hands on with NX-OS, Part#1

This article describes how to identify and resolve problems that might occur when upgrading or restarting.

Guide ContentsTroubleshooting OverviewTroubleshooting Installs, Upgrades, and Reboots (this section)Troubleshooting LicensingTroubleshooting VDCsTroubleshooting CFSTroubleshooting PortsTroubleshooting vPCsTroubleshooting VLANsTroubleshooting STPTroubleshooting RoutingTroubleshooting Unicast TrafficTroubleshooting WCCPTroubleshooting MemoryTroubleshooting Packet Flow IssuesTroubleshooting FCoEBefore Contacting Technical SupportTroubleshooting Tools and Methodology

Contents1 Information About Upgrades and Reboots 2 Upgrades and Reboot Checklist 3 Verifying Software Upgrades 4 Verifying a Nondisruptive Upgrade

4.1 Using ROM Monitor Mode

5 Troubleshooting Software Upgrades and Downgrades5.1 Software Upgrade Ends with Error 5.2 Upgrading Cisco NX-OS Software

6 Troubleshooting Software System Reboots6.1 Power-On or Switch Reboot Hangs 6.2 Corrupted Bootflash Recovery 6.3 Recovery from the loader> Prompt on SupervisorModules

6.4 Recovery from the loader> Prompt 6.5 Recovery from the switch(boot)# Prompt 6.6 Recovery for Systems with Dual SupervisorModules

6.6.1 Recovering One Supervisor ModuleWith Corrupted Bootflash

6.6.2 Recovering Both Supervisor Moduleswith Corrupted Bootflash

6.7 System or Process Resets


External Links 10

6.8 Recoverable System Restarts 6.9 Unrecoverable System Restarts 6.10 Standby Supervisor Fails to Boot 6.11 Recovering the Administrator Password

7 See Also 8 Further Reading 9 External Links

Information About Upgrades and RebootsCisco NX-OS consists of two images--the kickstart image and the system image. In order to bring up the system, both imagesshould have the same image version.

Upgrades and reboots are ongoing network maintenance activities. You should try to minimize the risk of disrupting the networkwhen performing these operations in production environments and to know how to recover quickly when something does gowrong.

Note: This publication used the term upgrade to refer to both Cisco NX-OS upgrades and downgrades.

Upgrades and Reboot ChecklistUse the following checklist to prepare for an upgrade:

Checklist Checkoff

Read the Release Notes for the release that you are upgrading or downgrading to.Ensure that an FTP or TFTP server is available to download the software images.Copy the new image onto your supervisor modules in bootflash: or slot0:.Use the show install all impact command to verify that the new image is healthy and the impact that the new load willhave on any hardware with regards to compatibility. Check for compatibility.Copy the startup-config file to a snapshot configuration in NVRAM. This step creates a backup copy of thestartup-config file (see the Rollback chapter in the Cisco NX-OS System Management Configuration Guide).Save your running configuration to the startup configuration.Back up a copy of your configuration to a remote TFTP server.Schedule your upgrade during an appropriate maintenance window for your network.

After you have completed the checklist, you are ready to upgrade the systems in your network.

Note: It is normal for the active supervisor to become the standby supervisor during an upgrade.Note: Log messages are not saved across system reboots. However, a maximum of 100 log messages with a severity level of

critical and below (levels 0, 1, and 2) are saved in NVRAM. You can view this log at any time by entering the showlogging nvram command.

Verifying Software UpgradesYou can use the show install all status command to watch the progress of your software upgrade or to view the ongoing installall command or the log of the last installed install all command from a console, SSH, or Telnet session. This command shows theinstall all output on both the active and standby supervisor module even if you are not connected to the console terminal.


Contents 11

switch# show install all statusThere is an on-going installation...

If you need further assistance to determine why an upgrade is unsuccessful, you should collect the details from the showtech-support command output and the console output from the installation, if available, before you contact your technical supportrepresentative.

Using ROM Monitor Mode

If your device does not find a valid system image to load, the system will start in ROM monitor mode. ROM monitor mode canalso be accessed by interrupting the boot sequence during startup. From ROM monitor mode, you can boot the device or performdiagnostic tests.

On most systems, you can enter ROM monitor mode by entering the reload EXEC command and then pressing the Break key onyour keyboard or by using the Break key-combination (the default Break key combination is Ctrl-C) during the first 60 seconds ofstartup.

Troubleshooting Software Upgrades and DowngradesThis section describes how to troubleshoot a software installation upgrade or downgrade failure.

Software Upgrade Ends with Error

Problem Possible Cause Solution

The upgradeends with anerror.

The standby supervisor modulebootflash: file system does not havesufficient space to accept the updatedimage.

Use the delete command to remove unnecessary files from the filesystem.

The specified system and kickstartimages are not compatible.

Check the output of the installation process for details on theincompatibility. Possibly update the kickstart image before updating thesystem image.

The install all command is entered onthe standby supervisor module. Enter the command on the active supervisor module only.

A module was inserted while theupgrade was in progress. Restart the installation.

The system experienced a powerdisruption while the upgrade was inprogress.

Restart the installation.

An incorrect software image path wasspecified. Specify the entire path for the remote location accurately.

Another upgrade is already inprogress.

Verify the state of the system at every stage and restart the upgrade after10 seconds. If you restart the upgrade within 10 seconds, the command isrejected. An error message displays, indicating that an upgrade iscurrently in progress.

Module failed to upgrade. Restart the upgrade or use the install module command to upgrade thefailed module.


Verifying a Nondisruptive Upgrade 13

Upgrading Cisco NX-OS Software

To perform an automated software upgrade on any system from the CLI, follow these steps:

Log into the system through the console, Telnet, or SSH port of the active supervisor.1. Create a backup of your existing configuration file, if required.2. Perform the upgrade by entering the install all command.3. Exit the system console and open a new terminal session to view the upgraded supervisor module by using the showmodule command.

4.

Tip: Always carefully read the output of the install all compatibility check command. This compatibility check tells you exactlywhat needs to be upgraded (such as the BIOS, loader, or firmware) and what modules will experience a disruptive upgrade. Ifthere are any questions or concerns about the results of the output, type n to stop the installation and contact the next level ofsupport.

The following example shows an upgrade using the install all command with the source images located on an SCP server.

switch# install all system scp://testuser@tftp-server1/tftpboot/rel/qa/4.0/final/m95 00-sf1ek9-mz.4.0.bin kickstartscp://testuser@tftp-server1/tftpboot/rel/qa/4.0/fin al/n7000-s1-kickstart-mz.4.0.bin

For scp://testuser@tftp-server1, please enter password: For scp://testuser@tftp-server1, please enter password:

Copying image from scp://testuser@pal/tftpboot/rel/qa/4.0/final/n7000-s1 -kickstart-mz.4.0.bin to bootflash:///n7000-s1-kickstart-mz.4.0.bin. [####################] 100% -- SUCCESS

Copying image from scp://testuser@pal/tftpboot/rel/qa/4.0/final/n7000-s1 -mz.4.0.bin to bootflash:///n7000-s1-mz.4.0.bin. [####################] 100% -- SUCCESS

Verifying image bootflash:///n7000-s1-kickstart-mz.4.0.bin [####################] 100% -- SUCCESS

Verifying image bootflash:///n7000-s1-mz.4.0.bin [####################] 100% -- SUCCESS

Extracting "slc" version from image bootflash:///n7000-s1-mz.4.0.bin. [####################] 100% -- SUCCESS

Extracting "ips" version from image bootflash:///n7000-s1-mz.4.0.bin. [####################] 100% -- SUCCESS

Extracting "svclc" version from image bootflash:///n7000-s1-mz.4.0.bin. [####################] 100% -- SUCCESS

Extracting "system" version from image bootflash:///n7000-s1-mz.4.0.bin. [####################] 100% -- SUCCESS

Extracting "kickstart" version from image bootflash:///n7000-s1-kickstart-mz .4.0.bin. [####################] 100% -- SUCCESS

Extracting "loader" version from image bootflash:///n7000-s1-kickstart-mz.2. 1.1a.bin. [####################] 100% -- SUCCESS

Compatibility check is done: Module bootable Impact Install-type Reason ------ -------- -------------- ------------ ------


Upgrading Cisco NX-OS Software 14

1 yes non-disruptive rolling 2 yes non-disruptive rolling 3 yes disruptive rolling Hitless upgrade is not supported 4 yes disruptive rolling Hitless upgrade is not supported 5 yes non-disruptive reset 6 yes non-disruptive reset

Images will be upgraded according to following table: Module Image Running-Version New-Version Upg-Required ------ ---------- -------------------- -------------------- ------------

1 slc 2.0(2b) 2.1(1a) yes 1 bios v1.1.0(10/24/03) v1.1.0(10/24/03) no 2 slc 2.0(2b) 2.1(1a) yes 2 bios v1.1.0(10/24/03) v1.1.0(10/24/03) no 3 ips 2.0(2b) 2.1(1a) yes 3 bios v1.1.0(10/24/03) v1.1.0(10/24/03) no 4 svclc 2.0(2b) 2.1(1a) yes 4 svcsb 1.3(5m) 1.3(5m) no 4 svcsb 1.3(5m) 1.3(5m) no 4 bios v1.1.0(10/24/03) v1.1.0(10/24/03) no 5 system 2.0(2b) 2.1(1a) yes 5 kickstart 2.0(2b) 2.1(1a) yes 5 bios v1.1.0(10/24/03) v1.1.0(10/24/03) no 5 loader 1.2(2) 1.2(2) no 6 system 2.0(2b) 2.1(1a) yes 6 kickstart 2.0(2b) 2.1(1a) yes 6 bios v1.1.0(10/24/03) v1.1.0(10/24/03) no 6 loader 1.2(2) 1.2(2) no

Do you want to continue with the installation (y/n)? [n] '''y''' Install is in progress, please wait.

Syncing image bootflash:///n7000-s1-kickstart-mz.4.0.bin to standby. [####################] 100% -- SUCCESS

Syncing image bootflash:///n7000-s1-mz.4.0.bin to standby. [####################] 100% -- SUCCESS

Setting boot variables. [####################] 100% -- SUCCESS

Performing configuration copy. [####################] 100% -- SUCCESS

Module 5: Waiting for module online. 2005 May 20 15:46:03 ca-9506 %KERN-2-SYSTEM_MSG: mts: HA communication with standby terminated. Please check the standby supervisor. -- SUCCESS

"Switching over onto standby".

If the configuration meets all guidelines when the install all command is used, all modules (supervisor and switching) areupgraded.

Troubleshooting Software System RebootsThis section describes how to troubleshoot software reboots.


Troubleshooting Software System Reboots 15

Power-On or Switch Reboot Hangs


A power-on or switch reboothangs for a dual supervisorconfiguration.

The bootflash iscorrupted. Use the Recovery for Systems with Dual Supervisor Modules procedure.

The BIOS iscorrupted.

Replace this module. Contact your customer support representative to return thefailed module.

The kickstartimage iscorrupted.

Power cycle the switch if required and enter CTRL-C when the switch says"Loading Boot Loader" to interrupt the boot process at the >loader prompt. Usethe Recovery from the loader> Prompt on Supervisor Modules procedure toupdate the kickstart image.

Boot parametersare incorrect. Verify and correct the boot parameters and reboot.

The systemimage iscorrupted.

Power cycle the switch if required and enter CTRL-] when the switch says"Checking all filesystems....r. done." to interrupt the boot process at theswitch#boot prompt. Use the Recovery from the switch(boot)# Promptprocedure to update the system image..

Corrupted Bootflash Recovery

All device configurations reside in the internal bootflash. If you have a corrupted internal bootflash, you could potentially loseyour configuration. Be sure to save and back up your configuration files periodically. The regular system boot goes through thefollowing sequence (see Figure 1):

The basic input/output system (BIOS) loads the loader.1. The loader loads the kickstart image into RAM and starts the kickstart image.2. The kickstart image loads and starts the system image.3. The system image reads the startup-configuration file.4.

Figure 1 Regular Boot Sequence


Power-On or Switch Reboot Hangs 16

If the images on your system are corrupted and you cannot proceed (error state), you can interrupt the system boot sequence andrecover the image by entering the BIOS configuration utility described in the following section. Access this utility only whenneeded to recover a corrupted internal disk.

Caution: The BIOS changes explained in this section are required only to recover a corrupted bootflash.Recovery procedures require the regular sequence to be interrupted. The internal sequence goes through four phases between thetime that you turn on the system and the time that the system prompt appears on your terminal--BIOS, boot loader, kickstart, andsystem.

Recovery Interruption

Phase

NormalPrompt--appears atthe end of eachphase.

RecoveryPrompt--appears whenthe system cannotprogress to the nextphase.

Description

BIOS loader> No bootable device

The BIOS begins the power-on self test, memory test, and otheroperating system applications. While the test is in progress, pressCtrl-C to enter the BIOS configuration utility and use the netbootoption.

Bootloader Starting kickstart loader>

The boot loader uncompresses the loaded software to boot an imageusing its filename as a reference. These images are made availablethrough bootflash. When the memory test is over, press Esc to enterthe boot loader prompt.

Kickstart Uncompressingsystem switch(boot)#

When the boot loader phase is over, press Ctrl-] (Control key plusright bracket key) to enter the switch(boot)# prompt. Depending onyour Telnet client, these keys may be reserved, and you may need toremap the keystroke. See the documentation provided by your Telnetclient. If the corruption causes the console to stop at this prompt, copythe system image and reboot the system.

System Login: -- The system image loads the configuration file of the last savedrunning configuration and returns a switch login prompt.

Figure 2 Regular and Recovery Sequence


Corrupted Bootflash Recovery 17

Recovery from the loader> Prompt on Supervisor Modules

Caution: This procedure uses the init system command, which reformats the file system of the device. Be sure that you havemade a backup of the configuration files before you begin this procedure.

The loader> prompt is different from the regular switch# prompt. The CLI command completion feature does not work at theloader> prompt and may result in undesired errors. You must type the command exactly as you want the command to appear.

Note: If you boot over TFTP from the loader> prompt, you must supply the full path to the image on the remote server.Note: The TFTP boot method is available only as a backup for diagnostics and for repairing bootflash corruption. The TFTP

boot method is not intended to bring up the system to a fully operational state. Reloading the system is mandatory afterall diagnostics and repairs have been completed.

Use the help command at the loader> prompt to display a list of commands available at this prompt or to obtain more informationabout a specific command in that list.

To recover a corrupted kickstart image (system error state) for a system with a single supervisor module, follow these steps:

1. Enter the local IP address and subnet mask for the system at the loader> prompt, and press Enter.

loader> set ip 172.16.1.2 255.255.255.0

2. Specify the IP address of the default gateway.

loader> set gw 172.16.1.1

3. Boot the kickstart image file from the required server.

loader> boot tftp://172.16.10.100/tftpboot/n7000-s1-kickstart-4.0.bin


Recovery from the loader> Prompt on Supervisor Modules 18

In this example, 172.16.10.100 is the IP address of the TFTP server, and n7000-s1-kickstart-4.0.bin is the name of the kickstartimage file that exists on that server.

The switch(boot)# prompt indicates that you have a usable Kickstart image.

4. Enter the init system command at theswitch(boot)# prompt.

switch(boot)# init system

Caution: Be sure that you have made a backup of the configuration files before you enter this command.5. Follow the procedure specified in the Recovery from the switch(boot)# Prompt procedure.

Recovery from the loader> Prompt

Caution: This procedure uses the init system command, which reformats the file system of the device. Be sure that you havemade a backup of the configuration files before you begin this procedure.

Note: The loader>prompt is different from the regular switch# or switch(boot)# prompt. The CLI command completionfeature does not work at the loader> prompt and may result in undesired errors. You must type the command exactly asyou want the command to appear.

Note: If you boot over TFTP from the loader> prompt, you must supply the full path to the image on the remote server.Tip: Use the help command at the loader> prompt to display a list of commands available at this prompt or to obtain moreinformation about a specific command in that list.

To recover a corrupted kickstart image (system error state) for a system with a single supervisor module, follow these steps:

1. Specify the local IP address and the subnet mask for the system.

loader> set ip 172.21.55.213 255.255.255.224set ip 172.21.55.213 255.255.255.224 Correct - ip addr is 172.21.55.213, mask is 255.255.255.224Found Intel 82546GB [2:9.0] at 0xe040, ROM address 0xf980Probing...[Intel 82546GB]Management interfaceLink UP in 1000/full modeEthernet addr: 00:1B:54:C1:28:60Address: 172.21.55.213Netmask: 255.255.255.224Server: 0.0.0.0Gateway: 172.21.55.193


loader> set gw 172.21.55.193 Correct gateway addr 172.21.55.193Address: 172.21.55.213Netmask: 255.255.255.224Server: 0.0.0.0Gateway: 172.21.55.193


loader> loader> '''boot tftp://172.28.255.18/tftpboot/n7000-s1-kickstart.4.0.3.gbin ''' Address: 172.21.55.213 Netmask: 255.255.255.224


Recovery from the loader> Prompt 19

Server: 172.28.255.18 Gateway: 172.21.55.193 Filesystem type is tftp, using whole disk Booting: /tftpboot/n7000-s1-kickstart.4.0.3.gbin console=ttyS0,9600n8nn quiet loader _ver="3.17.0".... .............................................................................Im age verification OK

Starting kernel... INIT: version 2.85 booting Checking all filesystems..r.r.r.. done. Setting kernel variables: sysctlnet.ipv4.ip_forward = 0 net.ipv4.ip_default_ttl = 64 net.ipv4.ip_no_pmtu_disc = 1 .

Setting the System Clock using the Hardware Clock as reference...System Clock set. Local time: Wed Oct 1 11:20:11 PST 2008 WARNING: image sync is going to be disabled after a loader netboot Loading system software No system image Unexporting directories for NFS kernel daemon...done. INIT: Sending processes the KILL signal Cisco Nexus Operating System (NX-OS) Software TAC support: http://www.cisco.com/tac Copyright (c) 2002-2008, Cisco Systems, Inc. All rights reserved. The copyrights to certain works contained in this software are owned by other third parties and used and distributed under license. Certain components of this software are licensed under the GNU General Public License (GPL) version 2.0 or the GNU Lesser General Public License (LGPL) Version 2.1. A copy of each such license is available at http://www.opensource.org/licenses/gpl-2.0.php and http://www.opensource.org/licenses/lgpl-2.1.php switch(boot)#

The switch(boot)# prompt indicates that you have a usable kickstart image.

4. Enter the init system command at the switch(boot)# prompt.

switch(boot)# init system

Caution: Be sure that you have made a backup of the configuration files before you enter this command.5. Follow the procedure specified in the Recovery from the switch(boot)# Prompt.

Recovery from the switch(boot)# PromptTo recover a system image using the kickstart image for a system with a single supervisor module, follow these steps:

1. Change to configuration mode and configure the IP address of the mgmt0 interface.

switch(boot)# config tswitch(boot)(config)# interface mgmt0

2. Follow this step if you entered an init system command. Otherwise, skip to Step 3.

a. Enter the ip address command to configure the local IP address and the subnet mask for the system.

switch(boot)(config-mgmt0)# ip address 172.16.1.2 255.255.255.0


Recovery from the switch(boot)# Prompt 20

b. Enter the ip default-gateway command to configure the IP address of the default gateway.

switch(boot)(config-mgmt0)# ip default-gateway 172.16.1.1

3. Enter the no shutdown command to enable the mgmt0 interface on the system.

switch(boot)(config-mgmt0)# no shutdown

4. Enter end to exit to EXEC mode.

switch(boot)(config-mgmt0)# end

5. If you believe there are file system problems, enter the init system check-filesystem command. This command checks allinternal file systems and fixes any errors that are encountered. This command takes a few minutes to complete.

switch(boot)# init system check-filesytem

6. Copy the system image from the required TFTP server.

switch(boot)# copy tftp://172.16.10.100/system-image1 bootflash:system-image1

7. Copy the kickstart image from the required TFTP server.

switch(boot)# copy tftp://172.16.10.100/kickstart-image1 bootflash:kickstart-image1

8. Verify that the system and kickstart image files are copied to your bootflash: file system.

switch(boot)#''' dir bootflash: ''' 12456448 Jul 30 23:05:28 1980 kickstart-image1 12288 Jun 23 14:58:44 1980 lost+found/ 27602159 Jul 30 23:05:16 1980 system-image1

Usage for bootflash://sup-local 135404544 bytes used 49155072 bytes free 184559616 bytes total

9. Load the system image from the bootflash: files system.

switch(boot)# '''load bootflash:system-image1''' Uncompressing system image: bootflash:/system-image1 CCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCC

Would you like to enter the initial configuration mode? (yes/no): yes

Note: If you enter no, you will return to the switch# login prompt, and you must manually configure the system.

Recovery for Systems with Dual Supervisor Modules

This section describes how to recover when one or both supervisor modules in a dual supervisor system have corrupted bootflash.

Recovering One Supervisor Module With Corrupted Bootflash

If one supervisor module has a functioning bootflash and the other has a corrupted bootflash, follow these steps:


Recovery for Systems with Dual Supervisor Modules 21

Boot the functioning supervisor module and log on to the system.1. At the switch# prompt on the booted supervisor module, enter the reload module slot force-dnld command, where slot isthe slot number of the supervisor module with the corrupted bootflash.

2.

The supervisor module with the corrupted bootflash performs a netboot and checks the bootflash for corruption. When the bootupscripts discover that the bootflash is corrupted, it generates an init system command, which fixes the corrupt bootflash. Thesupervisor boots as the HA Standby.

Caution: If your system has an active supervisor module currently running, you must enter the system standby manual-bootcommand in EXEC mode on the active supervisor module before entering the init system command on the standbysupervisor module to avoid corrupting the internal bootflash:. After the init system command completes on thestandby supervisor module, enter the system no standby manual-boot command in EXEC mode on the activesupervisor module.

Recovering Both Supervisor Modules with Corrupted Bootflash

If both supervisor modules have corrupted bootflash, follow these steps:

1. Boot the system and press the Esc key after the BIOS memory test to interrupt the boot loader.

Note: Press Esc immediately after you see the following message: 00000589K Low Memory Passed00000000K ExtMemory PassedHit ^C if you want to run SETUP....Wait.....If you wait too long, you will skipthe boot loader phase and enter the kickstart phase.

You see the loader> prompt.

Caution: The loader> prompt is different from the regular switch# or switch(boot)# prompt. The CLI command completionfeature does not work at the loader> prompt and may result in undesired errors. You must type the command exactlyas you want the command to appear.

Tip: Use the help command at the loader> prompt to display a list of commands available at this prompt or to obtain moreinformation about a specific command in that list.

2. Specify the local IP address and the subnet mask for the system.

loader> set ip 172.21.55.213 255.255.255.224set ip 172.21.55.213 255.255.255.224 Correct - ip addr is 172.21.55.213, mask is 255.255.255.224Found Intel 82546GB [2:9.0] at 0xe040, ROM address 0xf980Probing...[Intel 82546GB]Management interfaceLink UP in 1000/full modeEthernet addr: 00:1B:54:C1:28:60Address: 172.21.55.213Netmask: 255.255.255.224Server: 0.0.0.0Gateway: 172.21.55.193


loader> set gw 172.21.55.193 Correct gateway addr 172.21.55.193Address: 172.21.55.213Netmask: 255.255.255.224Server: 0.0.0.0Gateway: 172.21.55.193



Recovering One Supervisor Module With Corrupted Bootflash 22

loader> loader> '''boot tftp://172.28.255.18/tftpboot/n7000-s1-kickstart.4.0.3.gbin ''' Address: 172.21.55.213 Netmask: 255.255.255.224 Server: 172.28.255.18 Gateway: 172.21.55.193 Filesystem type is tftp, using whole disk Booting: /tftpboot/n7000-s1-kickstart.4.0.3.gbin console=ttyS0,9600n8nn quiet loader _ver="3.17.0".... .............................................................................Im age verification OK

Starting kernel... INIT: version 2.85 booting Checking all filesystems..r.r.r.. done. Setting kernel variables: sysctlnet.ipv4.ip_forward = 0 net.ipv4.ip_default_ttl = 64 net.ipv4.ip_no_pmtu_disc = 1 .

Setting the System Clock using the Hardware Clock as reference...System Clock set. Local time: Wed Oct 111:20:11 PST 2008 WARNING: image sync is going to be disabled after a loader netboot Loading system software No system image Unexporting directories for NFS kernel daemon...done. INIT: Sending processes the KILL signal Cisco Nexus Operating System (NX-OS) Software TAC support: http://www.cisco.com/tac Copyright (c) 2002-2008, Cisco Systems, Inc. All rights reserved. The copyrights to certain works contained in this software are owned by other third parties and used and distributed under license. Certain components of this software are licensed under the GNU General Public License (GPL) version 2.0 or the GNU Lesser General Public License (LGPL) Version 2.1. A copy of each such license is available at http://www.opensource.org/licenses/gpl-2.0.php and http://www.opensource.org/licenses/lgpl-2.1.php switch(boot)#

The switch(boot)# prompt indicates that you have a usable kickstart image.

Note: If you boot over TFTP from the loader> prompt, you must supply the full path to the image on the remote server.5. Enter the init-system command to repartition and format the bootflash.

6. Perform the steps in the Recovery from the switch(boot)# Prompt procedure.

7. Perform the steps in the Recovering One Supervisor Module With Corrupted Bootflash procedure to recover the othersupervisor module.

Note: If you do not enter the reload module command when a boot failure has occurred, the active supervisor moduleautomatically reloads the standby supervisor module within 3 to 6 minutes after the failure.

System or Process Resets

When a recoverable or nonrecoverable error occurs, the system or a process on the system may reset. See Table 2-4 for possiblecauses and solutions.


The system or aprocess on the systemresets.

A recoverable error occurred on thesystem or on a process in the system.

The system has automatically recovered from the problem. Usethe Recoverable System Restarts procedure and the System orProcess Resets procedure.


Recovering Both Supervisor Modules with Corrupted Bootflash 23

A nonrecoverable error occurred onthe system.

The system cannot recover automatically from the problem. Usethe Recoverable System Restarts procedure to determine thecause.

A clock module failed. Verify that a clock module failed. Replace the failed clock moduleduring the next maintenance window.Recoverable System Restarts

Every process restart generates a syslog message and a Call Home event. Even if the event does not affect service, you shouldidentify and resolve the condition immediately because future occurrences could cause a service interruption.

To respond to a recoverable system restart, follow these steps:

1. Check the syslog file to see which process restarted and why it restarted.

switch# show log logfile | include error

For information about the meaning of each message, see the Cisco NX-OS System Messages Reference. The system output lookslike the following example:

Sep 10 23:31:31 dot-6 % LOG_SYSMGR-3-SERVICE_TERMINATED: Service "sensor" (PID 704) has finished with error code SYSMGR_EXITCODE_SY.switch# show logging logfile | include failJan 27 04:08:42 88 %LOG_DAEMON-3-SYSTEM_MSG: bind() fd 4, family 2, port 123, addr 0.0.0.0, in_classd=0 flags=1 fails: Address already in useJan 27 04:08:42 88 %LOG_DAEMON-3-SYSTEM_MSG: bind() fd 4, family 2, port 123, addr 127.0.0.1, in_classd=0 flags=0 fails: Address already in useJan 27 04:08:42 88 %LOG_DAEMON-3-SYSTEM_MSG: bind() fd 4, family 2, port 123, addr 127.1.1.1, in_classd=0 flags=1 fails: Address already in useJan 27 04:08:42 88 %LOG_DAEMON-3-SYSTEM_MSG: bind() fd 4, family 2, port 123, addr 172.22.93.88, in_classd=0 flags=1 fails: Address already in useJan 27 23:18:59 88 % LOG_PORT-5-IF_DOWN: Interface fc1/13 is down (Link failure or not-connected)Jan 27 23:18:59 88 % LOG_PORT-5-IF_DOWN: Interface fc1/14 is down (Link failure or not-connected)Jan 28 00:55:12 88 % LOG_PORT-5-IF_DOWN: Interface fc1/1 is down (Link failure or not-connected)Jan 28 00:58:06 88 % LOG_ZONE-2-ZS_MERGE_FAILED: Zone merge failure, Isolating port fc1/1 (VSAN 100)Jan 28 00:58:44 88 % LOG_ZONE-2-ZS_MERGE_FAILED: Zone merge failure, Isolating port fc1/1 (VSAN 100)Jan 28 03:26:38 88 % LOG_ZONE-2-ZS_MERGE_FAILED: Zone merge failure, Isolating port fc1/1 (VSAN 100)Jan 29 19:01:34 88 % LOG_PORT-5-IF_DOWN: Interface fc1/1 is down (Link failure or not-connected)switch#

2. Identify the processes that are running and the status of each process.

switch# show processes

The following codes are used in the system output for the state (process state):

D = uninterruptible sleep (usually I/O) R = runnable (on run queue) S = sleeping


System or Process Resets 24

T = traced or stopped Z = defunct (zombie) process NR = notrunning ER = should be running but currently notrunning

Note: ER usually is the state that a process enters if it has been restarted too many times and has been detected as faulty by thesystem and disabled.

The system output looks like the following example. (This output has been abbreviated to be more concise.)

PID State PC Start_cnt TTY Process----- ----- -------- ----------- ---- -------------

1 S 2ab8e33e 1 - init 2 S 0 1 - keventd 3 S 0 1 - ksoftirqd_CPU0 4 S 0 1 - kswapd 5 S 0 1 - bdflush 6 S 0 1 - kupdated 71 S 0 1 - kjournald 136 S 0 1 - kjournald 140 S 0 1 - kjournald 431 S 2abe333e 1 - httpd 443 S 2abfd33e 1 - xinetd 446 S 2ac1e33e 1 - sysmgr 452 S 2abe91a2 1 - httpd 453 S 2abe91a2 1 - httpd 456 S 2ac73419 1 S0 vsh 469 S 2abe91a2 1 - httpd 470 S 2abe91a2 1 - httpd

3. Show the processes that have had abnormal exits and to if there is a stack-trace or core dump.

switch# show process logProcess PID Normal-exit Stack-trace Core Log-create-time---------------- ------ ----------- ----------- ------- ---------------

ntp 919 N N N Jan 27 04:08snsm 972 N Y N Jan 24 20:50

4. Show detailed information about a specific process that has restarted.

switch# show processes log pid 898Service: idehsdDescription: ide hotswap handler DaemonStarted at Mon Sep 16 14:56:04 2002 (390923 us)Stopped at Thu Sep 19 14:18:42 2002 (639239 us)Uptime: 2 days 23 hours 22 minutes 22 secondsStart type: SRV_OPTION_RESTART_STATELESS (23)Death reason: SYSMGR_DEATH_REASON_FAILURE_SIGTERM (3)Exit code: signal 15 (no core)CWD: /var/sysmgr/workVirtual Memory:CODE 08048000 - 0804D660 DATA 0804E660 - 0804E824 BRK 0804E9A0 - 08050000 STACK 7FFFFD10Register Set:EBX 00000003 ECX 0804E994 EDX 00000008 ESI 00000005 EDI 7FFFFC9C EBP 7FFFFCAC EAX 00000008 XDS 0000002B XES 0000002B EAX 00000003 (orig) EIP 2ABF5EF4 XCS 00000023 EFL 00000246 ESP 7FFFFC5C XSS 0000002B


Recoverable System Restarts 25

Stack: 128 bytes. ESP 7FFFFC5C, TOP 7FFFFD100x7FFFFC5C: 0804F990 0804C416 00000003 0804E994 ................0x7FFFFC6C: 00000008 0804BF95 2AC451E0 2AAC24A4 .........Q.*.$.*0x7FFFFC7C: 7FFFFD14 2AC2C581 0804E6BC 7FFFFCA8 .......*........0x7FFFFC8C: 7FFFFC94 00000003 00000001 00000003 ................0x7FFFFC9C: 00000001 00000000 00000068 00000000 ........h.......0x7FFFFCAC: 7FFFFCE8 2AB4F819 00000001 7FFFFD14 .......*........0x7FFFFCBC: 7FFFFD1C 0804C470 00000000 7FFFFCE8 ....p...........0x7FFFFCCC: 2AB4F7E9 2AAC1F00 00000001 08048A2C ...*...*....,...PID: 898SAP: 0UUID: 0switch#

5. Determine if the restart recently occurred.

switch# show system uptime Start Time: Fri Sep 13 12:38:39 2002Up Time: 0 days, 1 hours, 16 minutes, 22 seconds

To determine if the restart is repetitive or a one-time occurrence, compare the length of time that the system has been up with thetime stamp of each restart.

6. View the core files.

switch# show coresModule-num Process-name PID Core-create-time---------- ------------ --- ----------------

5 fspf 1524 Jan 9 03:116 fcc 919 Jan 9 03:098 acltcam 285 Jan 9 03:098 fib 283 Jan 9 03:08

The output shows all cores that are presently available for upload from the active supervisor. The module-num column shows theslot number on which the core was generated. In the previous example, an FSPF core was generated on the active supervisormodule in slot 5. An FCC core was generated on the standby supervisory module in slot 6. Core dumps generated on the modulein slot 8 include ACLTCAM and FIB.

Copy the FSPF core dump to a TFTP server with the IP address 1.1.1.1, as follows:

switch# copy core://5/1524 tftp::/1.1.1.1/abcd

Display the file named zone_server_log.889 in the log directory as follows:

switch# '''show pro log pid 1473''' ======================================================

Service: ips Description: IPS Manager

Started at Tue Jan 8 17:07:42 1980 (757583 us) Stopped at Thu Jan 10 06:16:45 1980 (83451 us) Uptime: 1 days 13 hours 9 minutes 9 seconds

Start type: SRV_OPTION_RESTART_STATELESS (23) Death reason: SYSMGR_DEATH_REASON_FAILURE_SIGNAL (2) Exit code: signal 6 (core dumped) CWD: /var/sysmgr/work



Virtual Memory:

CODE 08048000 - 080FB060 DATA 080FC060 - 080FCBA8 BRK 081795C0 - 081EC000 STACK 7FFFFCF0 TOTAL 20952 KB

Register Set:

EBX 000005C1 ECX 00000006 EDX 2AD721E0 ESI 2AD701A8 EDI 08109308 EBP 7FFFF2EC EAX 00000000 XDS 0000002B XES 0000002B EAX 00000025 (orig) EIP 2AC8CC71 XCS 00000023 EFL 00000207 ESP 7FFFF2C0 XSS 0000002B

Stack: 2608 bytes. ESP 7FFFF2C0, TOP 7FFFFCF0

0x7FFFF2C0: 2AC8C944 000005C1 00000006 2AC735E2 D..*.........5.* 0x7FFFF2D0: 2AC8C92C 2AD721E0 2AAB76F0 00000000 ,..*.!.*.v.*.... 0x7FFFF2E0: 7FFFF320 2AC8C920 2AC513F8 7FFFF42C ... ..*...*,... 0x7FFFF2F0: 2AC8E0BB 00000006 7FFFF320 00000000 ...*.... ....... 0x7FFFF300: 2AC8DFF8 2AD721E0 08109308 2AC65AFC ...*.!.*.....Z.* 0x7FFFF310: 00000393 2AC6A49C 2AC621CC 2AC513F8 .......*.!.*...* 0x7FFFF320: 00000020 00000000 00000000 00000000 ............... 0x7FFFF330: 00000000 00000000 00000000 00000000 ................ 0x7FFFF340: 00000000 00000000 00000000 00000000 ................ 0x7FFFF350: 00000000 00000000 00000000 00000000 ................ 0x7FFFF360: 00000000 00000000 00000000 00000000 ................ 0x7FFFF370: 00000000 00000000 00000000 00000000 ................ 0x7FFFF380: 00000000 00000000 00000000 00000000 ................ 0x7FFFF390: 00000000 00000000 00000000 00000000 ................ 0x7FFFF3A0: 00000002 7FFFF3F4 2AAB752D 2AC5154C . ... output abbreviated ... Stack: 128 bytes. ESP 7FFFF830, TOP 7FFFFCD0

7. Enter the system cores tftp:[//servername][/path] command to configure the system to use TFTP to send the core dump to aTFTP server.

This command causes the system to enable the automatic copy of core files to a TFTP server. For example, the followingcommand sends the core files to the TFTP server with the IP address 10.1.1.1:

switch(config)# system cores tftp://10.1.1.1/cores

The following conditions apply:

The core files are copied every 4 minutes. This time interval is not configurable.

The copy of a specific core file to a TFTP server can be manually triggered, by using the command copycore://module#/pid# tftp://tftp_ip_address/file_name.



The maximum number of times that a process can be restarted is part of the high-availability (HA) policy for any process.(This parameter is not configurable.) If the process restarts more than the maximum number of times, the older core filesare overwritten.

The maximum number of core files that can be saved for any process is part of the HA policy for any process. (Thisparameter is not configurable, and it is set to three.)

8. Determine the cause and resolution for the restart condition by contacting your technical support representative and asking therepresentative to review your core dump.

See the Cisco NX-OS High Availability and Redundancy Guide for more information on high-availability policies.

Unrecoverable System Restarts

An unrecoverable system restart might occur in the following cases:

A critical process fails and is not restartable. A process restarts more times than is allowed by the system configuration. A process restarts more frequently than is allowed by the system configuration.

The effect of a process reset is determined by the policy configured for each process. An unrecoverable reset may causefunctionality loss, the active supervisor to restart, a supervisor switchover, or the system to restart.

To respond to an unrecoverable reset, see the Troubleshooting Cisco NX-OS Software System Reboots procedure.

The show system reset-reason command displays the following information:

The last four reset-reason codes for the supervisor modules are displayed. If either supervisor module is absent, thereset-reason codes for that supervisor module are not displayed.

The show system reset-reason module number command displays the last four reset-reason codes for a specific modulein a given slot. If a module is absent, then the reset-reason codes for that module are not displayed.

The overall history of when and why expected and unexpected reloads occur The time stamp of when the reset or reload occurred The reason for the reset or reload of a module The service that caused the reset or reload (not always available) The software version that was running at the time of the reset or reload

switch# show system reset-reason module 6----- reset reason for Supervisor-module 6 (from Supervisor in slot 6) ---1) At 281000 usecs after Wed Jun 25 20:16:34 2008 Reason: Reset Requested by CLI command reload Service: Version: 4.0(2.45)2) At 791071 usecs after Wed Jun 25 20:04:50 2008 Reason: Reset Requested by CLI command reload Service: Version: 4.0(2.45)3) At 70980 usecs after Wed Jun 25 19:55:52 2008 Reason: Reset Requested by CLI command reload Service: Version: 4.0(2)4) At 891463 usecs after Wed Jun 18 23:44:48 2008 Reason: Reset Requested by CLI command reload Service: Version: 4.0(2)


Unrecoverable System Restarts 28

Standby Supervisor Fails to Boot

The standby supervisor does not boot after an upgrade. You may see the following system message:

Error Message SYSMGR-2-STANDBY_BOOT_FAILED: Standby supervisor failed to boot up.

Explanation This message is printed if the standby supervisor doesn't complete its boot procedure (i.e. it doesn't reach thelogin prompt on the local console) 3 to 6 minutes after the loader has been loaded by the BIOS. This message is usuallycaused by boot variables not properly set for the standby supervisor. This message can also be caused by a userintentionally interrupting the boot procedure at the loader prompt (by means of pressing ESC).

Recommended Action Connect to the local console of the standby supervisor. If the supervisor is at the loader prompt,try to use the boot command to continue the boot procedure. Otherwise, issue a reload command for the standbysupervisor from a vsh session on the active supervisor, specifying the force-dnld option. Once the standby is online, fixthe problem by setting the boot variables appropriately.

Symptom Possible Cause SolutionStandby supervisor does notboot.

Active supervisor kickstart image booted fromTFTP.

Reload the active supervisor frombootflash:.

Recovering the Administrator Password

You can access the system if you forget the administrator password.

Problem SolutionYou forgot the administrator password foraccessing.

Use the Password Recovery procedure to recover the password using a localconsole connection.

See AlsoCisco NX-OS/IOS Configuration Fundamentals Comparison


Cisco Nexus 7000 Series Upgrade/Downgrade Guides

Cisco Nexus 7000 Series Release Notes


Nexus 7000 NX-OS Upgrade (walkthru example)


Standby Supervisor Fails to Boot 29

NX-OS Intro (part 1) (video)

This article describes how to troubleshoot licensing on a Cisco NX-OS device.

Guide ContentsTroubleshooting OverviewTroubleshooting Installs, Upgrades, and RebootsTroubleshooting Licensing (this section)Troubleshooting VDCsTroubleshooting CFSTroubleshooting PortsTroubleshooting vPCsTroubleshooting VLANsTroubleshooting STPTroubleshooting RoutingTroubleshooting Unicast TrafficTroubleshooting WCCPTroubleshooting MemoryTroubleshooting Packet Flow IssuesTroubleshooting FCoEBefore Contacting Technical SupportTroubleshooting Tools and Methodology

Contents1 Information About Troubleshooting Licensing Issues

1.1 Chassis Serial Numbers 1.2 Swapping out a Chassis 1.3 Grace Period

2 Licensing Guidelines 3 Initial Troubleshooting Checklist 4 Displaying License Information Using the CLI

4.1 Example: Displays Information About CurrentLicense Usage

4.2 Example: Displays the List of Features in aSpecified Package

4.3 Example: Displays the Host ID for the License 4.4 Example: Displays All Installed License KeyFiles and Contents

5 Licensing Installation Issues5.1 Serial Number Issues 5.2 RMA Chassis Errors or License TransfersBetween Systems

5.3 Receiving Grace Period Warnings After LicenseInstallation

5.4 Grace Period Alerts 5.5 License Listed as Missing



External Links 30

Information About Troubleshooting Licensing IssuesCisco NX-OS requires licenses select features. The icenses enable those features on your system. You must purchase a license foreach system that you want to enable the licensed features on.

Note: You can enable a feature without installing the license. Cisco NX-OS provides a grace period that allows you to try outthe feature before purchasing the license.

Chassis Serial Numbers

Licenses are created using the serial number of the chassis where the license file is to be installed. Once you order a license basedon a chassis serial number, you cannot use this license on any other system.

Swapping out a Chassis

If you swap out a chassis which included licenses, you must contact TAC to generate a new license. The old license was based onthe chassis serial number and will not work with the new chassis.

Grace Period

If you use a feature that requires a license but you have not installed a license for that feature, you are given a 120-grace period toevaluate the feature. You must purchase and install the number of licenses required for that feature before the grace period ends orCisco NX-OS will disable the feature at the end of the grace period.

License packages can contain several features. If you disable a feature during the grace period and there are other features in thatlicense package that are still enabled, the clock does not stop for that license package. To suspend the grace period countdown fora licensed feature, you must disable every feature in that license package. Use the show license usage command to determinewhich features are enabled for a license package.

Licensing GuidelinesFollow these guidelines when dealing with licenses for Cisco NX-OS:

Do not ignore the grace period expiration warnings. Allow 60 days before the grace period expires to allow time forordering, shipping, and installation for a new license purchase.

Carefully determine the license(s) that you require based on the features that require a license. Order your license accurately, as follows:

Enter the Product Authorization Key that appears in the Proof of Purchase document that comes with yoursystem.

Enter the correct chassis serial number when ordering the license. The serial number must be for the same chassisthat you plan to install the license on. Use the show license host-id command to obtain your chassis serialnumber.

Enter serial numbers accurately. Do not use the letter "O" instead of a zero in the serial number. Order the license that is specific to your chassis.


Information About Troubleshooting Licensing Issues 31

Back up the license file to a remote, secure place. Archiving your license files ensures that you will not lose the licenses inthe case of a failure on your system.

Install the correct licenses on each system, using the licenses that were ordered using that system's serial number. Licensesare serial-number specific and platform specific.

Use the show license usage command to verify the license installation. Never modify a license file or attempt to use it on a system that it was not ordered for. If you return a chassis, contact yourcustomer support representative to order a replacement license for the new chassis.

Initial Troubleshooting ChecklistBegin troubleshooting license issues by checking the following issues first:

Checklist Checkoff

Verify the chassis serial number for all licenses ordered.Verify the platform or module type for all licenses ordered.Verify that the Product Authorization Key that you used to order the licenses comes from the same chassis that youretrieved the chassis serial number on.Verify that you have installed all licenses on all systems that require the licenses for the features you enable.

Displaying License Information Using the CLIUse the show license commands to display all license information configured on this system.

Example: Displays Information About Current License Usage

switch(config)# show license usageFeature Ins Lic Status Expiry Date Comments Count --------------------------------------------------------------------------------

LAN_ADVANCED_SERVICES_PKG No - In use Grace 102D 0H LAN_ENTERPRISE_SERVICES_PKG No - In use Grace 103D 22H ------------------------------------------------------------------------------------------

Example: Displays the List of Features in a Specified Package

switch(config)# show license usage LAN_ENTERPRISE_SERVICES_PKGApplication -----------

pbr Tunnel -----------


Licensing Guidelines 32

Example: Displays the Host ID for the License

switch# show license host-idLicense hostid: VDH=FOX0646S017

Note: Use the entire ID that appears after the colon (:) . The VHD is the Vendor Host ID.

Example: Displays All Installed License Key Files and Contents

switch# show license entp.lic: SERVER this_host ANY VENDOR cisco INCREMENT LAN_ENTERPRISE_SERVICES_PKG cisco 1.0 permanent uncounted \ VENDOR_STRING=MDS_SWIFTN7K-LAN1K9= \ HOSTID=VDH=TBC10412106 \ > NOTICE="20071025133322456LicLineID>1/LicLineID> \

Licensing Installation IssuesCommon problems with licenses usually occur from incorrectly ordering the license file, installing the license file on an incorrectsystem, or not ordering the correct number of licenses for your fabric.

Serial Number Issues

Make sure that you use the correct chassis serial number when ordering your license. Use the show license host-id command toobtain the correct chassis serial number for your system using the CLI.

If you use a license meant for another chassis, you may see the following system message:

Error Message: LICMGR-3-LOG_LIC_INVALID_HOSTID: Invalid license hostid VDH=[chars] for feature [chars].

Explanation: The feature has a license with an invalid license Host ID. This can happen if a supervisor module with licensedfeatures for one system is installed on another system.

Recommended Action: Reinstall the correct license for the chassis where the supervisor module is installed.

When entering the chassis serial number during the license ordering process, do not use the letter "O" instead of any zeros in theserial number.


Example: Displays the Host ID for the License 33

RMA Chassis Errors or License Transfers Between Systems

A license is specific to the system for which it is issued and is not valid on any other system. If you need to transfer a license fromone system to another, contact your technical support representative.

Receiving Grace Period Warnings After License Installation

If the license installation does not proceed correctly, or if you are using a feature that exists in a license package that you have notinstalled, you will continue to get grace period warnings.

Symptom Possible Cause Solution

You are receiving grace periodwarnings after a licenseinstallation.

The license file is copied tothe system but is notinstalled.

Use the license install command to install the license.

The license installationfailed.

Check your logs for any system messages for a failed licenseinstallation. Use the show license usage command to determinewhich feature is in use without a license.

Grace Period Alerts

Cisco NX-OS gives you a 120-day grace period. This grace period starts or continues when you are evaluating a feature for whichyou have not installed a license.

The grace period stops if you disable a feature that you are evaluating. If you enable that feature again without a valid license, thegrace period countdown continues where it left off.

The grace period operates across all features in a license package. License packages can contain several features. If you disable afeature during the grace period and there are other features in that license package that are still enabled, the countdown does notstop for that license package. To suspend the grace period countdown for a license package, you must disable every feature in thatlicense package.

The Cisco NX-OS license counter keeps track of all licenses on a system. If you are evaluating a feature and the grace period hasstarted, you will receive console messages, SNMP traps, system messages, and daily Call Home messages.

Beyond that, the frequency of these messages become hourly during the last seven days of the grace period. The followingexample uses the VDC feature. On January 30th, you enabled the VDC feature, using the 120-day grace period. You will receivegrace period ending messages as follows:

Daily alerts from January 30th to May 21st Hourly alerts from May 22nd to May 30th

On May 31st, the grace period ends, and the VDC feature is automatically disabled. You will not be allowed to use multiple VDCs


RMA Chassis Errors or License Transfers Between Systems 34

until you purchase a valid license.

Note: You cannot modify the frequency of the grace period messages.

Caution: After the final seven days of the grace period, the feature is turned off and your network traffic may be disrupted.Any future upgrade will enforce license requirements and the 120-day grace period.

If you try to use an unlicensed feature, you may see one of the following system messages:

Error Message: LICMGR-2-LOG_LIC_GRACE_EXPIRED: Grace period expired for feature [chars].

Explanation: The unlicensed feature has exceeded its grace time period. Applications using this license will be shut downimmediately.

Recommended Action: Install the license file to continue using the feature.

Error Message: LICMGR-3-LOG_LICAPP_NO_LIC: Application [chars] running without [chars] license, shutdown in [dec]days.

Explanation: The Application [chars1] has not been licensed. The application will work for a grace period of [dec] days afterwhich it will be shut down unless a license file for the feature is installed.

Recommended Action: Install the license to continue using the feature.

Error Message: LICMGR-3-LOG_LIC_LICENSE_EXPIRED: Evaluation license expired for feature [chars].

Explanation: The feature has exceeded its evaluation time period. The feature will be shut down after a grace period.


Error Message: LICMGR-3-LOG_LIC_NO_LIC: No license(s) present for feature [chars]. Application(s) shutdown in [dec]days.

Explanation: The feature has not been licensed. The feature will work for a grace period, after which the application(s) using thefeature will be shut down.


Error Message: LICMGR-6-LOG_LICAPP_EXPIRY_WARNING: Application [chars] evaluation license [chars] expiry in [dec]days.

Explanation: The application will exceed its evaluation time period in the listed number of days and will be shut down unless apermanent license for the feature is installed.

Recommended Action: Install the license file to continue using the feature.


Grace Period Alerts 35

Use the show license usage command to display grace period information for a system.

switch(config)# show license usage Feature Ins Lic Status Expiry Date Comments Count --------------------------------------------------------------------------------

LAN_ADVANCED_SERVICES_PKG No - In use Grace 102D 0H LAN_ENTERPRISE_SERVICES_PKG No - In use Grace 103D 22H ------------------------------------------------------------------------------------------

License Listed as Missing

After a license is installed and operating properly, it may show up as missing if you modify your system hardware or encounter abootflash: issue.

Symptom Possible Causes Solutions

A license is listed asmissing.

The supervisor module was replacedafter the license was installed. Use the Corrupted Bootflash Recovery procedure to recover from

the corrupted bootflash:. Reinstall the license.The supervisor bootflash: is corrupted

See AlsoBefore Contacting Technical Support


Cisco Nexus 7000 Series Licensing Information


This article describes how to troubleshoot virtual device contexts (VDCs).

Guide ContentsTroubleshooting OverviewTroubleshooting Installs, Upgrades, and RebootsTroubleshooting LicensingTroubleshooting VDCs (this section)Troubleshooting CFS


License Listed as Missing 36

Troubleshooting PortsTroubleshooting vPCsTroubleshooting VLANsTroubleshooting STPTroubleshooting RoutingTroubleshooting Unicast TrafficTroubleshooting WCCPTroubleshooting MemoryTroubleshooting Packet Flow IssuesTroubleshooting FCoEBefore Contacting Technical SupportTroubleshooting Tools and Methodology

Contents1 Information About Troubleshooting VDCs 2 Initial Troubleshooting Checklist 3 VDC Issues

3.1 You Cannot Create a VDC 3.2 You Cannot Log into a Device 3.3 You Cannot Switch to a VDC 3.4 You Cannot Delete a VDC 3.5 You Cannot Allocate an Interface to a VDC

3.5.1 Table: Port Numbers for Cisco Nexus 7000 Series 32-port10-Gbps Ethernet module

3.6 The VDC Does Not Reflect a Resource Template Change 3.7 The VDC Remains in a Failed State 3.8 You Cannot Copy the Running-Config File to the Startup-Config File ina VDC


Information About Troubleshooting VDCsCisco NX-OS supports VDCs, which you can use to divide the physical NX-OS device into separate virtual devices. Each VDCappears as a unique device to the connected users. A VDC runs as a separate logical entity within the physical NX-OS device,maintains its own unique set of running software processes, has its own configuration, and can be managed by a separateadministrator.

VDC issues may not be directly related to VDC management. See the troubleshooting chapter that reflects your symptoms to findother issues related to VDCs. For instance, if you configure a VDC template that limits the number of port channels in that VDC,you may experience problems if you try to create more port channels than the VDC template allows.

VDC templates set limits on the following features:

Port channels SPAN sessions IPv4 route map memory


External Links 37

VLANs Virtual routing and forwarding instances (VRFs)

The minimum resource value configures the guaranteed limit for that feature. The maximum resource value representsoversubscription for the feature and is available on a first-come,first-served basis.

Note: When you allocate an interface to a VDC, Cisco NX-OS removes all configuration for that interface.

See the Cisco NX-OS Virtual Device Context Configuration Guide for more information on VDCs or for details on any VDCconfiguration changes recommended in this article.

Initial Troubleshooting ChecklistBegin troubleshooting VDC issues by checking the following issues first:

Checklist Check offVerify that you are logged into the device as network-admin if you are creating or modifying VDCs.Verify that you are in the correct VDC. You must be in the default VDC to configure VDCs.Verify that you have installed the Advanced Services license to configure VDCs.Verify that you are not attempting to create more than three VDCs.

Use the following commands to display VDC information:

show vdc membership - Displays information about which interfaces are assigned to a VDC. show vdc resource - Displays information about the resources assigned (Command is available only in the default VDC). show vdc current-vdc - Displays the VDC you are currently in.

VDC IssuesProblems with VDCs usually occur from logging into the incorrect VDC or misallocating resources for a VDC.

You Cannot Create a VDC

When you have a problem with creating a VDC, you may see one of the following system messages:

Error Message: VDC_MGR-2-VDC_BAD: vdc_mgr: There has been a failure at res_mgr

Explanation: You cannot create a VDC because not enough resources are available based on the template configuration. If notemplate is used, a default template is applied.


Information About Troubleshooting VDCs 38

Recommended Action: Verify that you have sufficient resources available to create this VDC by using the show vdc resources[detail] or show vdc resource template command. Modify the template that you are using to create the VDC or create a newtemplate with resource limits that are currently available.

Error Message: VDC_MGR-2-VDC_BAD: vdc_mgr: : There has been a failure at sys_mgr

Explanation: Some services crashed or failed to come up because of insufficient system resources other than what can bereserved using the resource templates. These dynamic resources are based on system utilization and may not be available tosupport a new VDC.

Recommended Action: Use the show system internal sysmgr service running command to determine what caused the failure.


You cannotcreate a VDC.

You are not logged in asnetwork-admin. Log into the device with an account that has network-admin privileges.

You are not logged intothe default VDC. Use the switchto command to switch to the default VDC to allocate resources.

There are not enoughresources.

Use the show vdc resources [detail] or show vdc resource template command todetermine your available resources. Modify your template or create a VDC with fewerresources by using the limit-resource command in VDC configuration mode.

You Cannot Log into a Device

You may have a problem when logging into a device.


You cannot loginto a device.

There is no accountinformation for the VDC.

Log into the device as network-admin and use the switchto command to switchto the VDC and configure the password and network connectivity for this VDC.

You are using an incorrectVDC username. Log into the device with the account created for that VDC.

You Cannot Switch to a VDC

You may have a problem when you switch to another VDC.

Symptom Possible Cause SolutionYou cannot switch to aVDC.

You are not logged in as network-admin ornetwork-operator.

Log into the device with an account that has thecorrect privileges.


You Cannot Create a VDC 39

You Cannot Delete a VDC

When you have a problem with deleting a VDC, you may see one of the following system messages:

Error Message: VDC_MGR-2-VDC_UNGRACEFUL: vdc_mgr: Ungraceful cleanup request received for vdc [dec], restartcount for this vdc is [dec]

Explanation: Vdc_mgr has begun an ungraceful cleanup for a VDC.

Recommended Action: No action is required.

Error Message: VDC_MGR-2-VDC_OFFLINE: vdc [dec] is now offline

Explanation: Vdc_mgr has finished deleting a VDC.

Recommended Action: No action is required.


You cannot delete aVDC.

You attempted to delete the default VDC. You cannot delete the default VDC.Unknown errors occurred when deletinga VDC.

Use the show tech-support VDC command to gather moreinformation.

You Cannot Allocate an Interface to a VDC

When you have a problem with creating a VDC, you may see the following system message:

Error Message: VDC_MGR-2-VDC_BAD: vdc_mgr: There has been a failure at gim (port_affected_list).

Explanation: An interface allocation has failed.

Recommended Action: Use the show vdc membership status or show interface brief command to gather more information.


You cannotallocate aninterface to aVDC.

You are not logged in as network-admin. Log into the device with an account that has the correct privileges.

You are not logged into the correct VDC. Use the switchto command to switch to the default VDC to allocateresources.

The interface is part of a dedicated portgroup.

Use the show interface capabilities command to determine if theport is dedicated. All ports in a dedicated port group must be in thesame VDC.

The interface is on the Cisco Nexus 7000Series 32-port 10-Gbps Ethernet module(N7K-M132XP-12).

You must allocate all ports in a port group to the same VDC forthis module. For information about the port number to port groupmapping, see Table: Port Numbers for Cisco Nexus 7000 Series32-port 10-Gbps Ethernet module.


You Canno

Documents

n7kts