Manuals for HP 3PAR Service Processors

7/23/2019 Manuals for HP 3PAR Service Processors

1/17

HP 3PAR Service Processor TroubleshootingGuide

AbstractThis guide provides information for troubleshooting an HP 3PAR Service Processor.

HP Part Number: QR483-96013Published: December 2012


2/17

Copyright 2011, 2012 Hewlett-Packard Development Company, L.P.

Confidential computer software. Valid l icense from HP required for possession, use or copying. Consistent with FAR 12.211 and 12.212, Commercial

Computer Software, Computer Software Documentation, and Technical Data for Commercial Items are licensed to the U.S. Government under

vendor's standard commercial license.

The information contained herein is subject to change without notice. The only warranties for HP products and services are set forth in the express

warranty statements accompanying such products and services. Nothing herein should be construed as constituting an additional warranty. HP shall

not be liable for technical or editorial errors or omissions contained herein.

Acknowledgments

Intel, Itanium, Pentium, Intel Inside, and the Intel Inside logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries

in the United States and other countries.

Microsoft, Windows, Windows XP, and Windows NT are U.S. registered trademarks of Microsoft Corporation.

Adobe and Acrobat are trademarks of Adobe Systems Incorporated.

Java and Oracle are registered trademarks of Oracle and/or its affiliates.

UNIX is a registered trademark of The Open Group.


3/17

Contents

1 Introduction...............................................................................................4Supported Service Processors.....................................................................................................4

2 Guidelines for Troubleshooting the Service Processor.......................................5Troubleshooting Guidelines........................................................................................................5

Troubleshooting Tools................................................................................................................5Audit and Logging Information..............................................................................................5SPLOR................................................................................................................................6

Running a SPLOR from SPOCC.........................................................................................6Using a SPLOR................................................................................................................6

3 Troubleshooting Service Processor Issues........................................................8Troubleshooting IP Address Problems...........................................................................................8

Duplicate IP Address ...........................................................................................................8Page Not Found Message..................................................................................................8

4 Support and Other Resources.......................................................................9Contacting HP..........................................................................................................................9HP 3PAR documentation............................................................................................................9Typographic conventions.........................................................................................................12HP 3PAR branding information.................................................................................................12

5 Documentation feedback...........................................................................13A Service Processor Process Overview............................................................14B Service Processor Messages in SPMAINT.....................................................15C IP Ports Used by HP 3PAR..........................................................................17

Contents 3


4/17

1 IntroductionThis guide provides information for troubleshooting an HP 3PAR Service Processor (SP).

Supported Service ProcessorsThe following table lists supported SPs.

Table 1 Supported Service ProcessorsSP ID RangeDescriptionPart Number

SP00001SP00199Service Processor, 2U (Cal Digital)F970 0017 01

SP00300SP00399Service Processor, 1U (DELL 650)F970 0085

SP00400SP00899Service Processor, 1U (DELL 750)F970 0088

SP00900SP01999Service Processor, 1U (WINTEC)F975 0009 50 R5

SP02000SP02999Service Processor, 1U (SuperMicro)F979 200051

SP03000SPxxxxService Processor, 1U (SuperMicro II)F975-200010

Varies. See note.Service Processor, 1U (ProliantDL360e)

QR516A

NOTE: For the SP 4.1 release, the SP serial number range uses one of the following formats:

7-character SP ID: The literal SP string (2 characters) + 5 digits (for example, SP09997).This ID format is used for legacy SPs.

12-character SP ID: The literal SP string (2 characters) + 10 alphanumeric characterscontaining the HP serial number (for example, SP5463AB1253). This ID format is used forSPs that run on HP Proliant DL360e.

41-character SP ID: The literal SP string (2 characters) + 32 alphanumeric characters + 7

dashes (-). The dashes delimit the 32 character alphanumeric strings into group of 4 characters(for example, SP1111-bbbb-3333-DDDD-5555-6666-7777-8888). This ID format is used forthe virtual SP.

4 Introduction


5/17

2 Guidelines for Troubleshooting the Service ProcessorThis chapter provides troubleshooting guidelines and explains how to use this document fortroubleshooting Service Processor (SP) issues.

Troubleshooting GuidelinesGeneral troubleshooting guidelines are as follows:

Identify the problem or problems.

Assess the situation.

Determine how the criticality of the problem. If more than one problem exists, determine whichproblem is the most significant in terms of impact.

Collect as much information as possible relevant to the problem.

Generate an action plan and then execute it.

If the action does not resolve the problem, escalate.

Troubleshooting ToolsYou can use the following tools to collect data from the SP: Audit and Logging Information - Provides audit information regarding SP usage.

SPLOR - Gathers files to diagnose SP issues.

Audit and Logging InformationHP 3PAR SP Audit Information is contained in the SP_USER_ACTIVITYlog file, which providesthe following audit information:

Users who accessed the SP

Logon and logoff times The functionality used, such as Interactive CLI, Added System, and Started Guided Maintenance

Available through SPOCC-Reports, this file is gathered as part of a SPLOR and can be viewed byHP Support personnel using Service Tools and Technical Support (STaTS).

Output examples:

yy/mm/dd 15:12:48 Remote Operation requested by userID as SP user #yy/mm/dd 15:13:20 #( userID-pts0 )/spmaint -- spmaint starting

yy/mm/dd 15:14:47 #( userID-pts0 )/spmaint -- InFormOS validating update status --InServ: myInServ. RC: 0yy/mm/dd 16:20:56 #( userID-pts0 )/spmaint -- interactive CLI requested for InServ:

myInServyy/mm/dd 16:22:58 #( userID-pts0 )/spmaint -- csstatus launched for myInServyy/mm/dd 16:35:36 Remote Operation terminated for userID as SP user #

The actual commands that were executed can be found in other logs that are available to HP 3PARSupport personnel.

HP 3PAR SP command logging includes the following:

Commands that were executed using SPMAINT are logged in SPMNTLOG, which is availableas a unique file in STaTS.

Commands that were initiated from the SP GUI (SPOCC) are logged in SPOCCMNTLOG.This log is available as part of SPLOR.

Troubleshooting Guidelines 5


6/17

SPLORA SPLOR gathers files to help diagnose SP issues. You run a SPLOR from SPOCC.

Running a SPLOR from SPOCC

1. Log in to SPOCC as user 3parcust. Enter the IP address of the SP in a web browser, andthen pressENTER.

2. ClickSupporton the left-side of the screen.

3. UnderService Processor, clickLaunch SPLOR.4. To view the SPLOR results, clickView SPLOR Contents.

Using a SPLOR

A SPLOR provides the information for troubleshooting an SP. The following table provides a list offile names and descriptions in the SPLOR.

Table 2 SPLOR file names and description

DescriptionFile name

Connection Portal modem numberDial-numbers

Boot messagesdmesg

ST Process errorsdserrors

Sptransfer, SPup2date, stAnalyze, stBackDoor, stGrpNotdumplist

STlogger, stMonitor, stNotifyetc_filelist

SP hosts fileetc_hosts

Apache web server configuration fileetc_httpd_conf_httpd

Network configuration informationetc_network

Eth0 network configuration information (SP Public networkinterface)

etc_network_ifcfg-eth0

Name service switch configuration fileetc_nsswitch.conf

Network Time Protocol (NTP) configuration fileetc_ntp.conf

SP Firewall configuration fileetc_sp_fw

Syslog configuration fileetc_syslog_conf

rsync service configurationetc_xinetd_rsync

tftp service configurationetc_xinetd_tftp

Event filter file used by the spevent process to controlcollection of storage system event files

filter

Global values used for control purposesgdda

SP OS Update log fileinstallWork/pkgLog

SP OS Update log fileinstallWork/SPOSupdlog

Interprocess communication statusipcs

SP Message logMessageLog340.100407.144344.txt

Active SP Message logMessageLog341.txt

Login configuration; describes which logins are allowedmgetty_login_config

6 Guidelines for Troubleshooting the Service Processor


7/17

Table 2 SPLOR file names and description(continued)

DescriptionFile name

Network status information for connections, protocol, ports,and sockets

netstat_ifconfig

Network route informationnetwork_routes

SP hardware configuration informationplatform

Process status informationps

Process message queue file listingsrmsgq

Software package manager listrpms

Internal files used during upgrades and installationssp_tmp_installWork_filelst

SP user activity log filesp_user_activity

Internal files used during upgrades and installationssp_var_install_filelist

SP configuration file (useful if an SP needs to be rebuilt)spconfig.100404.042520

Software installation history log filespinHistory

Report generated from "splor" file executionsplor

SP maintenance logspmntlog

SP maintenance log regarding transfersspmntlog.xfer

Process activity statisticstop

Postgres database file listingvar_lib_pgsql

Postgres database log filevar_lib_pgsql_logfile

Logs from SP booting upvar_log_boot

Cron job logfilevar_log_cron

HTTP access log filevar_log_httpd_access_log

HTTP error log filevar_log_httpd_error_log

Log of mails to customer for eventsvar_log_maillog

Linux message log filevar_log_messages

Authentication log for secure servicevar_log_secure

Printer logvar_log_spooler

Listing of weekly data file collectionweekly

Work queue statisticsWorkqMaint

Troubleshooting Tools 7


8/17

3 Troubleshooting Service Processor IssuesThis chapter provides instructions for troubleshooting general problems that may occur with aService Processor (SP).

Troubleshooting IP Address Problems

Duplicate IP AddressIf the wizard cannot configure the permanent IP address you entered because it is already in use:

1. ClickStopto stop theApply Settingsprocess and return to the Service Processor Setup wizard.2. Click thePrevbutton until you reach Step 2: SP Networking.3. Determine an available IPv4 address to use for the Service Processor, and then enter that IP

address in theIP Addresstext box.4. ClickNextuntil you return to Step 7: Apply Settings. (You do not need to re-enter any other

Service Processor settings.)

The wizard will automatically begin to apply the settings again.

Page Not Found MessageIf the Service Processor displays aPage Not Foundor similar message, the permanent ServiceProcessor IP address you entered is not a valid address. You must set up the Service Processoragain.

1. Restart the SP Setup wizard:

If you are using a physical Service Processor, return to the Set IP Address wizard and setup a new, valid IP address.

After you have set up the new IP address, use that new permanent IP address to restartthe Service Processor Setup wizard (in other words, enter the URL https:///sp/SpSetupWizard.html).

If you are using a virtual Service Processor (VSP), use the temporary SP IP address torestart the Service Processor Setup wizard (in other words, enter the URL https:///sp/SpSetupWizard.html).

2. Proceed through the Service Processor Setup wizard, re-entering your Service Processor settings.(The Service Processor ID is already set; you do not need to reset the ID.)

8 Troubleshooting Service Processor Issues


9/17

4 Support and Other Resources

Contacting HPFor worldwide technical support information, see the HP support website:

http://www.hp.com/support

Before contacting HP, collect the following information: Product model names and numbers

Technical support registration number or Service Agreement ID (if applicable)

Product serial numbers

Error messages

Operating system type and revision level

Detailed questions

Specify the type of support you are requesting:

Support requestHP 3PAR storage system

StoreServ 7000 StorageHP 3PAR StoreServ 7200 and 7400 Storage systems

3PAR or 3PAR StorageHP 3PAR StoreServ10000 Storage systems

HP 3PAR T-Class storage systems

HP 3PAR F-Class storage systems

HP 3PAR documentation

See:For information about:

The Single Point of Connectivity Knowledge for HPStorage Products (SPOCK) website:

Supported hardware and software platforms

http://www.hp.com/storage/spock

The HP 3PAR support website:Locating HP 3PAR documents

http://www.hp.com/3par/support_manuals

HP 3PAR storage system software

HP 3PAR StoreServ Storage Concepts GuideStorage concepts and terminology

HP 3PAR Management Console User's GuideUsing the HP 3PAR Management Console (GUI) to configureand administer HP 3PAR storage systems

HP 3PAR Command Line Interface AdministratorsManual

Using the HP 3PAR CLI to configure and administer storagesystems

HP 3PAR Command Line Interface ReferenceCLI commands

HP 3PAR System Reporter Software User's GuideAnalyzing system performance

HP 3PAR Host Explorer Users GuideInstalling and maintaining the Host Explorer agent in orderto manage host configuration and connectivity information

HP 3PAR CIM API Programming ReferenceCreating applications compliant with the Common InformationModel (CIM) to manage HP 3PAR storage systems

HP 3PAR-to-3PAR Storage Peer Motion GuideMigrating data from one HP 3PAR storage system to another

Contacting HP 9
http://www.hp.com/supporthttp://www.hp.com/storage/spockhttp://www.hp.com/3par/support_manualshttp://www.hp.com/3par/support_manualshttp://www.hp.com/storage/spockhttp://www.hp.com/support


10/17


HP 3PAR Secure Service Custodian Configuration UtilityReference

Configuring the Secure Service Custodian server in order tomonitor and control HP 3PAR storage systems

HP 3PAR Remote Copy Software Users GuideUsing the CLI to configure and manage HP 3PAR RemoteCopy

HP 3PAR Upgrade Pre-Planning GuideUpdating HP 3PAR operating systems

HP 3PAR F-Class, T-Class, and StoreServ 10000 StorageTroubleshooting Guide

Identifying storage system components, troubleshootinginformation, and detailed alert information

HP 3PAR Policy Server Installation and Setup GuideInstalling, configuring, and maintaining the HP 3PAR PolicyServer HP 3PAR Policy Server Administration Guide



11/17


Planning for HP 3PAR storage system setup

Hardware specifications, installation considerations, power requirements, networking options, and cabling informationfor HP 3PAR storage systems

HP 3PAR StoreServ 7000 Storage Site Planning ManualHP 3PAR 7200 and 7400 storage systems

HP 3PAR StoreServ 10000 Storage Physical PlanningManual

HP 3PAR 10000 storage systems

HP 3PAR StoreServ 10000 Storage Third-Party RackPhysical Planning Manual

Installing and maintaining HP 3PAR 7200 and 7400 storage systems

HP 3PAR StoreServ 7000 Storage Installation GuideInstalling 7200 and 7400 storage systems and initializingthe Service Processor HP 3PAR StoreServ 7000 Storage SmartStart Software

Users Guide

HP 3PAR StoreServ 7000 Storage Service GuideMaintaining, servicing, and upgrading 7200 and 7400storage systems

HP 3PAR StoreServ 7000 Storage Troubleshooting GuideTroubleshooting 7200 and 7400 storage systems

HP 3PAR Service Processor Software User GuideMaintaining the Service Processor

HP 3PAR Service Processor Onsite Customer Care(SPOCC) User's Guide

HP 3PAR host application solutions

HP 3PAR Recovery Manager Software for Oracle User'sGuide

Backing up Oracle databases and using backups for disasterrecovery

HP 3PAR Recovery Manager Software for MicrosoftExchange 2007 and 2010 User's Guide

Backing up Exchange databases and using backups fordisaster recovery

HP 3PAR Recovery Manager Software for Microsoft SQLServer Users Guide

Backing up SQL databases and using backups for disasterrecovery

HP 3PAR Management Plug-in and Recovery ManagerSoftware for VMware vSphere User's Guide

Backing up VMware databases and using backups fordisaster recovery

HP 3PAR VSS Provider Software for Microsoft WindowsUser's Guide

Installing and using the HP 3PAR VSS (Volume Shadow CopyService) Provider software for Microsoft Windows

HP 3PAR Storage Replication Adapter for VMwarevCenter Site Recovery Manager Implementation Guide

Best practices for setting up the Storage Replication Adapterfor VMware vCenter

HP 3PAR Storage Replication Adapter for VMwarevCenter Site Recovery Manager Troubleshooting Guide

Troubleshooting the Storage Replication Adapter for VMwarevCenter Site Recovery Manager

HP 3PAR VAAI Plug-in Software for VMware vSphereUser's Guide

Installing and using vSphere Storage APIs for ArrayIntegration (VAAI) plug-in software for VMware vSphere

HP 3PAR documentation 11


12/17

Typographic conventionsTable 3 Document conventions

ElementConvention

Bold text Keys that you press

Text you typed into a GUI element, such as a text box

GUI elements that you click or select, such as menu items, buttons,

and so on

Monospace text File and directory names

System output

Code

Commands, their arguments, and argument values

Code variables

Command variables

Bold monospace text Commands you enter into a command line interface

System output emphasized for scannability

WARNING! Indicates that failure to follow directions could result in bodily harm or death, or inirreversible damage to data or to the operating system.

CAUTION: Indicates that failure to follow directions could result in damage to equipment or data.

NOTE: Provides additional information.

Required

Indicates that a procedure must be followed as directed in order to achieve a functional andsupported implementation based on testing at HP.

HP 3PAR branding information The server previously referred to as the "InServ" is now referred to as the "HP 3PAR StoreServ

Storage system."

The operating system previously referred to as the "InForm OS" is now referred to as the "HP3PAR OS."

The user interface previously referred to as the "InForm Management Console (IMC)" is nowreferred to as the "HP 3PAR Management Console."

All products previously referred to as 3PAR products are now referred to as "HP 3PAR"products.



13/17

5 Documentation feedbackHP is committed to providing documentation that meets your needs. To help us improve thedocumentation, send any errors, suggestions, or comments to Documentation Feedback([email protected]). Include the document title and part number, version number, or the URLwhen submitting your feedback.

13
mailto:[email protected]:[email protected]


14/17

A Service Processor Process OverviewThe following list provides an overview of the Service Processor (SP) processes.

SPCOLLECT - Periodically collects the storage system CLI data. The process also creates filesthat contain the collected data, including the following file types:

Status

Config

Evtlog

Perform

Environ

EEPROM

Biosmsg

The file types correlate to the file extensions found in Service Tools and Technical Support

(STaTS). The period of collection varies with each file type. SPCOMMCTLR - Controls the communication from the SP to the Connection Portal. This process

checks the configuration of Customer Controlled Access and determines the method of transfer:Ethernet or modem.

SPNETMON - Monitors the SP network and gathers active network interface statistics.

SPTRANSFER - Manages the transfer of files from the SP to the Connection Portal. This processinterfaces with modem or an Ethernet network to accomplish transfer.

SPUP2DATE - Monitors Connection Portal for software updates staged to transfer to the SP.

STANALYZE - Analyzes event files created by spevent. Develops the text abstract that describesthe event or alert. This process sends a message to the notification process.

STBACKDOOR - Sends information about the SP process problems. STGRPNOT - Scans notification database and issues any grouped notifications that have met

their timing parameters. Low-urgency notifications are grouped before they are sent.

STLOGGER - A process used by all other processes to record process logging information.This information is stored in the MessageLog.

STMONITOR - Monitors various services and checks for normal operation. This process issuesback-door notifications for abnormal conditions.

STNOTIFY - Sends email notifications to users who have subscribed to the local notificationfeature.

14 Service Processor Process Overview


15/17

B Service Processor Messages in SPMAINTThis appendix describes Service Processor (SP) messages in spmntlogfile.

Acronyms used in this solution:

CCA - Customer Controlled Access (formerly called Big Yellow Switch [BYS])

CP - Connection Portal SP - Service Processor

ST - Service Tools groupWhen an SP transfers (or attempts to transfer) files from the customer site to the Connection Portal,it logs messages to the spmntlogfile.

The SP message table provides the following information:

The message that is logged in the spmntlogfile

The meaning of each message

Remedial action that may be taken

NOTE: The following table is not an exhaustive list of the messages you may see. If you see amessage that is not on the list and that is causing a problem, contact HP Support.

Table 4 SP messages in SPMAINT

Remedial ActionMeaningMessage

If remote access is available, run aSPLOR.

The code is doing some housekeepingthat does not require external

Initializing

If remote access is not available, sendan FE onsite to log on to the SP console

communications. If it persists beyond30 seconds, something is probablyamiss. and capture any errors. If the problem

persists, escalate to your HP 3PARAuthorized Service Provider.

If the problem persists, escalate to your

HP 3PAR Authorized Service Provider.

The text DB on the SP failed to

produce the SP serial number, and we

gddabatch failed on get for

SP_IDcannot continue without a serialnumber. This is a logic failure and afatal error.

It is very unlikely that a user willencounter this message.

If the problem persists, escalate to yourHP 3PAR Authorized Service Provider.

Trying to query the CommunicationController to determine whether

Failed to get state of xfer

media. Reason:...transfers are allowed. This issue resultsfrom a logic failure.

It is unlikely that a user will encounterthis message.


SPcommCtlris not providing themedia (Ethernet or modem) that is

keyChange GetVarvarXFER_MEDIA failed...

needed to do the exchange. This is alogic failure and a fatal error.


Program cannot get the state of theCustomer Controlled Access (CCA),

keyChange GetVar varBYSfailed...

which indicates whether it is legal todo transfers. This is a logic failure.

Change the CCA to allow outbound orbidirectional traffic (depending on yourpreference).

Customer Controlled Access (CCA) isin the OFF state. Communications withthe Connection Portal are not allowed

keyChange BYS state is OFF- sleeping for state change

in either direction, and a key

15


16/17

Table 4 SP messages in SPMAINT (continued)

Remedial ActionMeaningMessage

exchange is not allowed either. TheCCA will remain in this stateindefinitely, retrying every 60 seconds.

Verify network connectivity. If theproblem persists, escalate to your HP3PAR Authorized Service Provider.

CCA is OK, and the SP is attemptingto contact the CP using SSH to performsome trivial operation to make sure

Testing connection...

that a connection is possible. Thisshould be a short lived state.

If this state remains for more than fiveminutes, the issue could be a logicflaw. It could also be a network issue,but generally, if the networkconnection is not available, SSH timesout, in which case the system wouldbe in a different state.

Transfer media is via modem andthere was a problem retrying the

sleeping for modem problems- ...

Reboot the SP.

Check the phone line.operation. This could be due to a

modem issue in some older SPs or afaulty phone line.


The CCA state is OK, but the firewallis still not configured to allow the

sleeping for firewallaccess!

communication to the CP. This is alogic flaw.

Verify the firewall configuration and/orresolve any network routing issues.

CCA is appropriately set forcommunications to occur, but SP

keyChange: state OK, but sshconnection fails - sleeping

cannot reach the Connection Portal.This issue is most likely due tocustomer firewall or routing problems.CCA will stay in this state forever(re-trying every two minutes) until the

SSH connection works.

No action required.Everything is OK for the SP to talk tothe CP. The SP is about to invoke the

Performing exchange

backend process to install the newkey.


A miscommunication between the SPand CP has occurred. This message

spcmd communication problem

- sleepingrepresents a number of possible butobscure issues.

16 Service Processor Messages in SPMAINT


17/17

C IP Ports Used by HP 3PARTable 5 (page 17)describes the TCP/IP port assignments for communication between variouscomponents.

Table 5 TCP/IP port assignments

Flow of TrafficUsagePort

Third-party SSH client HP 3PARService Processor

Used for storage system monitoringand configuration through SSH

22: The Secure Shell (SSH) Protocolin Legacy Mode

Third-party SSH client HP 3PARStorage System CLI client

connections by the followingcomponents:

443: The Secure Socket Layer (SSL)Protocol in Secure Service AgentMode HP 3PAR Service Processor HP 3PAR Connection Portal HP

3PAR Service Processor HP 3PAR OS CLI Client

HP 3PAR Connection Portal

HP 3PAR Service Processor --> WWWUsed by the HP 3PAR ServiceProcessor component to communicateusing the HTTP or HTTPS protocol.

80: World Wide Web (WWW) forHTTP in Legacy Mode

443: WWW for HTTPS in SecureService Agent Mode

Service Processor Storage SystemEvent Service

Passes unsolicited events from thestorage system and is used by the HP3PAR Service Processor component.

5781: 3PAR Event Reporting Service

(This port was used for HP 3PARManagement Console events prior toOS version 2.3.1)

HP 3PAR IMC HP 3PAR StorageSystem CLI Server

Used by the following components forstorage system monitoring and

5782: 3PAR Management Service(unsecured)

HP 3PAR CLI HP 3PAR StorageSystem CLI Server

configuration over an unsecuredchannel:

HP 3PAR Management Console Recovery Manager HP 3PARStorage System CLI Server HP 3PAR OS CLI

HP 3PAR System Reporter HP 3PARStorage System CLI Server

HP 3PAR Recovery Manager

HP 3PAR System ReporterHP 3PAR Service Processor --> HP 3PARStorage System CLI Server HP 3PAR Service Processor

(Port 2540 was used prior to SPversion 2.2.4.)

NOTE: If configured to do so, the storage system is capable of communication with acustomer-defined host via UDP port 514 (syslog).

Documents

Manuals for HP 3PAR Service Processors