EMC ControlCenter 6...EMC ControlCenter 6.0 Installation and Operational Troubleshooting Guide Software Technical Support Group 6.0.0 Thursday, June 21, 2007 P/N 300-004-815 Section

EMC ControlCenter 6.0 Installation and Operational Troubleshooting Guide

Software Technical Support Group 6.0.0

Thursday, June 21, 2007

P/N 300-004-815 Rev A01

EMC ControlCenter 6.0 Installation and Operational Troubleshooting Guide 1

About This Document This document provides 6.0 version specific troubleshooting guidelines for resolving component installation, discovery, ControlCenter component operation, and ControlCenter product functionality problems.

Version specific exceptions are noted, and others will be added as future features and functionality are introduced.

In addition, this document provides information to assist with documenting problem details and outlines relevant log collections based on the product or component type the problem is being reported against.

Completion of the troubleshooting steps, and collection of all specified log files outlined in this document are necessary to sufficiently root cause ControlCenter component operation failures or functionality problems. It is highly recommended that these steps be performed in a timely fashion following the initial problem occurrence, and that the log files collected contain data from the period of time in which the problem was first encountered. Additionally, a thorough description of the steps necessary to reproduce the problem, a screen capture displaying the problem, and a comprehensive description of the problem environment should be collected prior to opening a case with the EMC Software Support Center.

This document refers specifically to the troubleshooting of EMC ControlCenter software.

Given the complexity of ControlCenter distributed systems and their interfaces, this document is intended to be used in conjunction with other ControlCenter documentation.

Locations of other ControlCenter documentation and reference material An end user viewable set of the ControlCenter documentation is available on Powerlink:

http://powerlink.emc.com Audience This document is intended for use by EMC service personnel. Copyright © 2006 EMC® Corporation. All rights reserved.

EMC believes the information in this publication is accurate as of its publication date. The information is subject to change without notice.

THE INFORMATION IN THIS PUBLICATION IS PROVIDED “AS IS.” EMC CORPORATION MAKES NO REPRESENTATIONS OR WARRANTIES OF ANY KIND WITH RESPECT TO THE INFORMATION IN THIS PUBLICATION, AND SPECIfICALLY DISCLAIMS IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.

Use, copying, and distribution of any EMC software described in this publication requires an applicable software license.


http://powerlink.emc.com/

Section I Component Installation ..........................................................................6

Installation (General/All Components)...............................................................6 Installation (Repository).....................................................................................9 Installation (Server) .........................................................................................11 Installation (Media Repository) ........................................................................13 Installation (Web Server).................................................................................15 Installation (NAT Environment)........................................................................17 Installation (Multihome Server)........................................................................21 Installation (Store) ...........................................................................................25 Installation (WebConsole Server)....................................................................27 Installation (API Server) ..................................................................................29 Installation (StorageScope) .............................................................................31 Installation (Performance Manager) ................................................................34 Installation (Master Agent on Infrastructure) ...................................................37 Installation (Master Agent Non-Infrastructure).................................................39 Installation (Console).......................................................................................42 ControlCenter 5.2 to 6.0 Differences ...............................................................44 Agents no Longer Supported in ControlCenter 6.0..........................................45 ControlCenter 6.0 Infrastructure Log File Locations ........................................46 Solutions Enabler Media Repository Load Kit Installation ...............................47 Solutions Enabler Push Installation .................................................................48 Agent Remote Install-Uninstall ........................................................................50 Upgrade 5.2 Infrastructure to 6.0.....................................................................53 Upgrade Service Pack.....................................................................................55

Section II Discovering Objects ............................................................................57 CLARiiON Array Discovery .............................................................................57 Symmetrix Array Local Discovery....................................................................61 Symmetrix Array Proxy Discovery ...................................................................64 IBM ESS Array Discovery ...............................................................................69 HDS Array Discovery ......................................................................................72 StorageWorks Array Discovery .......................................................................76 SMI Array Discovery........................................................................................80 SMI Arrays Provider ........................................................................................83 Fujitsu Siemens BS2000 Host Discovery ........................................................87 Invista Discovery .............................................................................................90 Centera Discovery...........................................................................................94 Celerra Discovery............................................................................................98 Host Discovery ..............................................................................................103 VMWare ESX Server Discovery ....................................................................107


Switch Discovery ...........................................................................................112 Oracle Database Discovery...........................................................................117 Oracle Database Proxy Discovery.................................................................121 Oracle Database Cluster Discovery ..............................................................125 Database Discovery using Common Mapping Agent (non-Oracle) ...............128 File system Discovery ...................................................................................132

Section III Runtime Issues ................................................................................135 Data Collection Policies.................................................................................135 Alerts .............................................................................................................138 Windows Based Authentication .....................................................................142 LDAP Authentication .....................................................................................144 Managed Object Deletion..............................................................................147 Managed Object Exploring ............................................................................150 Real-Time Performance Data Retrieval.........................................................153 Symmetrix Optimizer .....................................................................................156 Symmetrix Configuration Management – Configuration Changes.................159 Symmetrix TimeFinder - BCVs......................................................................162 Symmetrix TimeFinder - Clone......................................................................165 Symmetrix TimeFinder - SNAP .....................................................................168 Symmetrix SRDF...........................................................................................171 Storage Provisioning Service – Symmetrix, CLARiiON, StorageWorks ........174 Storage Provisioning Service – HP StorageWorks XP ..................................179 SAN Manager Zoning....................................................................................182 SAN Manager Masking Operations ...............................................................185 ControlCenter Server ....................................................................................188 ControlCenter Store ......................................................................................191 ControlCenter Repository Database..............................................................194 ControlCenter ECC API.................................................................................196 ControlCenter Web Console Server ..............................................................197 ControlCenter Web Console .........................................................................199 Agent Run Time ............................................................................................201 Agents Performance – Resource Usage .......................................................204 Agents Cluster Failover .................................................................................207 Agents Load Balancing .................................................................................210 FCC Agent SNMP Port Configurations..........................................................213 Veritas NetBackup Agent ..............................................................................215 ControlCenter 3rd Party Integration Packages ...............................................218 ControlCenter Workload Analyzer – Performance Manager .........................220 ControlCenter StorageScope/SRM ...............................................................223 ControlCenter StorageScope Data Migration Utility ......................................226

Appendix A .......................................................................................................229


EMC ControlCenter Log Collection Wizard ...................................................229


Section I Component Installation

Installation (General/All Components) How do I troubleshoot a ControlCenter 6.0 Install? What do I verify if I have problems installing ControlCenter 6.0? Troubleshooting StepsVerify you have completed the Generic Install Procedure steps:

• Disable the ‘Terminal Services’ service and, if it exists, the ‘Terminal Server Licensing’ service and then reboot. • Stop any active anti-virus software until installation is complete. • Stop Distributed Transaction Coordinator service until installation is complete. This is automatically checked at

the start of installation, but you may want to verify that it is still disabled. • Ensure %SYSTEMROOT%, for instance c:\WINNT\system32, is present in the PATH environment variable.

o In a command window, run set, check Path and SystemRoot. • Run in English, Japanese, French, German, or Korean language environment only. (Infrastructure only.)

o Reference the ControlCenter Support Matrix for a list of the ControlCenter components that German, French, Japanese, and Korean language versions of operating systems can tolerate. Only US English-OS versions of Windows 2000 and Windows 2003 are supported, with the regional Locale set to the supported languages. Because ControlCenter does not support German, French, Japanese, or Korean language usage, use regional Locale English during the installation so the pathname, folder name, and username are correct.

• If a component does not appear in the Available Components List, it could be one of the following issues: o The component is already installed. o The prerequisites are not installed. o Depending on the component, you may be only allowed to install it once in one location.

Check for any CD or disk related problems in the Windows Event Viewer. Stop any wperl.exe and javaw.exe processes running in Task Manager, then re-launch the

Install Wizard. • Close Microsoft “Services” window prior to installation. • Close Microsoft “Computer Management” window prior to installation. • Ping the Repository using host name and IP and echo to a file. In some environments, security policies may

mandate that ICMP (Internet Control Message Protocol) be disabled. This is not controlled by ControlCenter but is needed for the Ping command. Verify ICMP is enabled for communication between distributed infrastructure components.

• Verify the following two Repository services are installed, set to automatic startup type, and their Status is ‘Started’:

o OracleECCREP_HOMETNSListener o OracleServiceRAMBDB

• Confirm installed components by running the following utility and redirecting the output to a file: o %ECC_INSTALL_ROOT%\tools\utils\regutil All show > regutil_all_show.txt

Information to Collect Note: All logs must be time relevant to the problem event. In addition, Server and Store log file sets must contain all available trace files. Using the Log Collection Wizard please select the following category and subcategories which will collect the recommended log files indicated below. Deselect any log types or log files that are not applicable to your environment. For instructions on using the Log Collection Wizard see emc154217. LCW Category: Installations and Upgrades

LCW Subcategory: Installation PRINCIPAL_INSTALL_LOG -FOR ALL INSTALL ISSUES

• %SystemDrive%\TEMP\EMC ControlCenter 6.*.log


INFRASTRUCTURE_SERVICES • %SystemDrive%\TEMP\ECCWebConsole_Service.log • %SystemDrive%\TEMP\ECCAPIServer_Service.log • %SystemDrive%\TEMP\ECCTomcat_Service.log • %SystemDrive%\TEMP\ECCSTORE.log • %SystemDrive%\TEMP\ECCSERVER.log • %SystemDrive%\TEMP\ccinstall_log.txt • %SystemDrive%\TEMP\replaceLine.txt

INFRASTRUCTURE_LOGS • %ECC_INSTALL_ROOT%\cert.log • %ECC_INSTALL_ROOT%\log\*

REPOSITORY_INSTALL • %ECC_INSTALL_ROOT%\Repository\cfgtoollogs\dbca\rambdb\*.log • %ECC_INSTALL_ROOT%\Repository\cfgtoollogs\dbca\trace.log

REPOSITORY • %ECC_INSTALL_ROOT%\Repository\NETWORK\log\listener.log • %ECC_INSTALL_ROOT%\Repository\NETWORK\ADMIN\tnsnames.ora • %ECC_INSTALL_ROOT%\Repository\NETWORK\ADMIN\listener.ora • %ECC_INSTALL_ROOT%\Repository\NETWORK\ADMIN\sqlnet.ora • %ECC_INSTALL_ROOT%\Repository\admin\Ramb_scripts\upgrade\*.log • %ECC_INSTALL_ROOT%\Repository\admin\Ramb_scripts\ecc_getversion.log • %ECC_INSTALL_ROOT%\Repository\admin\rambdb\bdump\alert_rambdb.log • %ECC_INSTALL_ROOT%\Repository\admin\rambdb\bdump\*.trc • %ECC_INSTALL_ROOT%\Repository\admin\rambdb\pfile\* • %ECC_INSTALL_ROOT%\Repository\assistants\dbca\logs\*

ORACLE • %ProgramFiles%\Oracle\Inventory\logs\* • %ProgramFiles%\Oracle\Inventory\logs\results\db\*

CONSOLE_INSTALL_WINDOWS • %ProgramFiles%\ecc\console\ecclog.txt

CONSOLE_INSTALL_SOLARIS • /opt/ecc/console/ecclog.txt

STORAGESCOPE_INSTALL • %SystemDrive%\TEMP\ECCWebConsole_Service.log • %SystemDrive%\TEMP\ECCAPIServer_Service.log • %SystemDrive%\TEMP\ECCTomcat_Service.log

STS_REPOSITORY_INSTALL • %ECC_INSTALL_ROOT%\Repository\cfgtoollogs\dbca\emcstsdb\*.log • %ECC_INSTALL_ROOT%\Repository\cfgtoollogs\dbca\trace.log

STS_REPOSITORY • %ECC_INSTALL_ROOT%\Repository\admin\emcsts_scripts\*.log • %ECC_INSTALL_ROOT%\Repository\admin\emcsts_scripts\*.err • %ECC_INSTALL_ROOT%\Repository\admin\emcsts_scripts\*.INI • %ECC_INSTALL_ROOT%\Repository\admin\emcstsdb\bdump\alert_emcstsdb.log • %ECC_INSTALL_ROOT%\Repository\admin\emcstsdb\bdump\*.trc • %ECC_INSTALL_ROOT%\Repository\NETWORK\log\listener.log • %ECC_INSTALL_ROOT%\Repository\NETWORK\ADMIN\tnsnames.ora • %ECC_INSTALL_ROOT%\Repository\NETWORK\ADMIN\listener.ora • %ECC_INSTALL_ROOT%\Repository\NETWORK\ADMIN\sqlnet.ora • %ECC_INSTALL_ROOT%\Repository\assistants\dbca\logs\*

PERFORMANCEMANAGER_INSTALL • %SystemDrive%\TEMP\ECCWebConsole_Service.log • %SystemDrive%\TEMP\ECCAPIServer_Service.log • %SystemDrive%\TEMP\ECCTomcat_Service.log

MGA -MASTER AGENT • [ECC_HOME]\*.log • [ECC_HOME]\*.rcfile • [ECC_HOME]\ctg.ini • [ECC_HOME]\master.ini


• [ECC_HOME]\MGA6*\MGA.log • [ECC_HOME]\MGA6*\MGA_B*.log • [ECC_HOME]\MGA6*\MGA_Err.log • [ECC_HOME]\MGA6*\installs\* • [ECC_HOME]\MGA6*\prereqs\*

MASTERAGENT_INSTALL_WINDOWS • %ECC_INSTALL_ROOT%\EMC ControCenter Master Agent_Log_*.txt

MASTERAGENT_INSTALL_UNIX • /usr/ecc/installMA.log • /usr/ecc/installMTR* • /usr/ecc/tmpunzip*

MRLK_INSTALL • %ECC_INSTALL_ROOT%\installSEK* • %ECC_INSTALL_ROOT%\media_rep\remoteinstall\SEK6*\create_ecc_kits.log

INTEGRATION_GATEWAY • %ECC_INSTALL_ROOT%\InstallIntPkg\ecc3pi\Win32\SETUP.INI • %ECC_INSTALL_ROOT%\InstallIntPkg\ecc3pi\Win32\*.log

OS Commands: (From EMC ControlCenter Server host)

• netstat -an • ipconfig /all • winmsd nfo file • set • regedit /e

Repository Exports: None specified. Additional Files:

• From the host where the EMC ControlCenter Server is installed: o %ECC_INSTALL_ROOT%\tools\utils\regutil All show > regutil_all_show.txt

• From the host where the Installation Wizard is being run: o Capture a screen shot of the EMC ControlCenter install GUI at the time of the failure

• From all non-EMC ControlCenter Server hosts involved in the installation o Windows: In a command prompt window, capture the output of the following commands redirected

to a filename. Make output file names unique if multiple hosts. dir /s %ECC_Install_ROOT% > eccdir.txt nslookup <hostname> > nslookup.txt for all hosts involved for this problem netstat –an > netstat_an.txt ipconfig /all > ipconfig_all.txt winmsd /s c:\tmp\<hostname>.nfo set > set.txt regedit /e regedit_e.txt

• From all hosts involved in the installation: o Capture the Windows event viewer logs:

%windir%\system32\config\SysEvent.Evt %windir%\system32\config\AppEvent.Evt %windir%\System32\config\SecEvent.Evt


Installation (Repository) How do I troubleshoot a ControlCenter 6.0 Repository Install? What do I verify if I have problems installing ControlCenter 6.0 Repository? Troubleshooting Steps Verify you have completed the Generic Install Procedure steps:

• Disable the ‘Terminal Services’ service and, if it exists, the ‘Terminal Server Licensing’ service, then reboot. • Stop any active anti-virus software until installation is complete. • Stop Distributed Transaction Coordinator service until installation is complete. This is automatically checked at






Install Wizard. • Close Microsoft “Services” window prior to installation. • Close Microsoft “Computer Management” window prior to installation.

These steps are just a guide and should be done prior to installation. If the installation has started the suggested steps may not be appropriate, in some cases. This is the relational database that will hold current and historical data about EMC ControlCenter™. This is the main Repository, and there is only one main Repository per implementation. The main Repository is the first infrastructure component to be installed. If the Repository install exits with a message indicating that the Oracle product is on the system, do the following: Ensure this is the correct system and uninstall Oracle from this system. Engage site administrators.

• Make a backup of the registry and place the backup on external media. • Delete the following Oracle registry keys: (Open regedit and search for ‘Oracle’). Use extreme caution!

o HKEY_LOCAL_MACHINE\SOFTWARE\ORACLE o HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Oracle*

• Delete the Oracle folder [by default C:\Program Files\Oracle] • Install locally on the host or via a supported console redirection method. Do NOT use Terminal Services. • Ensure that any ‘Terminal Server’ Services are not running. • If this registry key exists, verify it has not been corrupted: REPOSITORY_HOSTNAME at the following location:

o HKEY_LOCAL_MACHINE\SOFTWARE\EMC Corporation\EMC CONTROL CENTER\6.0 • Verify the following two Repository services are installed and set to automatic startup type:



Information to Collect Note: All logs must be time relevant to the problem event. In addition, Server and Store log file sets must contain all available trace files.


Using the Log Collection Wizard please select the following category and subcategories which will collect the recommended log files indicated below. Deselect any log types or log files that are not applicable to your environment. For instructions on using the Log Collection Wizard see emc154217. LCW Category: Installations and Upgrades


• %SystemDrive%\TEMP\EMC ControlCenter 6.*.log INFRASTRUCTURE_LOGS

• %ECC_INSTALL_ROOT%\cert.log • %ECC_INSTALL_ROOT%\log\*














Installation (Server) How do I troubleshoot a ControlCenter 6.0 Server Install? What do I verify if I have problems installing ControlCenter 6.0 Server? Troubleshooting Steps Verify you have completed the Generic Install Procedure steps:











• Confirm installed components by running the following utility and redirecting the output to a file: %ECC_INSTALL_ROOT%\tools\utils\regutil All show > regutil_all_show.txt



• %SystemDrive%\TEMP\EMC ControlCenter 6.*.log INFRASTRUCTURE_SERVICES

• %SystemDrive%\TEMP\ECCSERVER.log • %SystemDrive%\TEMP\ccinstall_log.txt


• %SystemDrive%\TEMP\replaceLine.txt INFRASTRUCTURE_LOGS















Installation (Media Repository) How do I troubleshoot a ControlCenter 6.0 Media Repository Install? What do I verify if I have problems installing ControlCenter 6.0 Media Repository? Troubleshooting Steps Verify you have completed the Generic Install Procedure steps:

• Disable the ‘Terminal Services’ service and, if it exists, the ‘Terminal Server Licensing’ service, then reboot • Stop any active anti-virus software until installation is complete. • Stop Distributed Transaction Coordinator service until installation is complete. This is automatically checked at














• %SystemDrive%\TEMP\ECCSERVER.log • %SystemDrive%\TEMP\ccinstall_log.txt


• %SystemDrive%\TEMP\replaceLine.txt INFRASTRUCTURE_LOGS















Installation (Web Server) How do I troubleshoot a ControlCenter 6.0 Web Server Install? What do I verify if I have problems installing ControlCenter 6.0 Web Server? Troubleshooting Steps Verify you have completed the Generic Install Procedure steps:















• %SystemDrive%\TEMP\ECCTomcat_Service.log • %SystemDrive%\TEMP\ECCSERVER.log


• %SystemDrive%\TEMP\ccinstall_log.txt • %SystemDrive%\TEMP\replaceLine.txt















Installation (NAT Environment) How do I troubleshoot a ControlCenter 6.0 NAT environment Install? What do I verify if I have problems installing ControlCenter 6.0 in a NAT environment? Background ControlCenter Network Address Translation (NAT) support encompasses the following:

• Only Static NAT address translation is supported. • Dynamic NAT is not supported. • Port Address Translation (PAT) is not supported. • All ControlCenter agents are supported outside the NAT firewall. • ControlCenter Agents that communicate through a NAT firewall to a WLA Archiver agent are not supported.

Hence, WLA DCPs of agents deployed outside of NAT firewall should be disabled. • ControlCenter infrastructure components such as Repository, Server, Store, Web Console, and Console are not

supported outside the NAT firewall. • WLA Archiver component is not supported outside the NAT firewall.

When ControlCenter components reside in a private inside network behind a NAT firewall, and agents are behind another firewall with the Internet between the two NAT firewalls, you must modify ControlCenter component configurations to allow communications to occur between ControlCenter components across the firewall. NAT firewalls are set up to conceal the internal network addresses from the outside networks.

• ControlCenter components (ControlCenter Agents) that reside outside the NAT firewall must communicate with the ECC Server host using the outside global IP address of the ECC Server.

• Only ControlCenter Agents are supported outside the NAT firewall. Hence, only the Master Agent's ctg.ini file needs modification to support NAT communication.

Note: This information is located in the ControlCenter 6.0 Planning and Installation Guide Volume 1. For setup examples including illustrations, please refer to this guide which is available on Powerlink. Master Agent When the Master Agent is installed and communicating to the ControlCenter infrastructure though a NAT firewall, modify the agent ctg.ini file to use the ECC Server outside global address as follows: Note: The fully qualified domain name (FQDN) hostname should be used, and is recommended, in place of the IP address.

1. Stop all ControlCenter agents and the Master Agent. 2. Edit the following file, where <install_root> is the root ControlCenter install directory:

<install_root>/exec/ctg.ini 3. In ctg.ini, update the Server Host parameter to specify the outside global IP address of the ECC Server (NIC1)

that will be used to communicate with the agent host. 4. Restart all ControlCenter agents and the Master Agent.

MVS Master Agent The MVS Master Agent (MMNSERVR) determines the network location of the ECC server from a dataset determined by the installation's dataHLQ, as follows:

dataHLQ.SSTINI This exact specification for that dataset will be found in the member MMNSERVR in either the dataHLQ.PROCLIB or in the host's current JES proclib concatenation, and it should look something like:

//ECCINI DD DISP=SHR,DSN=dataHLQ.SSTINI(MMN600) The MMN600 member must be changed as follows:

1. Find the line containing the string: Server host = PARM 2. Change the current PARM parameter after the = sign to reflect the outside global host name or IP address of

the ECC server. For example: Server host = ECCHOSTA or Server host = 40.1.1.40 3. Restart all ControlCenter agents and the Master Agent.

Note: In order for communications to occur with specific agents, you must ensure that ports enabling firewall communication are open for those agents. Refer to the ControlCenter 6.0 Planning and Installation Guide Volume 1 for detailed port information.


Troubleshooting Steps Verify you have completed the Generic Install Procedure steps:















• %SystemDrive%\TEMP\ECCWebConsole_Service.log • %SystemDrive%\TEMP\ECCAPIServer_Service.log • %SystemDrive%\TEMP\ECCTomcat_Service.log • %SystemDrive%\TEMP\ECCSTORE.log • %SystemDrive%\TEMP\ECCSERVER.log • %SystemDrive%\TEMP\ccinstall_log.txt • %SystemDrive%\TEMP\replaceLine.txt

INFRASTRUCTURE_LOGS






CONSOLE_INSTALL_WINDOWS • %ProgramFiles%\ecc\console\ecclog.txt

CONSOLE_INSTALL_SOLARIS • /opt/ecc/console/ecclog.txt

MGA -MASTER AGENT • [ECC_HOME]\*.log • [ECC_HOME]\*.rcfile • [ECC_HOME]\ctg.ini • [ECC_HOME]\master.ini • [ECC_HOME]\MGA6*\MGA.log • [ECC_HOME]\MGA6*\MGA_B*.log • [ECC_HOME]\MGA6*\MGA_Err.log • [ECC_HOME]\MGA6*\installs\* • [ECC_HOME]\MGA6*\prereqs\*



OS Commands (From EMC ControlCenter Server host):






to a filename. Make output file names unique if multiple hosts. dir /s %ECC_Install_ROOT% > eccdir.txt


nslookup <hostname> > nslookup.txt for all hosts involved for this problem netstat –an > netstat_an.txt ipconfig /all > ipconfig_all.txt winmsd /s c:\tmp\<hostname>.nfo set > set.txt regedit /e regedit_e.txt




Installation (Multihome Server) How do I troubleshoot a ControlCenter 6.0 Multihome Server Install? What do I verify if I have problems installing ControlCenter 6.0 Multihome Server? Background Deployment of multihome should be planned prior to starting the installation as documented in the EMC ControlCenter 6.0 Installation and Planning Guide Volume 1. During the initial installation of ControlCenter, the installation menu prompts for selection of multihome hostname/IP addresses. For cases when the planning is not done at the initial installation of ControlCenter or when the environment has been changed for multihome, this section describes the modifications needed to the infrastructure .ini files to support multihome. Note: This information is located in the ControlCenter 6.0 Planning and Installation Guide Volume 1. For setup examples including illustrations, please refer to this guide which is available on Powerlink.. Repository Follow this procedure to enable ControlCenter multihome support:

1. On the ControlCenter Repository host, select Start, Settings, Control Panel, and then open Administrative Tools. Open Services and stop the following Oracle database services:

o OracleServiceRAMBDB o OracleECCREP_HOMETNSListener

2. On the ControlCenter Repository host, edit the following file, where %ECC_INSTALL_ROOT% is the root ControlCenter install directory: %ECC_INSTALL_ROOT%\Repository\NETWORK\ADMIN\listener.ora

3. In listener.ora, update the ECC_Server_IP_Address field (which follows) with the IP address of NIC1 and NIC2 from which the Repository listens to the DB connection requests.

o (ADDRESS = (PROTOCOL = TCP) (HOST = <ECC_Server_IP_Address>) (PORT = 1521) o (ADDRESS = (PROTOCOL = TCP) (HOST = <ECC_Server_IP_Address>) (PORT = 1521)

Sample code from the listener.ora file: LISTENER =

(DESCRIPTION_LIST = (DESCRIPTION =

(ADDRESS_LIST = (ADDRESS = (PROTOCOL = IPC)(KEY = EXTPROC0))

) (ADDRESS_LIST =

(ADDRESS = (PROTOCOL = TCP)(HOST = 150.1.1.50)(PORT = 1521) (ADDRESS = (PROTOCOL = TCP)(HOST = 170.1.1.70)(PORT = 1521)

) ) ) )

4. Save and exit listener.ora. 5. Restart the Oracle services that you stopped earlier in this procedure.

ECC server.ini File Modifications for Multihome

1. On the ECC Server host, select Start>Settings>Control Panel>Administrative Tools. Open Services and stop the following service: EMC ControlCenter Server

2. On the ECC Server host, edit the following file, where %ECC_INSTALL_ROOT% is the root ControlCenter install directory: %ECC_INSTALL_ROOT%\ecc_inf\data\ecc_inf_hostname\data\ecc_server.ini

3. In ecc_server.ini, update the ECC_Server_IP_Address field shown below to specify the IP address for an ECC Server listening port:

[Voyager] name server url = //<ECC_Server_IP_Address>:30751

4. Specify the additional NIC IP addresses on the name server aliases parameter in ecc_server.ini. The hostnames or IP addresses specified on name server aliases parameter must be different than the one specified on name


server url. By populating name server aliases, it will allow Consoles (for example) on the additional network to communicate with the Server.

5. Specify the IP addresses of the ECC Server NIC1 and NIC2 on the Database Host Name parameter in ecc_server.ini. For example: Database Host Name = 150.1.1.50, 170.1.1.70

6. Save and exit ecc_server.ini. 7. Restart the EMC ControlCenter Server service that you stopped earlier in this procedure.

ECC store.ini File Modifications for Multihome It important to understand that a multihome Store can use one of two paths (for example: NIC1 or NIC2) to communicate to the ECC Server and Repository. This applies to a Store in the single host infrastructure or an external Store in a distributed configuration. If NIC1 is the communication path from the ECC Store, it is important that the Repository is also listening on NIC1 at port 1521 and the Server is listening on NIC1 at port 30751 to complete the communication paths for the Store connection request.

1. On the ECC Store host, select Start>Settings>Control Panel>Administrative Tools. Open Services and stop the following service: EMC ControlCenter Store

2. On the ECC Store host, edit the following file, where %ECC_INSTALL_ROOT% is the root ControlCenter install directory: %ECC_INSTALL_ROOT%\ecc_inf\data\ecc_inf_hostname\data\ecc_store.ini

3. In ecc_store.ini, update the ECC_Server_IP_Address field shown next to specify the IP address for an ECC Server listening port: Server host= <ECC_Server_IP_Address>

4. In ecc_store.ini, update the ECC_Server_IP_Address field shown next to specify the IP address for an ECC Server listening ports:

[Voyager] name server url = //<ECC_Server_IP_Address>:30751 login url = //<ECC_Server_IP_Address>:30721

5. Save and exit ecc_store.ini. 6. Restart the EMC ControlCenter Store service that you stopped earlier in this procedure.

Troubleshooting Steps Verify you have completed the Generic Install Procedure steps:





















OS Commands (From EMC ControlCenter Server host):





• From the host where the Installation Wizard is being run:

o Capture a screen shot of the EMC ControlCenter install GUI at the time of the failure • From all non-EMC ControlCenter Server hosts involved in the installation

o Windows: In a command prompt window, capture the output of the following commands redirected to a filename. Make output file names unique if multiple hosts.

dir /s %ECC_Install_ROOT% > eccdir.txt nslookup <hostname> > nslookup.txt for all hosts involved for this problem netstat –an > netstat_an.txt ipconfig /all > ipconfig_all.txt winmsd /s c:\tmp\<hostname>.nfo set > set.txt regedit /e regedit_e.txt




Installation (Store) How do I troubleshoot a ControlCenter 6.0 Store Install? What do I verify if I have problems installing ControlCenter 6.0 Store? Troubleshooting Steps Verify you have completed the Generic Install Procedure steps:















• %SystemDrive%\TEMP\ECCSTORE.log


• %SystemDrive%\TEMP\ECCSERVER.log • %SystemDrive%\TEMP\ccinstall_log.txt • %SystemDrive%\TEMP\replaceLine.txt















Installation (WebConsole Server) How do I troubleshoot a ControlCenter 6.0 WebConsole Server Install? What do I verify if I have problems installing ControlCenter 6.0 WebConsole Server? Troubleshooting Steps Verify you have completed the Generic Install Procedure steps:











• Confirm installed components by running the following utility and redirecting the output to a file: %ECC_INSTALL_ROOT%\tools\utils\regutil All show > regutil_all_show.txt




• %SystemDrive%\TEMP\ECCWebConsole_Service.log • %SystemDrive%\TEMP\ECCAPIServer_Service.log


• %SystemDrive%\TEMP\ECCTomcat_Service.log • %SystemDrive%\TEMP\ECCSERVER.log • %SystemDrive%\TEMP\ccinstall_log.txt • %SystemDrive%\TEMP\replaceLine.txt















Installation (API Server) How do I troubleshoot a ControlCenter 6.0 API Server Install? What do I verify if I have problems installing ControlCenter 6.0 API Server? Troubleshooting Steps Verify you have completed the Generic Install Procedure steps:















• %SystemDrive%\TEMP\ECCWebConsole_Service.log • %SystemDrive%\TEMP\ECCAPIServer_Service.log


• %SystemDrive%\TEMP\ECCTomcat_Service.log • %SystemDrive%\TEMP\ECCSERVER.log • %SystemDrive%\TEMP\ccinstall_log.txt • %SystemDrive%\TEMP\replaceLine.txt















Installation (StorageScope) How do I troubleshoot a ControlCenter 6.0 StorageScope Install? What do I verify if I have problems installing ControlCenter 6.0 StorageScope? Background EMC ControlCenter StorageScope 6.0 provides a variety of features (snapshots, views, built-in and custom reports, database queries) to help you:

• Assess your current storage environment • Determine future storage needs based on historical usage and trending reports • Reclaim unused or under utilized storage resources

StorageScope license keys are provided with the EMC ControlCenter software package. Two licenses are provided: • StorageScope license - allows you to deploy agents and enables all StorageScope functions with the exception

of file level reporting • Optional StorageScope File Level Reporting license - enables file level data collection policies (DCPs) and file

level reporting. The ControlCenter 6.0 Planning and Installation Guide Volume 1 provides detailed information about which features and applications are enabled with each license. Troubleshooting Steps Verify you have completed the Generic Install Procedure steps:












• Verify that the host: o Has access to the ControlCenter Repository host through TCP/IP.


o Uses DNS to resolve names and IP addresses. o MDAC 2.8 (or above) is required for hosts running StorageScope. It can be downloaded from

Microsoft’s website. Information to Collect Note: All logs must be time relevant to the problem event. In addition, Server and Store log file sets must contain all available trace files. Using the Log Collection Wizard please select the following category and subcategories which will collect the recommended log files indicated below. Deselect any log types or log files that are not applicable to your environment. For instructions on using the Log Collection Wizard see emc154217. LCW Category: Installations and Upgrades


• %SystemDrive%\TEMP\EMC ControlCenter 6.*.log REPOSITORY

• %ECC_INSTALL_ROOT%\Repository\NETWORK\log\listener.log • %ECC_INSTALL_ROOT%\Repository\NETWORK\ADMIN\tnsnames.ora • %ECC_INSTALL_ROOT%\Repository\NETWORK\ADMIN\listener.ora • %ECC_INSTALL_ROOT%\Repository\NETWORK\ADMIN\sqlnet.ora • %ECC_INSTALL_ROOT%\Repository\admin\Ramb_scripts\upgrade\*.log • %ECC_INSTALL_ROOT%\Repository\admin\Ramb_scripts\ecc_getversion.log • %ECC_INSTALL_ROOT%\Repository\admin\rambdb\bdump\alert_rambdb.log • %ECC_INSTALL_ROOT%\Repository\admin\rambdb\bdump\*.trc • %ECC_INSTALL_ROOT%\Repository\admin\rambdb\pfile\* • %ECC_INSTALL_ROOT%\Repository\assistants\dbca\logs\*


STORAGESCOPE_INSTALL • %SystemDrive%\TEMP\ECCWebConsole_Service.log • %SystemDrive%\TEMP\ECCAPIServer_Service.log • %SystemDrive%\TEMP\ECCTomcat_Service.log

STS_REPOSITORY_INSTALL • %ECC_INSTALL_ROOT%\Repository\cfgtoollogs\dbca\emcstsdb\*.log • %ECC_INSTALL_ROOT%\Repository\cfgtoollogs\dbca\trace.log


OS Commands: None specified. Repository Exports: None specified. Additional Files:









Installation (Performance Manager) How do I troubleshoot a ControlCenter 6.0 Performance Manager Install? What do I verify if I have problems installing ControlCenter 6.0 Performance Manager? Troubleshooting Steps Verify you have completed the Generic Install Procedure steps:












• The WLA.ini file contains the Alias (IP address of the ControlCenter Repository) and the Server (ecc) IP. • Default Performance View/Performance Manager install location:

%ECC_Install_ROOT%\Client\PerformanceManager • WLA Performance View/Performance Manager is an Oracle client and will install an Oracle Client in C:\Program

Files\Oracle and %ECC_Install_ROOT%\Client, if an Oracle client is not found. Information to Collect Note: All logs must be time relevant to the problem event. In addition, Server and Store log file sets must contain all available trace files. Using the Log Collection Wizard please select the following category and subcategories which will collect the recommended log files indicated below. Deselect any log types or log files that are not applicable to your environment. For instructions on using the Log Collection Wizard see emc154217.


LCW Category: Installations and Upgrades LCW Subcategory: Installation

PRINCIPAL_INSTALL_LOG -FOR ALL INSTALL ISSUES • %SystemDrive%\TEMP\EMC ControlCenter 6.*.log

INFRASTRUCTURE_SERVICES • %SystemDrive%\TEMP\ECCSERVER.log • %SystemDrive%\TEMP\ccinstall_log.txt • %SystemDrive%\TEMP\replaceLine.txt





PERFORMANCEMANAGER_INSTALL • %SystemDrive%\TEMP\ECCWebConsole_Service.log • %SystemDrive%\TEMP\ECCAPIServer_Service.log • %SystemDrive%\TEMP\ECCTomcat_Service.log












Installation (Master Agent on Infrastructure) How do I troubleshoot a ControlCenter 6.0 Master Agent Install on the Infrastructure? What do I verify if I have problems installing ControlCenter 6.0 Master Agent Install on the Infrastructure? Troubleshooting Steps Verify you have completed the Generic Install Procedure steps:






Check for any CD or disk related problems in the Windows Event Viewer. Stop any wperl.exe and javaw.exe processes running in Task Manager, then relaunch

Install Wizard. • Close Microsoft “Services” window prior to installation. • Close Microsoft “Computer Management” window prior to installation. • Check that the following Registry key is created: HKEY_LOCAL_MACHINE\SOFTWARE\EMC Corporation\EMC

CONTROL CENTER\6.0 points to a string value: MA_ROOT. MA_ROOT refers to the root ECC install directory which is created.




• %SystemDrive%\TEMP\ccinstall_log.txt • %SystemDrive%\TEMP\replaceLine.txt



• netstat -an • ipconfig /all • winmsd nfo file


• set • regedit /e









Installation (Master Agent Non-Infrastructure) How do I troubleshoot a ControlCenter 6.0 Master Agent Install on an agent host? What do I verify if I have problems installing ControlCenter 6.0 Master Agent Install on an agent host? Troubleshooting Steps Identify the installation methodology performed:

• Windows Installation using ControlCenter installation GUI (MasterAgent.exe on disk #4) • Windows Native installer via the CLI command only • Windows Installation using command line • UNIX Installation using command line • PushMA utility (available on all platforms) • Native Install method on UNIX

Install via the ControlCenter installation GUI (MasterAgent.exe on disk #4) • Disable the ‘Terminal Services’ service and, if it exists, the ‘Terminal Server Licensing’ service, then reboot. • Stop any active anti-virus software until installation is complete. • Stop Distributed Transaction Coordinator service until installation is complete. This is automatically checked at






Install Wizard. • Close Microsoft “Services” window prior to installation. • Close Microsoft “Computer Management” window prior to installation. • Check that the following Registry key is created: HKEY_LOCAL_MACHINE\SOFTWARE\EMC Corporation\EMC

CONTROL CENTER\6.0 points to a string value: MA_ROOT. MA_ROOT refers to the root ECC install directory which is created.

• If ‘EMC ControlCenter Master Agent’ is not listed in Add/Remove Programs, o Check for the presence of :

A non-empty MA_ROOT folder o The installation performs a check for the presence of these. Most probable cause is an incomplete

uninstall of an older version of ControlCenter. o Remove the exec directory. o Check for the presence of the following Registry entry:

HKEY_LOCAL_MACHINE\SOFTWARE\EMC Corporation\EMC CONTROL CENTER\6.0 o Delete the String value: MA_ROOT if the above Registry key is found. o Restart the Master Agent installation.

Install via Windows Native Installer (setup.exe in ‘WinNativeInstall’ folder)

• Setup.exe must be run from CLI with the command line syntax specified in the Installation Guide • Ensure that the Master Agent is not already installed on the host


Windows Install via Command Line (install_master.bat) • Checks for the presence of the following Registry entry:

o HKEY_LOCAL_MACHINE\SOFTWARE\EMC Corporation\EMC CONTROL CENTER\6.0 • Delete the value for <MA_ROOT> if the above Registry key is found. • The installation performs a check for the presence of this directory. Most probable cause is an incomplete

uninstall of an older version of ControlCenter. • Remove the exec directory, and restart the Master Agent installation. • Collect the following documentation:

o <MA_ROOT>\installMA.log This log file contains the results of the actions performed, but does not contain a record of

what the user answered for the interactive questions. o Run regedit(32) export keys HKEY_LOCAL_SYSTEM\SYSTEM\CurrentcontrolSet\Services

UNIX Installation using command line:

• Determine the value of <MA_ROOT> on UNIX by examining the contents of the file: /var/emc/.eccrc. There will be no environment variable associated with MA_ROOT.

• Check for the presence of : <MA_ROOT>/exec o The installation performs a check for the presence of this directory. Most probable cause is an

incomplete uninstall of an older version of ControlCenter. • Remove the /var/emc/eccrc file, and the <MA_ROOT>/exec directory, and restart the Master Agent installation. • Collect the following documentation:

o <MA_ROOT>/installMA.log This log file contains the results of the actions performed by the install, but does not

contain a record of what the user answered for the interactive questions. o <MA_ROOT>/exec/ *

pushMA utility

• Supported only if the user installs the first instance of the Master Agent using the command line. • Supported on all operating systems, but can only be used to push to the same OS type and version, i.e., a user

would need to create one pushMA package for each OS needed. • No log files are created by this utility. Output is sent only to the screen.

Native install method

• Use of native install methods requires that the Native Install Package option be selected during at least one Master Agent installation on the desired UNIX OS, from the CD.

• Collect the following documentation: o Master Agent uninstall log files

Log files are determined by the operating system's native installer engine. o Windows Uninstall using GUI:

%ECC_Install_ROOT%\uninstallMTR*.log Command line uninstall for Windows

• No log files created. Output goes directly to screen. Command line uninstall for UNIX

• No log files created. Output goes directly to screen. Information to Collect Note: All logs must be time relevant to the problem event. In addition, Server and Store log file sets must contain all available trace files. Using the Log Collection Wizard please select the following category and subcategories which will collect the recommended log files indicated below. Deselect any log types or log files that are not applicable to your environment. For instructions on using the Log Collection Wizard see emc154217.


LCW Category: Installations and Upgrades LCW Subcategory: Installation

PRINCIPAL_INSTALL_LOG -FOR ALL INSTALL ISSUES • %SystemDrive%\TEMP\EMC ControlCenter 6.*.log




OS Commands: (From EMC ControlCenter Server host) None specified. Repository Exports: None specified. Additional Files:

• From the host where the Master Agent install was attempted: o Windows: In a command prompt window, capture the output of the following commands redirected

to a filename. Make output file names unique if multiple hosts. dir /s %ECC_Install_ROOT% > eccdir.txt winmsd /s c:\tmp\<hostname>.nfo (Windows) set > set.txt (Windows) env > env.txt (Unix) regedit /e regedit_e.txt (Windows)


Installation (Console) How do I troubleshoot a ControlCenter 6.0 Console Install? What do I verify if I have problems installing a ControlCenter 6.0 Console? Troubleshooting Steps

• The user logged on to the client host must have the appropriate privileges to install new software, write to ‘Program Files’, and create desktop icons, such as a local or domain Administrator.

o If a folder other than ‘Program Files’ is selected and is writable by the non-Administrator user, the install will complete and fail to create the desktop icons and start menu shortcuts, however the console install will complete.

o To start a console in this state: Open a command prompt and navigate to:<console_install_directory>\bin Type: createprocessfg launch.bat If the console starts up and the user is able to authenticate to the ControlCenter server,

the console software was successfully installed with the exception of shortcut icons and default installation paths.

• Verify ‘EMC ControlCenter Web Server’ service is started on EMC ControlCenter Server host. o The ‘EMC ControlCenter Web Server’ service will accept client connections and allow the remote

download of the ControlCenter console install. o Allow a 1-5 mintues for the service to fully initialize and begin accepting client connections

• Verify ‘EMC ControlCenter Server’ service is started on EMC ControlCenter Server host. o The ‘EMC ControlCenter Server’ service is not needed to install the Console, but it is needed to run

the Console. • Ping the EMC ControlCenter Server host using both host name and IP address to prove correct resolution.

o Be sure ICMP is not being blocked on the network when performing this test • Using Internet Explorer, connect to the ControlCenter Server URL via host name or IP address.

o http://<ecc_server_hostname>/webinstall or http://<ecc_server_IP>/webinstall o If message “Page not found” appears, determine if a firewall exists. Refer to the EMC ControlCenter

6.0 Planning and Installation Guide, Volume 1 for more information. o If a firewall exists within ControlCenter environment, request port 30000 is opened.

• Verify etc/hosts file (if not using DNS) to ensure infrastructure host names and IP addresses are included and are correctly matched.


LCW Subcategory: Installation CONSOLE_INSTALL_WINDOWS

• %ProgramFiles%\ecc\console\ecclog.txt CONSOLE_INSTALL_SOLARIS

• /opt/ecc/console/ecclog.txt Note: The ecclog.txt file will only exist if there has been an error or exception during install. OS Commands: (From EMC ControlCenter Server host)




• From the host where the console install was attempted: o Host OS type [Windows version (run winver), Solaris version (run uname -a)]


• From the host where the Installation Wizard is being run: o Record any error messages

• Indicate if the console is a remote console or on the infrastructure host. • From the host where the Installation Wizard is being run:

o From a command prompt, capture the output of the following commands redirected to a filename. Make output file names unique if multiple hosts.

Host OS type [Windows version (run winver), Solaris version (run uname -a)] nslookup <hostname> > nslookup.txt for all hosts involved for this problem netstat –an > netstat_an.txt ipconfig /all > ipconfig_all.txt winmsd /s c:\tmp\<hostname>.nfo set > set.txt regedit /e regedit_e.txt (if a Windows host)


ControlCenter 5.2 to 6.0 Differences

What services have changed in ControlCenter 6.0? What are the ControlCenter 6.0 services? What are the differences between a ControlCenter 5.2.x infrastructure and a ControlCenter 6.0 infrastructure? What do I need to consider when upgrading ControlCenter 5.2.x to ControlCenter 6.0? Comparison: 5.2.x Services and 6.0 Services ControlCenter 5.2 ControlCenter 6.0 Notes EMC ControlCenter ECCAPI Server EMC ControlCenter API Server Name Change Only EMC ControlCenter Master Agent EMC ControlCenter Master Agent No Change EMC ControlCenter Performance Manager Reports

EMC ControlCenter Web Server Web Server consolidates WebApplications Server and Performance Manager Server 5.2.x services into one service for 6.0.

EMC ControlCenter WebApplications Server

EMC ControlCenter Web Server Handles downloads of Console and Integrations Packages programs. New install link: http://<hostname>/webinstall

EMC ControlCenter WebConsole Server

EMC ControlCenter WebConsole Server

No Change

EMC ControlCenter Store EMC ControlCenter Store No Change EMC ControlCenter Server EMC ControlCenter Server No Change EMC ControlCenter StorageScope EMC StorageScope Server Supports StorageScope 6.0 scheduling and

reporting. OracleECCREP_HOMETNSListener OracleECCREP_HOMETNSListener No Change OracleServiceRAMBDB OracleServiceRAMBDB No Change OracleServiceEMCSTSDB New service to support the StorageScope

Repository. Note: Information regarding changes to the ECC Server, Repository, and Store upgrades can be found in the ControlCenter 6.0 Upgrade Guide.


Agents no Longer Supported in ControlCenter 6.0 Unsupported agents The following agents are no longer supported and will not run in ControlCenter 6.0. If these agents were installed in previous versions of ControlCenter, you can uninstall them from the host if you do not want them appearing (even though they are not functioning) in the ControlCenter Console. Note: Some of these agents were generally unavailable but still supported in ControlCenter 5.2.x. For ControlCenter 6.0 and higher, none of these agents will function at all, although some will appear in the Console until you uninstall them. Backup Agent for TSM (mainframe agent only)

• Logical Agent for MVS • Storage Agent for IBM ESS — moved to Open Systems at 5.2.0 and now called Storage Agent for ESS

(Windows 2000 only). The Storage Agent for IBM ESS cache alerts were moved under the Physical Agent for MVS at that time.

• Storage Agent for RVA/SVA • Tape Agent for MVS • Host Agent for Active directory • Host Agent for MVS HSM • Host Agent for MVS SMS • Database Agent for DB2

Note: ControlCenter 6.0 Mainframe support is provided by the Master Agent and the Physical Agent for MVS. Agent backward compatibility support ControlCenter 6.0 provides agent backwards compatibility to ControlCenter 5.2 SP3, SP4, and SP5 agents except as noted in this section (and the previous section that lists agents that are no longer supported). Backwards compatibility means that the 5.2.x agents will continue to function as designed, yet with the 6.0 infrastructure. However, you must update the 5.2.x agents to 6.0 to make use of new 6.0 functionality. Note: The ControlCenter 6.0 infrastructure is backwards compatible with SP3 agents and higher. Older versions of ControlCenter agents are not supported. The following ControlCenter 5.2 SP3, SP4, or SP5 agents are not backwards compatible with the 6.0 infrastructure:

• Storage Agent for HDS • FCC Agent

Required agent upgrades After you upgrade the ECC Server and other infrastructure components, you must:

• Upgrade all agents that are at pre-5.2 SP3 release levels. Agents at pre-5.2 SP3 release cannot run with a 6.0 infrastructure (with the exception of the Master Agent for the purpose of running the upgrade).

• Upgrade pre-6.0 versions of the Fibre Channel Connectivity Agent or Storage Agent for HDS. Only the 6.0 versions of the Fibre Channel Connectivity Agent and Storage Agent for HDS can run with the 6.0 infrastructure.

• The agent upgrade wizard automatically sends upgrades for each agent installed on an agent host when you select that host for upgrade. You need not take individual action for each agent on a host.

Note: For more information, see the EMC ControlCenter 6.0 Upgrade Guide on Powerlink


ControlCenter 6.0 Infrastructure Log File Locations Log File Location on ControlCenter 6.0 Infrastructure Hosts: The directory structure for log file collection has changed in 6.0. The infrastructure “hostname” directory is no longer variable based on the name of the infrastructure host. In 6.0, the directory will be literally named ecc_inf_hostname. For example, to collect a server.trc file from a 6.0 infrastructure where the installation is in the default location on the C drive, go to: C:\ECC\ecc_inf\data\ecc_inf_hostname\data\server.trc

• SERVER o %ECC_INSTALL_ROOT%\ecc_inf\data\ecc_inf_hostname\data\server.trc o %ECC_INSTALL_ROOT%\ecc_inf\data\ecc_inf_hostname\data\server.trc.*.gz o %ECC_INSTALL_ROOT%\ecc_inf\data\ecc_inf_hostname\data\ecc_server.ini

• STORE o %ECC_INSTALL_ROOT%\ecc_inf\data\ecc_inf_hostname\data\store.trc o %ECC_INSTALL_ROOT%\ecc_inf\data\ecc_inf_hostname\data\store.trc.*.gz o %ECC_INSTALL_ROOT%\ecc_inf\data\ecc_inf_hostname\data\ecc_store.ini

• ECCAPISERVER o %ECC_INSTALL_ROOT%\ECCAPIServer\data\apiserver.trc o %ECC_INSTALL_ROOT%\ECCAPIServer\data\apiserver.trc.*.gz o %ECC_INSTALL_ROOT%\ECCAPIServer\data\ecc_apiserver.ini

• TOMCAT o %ECC_INSTALL_ROOT%\tomcat\logs\*

New StorageScope Log File Locations:

• FLR ARCHIVER o [ECC_HOME]\SGS6*\SGS.log o [ECC_HOME]\SGS6*\SGS_B*.log.gz o [ECC_HOME]\SGS6*\SGS.ini o [ECC_HOME]\SGS6*\SGS_Err.log o [ECC_HOME]\SGS6*\SGS_B*_Err.log

• STORAGESCOPE CONSOLE

o %ECC_INSTALL_ROOT%\STS6*\SrmConsole\log\* o %ECC_INSTALL_ROOT%\STS6*\SrmConsole\reports\Oracle\*

• STORAGESCOPE SERVICE

o %ECC_INSTALL_ROOT%\Tomcat\webapps\srm\conf\* o %ECC_INSTALL_ROOT%\Tomcat\webapps\srm\logs\* o %ECC_INSTALL_ROOT%\Tomcat\webapps\srm\data\*

• STS REPOSITORY

o %ECC_INSTALL_ROOT%\Repository\admin\emcsts_scripts\*.log o %ECC_INSTALL_ROOT%\Repository\admin\emcsts_scripts\*.err o %ECC_INSTALL_ROOT%\Repository\admin\emcsts_scripts\*.INI o %ECC_INSTALL_ROOT%\Repository\admin\emcstsdb\bdump\alert_emcstsdb.log o %ECC_INSTALL_ROOT%\Repository\admin\emcstsdb\bdump\*.trc

Note: Use the ControlCenter 6.0 Log Collection Wizard to assist in collecting ControlCenter 6.0 log files. For instructions on using the Log Collection Wizard see Appendix A in this guide or Primus emc154217.


Solutions Enabler Media Repository Load Kit Installation How do I troubleshoot a ControlCenter 6.0 MRLK Install? What do I verify if I have problems installing the MRLK on ControlCenter 6.0? Using the Media Repository Load Kit (MRLK)

• Solutions Enabler 6.3 is the minimum version for ControlCenter 6.0 • Follow instructions included with the MRLK (Media Repository Load kit). This is available from Powerlink™ only.

To install the MRLK into the Media Repository:

• Download the MRLK zip file to a temporary directory on the ControlCenter Server host. • Unzip the MRLK zip file. • Execute the exe file (Wait for the install wizard to start) • Click Next at the Welcome screen, the install will begin. • Click Finish when the install is complete.

To push out the new Solutions Enabler to hosts, follow the instructions below:

• Ensure all applications which use the SYMAPI libraries are stopped. This includes SYMCLI scripts, symapisrv and stor deamons, Pegasus CIM Server, the CMA (Common Mapping Agent), EGS, Oracle, and host agents.

• Using the ControlCenter Console Agent Administration Wizard, stop all agents that are running on the agent hosts where Solutions Enabler is to be deployed.

• Use the Console Agent Administration Wizard, install Solutions Enabler on the desired agent hosts. • Start the agents.


LCW Subcategory: Installation MRLK_INSTALL

• %ECC_INSTALL_ROOT%\installSEK* • %ECC_INSTALL_ROOT%\media_rep\remoteinstall\SEK6*\create_ecc_kits.log

OS Commands (from EMC ControlCenter Server host): None specified. Repository Exports: None specified. Additional Files:

• From the ControlCenter infrastructure, collect a screen capture of the error message.


Solutions Enabler Push Installation How do I troubleshoot a ControlCenter 6.0 Solutions Enabler push Install? What do I verify if I have problems installing Solutions Enabler from ControlCenter 6.0? Troubleshooting Steps Ensure all applications which use the SYMAPI libraries are stopped. This includes SYMCLI scripts, symapisrv and stor deamons, Pegasus CIM Server, the CMA (Common Mapping Agent), EGS (Symmetrix), and host agents. Starting with 5.2, you have the ability to choose a non-default Solutions Enabler install directory. Information to Collect Note: All logs must be time relevant to the problem event. In addition, Server and Store log file sets must contain all available trace files. Using the Log Collection Wizard please select the following category and subcategories which will collect the recommended log files indicated below. Deselect any log types or log files that are not applicable to your environment. For instructions on using the Log Collection Wizard see emc154217. LCW Category: Installations and Upgrades

LCW Subcategory: Solutions Enabler Remote Installation SE_INSTALL

• [ECC_HOME]\E*EslntInst.log • [ECC_HOME]\E*E6*.rcfile • [ECC_HOME]\ctg.ini

SYMAPI_UNIX • /var/symapi/log/*

SYMAPI_WINDOWS • %ProgramFiles%\EMC\SYMAPI\log\symapi*.log • %ProgramFiles%\EMC\SYMCLI\*.log


SERVER • %ECC_INSTALL_ROOT%\ecc_inf\data\ecc_inf_hostname\data\server.trc • %ECC_INSTALL_ROOT%\ecc_inf\data\ecc_inf_hostname\data\server.trc.*.gz • %ECC_INSTALL_ROOT%\ecc_inf\data\ecc_inf_hostname\data\ecc_server.ini

STORE • %ECC_INSTALL_ROOT%\ecc_inf\data\ecc_inf_hostname\data\store.trc • %ECC_INSTALL_ROOT%\ecc_inf\data\ecc_inf_hostname\data\store.trc.*.gz • %ECC_INSTALL_ROOT%\ecc_inf\data\ecc_inf_hostname\data\ecc_store.ini

ECCAPISERVER • %ECC_INSTALL_ROOT%\ECCAPIServer\data\apiserver.trc • %ECC_INSTALL_ROOT%\ECCAPIServer\data\apiserver.trc.*.gz • %ECC_INSTALL_ROOT%\ECCAPIServer\data\ecc_apiserver.ini

REPOSITORY • %ECC_INSTALL_ROOT%\Repository\NETWORK\log\listener.log • %ECC_INSTALL_ROOT%\Repository\NETWORK\ADMIN\tnsnames.ora • %ECC_INSTALL_ROOT%\Repository\NETWORK\ADMIN\listener.ora • %ECC_INSTALL_ROOT%\Repository\NETWORK\ADMIN\sqlnet.ora • %ECC_INSTALL_ROOT%\Repository\admin\Ramb_scripts\upgrade\*.log


• %ECC_INSTALL_ROOT%\Repository\admin\Ramb_scripts\ecc_getversion.log • %ECC_INSTALL_ROOT%\Repository\admin\rambdb\bdump\alert_rambdb.log • %ECC_INSTALL_ROOT%\Repository\admin\rambdb\bdump\*.trc • %ECC_INSTALL_ROOT%\Repository\admin\rambdb\pfile\* • %ECC_INSTALL_ROOT%\Repository\assistants\dbca\logs\*


CONSOLE_WINDOWS • %SystemDrive%\Documents and Settings\*\ecc\data\*\console.trc • %SystemDrive%\Documents and Settings\*\ecc\data\*\console.trc.*.gz • %SystemDrive%\Documents and Settings\*\ecc\data\*\consoleLaunch.log • %SystemDrive%\Documents and Settings\*\ecc\data\*\ecc_console.ini

CONSOLE_SOLARIS • $HOME/ecc/data/*/console.trc • $HOME/ecc/data/*/console.trc.*.gz • $HOME/ecc/data/*/consoleLaunch.log • $HOME/ecc/data/*/ecc_console.ini

Keep in mind in order for the Console.trc file to be created, the user must enable Console tracing when launching the Console session. Also keep in mind that the console.trc file will “roll-over” and zip itself as soon as the EMC ControlCenter console is closed. The new zip file will be: console.trc.1.gz OS Commands: (From EMC ControlCenter Server host) None specified. Repository Exports: None specified. Additional Files:








Agent Remote Install-Uninstall How do I troubleshoot installing or uninstalling ControlCenter 6.0 Agents? What to verify and collect for ControlCenter 6.0 Agent install or uninstall issues? Troubleshooting Steps This uses the “Installation Wizard,” each selected agent is checked for disk space and prerequisites.

• If an agent does not appear in the Available Agents List, it could be one of the following issues: o The agent is already installed

View the ctg.ini file to see which agents are installed [ECC_HOME]\ctg.ini o The agent support is not licensed in the console under ECC Administration > Install > Edit Licenses menu.

The ControlCenter Planning and Installation Guide outlines the licenses needed for various agents.

o The agent is not available for this host OS type. • Verify the required agent disk space is available in the installation file system. A check is run at install, but may

not cover all possibilities. The ControlCenter Performance and Scalability Guidelines provide resource requirement information for agents.

• Verify the Master Agent is installed, and started on the distributed host. o UNIX: To check if Master Agent is started run: ps -ef | grep mstr

To start Master Agent run: <Install Root>/exec/start_master.csh o Windows: To check if Master Agent process appears under the 'Processes' tab: taskmgr

Start the 'EMC ControlCenter Master Agent' from Services • If the installation was successful, but the Console doesn’t show the agent, it is likely that the agent did not start

up correctly. In that case, it is necessary to collect the specific agent’s log files in addition to the Master Agent files listed below.

• If troubleshooting a specific application’s agent such as the VERITAS NetBackup Agent or LEGATO NetWorker then:

o Verify the absolute pathname of the application was entered correctly during the application agent install. If the application path was entered incorrectly the agent will not be displayed on Console as an installed agent on that host, however the agent acronym directory would have been created in %ECC_Install_ROOT%\exec.

o The absolute pathname can be verified in the agents acronym xxx.ini file. Information to Collect Remote agent pushes require logs on up to 3 hosts. (The Console host, ControlCenter Server host, and the remote [target] host.) Please be sure to include the Sub-Agent logs that match the Agent type you are attempting to install. Note: All logs must be time relevant to the problem event. In addition, Server and Store log file sets must contain all available trace files. Using the Log Collection Wizard please select the following category and subcategories which will collect the recommended log files indicated below. Deselect any log types or log files that are not applicable to your environment. For instructions on using the Log Collection Wizard see emc154217. LCW Category: Installations and Upgrades

LCW Subcategory: Agent Remote Install-Uninstall CONSOLE_WINDOWS

• %SystemDrive%\Documents and Settings\*\ecc\data\*\console.trc • %SystemDrive%\Documents and Settings\*\ecc\data\*\console.trc.*.gz • %SystemDrive%\Documents and Settings\*\ecc\data\*\consoleLaunch.log • %SystemDrive%\Documents and Settings\*\ecc\data\*\ecc_console.ini






Keep in mind in order for the Console.trc file to be created, the user must enable Console tracing when launching the Console session. Also keep in mind that the console.trc file will “roll-over” and zip itself as soon as the EMC ControlCenter console is closed. The new zip file will be: console.trc.1.gz Note: A specific environment variable for ECC’s install root does not exist on UNIX hosts. You can find out where the Master Agent was installed by looking at the contents of the /var/emc/.eccrc file. OS Commands: (From EMC ControlCenter Server host)



• From the host to which the agent push was attempted: o Windows: In a command prompt window, capture the output of the following commands redirected

to a filename. Make output file names unique if multiple hosts. dir /s %ECC_Install_ROOT%/exec > eccdir.txt (on Windows) ls –lR <ecc_install_root>/exec > eccdir.txt (on UNIX) /usr/bin/ps –ef > ps.txt (on UNIX) nslookup <hostname> > nslookup.txt netstat –an > netstat_an.txt winmsd /s c:\tmp\<hostname>.nfo (on Windows) set > set.txt (on Widnows) env > env.out (on Unix) regedit /e regedit_e.txt (on Windows)

• From the ControlCenter Server host: o nslookup <hostname> > nslookup.txt

• From the EMC ControlCenter console: • Collect an export of the Alerts table view:

1. Open the ControlCenter Console 2. Select ECC Administration on the blue toolbar 3. Click the dropdown arrow next to ECC Administration, click Alerts 4. In the Tree Panel on the left side of the Console, put a check next to the Hosts folder. Make

sure there are no other items checked.


5. In the Target Panel on the right you will see the Alerts view. On the top of that view, click the Table button.

6. Click inside the table to make it the active pane 7. Go to the top left hand corner of the console screen and select File > Export > Browse and save

export to a location such as your desktop or C: drive as a .csv file. • Collect an export of the Policies table view:

1. Open the ControlCenter Console 2. Select ECC Administration on the blue toolbar 3. Click the dropdown arrow next to ECC Administration, click Policies 4. In the Tree Panel on the left side of the Console, put a check next to the affected host in the

Hosts folder. Make sure there are no other items checked. 5. In the Target Panel on the right you will see the Policies view. On the top of that view, click the

Table button. 6. Click inside the table to make it the active pane 7. Go to the top left hand corner of the console screen and select File > Export > Browse and save

export to a location such as your desktop or C: drive as a .csv file. • Collect an export of the Agents table view:

1. Open the ControlCenter Console 2. Select ECC Administration on the blue toolbar 3. Click the dropdown arrow next to ECC Administration, click Agents 4. In the Tree Panel on the left side of the Console, put a check next to the folder name Hosts.

Make sure there are no other items checked. 5. In the Target Panel on the right you will see the Agents – Relationships view. On the top of that

view, click the Table button. 6. Click inside the table to make it the active pane 7. Go to the top left hand corner of the console screen and select File > Export > Browse and save

export to a location such as your desktop or C: drive as a .csv file.


Upgrade 5.2 Infrastructure to 6.0 How do I troubleshoot upgrading ControlCenter Infrastructure components to 6.0? What to verify and collect for ControlCenter Infrastructure upgrade issues? Troubleshooting Steps Verify you have completed the Generic Install Procedure steps:

• To upgrade a 5.2 infrastructure to 6.0, follow the instructions outlined in the EMC ControlCenter 6.0 Upgrade Guide available on Powerlink.







Install Wizard. • Close Microsoft “Services” window prior to installation. • Ensure EMC ControlCenter Server (ECCService_Serv on W2K, ECCService_Server.exe on W2003) and EMC

ControlCenter Store (ECCService_Stor on W2k, ECCService_Store.exe on W2003) and any distributed Store services have been stopped. Provide enough time to ensure the processes have completed their shut down routines.

• Verify Repository host has been rebooted after initial install, prior to upgrade attempt. (This ensures environment variables have been set properly.)

• Correct the failure condition outlined in upgrade logs (eccservservicesetup.log and eccstoreservicesetup.log) and resume upgrade.

Differences Between 5.2 and 6.0 Reference Primus emc154379 “ControlCenter 5.2 to 6.0 Differences”, the EMC ControlCenter 6.0 Overview Guide, and the EMC ControlCenter 6.0 Release Notes. Information to Collect Note: All logs must be time relevant to the problem event. In addition, Server and Store log file sets must contain all available trace files. Using the Log Collection Wizard please select the following category and subcategories which will collect the recommended log files indicated below. Deselect any log types or log files that are not applicable to your environment. For instructions on using the Log Collection Wizard see emc154217. LCW Category: Installations and Upgrades

LCW Subcategory: Upgrade 5.2 Infrastructure PRINCIPAL_INSTALL_LOG -FOR ALL INSTALL ISSUES







UPGRADE_LOGS • %ECC_INSTALL_ROOT%\eccservservicesetup.log • %ECC_INSTALL_ROOT%\eccstoreservicesetup.log








• From the host where the Repository is installed: o Windows: In a command prompt window, capture the output of the following commands redirected

to a filename. %ECC_Install_ROOT%\Repository\bin\LSNRCTL.exe status > listener.txt




Upgrade Service Pack How do I troubleshoot upgrading ControlCenter 6.0 Service Pack? What to verify and collect for ControlCenter Service Pack install issues? Troubleshooting Steps Verify you have completed the Generic Install Procedure steps:







Install Wizard. • Close Microsoft “Services” window prior to installation.


LCW Subcategory: Service Pack Installation PRINCIPAL_INSTALL_LOG -FOR ALL INSTALL ISSUES





REPOSITORY




Installation instructions for Service Packs, Patches (hot fixes) will vary. Installation instructions are included with the distribution when it is made available. Ensure the installation instructions were followed. If a Service Alert for a hot fix was created, this too will outline the installation procedure. Service Pack Log files:

• Service Packs logs are in %ECC_INSTALL_ROOT% in the form of “ecc6xxSPx_install.txt” Patches • In some cases where the patch is an executable, (but not all cases) a resulting Hotfix#.log file will be

generated in the directory where the executable was run. Generally in case of error or exceptions, the InstallShield log will be in %ECC_INSTALL_ROOT% in the form of eccPatch6xxxx_install.txt

OS Commands: (From EMC ControlCenter Server host) • netstat -an • ipconfig /all • winmsd nfo file • set • regedit /e


• From the host where the Installation Wizard is being run: o Capture a screen shot of the EMC ControlCenter install GUI at the time of the failure o %ECC_INSTALL_ROOT%\eccservservicesetup.log o %ECC_INSTALL_ROOT%\eccstoreservicesetup.log





• From the host where the EMC ControlCenter Repository is installed: o %ECC_Install_ROOT%\Repository\bin\LSNRCTL.exe status > listener.txt


Section II Discovering Objects

CLARiiON Array Discovery How do I troubleshoot discovering a CLARiiON in ControlCenter 6.0? What do I need to verify and collect for ControlCenter 6.0 CLARiiON discovery issues? Troubleshooting Steps • Verify the FC/CX/AX series model is supported.

o AX Series arrays require an upgrade from Navisphere Express to Navisphere Manager for CLARiiON Agent discovery.

• Verify prerequisites to run CLARiiON agent are met. Reference the ControlCenter Support Matrix. o JDK 1.4.x (1.4.2 recommended) use command prompt window “java –version.” (This is checked by the

install script.) MetaLUN support requires JRE 1.4.2_06

o Verify Solutions Enabler 6.3 or later in installed on the host where the Storage Agent for CLARiiON is running.

o Verify Navisphere® CLI software is installed on the host where Storage Agent for CLARiiON is running Navisphere CLI 6.7, (6.16, 6.19, 6.22). Use command prompt window “navicli –help.” (This is checked by the install script.)

On Windows hosts use Control Panel —Add remove software look for Navisphere CLI. On Solaris UNIX Hosts do pkginfo | grep Navisphere. On HP UX do swlist | grep Navisphere. On IBM AIX run lslpp -l | grep Navisphere CLI.

o Minimum version of Navisphere CLI required to support the CX3 series arrays is 6.22. o If the Navisphere Agent was not installed on the host along with Navisphere CLI, the host must be

manually registered with the CLARiiON o Verify Navisphere SP Agent is installed and running on the FC4700/CX storage processors (SPs), or the

FC4500/FC5300 SP hosts. o Issue the following Navicli command to verify agent is installed, version and running "Navicli -h SP name

or IP address getagent." o Verify Access Logix™ software is installed and enabled on the FC4700/CX SPs or the FC4500/FC5300

hosts. o Verify for UNIX hosts, the path of the Navisphere CLI software is added to the PATH.

"echo $PATH" look for /opt/Navisphere/bin" in path o Verify for SnapView™ support on FC 4700/CX arrays, SnapView must be installed on the array

Use the following command “navicli -h SP name or IP address -ndu list” (look for SnapView module this will also list the version).

• Verify that the storage processors are accessible via the network, either via the SP host name or IP address. o From the agent host, Ping the SP name or IP address. o Verify SP’s Navicli username and password; Default: emc/emc.

This is often changed by the end user. Obtain the correct username and password. (User must have administrator role and GLOBAL scope). If not correct or left blank, results in partial discovery.

• Verify there is at least one active Store. o ECCService_Stor on W2k o ECCService_Store.exe on W2003 o MS Services and Task manager should not be used to make this determination. They may be unreliable.

Check to see if Store is active on Console or by verifying if store.trc file is active and being updated. • Verify DCP discovery_00 for Storage Agent for CLARiiON is enabled. • Check discovery progress by going to Console->Discovery->Review Progress • Document approximate time of discovery.


Information to Collect Note: All logs must be time relevant to the problem event. In addition, Server and Store log file sets must contain all available trace files. Using the Log Collection Wizard please select the following category and subcategories which will collect the recommended log files indicated below. Deselect any log types or log files that are not applicable to your environment. For instructions on using the Log Collection Wizard see emc154217. LCW Category: Object Discovery and Monitoring

LCW Subcategory: CLARiiON –Array MGL -CLARIION

• [ECC_HOME]\MGL6*\MGL.log • [ECC_HOME]\MGL6*\MGL_B*.log.gz • [ECC_HOME]\MGL6*\MGL.ini • [ECC_HOME]\MGL6*\MGL_Err.log • [ECC_HOME]\MGL6*\MGL_B*_Err.log • [ECC_HOME]\MGL6*\MGL_Discovery.log • [ECC_HOME]\MGL6*\MGL_Discovery.log.gz




EMC_SMI_PROVIDER • %ProgramFiles%\EMC\SYMCLI\logs\*




TOMCAT • %ECC_INSTALL_ROOT%\tomcat\logs\*

REPOSITORY • %ECC_INSTALL_ROOT%\Repository\NETWORK\log\listener.log • %ECC_INSTALL_ROOT%\Repository\NETWORK\ADMIN\tnsnames.ora • %ECC_INSTALL_ROOT%\Repository\NETWORK\ADMIN\listener.ora • %ECC_INSTALL_ROOT%\Repository\NETWORK\ADMIN\sqlnet.ora • %ECC_INSTALL_ROOT%\Repository\admin\Ramb_scripts\upgrade\*.log • %ECC_INSTALL_ROOT%\Repository\admin\Ramb_scripts\ecc_getversion.log • %ECC_INSTALL_ROOT%\Repository\admin\rambdb\bdump\alert_rambdb.log • %ECC_INSTALL_ROOT%\Repository\admin\rambdb\bdump\*.trc • %ECC_INSTALL_ROOT%\Repository\admin\rambdb\pfile\*


• %ECC_INSTALL_ROOT%\Repository\assistants\dbca\logs\* ORACLE

• %ProgramFiles%\Oracle\Inventory\logs\* • %ProgramFiles%\Oracle\Inventory\logs\results\db\*



Keep in mind in order for the Console.trc file to be created, the user must enable Console tracing when logging on prior to the discovery. Also keep in mind that the console.trc file will “roll-over” and zip itself as soon as the EMC ControlCenter console is closed. The new zip file will be: console.trc.1.gz OS Commands: (From EMC ControlCenter Server host)

• None specified. Repository Exports:

• ControlCenter Repository o Use existing Repository dump file

Additional Files: Collect from Console Workstation where discovery was initiated:

• Export Agents Policy View 1. Open the ControlCenter Console 2. Select ECC Administration on the blue toolbar 3. Click the dropdown arrow next to ECC Administration, click Agents 4. In the Tree Panel on the left side of the Console, put a check next to the folder name Hosts. Make

sure there are no other items checked. 5. In the Target Panel on the right you will see the Agents – Relationships view. On the top of that



1. Open the ControlCenter Console 2. Select ECC Administration on the blue toolbar 3. Click the dropdown arrow next to ECC Administration, click Policies 4. In the Tree Panel on the left side of the Console, put a check next to the affected host in the Hosts

folder. Make sure there are no other items checked. 5. In the Target Panel on the right you will see the Policies view. On the top of that view, click the Table

button. 6. Click inside the table to make it the active pane 7. Go to the top left hand corner of the console screen and select File-->Export-->Browse and save


1. Open the ControlCenter Console 2. Select ECC Administration on the blue toolbar 3. Click the dropdown arrow next to ECC Administration, click Agents 4. In the Tree Panel on the left side of the Console, put a check next to the folder name Hosts. Make


view, click the Table button.


6. Click inside the table to make it the active pane 7. Go to the top left hand corner of the console screen and select File-->Export-->Browse and save



Symmetrix Array Local Discovery How do I troubleshoot discovering a local Symmetrix in ControlCenter 6.0? What do I need to verify and collect for ControlCenter 6.0 Symmetrix local discovery issues? Troubleshooting Steps

• Verify that there is an active Symmetrix agent. • Verify there is at least one active Store.

o ECCService_Stor on W2k o ECCService_Store.exe on W2003 o MS Services and Task manager should not be used to make this determination. They may be

unreliable. Check to see if Store is active on the Console or verify if store.trc file is active and being updated.

• Verify through agents view which host is running the primary Symmetrix agent for that Symmetrix serial number.

• Verify through ‘syminq’ whether this host has access to the Symmetrix in question. • Verify through ‘symcfg list’ that this host can see Symmetrix via symapi.

o You may need to run ‘symcfg discover’ prior to running the ‘symcfg list’ command. • Verify that the Local Discovery_00, and Configuration_00 policies are enabled and that the host running the

primary Symmetrix agent is listed in the ApplyTo tab of the Local Discovery policy. • Verify that the proper version of Solutions Enabler is installed on primary Symmetrix agent host. (Check

Support Matrix) • Verify Zoning

o Check Switch to see if HBA is logged in o Check Switch to see if Symmetrix port is logged in. o Verify HBA WWN is listed under correct FA

• Verify Masking o Run ‘symmaskdb list –sid xxx’ o Check that device given access is mapped to FA o Verify Symmetrix for proper Volume Logix setup

Is the VCM bit turned on for the FA? Is there a VCM device defined as a VCMDB? Is the VCMDB assigned to the FA as LUN 0?

• Check if (ACL’s) Access Controls are enabled? o If yes, run: symcfg –sid xxx list –v o Verify if the host is in the list

• Determine what portion of the Symmetrix (if any) has been discovered. • Document approximate time of discovery. • Determine if there is a pink diamond next to the Symmetrix which indicates that it is not fully discovered. If

Symmetrix is in console, but not fully discovered, stop Symmetrix agents, delete Symmetrix, and then restart one Symmetrix agent. Allow Symmetrix agent to run for several minutes.


LCW Subcategory: Symmetrix –Array EGS -SYMMETRIX

• [ECC_HOME]\EGS6*\EGS.log


• [ECC_HOME]\EGS6*\EGS_B*.log.gz • [ECC_HOME]\EGS6*\EGS.ini • [ECC_HOME]\EGS6*\EGS_Err.log • [ECC_HOME]\EGS6*\EGS_B*_Err.log • [ECC_HOME]\EGS6*\EGS_Discovery.log • [ECC_HOME]\EGS6*\EGS_Discovery.log.gz












CONSOLE_SOLARIS • $HOME/ecc/data/*/console.trc • $HOME/ecc/data/*/console.trc.*.gz


• $HOME/ecc/data/*/consoleLaunch.log • $HOME/ecc/data/*/ecc_console.ini

Keep in mind in order for the Console.trc file to be created, the user must enable Console tracing when launching the console session. Also keep in mind that the console.trc file will “roll-over” and zip itself as soon as the EMC ControlCenter console is closed. The new zip file will be: console.trc.1.gz OS Commands: (From EMC ControlCenter Server host) None specified. Repository Exports: None specified. Additional Files: Collect information using ControlCenter Console:

• Collect an export of the Alerts table view: 1. Open the ControlCenter Console 2. Select ECC Administration on the blue toolbar 3. Click the dropdown arrow next to ECC Administration, click Alerts 4. In the Tree Panel on the left side of the Console, put a check next to the Hosts folder. Make

sure there are no other items checked. 5. In the Target Panel on the right you will see the Alerts view. On the top of that view, click the












Symmetrix Array Proxy Discovery How do I troubleshoot discovering a proxy Symmetrix in ControlCenter 6.0? What do I need to verify and collect for ControlCenter 6.0 Symmetrix proxy discovery issues? Background Proxy discovery is the process of discovering information about Symmetrix arrays that are attached to hosts not running a Symmetrix agent (Storage Agent for Symmetrix). In addition, this policy retrieves information about device groups on the host. You can configure a Symmetrix Agent operating on one host to act – by proxy – as the Symmetrix Agent for another host. The agent will then perform all the usual tasks for the Symmetrix arrays attached to that second host. You can delete or disable this policy, or assign the policy to the Symmetrix Agent on another host. Notes

• Solutions Enabler SYMAPI server must be installed on the second host and there must be a connection between the second host and the Symmetrix.

• Performance Manager requires this data collection policy to collect information about Symmetrix arrays connected to hosts that are not running the Storage Agent for Symmetrix. This applies to all mainframe hosts and those Open Systems hosts that (for whatever reason) do not have the Storage Agent for Symmetrix running.

• Symmetrix Manager requires this data collection policy to manage device groups on hosts where a Storage Agent for Symmetrix is not running, as well as managing Symmetrix arrays that are attached to the second host.

Terminology The following terms are used in this discussion:

• Local Host: is the host on which the Storage Agent for Symmetrix is running • Console Host: is the host on which the Console is running • Proxied Host: is the host that has the direct connection to the Symmetrix with which you want to communicate • Proxied Symmetrix: is the object of the proxy effort; a Symmetrix that can be found in your network, but is not

directly attached to the local host running the Symmetrix agent Purpose This policy defines the frequency with which Symmetrix discovery is executed for the second host. This policy is not enabled by default. Determining host names Because ControlCenter uses host names for unique database keys, it is important to select the correct host name under the Source tab. Otherwise, you may create two host records in the database and you will not be able to access some information, nor will you be able to manage device groups on the host running the SYMAPI server) If an agent is running on the proxied host: The host name used by a ControlCenter host and the agent running on the host is often the fully-qualified host name, but could have some other value.

• Open Systems: To determine the correct host name to specify for the proxy policy for open systems hosts, open the Properties view for the proxied host, and you will see the Host and Host Name columns. The Host column is a shortened display name. The Host Name column is a longer name, and is the name you should use.

• Mainframe: The case-sensitive host name must be identical to the actual full host name of the MVS system (hostname.domain). To verify the host name and domain, an MVS systems administrator can view the output from HOMETEST (HOMETEST is available with IBM TCP/IP only). Ensure that the host name in the policy matches case with the host name as it appears in the Console tree. Ensure that an approved version of Solutions Enabler is installed on the MVS host. See the Release Notes or Support Matrix for approved versions.

If no agent is installed on the proxied host: • If there are plans to install a ControlCenter agent on the host, the agent should be installed, and you should

then follow the previous procedure.


• If there is no ControlCenter agent running on the proxied host, you should enter the fully-qualified name for the host. If the names used by the Proxy Discovery DCP and later agent installations are different, the result will be multiple host records in the ControlCenter database, with incomplete data associated with each record.

Configuration requirements and procedure Requirements In order to proxy for another host, the remote host must be running the Solutions Enabler SYMAPI Server. You will need to supply:

• The correct name of the remote host • The IP address of the remote host • The port being used by SYMAPI server on the remote host (The default is 2707.) • The name of the host on which the Storage Agent for Symmetrix (that will be used for proxying) is running

Procedure After you have installed the ControlCenter infrastructure, you should follow these steps to set up a Storage Agent for Symmetrix in proxy mode.

1. Determine the Symmetrix you want to monitor, the host to which it is attached, and the host on which you want to run the Storage Agent for Symmetrix in proxy mode.

2. If not already installed, install Solutions Enabler on each proxy host connected to the proxied Symmetrix array. See the EMC ControlCenter Release Notes or Support Matrix for acceptable versions of Solutions Enabler.

3. If not already installed, install (from the Console) the Storage Agent for Symmetrix on the local host. 4. Mainframe: It is critical that the agents are started in a certain order. If the proxy Symmetrix Agent is started

first and populates the database before the MVS based agents (CSMAGENTS), the information from the MVS based agents will not populate the database and all data gathered regarding the MVS attached Symmetrix will be erroneous.

5. Mainframe: The mainframe agents should be installed and started on the mainframe. The Storage Agent for Symmetrix for the mainframe environment actually is a proxy agent, which in turn runs on a Windows or UNIX host. Reference the ControlCenter Support Matrix for supported operating systems.

6. Install the proxy agent by pushing (installing) the Symmetrix Agent from the Console on a UNIX or Windows host. The host does not need to be connected to a Symmetrix.

7. From the tree panel under Administration, Data Collection Policies, Policy Templates, Storage Agent for Symmetrix locate Proxy Discovery. Right click on this and select New. You need to configure this data collection policy so that the Storage Agent for Symmetrix can communicate with the proxied host and the Symmetrix arrays connected to it.

8. In the Policy Description dialog box, there are four tabs. In the Source tab, enter the case-sensitive host name, IP address and port number under which Solutions Enabler is running on the proxied host. Both the case-sensitive host name and host IP address must be specified. The case-sensitive host name has to match exactly with what would be generated by an agent running locally on the host. You can normally use the hostname command or the host names column of the Properties view to derive this value. You should match the case exactly. Mainframe: The case-sensitive host name must be identical to the actual full host name of the MVS system (hostname.domain). To verify the host name and domain, an MVS systems administrator can view the output from HOMETEST (HOMETEST is available with IBM TCP/IP only). Ensure that the host name in the policy matches case with the host name as it appears in the Console tree. Ensure that an approved version of Solutions Enabler is installed on the MVS host. See the release notes for approved versions.

9. In the Apply To tab, uncheck Apply this policy to all applicable hosts (it will show all hosts that have a Storage Agent for Symmetrix installed).

10. Select the host with the Storage Agent for Symmetrix you are using as the proxy and click Move to move it to the right side.

11. Verify that Policy Enabled is checked and click OK. This will start the proxy agent. The newly created policy will appear under the Storage Agent for Symmetrix folder located in the Policy Definitions folder (not the Policy Templates folder).

Monitored resources: This policy collects discovery information from Symmetrix arrays attached to hosts that are not running a Storage Agent for Symmetrix.



LCW Subcategory: Symmetrix –Array EGS -SYMMETRIX

• [ECC_HOME]\EGS6*\EGS.log • [ECC_HOME]\EGS6*\EGS_B*.log.gz • [ECC_HOME]\EGS6*\EGS.ini • [ECC_HOME]\EGS6*\EGS_Err.log • [ECC_HOME]\EGS6*\EGS_B*_Err.log • [ECC_HOME]\EGS6*\EGS_Discovery.log • [ECC_HOME]\EGS6*\EGS_Discovery.log.gz































IBM ESS Array Discovery How do I troubleshoot discovering IBM ESS arrays in ControlCenter 6.0? What do I need to verify and collect for ControlCenter 6.0 IBM ESS array discovery issues? Troubleshooting Steps

• Verify there is at least one active Store. o ECCService_Stor on W2k o ECCService_Store.exe on W2003 o MS Services and Task manager should not be used to make this determination. They may be

unreliable. Check to see if Store is active on Console or by verifying if store.trc file is active and being updated. Verify the ESS agent is installed and active (MGC)

• Verify IBM TotalStorage Enterprise Storage Server CLI (ESS CLI) is installed on the agent host. From a command prompt; Enter esscli. Returns: “date/time IBM ESSCLI version” and a long “help” list

• Verify the User and Password on the IBM ESS server is valid. User has to have at least View-level access configured on the ESS Clusters. See the ESS administrator. Review the Discovery.log to see what the agent received (search for esscli)

• Verify Ping from the ESS Agent Host to the ESS Storage Specialist server that runs in the ESS Clusters Information to Collect Note: All logs must be time relevant to the problem event. In addition, Server and Store log file sets must contain all available trace files. Using the Log Collection Wizard please select the following category and subcategories which will collect the recommended log files indicated below. Deselect any log types or log files that are not applicable to your environment. For instructions on using the Log Collection Wizard see emc154217. LCW Category: Object Discovery and Monitoring

LCW Subcategory: IBM ESS –Array MGC -IBM ESS

• [ECC_HOME]\MGC6*\MGC.log • [ECC_HOME]\MGC6*\MGC_B*.log.gz • [ECC_HOME]\MGC6*\MGC.ini • [ECC_HOME]\MGC6*\MGC_Err.log • [ECC_HOME]\MGC6*\MGC_B*_Err.log • [ECC_HOME]\MGC6*\MGC_Discovery.log • [ECC_HOME]\MGC6*\MGC_Discovery.log.gz




ECCAPISERVER


• %ECC_INSTALL_ROOT%\ECCAPIServer\data\apiserver.trc • %ECC_INSTALL_ROOT%\ECCAPIServer\data\apiserver.trc.*.gz • %ECC_INSTALL_ROOT%\ECCAPIServer\data\ecc_apiserver.ini






Keep in mind in order for the Console.trc file to be created, the user must enable Console tracing when launching the console session. Also keep in mind that the console.trc file will “roll-over” and zip itself as soon as the EMC ControlCenter console is closed. The new zip file will be: console.trc.1.gz OS Commands: (From EMC ControlCenter Server host) None specified. Repository Exports: None specified. Additional Files: Collect information using the ControlCenter Console:





1. Open the ControlCenter Console 2. Select ECC Administration on the blue toolbar 3. Click the dropdown arrow next to ECC Administration, click Policies


4. In the Tree Panel on the left side of the Console, put a check next to the affected host in the Hosts folder. Make sure there are no other items checked.

5. In the Target Panel on the right you will see the Policies view. On the top of that view, click the Table button.








HDS Array Discovery How do I troubleshoot discovering HDS and HP XP arrays in ControlCenter 6.0? What do I need to verify and collect for ControlCenter 6.0 HDS / HP XP array discovery issues? Troubleshooting Steps


unreliable. Check to see if Store is active on Console or by verifying if store.trc file is active and being updated.

• Verify DCP discovery_00 for Storage Agent for HDS is enabled. • Use ping command to verify the Agent host has network access to either the CommandView Server or the

HiCommand server (depending on array type to be discovered). • Verify all ADA menu items data is correct. Check for alerts that indicate trouble connecting with CommandView

or HiCommand servers. Official Names

• Hitachi HiCommand Device Manager Server (API) • HP StorageWorks Command View XP Command Line Interface (CLI) Device Manager Server

HDS

9200 9500V Series (9570V, 958xV) 9900 Series (9910, 9960) 9900V Series (9970V, 9980V) TagmaStore

HiCommand

Sun 9900 Series (9910, 9960, 9970, 9980, 9990) HiCommand

HP StorageWorks XP48, XP128, XP512, XP1024, 12000 CommandView

Need licenses for HP-XP 512 Active Command support:

• LUN Configuration Manager XP • LUN Security Manager XP

Hitachi Array Support

• Verify for HDS, that the HiCommand server (contains the API) is installed and functioning correctly. The CLI may be installed but is not required by the agent. Agent must be able to access the HiCommand server. The HiCommand API is part of the HiCommand server.

1. On the agent host, the easiest way for a user to verify if HiCommand server is running is to use a browser and connect to it: http://<HiCommandServerAddress>:<PortNumber-2001Default>. If the login prompt appears, it means the server is up and running and hence the API is available.

2. If Web server connection fails, ping the HiCommand API server from the agent host. 3. Verify for WLA data collection that Hitachi Tuning manager is installed. This is done by using a

browser to connect to the following URLs: Http://<hostname>:<port number>/TuningManager/login.do Http://<hostname>:<port number>/TuningService/TrueNorth

HP XP Arrays 1. Verify for HP XP, CommandView server, and CLI are installed and functioning correctly. 2. Verify MGD (mgdagent) Agent is installed on the same host as CommandView CLI is installed and configured to

access the CommandView server. Use command cvcli -p <user><password>. 3. Verify mgdagent.exe process is running. 4. Verify that the file CVCLI.properties is set up to point to the right CommandView server. 5. Ensure that the path to the cvcli.bat file was specified correctly (check for alerts relating to inability to connect

to CommandView server).


6. Verify that JRE 1.4.2 or higher is installed and PATH is pointing to the right version of JRE. Use cmd; java –version.

7. Specify CommandView “administrator” account if you ever intend to perform active configuration changes, otherwise “user” account is sufficient. CommandView server limits configuration changes to one active user at a time.

8. Change EMC Master Agent service from starting as system to starting as administrator. This must be done because HDS agent is started by Master Agent and Master Agent must be started as administrator. CommandView CLI requires agent to run as administrator for access.

9. Document approximate time of discovery.


LCW Subcategory: HDS –Array MGD -HDS

• [ECC_HOME]\MGD6*\MGD.log • [ECC_HOME]\MGD6*\MGD_B*.log.gz • [ECC_HOME]\MGD6*\MGD.ini • [ECC_HOME]\MGD6*\MGD_Err.log • [ECC_HOME]\MGD6*\MGD_B*_Err.log • [ECC_HOME]\MGD6*\MGD_Discovery.log • [ECC_HOME]\MGD6*\MGD_Discovery.log.gz


SERVER • %ECC_INSTALL_ROOT%\ecc_inf\data\ecc_inf_hostname\data\server.trc • %ECC_INSTALL_ROOT%\ecc_inf\data\ecc_inf_hostname\data\ecc_server.ini • %ECC_INSTALL_ROOT%\ecc_inf\data\ecc_inf_hostname\data\server.trc.*.gz





REPOSITORY • %ECC_INSTALL_ROOT%\Repository\NETWORK\log\listener.log • %ECC_INSTALL_ROOT%\Repository\NETWORK\ADMIN\tnsnames.ora


• %ECC_INSTALL_ROOT%\Repository\NETWORK\ADMIN\listener.ora • %ECC_INSTALL_ROOT%\Repository\NETWORK\ADMIN\sqlnet.ora • %ECC_INSTALL_ROOT%\Repository\admin\Ramb_scripts\upgrade\*.log • %ECC_INSTALL_ROOT%\Repository\admin\Ramb_scripts\ecc_getversion.log • %ECC_INSTALL_ROOT%\Repository\admin\rambdb\bdump\alert_rambdb.log • %ECC_INSTALL_ROOT%\Repository\admin\rambdb\bdump\*.trc • %ECC_INSTALL_ROOT%\Repository\admin\rambdb\pfile\* • %ECC_INSTALL_ROOT%\Repository\assistants\dbca\logs\*




Keep in mind in order for the Console.trc file to be created, the user must enable Console tracing when launching the console session. Also keep in mind that the console.trc file will “roll-over” and zip itself as soon as the EMC ControlCenter console is closed. The new zip file will be: console.trc.1.gz OS Commands: (From EMC ControlCenter Server host) None specified. Repository Exports: None specified. Additional Files: Collect information using the ControlCenter Console:



Table button. 6. Click inside the table to make it the active pane 7. Go to the top left hand corner of the console screen and select File-->Export-->Browse and

save export to a location such as your desktop or C: drive as a .csv file. • Collect an export of the Policies table view:



Table button. 6. Click inside the table to make it the active pane 7. Go to the top left hand corner of the console screen and select File-->Export-->Browse and

save export to a location such as your desktop or C: drive as a .csv file. • Collect an export of the Agents table view:




view, click the Table button. 6. Click inside the table to make it the active pane 7. Go to the top left hand corner of the console screen and select File-->Export-->Browse and

save export to a location such as your desktop or C: drive as a .csv file.


StorageWorks Array Discovery How do I troubleshoot discovering HP StorageWorks arrays in ControlCenter 6.0? What do I need to verify and collect for ControlCenter 6.0 HP StorageWorks array discovery issues? Background This agent monitors StorageWorks arrays and logs general information about StorageWorks subsystems and units. You can display information that this agent collects about subsystems, controllers, storagesets, remote copy sets, connections, units, and configured hosts. You can also disable or enable host access to a unit or set the offset for a connection. This agent is available with the following license keys:

• StorageScope (full functionality, except LUN Masking). • SAN Manager (discovery, monitoring, and LUN masking).

The following configurations may be used: • A host that is either physically connected to StorageWorks arrays and running the requisite API. • A standalone host running the requisite API that is connected to HP StorageWorks arrays. You can install this

agent on the ECC Server host. • If you are using the StorageWorks Command Console Agent, you can connect over the network through that

agent. • If you have a SANworks Management Appliance or OpenView Management Appliance in your environment, you

can connect to the HSG Element Manager Agent that is running on either of these appliances. • When you connect to one of these products, you establish connections to all the StorageWorks subsystems to

which these products are connected. Troubleshooting Steps The software formerly known as "steam agent" is now “Command Console Agent” (for Storageworks Command Console) or “Element Manager Agent for Management Appliance”

• Ensure that the agent host is running Windows (refer to the EMC ControlCenter Support Matrix on Powerlink for supported versions). Do the following to ensure that the host can access the StorageWorks Command Console Agent:

o Run the Agent Configuration Utility (provided by HP) on the host running the StorageWorks Command Console Agent.

o In the utility, add the name of the target agent host to the list of hosts allowed to access the StorageWorks Command Console Agent over the network. Specify Configuration as the Access Privilege for the newly added host.

• Configure the SANworks Management Appliance so Storage Agent for HP StorageWorks can access the HSG Element Manager Agent:

o Start HSG Element Manager. o Select Options, Agent Options. The HSG Element Manager Agent Options panel appears. o In the Add Client row, for Host Name, enter the name of the host where Storage Agent for HP

StorageWorks is to be installed. Select the Agent Config access level. Then click Submit. • For Tru64 support, ensure that the array has HP Array Control software installed. • Verify a successfully installed HP StorageWorks (SW) Agent is present and running. (Check services for Steam

Agent) • Verify connection type (reference the ControlCenter Support Matrix for supported versions)

o Management Appliance: HSG Element Manager SANworks Management Appliance OpenView Management Appliance

o Network connection: ACS Solutions Software • Verify there is at least one active Store.




• Verify the host running the ControlCenter agent is authorized to communicate with the HP steam agent. • Verify host running HP steam agent has physical access to the HP SW array (inq). • Verify DCP discovery_00 for Storage Agent for HP StorageWorks is enabled. • Verify assisted discovery has completed successfully. • Document approximate time of discovery.


LCW Subcategory: StorageWorks –Array MNQ -STORAGEWORKS

• [ECC_HOME]\MNQ6*\MNQ.log • [ECC_HOME]\MNQ6*\MNQ_B*.log.gz • [ECC_HOME]\MNQ6*\MNQ.ini • [ECC_HOME]\MNQ6*\MNQ_Err.log • [ECC_HOME]\MNQ6*\MNQ_B*_Err.log • [ECC_HOME]\MNQ6*\MNQ_Discovery.log • [ECC_HOME]\MNQ6*\MNQ_Discovery.log.gz












Keep in mind in order for the Console.trc file to be created, the user must enable Console tracing when launching the console session. Also keep in mind that the console.trc file will “roll-over” and zip itself as soon as the EMC ControlCenter console is closed. The new zip file will be: console.trc.1.gz


• None specified. Repository Exports:


Additional Files: Collect from Console Workstation where discovery was initiated:

• Export Agents Policy View 1. Open the ControlCenter Console 2. Select ECC Administration on the blue toolbar 3. Click the dropdown arrow next to ECC Administration, click Agents 4. In the Tree Panel on the left side of the Console, put a check next to the folder name Hosts. Make


view, click the Table button. 6. Click inside the table to make it the active pane 7. Go to the top left hand corner of the console screen and select File-->Export-->Browse and save




button. 6. Click inside the table to make it the active pane 7. Go to the top left hand corner of the console screen and select File-->Export-->Browse and save






6. Click inside the table to make it the active pane 7. Go to the top left hand corner of the console screen and select File-->Export-->Browse and save

export to a location such as your desktop or C: drive as a .csv file. • Collect steam agent log

o Windows this is the application event log. o Collect from UNIX platforms /opt/steam/

• From ControlCenter console – Discover – Review Progress – StorageWorks – need a stored result with an object for the array in question.


SMI Array Discovery How do I troubleshoot discovering SMI arrays in ControlCenter 6.0? What do I need to verify and collect for ControlCenter 6.0 SMI array discovery issues? Background CIM is based on a client-server architecture. The clients send requests over the network to the server (CIMOM). The server examines the requests and forwards them to the appropriate provider. Native ECC storage agent will override SMI Discovered objects if that storage agent type is installed. Refer to the EMC ControlCenter 6.0 Planning and Installation Guide, Volume 1, for additional information. SMI Provider is developed by individual storage and switch vendors not supported by CC’s SMI agent. The Provider uses native APIs to monitor and control the arrays and switches. For Symmetrix and CLARiiON® arrays, the provider is in an option in Solutions Enabler. A Provider populates a namespace in the CIM Object Manager. EMC Namespace default is root/emc. The agent and provider do not have to be on the same host. To install the SMI option of Solutions Enabler requires a local install of Solution Enabler from CDs. Reference the EMC ControlCenter 6.0 Support Matrix for a list of the array and tape hardware vendors with version 1.1.0 SMI-S providers that are supported by EMC ControlCenter Storage Agent for SMI. Current EMC SMI-S providers are available for download on Powerlink. Refer to SNIA.org for the specific product families and models for each vendor listed in the Support Matrix. EMC cannot vouch for the accuracy or consistency of the data provided by SMI-S 1.1.0 compliant providers not listed in the Support Matrix. Troubleshooting Steps

• Verify that the agent host can resolve the SMI provider host name. • Verify the SMI provider software/firmware (vendor specific) is installed, the listening port, the namespace, user,

and password is correct. • Verify if SSL security is active; verify certificate file name including the file name and the full path. • Verify that there is an active SMI agent running (mgxagent.exe). • Verify the discoveryType matches the array type. For instance: If a user uses discovery type HDS, but the type

of the array that the SMI agent discovered is CLARiiON, the agent will not manage it. If users don't know or are not sure of the array type, they should use DiscoveryType "SMI Arrays.”



• MS Services and Task Manager should not be used to make this determination. They are proven to be unreliable.

• Check to see if Store is active on Console or by verifying if store.trc file is active and being updated. • Verify proper version of Solutions Enabler is installed with the SMI option for Symmetrix and CLARiiON arrays.

(Solutions Enabler 5.4 with the SMI option was the first version to support this functionality.) • Verify that the Pegasus CIM Server EMC SMI provider is running for Symmetrix and CLARiiON arrays. • Other vendors require looking at their documentation to see how they have implemented this functionality. • Review Discovery progress in Discover->Review Progress pull down, then select DiscoverType, provider host. • Document approximate time of discovery. • Check the agent view to see the arrays being managed by the SMI agent. The Properties view for the

Symmetrix will indicate Managed= “No Symmetrix Agent”. Information to Collect Note: All logs must be time relevant to the problem event. In addition, Server and Store log file sets must contain all available trace files.


Using the Log Collection Wizard please select the following category and subcategories which will collect the recommended log files indicated below. Deselect any log types or log files that are not applicable to your environment. For instructions on using the Log Collection Wizard see emc154217. LCW Category: Object Discovery and Monitoring

LCW Subcategory: SMI –Array MGX -SMI ARRAYS

• [ECC_HOME]\MGX6*\MGX.log • [ECC_HOME]\MGX6*\MGX_B*.log.gz • [ECC_HOME]\MGX6*\MGX.ini • [ECC_HOME]\MGX6*\MGX_Err.log • [ECC_HOME]\MGX6*\MGX_B*_Err.log • [ECC_HOME]\MGX6*\MGX_Discovery.log • [ECC_HOME]\MGX6*\MGX_Discovery.log.gz














OS Commands: (From EMC ControlCenter Server host) None specified. Repository Exports: None specified. Additional Files: None specified.


SMI Arrays Provider How do I troubleshoot the SMI provider in ControlCenter 6.0? Background SMI Provider is developed by individual storage and switch vendors. The provider uses native APIs to monitor and control the arrays and switches. For Symmetrix and CLARiiON arrays, the provider is in an option in Solutions Enabler 5.4. A provider populates a namespace in the CIM Object Manager. EMC Namespace default is root/cimv28. The agent and provider do not have to be on the same host. The SMI agent only supports storage arrays. To install the SMI option of Solutions Enabler requires a local install of Solution Enabler from CDs. The following information came from the troubleshooting provided by Solutions Enabler engineering. Architecture CIM is based on a client-server architecture. The clients send requests over the network to the server (CIMOM). The server examines the requests and forwards them to the appropriate provider. Common Error Messages The CIM client displays a variety of error messages. This section lists the most common error messages and the reason for the error:

• Error: Cannot connect to localhost:5988. Connection failed. o Reason: CIM Server is not running or it is listening on a different port.

• Error: CIM_ERR_ACCESS_DENIED: Access to a CIM resource was not available to the client: "provider blocked."

o Reason: The provider is disabled. Use the command "cimprovider -l -s" to display the status of the provider. The command cimprovider is also used to enable the provider (to get the syntax, issue the command "cimprovider -h"). However, try to find out why the provider was disabled first.

• Error: CIM_ERR_ACCESS_DENIED: Access to a CIM resource was not available to the client: "Remote privileged user access is not enabled."

o Reason: In the Pegasus configuration file, cimserver_planned.conf, the entry enableAuthentication is set to true; however, the entry enableRemotePrivilegedUserAccess, in the same file, is set to false.

• Error: CIM_ERR_FAILED: A general error occurred that is not covered by a more specific error code: "provider initialization failed."

o Reason: Pegasus is unable to fire up the provider. See the Debugging Techniques section. • Error: CIM_ERR_FAILED: A general error occurred that is not covered by a more specific error

code: "The requested feature is not presently available." o Reason: Operation denied by CLARiiON array. Most likely, the SMI-S provider is not a privileged user

on the CLARiiON storage array. Therefore, the SMI-S provider can only issue read only operations against the array (as opposed to active control operations). See the Log Files section.

• Error: CIM_ERR_FAILED: A general error occurred that is not covered by a more specific error code.

o Reason: Generally, due to a failed return from a storapi call. Most likely, the client attempted to access a device/entity/object that does not exist, inaccessible, no longer valid, etc. See the Debugging Techniques section.

• Error: CIM_ERR_FAILED: A general error occurred that is not covered by a more specific error code: "Provider not available."

o Reason: The requested method name is either incorrect or not registered in the Repository (method names are case sensitive).

• Error: CIM_ERR_FAILED: A general error occurred that is not covered by a more specific error code: "Unknown error." ...

o Reason: There are a number of reasons for this error message. Some of the reasons are: wrong DLLs or libraries, wrong path, etc. Be sure to verify the correct DLLs and/or libraries are getting loaded. If you have installed multiple versions of the kit, the cimserver may be finding an incorrect version of the libraries in its search path. Also, see the next error.

• Error: CIM_ERR_FAILED: A general error occurred that is not covered by a more specific error code: "Unknown error."


o Reason: The called storapi returned a status not known to the provider. See the Debugging Techniques section.

• Error: CIM_ERR_INVALID_CLASS: The specified class does not exist: "<class-name>" o Reason: The class name is either incorrect (class names are case sensitive) or not registered in the

Repository. • Error: CIM_ERR_INVALID_NAMESPACE: The target namespace does not exist: "<namespace-

name>" o Reason: The namespace name is either incorrect or not registered in the Repository.

• Error: CIM_ERR_INVALID_NAMESPACE: The target namespace does not exist: "root/PG_Internal" o Reason: The root/PG_Internal namespace in the Repository is required for authentication and

cimconfig commands. • Error: CIM_ERR_INVALID_PARAMETER: One or more parameter values passed to the method were

invalid. o Reason: A storapi was called with an invalid parameter. See the Debugging Techniques section.

• Error: CIM_ERR_NOT_FOUND: The requested object could not be found. o Reason: An incorrect entity was supplied to a storapi or the entity does not exist.

• Error: HTTP Error (401 Unauthorized). o Reason: The supplied username and password are incorrect. In the Pegasus configuration file,

cimserver_planned.conf, the entry enableAuthentication is set to true. This means Pegasus checks the username and password against its authorization file, i.e., cimserver.passwd.

• Error: SSL Exception: Could not get server certificate. o Reason: The server is unable to locate the default certificate file (server.pem) in the configuration

directory (EMC\SYMAPI\config). The server locates the certificate file by examining the sslCertificateFilePath parameter in the file cimserver_planned.conf in the installation directory EMC\SYMCLI.

• Error: SSL Exception: Could not load certificates in to trust store. o Reason: The client is unable to locate its certificate file.

Symptoms This section outlines the reported problems in terms of the observed behavior and gives you some ideas as to what might be the problem as well as suggesting areas you can examine.

• Cannot connect to the http or https ports. o Verify the port numbers are correct and match the port numbers specified in the Pegasus's

configuration file (cimserver_planned.conf). Also, in the same file, verify the following entries: enableHttpsConnection=true enableHttpConnection=true

• The provider and/or the client appear to be hung. o It may be that the provider is processing a complex operation such as an association that involves

many classes against a number of arrays that have many devices. Try to issue a simpler request from another client for example, "EnumerateInstanceNames for the class EMC_StorageSystem". If the second client completes its operation, most likely the first client eventually completes its operation in due time as well. If on the other hand the second client also hangs, enable Pegasus tracing (see the Debugging Techniques section) and issue another request to get an idea of what happens to the clients' requests.

• The client receives an exception. o You need to figure out where the exception was thrown. Here are the potential sources of an

exception: The exception could have been thrown by the client itself. Pegasus PAL The provider C++ runtime

o Generally, storapi does not throw an exception. However, if an API returns an unsuccessful status, the caller of the API in the provider may in turn throw an appropriate exception. In any case, the best approach is to turn on various tracing mechanisms (see the Debugging Techniques section) and repeat the operation that resulted in the exception in the hope of identifying the source of the problem.

• cimserver (Pegasus CIM Object Manager - EMC_SMI_PROVIDER) service (daemon) does not start or immediately stops.


o Assuming the correct and the necessary libraries (or DLLs) are not missing and are not corrupt (or of 0 length), verify the following:

The Repository directory exists and populated (EMC\SYMCLI\repository). There is sufficient storage space on the installation disk for various log files. The certificate file exists and is not of 0 length (for example,

EMC\SYMAPI\config\server.pem). The ports (for example, 5988 and 5989) are not in use by some other processes. The various file names and paths in the Pegasus configuration file

(EMC\SYMCLI\cimserver_current.conf) are valid. Your process has adequate priviledges to start a system process.

o Sometimes, it might be helpful to run the cimserver in a "console" window (terminal), as a non-service (non-daemon) process and watch for any unusual error messages. To do so, in the EMC\SYMCLI\storbin directory:

On Windows, type the command: cimserver On UNIX, issue this command: cimserver daemon=false

Debugging Techniques

• Debugging CIMOM and the providers is a complex task. The key to successfully debug a reported problem is to collect as much debugging information as possible.

• One of the first things you need to establish is which component of the architecture is issuing the error message, or, in the situation where the client appears not to receive any replies (may seem hung), you need to determine how far the client's request are getting.

• Fortunately, there are some "knobs" available to allow you to trace the flow of messages through the system. Pegasus Trace Facility

• Use the Pegasus's utility cimconfig to turn on Pegasus trace data. However, before adjusting any of the Pegasus' configuration and trace data, be sure to make a copy of the file EMC\SYMCLI\cimserver_planned.conf so that you can restore it back once your troubleshooting work is completed.

• By default, the trace data is written into a file called cimserver.trc in Pegasus' home directory. • Some examples:

o cimconfig -s traceLevel= 4 // Set the current traceLevel o cimconfig -s traceComponents=ALL // Set the current traceComponents o cimconfig -s traceComponents=ALL -p // Set the planned traceComponents

• Other useful options for cimconfig: o cimconfig // to get help o cimconfig -l -c // to list the current settings o cimconfig -l -p // to list all possible (planned) settings and their values

• Hints: o Check the cimserver.trc to be sure the requests are passed on to the provider. In the trace file, you

want to look for entries indicating the provider is found, the appropriate method of the provider is invoked, and there was a response back from the provider. For example:

Provider for class EMC_StorageSystem provider emc_provider found. providerName = emc_provider found. Calling provider.enumerateInstanceNames: emc_provider Entering method CIMOperationResponseEncoder::sendResponse()

o The cimserver.trc file also includes the XML messages (XmlIO) from the clients and the responses back to the clients (<SIMPLEREQ> and <SIMPLERSP> tags).

o Generally, you can explain unusual behavior by studying the XML messages. Look for missing values, incorrect syntax, extra punctuation characters, etc.

Provider Trace Facility

• Edit the file EMCProvider.properties and set the property com.emc.storage.provider.trace.level to 4, as follows: o com.emc.storage.provider.trace.level = 4

• You must restart the provider for the new settings to take hold. • The trace file is in the SYMAPI log subdirectory with the following name:

o emc_provider-<current-date>.log • Hints:

o Check the provider's trace file and look for any error messages, unknown exceptions, and the general flow of activities within the provider.


storapi Trace Facility

• If you believe the problem has something to do with storapi, you can turn on the trace output of storapi by setting the following environment variables and restarting the cimserver:

o symapi_debug=-1 o symapi_debug_filename=filename

• Hints: o The storapi log files generally have useful messages as to the source of the problem. These files are

in the log directory of the SYMAPI directory tree, e.g., EMC\SYMAPI\log\symapi-<current-date>.log. Entries from the SMI-S provider are identified by "EMC:SMI Provider".

Log Files It would be helpful to examine the various log files periodically to be sure there are no problems reported.

• Pegasus: The Pegasus' log files are placed in the directory EMC\SYMCLI\logs. • SMI-S Provider: The Provider's daily log files are in the directory EMC\SYMAPI\log with the file format of

emc_provider-<current-date>.log. • Storapi: Under normal conditions, storapi records significant events and error messages in its daily log file in

the SYMAPI log directory (EMC\SYMAPI\log\symapi-<current-date>.log). EMC Utilities The program test (not included in the installation kit) is a CIM client with a simple self-explanatory menu system that allows you to interact with the cimserver. It is a must for any troubleshooting activity. Pegasus Utilities The following utilities are placed in the EMC\SYMCLI\storbin directory during the installation of the cimserver. To get help, simply invoke the program with the "-h" switch:

• cimauth // add authorization for a CIM user • cimconfig // manipulate cimserver's settings • cimmof // connect to cimserver to update the Repository • cimmofl // directly update the Repository (the server has to be restarted to see the changes) • cimprovider // enable/disable/display providers statuses • cimuser // add new user and password for authentication


Fujitsu Siemens BS2000 Host Discovery How do I troubleshoot discovering a Fujitsu Siemens BS2000 host in ControlCenter 6.0? What do I need to verify and collect for ControlCenter 6.0 Fujitsu Siemens BS2000 discovery issues? Troubleshooting Steps The SE/API libraries for the BS2000 are shipped directly by Fujitsu Siemens to its customers as part of its software products. Refer to the EMC ControlCenter Support Matrix for the supported OS level.

• Verify the Common Mapping Agent (CMA) is running. • Verify Network connective between Agent host & BS2000 system for each VM (use ping). • On every VM to be discovered:

o Verify that the POSIX subsystem is installed and started (all sinlib libraries for the network, symapi and ssca-bs2 should be installed in the POSIX environment).

o Verify that the SYMAPI-D subsystem is installed and started. o Verify that symapisrv is running in the backgroup (use /STA P in BS2000 or ps -f in POSIX) o Verify that the POSIX file /var/sccabs2/config/nodewwn (for S1xx systems) or the POSIX file

/var/sccabs2/config/hostconf (for SXxx systems) contain the correct information.

• For further installation details or instruction, please consult the FSC manual on the SHC-OSD. • Verify there is at least one active Store.



• Document approximate time of discovery. Information to Collect Note: All logs must be time relevant to the problem event. In addition, Server and Store log file sets must contain all available trace files. Using the Log Collection Wizard please select the following category and subcategories which will collect the recommended log files indicated below. Deselect any log types or log files that are not applicable to your environment. For instructions on using the Log Collection Wizard see emc154217. LCW Category: Object Discovery and Monitoring

LCW Subcategory: Common Mapping Agent EGZ -COMMON MAPPING

• [ECC_HOME]\EGZ6*\EGZ.log • [ECC_HOME]\EGZ6*\EGZ_B*.log.gz • [ECC_HOME]\EGZ6*\EGZ.ini • [ECC_HOME]\EGZ6*\EGZ_Err.log • [ECC_HOME]\EGZ6*\EGZ_B*_Err.log • [ECC_HOME]\EGZ6*\EGZ_Discovery.log • [ECC_HOME]\EGZ6*\EGZ_Discovery.log.gz


SYMAPI_UNIX


• /var/symapi/log/* SYMAPI_WINDOWS

• %ProgramFiles%\EMC\SYMAPI\log\symapi*.log • %ProgramFiles%\EMC\SYMCLI\*.log










OS Commands: (From EMC ControlCenter Server host) None specified. Repository Exports: None specified. Additional Files:

• /var/symapi/log/symapi-yyyymmdd.log from day of discovery • From BS2000VM Host, collect:

o Symapi logs o sccabs2 logs


o These can be found under the POSIX directories /var/symapi/log and /var/sccabs2/log. Files are symapi_yyyymmdd.log and sccabs2_yyyymmdd.log


Invista Discovery How do I troubleshoot discovering an Invista in ControlCenter 6.0? What do I need to verify and collect for ControlCenter 6.0 Invista discovery issues? Background

• The EMC Invista Agent runs on a server networked to an EMC Invista. The agent collects detailed information about the Invista which you can view through ControlCenter Properties views. It does not have any capabilities for configuring or managing an Invista instance. For those functions, use the Invista’s native management software, which can be launched from within ControlCenter by right-clicking an Invista instance and selecting Management Tools.

• This agent is available with the following license keys: o StorageScope o SAN Manager

Troubleshooting Steps

• Before deploying Storage Agent for Invista: o Ensure the Master Agent has been installed on the host. o Ensure that the target host is running a supported Windows operating system

Refer to EMC ControlCenter Support Matrix on Powerlink for supported versions o The host is running the Invista CLI. o The host is running a supported version of the Java JRE or SDK

With the path to the Java executable in the PATH environment variable • Verify there is at least one active Store.



• Invista is an Assisted Discovery. Provide the following information in the Discover Other Objects dialog box: o SPA Network Name—Network name of the storage processor A (SPA) of the Control Path Processor

of the EMC Invista instance to be discovered. o SPB Network Name—Network name of the SPB of the Control Path Processor of the EMC Invista

instance to be discovered. o Username—Username to access the EMC Invista. o Password—Password to access the EMC Invista.

• Verify that the Discovery_00 policy is enabled and that the host running the primary Invista agent is listed in the ApplyTo tab of the LocalDiscovery policy. ( This policy is Enabled by default )

• Determine what, if any, portion of the Invista has been discovered. • Document approximate time of discovery. • Determine if there is an orange diamond next to the Invista which indicates that it is not fully discovered. If

Invista is in the console, but not fully discovered, stop Invista agents, delete Invista, and then restart one Invista agent. Allow Invista agent to run for several minutes.



LCW Subcategory: EMC Invista -Storage Virtualization FGX -INVISTA

• [ECC_HOME]\FGX6*\FGX.log • [ECC_HOME]\FGX6*\FGX_B*.log.gz • [ECC_HOME]\FGX6*\FGX.ini • [ECC_HOME]\FGX6*\FGX_Err.log • [ECC_HOME]\FGX6*\FGX_B*_Err.log • [ECC_HOME]\FGX6*\FGX_Discovery.log • [ECC_HOME]\FGX6*\FGX_Discovery.log.gz
























view, click the Table button. 6. Click inside the table to make it the active pane


7. Go to the top left hand corner of the console screen and select File > Export > Browse and save export to a location such as your desktop or C: drive as a .csv file.


Centera Discovery How do I troubleshoot discovering Centera arrays in ControlCenter 6.0? What do I need to verify and collect for ControlCenter 6.0 Centera array discovery issues? Centera Components Background Centera

• Content Addressed Storage (CAS) • Runs Linux for an OS • Runs CentraStar Software on top of Linux OS

CentraStar • Centera Software • Like Symmetrix Enginuity Code • Like CLARiiON Flare Code

CenteraViewer • GUI and Command Line Interface for Centera • Like Navisphere Manager for CLARiiON

CenteraPing • Specialized Ping Command for Centeras • Use CenteraPing to check communication with "Access Nodes"

Access Nodes and Storage Nodes • All Centeras are made up of a collection of Rack Mounted PCs • Each of these Rack Mounted PCs is identical • Some of them are assigned the Role of "Access Node" • Some of them are assigned the Role of "Storage Node"




• Verify Centera Agent Discovery_00 Data Collection Policy (DCP) is Enabled. • Verify Primary Access Node and Failover IP Addresses. Use CenteraVerify and CenteraPing • Verify Path to Authorization File (.pea) if implemented. (Engage Centera Support) • Should be in the Centera Agent Directory • Document approximate time of Discovery • Verify Centera agent is running, CGCagent. • In the agents log, search for MoPI (Monitoring API) part of the Centera SDK responsible for sending the

discovery information which the agent uses to populate the Repository. A log file message with the MoPI tag is an error message from the Centera SDK.

Additional logging information:

• FPLIBRARY_LOG_DEBUG_FILTER = ALL • FPLIBRARY_LOG_FILE_DEBUG = <logfile> • CASScript • http://groups.yahoo.com/group/CenteraTechGroup/files/CAS%20Tools/CASScript/ • Logs from Centera are only available to support personnel, engage Centera Support.


Change ECC Template: • Run this script if the Centera Cluster is monitored by ECC 5.2 Agent (ask the customer for this information). • In CV, follow the path Tools > Service > MoPI > and then select one of the following:

o Install ECC 5.2 template - for ECC versions 5.2, 5.2SP1 and 5.2SP2. o Install ECC 5.2SP3 template - for ECC versions 5.2SP3 and above.

• Note: Customers that use the ECC 5.2.x Agent for Centera to monitor their infrastructure cannot use the new Health Report format.


LCW Subcategory: Centera -CAS CGC -CENTERA

• [ECC_HOME]\CGC6*\CGC.log • [ECC_HOME]\CGC6*\CGC_B*.log.gz • [ECC_HOME]\CGC6*\CGC.ini • [ECC_HOME]\CGC6*\CGC_Err.log • [ECC_HOME]\CGC6*\CGC_B*_Err.log • [ECC_HOME]\CGC6*\CGC_Discovery.log • [ECC_HOME]\CGC6*\CGC_Discovery.log.gz














Keep in mind in order for the Console.trc file to be created, the user must enable Console tracing when launching the console session. Also keep in mind that the console.trc file will “roll-over” and zip itself as soon as the EMC ControlCenter console is closed. The new zip file will be: console.trc.1.gz OS Commands (from EMC ControlCenter Server host): None specified. Repository Exports:


Additional Files:

• From the ControlCenter Console collect an export of the Alerts table view: 1. Open the ControlCenter Console 2. Select ECC Administration on the blue toolbar 3. Click the dropdown arrow next to ECC Administration, click Alerts 4. In the Tree Panel on the left side of the Console, put a check next to the Hosts folder. Make sure

there are no other items checked. 5. In the Target Panel on the right you will see the Alerts view. On the top of that view, click the Table

button. 6. Click inside the table to make it the active pane 7. Go to the top left hand corner of the console screen and select File > Export > Browse and save

export to a location such as your desktop or C: drive as a .csv file. • From the ControlCenter Console collect an export of the Policies table view:




export to a location such as your desktop or C: drive as a .csv file. • From the ControlCenter Console collect an export of the Agents table view:







Celerra Discovery How do I troubleshoot discovering Celerra NAS devices in ControlCenter 6.0? What do I need to verify and collect for ControlCenter 6.0 Celerra NAS discovery issues? Background Prerequisites You must use assisted discovery with this agent (refer to the ControlCenter 6.0 Planning and Installation Guide Volume 1 for details). Before deploying this agent:

• Ensure that this agent has exclusive access to port 162 (the default) where it listens for SNMP traps from the devices it manages. If other processes (such as Fibre Channel Connectivity Agent or Microsoft SNMP Trap Service) use port 162, they can prevent this agent from receiving traps and subsequently sending ControlCenter alerts. If necessary, you can change the Storage Agent for NAS listening port—refer to the ControlCenter 6.0 Planning and Installation Guide Volume 1 for detailed information on controlling Port 162 Usage for SNMP.

Do the following when using this agent in a proxy setup: • Ensure that Symmetrix serial numbers do not exceed nine digits. If a Celerra Network Server is attached to a

Symmetrix array with a serial number that has more than nine digits, Symmetrix Fibre Channel information returned by the Celerra might be incorrect or not displayed.

• Ensure that, when Symmetrix or CLARiiON arrays are attached to a Celerra Network Server, disks are mapped from the storage array to the Celerra system and that disks are mapped to Symmetrix Fibre Channel ports and CLARiiON front-end ports. Otherwise ControlCenter will not display information for those ports (even if a physical connection exists between the array and the Celerra).

• Beware that when CLARiiON front-end ports are discovered via proxy by this agent, information about those ports will not contain the WWN attribute.

Configuration Detailed information is available in the ControlCenter 6.0 Planning and Installation Guide Volume 1, under “Plan for Storage Agent for NAS”. Included is information on using emcplink to specifiy security policies for secure shell client authentication (ssh) and how to configure Celerra to send SNMP traps to ControlCenter. Troubleshooting Steps Note: Command outputs are for reference only. Command examples do not imply the command is supported for all versions.

• Verify that the NAS agent is installed and active NAS agent (CNN) • Verify that the NAS agent is licensed

o Administration>Install>Edit Licesnes • Verify from the Agent properties view the version of CNN agent and the host that it is on • Verify the version of Control Station. Use cmd /nas/bin/nas_version [returns x.x.x-x like 5.3.7-0] • Verify the version of Linux. Use cmd uname -r {returns format like 2.4.9-34.5303.EMC] • Verify the version of NAS code. Use cmd /nas/bin/server_version ALL [returns multiple lines like]

server_2 : Product: EMC Celerra File Server Version: T5.3.7.0 server_3 : Product: EMC Celerra File Server Version: T5.3.7.0

• Verify that the Java Process is running on the Control Station. Use cmd ps -aex |grep java [returns format such as:]

21927 ? S 0:15 /nas/http/webui/tools/j2sdk1.4.2_01/bin/java –server 22200 ? S 0:00 /nas/http/webui/tools/j2sdk1.4.2_01/bin/java -server (output omitted)

• Verify there is at least one active Store. o ECCService_Stor on W2k o ECCService_Store.exe on W2003


o MS Services and Task manager should not be used to make this determination. They may be unreliable. Check to see if Store is active on Console or by verifying if store.trc file is active and being updated.

• Verify DCP discovery_00 for Storage Agent for NAS is enabled. • If discovering through telnet, verify telnet is enabled. If you cannot telnet to the Celerra Control Station, telnet

may be disabled. To enable Telnet: o Use an SSH client (putty, etc) to connect to the Celerra Control Station. Log in as "root"

1. Check to see if Telnet is enabled: /sbin/chkconfig telnet --list

2. Enable Telnet: /sbin/chkconfig telnet on /sbin/service/xinetd start

o If discovering through ssh, you need to make sure the ssh key is authenticated for the host machine

where the NAS agent is installed. Also, if the key changes, you will need to authenticate again. On the host with the NAS agent run:

1. emcplink -ssh -interactive -<ssh version> -pw xxx sshuser@<ip_address>

o If discovering through ssh ensure that the ssh daemon is running: 1. Log in to the Control Station 2. Type the following command:

ps -aex |grep sshd

3. If running, you will see a line similar to the following: 882 ? S 0:00 /usr/sbin/sshd

• The emcplink utility allows you to specify security policies for secure shell (ssh) client authentication (required

for Storage Agent for NAS to discover Celerra). See the ControlCenter 6.0 Planning and Installation Guide Volume 1, “Plan for Storage Agent for NAS” for details on configuration.

• Note: NAS code 4.1.4 and above Telnet is turned off. Below 4.1.4 Telnet is enabled by default. User will need to use either ssh (secure shell protocol) or Telnet.

• If you are unable to discover a Celerra using SSH, reference emc95783. • Verify if Celerra is connected to a Symmetrix, a CLARiiON, or both. Use the command:

nas_storage -l id acl name serial_number 1 0 APM00023801040 APM0002380104

• Verify the user name and password when doing the assisted discovery. Try logging on. • When doing assisted discovery for the NAS agent, verify the contents of the review progress bar for NAS

Container. o Does the Celerra show up under storage systems? o Is the Celerra fully discovered? o Are the file systems populated? o Under Data Mover are the exports and mount points populated? o Are the NAS devices populated? o Are the volumes populated?

• Clear the Review Progress cache and remove all failed attempts from the Review Progress window before retrying assisted discovery.

• Stop and restart the NAS agent. Allow NAS Agent to run for several minutes. • Document approximate time of discovery.

Also See: Article emc139913: Deleted Celerra file systems cause incorrect storage reporting in ControlCenter and StorageScope Information to Collect Note: All logs must be time relevant to the problem event. In addition, Server and Store log file sets must contain all available trace files.



LCW Subcategory: Celerra -NAS CNN -CELERRA-NAS

• [ECC_HOME]\CNN6*\CNN.log • [ECC_HOME]\CNN6*\CNN_B*.log.gz • [ECC_HOME]\CNN6*\CNN.ini • [ECC_HOME]\CNN6*\CNN_Err.log • [ECC_HOME]\CNN6*\CNN_B*_Err.log • [ECC_HOME]\CNN6*\CNN_Discovery.log • [ECC_HOME]\CNN6*\CNN_Discovery.log.gz










CONSOLE_WINDOWS • %SystemDrive%\Documents and Settings\*\ecc\data\*\console.trc


• %SystemDrive%\Documents and Settings\*\ecc\data\*\console.trc.*.gz • %SystemDrive%\Documents and Settings\*\ecc\data\*\consoleLaunch.log • %SystemDrive%\Documents and Settings\*\ecc\data\*\ecc_console.ini



• From the Repository host obtain a current export by running: Start, Programs, EMC, EMC ControlCenter, Repository Maintenance, ControlCenter Repository, Export Database

o %RAMBBACKUPDIR%\exp\rambdb_exp.dmp Additional Files:

• From the ControlCenter Console, select Discover, Review Progress for NAS Container and capture the output of the review results.














• From the Celerra Control Station: o /nas/bin/nas_version /nas/bin/nas_symm -l /nas/bin/nas_symm -i <symmetrix serial number> (for

each Symmetrix attached) o /nas/sbin/rootnas_fs -i -s fs01_snap1 o /nas/log/cmd_log.err o /nas/log/sys_log o server_mount ALL (if mount points are not showing) o server_mountpoint ALL -l (if mount points are not showing) o server_df ALL (if files systems are not being populated) o server_export ALL (if exports are not being populated)


Host Discovery How do I troubleshoot discovering hosts in ControlCenter 6.0? What do I need to verify and collect for ControlCenter 6.0 host discovery issues? Troubleshooting Steps



• If the host does not appear at all, the problem is with the Master Agent on the host: o Stop Master Agent on host. o Restart Master Agent on host. o Collect [ECC_HOME]\MGA6*\MGA.log.

• If the host appears, but is not fully discovered: o Determine what folders under the host are not discovered. o Verify that the HBA types and drivers are supported in the ControlCenter Support Matrix. o Verify that the SNIA compliant libraries are loaded on the system.

HBA Troubleshooting: General Notes:

• NIC Cards are discovered by the Master Agent • HBAs are discovered by the Host Agent • After you add Devices to a Host, you must Re-Boot the Host before it will see them

SNIA Library Locations: WINDOWS

• The default library paths are specified for each host operating system in the esnapi.cfg file located: %ECC_Install_ROOT%\exec\esnapi\esnapi.cfg.

• Default HBA SNIA locations o Emulex: path=C:\WINNT\system32\emulexhbaapi.dll o qlogic: path=C:\WINNT\system32\qlsdm.dll

OR path=C:\Program Files\Qlogic Corporation\Qlsdm\Lib\qlsdm.dll

Also see solution emc145122, "How to install the EmulexHBAAPI.dll on Windows 2003 with STORport driver 1.20a3." SOLARIS

• For all UNIX type Hosts check the hba.conf file for SNIA Library Locations: /etc/hba.conf • Emulex - (Possible Location, check hba.conf): path=/usr/lib/libemulexhbaapi.so • qlogic - (Possible Location, check hba.conf): path=/usr/lib/libqlsdm.so • JNI - (Possible Location, check hba.conf): path=/opt/JNIsnia/Solaris/Jni/32bit/JniHbaLib.so

HP_UX

• HP HBA - (Possible Location, check hba.conf): path=/usr/lib/libhbaapihp.sl LINUX See solution emc97044, "How to ensure a Linux host is configured correctly for Fibre Channel HBA discovery in ControlCenter."

• Ensure that an EMC supported version of the Fibre Channel HBA Driver is installed.


• Ensure that the application support libraries (SNIA) have been installed. • For many vendors, the libraries are shipped with the driver however, they may need to be installed separately. • Check the driver installation manual and release notes for information on installing the libraries. • If SNIA libraries are not available for the HBA, then ControlCenter will not be able to import its World Wide

Names. • If a supported driver and its support libraries have been installed, check the contents of the file /etc/hba.conf. • This file is used by Solutions Enabler and ControlCenter Host Agent to identify the location of the SNIA libraries. • The file is created automatically by the HBA driver installation utility, and should appear something like this:

o qla2x00 /usr/lib/libqlsdm.so o com.emulex.emulexapilibrary /usr/lib/libemulexhbaapi.so

• If the hba.conf file does not exist or is empty, re-installing the application libraries will resolve the issue. • If you are sure that the SNIA-compliant libraries are present, use the above example to populate the file based

on the type of HBA installed in the host. • Ensure that the paths on each line point to a file that exists. • Once the modifications have been performed, restart the Linux Host Agent and the missing information will be

populated into the ControlCenter Console. Procedure to Replace HBAs:

1. Stop ControlCenter Host Agent on Host where HBA will be swapped. 2. Remove OLD HBA from Host. 3. Install NEW HBA on Host. 4. From ControlCenter Console Tree Panel -- Search for OLD HBA WWN. 5. Delete each instance of the OLD HBA WWN found in the Tree Search. (this will ensure the OLD HBA WWN is

not left behind in a Zone, etc. and is removed from the ControlCenter Repository) 6. Re-Boot Host with NEW HBA installed. 7. Start ControlCenter Host Agent. 8. Allow time for ControlCenter Host Agent to re-discover the Host and NEW HBA.

Note: If the Host Agent will not Discover the New HBA and HBAnywhere has been loaded, then search for C:\WINNT\system32\emulexhbaapi.dll. If emulexhbaapi.dll is not found in the system32 Directory: Copy it in there from another Host that does have it, then stop and re-start the Host Agent.

9. After replacing HBA run the following command to enter the New HBA WWN into the VCMDB: C:\Program Files\EMC\SYMCLI\symmask -sid <SymmID> -wwn <Old-WWN> replace <New-WWN>

The equivalent Procedure to enter the New WWN into the VCMDB through ControlCenter is:

1. Click Hosts 2. Select Adapter 3. Right-Click on HBA WWN 4. Select Masking from Pop-Up Menu 5. Select Replace from Pop-Up Menu 6. Source –Select Port Section:

1. Highlight OLD WWN 7. Target- Replace with Port Section:

1. Select Storage Array: Highlight Appropriate Storage Array 2. Select Port to replace with: 3. Highlight NEW WWN 4. Click Add Action Button 5. Click Continue Button

Procedure to Clean-up a Cloned Host in ControlCenter: Note: This Procedure CANNOT be performed on the Infrastructure Repository Host. See solution emc92671, "Hosts are disappearing from ControlCenter Console." Information to Collect Note: All logs must be time relevant to the problem event. In addition, Server and Store log file sets must contain all available trace files.



LCW Subcategory: Windows –Host | AIX –Host | HP-UX –Host | Solaris –Host | Linux –Host | Novell –Host XXX –Host Agent

• [ECC_HOME]\MXR6*\MXR.log • [ECC_HOME]\MXR6*\MXR_B*.log.gz • [ECC_HOME]\MXR6*\MXR.ini • [ECC_HOME]\MXR6*\MXR_Err.log • [ECC_HOME]\MXR6*\MXR_B*_Err.log • [ECC_HOME]\MXR6*\MXR_Discovery.log • [ECC_HOME]\MXR6*\MXR_Discovery.log.gz Where x represents the agent O/S acronym


ESNAPI • [ECC_HOME]\esnapi\data\*







CONSOLE_WINDOWS • %SystemDrive%\Documents and Settings\*\ecc\data\*\console.trc • %SystemDrive%\Documents and Settings\*\ecc\data\*\console.trc.*.gz • %SystemDrive%\Documents and Settings\*\ecc\data\*\consoleLaunch.log


• %SystemDrive%\Documents and Settings\*\ecc\data\*\ecc_console.ini CONSOLE_SOLARIS

• $HOME/ecc/data/*/console.trc • $HOME/ecc/data/*/console.trc.*.gz • $HOME/ecc/data/*/consoleLaunch.log • $HOME/ecc/data/*/ecc_console.ini

Keep in mind in order for the Console.trc file to be created, the user must enable Console tracing when launching the console session. Also keep in mind that the console.trc file will “roll-over” and zip itself as soon as the EMC ControlCenter console is closed. The new zip file will be: console.trc.1.gz OS Commands: (From EMC ControlCenter Server host) None specified. Repository Exports:


Additional Files: Collect information using the ControlCenter Console:














VMWare ESX Server Discovery How do I troubleshoot discovering a VMWare ESX server in ControlCenter 6.0? What do I need to verify and collect for ControlCenter 6.0 VMWare ESX server discovery issues? Background This agent discovers VMware ESX Servers, including the virtual machines, file systems, snapshots, and virtual disks. To discover and manage guest operating systems or applications running on a virtual machine, install a Master Agent and the appropriate host agent. One instance of the VMware Agent on a Windows host can manage multiple objects. One instance of the appropriate host agent on the guest operating system can perform discovery and management of the guest operating system or applications on a virtual machine. You can also deploy this agent on a ControlCenter infrastructure host. Do not deploy this agent on the ESX Server host itself. To create or manage virtual machines, including provisioning storage to a virtual machine, use VirtualCenter, an application outside of ControlCenter for managing the VMware ESX Server. To enable trap monitoring on the ESX Server using the ESX Management Interface or service console, refer to the ESX Server Administration Guide. Troubleshooting Steps

• Before installing the VMware Agent, ensure that: o The ControlCenter Server, Repository, Store, and Console are installed and running. o The Master Agent is deployed and running on the host where you plan to deploy the VMware Agent. o The host where you want to deploy this agent is running the Windows operating system and has

Ethernet connectivity to the ESX Server. Refer to the EMC ControlCenter Support Matrix on Powerlink for supported versions.

o There is sufficient disk space on the target host. • Before attempting a discovery of an ESX Server

o The ESX Server FQDN/hostname is resolvable from the host where the VMWare Agent resides. o The /etc/vmware/hostd/config.xml file on the ESX Server must be modified as follows:

Example of the section of the original config.xml file that requires modification: 1. Make a backup copy of the config.xml before editing. 2. Delete the SDK redirection 3. Move the SDK namespace from the <https> section to the <http> section

Red highlight = Delete text section Green highlight = add text to section

<http> <port>80</port> <proxyDatabase> <server id="0"> <namespace> / </namespace> <host> localhost </host> <port> 9080 </port> </server>

<server id="1"> <namespace> /sdk </namespace> <host> localhost </host> <port> 8085 </port> </server> <redirect id="0"> /ui </redirect> <redirect id="1"> /mob </redirect> <redirect id="2"> /sdk </redirect> </proxyDatabase> </http> <https> <port>443</port> <proxyDatabase>


<server id="0"> <namespace> / </namespace> <host> localhost </host> <port> 9080 </port> </server> <server id="1"> <namespace> /sdk </namespace> <host> localhost </host> <port> 8085 </port> </server> <server id="2"> <namespace> /ui </namespace> <host> localhost </host> <port> 8080 </port> </server> <server id="3"> <namespace>/mob</namespace> <host>localhost</host> <port>8087</port> </server> </proxyDatabase> </https>

4. Restart the vmware-mgmt service by running: /etc/rc.d/init.d/mgmt-vmware restart

• Before attempting a discovery of a VirtualCenter host

o Verify the above steps in ‘ESX Server discovery” have been performed. o The %ALLUSERSPROFILE%\Application Data\VMWare\VMWare VirtualCenter\vpxd.cfg file on the

VirtualCenter host must be modified as follows: Example of the section of the original vpxd.cfg file that requires modification:

1. Make a backup copy of the config.xml before editing. 2. Comment out SDK redirection 3. Uncomment SDK Server definition

Red highlight = Comment out text section Green highlight = Uncomment text section

<http>      <proxyDatabase> <server id="0"> <namespace> / </namespace> <host> localhost </host> <port> -1 </port> </server>  <redirect id="2">/ui</redirect> <redirect id="3">/mob</redirect>   <server id="1"> <namespace> /sdk </namespace> <host> localhost </host> <port> -2 </port> </server>


     </proxyDatabase>

</http>

1. Restart the ‘VMWare VirtualCenter Server’ service

• The VMWare agent requires an Assisted Discovery. Provide the following information in the Discover Other Objects dialog box:

o ESX Server Provider Host Name—Network name of the host where the VMware ESX provider resides. This must be the fully qualified domain name or IP address of the ESX Server.

o ESX Server Provider Username—Username of the host where the VMware ESX provider resides. o ESX Server Provider Password—Password for the specified username.

• For discovery of the ESX Server VirtualCenter Server, provide the following information in the Discover Other Objects dialog box:

o VirtualCenter Server Name—The VirtualCenter Server Name can be an FQDN or an IP address. o VirtualCenter Server Username—Username of the VirtualCenter Server. o VirtualCenter Server Password—Password for the specified username.



• The VMware Tools service for a VMware Guest must be set to start up automatically or the ControlCenter Console may not be able to display accurate information about the state of the VMware guest and VMware tools.


LCW Subcategory: VMWare -Virtual Machine EGV -VMWARE

• [ECC_HOME]\EGV6*\EGV.log • [ECC_HOME]\EGV6*\EGV_B*.log.gz • [ECC_HOME]\EGV6*\EGV.ini • [ECC_HOME]\EGV6*\EGV_Err.log • [ECC_HOME]\EGV6*\EGV_B*_Err.log • [ECC_HOME]\EGV6*\EGV_Discovery.log • [ECC_HOME]\EGV6*\EGV_Discovery.log.gz

MGA -MASTER AGENT • [ECC_HOME]\*.log • [ECC_HOME]\*.rcfile • [ECC_HOME]\ctg.ini • [ECC_HOME]\master.ini • [ECC_HOME]\MGA6*\MGA.log • [ECC_HOME]\MGA6*\MGA_B*.log • [ECC_HOME]\MGA6*\MGA_Err.log • [ECC_HOME]\MGA6*\installs\*


• [ECC_HOME]\MGA6*\prereqs\* ESNAPI

• [ECC_HOME]\esnapi\data\* SERVER

• %ECC_INSTALL_ROOT%\ecc_inf\data\ecc_inf_hostname\data\server.trc • %ECC_INSTALL_ROOT%\ecc_inf\data\ecc_inf_hostname\data\server.trc.*.gz • %ECC_INSTALL_ROOT%\ecc_inf\data\ecc_inf_hostname\data\ecc_server.ini










Additional Files: Collect from the ESX Server host:

• /etc/vmware/hostd/config.xml file • /etc/resolv.conf file


• /etc/hosts file • /sbin/ifconfig > ifconfig.txt • /bin/netstat –an > netstat_an.txt • ps –ef > ps_ef.txt

Collect from the VMWare VirtualCenter host (if available): • %ALLUSERSPROFILE%\Application Data\VMWare\VMWare VirtualCenter\vpxd.cfg file • %SystemRoot%\system32\drivers\etc\hosts file • Ipconfig /all > ipconfig_all.txt • netstat –an > netstat_an.txt • winmsd /nfo winmsd.nfo • winmsd /report winmsd.txt

Collect information using the ControlCenter Console: • Collect an export of the Alerts table view:














Switch Discovery How do I troubleshoot discovering Fibre Channel switches in ControlCenter 6.0? What do I need to verify and collect for Fibre Channel switch discovery issues? Troubleshooting Steps

• Verify there is an active FCC Agent running • Verify there is at least one active Store.



This process assumes that the environment is a supported ControlCenter environment. Reference the EMC ControlCenter Support Matrix to verify the environment is fully supported.

McData:

1. Verify if it is a full discovery or a rediscovery of the switch that is failing. o If it is a rediscovery then go to McData step 13

2. If it is a full discovery, does a switch Icon show up in the ControlCenter Console? 3. If a switch icon does not show up, this is an indication of an SNMP issue.

o If switch icon does appear, go to McData step 9. 4. Verify that SNMP is enabled at the service processor. 5. Try and ping the switch. If you cannot ping the switch, have the site network personnel configure network

connectivity to the switch. 6. If you can ping the switch, do a walkmib (or use any other SNMP query tool) to the switch to verify SNMP

communication. 7. If the walkmib times out, then have the site network personnel verify that the correct TCP/IP ports are open.

o Refer to the EMC ControlCenter Planning and Installation Guide Volume 1 for ports. 8. If the walkmib returns correct information, collect the FCC agent, trace_x, trace_y, store, server, console.trc,

and export of the repository. 9. If the switch Icon does show up in the console this is an indication that the SNMP portion of the discovery is

working. The next step would be to verify that the FibreZone Bridge is functioning properly. 10. To verify the bridge is working properly request Customer Service dial into (or Webex into) the service

processor and check the FibreZone Bridge install log to make sure the proper version bridge is installed and that it is installed in non secure mode. Also check the log to make sure the Noblenet Port Mapper is installed.

o Default location for FibreZoneBridge install log: C:\Program Files\EMC\FibreZoneBridge\fz_bridge_install.log

11. Stop and restart the FibreZone Bridge service to reset it. 12. Check the master log in Connectrix Manager to verify that there are no errors being logged. If there are issues

with Connectrix Manager losing communication with the switches, loss of manageability errors (among others) will cause problems for ECC. If you see any errors logged have Customer Service Switch support personnel check the errors to verify that the switches are healthy. If there appears to be a problem with the switch, then collect logs from the switch. To collect the switch logs for a McData go to Connectrix Manager and run a data collection for every switch in the fabric.

13. If you are attempting a rediscovery, what is the error that is being logged? o If it is a third party library error, that is an indication of a problem with the communication between

ESNAPI and the switch. ESNAPI contains the switch vendor api’s. Edit the esnapi.cfg file to enable debug for McData (see emc139091 - McData / EMC ControlCenter Discovery Troubleshooting Checklist). After enabling ESNAPI debug, run the discovery again and when the error appears collect the egn.log, trace_x, trace_y logs.


14. To get information on what part of the discovery is failing, watch the discovery as it progresses in the discovery window of the console. For example, if it fails while importing the zoning then start troubleshooting the FibreZone Bridge because that is what ControlCenter communicates with to pull the zoning into ECC.

Cisco: Note: The Cisco discovery is done completely with SNMP.

1. Use the EMC ControlCenter Support Matrix to verify that the switch models and firmware are supported by ControlCenter.

2. An icon for the switch appearing in the console is an indication that SNMP is configured properly and there is at least some SNMP communication occurring.

o Collect the egn.logs, server.trc’s, store.trc’s console.trc’s from the time of the discovery and an export of the repository.

o Review the logs and reference Primus to help troubleshoot any errors found. 3. If the issue appears to be SNMP related, use telnet commands to verify that SNMP is configured properly on

the Cisco switch. 4. If SNMP is configured properly then use an SNMP mibwalk tool to try and walk the mib. 5. If you are unable to walk the mib, verify if there is a firewall. If there is a firewall, then refer to the install

documentation to verify the proper ports are configured. o Refer to the EMC ControlCenter Planning and Installation Guide Volume 1 for ports.

6. If there is no firewall or you have verified that the correct ports are open, engage the site network personnel to get a network trace while the discovery is running to determine why the SNMP requests to the switch are not working.

7. If the network trace shows the SNMP requests are not getting to the switch, then request the site network personnel rectify the network issue.

8. If the SNMP requests are getting to the switch, but the switch is not responding then collect a show tech-support details output from the switch(es). Have a Customer Service Switch support personnel review the output for errors.

9. If the SNMP requests and responses from the switch appear to be working correctly and the switch discovery still fails the problem is internal to ControlCenter. Collect the EGN.logs, Server.trc’s, Store.trc’s console.trc’s from the time of the discovery and an export of the repository.

Brocade: 1. If you are receiving an error during discovery search Primus to see if it is a known issue. 2. Use the EMC ControlCenter Support Matrix to verify the switch models and firmware are supported for the

version of ControlCenter. 3. If during the discovery you receive an SNMP error or if a switch icon does not appear in the console, check to

make sure that SNMP is configured properly on the Brocade switches. If a switch Icon does show up in the console go to Brocade step 9.

4. If SNMP is configured properly, then use walkmib (or some other SNMP tool) to try and walk the mib. 5. If SNMP is configured properly and you are able to walk the mib, then collect the FCC agent logs, server.trc’s,

store.trc’s, trace_x, and trace_y logs. 6. If SNMP is configured properly and you are not able to walk the mib, verify if there is a firewall or if there are

ports being blocked between the FCC agent and the switches. 1. Refer to the EMC ControlCenter Planning and Installation Guide Volume 1 for ports. 2. Try pinging the switches to confirm some level of network connectivity.

7. If there is no firewall or you have verified that the correct ports are open, engage the site network personnel to get a network trace while trying to walk the mib.

8. If the problem with SNMP communication doesn’t appear to be network related telnet to the switch and capture the output of a supportshow command and have Customer Service switch support personnel review the supportshow to try and determine if there is a problem with the switch.

9. If the switch icon shows up in the console it is an indication that the SNMP portion of the discovery is working. After the SNMP portion of the discovery is completed the Brocade API called “FAL” (Fabric Access Layer) takes over. 1. To troubleshoot the FAL check the esnapi trace_x trace_y logs for errors. 2. If there are no errors in the esnapi logs during the time of discovery, put the esnapi in debug mode for

Brocade.


3. Rename the trace_x and trace_y log files so that you can generate new clean files and try the discovery again.

4. Then check the trace_x and trace_y files for errors during the time of discovery. 5. If you see errors check Primus to see if they are known errors. 6. If the errors are not in Primus, then collect a supportshow from all the switches in the fabric to determine

if there is any problem on the switches. 7. Verify again that the switch models and firmware are supported. Escalate as necessary.

10. If the SNMP portion of the discovery and the FAL portion of the discovery seem to be working correctly then the problem is internal to ControlCenter. Collect the egn.logs, sever.trc, store.trc, console.trc and get an export of the repository.


LCW Subcategory: McData –Switch | Brocade –Switch | Cisco –Switch EGN -SWITCHES

• [ECC_HOME]\EGN6*\EGN.log • [ECC_HOME]\EGN6*\EGN_B*.log.gz • [ECC_HOME]\EGN6*\EGN.ini • [ECC_HOME]\EGN6*\EGN_Err.log • [ECC_HOME]\EGN6*\EGN_B*_Err.log • [ECC_HOME]\EGN6*\EGN_Discovery.log • [ECC_HOME]\EGN6*\EGN_Discovery.log.gz








TOMCAT


• %ECC_INSTALL_ROOT%\tomcat\logs\* REPOSITORY





Keep in mind in order for the Console.trc file to be created, the user must enable Console tracing when launching the console session. Also keep in mind that the console.trc file will “roll-over” and zip itself as soon as the EMC ControlCenter console is closed. The new zip file will be: console.trc.1.gz OS Commands: (From EMC ControlCenter Server host) None specified. Repository Exports: None specified. Additional Files:

Collect from Switch: • For Brocade switch SupportShow from each switch within a fabric. • For McDATA (Connectrix) follow process for data collection from Connectrix Manager. • For Cisco show tech-support from each switch within a fabric

Collect information using the ControlCenter Console: • Collect an export of the Alerts table view:





1. Open the ControlCenter Console 2. Select ECC Administration on the blue toolbar 3. Click the dropdown arrow next to ECC Administration, click Policies


4. In the Tree Panel on the left side of the Console, put a check next to the affected host in the Hosts folder. Make sure there are no other items checked.

5. In the Target Panel on the right you will see the Policies view. On the top of that view, click the Table button.








Oracle Database Discovery How to set up EMC ControlCenter to perform Oracle database discovery? How do I discover an Oracle database in ControlCenter 6.0? Background Database Agent for Oracle allows you to discover, monitor, and view configuration and allocation information for Oracle databases. A single agent supports all the database instances installed on the agent host (subject to scalability limitations) or any machines that can communicate with the agent host. This agent is available with Open Integration Components, the ControlCenter base package (discovery and alerts only). Optionally enter a StorageScope or ARM license key to enable full functionality. This agent should be installed on hosts that can access the Oracle databases to be monitored and managed.

• Do not install the Database Agent for Oracle on the same host as the ControlCenter Repository. • Do not deploy Database Agent for Oracle on hosts running 64-bit versions of AIX—the agent is supported only

for 32-bit architecture hosts. • Refer to the ControlCenter Performance and Scalability Guidelines for additional information.

Troubleshooting Steps • If you plan to monitor Oracle databases on local hosts:

o Deploy the Solutions Enabler client libraries, and ensure that Oracle or the Oracle Client is installed, on the local host where Database Agent for Oracle is to reside.

o Ensure that the SYMAPI Server component of Solutions Enabler is running on the local host. • Database Agent for Oracle grants only select privileges on some SYS views and tables to Oracle Agent User. Review

the access privileges in the sys8.sql file and object creation in the <ECC_HOME>/exec/EGO600/oraagent8.sql script. • Define and enable Database Agent for Oracle DCPs. Data collection policies enable the Database Agent for Oracle to

retrieve information about discovered Oracle database instances in specified intervals. You must define and enable the data collection policies yourself to begin collecting data with the agent.

o Collected data appears under the Database Instances folder in the tree panel of the main Console window. o Refer to the “Database Agent for Oracle data collection policies” topic in the ControlCenter Console Online

Help for details. • Verify there is at least one active Store.

o ECCService_Stor on W2k o ECCService_Store.exe on W2003 o MS Services and Task manager should not be used to make this determination. They may be unreliable.

Check to see if Store is active on Console or by verifying if store.trc file is active and being updated. • Verify the Oracle agent is installed and running. • Ensure the discovery settings in the Assisted Discovery dialog are filled out accurately and completely • Review Discovery progress in Discover->Review Progress pulldown, then select database. • Document approximate time of discovery.

o When the Review Discovery Results dialog box shows that an object has been detected, the object does not appear in the ControlCenter tree panel until an agent has executed the data collection policy for that object.

• Ensure that you can connect to the database with the provided username and password via sqlplus Optional Configuration Procedures This information is located in the ControlCenter Installation and Planning Guide Volume 1. Change the tablespace for Database Agent for Oracle (optional) By default, Database Agent for Oracle control data is stored in the USERS tablespace of each database that it monitors. Do the following to specify a different tablespace for that data:

1. Stop Database Agent for Oracle, using the ControlCenter Console. 2. Open the following file with a text editor:

<Install_Root>\exec\EGO.ini, where <Install_Root> is the ECC infrastructure install directory. 3. Update the following field to specify the new tablespace:

Tablespace to store ECC Management info = MY_TSPACE If the specified tablespace does not exist, the agent uses the SYSTEM tablespace.

4. Save and exit EGO.ini.


5. Restart Database Agent for Oracle. Manually run Database Agent for Oracle (optional) To manually run Database Agent for Oracle without providing sys password:

1. Locate ManualSetup.sql in <ECC_HOME>\exec\EGO600. Replace all occurrences of OUSER with the username Database Agent for Oracle is to use for database connections.

2. If you would like ManualSetup.sql to create the user account, uncomment the appropriate lines beginning with CREATE USER and set the password and DEFAULT TABLESPACE for the user in ManualSetup.sql. Alternatively, you could logon to the oracle instance and manually create the user with another tool and leave the CREATE USER lines in ManualSetup.sql commented out.

3. Using SQLPLUS, connect as user sys and run ManualSetup.sql. Ignore the error while executing the drop table ecc_fet. At this point there should be a number of views created in the schema of the new user you created.

4. Start Database Agent for Oracle, using the Agent Administration Wizard in the ControlCenter Console. 5. Select Discover>Other>Database>Oracle Agent and provide the database connection information for

Database Agent for Oracle. Provide a dummy SYS password (required). For Oracle Agent User Name, provide the database username and password you created in the first step of this procedure.


LCW Subcategory: Oracle –Database EGO -ORACLE

• [ECC_HOME]\EGO6*\EGO.log • [ECC_HOME]\EGO6*\EGO_B*.log.gz • [ECC_HOME]\EGO6*\EGO.ini • [ECC_HOME]\EGO6*\EGO_Err.log • [ECC_HOME]\EGO6*\EGO_B*_Err.log • [ECC_HOME]\EGO6*\EGO_Discovery.log • [ECC_HOME]\EGO6*\EGO_Discovery.log.gz













Additional Files: Collect information using ControlCenter Console:







Table button. 6. Click inside the table to make it the active pane



• Collect an export of the Agents table view: 1. Open the ControlCenter Console 2. Select ECC Administration on the blue toolbar 3. Click the dropdown arrow next to ECC Administration, click Agents 4. In the Tree Panel on the left side of the Console, put a check next to the folder name Hosts.





Oracle Database Proxy Discovery How to set up EMC ControlCenter 6.0 to perform an Oracle database discovery over proxy? How do I discover an Oracle database installed on a host without an Oracle Agent? EMC ControlCenter does not support running the Oracle Agent on AIX 5.2 or higher. How do I discover my Oracle database? Troubleshooting Steps Keep the following requirements in mind:

• Ensure the version of Oracle is one supported by ControlCenter. Review the appropriate EMC ControlCenter Support Matrix.

• Ensure a copy of the Oracle client is installed on the host with the Oracle Agent. If the client is not present, the discovery will fail.

• Verify the hostname of the database host is resolvable through DNS. Use "ping" and "nslookup" commands to test.

• Proxy Discovery cannot be performed using an Oracle Agent that is already being used to discover a local database. If the Oracle Agent is being used to discover a database on the same host as the agent, you will need to install the Oracle Agent onto another supported host that does not have an Oracle database. An Oracle Agent cannot do both Local and Proxy discoveries.

• The same version of Solutions Enabler is required on both the Oracle Agent host and the Oracle database host Odd discovery problems will occur if the Solutions Enabler versions differ.

• The SYMAPI Server must be running on the Oracle database host. Upon install, Solutions Enabler will ask if you want to run the SYMAPI Server.

• When installing Solutions Enabler, be sure to select the Oracle ‘SRM Oracle Database Components’. By default this component is not selected.

• Pushing Solutions Enabler via ControlCenter’s MRLK will not install the necessary binaries to support the proxy discovery of the database.

• If the database resides on a 64-bit system host, ensure that the 64-bit version of Solutions Enabler is installed. • You will need the SYS password for discovery. If you do not want to specify the SYS password during

discovery, you can use the following procedure: o To manually run Database Agent for Oracle without providing sys password:

1. Locate ManualSetup.sql in <ECC_HOME>\exec\EGO600. Replace all occurrences of OUSER with the username Database Agent for Oracle is to use for database connections.

2. If you would like ManualSetup.sql to create the user account, uncomment the appropriate lines beginning with CREATE USER and set the password and DEFAULT TABLESPACE for the user in ManualSetup.sql. Alternatively, you could logon to the oracle instance and manually create the user with another tool and leave the CREATE USER lines in ManualSetup.sql commented out.

3. Using SQLPLUS, connect as user sys and run ManualSetup.sql. Ignore the error while executing the drop table ecc_fet. At this point there should be a number of views created in the schema of the new user you created.

4. Start Database Agent for Oracle, using the Agent Administration Wizard in the ControlCenter Console.

5. Select Discover>Other>Database>Oracle Agent and provide the database connection information for Database Agent for Oracle. Provide a dummy SYS password (required). For Oracle Agent User Name, provide the database username and password you created in the first step of this procedure.

Find Hosts

1. Choose a supported host that will run the Database Agent for Oracle. See the appropriate EMC ControlCenter Support Matrix for more information. Make sure the host does not have an Oracle database. This host will be referred to as the "Oracle Agent host".

2. Note the name of the host running the Oracle database. This host will be referred to as the "Oracle database host".


Install Solutions Enabler 1. Download the latest supported version of Solutions Enabler for both hosts. For supported versions, see the

appropriate EMC ControlCenter Support Matrix. 2. Install Solutions Enabler on both the Oracle Agent host and the Oracle database host. Be sure to specify the

installation of the SYMAPI Server service on the Oracle database host. Also be sure to install ‘SRM Oracle Database Components’.

Install the Oracle Client

1. Locate a copy of the Oracle database client that is compatible with the version of Oracle you will be discovering. Usually, this means the version of the client should be equal to or higher than the version of the database.

2. Install the Oracle client software onto the Oracle Agent host. Install the Database Agent for Oracle

1. If not present already, install the EMC ControlCenter Master Agent onto the designated Oracle Agent host using the ControlCenter CDs. The EMC ControlCenter Planning and Installation Guide explains how to correctly install a Master Agent. Once the Agent is installed and the host appears in EMC ControlCenter, proceed to the next step.

2. In the EMC ControlCenter Console, right-click on the host, go to Agents -> Install. Use the wizard to install the Database Agent for Oracle. If the Database Agent for Oracle does not start, ensure you have met the requirements per the Planning and Installation Guide, as well as the Support Matrix.

Perform Proxy Discovery

1. In EMC ControlCenter Console, click ECC Administration in the blue bar at the top of the screen. 2. Above the blue bar, click Discover -> Assisted. 3. In the left-hand column, click Database. 4. On the right-hand side, click the Oracle Agent tab. 5. Fill in the form with information describing the Oracle database host. Be sure to fill in all required fields. 6. In the Send To section, use the drop-down to select the Oracle Agent that will be used for Proxy Discovery. 7. When finished, click the OK button. 8. Above the blue bar, click Discover -> Review Progress. 9. Click Database in the left-hand column. A list of all discoveries for databases is listed on the right-hand

side. Select the current discovery entry to see its status. When finished, the database will appear under the Databases folder in the Console's tree panel.




MGA -MASTER AGENT • [ECC_HOME]\*.log • [ECC_HOME]\*.rcfile • [ECC_HOME]\ctg.ini • [ECC_HOME]\master.ini • [ECC_HOME]\MGA6*\MGA.log


• [ECC_HOME]\MGA6*\MGA_B*.log • [ECC_HOME]\MGA6*\MGA_Err.log • [ECC_HOME]\MGA6*\installs\* • [ECC_HOME]\MGA6*\prereqs\*









OS Commands: (From EMC ControlCenter Server host) None specified. Repository Exports:




sure there are no other items checked.


5. In the Target Panel on the right you will see the Alerts view. On the top of that view, click the Table button.












Oracle Database Cluster Discovery How do I troubleshoot discovering an Oracle cluster in ControlCenter 6.0? What do I need to verify and collect for ControlCenter 6.0 Oracle cluster discovery issues? Troubleshooting Steps

1. Review emc153947, all requirements have been taken into account. (this is the proxy discovery – replace with 6.0 version)

2. Verify SYMAPISVR is installed on all Nodes. 3. Review Discovery progress in Discover->Review Progress pull down, then select database. 4. Document approximate time of discovery.



























• Collect an export of the Agents table view: 1. Open the ControlCenter Console 2. Select ECC Administration on the blue toolbar 3. Click the dropdown arrow next to ECC Administration, click Agents 4. In the Tree Panel on the left side of the Console, put a check next to the folder name Hosts.





Database Discovery using Common Mapping Agent (non-Oracle) How do I troubleshoot discovering Microsoft SQL, Sybase, Informix, and DB2 databases in ControlCenter 6.0? What do I need to verify and collect for ControlCenter 6.0 Microsoft SQL, Sybase, Informix, and DB2 discovery issues? Background Common Mapping Agent allows you to discover and monitor (but not manage) databases and host configuration information locally on a single host or on remote hosts (via proxy). When operating on one host, CMA can look for databases and file systems on remote hosts, so only one CMA is required in place of individual agents for the desired databases and host systems (which can improve host performance and reduce overhead). Data collected by this agent is used in the Topology, Properties, and Relationship views in the ControlCenter Console or Web Console. The following features, which are available with host agents, are not supported with Common Mapping Agent.

• Issuing active commands • Performing real-time explore • Sending alerts • Displaying host devices properties: Free size, tracks per cylinder, and sectors per track • Displaying the following for VERITAS Volume Manager: VM disk count, maximum free space, utility state, VM

Disks, and details about VERITAS sub-disks, plexes, etc. • Displaying the following for HP-UX: Allocation state for logical volumes • Windows Drive size • Windows: In the ControlCenter Console Relationship View, the parent and child relationship for PowerPath

cannot be displayed • Informix databases: The agent does not collect information about temporary tablespace data files (the Console

displays the temporary tablespace names, but no data files) Troubleshooting Steps

• The Common Mapping agent can be installed on a Windows or Solaris host. The agent can detect and monitor databases on HP-UX and AIX hosts via proxy. The agent can detect and monitor file systems on AIX, HP-UX, Linux, and Tru64 hosts via proxy, and report information about logical volumes, storage devices, partitions, and volume groups. The agent can report information about storage devices, partitions, host devices, and any volume serial number (VSN) associated with those host devices for BS2000 hosts (but not file system information).

• Verify user account has been specified for Common Mapping Agent. (Referred to as CMA or agent acronym EGZ).

• Do the following before you deploy Common Mapping Agent (CMA). Refer to EMC ControlCenter Support Matrix on Powerlink for supported software versions.

• Deploy EMC Solutions Enabler 6.3 or higher on the target host, the host where the agent will reside. o When you deploy Solutions Enabler through Control Center Media Repository (after installation of

Media Repository Load Kit), all required components are installed by default, and SYMAPI Server (which must be running for the agent to function properly) is started.

o If you deploy Solutions Enabler using a different method (necessary in a proxy setup when the Master Agent is not deployed on the target host), ensure SYMAPI Server is started. Select Yes, for the Install SRM Database Components during installation, and start SYMAPI server after installation.

o If Common Mapping Agent is to act as a proxy for remote hosts, Solutions Enabler is required on both the local and remote host(s).

• To monitor MS SQL, Sybase, Informix, or DB2 database instances, ensure the databases are running on the agent host (if CMA is installed locally) or on the proxy host (if CMA is acting as a proxy for one or more remote hosts).

• This agent will not discover a Sybase instance on a UNIX host unless you add the Sybase executable file to the PATH and the location of the Sybase libraries to LD_LIBRARY_PATH, using the following procedure: o While logged on to the agent host as root, permanently set the following environment variables:

export SYBASE=(SYBASE_HOME); PATH=$PATH:$SYBASE/ASE-12_5(or your base directory)/bin; export SYBASE_OCS=OCS-12_5;


export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$SYBASE/OCS-12_5 (or your OCS directory)/lib;

o Stop this agent, the Master Agent, and SYMAPI Server. Then restart those components. o Using the ControlCenter Console, rediscover the Sybase database(s).

Microsoft SQL guidelines

• User account for SQL server or Sybase can use ‘sa’ or create a unique user for the CMA. • To discover the default SQL Server instance on a host, enter 'default' instead of the host name. • To use Windows trusted user authentication rather than SQL Server authentication, enter 'trusted_user' as the

user. • User needs access to all of the databases. • SQL Server Authentication set as “SQL Server and Windows.” • SQL User Authentication set as “SQL Server Authentication” only. • All databases must be online and running. • Solutions Enabler 6.3 or higher is required for host that has CMA running. • Symapisrv must be running on Db host, if CMA agent is not on same host as DB host. SQL Server discovery

uses Common Mapping Agent talking to symapisrv running on the SQL machine. During installation of Solution Enabler, you need to install the database SRM for the respective database and SYMAPISRV and Solutions Mapping licenses on the remote hosts for cmd line access (where the SQL DB is located).

• Port of symapisrv match that of port specified in discovery window. • When CMA monitors two or more SQL Server instances on the same host, specify unique database

username/password combinations for each instance. Otherwise, the agent uses default database instance information for all SQL Server instances.

• Check the Data Collection Policy for the database. • Verify there is at least one active Store.



• Review Discovery progress in Discover->Review Progress pull down, then select database. • Document approximate time of discovery.

Best practices for the Common Mapping Agent Limit the number of managed objects per Common Mapping Agent When the number of managed objects exceeds the managed object counts, consider adding one or more Common Mapping Agents to sustain optimum performance. For example, one Common Mapping agent can manage up to a total count (100) before adding a second Common Mapping Agent. Discovery polling will be done in five hours during off-peak time. The Common Mapping Agent may coexist with other agents or Infrastructure components. The limitation is based on an agent host with two CPUs (700 MHz to 1.0 GHz) and 1 GB of memory. Refer to the ControlCenter Performance and Scalability Guidelines for additional information. Information to Collect Note: All logs must be time relevant to the problem event. In addition, Server and Store log file sets must contain all available trace files. Using the Log Collection Wizard please select the following category and subcategories which will collect the recommended log files indicated below. Deselect any log types or log files that are not applicable to your environment. For instructions on using the Log Collection Wizard see emc154217. LCW Category: Object Discovery and Monitoring

LCW Subcategory: Common Mapping Agent | Sybase –Database | SQL Server –Database | Informix –Database | DB2 –Database

EGZ -COMMON MAPPING • [ECC_HOME]\EGZ6*\EGZ.log • [ECC_HOME]\EGZ6*\EGZ_B*.log.gz


• [ECC_HOME]\EGZ6*\EGZ.ini • [ECC_HOME]\EGZ6*\EGZ_Err.log • [ECC_HOME]\EGZ6*\EGZ_B*_Err.log • [ECC_HOME]\EGZ6*\EGZ_Discovery.log • [ECC_HOME]\EGZ6*\EGZ_Discovery.log.gz





























File system Discovery How do I troubleshoot discovering remote file systems (using the Common Mapping Agent) in ControlCenter 6.0? What do I need to verify and collect for ControlCenter 6.0 remote file system (using the Common Mapping Agent) discovery issues? Troubleshooting Steps

1. Verify Symapisrv is running on remote host using proxy. 2. Verify the symapisrv port matches that of port specified in discovery window using netstat. 3. Verify there is at least one active Store.

1. ECCService_Stor on W2k 2. ECCService_Store.exe on W2003 3. MS Services and Task manager should not be used to make this determination. They may be


4. Review Discovery progress in Discover->Review Progress pull down, then select host. 5. Check the data collection policy for proxy host discovery, if applicable. 6. Document approximate time of discovery. 7. Collect the $SYBASE_HOME/ASE-yourversion directory from the Common Mapping agent host.


LCW Subcategory: Common Mapping Agent EGZ -COMMON MAPPING

• [ECC_HOME]\EGZ6*\EGZ.log • [ECC_HOME]\EGZ6*\EGZ_B*.log.gz • [ECC_HOME]\EGZ6*\EGZ.ini • [ECC_HOME]\EGZ6*\EGZ_Err.log • [ECC_HOME]\EGZ6*\EGZ_B*_Err.log • [ECC_HOME]\EGZ6*\EGZ_Discovery.log • [ECC_HOME]\EGZ6*\EGZ_Discovery.log.gz


SYMAPI_UNIX • /var/symapi/log/* • SYMAPI_WINDOWS • %ProgramFiles%\EMC\SYMAPI\log\symapi*.log • %ProgramFiles%\EMC\SYMCLI\*.log

SERVER • %ECC_INSTALL_ROOT%\ecc_inf\data\ecc_inf_hostname\data\server.trc • %ECC_INSTALL_ROOT%\ecc_inf\data\ecc_inf_hostname\data\server.trc.*.gz


• %ECC_INSTALL_ROOT%\ecc_inf\data\ecc_inf_hostname\data\ecc_server.ini STORE

• %ECC_INSTALL_ROOT%\ecc_inf\data\ecc_inf_hostname\data\store.trc • %ECC_INSTALL_ROOT%\ecc_inf\data\ecc_inf_hostname\data\store.trc.*.gz • %ECC_INSTALL_ROOT%\ecc_inf\data\ecc_inf_hostname\data\ecc_store.ini









Additional Files:

• From the Common Mapping agent host, collect the $SYBASE_HOME/ASE-yourversion directory. • From the ControlCenter Console collect an export of the Alerts table view:

1. Open the ControlCenter Console 2. Select ECC Administration on the blue toolbar 3. Click the dropdown arrow next to ECC Administration, click Alerts 4. In the Tree Panel on the left side of the Console, put a check next to the Hosts folder. Make sure


button. 6. Click inside the table to make it the active pane



• From the ControlCenter Console collect an export of the Policies table view: 1. Open the ControlCenter Console 2. Select ECC Administration on the blue toolbar 3. Click the dropdown arrow next to ECC Administration, click Policies 4. In the Tree Panel on the left side of the Console, put a check next to the affected host in the Hosts









Section III Runtime Issues

Data Collection Policies How do I troubleshoot Data Collection Policies (DCPs) in ControlCenter 6.0? What do I need to verify and collect for ControlCenter 6.0 Data Collection Policy (DCP) issues? Troubleshooting Steps

1. Record which DCPs are enabled for agents that are installed and running on the subject host (including WLA data collections).

2. If the issue involves Workload Analyzer (WLA), record how many objects, and the size of the objects involved in the WLA collections. See the WLA DCPs subsection of the Runtime Issues section of the ControlCenter 6.0 Troubleshooting Guide (emc153987) for further WLA problem diagnosis.

3. Check for gatekeepers. At least one gatekeeper is required for hosts where there is a Symmetrix Storage Agent installed.

4. Check semaphore settings on UNIX hosts. The Solutions Enabler Installation Guide provides semaphore information for UNIX hosts.

Information to Collect Agent files are located [ECC_HOME]\xxx600

• xxx represents the agent’s acronym. • Example: Symmetrix Agent C:\ECC\exec\EGS600\* • The ctg.ini file will correlate the agent’s acronym to the full agent name.

Note: All logs must be time relevant to the problem event. In addition, Server and Store log file sets must contain all available trace files. Using the Log Collection Wizard please select the following category and subcategories which will collect the recommended log files indicated below. Deselect any log types or log files that are not applicable to your environment. For instructions on using the Log Collection Wizard see emc154217. LCW Category: Object Discovery and Monitoring

LCW Subcategory: Select all Subcategories that correspond to the agent(s) concerned. For example, if the issue is related to the CLARiiON agent, select CLARiiON –Array.

AGENT SPECIFIC LOGS

• [ECC_HOME]/xxx6*/xxx.log • [ECC_HOME]/xxx6*/xxx_B*.log.gz • [ECC_HOME]/xxx6*/xxx.ini • [ECC_HOME]/xxx6*/xxx_Err.log • [ECC_HOME]/xxx6*/xxx_B*_Err.log • [ECC_HOME]/xxx6*/xxx_Discovery.log • [ECC_HOME]/xxx6*/xxx_Discovery.log.gz

MGA -MASTER AGENT • [ECC_HOME]\*.log • [ECC_HOME]\*.rcfile • [ECC_HOME]\ctg.ini • [ECC_HOME]\master.ini • [ECC_HOME]\MGA6*\MGA.log • [ECC_HOME]\MGA6*\MGA_B*.log • [ECC_HOME]\MGA6*\MGA_Err.log


• [ECC_HOME]\MGA6*\installs\* • [ECC_HOME]\MGA6*\prereqs\*














OS Commands (from EMC ControlCenter Server host): None specified. Repository Exports:


Additional Files:

• From all UNIX hosts involved in the issue, collect an EMCGrab. • From all Windows hosts involved in the issue, run:

o winmsd /s C:\tmp\<hostname>.nfo • From the ControlCenter Console collect an export of the Alerts table view:














Alerts How do I troubleshoot alerts in ControlCenter 6.0? What do I need to verify and collect for ControlCenter 6.0 alerting issues? Troubleshooting StepsThe following are suggestions for identifying why expected alerts are not occurring. Actions in response to alert messages from the environment depend on the alert message itself, and therefore are not discussed here.

• See online help “Viewing and setting alerts and autofixes” for alert administration help. • See online help “Alert Descriptions” for information about the meaning of the alerts. • Refer to online help “Alert Responses” for alert response overview. • Verify the version of any agent prerequisite software is at least minimum version required for running version of

ControlCenter. o Reference the ControlCenter Support Matrix for information regarding agent prerequisite software

• Verify the version of the distributed agent responsible for monitoring the alert condition is consistent with the version of the infrastructure.

• Verify the alert(s) in question are enabled for the desired object(s). o This can be verified from the Alert Definition -> edit. The alert is enabled when checked on the top-

right corner, or when the alert definition has a red bell icon. The Apply To tab will display the objects the alert will report against.

• If necessary, add or remove the desired object(s) from the Alert Definition. • If applicable, verify the Alert Management Policy is created and applied to the desired alert(s).

o This can be verified from the Alert Definition -> edit -> Actions Tab • Verify if the alert in question has ever been reported by viewing Alert History. • If alerts are not showing on Console for McDATA switches, verify if DSM Connect is being used to house the

Switch Application Software. Then verify the number of NIC cards installed on the DSM Connect workstation. DSM Connect has been known to accept incoming traffic on one NIC card, but try to respond from the other. This has been known to prevent McDATA switch alerts from being processed on Console.

• If the alert is found in Alert History, note the “Last Modified” date and verify the alert state by double clicking the alert.

Possible Alert states:

• New • Signed • Acknowledged • Cleared






AGENT SPECIFIC LOGS • [ECC_HOME]/xxx6*/xxx.log • [ECC_HOME]/xxx6*/xxx_B*.log.gz • [ECC_HOME]/xxx6*/xxx.ini • [ECC_HOME]/xxx6*/xxx_Err.log • [ECC_HOME]/xxx6*/xxx_B*_Err.log • [ECC_HOME]/xxx6*/xxx_Discovery.log • [ECC_HOME]/xxx6*/xxx_Discovery.log.gz










ORACLE







Additional Files:











1. Open the ControlCenter Console 2. Select ECC Administration on the blue toolbar 3. Click the dropdown arrow next to ECC Administration, click Agents


4. In the Tree Panel on the left side of the Console, put a check next to the folder name Hosts. Make sure there are no other items checked.

5. In the Target Panel on the right you will see the Agents – Relationships view. On the top of that view, click the Table button.




Windows Based Authentication How do I troubleshoot Windows authentication in ControlCenter 6.0? What do I need to verify and collect for ControlCenter 6.0 Windows authentication issues? Troubleshooting Steps

• For setting up user and group accounts refer to online help “Administering ControlCenter Users.” • Authentication failures may occur for reasons other than invalid usernames or passwords entered by the user.

The reasons differ for each authentication methodology used. Regardless of which methodology is being used, authentication failures are written to the server.trc file.

• Windows-Based Authentication Failures o When a ControlCenter user cannot log on and the username and password are known to be good,

there may be a Windows system configuration error. The server.trc file from the time of the authentication failure during logon will contain an entry consisting of the text “User authentication failure for username <errorcode>”. The error code will provide a clue to what has happened.

• There have been two specific failures for Window authentication observed. o A failure of the ControlCenter server connecting to the active directory. This type of problem will also

prevent a ControlCenter administrative user from adding new domain users to ControlCenter. It may also prevent a user from logging on.

o The ControlCenter server host is not properly configured as a member of the domain in which it is suppose to reside. This will usually result in an error code of 178x associated with the authentication failure. The local system administrator will need to be engaged to correct this condition.

• Verify the ControlCenter user account exists. ControlCenter user accounts may be local or domain usernames or they may be LDAP usernames if the customer has selected LDAP user authentication.

• Document clearly what authorization rules apply to each ControlCenter user and user group. • Document each authorization rule. This may be done by selecting a properties view and checking the check box

to the left of the Authorization Rules folder. The full set of authorization rules will be displayed. Note: To do this, you must be logged on to ControlCenter with an administrative user account. (Actions outside the user / user group authorization rules are "available" to users, but upon command execution the authorization limitations will be enforced and reported.)

• Using a Console logged on with a ControlCenter administrative account, such as eccadmin, select a properties view and check the checkbox next to the ECC Users Groups folder. Document which users are members of each user group.

• Record and capture what actions have been granted for the user or user group. Information to Collect Note: All logs must be time relevant to the problem event. In addition, Server and Store log file sets must contain all available trace files. Using the Log Collection Wizard please select the following category and subcategories which will collect the recommended log files indicated below. Deselect any log types or log files that are not applicable to your environment. For instructions on using the Log Collection Wizard see emc154217. LCW Category: ControlCenter Components

LCW Subcategory: Infrastructure Components and Consoles SERVER












Additional Files:

• From the ControlCenter console collect a screen capture of the properties view for the ECC Users Groups. • From the ControlCenter Console collect an export of the Alerts table view:






LDAP Authentication How do I troubleshoot LDAP Authentication in ControlCenter 6.0? What do I need to verify and collect for ControlCenter 6.0 LDAP Authentication issues? Setup Procedure When LDAP authentication is being used, sudden authentication (Console logon) failures may be due to the expiration of an LDAP server’s X.509 digital certificate. Engage the LDAP administrator for more information. Also in such a case, search the server.trc files for the text entry “LDAP”. The expiration of a digital certificate will result in a “connect error” that occurs when a ControlCenter user attempts to log on to the Console. Determining if the X.509 certificates have expired ControlCenter user logon may continue to work if you have a backup LDAP server with its own X.509 certificate. If you find LDAP connect errors in the Server trace files, use the following procedure to determine if one or more of the X.509 certificates have expired.

1. In the %ECC_Install_ROOT%\ecc_inf\data\<host_name>\data directory find two files: ECCkeystore ssl.properties

2. Read the ssl.properties file for the name of the keystore file (if it is not ECCkeystore) and the password for the keystore file.

3. Run the following keytool command (found in %ECC_Install_ROOT%\tools\jre\nt\1.4.2\bin) to list the certificates in the keystore file: keytool –list –v -keystore ECCkeystore –storepass <password>

4. Read the “Valid from:” entry for each certificate which will list the first and last dates for which the certificate is valid. Note the “alias name” entry for any expired certificate.

5. Obtain a new certificate file from the LDAP administrator for each certificate that has expired. (It is possible that a single certificate signed by a third-party certificate authority has replaced more than more certificate.) Follow the LDAP or security administrator’s advice if unsure.

6. Remove each expired certificate by alias name as follows: keytool –delete –v –alias cert-alias –keystore ECCkeystore –storepass <password>

7. Add the new certificate file (individually if there is more than one) as follows: keytool –import –v –alias cert-alias –file cert-file –keystore ECCkeystore –storepass <password> Note: The keytool –help command provides a complete list of options and syntax for keytool. Stopping and restarting ECC Server should not be required for these changes to take effect. Verify the ControlCenter user account exists. ControlCenter user accounts may be local or domain usernames or they may be LDAP usernames if the customer has selected LDAP user authentication.


1. Document clearly what authorization rules apply to each ControlCenter user and user group. 2. Document each authorization rule. This may be done by selecting a properties view and checking the check box

to the left of the Authorization Rules folder. The full set of authorization rules will be displayed. Note: To do this, you must be logged on to ControlCenter with an administrative user account. (Actions outside the user / user group authorization rules are "available" to users, but upon command execution the authorization limitations will be enforced and reported.)

3. Using a Console logged on with a ControlCenter administrative account, such as eccadmin, select a properties view and check the checkbox next to the ECC Users Groups folder. Document which users are members of each user group.

4. Record and capture what actions have been granted for the user or user group. Also See:


Article emc107004: Why are ControlCenter users unable to use their new passwords after they have been changed on LDAP? Information to Collect Note: All logs must be time relevant to the problem event. In addition, Server and Store log file sets must contain all available trace files. Using the Log Collection Wizard please select the following category and subcategories which will collect the recommended log files indicated below. Deselect any log types or log files that are not applicable to your environment. For instructions on using the Log Collection Wizard see emc154217. LCW Category: ControlCenter Components










Keep in mind in order for the Console.trc file to be created, the user must enable Console tracing when launching the console session. Also keep in mind that the console.trc file will “roll-over” and zip itself as soon as the EMC ControlCenter console is closed. The new zip file will be: console.trc.1.gz OS Commands (from EMC ControlCenter Server host): None specified.


Repository Exports:


Additional Files:

• From the ControlCenter console collect a screen capture of the properties view for the ECC Users Groups. • From the ControlCenter Console collect an export of the Alerts table view:






Managed Object Deletion How do I troubleshoot deleting managed objects in ControlCenter 6.0? What do I need to verify and collect for ControlCenter 6.0 managed object deletion issues? Troubleshooting Steps

• Collect a screenshot of the error. • If an error message contains a “details” button, bring up this detailed message and take a screenshot. • Be sure all Agents that are responsible for the Managed Object you are attempting to delete are stopped. • Log out of the console and log back in to verify that the object was deleted. • Please be aware that during times of high activity in the Infrastructure components (repository, server, store),

there may be a delay in deleting a managed object. Check CPU and memory usage by these processes for abnormally high levels.

• Part of the process of deleting a Managed Object requires that all associated entries in the database are removed via an internal SQL statement. If, for any reason, a database-related error occurs during a SQL transaction, Oracle will automatically “roll-back” the last transaction to restore the condition of the tables prior to the error. This is done to maintain database consistency and is the default behavior of Oracle. This applies to Managed Object discovery as well as deletion. To determine if an error occurred during the SQL transaction the Server and Store traces from the same time-frame of the deletion will be required.

Information to Collect Note: All logs must be time relevant to the problem event. In addition, Server and Store log file sets must contain all available trace files. Using the Log Collection Wizard please select the following category and subcategories which will collect the recommended log files indicated below. Deselect any log types or log files that are not applicable to your environment. For instructions on using the Log Collection Wizard see emc154217. LCW Category: ControlCenter Components







ORACLE • %ProgramFiles%\Oracle\Inventory\logs\*


• %ProgramFiles%\Oracle\Inventory\logs\results\db\* CONSOLE_WINDOWS





Additional Files:

• From the ControlCenter console collect a screen capture of the error message. • From the ControlCenter Console collect an export of the Alerts table view:











view, click the Table button. 6. Click inside the table to make it the active pane




Managed Object Exploring How do I troubleshoot exploring managed objects in ControlCenter 6.0? What do I need to verify and collect for ControlCenter 6.0 managed object exploration issues? Troubleshooting Steps

1. Verify explore operation has the appropriate license(s) applied. 2. Verify the agents involving explore operation are active via the agents table view. 3. Document approximate time explore was being run. 4. Collect appropriate agent logs based on explore operation. (i.e., exploring Solaris host file systems, Collect MSR

agent logs.) 5. Compare native command output (example df –k) with the ControlCenter view.

o Capture screen shot of the issue on the Console o Redirect the native command output to a file





AGENT SPECIFIC LOGS




SYMAPI_WINDOWS














Repository Exports: • ControlCenter Repository

o Use existing Repository dump file Additional Files:

• From the Console capture a screen shot of the problem. • From the hosts involved in the issue, redirect the native command output to a file. • From all UNIX hosts involved in the issue, collect an EMCGrab. • From all Windows hosts involved in the issue, run:















Real-Time Performance Data Retrieval How do I troubleshoot using ControlCenter 6.0 Real-Time Performance data retrieval? What do I verify and collect for ControlCenter 6.0 Real-Time Performance data retrieval problems?? Troubleshooting Steps

• Verify Performance Statistics DCP is configured properly and is enabled. • Verify Symmetrix Storage Agent is active for specific Symmetrix MO. (Symmetrix data) • Verify host agent is active for specific host. (Host data.) • Verify for host data that the host has a relationship to a Symmetrix. (Host real-time performance is reported

only as it relates to Symmetrix devices.) • Verify Fibre Channel Connectivity agent is active. (Switch data.) • Verify %ECC_Install_ROOT%\exec\XXX600\gpdata directory has updated files. XXX represents the agent’s

acronym. • Collect screenshot of problem. • Document approximate time data retrieval was attempted.





AGENT SPECIFIC LOGS



SYMAPI_UNIX















Repository Exports:


Additional Files:

• Collect screenshot of problem with Real-Time data. • Collect an export of the Alerts table view:














Symmetrix Optimizer How do I troubleshoot using Symmetrix Optimizer in ControlCenter 6.0? What do I need to verify and collect for ControlCenter 6.0 Symmetrix Optimizer issues? Background SymmOptimizer communicates to Symmetrix Storage Agent via the agent PID (Agent host process ID). If the Symmetrix Agent stops and is restarted via the autostart parameter setting (causing the agent to be operating under a new PID) Optimizer and the Agent are no longer talking to each other. Troubleshooting Steps

• Use the agents view to determine which host is running the Primary Symmetrix agent for that Serial number. • Contact Customer Support to access the Symmetrix Service processor:

o Verify if this condition exists by reviewing the symmoptm.log on the Symmetrix service processor. o Verify Symmwin is running on Symmetrix service processor. o Verify Optimizer is running on Symmetrix service processor. o Verify the version of Optimizer running on the Symmetrix service processor. o Verify if a Symmwin script was running at the time of the Optimizer failure.

This can be accomplished by looking through the symmoptm.log for an error string noting the time of the failure, and then compare with the symmwin.log file.

o Note the name and step of the script running during the failure. If failure during swapvols script and DRVs are still established to STDs, do NOT restart Optimizer.

o Note release of code, and engage a Customer Service Product Support Enginer to assist with recovery. If DRVs are not established to STD, restart Optimizer application and monitor for failed script or application to complete successfully.

o On the Symmetrix service processor: if Symmwin was not running, and it was not shut down intentionally:

Check the Microsoft Event log for Symmwin application failure. Check for unexpected SP reboot (0476 or 0497 Symptom Codes). This may indicate an SP

replacement is required. Engage PSE for further analysis. • If the problem involves manipulation of Optimizer from the Console GUI, include the Console trace files in

addition to other Optimizer files.

Information to Collect Note: All logs must be time relevant to the problem event. In addition, Server and Store log file sets must contain all available trace files. Using the Log Collection Wizard please select the following category and subcategories which will collect the recommended log files indicated below. Deselect any log types or log files that are not applicable to your environment. For instructions on using the Log Collection Wizard see emc154217. LCW Category: Object Management

LCW Subcategory: Symmetrix Management EGS -SYMMETRIX


MGA -MASTER AGENT • [ECC_HOME]\*.log • [ECC_HOME]\*.rcfile


• [ECC_HOME]\ctg.ini • [ECC_HOME]\master.ini • [ECC_HOME]\MGA6*\MGA.log • [ECC_HOME]\MGA6*\MGA_B*.log • [ECC_HOME]\MGA6*\MGA_Err.log • [ECC_HOME]\MGA6*\installs\* • [ECC_HOME]\MGA6*\prereqs\*




STORE • %ECC_INSTALL_ROOT%\ecc_inf\data\ecc_inf_hostname\data\store.trc • %ECC_INSTALL_ROOT%\ecc_inf\data\ecc_inf_hostname\data\store.trc.*.gz • %ECC_INSTALL_ROOT%\ecc_inf\data\ecc_inf_hostname\data\ecc_store.ini • ECCAPISERVER • %ECC_INSTALL_ROOT%\ECCAPIServer\data\apiserver.trc • %ECC_INSTALL_ROOT%\ECCAPIServer\data\apiserver.trc.*.gz • %ECC_INSTALL_ROOT%\ECCAPIServer\data\ecc_apiserver.ini








Repository Exports: • ControlCenter Repository


• Contact EMC Customer Service to collect from Symmetrix Service Processor (Symm4 / Symm5 - 67 and 68) o C:\EMC\<SerialNumber>\Optimizer\logs\optdbg.log o C:\symmwin\logs\logall.log o C:\symmwin\logs\logall0n.log

• Contact EMC Customer Service to collect from Symmetrix Service Processor (Symm6 / RMS - 69 code) o C:\EMC\<SerialNumber>\Optimizer\logs\optdbg.log o C:\EMC\<SerialNumber>\symmwin\logs\continuous\logall_dateXXXX_timeYYYYY.log

• Contact EMC Customer Service to collect from Symmetrix Service Processor (Symm7) o O:\emc\<SN>\Optimzer\logs\optdbg.log


Symmetrix Configuration Management – Configuration Changes How do I troubleshoot using Symmetrix configuration management operations in ControlCenter 6.0? What do I need to verify and collect for ControlCenter 6.0 Symmetrix configuration management issues? Troubleshooting Steps Document the following for all ECC configuration operations:

• Serial numbers of the Symmetrix arrays involved in the configuration change • Microcode versions of the Symmetrix arrays involved in the configuration change • Approximate time of the failed operation • Primary Symmetrix Agent host name and IP Address

o Use the agents view to determine which host is running as Primary Symmetrix agent for that serial number

• Version of ControlCenter • Version of SYMAPI/Solutions Enabler

o Be sure to verify the version on the host where the primary Symmetrix agent was running at the time of failure

• Specific Symmetrix devices involved in the configuration change • Specific directors and ports involved in the configuration change • Exact configuration change (symmconfigure) operation being performed at the time of failure

o For example: map device, form meta, set port flags, etc. • Specific error message returned

o Collect a screen capture of error message • At which point did the failure occur?

o Prepare, Preview, or Commit • Verify if any configuration locks are remaining on the Symmetrix

o symcfg list -lockn all -sid xxx o Record the lock number of any locks that are enabled o IMPORTANT: Be sure to only release stale configuration locks (lock 15 or F)

• Dial in and determine if a Symmwin script was running at the time of failure • Dial in and verify if the configuration change completed internally to the Symmetrix • Run symdev list | more from Primary Symmetrix Agent host to verify if the change is reflected in the symapi

database o If the configuration change is visible on the Symmetrix but not in the symapi database, run a symcfg

discover and check the database again. • Document specifically what change is not reflected on the Console and provide supporting screens shots • Provide a timeline for how long the commit took to complete and how long until the DB refresh took to

complete Information to Collect Note: All logs must be time relevant to the problem event. In addition, Server and Store log file sets must contain all available trace files. Using the Log Collection Wizard please select the following category and subcategories which will collect the recommended log files indicated below. Deselect any log types or log files that are not applicable to your environment. For instructions on using the Log Collection Wizard see emc154217. LCW Category: Object Management



















Additional Files:

• From the Symmetrix service processor collect: o For Symmetrix running 5x69/5x70/5x71 code: C:\EMC\symm s/n \SYMMWIN\logs\silconfig.log o For Symmetrix running 5x68 code and below: C:\SYMMWIN\LOGS\silconfig.log


o Open the ControlCenter Console o Select ECC Administration on the blue toolbar o Click the dropdown arrow next to ECC Administration, click Alerts o In the Tree Panel on the left side of the Console, put a check next to the Hosts folder. Make sure

there are no other items checked. o In the Target Panel on the right you will see the Alerts view. On the top of that view, click the Table

button. o Click inside the table to make it the active pane o Go to the top left hand corner of the console screen and select File > Export > Browse and save


o Open the ControlCenter Console o Select ECC Administration on the blue toolbar o Click the dropdown arrow next to ECC Administration, click Policies o In the Tree Panel on the left side of the Console, put a check next to the affected host in the Hosts

folder. Make sure there are no other items checked. o In the Target Panel on the right you will see the Policies view. On the top of that view, click the Table



o Open the ControlCenter Console o Select ECC Administration on the blue toolbar o Click the dropdown arrow next to ECC Administration, click Agents o In the Tree Panel on the left side of the Console, put a check next to the folder name Hosts. Make

sure there are no other items checked. o In the Target Panel on the right you will see the Agents – Relationships view. On the top of that

view, click the Table button. o Click inside the table to make it the active pane o Go to the top left hand corner of the console screen and select File > Export > Browse and save



Symmetrix TimeFinder - BCVs How do I troubleshoot using Symmetrix TimeFinder operations in ControlCenter 6.0? What do I need to verify and collect for ControlCenter 6.0 Symmetrix TimeFinder issues? Troubleshooting Steps Document the following for all ECC TimeFinder operations:

• Serial number of the Symmetrix array involved in the TimeFinder operation • Microcode version of the Symmetrix array involved in the TimeFinder operation • Approximate time of the failed operation • Primary Symmetrix Agent host name and IP Address




• Specific Symmetrix Standard and BCV devices involved in the operation • Exact TimeFinder operation being performed at the time of failure

o For example: establish, split, restore, etc. • Specific error message returned

o Collect a screen capture of error message • Dial in and verify if the operation completed internally to the Symmetrix • Run symmir list | more from Primary Symmetrix Agent host to verify if the change is reflected in the symapi

database o The TimeFinder symcli license must be active to run symmir commands. The ControlCenter licenses

do not enable symcli commands. o If the change is visible on the Symmetrix but not in the symapi database, run a symcfg discover and

check the database again. • Document specifically what change is not reflected on the Console and provide supporting screens shots • Provide a timeline for how long the operation took to complete and how long until the DB refresh took to

complete Information to Collect Note: All logs must be time relevant to the problem event. In addition, Server and Store log file sets must contain all available trace files. Using the Log Collection Wizard please select the following category and subcategories which will collect the recommended log files indicated below. Deselect any log types or log files that are not applicable to your environment. For instructions on using the Log Collection Wizard see emc154217. LCW Category: Object Management


















Additional Files:















Symmetrix TimeFinder - Clone How do I troubleshoot using Symmetrix Clone operations in ControlCenter 6.0? What do I need to verify and collect for ControlCenter 6.0 Symmetrix Clone issues? Background TimeFinder/Clone functionality is controlled via copy sessions, which pair the source and target devices. Sessions are maintained on the Symmetrix array and can be queried to verify the current state of the device pairs. A copy session must first be created to define and set up the clone devices. The session is then activated, enabling the target device to be accessed by its host. When the information is no longer needed, the session can be terminated. Because various other ongoing operations can conflict with your clone session, certain rules must be considered. The availability of some clone copy operations depends on the current state of SRDF and BCV pairs. The following rules apply to certain BCV pair states:

• If the source or target of a symclone create or activate operation is a BCV, the BCV pair state must be split. The split must be totally complete before the operation is allowed.

• A TimeFinder standard device cannot be created or activated in a clone session if the BCV pair state is SplitBfrRest or RestInProg.

• The symclone terminate command is allowed for all TimeFinder BCV pair states. For the TimeFinder pair states that rule TimeFinder/Clone copy sessions, a description of each BCV pair state, or specific information regarding possible SRDF pair state conflicts, refer to the Solutions Enabler Symmetrix TimeFinder Family CLI Product Guide available on Powerlink. Troubleshooting Steps

• Use the agents view to determine which host is running the Primary Symmetrix agent for that Serial number. • Serial number of the Symmetrix array involved in the Clone operation • Microcode version of the Symmetrix array involved in the Clone operation • Approximate time of the failed operation • Primary Symmetrix Agent host name and IP Address




• Specific Symmetrix Standard and Clone devices involved in the operation • Exact Clone operation being performed at the time of failure

o For example: create, activate, terminate • Specific error message returned

o Collect a screen capture of error message • Dial in and determine if a Symmwin script was running at the time of failure. • Dial in and verify if the operation completed internally to the Symmetrix • Use the symclone command from the Primary Symmetrix Agent host to verify if the change is reflected in the

symapi database. Reference the EMC Solutions Enabler Symmetrix CLI Command Reference guide for information regarding the appropriate options and arguments needed to display the desired information.

o The Solutions Enabler feature must be licensed on the host to run symcli commands. ControlCenter licenses do not activate the command line.

o The Solutions Enabler Symmetrix CLI Command Reference Guide provides details on available commands and their functions.

o If the change is visible on the Symmetrix but not in the symapi database, run a symcfg discover and check the database again.

• Document specifically what change is not reflected on the Console and provide supporting screens shots • Provide a timeline for how long the operation took to complete and how long until the DB refresh took to

complete

Information to Collect Note: All logs must be time relevant to the problem event. In addition, Server and Store log file sets must contain all available trace files.


Using the Log Collection Wizard please select the following category and subcategories which will collect the recommended log files indicated below. Deselect any log types or log files that are not applicable to your environment. For instructions on using the Log Collection Wizard see emc154217. LCW Category: Object Management











CONSOLE_WINDOWS






Additional Files:















Symmetrix TimeFinder - SNAP How do I troubleshoot using Symmetrix SNAP operations in ControlCenter 6.0? What do I need to verify and collect for ControlCenter 6.0 Symmetrix SNAP issues? Background Since Enginuity Version 5669, TimeFinder/Snap operations allow you to create and manage virtual copy sessions between a source device and multiple virtual (VDEV) target devices. The source device can be either a standard device or a BCV device. The target device(s) must be virtual device(s). A virtual device is a host-accessible device containing track-level location information (pointers) that indicate where the copy session data is located in the physical storage. Virtual devices consume minimal physical disk storage, as they store only the address pointers to the data stored on the source device or a pool of SAVE devices. Copies of striped or concatenated meta devices can also be created as long as the source and target virtual meta devices are completely identical in stripe count, stripe size, and capacity. Once the copy session has been activated, the device copy can be accessed via the virtual target device instantly. Detailed information is available in the EMC Solutions Enabler TimeFinder Family CLI Product Guide on Powerlink. Troubleshooting Steps

• Use the agents view to determine which host is running the Primary Symmetrix agent for that Serial number. • Serial number of the Symmetrix array involved in the SNAP operation • Microcode version of the Symmetrix array involved in the SNAP operation • Approximate time of the failed operation • Primary Symmetrix Agent host name and IP Address




• Specific Symmetrix Standard, VDEV, and Save devices involved in the operation • Exact SNAP operation being performed at the time of failure

o For example: create, activate, terminate • Specific error message returned

o Collect a screen capture of error message • Dial in and verify if the operation completed internally to the Symmetrix • Use the symsnap command from the Primary Symmetrix Agent host to verify if the change is reflected in the

symapi database. Reference the EMC Solutions Enabler Symmetrix CLI Command Reference guide for information regarding the appropriate options and arguments needed to display the desired information.

o If the change is visible on the Symmetrix but not in the symapi database, run a symcfg discover and check the database again.

• Document specifically what change is not reflected on the Console and provide supporting screens shots • Provide a timeline for how long the operation took to complete and how long until the DB refresh took to

complete



• [ECC_HOME]\EGS6*\EGS.log • [ECC_HOME]\EGS6*\EGS_B*.log.gz


• [ECC_HOME]\EGS6*\EGS.ini • [ECC_HOME]\EGS6*\EGS_Err.log • [ECC_HOME]\EGS6*\EGS_B*_Err.log • [ECC_HOME]\EGS6*\EGS_Discovery.log • [ECC_HOME]\EGS6*\EGS_Discovery.log.gz














Additional Files:















Symmetrix SRDF How do I troubleshoot using Symmetrix SRDF functions in ControlCenter 6.0? What do I need to verify and collect for ControlCenter Symmetrix SRDF issues? Troubleshooting Steps

• Use the agents view to determine which host is running the Primary Symmetrix agent for that Serial number. • SRDF configuration type (bi-directional, multi-hop, concurrent, etc.) • Symmetrix serial numbers and responsibility involved in the configuration. • Symmetrix microcode version(s). • Version of ControlCenter. • Version of SYMAPI. • Specific Symmetrix device group or devices involved in the RDF operation. • SRDF operation being performed at the time of failure. • Approximate time of the failed operation. • Specific error message returned. (Screen capture of error message.) • At which point did the failure occur? (Ex: merging track tables, enable / disable volumes, etc.) • Primary Symmetrix Agent host name and IP Address. (Use the agents view to determine which host is running

as Primary Symmetrix agent for the Serial numbers involved.) • Verify if any configuration locks are remaining on the Symmetrix

o symcfg list -lockn all -sid xxx o Record the lock number of any locks that are enabled o IMPORTANT: Be sure to only release stale locks o If necessary, engage a PSE to assist with hardware status and recovery






SYMAPI_WINDOWS • %ProgramFiles%\EMC\SYMAPI\log\symapi*.log


• %ProgramFiles%\EMC\SYMCLI\*.log SERVER










Additional Files:


o Open the ControlCenter Console o Select ECC Administration on the blue toolbar o Click the dropdown arrow next to ECC Administration, click Alerts


o In the Tree Panel on the left side of the Console, put a check next to the Hosts folder. Make sure there are no other items checked.

o In the Target Panel on the right you will see the Alerts view. On the top of that view, click the Table button.

o Click inside the table to make it the active pane o Go to the top left hand corner of the console screen and select File > Export > Browse and save











Storage Provisioning Service – Symmetrix, CLARiiON, StorageWorks How do I troubleshoot using Storage Provisioning Services (SPS) for Symmetrix, CLARiiON, and StorageWorks arrays in ControlCenter 6.0? What do I need to verify and collect for ControlCenter Storage Provisioning Services (SPS) issues with Symmetrix, CLARiiON, or StorageWorks arrays? Background Storage Provisioning Services operations are reliant on many agents. (Storage agents, SYMAPI, Host agents, FCC Agent, SDM agent, NaviCLI, Steam Agent). All SPS operations require the user to create, or assign, an existing task prior to executing. It is this Task List that will help in determining where the operation failure occurred, and which log files are most relevant in determining the cause of the failure. The Storage Policy represents the allocation parameters. The storage device configurations and protection level can prevent SPS from finding any matching devices based on how the Storage Policy is configured. The ControlCenter Console is the primary user interface for the Storage Provisioning Services. The Console provides wizards for performing both storage allocation and deallocation tasks, and the Console provides the only interface for some Storage Provisioning Services functions, such as:

• Managing permissions • Creating and editing storage pools • Creating allocation and deallocation policies • Performing deallocation tasks

You can use the CLI to perform the following provisioning functions: • Allocate storage to a host • Extend volume groups • Extend logical volumes • Extend file systems • View storage allocation policies and details of allocation tasks

ControlCenter CLI components: • EACLI Client provides an interactive or batch interface for executing CLI commands. You must install the

EACLI on a client before you can use it. • ECCAPI Server processes commands from CLIs, APIs, and other ControlCenter components, like the Web

Console, and sends them to the ECC Server. The ECCAPI Server must be installed and running to use the EACLI. Refer to the current EMC ControlCenter 6.0 Planning and Installation Guide, Volume 1 available on Powerlink.

• ECCAPI Plugin Manager processes commands for plugins to the ECCAPI Server. The ECCAPI Plugin Manager ensures that functions such as user authentication and batch mode operation are handled consistently.

• The ECC Server is the central communication point between ControlCenter components. Refer to EMC ControlCenter 6.0 Overview for more information on the ControlCenter architecture.

Troubleshooting Steps ControlCenter Console SPS

• Use the agents view to determine which host is running the Primary agent for that Serial number. • Verify the customer has created a Storage Pool. • Verify the customer has created a Storage Policy. • Verify the customer has a default fibre zoning policy. • Verify the storage device configurations match the parameters specified in the Storage Policy. • Collect a screen capture of the expanded task list from target panel showing the failed step. • The following considerations apply to the physical environment:

o Fabric — Ensure that hosts and storage systems are physically connected to the SAN fabric. The Fibre Channel Connectivity Agent must be used to discover all connectivity in the fabric.

o Host Types — Windows 2000, Windows 2003, HP-UX, Sun Solaris, and AIX hosts are supported. Refer to the EMC ControlCenter Support Matrix on Powerlink® to ensure that the HBA drivers are at the minimum supported revisions.


o VMware hosts — For VMware hosts running ESX Server 3.0, SPS can allocate storage up to the point of the host bus adapter of the server. VMware Virtual Infrastructure Client 2.0 can then be used to pick up the storage that has been allocated to the server.

o TES (Topology Editing Service) — If using the TES to create user-defined (virtual) hosts, SPS can allocate storage to a virtual host. When an actual physical host is added to the SAN the storage will be automatically available to the host, provided the correct host bus adapter details were used when creating the virtual host.

o Array Types — The supported arrays include Symmetrix, CLARiiON, and HP StorageWorks EMA with HSG80 controllers. In all cases, logical devices must be created ahead of time; SPS provides services to allocate those devices to (or deallocate from) supported hosts.

Note: Some functions require a minimum level of array microcode. Refer to the EMC ControlCenter Support Matrix on Powerlink for specific version information.

• The Automated Resource Manager (ARM) license is required for SPS. • The native agents for each of the storage arrays used for provisioning must be installed and operating, for

example, Storage Agent for HP StorageWorks or Storage Agent for CLARiiON. SPS cannot allocate or deallocate storage on arrays discovered by the Storage Agent for SMI.

• Refer to the EMC ControlCenter Support Matrix on Powerlink for the minimum version of Solutions Enabler. • Support for multipathing varies with the storage array or software used to support it.

o Symmetrix and CLARiiON: PowerPath 3.x and 4.x are supported in SPS. The policy field specifying the number of

paths is used to configure devices. PowerPath will manage I/O to those devices over those paths.

o StorageWorks: Version 5.2 supports a single path only.

Command Line SPS

• Before you use the CLI, ensure that the ControlCenter environment is configured for provisioning, for example: o Infrastructure components such as the ECC Server, Repository, and ECCAPI Server are running o Required agents, especially storage agents, are running o Hosts and storage systems are physically connected to the SAN fabric o For a complete discussion about configuring your environment for provisioning, refer to EMC

ControlCenter 6.0 Storage Provisioning Services Allocating and Deallocating Storage • Verify JRE 1.4.2 is installed on the host running the ecccli jar files (where the ECCAPIServer is installed).

o At a command prompt type: java –version • Capture the cmd prompt outputs for the failing operation with the verbose option enabled when executing the

cmd. eacli –ecchost <hostname> -eccport 38988 -eccverbose See Also: Primus emc134010: Unable to perform any config operation of Symmetrix via ControlCenter Console. Information to Collect Agent files are located [ECC_HOME]\xxx600


Note: All logs must be time relevant to the problem event. In addition, Server and Store log file sets must contain all available trace files. Using the Log Collection Wizard please select the following category and subcategories which will collect the recommended log files indicated below. Deselect any log types or log files that are not applicable to your environment. For instructions on using the Log Collection Wizard see emc154217. LCW Category: Object Management

LCW Subcategory: Select all Subcategories that correspond to the agent(s) concerned. For example, if the issue is related to the CLARiiON agent, select CLARiiON Management. SPS operations usually include multiple types of agents, be sure to include all agent types needed for your environment.


AGENT SPECIFIC LOGS

















Additional Files:











1. Open the ControlCenter Console 2. Select ECC Administration on the blue toolbar 3. Click the dropdown arrow next to ECC Administration, click Agents


4. In the Tree Panel on the left side of the Console, put a check next to the folder name Hosts. Make sure there are no other items checked.

5. In the Target Panel on the right you will see the Agents – Relationships view. On the top of that view, click the Table button.




Storage Provisioning Service – HP StorageWorks XP How do I troubleshoot using HP-XP active commands in ControlCenter 6.0? What do I need to verify and collect for HP-XP active commands? Troubleshooting Steps

• Only one active administrator can be connected to the array at a given time. This is a CommandView restriction.

• If using administrative userid/password and someone else is actively connected to the array as an administrator, you can discover, but you will not be able to make any configuration changes to the array. Most common error message will be: "Unable to initialize the session. Unable to obtain sessionid not in R/W mode after connect!

• To identify the IP address of who is connected to the array: From a CommandView prompt CV_CLI 30471 (R)> list session <serial#> Sending Request to CV server ... User Name,Host Aldress,Mode,Total Time,Idle Time,Session Id administrator,10.239.3.55,Downgraded Mode,00:00:09,00:00:06,mkgiha7an1 administrator,10.239.3.55,Modify Mode,00:03:24,00:00:30,3mte2p77x1 CV_CLI 30471 (R)>

Notes: • The time it takes to perform an active command is dependent on the response from CommandView server. • Before an active command can be issued, the CommandView server must be in refreshed mode. The agent will

repeatedly issue the CommandView CLI command: get_refresh_status <array_serrial#> until the data is refreshed in the CommandView Server. (This can be immediate or take several minutes.)

• Once the active CLI command is issued, if the command fails - the agent will immediately return an error description.

• If the command succeeds, the agent will again repeatedly issue the get_refresh_status <array_serial#> command until the data has been refreshed to the CommandView server (this will take several minutes). Once the CommandView server is refreshed, the agent will return a success to the console. The ECC server will then issue a REFRESHMO command to the agent to rediscover the array so that the change will be visible to the console.


LCW Subcategory: 3rd Party Array Management MNQ -STORAGEWORKS

• [ECC_HOME]\MNQ6*\MNQ.log • [ECC_HOME]\MNQ6*\MNQ_B*.log.gz • [ECC_HOME]\MNQ6*\MNQ.ini • [ECC_HOME]\MNQ6*\MNQ_Err.log • [ECC_HOME]\MNQ6*\MNQ_B*_Err.log • [ECC_HOME]\MNQ6*\MNQ_Discovery.log • [ECC_HOME]\MNQ6*\MNQ_Discovery.log.gz













Additional Files:


1. Open the ControlCenter Console


2. Select ECC Administration on the blue toolbar 3. Click the dropdown arrow next to ECC Administration, click Alerts 4. In the Tree Panel on the left side of the Console, put a check next to the Hosts folder. Make sure













SAN Manager Zoning How do I troubleshoot zoning with SAN Manager in ControlCenter 6.0? What do I need to verify and collect for SAN Manager zoning issues in ControlCenter 6.0? Troubleshooting Steps

• Verify there is an active FCC Agent running. • Verify via ping that the FCC Agent host can communicate with the switch. Ping not working is a symptom to

investigate, but it might be acceptable with some special firewall settings. • Determine if there is a firewall between the FCC Agent host and the switch. • If a firewall is present, verify proper port configuration as described in the Port Usage and Firewall

Configuration Appendix of the EMC ControlCenter Planning and Installation Guide, Volume 1 available on Powerlink.

• Determine if there is an orange diamond next to the switch. This indicates that it is not fully discovered, and you should collect the log files.

• If the switch appears, but is not fully discovered: o Determine what changes to the fabric/network environment may have occurred and how they

happened. o Verify on McData the FibreZone Bridge is running on the service processor or DSM Connect

workstation. By default the bridge is installed at C:\Program Files\EMC\FibreZoneBridge directory. The file fz_bridge_install.log records the version of the bridge. If the bridge is running, the EMCFibreZoneBridge service is started.

o Verify on Connectrix Manager that the McDATA switch is not grayed out. o Verify on McDATA the proper version of the FibreZone Bridge is installed. The file

fz_bridge_install.log records the version of the bridge. o Verify on Brocade and Cisco that you can Telnet from the FCC Agent host into the switch.

Note: In secure Brocade, for example, you may not be able to Telnet-login, but that is OK. o Verify on Brocade that Quick Loop is not in use.

• Collect screen capture of error message. Information to Collect Note: All logs must be time relevant to the problem event. In addition, Server and Store log file sets must contain all available trace files. Using the Log Collection Wizard please select the following category and subcategories which will collect the recommended log files indicated below. Deselect any log types or log files that are not applicable to your environment. For instructions on using the Log Collection Wizard see emc154217. LCW Category: Object Management

LCW Subcategory: Zoning Operations EGN -SWITCHES




• [ECC_HOME]\MGA6*\prereqs\* ESNAPI

• [ECC_HOME]\esnapi\data\* SYMAPI_UNIX

• /var/symapi/log/* • SYMAPI_WINDOWS • %ProgramFiles%\EMC\SYMAPI\log\symapi*.log • %ProgramFiles%\EMC\SYMCLI\*.log












Additional Files: • Collect from Brocade Switches

o SupportShow from each switch within a fabric. • Collect from McDATA (Connectrix) Switches

o Follow process for data collection from Connectrix Manager from emc153880 (Switch Discovery in this guide).















SAN Manager Masking Operations How do I troubleshoot masking with SAN Manager in ControlCenter 6.0? What do I need to verify and collect for SAN Manager masking issues in ControlCenter 6.0? Troubleshooting Steps

• Symmetrix o Verify there is an active SDM Agent running. o Verify there is an active Symmetrix Agent for the array you wish to LUN mask on. o Verify the SDM Agent host has an in-band connection to the array and the VCM Database device is

presented to the agent host. • CLARiiON

o Verify there is an active MGL Agent running. • HP StorageWorks (HGS80)

o Verify there is an active MNQ Agent running. • HDS (Xp512)

o Verify there is an active HDS Agent running. • Collect screen capture of error message. • If masking operation completed successfully, but the devices are not visible on the host:

o Verify that an FCC Agent is active and the associated switch is fully discovered. o Verify that the HBA types and drivers are supported in the EMC Support Matrix. o Verify that the SNIA compliant libraries are loaded on the FCC Agent system and the managed hosts.

The default library paths are specified for each host operating system in the esnapi.cfg file located: %ECC_INSTALL_ROOT%\exec\esnapi\esnapi.cfg.



Note: All logs must be time relevant to the problem event. In addition, Server and Store log file sets must contain all available trace files. Using the Log Collection Wizard please select the following category and subcategories which will collect the recommended log files indicated below. Deselect any log types or log files that are not applicable to your environment. For instructions on using the Log Collection Wizard see emc154217. LCW Category: Object Management

LCW Subcategory: Select all Subcategories that correspond to the array agent(s) concerned. Symmetrix Device Masking, CLARiiON Device Masking, Storageworks Device Masking, or HDS Device Masking

AGENT SPECIFIC LOGS


EGN -SWITCHES • [ECC_HOME]\EGN6*\EGN.log • [ECC_HOME]\EGN6*\EGN_B*.log.gz • [ECC_HOME]\EGN6*\EGN.ini • [ECC_HOME]\EGN6*\EGN_Err.log • [ECC_HOME]\EGN6*\EGN_B*_Err.log


• [ECC_HOME]\EGN6*\EGN_Discovery.log • [ECC_HOME]\EGN6*\EGN_Discovery.log.gz
















Additional Files:















ControlCenter Server How do I troubleshoot problems with the ControlCenter 6.0 Server component? What do I need to verify and collect for ControlCenter 6.0 Server issues? Troubleshooting Steps

• If there is no Server log or trace files and the ControlCenter Server stops after approximately 5 minutes, or the ControlCenter Server does not respond to a known working Console, suspect a Repository problem (refer to Primus emc154058 “ControlCenter Repository Database” .)

• Obtain a current RDA file by running: o Start/programs/EMC/EMC ControlCenter/RepositoryAdministration/Database RDA

• Run winmsd /s c:\tmp\<hostname>.nfo on all infrastructure hosts. o Verify from winmsd output displayed under Software Environment -> Running Tasks that there are

active JavaService.exe processes. Based on the implementation type, there maybe several Java processes running on an “all in one” host, such as WLAXML Servlet, StorageScope and Console (java.exe) are all Java-based components.

• Run drwtsn32 to verify the Log File Path and Crash Dump file locations. • Check Event Viewer -> Applications and System files for messages pertaining to Repository or Server • Check services to verify Server and Store are in a “started” status with an “automatic” startup type. • Verify the following two Repository services are installed and started as automatic startup type.


• The ECC Server requires the following ports (by default) to be accessible to the ControlCenter Console through a firewall:

o 5799 (ECC Server port) o 30721 (Login URL port) o 30750 (plain server port) o 30751 (name server port) o 30753

• Ports 30721, 30750, 30751, and 30753 must be open for Console access. Port 30753 is a derived value that is two higher than the greater of the plain server port and the secure server port. The secure server port is defined as 30751 in the following file:

o <install_root>\ecc_inf\data\<hostname>\data\ecc_server.ini o <install_root> is the root ControlCenter install directory.

• Port 30722 is also a derived port number opened by the ECC Server and is used by the Store when logging into the ECC Server. The ECC Server and Store are generally not separated by a firewall.

• The ECC Server listening ports are configured in the ecc_server.ini file, with the exception of the derived port values noted above, and may be changed if necessary to conform to needs of the environment. If you change port numbers, be aware of the derived port numbers and configure your firewalls according to your new ports.

• ControlCenter Console ports: 30760, 30769 • If the host with the EMC ControlCenter Server service has more than one network adapter, ensure that the

Server process has bound to the correct IP Address by checking the binding order of the adapter. The Adapter that the Server should bind to should be set as the first adapter in the Adapter Binding window.

o Also see Article emc133379: “EMC ControlCenter Server service does not continue to run”. o It is possible that after a reboot a secondary or newly installed network adapter is “picked up” first by

the OS and set as the first entry in the Adapter binding window Information to Collect Note: All logs must be time relevant to the problem event. In addition, Server and Store log file sets must contain all available trace files. Using the Log Collection Wizard please select the following category and subcategories which will collect the recommended log files indicated below. Deselect any log types or log files that are not applicable to your environment. For instructions on using the Log Collection Wizard see emc154217.


LCW Category: ControlCenter Components

LCW Subcategory: Infrastructure Components SERVER









Repository Exports: None Specified. Additional Files:





%windir%\system32\config\SysEvent.Evt %windir%\system32\config\AppEvent.Evt


%windir%\System32\config\SecEvent.Evt Collect information using ControlCenter Console:














ControlCenter Store How do I troubleshoot problems with the ControlCenter 6.0 Store component? What do I need to verify and collect for ControlCenter 6.0 Store issues? Troubleshooting Steps

• If there is no Server log or trace files and the ControlCenter Server stops after approximately 5 mins, or the ControlCenter Server does not respond to a known working Console, suspect a Repository problem (refer to Primus emc154058 “ControlCenter Repository Database”.)

• Obtain a current RDA file by running: o Start > Programs > EMC > EMC ControlCenter > RepositoryAdministration > Database RDA

• Run winmsd /s c:\tmp\<hostname>.nfo on all infrastructure hosts. o Verify from winmsd output displayed under Software Environment -> Running Tasks that there are

active JavaService.exe processes. Based on the implementation type, there maybe several Java processes running on an “all in one” host, such as WLAXML Servlet, StorageScope and Console (java.exe) are all Java-based components.

• Run drwtsn32 to verify the Log File Path and Crash Dump file locations. • Check Event Viewer -> Applications and System files for messages pertaining to Repository or Server • Check services to verify Server and Store are in a “started” status with an “automatic” startup type.

o The Server service must be running for the Store service to run. • Verify the following two Repository services are installed and started as automatic startup type.


• Please check for any Firewall changes where the following ports may be blocked: o ControlCenter Console: 30760, 30769 o ControlCenter Server: 5799, 30721, 30750, 30751, 30753

• Each instance of the Store has a configured port that is derived by adding two to the ECC Server's default configured port. In addition, the Store has a second listening port that is dynamic. The Store's configured port and the second port may be configured separately by adding the following lines to ecc_store.ini:

o use fixed port = true o store port = 5801 o client base port = 30730

• In this example, the Store opens two listening ports: 5801 and 30730. If use fixed port is false, the configured port is derived as described above, and a dynamic port is used for the second listening port.





ECCAPISERVER









• From the host where the EMC ControlCenter Server is installed: o %ECC_INSTALL_ROOT%\tools\utils\regutil All show > regutil_all_show.txt o route print > <file_name> for clustered and multi-homed environments

• From the host where the EMC ControlCenter Store(s) are installed: o Windows: In a command prompt window, capture the output of the following commands redirected


• From the host where the EMC ControlCenter Store(s) are installed: o Capture the Windows event viewer logs:


Collect information using ControlCenter Console: • Collect an export of the Alerts table view:



Table button. 6. Click inside the table to make it the active pane



• Collect an export of the Policies table view: 1. Open the ControlCenter Console 2. Select ECC Administration on the blue toolbar 3. Click the dropdown arrow next to ECC Administration, click Policies 4. In the Tree Panel on the left side of the Console, put a check next to the affected host in the









ControlCenter Repository Database How do I troubleshoot problems with the ControlCenter 6.0 database Repository? What do I need to verify and collect for ControlCenter 6.0 database Repository issues? Troubleshooting Steps Note: Do not run any Repository scripts from a Terminal Services session.

• If the following messages are present in the rambdbalrt.log: “media recovery needed”, “ora-01113”, or “end of file on communication channel is reached,” then run locally from the Repository command prompt %ECC_INSTALL_ROOT%\repository\admin\Ramb_Scripts\ramb_mediarecovery.bat. This will put the database in a usable state.

• If the Repository maintenance jobs are running at unexpected or inconsistent time intervals, run %ECC_INSTALL_ROOT%\repository\admin\Ramb_Scripts\ramb_change_jobs.bat locally from a DOS prompt. This script will adjust the maintenance job schedules.

• If the message “short term connection failure” or any database connection messages are found in the store.trc files, stop the Server and Store processes Verify connectivity to the repository by running %ECC_INSTALL_ROOT%\repository\admin\Ramb_Scripts\InstalledECCCompByComp or InstalledECCCompByHost or InstalledECCCompByVer and verify there are no errors in the corresponding error file and the corresponding log file is updated. If the Repository is not running, you may notice a current sqlnet.log file in the Ramb_Scripts directory. If the Repository is not running, verify the existence of any media recovery errors as stated above in the first bullet item and then restart the database, Server and Store processes.

• Verify the following two Repository services are installed and set to automatic startup type. o OracleECCREP_HOMETNSListener o OracleServiceRAMBDB

• Run Repository Backup from Start>Programs>EMC>EMC ControlCenter>Repository Maintenance>Backup Database.

• Run Repository Export DB from Start> Programs> EMC> EMC ControlCenter> Repository Maintenance> ControlCenter Repository> Export Database.

• Run LSNRCTL STATUS ><output_file_name> • Run netstat –an > <file_name> to verify port usage


LCW Subcategory: Infrastructure Components REPOSITORY






Additional Files:

• From all the host where the EMC ControlCenter Repository is installed: o Windows: In a command prompt window, capture the output of the following commands redirected


• Obtain a current RDA file by running o Start>Programs>EMC>EMC ControlCenter>RepositoryAdministration>Database RDA

• Capture the Windows event viewer logs: o %windir%\system32\config\SysEvent.Evt o %windir%\system32\config\AppEvent.Evt o %windir%\System32\config\SecEvent.Evt


ControlCenter ECC API How do I troubleshoot problems with the ControlCenter 6.0 ECC API component?? What do I need to verify and collect for ControlCenter 6.0 ECC API issues? Troubleshooting Steps

• The ECCAPI service must be restarted after the ARM license has been entered via the Java console. o Verify ECCAPI Sever is running.

EMC ControlCenter ECCAPI Server [(ECCService_ECCA) (ECCService_ECCAPI_Server.exe on W2003)] o Verify EMC ControlCenter server is running.

EMC ControlCenter Server [(ECCService_Serv on W2k) ( ECCService_Server.exe on W2003)] • Verify the EMC ControlCenter store is running.

o EMC ControlCenter Store [(ECCService_Stor on W2k) ( ECCService_Store.exe on W2003)] • Verify the following two Repository services are installed and set to automatic startup type.









ControlCenter Web Console Server How do I troubleshoot problems with the ControlCenter 6.0 Web Console Server component? What do I need to verify and collect for ControlCenter 6.0 Web Console Server issues? Troubleshooting Steps

• The WEB Console server can have 10 users. It is equivalent to one Java Console. So maximum number of users is 19 connected at the same time (10 Web users + 9 Java console).

• Record error messages. • Capture Screen shots with problem description and supporting data. • Run ping to EMC ControlCenter Server host using both host name and IP address to prove correct resolution. • Run tracert to EMC ControlCenter Server host. • Run nslookup > <output_file_name> • Windows: Check %SystemRoot%\system32\drivers\etc\hosts. • UNIX: Check etc/hosts. • If a firewall exists within the ControlCenter environment, verify that Oracle was installed with the firewall

enabled by checking the registry entry: USE_SHARED_SOCKET=TRUE in HKEY_LOCAL_MACHINE\SOFTWARE\ORACLE\HOME0

o You can view the setting by exporting the registry key to a file. From a command prompt run: 1. regedit /e C:\OracleKey.txt

HKEY_LOCAL_MACHINE\SOFTWARE\ORACLE\HOME0 2. Open the text file and verify the registry entry USE_SHARED_SOCKET=TRUE

o To add this setting to the registry, from a command prompt run: 1. regedit HKEY_LOCAL_MACHINE\SOFTWARE\ORACLE\HOME0

USE_SHARED_SOCKET=TRUE 2. Confirm to add entry when prompted.

• Record host OS type. • Run winmsd /s C:\tmp\<hostname>.nfo • If the Console launches and user can sign in but the Online Help menu item Documentation Library entry gets

a Page not found message in the browser window, verify the ECC WebApplications Server is running (tomcat.exe).

• Verify ECC server is running.[(ECCService_Serv on W2k) ( ECCService_Server.exe on W2003)] • Verify the ECC store is running. [(ECCService_Stor on W2k) ( ECCService_Store.exe on W2003)] • Verify the following two Repository services are installed and set to automatic startup type.


• Verify Browser has “popup Blocker” turned off. • Verify browser is supported. Reference the ControlCenter Support Matrix.

o Microsoft Internet Explorer 5.5 or 6.0 o Netscape Navigator 6.2.3 or 7.0

• Verify the supported JRE version is installed on the browser host. o Java Plug-in: JRE 1.4.2_06 or later is required for the Web Console browser. JRE 1.5.x is not

supported with ControlCenter 6.0. o Use command prompt window: java –version

• For proper functioning of the Web Console on Windows 2000 and Windows 2003 the user needs to make sure that the Java (Sun) 'Use Java 2 v1.4.2_0X for <applet> (requires restart)' is selected.

o Under IE Tools -> Internet Option -> Advanced Tab • Verify if a firewall is installed that the port is opened (Browser error message “Page not found”) • Check <install_root>ECCWebConsole\tomcat\conf\server.xml for correct port


o <Connector className="org.apache.coyote.tomcat4.CoyoteConnector" port="7070" minProcessors="5" maxProcessors="500" enableLookups="true" redirectPort="8443" acceptCount="100" debug="0" connectionTimeout="20000" useURIValidationHack="false" disableUploadTimeout="true" />









ControlCenter Web Console How do I troubleshoot problems with the ControlCenter 6.0 Web Console? What do I need to verify and collect for ControlCenter 6.0 Web Console issues? Troubleshooting Steps

• The WEB Console server can have 10 users. It is equivalent to one Java Console. So maximum number of users is 19 connected at the same time (10 Web users + 9 Java console).

• Record error messages. • Capture Screen shots with problem description and supporting data. • Run ping to EMC ControlCenter Server host using both host name and IP address to prove correct resolution. • Run tracert to EMC ControlCenter Server host. • Run nslookup > <output_file_name>. • Windows: Check %SystemRoot%\system32\drivers\etc\hosts. • UNIX: Check etc/hosts. • Verify if a firewall exists within ControlCenter environment, that Oracle was installed with the firewall enabled.

Run regedit HKEY_LOCAL_MACHINE\SOFTWARE\ORACLE\ HOME0 USE_SHARED_SOCKET=TRUE • Record host OS type. • Run winmsd /s C:\tmp\<hostname>.nfo • If the Console launches and user can sign in but the Online Help menu item Documentation Library entry gets a

Page not found message in the browser window, verify the ECC WebApplications Server is running (tomcat.exe).

• Verify ECC server is running.[(ECCService_Serv on W2k) ( ECCService_Server.exe on W2003)] • Verify the ECC store is running. [(ECCService_Stor on W2k) ( ECCService_Store.exe on W2003)] • Verify the following two Repository services are installed and set to automatic startup type.


• Verify Browser has “popup Blocker” turned off. • Verify browser is supported. Reference the ControlCenter Support Matrix.

o Microsoft Internet Explorer 5.5 or 6.0 o Netscape Navigator 6.2.3 or 7.0

• Verify the supported JRE version is installed on the browser host. o Java Plug-in: JRE 1.4.2_06 or later is required for the Web Console browser. JRE 1.5.x is not

supported with ControlCenter 6.0. o Use command prompt window: java –version

• For proper functioning of the Web Console on Windows 2000 and Windows 2003 the user needs to make sure that the Java (Sun) 'Use Java 2 v1.4.2_0X for <applet> (requires restart)' is selected.

o Under IE Tools -> Internet Option -> Advanced Tab • Verify if a firewall is installed that the port is opened (Browser error message “Page not found”)

Check <install_root>ECCWebConsole\tomcat\conf\server.xml for correct port <Connector className="org.apache.coyote.tomcat4.CoyoteConnector" port="7070" minProcessors="5" maxProcessors="500" enableLookups="true" redirectPort="8443" acceptCount="100" debug="0" connectionTimeout="20000" useURIValidationHack="false" disableUploadTimeout="true" />

Information to Collect Note: All logs must be time relevant to the problem event. In addition, Server and Store log file sets must contain all available trace files. Using the Log Collection Wizard please select the following category and subcategories which will collect the recommended log files indicated below. Deselect any log types or log files that are not applicable to your environment. For instructions on using the Log Collection Wizard see emc154217. LCW Category: Console

LCW Subcategory: Console Launch WEBCONSOLE

• %ECC_INSTALL_ROOT%\ECCWebConsole\tomcat\logs\*.txt TOMCAT


• %ECC_INSTALL_ROOT%\tomcat\logs\* ECCAPISERVER




Agent Run Time How do I troubleshoot Agent run time issues in ControlCenter 6.0? What do I need to verify and collect for ControlCenter 6.0 Agent run time issues? Troubleshooting Steps

• Export agent table view. • Export any alerts for that agent host. • Export Agent Policy View. • Restart Console session, and evaluate if Agent status has changed. • Verify status of Agent process from host command.

o Run: ps –ef | grep agent for UNIX o Check in Task Manager for Windows

• If troubleshooting a specific application’s agent such as the VERITAS NetBackup Agent or LEGATO NetWorker. Verify the absolute pathname of the application was entered correctly during install. If the path was entered incorrectly the Agent will not be displayed on Console as an installed Agent on that host, however the agent acronym directory would have been created in %ECC_INSTALL_ROOT%\exec.

o The absolute pathname can be verified in the Agent acronym xxx.ini file. Information to Collect Agent files are located [ECC_HOME]\xxx600




AGENT SPECIFIC LOGS




• [ECC_HOME]\MGA6*\prereqs\* SYMAPI_UNIX
















Additional Files:
















Agents Performance – Resource Usage How do I troubleshoot Agent performance or resource utilization issues in ControlCenter 6.0? What do I need to verify and collect for ControlCenter 6.0 Agent performance or resource utilization issues? Troubleshooting Steps

• Record specific hostname, OS, and IP of host which performance call is being placed. • Verify by what means the caller has determined an agent or other ControlCenter component is affecting

performance or system resources. o Collect native host command outputs displaying increased resource usage (examples include, but are

not limited to, Glance, Perfmon, sar, iostat). o Verify the particular agent(s) involved from native command output. o Verify if other application jobs such as backups were running during the times of high CPU utilization. o Document the approximate time of the issues.

• Check for gatekeepers, keeping in mind at least one gatekeeper is required for hosts where there is a Symmetrix Storage Agent installed.

• Check semaphore settings on UNIX hosts. • Export Agents Policy View. • Record which DCPs are enabled for agents installed, and running on subject host.

o Include WLA data collections o Record how many objects, and the size of the objects involved in the WLA collections. o Disable the WLA DCPs, and verify if the CPU Utilization remains high.

• Verify what Alerts are enabled for the host in question. o Disable the Alerts and verify if the CPU utilization remains high.

• What alerts are in the Alerts View (Active) for the host in question? o Look for “agent overloaded alerts.”

• If performance improves, enable DCPs, and Alerts one at a time while monitoring CPU utilization. • Collect EMCGrab for UNIX hosts • Run winmsd /s C:\tmp\<hostname>.nfo for Windows hosts





AGENT SPECIFIC LOGS














CONSOLE_SOLARIS


• $HOME/ecc/data/*/console.trc • $HOME/ecc/data/*/console.trc.*.gz • $HOME/ecc/data/*/consoleLaunch.log • $HOME/ecc/data/*/ecc_console.ini



Additional Files:
















Agents Cluster Failover How do I troubleshoot Agent cluster failover issues in ControlCenter 6.0? What do I need to verify and collect for ControlCenter 6.0 Agent cluster failover issues? Troubleshooting Steps Note: The following information is contained in the Engineering White Paper, “Customizing Hostnames in EMC ControlCenter” which is available on Powerlink. Reference this White Paper for detailed prerequisite and failover setup instructions. Instructions for installing ControlCenter agents in a clustered application environment may be found in the EMC ControlCenter 6.0 Planning and Installation Guide Volume 1.

• This is a completely manual process, EMC does not provide any automated process. • Verify agent is installed on a Symmetrix SRDF device (R1/R2). • ECC_HostID file MUST be created before starting Master Agent. • ECC_HostID file must contain “Valid” names. • ECC_HostID file must NOT contain the same name as Primary host (R1), or the same name as any other host. • Databases need to be discovered again on the altered host. • If Alerts and DCPs are not set to “Apply to all applicable objects”, you will need to add the altered host to the

Alert/DCP definition • Check for ECC_HostID file (Master Agent checks in the order in the table).

UNIX Host Locations Windows Host Locations %ECC_SYSFILE_HOME%

%ECC_SYSFILE_HOME%

/tmp/ECC %SYSTEMROOT%\ECC /var/ECC %ECC_HOME% %ECC_HOME% %TEMP%\ECC% %TMP\ECC% C:\tmp\ECC C:\temp\ECC





AGENT SPECIFIC LOGS

• [ECC_HOME]/xxx6*/xxx.log • [ECC_HOME]/xxx6*/xxx_B*.log.gz • [ECC_HOME]/xxx6*/xxx.ini


• [ECC_HOME]/xxx6*/xxx_Err.log • [ECC_HOME]/xxx6*/xxx_B*_Err.log • [ECC_HOME]/xxx6*/xxx_Discovery.log • [ECC_HOME]/xxx6*/xxx_Discovery.log.gz






Additional Files:









button. 6. Click inside the table to make it the active pane



• From the ControlCenter Console collect an export of the Agents table view: 1. Open the ControlCenter Console 2. Select ECC Administration on the blue toolbar 3. Click the dropdown arrow next to ECC Administration, click Agents 4. In the Tree Panel on the left side of the Console, put a check next to the folder name Hosts. Make





Agents Load Balancing How do I troubleshoot Agent load balancing issues in ControlCenter 6.0? What do I need to verify and collect for ControlCenter 6.0 Agent load balancing issues? Troubleshooting Steps

• Control of load balancing is set for all agent types by default. • Load balancing can be disabled for some agent types. • Some agent types can have specific control parameters for load balancing.

o Control is in the Server o %ECC_INSTALL_ROOT%\ecc_inf\data\ecc_inf_hostname \class\AgentManager.properties which is

read at Server startup only. o Check the parameters starting with LoadBalancer.countBasedLB.xxxx.





AGENT SPECIFIC LOGS




SYMAPI_WINDOWS • %ProgramFiles%\EMC\SYMAPI\log\symapi*.log


• %ProgramFiles%\EMC\SYMCLI\*.log EMC_SMI_PROVIDER

• %ProgramFiles%\EMC\SYMCLI\logs\* SERVER










• ControlCenter Repository
















FCC Agent SNMP Port Configurations How do I troubleshoot FCC Agent SNMP port configuration issues in ControlCenter 6.0? What do I need to verify and collect for ControlCenter 6.0 FCC Agent SNMP port configuration issues? FCC Agent SNMP Port Configurations

• The user can specify port to use for SNMP traffic at FCC Agent. • For fire wall support:

o Change the SNMP Config file setting. o Restart agent for change to take affect. o The switches also have to be configured. Refer to the vendor switch documentation to see if

configuration changes are allowed. Some don’t allow this. o Change: %ECC_INSTALL_ROOT%\exec\MGN600\conf\conf.txt

<SNMPTRAPPORT> 162 </SNMPTRAPPORT> <SNMPCLIENTPORT> Port #s (1024 to 65535) [if not using defaults reference Planning & Install Vol 1 Port Usage] </SNMPCLIENTPORT>


LCW Subcategory: McData –Switch, Brocade –Switch, or Cisco -Switch EGN -SWITCHES






SERVER • %ECC_INSTALL_ROOT%\ecc_inf\data\ecc_inf_hostname\data\server.trc


• %ECC_INSTALL_ROOT%\ecc_inf\data\ecc_inf_hostname\data\server.trc.*.gz • %ECC_INSTALL_ROOT%\ecc_inf\data\ecc_inf_hostname\data\ecc_server.ini







CONSOLE_SOLARIS • $HOME/ecc/data/*/console.trc • $HOME/ecc/data/*/console.trc.*.gz • $HOME/ecc/data/*/consoleLaunch.log • $HOME/ecc/data/*/ecc_console.iniCollect



Additional Files: None specified.


Veritas NetBackup Agent How do I troubleshoot Veritas NetBackup Agent issues in ControlCenter 6.0? What do I need to verify and collect for ControlCenter 6.0 Veritas NetBackup Agent issues? Troubleshooting Steps

• Collect tar file of entire agent directory on backup server. • Capture output of backup utilities display showing discrepancy. • Capture the outputs for the following commands from the backup server.

o Bpdbjobs –report –all_columns o Bpdbjobs -reports

• Collect screen shot of the StorageScope report (if applicable) from same date and time of other data collected. • If troubleshooting a specific application’s agent such as the VERITAS NetBackup Agent or LEGATO NetWorker.

Verify the absolute pathname of the application was entered correctly during install. If the path was entered incorrectly the Agent will not be displayed on Console as an installed Agent on that host, however the agent acronym directory would have been created in %ECC_INSTALL_ROOT%\exec.

o The absolute pathname can be verified in the Agent acronym xxx.ini file. Information to Collect Note: All logs must be time relevant to the problem event. In addition, Server and Store log file sets must contain all available trace files. Using the Log Collection Wizard please select the following category and subcategories which will collect the recommended log files indicated below. Deselect any log types or log files that are not applicable to your environment. For instructions on using the Log Collection Wizard see emc154217. LCW Category: Object Discovery and Monitoring

LCW Subcategory: Veritas NetBackup MNB -NETBACKUP-WINDOWS

• [ECC_HOME]\MNB6*\MNB.log • [ECC_HOME]\MNB6*\MNB_B*.log.gz • [ECC_HOME]\MNB6*\MNB.ini • [ECC_HOME]\MNB6*\MNB_Err.log • [ECC_HOME]\MNB6*\MNB_B*_Err.log • [ECC_HOME]\MNB6*\MNB_Discovery.log • [ECC_HOME]\MNB6*\MNB_Discovery.log.gz

MSB -NETBACKUP-SOLARIS • [ECC_HOME]\MSB6*\MSB.log • [ECC_HOME]\MSB6*\MSB_B*.log.gz • [ECC_HOME]\MSB6*\MSB.ini • [ECC_HOME]\MSB6*\MSB_Err.log • [ECC_HOME]\MSB6*\MSB_B*_Err.log • [ECC_HOME]\MSB6*\MSB_Discovery.log • [ECC_HOME]\MSB6*\MSB_Discovery.log.gz

MHB -NETBACKUP-HPUX • [ECC_HOME]\MHB6*\MHB.log • [ECC_HOME]\MHB6*\MHB_B*.log.gz • [ECC_HOME]\MHB6*\MHB.ini • [ECC_HOME]\MHB6*\MHB_Err.log • [ECC_HOME]\MHB6*\MHB_B*_Err.log • [ECC_HOME]\MHB6*\MHB_Discovery.log • [ECC_HOME]\MHB6*\MHB_Discovery.log.gz

MAB -NETBACKUP-IBM AIX • [ECC_HOME]\MAB6*\MAB.log • [ECC_HOME]\MAB6*\MAB_B*.log.gz • [ECC_HOME]\MAB6*\MAB.ini • [ECC_HOME]\MAB6*\MAB_Err.log


• [ECC_HOME]\MAB6*\MAB_B*_Err.log • [ECC_HOME]\MAB6*\MAB_Discovery.log • [ECC_HOME]\MAB6*\MAB_Discovery.log.gz












Repository Exports:


Additional Files:

• From the backup server o Collect tar file of entire agent directory. o Capture output of backup utilities display showing discrepancy. o Capture the outputs for the following commands:

Bpdbjobs –report –all_columns Bpdbjobs -reports

• From the StorageScope Console, collect screen shot of the StorageScope report (if applicable) from same date and time of other data collected.














ControlCenter 3rd Party Integration Packages How do I troubleshoot problems with the ControlCenter 6.0 3rd Party Integration Package? What do I need to verify and collect for ControlCenter 6.0 3rd Party Integration Package issues? Troubleshooting Steps • Verify the Framework product and host platform being used are supported by the version of ControlCenter.

o Note the versions of both the Framework application, and host O/S. o Compare versions with the ControlCenter Support Matrix.

• Verify the Integration Gateway Agent has been installed and is started. • Verify the Alert Management policy has been defined and enabled for the Integration Gateway. • Verify the Integration Package has been installed.

o Install via Web browser connecting to ControlCenter Server host o http:// <ControlCenter_Server_ip>/webinstall

• Verify from the end user if the Integration Packages have been customized in anyway. Information to Collect Note: All logs must be time relevant to the problem event. In addition, Server and Store log file sets must contain all available trace files. Using the Log Collection Wizard please select the following category and subcategories which will collect the recommended log files indicated below. Deselect any log types or log files that are not applicable to your environment. For instructions on using the Log Collection Wizard see emc154217. LCW Category: Object Discovery and Monitoring

LCW Subcategory: Integration Gateway CNG -GATEWAY WINDOWS

• [ECC_HOME]\CNG6*\CNG.log • [ECC_HOME]\CNG6*\CNG_B*.log.gz • [ECC_HOME]\CNG6*\CNG.ini • [ECC_HOME]\CNG6*\CNG_Err.log • [ECC_HOME]\CNG6*\CNG_B*_Err.log • [ECC_HOME]\CNG6*\CNG_Discovery.log • [ECC_HOME]\CNG6*\CNG_Discovery.log.gz

CSG -GATEWAY SOLARIS • [ECC_HOME]\CSG6*\CSG.log • [ECC_HOME]\CSG6*\CSG_B*.log.gz • [ECC_HOME]\CSG6*\CSG.ini • [ECC_HOME]\CSG6*\CSG_Err.log • [ECC_HOME]\CSG6*\CSG_B*_Err.log • [ECC_HOME]\CSG6*\CSG_Discovery.log • [ECC_HOME]\CSG6*\CSG_Discovery.log.gz


INTEGRATION_GATEWAY • %ECC_INSTALL_ROOT%\InstallIntPkg\ecc3pi\Win32\SETUP.INI • %ECC_INSTALL_ROOT%\InstallIntPkg\ecc3pi\Win32\*.log

SERVER • %ECC_INSTALL_ROOT%\ecc_inf\data\ecc_inf_hostname\data\server.trc


• %ECC_INSTALL_ROOT%\ecc_inf\data\ecc_inf_hostname\data\server.trc.*.gz • %ECC_INSTALL_ROOT%\ecc_inf\data\ecc_inf_hostname\data\ecc_server.ini








Keep in mind in order for the Console.trc file to be created, the user must enable Console tracing when launching the console session. Also keep in mind that the console.trc file will “roll-over” and zip itself as soon as the EMC ControlCenter console is closed. The new zip file will be: console.trc.1.gz OS Commands (from EMC ControlCenter Server host): None specified. Repository Exports: None specified. Additional Files: None specified.


ControlCenter Workload Analyzer – Performance Manager How do I troubleshoot problems with the ControlCenter 6.0 Workload Analyzer and Performance Manager components? What do I need to verify and collect for ControlCenter 6.0 Workload Analyzer and Performance Manager issues? Troubleshooting Steps Performance Manager If data provider window opens and the title bar has “Not Connected to Repository” message, Performance Viewer cannot connect to Repository.

• Verify the following two Repository services are installed and set to automatic startup type. o OracleECCREP_HOMETNSListener o OracleServiceRAMBDB

• Verify connectivity to the repository by running %ECC_INSTALL_ROOT%\repository\admin\Ramb_Scripts\ o InstalledECCCompByComp o InstalledECCCompByHost o InstalledECCCompByVer o Verify there are no errors in the corresponding error file and the corresponding log file is updated. o If the Repository is not running properly, refer to Repository runtime section for failure analysis.

• Verify the parameters in the %ECC_INSTALL_ROOT%\client\WLA\WLA.ini file are correct. o The WLA.ini file contains the Alias (IP address of the ControlCenter Repository host) and the userid (ecc)

• Verify the Oracle Client installed properly by checking the existence of: o C:\Program Files\Oracle o %ECC_INSTALL_ROOT%\Client o Run regedit, examine the HKEY_LOCAL_MACHINE\SOFTWARE\ORACLE

• If installing Performance Viewer on a non-Repository host, verify the existence and configuration of: o <%ECC_INSTALL_ROOT%\repository\network\admin\tnsnames.ora. o This is what the Oracle Client Listener will refer to for WLA.ini configuration.

• If the data provider window opens, and Performance Viewer is connected to the Repository, but you still get the data provider window:

o The WLA DCP’s may not be enabled o The first scheduled collection has not yet fired o Refer to the WLA DCP section below for DCP verification

Collecting Agents

• Record which Agents are acting as Primary Agent for which MO Agent types with WLA DCP capability: o Symmetrix o Hosts (Windows, Solaris, HP-UX, AIX, Linux) o Storage (Symmetrix, CLARiiON, HDS (native Hitachi only and requires Hitachi Tuning Manager) o Symmetrix o Fibre Channel Connectivity o Oracle o Symmetrix o Celerra

• Verify if the respective agent is active and collecting data by checking for archived files. o Archive files are placed in: o <path selected @ install time>\<MO type>\<MO Name>\interval o There should be .ttp and .btp formatted files present in this directory. o The .ttp files represent the text formatted agent collections. o The .btp files are the archiver binary converted files.

• If only .ttp files are present then the archiver failed to convert the text formatted files. • Verify the data parameters within the .ttp files are populated. • Verify the Archiver of the MO is active. • Look for the text “Process called for policy WLA” in agent log to find the beginning of a collection policy firing. • WLA Retention parameters can be found in %ECC_INSTALL_ROOT%\exec\ENW600\Retention.dat.


WLA Archiver • Record which WLA Archiver is assigned to which MO.

o Applicable in multi-archiver environments only • Determine if the interval data being collected is available in the appropriate directory. • The archived interval data files are located:

o <path selected @ install>\<MO type>\<MO Name>\interval o The archived data path is available in the agents table view for each ENW Agent

• Verify the presence of .btp formatted files. o The .btp files are archiver binary converted files.

WLA DCPs

• Specify which WLA DCPs are involved with the problem description. • Verify the WLA DCPs are set up properly and enabled for desired MO(s)

o Capture screenshots of DCPs to verify how they are set up • Types of WLA DCPs:

o Analyst o Daily o Revolving

• Test WLA functionality by creating a new analyst collection DCP, specify collection interval, and allow analyst collection to run for a minimum of two data samples.

o At least two internal periods specified. • Enable the Analyst DCP for the MO agent, and verify the MO desired is listed under the DCP “Apply to Tab”. • When the collection duration is reached, allow time for the archiver to convert the data before checking

Performance Viewer for the presence of the new Analyst collection file. Automation Reports

• AutoReports.log is located in %ECC_INSTALL_ROOT%\exec\ENW600. • Private data views are saved to the local %ECC_INSTALL_ROOT%\Client\WLA\Views\user.vds file by default. • %ECC_INSTALL_ROOT%\Client\WLA\Automation\jobsdef.bts contains the automation jobs definition. • View Automation Reports with a Web browser.

o The browser connects to ControlCenter Performance Manager Reports Service. o If you get a message that the page cannot be displayed, verify that ControlCenter Performance

Manager Reports Service is running and the URL is http://<host_name:30002>/wla51.xml?report=Automation

o If you still are unable to reach the Automation Report URL, try specifying the host IP address instead of host_name.

• If you encountered Error: 500 Internal Servlet error. o Verify the %ECC_INSTALL_ROOT%\Client\WLAXML\Servlet directory exists. o Verify the contents wla51.xml, wlaxml.ini, and an images directory in

%ECC_INSTALL_ROOT%\Client\WLAXML\Servlet\WebApps\wla Information to Collect Note: All logs must be time relevant to the problem event. In addition, Server and Store log file sets must contain all available trace files. Using the Log Collection Wizard please select the following category and subcategories which will collect the recommended log files indicated below. Deselect any log types or log files that are not applicable to your environment. For instructions on using the Log Collection Wizard see emc154217. LCW Category: WLA Performance Manager Operations

LCW Subcategory: Performance Manager, WLA Archiver, or Automation Reports PERFORMANCEMANAGER_SERVICE

• %ECC_INSTALL_ROOT%\tomcat\webapps\wla\conf\* • %ECC_INSTALL_ROOT%\tomcat\webapps\wla\logs\* • %ECC_INSTALL_ROOT%\tomcat\webapps\wla\data\*

PERFORMANCEMANAGER • %ECC_INSTALL_ROOT%\Client\PerformanceManager\Automation\jobsdef.bts • %ECC_INSTALL_ROOT%\Client\PerformanceManager\Views\*.vds

ENW -WLA ARCHIVER


• [ECC_HOME]\ENW6*\AutoReports.log • [ECC_HOME]\ENW6*\ENW.log • [ECC_HOME]\ENW6*\ENW_B*.log.gz • [ECC_HOME]\ENW6*\ENW.ini • [ECC_HOME]\ENW6*\ENW_Err.log • [ECC_HOME]\ENW6*\ENW_B*_Err.log • [ECC_HOME]\ENW6*\Retention.dat








OS Commands (from EMC ControlCenter Server host): None specified. Repository Exports: None specified.

Additional Files: None specified.


ControlCenter StorageScope/SRM How do I troubleshoot problems with the ControlCenter 6.0 StorageScope SRM components? What do I need to verify and collect for ControlCenter 6.0 StorageScope SRM issues? StorageScope Report Generation Troubleshooting Steps (6.0) On the EMC StorageScope Server machine check the following:

• Check errors or warnings about processes taking too long or still running after many hours in o %ECC_INSTALL_ROOT%\Tomcat\webapps\srm\logs\sts_application.log

• Check for Oracle errors in o %ECC_INSTALL_ROOT%\Tomcat\webapps\srm\logs\sts_application.log

On the EMC ControlCenter Repository machine check the following:

• Check to see if last modified date is less than 24 hours old: o %ECC_INSTALL_ROOT%\Repository\admin\Ramb_scripts\ramb_analyze_table.log

• Run %ECC_INSTALL_ROOT%\Repository\admin\Ramb_scripts\ramb_analyze_table.bat o Check the last line of ramb_analyze_table.log to see if the analyze completed successfully. o %ECC_INSTALL_ROOT%\Repository\admin\Ramb_scripts\ramb_analyze_table.log

STS GUI (6.0) Enable debug as follows:

• Make a backup copy of %ECC_INSTALL_ROOT%\storagescope\config\log4j.properties. • Edit %ECC_INSTALL_ROOT%\storagescope\config\log4j.properties.

o Find lines: log4j.logger.srm=INFO, STSAppender log4j.appender.STSAppender.Threshold=INFO

o And replace with: log4j.logger.srm=DEBUG, STSAppender log4j.appender.STSAppender.Threshold=DEBUG

• Reproduce the problem. • Collect %ECC_INSTALL_ROOT%\storagescope\logs\sts_application.log. • Replace %ECC_INSTALL_ROOT%\storagescope\config\log4j.properties with copy made previously.

STS Report Content (6.0)

• STS File Level Reporting • Verify that DCPs are enabled • Check if the value appears in Console or ARM the same way it appears in STS • Verify the definition of the metric, and its meaning

o Refer to Console On-Line Help “ControlCenter Glossary” o Example of common misconception: File system free space is not related to array allocated device

free space. • Make note of the following:

o Is the value wrong ONLY in the STS reports? o Otherwise investigate as an Agent or a Repository issue.

• Collect native host command outputs illustrating report discrepancy. • Collect a screen shot with text indicating the suspect value. • Check to see if last modified date is less than 24 hours old:

o %ECC_INSTALL_ROOT%\Repository\admin\Ramb_scripts\ramb_analyze_table.bat • Run %ECC_INSTALL_ROOT%\Repository\admin\Ramb_scripts\ramb_analyze_table.bat

o Check the last line of ramb_analyze_table.log to see if the analyze completed successfully. o %ECC_INSTALL_ROOT%\Repository\admin\Ramb_scripts\ramb_analyze_table.log




Note: All logs must be time relevant to the problem event. In addition, Server and Store log file sets must contain all available trace files. Using the Log Collection Wizard please select the following category and subcategories which will collect the recommended log files indicated below. Deselect any log types or log files that are not applicable to your environment. For instructions on using the Log Collection Wizard see emc154217. LCW Category: StorageScope SRM reports

LCW Subcategory: Select all Subcategories that correspond to the reports concerned. For example, if the issue is related to the CLARiiON agent, select CLARiiON –Array.

AGENT SPECIFIC LOGS






STORES • %ECC_INSTALL_ROOT%\ecc_inf\data\ecc_inf_hostname\data\store.trc • %ECC_INSTALL_ROOT%\ecc_inf\data\ecc_inf_hostname\data\store.trc.*.gz • %ECC_INSTALL_ROOT%\ecc_inf\data\ecc_inf_hostname\data\ecc_store.ini

ECCAPISERVER • %ECC_INSTALL_ROOT%\ECCAPIServer\data\apiserver.trc • %ECC_INSTALL_ROOT%\ECCAPIServer\data\apiserver.trc.*.gz • %ECC_INSTALL_ROOT%\ECCAPIServer\data\ecc_apiserver.ini.

SGS –STORAGESCOPE-SRM • [ECC_HOME]\SGS6*\SGS.log • [ECC_HOME]\SGS6*\SGS_B*.log.gz • [ECC_HOME]\SGS6*\SGS.ini • [ECC_HOME]\SGS6*\SGS_Err.log • [ECC_HOME]\SGS6*\SGS_B*_Err.log


STORAGESCOPE CONSOLE • %ECC_INSTALL_ROOT%\STS6*\SrmConsole\log\* • %ECC_INSTALL_ROOT%\STS6*\SrmConsole\reports\Oracle\*

STORAGESCOPE SERVICE • %ECC_INSTALL_ROOT%\Tomcat\webapps\srm\conf\* • %ECC_INSTALL_ROOT%\Tomcat\webapps\srm\logs\* • %ECC_INSTALL_ROOT%\Tomcat\webapps\srm\data\*

STS REPOSITORY • %ECC_INSTALL_ROOT%\Repository\admin\emcsts_scripts\*.log • %ECC_INSTALL_ROOT%\Repository\admin\emcsts_scripts\*.err • %ECC_INSTALL_ROOT%\Repository\admin\emcsts_scripts\*.INI • %ECC_INSTALL_ROOT%\Repository\admin\emcstsdb\bdump\alert_emcstsdb.log • %ECC_INSTALL_ROOT%\Repository\admin\emcstsdb\bdump\*.trc • %ECC_INSTALL_ROOT%\Repository\NETWORK\log\listener.log • %ECC_INSTALL_ROOT%\Repository\NETWORK\ADMIN\tnsnames.ora • %ECC_INSTALL_ROOT%\Repository\NETWORK\ADMIN\listener.ora • %ECC_INSTALL_ROOT%\Repository\NETWORK\ADMIN\sqlnet.ora • %ECC_INSTALL_ROOT%\Repository\assistants\dbca\logs\*


• EMC ControlCenter Repository • STS DSS Repository

Additional Files:

• Obtain a current RDA file by running: o Start > Programs > EMC > EMC ControlCenter > RepositoryAdministration > Database RDA

• %ECC_INSTALL_ROOT%\Tomcat\conf\server.xml • %ECC_INSTALL_ROOT%\Tomcat\webapps\srm\conf \logs\localhost_log.<date>.txt


ControlCenter StorageScope Data Migration Utility How do I migrate a 5.2.x XML Repository to a StorageScope 6.0 Repository? Using the Data Migration Utility To use the StorageScope Data Migration Utility to migrate data from the StorageScope 5.2.x XML Repository to the StorageScope 6.0 Repository: WARNING The StorageScope application must be shut down before running the data migration utility. Also, verify that there are no other data migrations in progress before running the utility. Before following the below procedure, be sure to have a backup of the current state of the STS Repository. In the event the migration fails or is aborted a backup will be necessary to restore the database. Click: Start > Programs > EMC > EMC ControlCenter > Repository Maintenance > StorageScope Repository > Backup Database

1. Before running the utility, review the prerequisite information in the EMC ControlCenter 6.0 Upgrade Guide. 2. On the StorageScope application server, open the Windows Command Prompt. 3. Navigate to <%ECC_INSTALL_ROOT%>\Tomcat\webapps\srm\bin.

• Location: %ECC_INSTALL_ROOT%\Tomcat\webapps\srm\bin. When the migration batch file exits it changes folders to the parent of bin: %ECC_INSTALL_ROOT%\Tomcat\webapps\srm

4. Enter STSMigrate.bat to launch the Data Migration Utility. 5. Enter the location of the XML Repository, either as a Windows directory or a UNC path; press <ENTER> to

accept it. • UNC stands for Universal Naming Convention, the format is \\<hostname or IP address of the machine

with the 5.x XML repository>\<share name> example: \\ecctest5\xmlrepository. Using the UNC is appropriate if the 5.x XML repository is on a different machine than the StoragScope application server. Make sure the xmlrepository folder is shared on the 5.x XML repository machine. The Windows directory path example would be C:\ECC\StorageScope\xmlrepository. The Windows directory path is appropriate if the 5.x XML repository is on the same machine as the StoragScope application server.

• Note: If there are multiple sites contained in the StorageScope 5.2.x Repository, the utility will warn you that it can only migrate data from the primary site.

• The migration utility must be repeated for each ControlCenter implementation. You may only collect the master site folder from each site. When you share out the xmlrepository folder the migration utility searches for and collects only the master site.

6. Enter the first date for which data should be migrated, in MM/DD/YYYY format. The default value is the first date that data exists; press <ENTER> to accept it. Be sure to pad months and days of the month with less than one digit. • Example: 9/6/2006 should be 09/06/2006.

7. Enter the last date for which data should be migrated, in MM/DD/YYYY format. The default value is the last date that data exists; press <ENTER> to accept it. • After you have entered the range of dates, the utility confirms the number of days for which data is being

migrated. If you want to migrate a non-contiguous range of dates you may either copy/move/rename the undesirable folders in the master site folder until the migration is done or do numerous small migrations.

8. Enter the name of the host on which the StorageScope Repository resides, or press <ENTER> to accept the default value of localhost. • If the localhost default does not work, just enter the hostname or IP address of the machine the

ControlCenter Server resides on instead. 9. Enter the name (SID) of the StorageScope Repository, or press <ENTER> to accept the default value.

• Default is emcstsdb (Where can this value be found, environmental variable on the StorageScope repository machine.)

10. Enter the port on the host through which the StorageScope repository is accessed. • The default Oracle port is 1521

11. When prompted, choose whether to migrate CAS (Content-Adressable Storage) data. 12. When prompted, choose whether to migrate User-Defined Group data. 13. When prompted, choose whether to proceed with the migration using the selected options. The migration utility

will run, and provide updates on the progress of the data migration.


14. After the migration has completed, re-start the StorageScope web application. They are referring to the EMC ControlCenter Web Server service.

15. Run ETL. To run ETL (Extract Transform Load), open a browser and point it to http://<IP address or hostname

of StorageScope server machine>/srm/login.do. Login, then navigate to the Utilities menu, select Settings > ETL Scheduler to open the ETL Scheduler page.

Aborting the Migration Utility If you need to abort the migration while the utility is running, follow the instructions below:

1. On the StorageScope application server, open the Windows Command Prompt. 2. Navigate to <%ECC_INSTALL_ROOT%>\Tomcat\webapps\srm\bin. 3. Enter STSMigrate.bat stop to stop the migration.

WARNING If you stop the migration, the operation will stop at its current point; no data will be rolled back, even if the transfer of data or the population of trending tables is incomplete. After aborting the migration, you must restore your StorageScope Repository to the last saved backup before re-running the utility or using StorageScope. Information to Collect Note: All logs must be time relevant to the problem event. In addition, Server and Store log file sets must contain all available trace files. Using the Log Collection Wizard please select the following category and subcategories which will collect the recommended log files indicated below. Deselect any log types or log files that are not applicable to your environment. For instructions on using the Log Collection Wizard see emc154217. LCW Category: StorageScope SRM Operations

LCW Subcategory: StorageScope Components SGS -STORAGESCOPE-SRM

• [ECC_HOME]\SGS6*\SGS.log • [ECC_HOME]\SGS6*\SGS_B*.log.gz • [ECC_HOME]\SGS6*\SGS.ini • [ECC_HOME]\SGS6*\SGS_Err.log • [ECC_HOME]\SGS6*\SGS_B*_Err.log

STORAGESCOPE_CONSOLE • %ECC_INSTALL_ROOT%\STS6*\SrmConsole\log\* • %ECC_INSTALL_ROOT%\STS6*\SrmConsole\reports\Oracle\*

STORAGESCOPE_SERVICE • %ECC_INSTALL_ROOT%\Tomcat\webapps\srm\conf\* • %ECC_INSTALL_ROOT%\Tomcat\webapps\srm\logs\* • %ECC_INSTALL_ROOT%\Tomcat\webapps\srm\data\*





Additional Files:

• From the system where the migration was run, collect a screen capture of the error message. • From the ControlCenter 5.2 StorageScope system, collect the 5.2 XML Repository.


Appendix A EMC ControlCenter Log Collection Wizard How to use the ControlCenter 6.0 Log Collection Wizard? How do I collect log files with the ControlCenter 6.0 Log Collection Wizard?

Introduction

Application objective: Facilitate the collection of files from an EMC ControlCenter environment for troubleshooting.

Application premise: EMC Customer Service personnel use the Log Collection Wizard (Internal version) to build an Instructions XML file which contains a list of data elements required for the resolution of an issue. The generated Instructions XML file is sent to the customer. On the EMC ControlCenter Server machine the customer opens the Instructions XML file which is then interpreted by the Log Collection Wizard (Customer version). Based on the data elements in the Instructions XML file, the Log Collection Wizard (Customer version) will attempt to collect the desired information from the EMC ControlCenter environment. All the collected information is placed in one zip file. The customer then sends the resulting zip file to EMC.

Application abstract: Customer Service personnel can install the Log Collection Wizard (Internal version) software on their work laptop/desktop machine. The Log Collection Wizard (Internal version) is used to create or open an Instructions XML file to send to a customer. The Instructions XML file contains a list of data elements required for the resolution of a customer issue. An Instruction XML file is like a recipe, it is a list of items EMC Customer Service would like to collect from a customer’s EMC ControlCenter implementation. The Instruction XML can only be interpreted by the Log Collection Wizard, it is not a script and cannot be executed. The items (data elements) may include collecting EMC ControlCenter log files, EMC ControlCenter Database exports, EMC ControlCenter database dump files, predefined OS Commands to run on the EMC ControlCenter Server machine and their associated output. The Instruction XML file generated by the Log Collection Wizard is well formed and valid. If it is altered, any invalid data elements will be ignored. The Instruction XML file is built in memory as you proceed through the screens of the Log Collection Wizard. Once the file is complete, the Log Collection Wizard (Internal version) can be used to Email or FTP the Instructions XML file to the customer. The customer may then launch the Log Collection Wizard (Customer version) on their EMC ControlCenter Server machine, browse to the location of the Instructions XML file sent by Customer Service and open it. Customer Service can then walk the customer through the remainder of the screens in the Log Collection Wizard (Customer version). The customer will be prompted with environmental specific queries such as which hosts to collect files from and any additional files such as screenshots of the issue that may be of use. Finally, the customer can use the Log Collection Wizard (Customer version) to FTP a zip of the files collected back to EMC.

Using the customer version of the Log Collection Wizard

Note: In order for the customer version of the Log Collection Wizard to work there must be an EMC ControlCenter 6.0 master Agent that has a record in the EMC ControlCenter repository. The customer version of the Log Collection Wizard will only work on the EMC ControlCenter server machine. Launch the Log Collection Wizard (Customer version) from the “EMC ControlCenter 6.0 Log Collection Wizard” desktop shortcut on your EMC ControlCenter Server machine then follow the steps below:

1. Create a new Instructions file or open an existing one: If an Instructions file already exists for the issue you are trying to address select the “Open and existing Instructions xml file” radio button, click the “Next>>” button then browse to the file Instructions XML file. Once you have clicked the “Open” button you may proceed to step 5 below. If you select the “Create a new Instructions xml file” radio button, click the “Next>>” button and follow the steps below. If you select the “Transfer an existing Log Collection zip file” radio button go to step 11.

2. Category screen: Select a Category that best fits the issue you are trying to address. You may only select one Category. Click the “Next>>” button after making your selection.


3. Subcategory screen: Select one or more Subcategories that best fit the issue you are trying to address. Click the “Next>>” button after making your selection.

4. Recommended Log Files screen. A list of Recommended Log Files appears, uncheck any log files that do not apply to the issue you are trying to resolve. Click the “Next>>” button after making your selection.

5. Hosts in Environment screen: The wizard will present a list of Agents Hosts (Any host with a master agent in the ControlCenter repository.). If you know any hosts that do not have the log files you are trying to collect, uncheck those agent hosts. Click the “Next>>” button after making your selection.

6. Log Type Validation screen: The Log Collection Wizard compares the Log Types you selected to those that are on the Agent Hosts you selected. Either all the Log Types are found, a subset of the requested Log Types are found or none of the requested Log Types are found. If a subset or none of the requested Log Types are found click the “<<Back” Button and select additional hosts. There is no guarantee the Log Types you select will exists since they may not be installed in the ControlCenter environment you are working with.

7. OS Commands screen: Select any OS commands that may be helpful. The OS commands selected will only be run on the EMC ControlCenter Server machine. Click the “Next>>” button after making your selection.

8. Repository Exports screen: You may either selecting a previously exported repository or export a new dump file. Click the “Next>>” button after making your selection.

9. Additional Files screen: Click the “Add” button if you would like to collect additional files such as screenshots.

10. Log Collection Menu: You have now created a new Instructions XML file. You may either click the “Run” button to run the actions in the Instructions XML file or click the “Save As…” button to save it for future use and run it later. If you clicked the “Run” button you are moved to the File Transfer screen at the end of a successful run.

11. File Transfer screen: Just enter a password (your Email address) and the Service Request Number of the issue you are working on to send a file to EMC's FTP server . After clicking the “Send” button the file will be posted on EMC's FTP server at \incoming\<Service request #>\<your collected logs zip file>. If you cannot access EMC's FTP server or prefer to send the collected log files elsewhere click the “Other FTP” radio button and enter the appropriate information, then click the “Send” button. Lastly, you may click the “Manual” radio button then the “Open” button to manually browse to the zip file that was created as a result of the log file collection.

Using the internal version of the Log Collection Wizard Scenario: As an internal user of the Log Collection Wizard you can install the software on your work PC/laptop and use the Log Collection Wizard (Internal version) to create an Instructions XML file that can be subsequently sent to a customer via Email or by posting it on ftp.emc.com. The customer may then launch the Log Collection Wizard (Customer version) on their EMC ControlCenter Server machine and open the Instructions XML file you sent them. You may then walk the customer through the remainder of the Log Collection Wizard to determine which hosts to collect files from and suggest any additional files such as screenshots of the issue that may be required. Finally, the customer can use the Log Collection Wizard (Customer version) to FTP a zip of the files collected back to EMC. As a Customer Support personnel launch the Log Collection Wizard (Internal version) from the “EMC ControlCenter 6.0 Log Collection Wizard” desktop shortcut on your desktop or laptop machine then follow the steps below:

1. Create a new Instructions file or open an existing one: If an Instructions file already exists for the issue you are trying to address select the “Open and existing Instructions xml file” radio button, click the “Next>>” button then browse to the file Instructions XML file. Once you have clicked the “Open” button you may proceed to step 7 below. If you select the “Create a new Instructions xml file” radio button, click the “Next>>” button and follow the steps below.

2. Category screen: Select a Category that best fits the issue you are trying to address. You may only select one Category. Click the “Next>>” button after making your selection.

3. Subcategory screen: Select one or more Subcategories that best fit the issue you are trying to address. Click the “Next>>” button after making your selection.

4. Recommended Log Files screen: A list of Recommended Log Files appears, uncheck any log files that do not apply to the issue you are trying to resolve. Click the “Next>>” button after making your selection.


5. OS Commands screen: Select any OS commands that may be helpful. The OS commands selected will only be run on the EMC ControlCeneter Server machine. Click the “Next>>” button after making your selection.

6. Repository Exports screen: You may either selecting a previously exported repository or export a new dump file. Click the “Next>>” button after making your selection.

7. You have now created a new Instructions XML file that you may Email to the customer, post on a FTP server for the customer to get, or save it off as part of an Instructions XML file library. If you click the “Email” button Outlook will launch, the instructions XML file you just built in the wizard will be automatically attached. If you select the “FTP” button you will be brought to the File Transfer screen. Just enter a password (your Email address) and the Service Request Number of the issue you are working on. After clicking the “Send” button the file will be posted on EMC's FTP server at \outgoing\<Service request #>\<your Instructions XML file>. Lastly, you may save the Instructions XML off as part of an Instructions file library by clicking the “Save As…” button.