Building a ControlCenter Infrastructure in a Windows 2003

EMC Proven Professional Knowledge Sharing 2010

Upgrading a multi Infrastructure EMC ControlCenter Environment from version 6.0 to 6.1 UB7 on a Windows 2003 Cluster Kevin F Atkin

Kevin F AtkinSAN Support EngineerThomson [email protected]

2010 EMC Proven Professional Knowledge Sharing 2

Contents

1 OVERVIEW ..................................................................................................................................... 4 1.1 HOW TO USE THIS ARTICLE ............................................................................................................ 4

2 INSTALLATION PRE-REQUISITES .......................................................................................... 4 2.1 EMC CONTROLCENTER SERVER INFRASTRUCTURE BUILD .................................................... 4 2.2 HP DL380 SERVERS ...................................................................................................................... 6

2.2.1 Install Windows 2003 Enterprise Edition .......................................... 6 2.3 HP DL360 SERVERS ...................................................................................................................... 6

2.3.1 Install Windows 2003 Standard Edition ............................................ 6

2.3.2 Install HBA cards ............................................................................... 6

2.3.3 Install Powerpath version 5.3 ............................................................. 7

2.3.4 Zone and Mask Servers into the SAN................................................ 7 2.4 USER ACCOUNTS ........................................................................................................................... 8

2.4.1 Windows Cluster and User Accounts ................................................ 8

2.4.2 User Rights Assignment for the Clusteradmin Service Account ....... 9

2.4.3 ControlCenter Users........................................................................... 9 2.5 NETWORKING ............................................................................................................................... 10

2.5.1 Private LAN Network Card Configuration ...................................... 10 2.6 SERVICE LAN NETWORK CARD CONFIGURATION ...................................................................... 15 2.7 ADVANCED NETWORK CONFIGURATION ..................................................................................... 19 2.8 DNS NAME RESOLUTION ............................................................................................................ 20 2.9 INSTALL CONTROLCENTER IN A CLUSTERED ENVIRONMENT ..................................................... 20

2.9.1 Create the Cluster ............................................................................. 20

2.9.2 Create the Cluster Resources ........................................................... 24

2.9.3 Create Cluster Node Names ............................................................. 25

2.9.4 Create the EMC CONTROLCENTERRG Resource Group ........... 25

2.9.5 Create the shared physical disk resource ......................................... 26

2.9.6 Create the IP Address Resource ....................................................... 27

2.9.7 Create the NetworkName Resource ................................................. 29

2.9.8 Install Java on all Servers ................................................................ 32

2.9.9 Install MDAC 2.8 on the third HP DL 380 Server .......................... 32

2.9.10 Patch 4014 (Pre 6.1 StorageScope user fix) ..................................... 32 2.10 PREPARE HOSTS FOR CONTROLCENTER INSTALLATION ......................................................... 32

2.10.1 Upgrade Preparations ....................................................................... 34

2.10.2 Install ControlCenter onto the First Cluster Node ........................... 36

2.10.3 Upgrade Remaining Infrastructure Store Hosts ............................... 43

2.10.4 Upgrade StorageScope Host ............................................................ 44


2.10.5 Install the ControlCenter 6.1 Console .............................................. 48

2.10.6 ControlCenter 6.1 Post Upgrade Cluster checks.............................. 51

2.10.7 Edit EMC ControlCenter_server.ini File ......................................... 52

2.10.8 Edit the Master Agent file ctg.ini ..................................................... 53

2.10.9 Edit the EMC ControlCenter_store.ini file ...................................... 53

2.10.10 ControlCenter Patching .................................................................... 53

2.10.11 Patch 4667 (EMC CONTROLCENTER Repository) ..................... 53

2.10.12 Patch 4668 (StorageScope Repository) ........................................... 53

2.10.13 Solutions Enabler 7.1.0.0 ................................................................. 54

2.10.14 Install Solutions Enabler 7.1 MRLK (CC4762) on the Active node

of the cluster. ........................................................................................................ 54

2.10.15 After EMC CONTROLCENTER 6.0 has Upgraded to EMC

CONTROLCENTER 6.1 ..................................................................................... 54

2.10.16 Update Bundle 7 (Patch 4673) ......................................................... 57

2.10.17 Patch 4744 ........................................................................................ 58

2.10.18 Patch 4754 ........................................................................................ 58

2.10.19 Add local ORA_DBA Group to each Cluster Node ........................ 58

2.10.20 Post UB7 Cluster Preparation .......................................................... 59

2.10.21 Confirm Master Agents are at the same patch level ........................ 60

2.10.22 Agent Patching ................................................................................. 60 2.11 INSTALL CONTROLCENTER LICENCES .................................................................................... 61 2.12 CREATE CONTROLCENTER RESOURCES FOR THE CLUSTER SERVICE .................................... 62

2.12.1 Load StorageScope .......................................................................... 68

2.12.2 Install the WLA Archive Agent ....................................................... 69

2.12.3 Install the NAS Agent ...................................................................... 70

2.12.4 Add Gatekeepers to Symmetrix® Agent Host .................................. 71

Disclaimer: The views, processes, or methodologies published in this compilation are those of the authors. They do not necessarily reflect EMC Corporation’s views, processes, or methodologies.


1 Overview 1.1 How to use this Article

I wrote this article based on my experience building an EMC ControlCenter® Cluster

in the Thomson Reuters Data Center environments. Use this article in conjunction

with the EMC Documentation available on PowerLink; I refer directly to EMC

documentation in part, where I describe the installation process.

This article s aimed at the Upgrade of EMC CONTROLCENTER 6.0 to 6.1 UB7 but

includes the steps required to build the initial cluster etc. For existing EMC

CONTROLCENTER 6.0 clustered installations, start from Section 2.9 to prepare the

upgrade.

2 Installation Pre-Requisites 2.1 EMC CONTROLCENTER Server Infrastructure Build

This article is based on a six server (two clustered) ControlCenter Infrastructure, but

also applies to any number of hosts depending upon requirements. In this example, I

have chosen the following server hardware. This procedure works equally well on HP

Blade Servers too.

• The EMC ControlCenter Server hardware will comprise 3 x HPDL380 Servers

and 3 x HPDL360 Servers.

• All servers that participate in EMC ControlCenter require the Windows 2003

Server. The HPDL380 Servers will require Windows 2003 Enterprise Edition

for the purposes of clustering and to utilise the 8GB of Memory that they will

be installed with. The HPDL360 Servers will be built using Windows 2003

Standard Edition.


• The ControlCenter Infrastructure will be installed as a 2 Node Cluster on

HPDL380 Server hardware in a distributed configuration. The remaining

HPDL380 Server will be used for Storage Scope and WLA (Work Load

Analyser) reporting. The HPDL360 Servers will communicate with the EMC

CONTROLCENTER Cluster to provide extra Console and Agent connectivity.

• A reliable Windows Domain infrastructure and a Cluster Service Account that

exists in the domain.

• Three network connections in each clustered server (one for the Service LAN

and two for Cluster Private LAN communication) with enough IP addresses to

support all required resources:

1 x Server IP Address per node

2 x Cluster Private IP Addresses per Node

1 x Cluster IP Address

1 x Resource Group (Virtual Server) IP Address per Resource Group

• Shared SAN storage that the clustered servers have access to and that are

configured to EMC best practices. This will comprise a 1x 68GB LUN and a 500MB Lun for the Quorum disk.

• Shared and Non shared 68GB LUNs will be assigned as follows:

A) 1 x 68GB LUN assigned to the Cluster servers for Repository and the ControlCenter Server. This is will be assigned the drive letter R

b) 1 x 68GB LUN assigned to the Storage Scope Repository on the third HPDL380 Server. This will be assigned the drive letter S

C) 1x 68GB LUN assigned to the fourth server (HP DL360) for Work Load Analyzer file storage. This will be assigned the drive letter T

Name the drive mappings as in Table 1

Q: SharedQ

Quorum

R: SharedR

EMC ControlCenter Repository and Server

S: StorageScope … …

Storage Scope™ Repository

T: WLArchive WLA Archives

Table 1


2.2 HP DL380 Servers

2.2.1 Install Windows 2003 Enterprise Edition Load Windows 2003 Enterprise Edition onto the HP DL380 Servers and name each server as follows: Site id-emccc01a Site id-emccc01b Site id-emccc02 Complete Table 2 for IP address details Hostname IP Address Subnet Mask Gateway Site-EMCCC01a SITE-EMCCC01b SITE-EMCCC02 SITE-EMCCC03 SITE-EMCCC04 SITE-EMCCC05 SITE-EMCCC01-VIP1 SITE-EMCCC01-VIP2

Table 2

Join all 3 servers to a Windows 2003 Domain. In this guide the Domain will be known as EMC ControlCenter

2.3 HP DL360 Servers

2.3.1 Install Windows 2003 Standard Edition Load Windows 2003 Standard Edition onto the HP DL360 Servers and name each server as follows: Site id-emccc03 Site id-emccc04 Site id-emccc05 Join all 3 servers to a Windows 2003 Domain. 2.3.2 Install HBA cards The EMC ControlCenter servers will each have dual port Emulex LP11002-E cards

installed.


2.3.3 Install Powerpath version 5.3 Run the installation of Powerpath and accept the default settings.

For the clustered servers only, you will need to set the Mode on the first HBA to

Active and the second HBA to Standby. Simply right click the HBA and change the

setting. Note this setting may no longer need to be applied as there was an issue

with Power Path that made this change necessary.

2.3.4 Zone and Mask Servers into the SAN

Zone each EMC ControlCenter Server to both SAN Fabrics. Select an appropriate

VSAN in the case of a Cisco MDS switch Fabric. The Cluster nodes will be masked

to 1 x 68GB Lun and the 0.5GB Quorum disk.

The third HPDL380 Server will be masked to 2 x 68GB LUNs. The remaining

HPDL360 Servers will not be using SAN attached storage.


2.4 User Accounts

2.4.1 Windows Cluster and User Accounts Create a Domain Cluster Service account called clusteradmin in the Windows

Domain that the EMC ControlCenter Servers will be member servers of. The

clusteradmin Domain account must be a member of the local Administrators group

on the cluster.

Create a Global Group in the Domain called EMC ControlCenter Administrators.

Create user accounts in the Domain for all EMC ControlCenter users and add them

to the Domain Global Group EMC ControlCener Administrators. Next, add the EMC

ControlCenter Administrators Group to the Local Administrators Group on each EMC

ControlCenter Server. Use the clusteradmin Domain account when creating the

cluster, as discussed in section 2.7.


2.4.2 User Rights Assignment for the Clusteradmin Service Account

Open the Administrative tools folder and Navigate to:

Security Settings -> Local Policies -> User Rights

The Cluster Service account must be given the following User Rights:

• Act as part of the operating system

• Back up files and directories

• Restore files and directories

• Adjust memory quotas for a process

• Log on as a service

• Increase scheduling priorities

Consult the Microsoft Technet Website for any further settings that may need to be

applied under Security Settings -> Local Policies -> Security Options

2.4.3 ControlCenter Users Initially, a Local user account, EMC ControlCenteradmin, must be created on each

EMC ControlCenter Server host. This EMC ControlCenteradmin account is used to

install ControlCenter .This user will have complete access rights to all ControlCenter

objects including ControlCenter administrative rights, which allows the user to add

new ControlCenter users and to set the permissions that govern a user’s access

rights within ControlCenter.

The user accounts to administer ControlCenter will be Domain accounts that are

members of the Global Group ControlCenter Administrators. Once ControlCenter has

been installed, the accounts are added to the EMC ControlCenter Users folder in the

format:

Domain\User


2.5 Networking

Server clustering requires a very specific network configuration in order to function.

They can be grouped into two specific connections to simplify the design:

• The Service LAN

• Cluster Private LAN

This is depicted in the following generic diagram. Please note that only Node 1 and

Node 2 are clustered in ControlCenter.

2.5.1 Private LAN Network Card Configuration

The Cluster Private LAN and Cluster Service LAN NICs have quite specific network

settings applied.


Verify the Bindings are as follows on the Private LAN:

Select only “Internet Protocol (TCP/IP)”.

Select “Internet Protocol (TCP/IP)” and click the Properties button.


Enter the required IP address of the Private LAN

Ensure there is no Default Gateway specified.

Ensure there are no DNS Servers specified.

Click the Advanced… button, and choose the DNS tab.


Verify that the “DNS suffix for this connection:” entry is blank.

Ensure that the “Register this connection’s addresses in DNS” is NOT selected.

Select the WINS tab.


Ensure there are NO WINS servers listed.

Select “Disable NetBIOS over TCP/IP”

Click OK,

Then click OK again.

Repeat this process for both NICs that are to be used for Cluster Private LAN

connections.


2.6 Service LAN Network Card Configuration

The network connection to be used for Service LAN Communications needs to be

renamed to Service LAN.

:

Open the Properties for each of the NICs:

Verify the Bindings are as follows:


. Select “Internet Protocol (TCP/IP)” and click the Properties button. Enter the IP Addressing as shown in the example as appropriate.

Click the Advanced… button, and choose the DNS tab


Verify that the “DNS suffix for this connection:” entry is correct.

Ensure that the “Register this connection’s addresses in DNS” is selected.

Next, select the WINS tab.


Add WINS servers (if used). Ensure that “Enable NetBIOS over TCP/IP” is selected.

Click OK,

And click OK again.

Repeat this process for all NICs that are to be used for Service LAN connections.


2.7 Advanced Network Configuration

In the Network Connections window, select the menu option Advanced and choose

Advanced Setting.

Ensure the network connections are listed in the Connections: list in the following

order:


Rename the network adapters on the two HP DL380 servers that are taking part in clustering as shown; disable the unused adapters.

2.8 DNS Name Resolution

Name resolution will be provided by DNS. Each ControlCenter (EMC ControlCenter)

Host will register into DNS when joining the Windows Domain. Verify that both

forward and reverse DNS lookup is properly configured and active

2.9 Install ControlCenter in a Clustered Environment

2.9.1 Create the Cluster

The shared SAN storage should only be masked to the first Node of the cluster before the cluster is created. Once the cluster is online, the remaining node can be

masked into the shared storage, and the new Node added to the cluster using the

clusteradmin utility.

Create the cluster using Microsoft’s Cluster Administrator; select the Domain that is

relevant to where ControlCenter is located. Give the Cluster a generic name in the

Cluster name field. This name identifies the cluster. An example follows.


Click Next


When prompted for an IP address, enter the Virtual IP address of the Cluster. This

Virtual IP is the first of 2 Virtual IP addresses used in this installation.

Click Next,

Enter the Domain cluster account, in this example, clusteradmin

This account requires special permissions as described previously.


Click Next, and the Wizard will check the validity of the cluster

This completes the cluster build.


Certain prerequisites need to be in place before ControlCenter can be installed into a

Microsoft Cluster. These are outlined in the following sections.

2.9.2 Create the Cluster Resources

The Cluster comprises a Cluster Group and a Cluster Resource Group called EMC CONTROLCENTERRG. Each Group must have an IP Address (Virtual), a Name,

and a shared Disk resource. Note that the Cluster Resource Group shown in the graphic is labelled as ECCRG

The screen shot below shows the Cluster Group that formed when the cluster is

created. The ControlCenter components will be added to a new group called EMC

CONTROLCENTERRG. This is created by right clicking on the Groups folder and

selecting New Group.


2.9.3 Create Cluster Node Names The nodes should be named in the format site-emccc01a, Site-emccc01b For Example:

Cluster Node #1: dtcs-emccc01a

Cluster Node #2: dtcs-emccc01b

2.9.4 Create the EMC CONTROLCENTERRG Resource Group Using Cluster Administrator create a new Resource Group called EMC

CONTROLCENTERRG. To do this, right click on the Cluster and select New Group.

The EMC CONTROLCENTERRG Group will be where the ControlCenter Repository

and Server will come under cluster control.

The screen shot below shows the EMC CONTROLCENTERRG and the associated

IPAddress, NetworkName, and Disk resources. Note: Generic Services are added AFTER ControlCenter has been upgraded. There should be no Generic services created for ControlCenter until after the full upgrade completes.


2.9.5 Create the shared physical disk resource Click and highlight the Disk Group menu tree icon to show the Disk resource group.

Drag the desired physical disk from the Disk Group to the EMC

CONTROLCENTERRG Resource group. When asked to confirm that the resource

should be moved and the group should be changed for the resource, reply Yes.

MSCS creates a resource for the shared physical disk (Disk R)


2.9.6 Create the IP Address Resource The screen shot shows the General tab of the IP Address resource

The next screen shot shows the Dependencies tab of the IP Address resource where

the IP Address shows a dependancy on the shared Disk R.


The next screen shot shows the Paramaters tab (where the IP Address is input). This

address is a Virtual IP. This is the address that ControlCenter runs on. When a Move

Group is executed, the IP address will transfer to the other node. It is important that

DNS is working correctly for this to happen.


2.9.7 Create the NetworkName Resource The screen shot shows the General tab of the NetworkName Resource

On the Dependencies screen, since NetworkName is dependent on

IP Address, select Modify. Then move the IP Address resource from the Available

Resources column to the Resource Dependencies column.

Then, click next.

This automatically establishes dependencies on all previously configured generic

resources.

The Dependencies tab is shown follows.


The Parameters tab shows the name that ControlCenter uses. This name is input

into ControlCenter Console when connecting.


Right click on each Cluster Private Lan under the Networks folder in Cluster

Administrator. Select the Internal cluster communications only (private network)

radio button.

This is shown below:

Select the “All communications” (mixed network) radio button for the Service LAN. Bring the cluster resource group(s) online.


In the MSCS tree panel, right click the cluster resource groups to be

brought online (in our example, EMC CONTROLCENTERRG is the only resource

group).

And click Bring Online.

Move each Physical Disk resource on to all nodes in the cluster (via the Move Group

command in the Cluster Administrator). This is done so that you may verify that the

drive letters are assigned correctly on all nodes. Use the Disk volumes labeled in

Table 1 to assist in this verification.

2.9.8 Install Java on all Servers Install Java jre-6u2-windows-i586-p.exe on each server. Run the executable and

accept the defaults.

2.9.9 Install MDAC 2.8 on the third HP DL 380 Server Download the latest version of MDAC from Microsoft and install on the third HPDL

380 Server which will be site-emccc02.

2.9.10 Patch 4014 (Pre 6.1 StorageScope user fix) Follow the readme for this patch. It may not be required.

2.10 Prepare hosts for ControlCenter installation

Ensure the following for hosts where infrastructure components

(EMC CONTROLCENTER Server, Repository, and Store) are to reside:

• The fully-qualified domain names (FQDNs) and hostnames do

Not contain underscores.

• No applications are currently running (to prevent interference

With the installation process, close any Windows Explorer windows and

active command prompts).

• All Oracle software is removed

• Terminal Services is not running

• Disable Anti Virus Software


• Stop and Disable DTC Service (Distributed Transaction Coordinator). This

should not be re-enabled after EMC CONTROLCENTER is installed.

• Stop all HP Services and set to manual

• PowerPath™ version 5.3 installs EMC PowerPath RSA Encryption Service

5.3.0. This should be disabled.

• Verify ControlCenter Server Secure Listening Port 10799 is OPEN to ensure

proper communication between agents and the infrastructure.

Edit the path statement in the Environment Variables section of the System

Applet in Control Panel and remove the following path variables:

Variable name:

R_SHLIB_LD_LIBRARY_PATH

Variable Value:

D:\Program Files\EMC\RSA\Rkm_Client\lib

• If Cisco Device Manager is installed, Stop and set to Manual PostgreSQL

service.

• The ControlCenter Repository upgrade executable requires the use of two

DLL’s ….libeay32.dll and ssleay32.dll. Other applications may also use these

DLL’s which can conflict with ControlCenter upgrade.

Note the existing settings for the system %PATH% variable, and then make

sure the following path exists at the beginning in the Environment variables

section BEFORE the upgrade commences:

%RAMBOH%\admin\Ramb_scripts;R:\EMC

CONTROLCENTER\Repository\bin

Once the upgrade has completed, revert back to the original %PATH%


2.10.1 Upgrade Preparations Quoting from the EMC documentation, “Ensure that all ControlCenter Repository maintenance tasks are stopped prior to starting the upgrade." Run the

ramb_jobstatus.bat job located in the

R:\EMC CONTROLCENTER\Repository\admin\Ramb_scripts folder to determine

which maintenance jobs has already been run. Open the ramb_jobstatus.log file

located in the same folder as the batch file, to check for any maintenance jobs that

have not been run yet and run any additional maintenance jobs you feel are

necessary”

• Copy the regutil.exe and regutil.pl from the Regutil\Regutil600 directory,

found of where the EMC CONTROLCENTER6.1 disks have been copied as

discussed in Sect 2.8.2 of this document.

Copy these files to: R:\EMC CONTROLCENTER\Tools\Utils folder. Do this on

ALL Infrastructure hosts.

• Disable all WLA DCP’s as per EMC Upgrade guide discussed earlier.

• Uninstall all Control Center 6.0 Consoles using Add / Remove programs in

Windows Control Panel.

• Backup StorageScope Repository. Run an ETL from within StorageScope

and run emcsts_export_db.bat

• Backup ControlCenter Repository. Run clustersave.bat from R:\EMC

CONTROLCENTER\tools\Utils folder. Next, copy the EMC

CONTROLCENTER folder to R:\Rollback\EMC CONTROLCENTER

• If this is an upgrade to an existing cluster, remove ALL EMC

CONTROLCENTERRG Generic cluster resources using clusteradmin. The

only resources that should be under cluster control prior to the upgrade are:

Disk R , EMC CONTROLCENTERRG IPaddress and EMC

CONTROLCENTERRG Networkname


From the cluster A node server and the StorageScope server Services window, Stop

and set to Manual the following Services:

• Disable Terminal Services

• Distributed Transaction Coordinator service

• Anti-Virus Software service(s)

• EMC ControlCenter API Server service

• EMC ControlCenter Master Agent service

• EMC ControlCenter StorageScope Server service

• EMC ControlCenter Web Server service

• EMC ControlCenter WebConsole Server service

• EMC ControlCenter Store service

• EMC ControlCenter Server service

Note: ControlCenter services will already be set to manual if the clustersave.bat

file has been run

From the ControlCenter Repository host (Node A) Services window, Stop and

Restart:

• OracleEMC CONTROLCENTERREP_HOMETNSListener

• OracleServiceRAMBDB

Note: Stop both the OracleEMC CONTROLCENTERREP_HOMETNSListener

service and the OracleServiceRAMBDB service. After both services are listed as

stopped, re-start the OracleEMC CONTROLCENTERREP_HOMETNSListener

service and the OracleServiceRAMBDB service in that order.

Verify that all the ControlCenter processes listed above have stopped From Task

Manager.


From the StorageScope Repository host Services window, Stop and

Restart:

• OracleEMC CONTROLCENTERREP_HOMETNSListener

• OracleServiceEMCSTSDB

Note: Stop both the OracleEMC CONTROLCENTERREP_HOMETNSListener

service and the OracleServiceEMCSTSDB service. After both services are listed as

stopped, re-start the OracleEMC CONTROLCENTERREP_HOMETNSListener

service and the OracleServiceEMCSTSDB service in that order.

2.10.2 Install ControlCenter onto the First Cluster Node

Download the latest A04 EMC CONTROLCENTER 6.1 ISO images from the EMC

Powerlink website. The most convenient way to install or upgrade EMC

CONTROLCENTER is to first copy all the ISO images and Patches (currently UB7

shown here as CC_4673) to a shared network drive as shown below:


Once all the pre-requisites discussed in this document and in EMC documentation

have been fulfilled, the upgrade is ready to proceed. Make sure that the current EMC CONTROLCENTER folder is backed up as per EMC documentation!

Using the clusteradmin utility, PAUSE the second node of the cluster by right-clicking

on Node B and selecting Pause Node.

On the first node of the cluster (Node A), run setup.exe from the shared location.

The Primary Network Name appears automatically if DNS is correctly set up.

Select Next,


If prompted for a Secondary Network, Select None.

Accept the installation path of R:\EMC CONTROLCENTER (in this example), and

Click Next.

Specify the existing ControlCenter Repository, and Click Next


Select Install The following screenshots will be observed as the upgrade progresses:




The Installation of the Repository can take up to an hour or more! This will be

dependant upon the size of the database being upgraded.


Select, “Yes I want to Restart my Computer now” then select Finish. This completes

the initial (pre patching) part of the Repository upgrade.

When the Server has rebooted, check that the Cluster Service is running (the B Node

should still be Paused) so that the cluster R drive is made available.

Check that ALL EMC ControlCenter Services can be started. They will be set to

Manual at this point.

2.10.3 Upgrade Remaining Infrastructure Store Hosts Run setup.exe against all remaining EMC CONTROLCENTER hosts that have a 6.0

Store installed, so that the Store upgrades to EMC CONTROLCENTER 6.1


2.10.4 Upgrade StorageScope Host Make sure that all ControlCenter services are running before going onto the

StorageScope server (site-emccc02) and running setup.exe from the shared drive

where EMC CONTROLCENTER6.1 disks were copied to.

Follow the screen prompts


The above Window will pop up and requires an answer. Select No if you have

already run the emcsts_export_db.bat script found in: S:\EMC

CONTROLCENTER\Repository\admin\emcsts_scripts folder

Once the upgrade completes, perform the following validation tasks:


Validate access to the ControlCenter Repository and StorageScope Repository databases

• From the Infrastructure Server (Node A), run EMC

CONTROLCENTER_GETVERSION and ensure the correct version

displays as follows:

R:\EMC CONTROLCENTER\Repository\admin\Ramb_scripts>EMC ControlCenter_getversion.bat EMC CONTROLCENTER_INSTALL_ROOT environment variable found is

==> R:\EMC CONTROLCENTER

return code = 0

EMC CONTROLCENTER Repository Version

--------------------------------------------------------------

EMC CONTROLCENTER Repository version information

External Version : Version 6.1.0

Internal Version : 6.1.0.4.71

KitSpec : GA6

Build Label : EMC CONTROLCENTER61040.16MAY2009.1146.224

Build Name : EMC CONTROLCENTER61040.16MAY2009.1146.224

Release Date : Sat May 16 14:34:00 2009

Install Date : Thu Sep 24 10:59:16 2009

• Run the following on the StorageScope server and check the result is

displayed as below:

S:\EMC

CONTROLCENTER\Repository\admin\emcsts_scripts>emcsts_getversio

n.bat

Value of userName got as input: sys


Initializing Lockbox in:S:\EMC CONTROLCENTER\lockbox\WEB.lb

Value of UserName passed to SQL: sys

Success!

EMCSTS is open

Get Schema Version from Single-Site StorageScope Repository Database

. . .

Value of userName got as input: emcsts

Initializing Lockbox in:S:\EMC CONTROLCENTER\lockbox\WEB.lb

Value of UserName passed to SQL: emcsts

Success!

SQL*Plus: Release 10.2.0.4.0 - Production on Tue Oct 20 16:07:46 2009

Copyright (c) 1982, 2007, Oracle. All Rights Reserved.

Connected to:

Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 - Production

With the Partitioning, OLAP, Data Mining and Real Application Testing

options

GETVERSIONINFO

--------------------------------------------------------------------------------

EMC CONTROLCENTER Repository version information

External Version : Version 6.1.0

Internal Version : 6.1.0.4.71

KitSpec : GA6

Build Label : EMC CONTROLCENTER61040.16MAY2009.1146.224

Build Name : EMC CONTROLCENTER61040.16MAY2009.1146.224

Release Date : Sat May 16 14:34:00 2009


Disconnected from Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 –

Production

With the Partitioning, OLAP, Data Mining and Real Application Testing options

S:\EMC CONTROLCENTER\Repository\admin\emcsts_scripts>

DO NOT Apply the Update Bundle if the above commands do not return the correct version. Contact EMC for support. Open the Services applet on the A Node and verify that the Oracle database

services have started.

• Make sure the following services are running (if not then Start them)

• EMC ControlCenter Key Management Server

• EMC ControlCenter Repository

• EMC ControlCenter Server

• EMC ControlCenter Store

• EMC ControlCenter Web Server

• EMC ControlCenter API Server

• EMC StorageScope Server

2.10.5 Install the ControlCenter 6.1 Console

Once ControlCenter has upgraded to version 6.1 then install a Console so that the

Master Agent can be upgraded as detailed in section 2.8 of this document.

Open a browser and connect to the cluster node name. For example:

https://dtcs-emccc01:30002/webinstall

Note that it may be necessary to use the IP address of the cluster if the cluster name

cannot be resolved at this point. This solution to this comes after the cluster

resources have been re instated later in this document.


Install the Console 6.1.0 from the on screen menu as shown below:

When the Console has installed, enter the Cluster name in the Server Host Name as

shown here.

It may be necessary to use the IP address of the cluster if an error appears stating

that EMC CONTROLCENTER server is offline at this point. This is addressed later in

this document.


Log into the ControlCenter Console.

Use a Domain account that has previously been added to the EMC

CONTROLCENTER Users in ControlCenter. If this was a new install then the only

account available would be the local EMC ControlCenteradmin account “eccadmin”

Note that the Console view shown here is from a UB5 instance of ControlCenter

Once logged in:

Select the Agents View, and then right click on the main Repository server (Node A)

in the Agents view window.

Select Agents in the drop down list followed by Upgrade.

Upgrade the Master Agent to ControlCenter 6.1 level before

beginning the next procedure.


2.10.6 ControlCenter 6.1 Post Upgrade Cluster checks Before installing the Update Bundle, prepare the cluster nodes to enable successful

failover of the ControlCenter resources.

Execute the following procedure on the A node where the original ControlCenter

installation was done.

The Repository is automatically started and online after installing ControlCenter. The

Repository must be shut down prior to performing the Move Group command in the

next step. Shut down any Repository services that might be running:

In the Services window, select the following services:

OracleServiceEMCSTSDB

OracleEMC CONTROLCENTERREP_HOMETNSListener

OracleServiceRAMBDB

And select Stop

1. Change directory to R:\EMC CONTROLCENTER\tools\utils

2. Run clustersave.bat.5 Upgrading ControlCenter 5.2 to 6.1 3. Using the clusteradmin utility, initiate a manual failover of the

EMC CONTROLCENTERRG resource group to the B node. Right-click the resource

group EMC CONTROLCENTERRG and choose Move Group.

From the B node, do the following:


2. Run clusterupgrade.bat. 3. Reboot the B node for the upgrade to take effect.

The EMC CONTROLCENTERRG ControlCenter resources need to be recreated in

the case of an upgrade, or created if installing for the first time, but only after applying

the Update Bundles. This will be discussed later in this document.

The following do not need to be recreated:


� EMC CONTROLCENTERRGPhysicalDisk

� EMC CONTROLCENTERRGIPAddress

� EMC CONTROLCENTERRGNetworkName

Use the Move Group command again to bring the Cluster Resources under the A

node again.

2.10.7 Edit EMC ControlCenter_server.ini File To enable correct name resolution when logging into the EMC CONTROLCENTER

Console after upgrading the cluster configuration, re-enter the IP aliases in the EMC ControlCenter_server.ini file.

Locate the EMC ControlCenter_server.ini file on the A node in

<Install_Root>\EMC ControlCenter_inf\EMC ControlCenter_inf_hostname\data

Where in this instance the <Install_Root> is the R: drive

Make a copy of the EMC ControlCenter_server.ini file to a backup location.

Then open the EMC ControlCenter_server.ini file and search for the line that contains

the string name server aliases and then add the physical IP addresses (not logical)

of Node A and Node B respectively as shown in the example below:

name server aliases = 10.20.2.4,10.20.2.74


2.10.8 Edit the Master Agent file ctg.ini Locate the ctg.ini file in the R:\EMC CONTROLCENTER\exec

Add the following entry:

[Default]

cluster = true

2.10.9 Edit the EMC ControlCenter_store.ini file Locate the file where each Store is installed.

For example in S:\EMC CONTROLCENTER\EMC ControlCenter_inf\data\EMC

ControlCenter_inf_hostname\data

Add the following entry:

use fixed port = true

2.10.10 ControlCenter Patching The following sections outline the necessary patching that is required to complete the

upgrade, starting with the Oracle OCPU patches.

2.10.11 Patch 4667 (EMC CONTROLCENTER Repository) This is an Oracle OCPU patch. Follow the readme document that is supplied with this

patch. It can take quite some time to load!

2.10.12 Patch 4668 (StorageScope Repository) This is an Oracle OCPU patch. Follow the readme document that is supplied with this

patch. It can take quite some time to load!


2.10.13 Solutions Enabler 7.1.0.0 Update Bundle 7 (UB7) requires SE 7.1.0.0 is installed into the media repository, the next section describes how this is achieved.

2.10.14 Install Solutions Enabler 7.1 MRLK (CC4762) on the Active node of the cluster.

This enables Solutions Enabler to be deployed to other hosts

Read the readme that comes with Patch CC 4762 and then run the EMC

CONTROLCENTER_SE71.exe file. Make sure the patch installs successfully.

Select Next

The MRLK install installs Solutions Enabler 7.1.0.0 into the media repository, so that

it can be deployed to other hosts during Agent installation.

2.10.15 After EMC CONTROLCENTER 6.0 has Upgraded to EMC CONTROLCENTER 6.1

Download Patch 4563 from EMC and apply to the following Infrastructure hosts:


EMC ControlCenter Server

EMC ControlCenter Repository

EMC ControlCenter Store(s)

EMC ControlCenter StorageScope

.

Before running this patch, check for the existence of the registry string MA_ROOT on

each of the above hosts found in:

HKEY_LOCAL_MACHINE\SOFTWARE\EMC CONTROL CENTER\6.1

This string contains the value used to specify the ControlCenter installation path of

the Master Agent. If the key is not present, add it.

Consult the readme documentation for Patch 4563 on how to apply this patch

Run Patch 4563 on the above hosts. Once the patch is installed you will see the

HF4563 folder appear. Run the backup.bat file on each server, found in the HF4563

folder, and if successful, reboot the server.

R:\EMC CONTROLCENTER\HF4563\backup.bat (Repository Server)


S:\EMC CONTROLCENTER\HF4563\backup.bat (StorageScope and Store Server)

T:\EMC CONTROLCENTER\HF4563 (Store Server)

After rebooting, copy the EMC CONTROLCENTER folder to another location.


Example: R:\EMC CONTROLCENTER copy to D:\Rollback-PreUB7\EMC

CONTROLCENTER

S:\EMC CONTROLCENTER copy to D:\Rollback-PreUB7\EMC

CONTROLCENTER

T:\EMC CONTROLCENTER copy to D:\Rollback-PreUB7\EMC

CONTROLCENTER

I recommend using a third party tool such as Robocopy, or EMCopy to ensure that all

files copy to the new backup location successfully.

Example:

emcopy R:\EMC CONTROLCENTER D:\Rollback-PreUB7\EMC

CONTROLCENTER /s /d /o /r:0 /w:1 /log+:c:\EMC ControlCentercopy.txt

2.10.16 Update Bundle 7 (Patch 4673) Execute patch6107254_4673.exe on the A node host first.


Click Next,

Follow the on screen prompts. This will take around an hour or so to complete!

2.10.17 Patch 4744 Consult the EMC readme file before applying this patch. You must have EMC

ControlCenter 6.1.0.7.254 installed before you apply Patch 4744 which upgrades the

following ControlCenter 6.1 components.

.

• (v,6.1.0.7.289) ControlCenter Server

• (v,6.1.0.7.289) ControlCenter Store

• (v,6.1.0.7.289) ControlCenter console

2.10.18 Patch 4754 This patch fixes an issue with the Ramb_hotback.bat script, which failed to delete archive files older than 2 days. Follow the readme document supplied with this patch. Make sure the patch completes successfully.

2.10.19 Add local ORA_DBA Group to each Cluster Node Add a local Group called ORA_DBA to each node. This is a solution to a problem

described in EMC203001


2.10.20 Post UB7 Cluster Preparation The cluster nodes need to be made ready for ControlCenter to be able to run on

either node. Perform the following procedure on the active node:

In the Services window, select the following services:

OracleServiceEMCSTSDB

OracleEMC CONTROLCENTERREP_HOMETNSListener

OracleServiceRAMBDB

Select Stop. 1. Change directory to R:\EMC CONTROLCENTER\tools\utils

2. Run clustersave.bat.5Upgrading ControlCenter 5.2 to 6.1 3. Using the clusteradmin utility, initiate a manual failover of the

EMC CONTROLCENTERRG resource group to the B node.

Right-click resource group EMC CONTROLCENTERRG and choose Move

Group.


From the B node, do the following:


2. Run clusterupgrade.bat. 3. Reboot the B node for the upgrade to take effect.

Use the Move Group command again to bring the Cluster Resources back under the

A node again.

2.10.21 Confirm Master Agents are at the same patch level Check that the Master Agent on both Nodes of the Cluster are at the same version.

You will see that the Active node of the cluster will show the Master Agent as Active,

but it will be Inactive on the B Node. This is expected. If the Master Agent on the B

Node has not upgraded to 6.1 UB7, then do the following:

On the Node that is not active (the Master Agent is not active on that node), rename

all files with .COMFILE and .lck extensions found in the C:\Windows\EMC

CONTROLCENTER directory.

Fail over the Cluster to restart the Master Agent on that node. The .COMFILE and

.lck files will be recreated by the Master Agent when it starts. The Master Agent

should now be active against that Node in the ControlCenter Console.

On the other Node (the Node that is now not active), rename all files with .COMFILE

and .lck extensions in the C:\Windows\EMC CONTROLCENTER directory.

Fail over the Cluster again back to the A node, and the Master Agent will create the

.COMFILE and .lck files again. The Master Agent should now be at the same patch

level on both nodes.

2.10.22 Agent Patching Patch the Agents using the ControlCentre Console. The Master Agent must be

patched first! Select the hosts where the Agents are to be installed. As a general rule

do not install the FCC Agent on the same host as the NAS agent because of port

conflicts. Note that the Cluster nodes Do Not have any Agents installed apart


from the Master and Host agents. The following table shows what I consider to be a

good spread of agents:

Host Server Agent

Site-emccc02

NAS Agent, Clariion Agent (requires Naviseccli), Symmetrix Agent, SDM Agent, Master Agent, Host Agent; IntegrationGateway (if sending alerts to third party management system)

Site-emccc03 Symmetrix Agent, SDM Agent, Master Agent, Host Agent, Site-emccc04 FCC Agent, Master Agent, Host Agent Site-emccc05 FCC Agent, Master Agent, Host Agent

The Agents should upgrade to version 6.1.0.7.254

2.11 Install ControlCenter Licences

Add the new EMC CONTROLCENTER 6.1 Licenses using the Console Install Menu.


2.12 Create ControlCenter Resources for the Cluster Service

Create the EMC CONTROLCENTERRG resources shown below. Use the Generic

Service. Each has a dependency on its predecessor. It is good practice to number

these Resources in the order in which they start. The Resources listed from 01 to 02

as shown below will already be created at this stage.

The Dependencies of each Resource are shown graphically below:






Note the API server is not generally required so this may be omitted.


The screen shot below shows the ControlCenter resources in the EMC

CONTROLCENTERRG in an Online state. Make sure this is the case.

This now completes the installation of the cluster. Execute a Move Group and check

that the EMC CONTROLCENTERRG moves to the other Node. Make sure all EMC

Services come online. You will have to restart the Store(s) manually after a Move

Group is executed.

2.12.1 Load StorageScope Connect to the StorageScope Server via a web browser as follows: https://site-emccc02:30002/srm


2.12.2 Install the WLA Archive Agent If performing an upgrade to EMC CONTROLCENTER6.1 from 6.0, this will already

be in place.

Install the WLA Archiver Agent on the third HPDL 380 Server. Configure the WLA

Archive and WLA Reports as folders on the T: drive.


2.12.3 Install the NAS Agent If required, install the NAS Agent on the same server as the Storage Scope server, in

this instance this will be dtcp-emccc02

Once installed, navigate to the agent folder CNN610 and change the entry in the

CNN.INI file under the [SSH] heading from Version = 1 to version = 2

Restart the NAS Agent.

Make sure the NAS is discovered using the Assisted Discovery wizard in

ControlCenter. Use the emcplink.exe utility if there are connection problems. This

can be found in the CNN610 Agent folder.


Next, run the following commands:

emcplink -setpolicy +EMC_SSH_KEY_SECURITY_NONE

emcplink -ssh -interactive -2 -pw nasadmin nasadmin@<IP Address of Control Stn>

Where nasadmin is the user logon, and password of nasadmin. Substitute the correct

values if different. This will set up an ssh session with the Control Station and log in.

Disconnect from the session.

Retry the Assisted Discovery and the Celerra™ should now discover.

2.12.4 Add Gatekeepers to Symmetrix® Agent Host Zone in and mask 6 different Gatekeeper devices for each Symmetrix Array, to each

HBA on the Symmetrix Agent hosts.

References and Related Documentation

Document / Material Location (Hyperlink/URL) 1 ControlCenter 6.1 Planning

Guide https://powerlink.emc.com/nsepn/webapps/btg548664833igtcuup4826/km/live1/en_US/Offering_Technical/Technical_Documentation/300-006-387.pdf?mtcs=ZXZlbnRUeXBlPUttQ2xpY2tDb250ZW50RXZlbnQsZG9jdW1lbnRJZD0wOTAxNDA2NjgwNDVjMDlmLGRvY3VtZW50VHlwZT1wZGYsbmF2ZU5vZGU9MGIwMTQwNjY4MDM1YWM5Yl9Hcmlk

Documents

Building a ControlCenter Infrastructure in a Windows 2003