UPGRADING FROM ORACLE ENTERPRISE MANAGER 10G TO CLOUD CONTROL 12C WITH ZERO DOWNTIME

Manageability

1 Session # 864

UU PGRADING FROM PGRADING FROM OO RACLE RACLE EE NTERPRISE NTERPRISE MM ANAGER ANAGER 1010 G TO G TO

CC LOUD LOUD CC ONTROL ONTROL 1212 C WITH C WITH ZZ ERO ERO DD OWNTIMEOWNTIME

Leighton L. Nelson, Mercy

Abstract

Oracle Enterprise Manager 12c is the comprehensive enterprise management product specially suited for Oracle infrastructure and applications including Oracle Database Server, Fusion Middleware and Oracle Linux. It has been significantly improved compared to previous versions. However, if you already have a previous version deployed and you need to take advantage of all the features in 12c you need to have a solid path for doing so. Fortunately, Enterprise Manager provides the framework for seamlessly upgrading your environment. In addition, if your currently deployed environment is vital to your IT datacenter operations and you have already invested in significant time and effort into enhancing your environment such as custom reporting, job scheduling and user-defined metrics, then you should ensure that there is minimal disruption to these services. The 2-System Method Upgrade method provides the framework for achieving these goals.

In this paper we will take a look at how Mercy used this method to migrate from AIX 6.1 to Red Hat Enterprise Linux 6.2, upgrade repository database version from 11.1.0.6 to 11.2.0.3 while upgrading to Enterprise Manager from Grid Control 10g to Cloud Control 12c.

Target Audience

This paper will benefit those responsible for configuring, managing and monitoring Oracle databases and Fusion Middleware components using Oracle Enterprise Manager. These are typically database administrators or enterprise manager administrators.

Executive Summary

Learners will be able to:

• Learn about the different upgrade methods for Enterprise Manager Cloud Control 12c

• See how the 2-System Method is used to migrate from an older version of Enterprise Manager to Cloud Control 12c with minimal downtime.

• Identify pitfalls and lessons learned from the upgrade.

Background

Our existing Enterprise Manager Grid Control 10g environment had the following configuration.

Current Architecture (IBM JS21 Blades)

• Single Management Service – 10.2.0.5 / AIX 6.1

• Single Instance Repository Database – 11.1.0.6 / AIX 6.1

• Agents – 100 (AIX + Linux)

• Targets – 500+

• Proposed Architecture (VMware vSphere)

• OMS – 12.1.0.2 / RHEL 6.2

• Repository Database – 11.2.0.3 / RHEL 6.2

Manageability

2 Session # 864



Proposed Architecture (VMware vSphere)

• 2 Management Services (load balanced) – 12.1.0.2 / RHEL 6.2

• Repository Database with local Data Guard – 11.2.0.3 / RHEL 6.2



Technical Discussions and Examples

1. Install Pre-Upgrade Console

PSU 3 or greater required to be installed. Check the inventory for applied patches and PSUs using opatch. In the example below. oemhost1[emrep]-/home/oracle >opatch lsinventory -bugs_fixed|grep PSU 9174904 9702579 Wed Nov 23 12:37:30 CST 2011 ENTERPRISE MANAGER OMS 10.2.0.5.2 PSU 9282397 9702579 Wed Nov 23 12:37:30 CST 2011 ENTERPRISE MANAGER OMS 10.2.0.5.3 PSU WITH OMS PSU3 INSTALLED 9702579 9702579 Wed Nov 23 12:37:30 CST 2011 ENTERPRISE MANAGER OMS 10.2.0.5.4 PSU ? EM CONTENT (PSU-4) JUL2010

Shutdown the OMS oemhost1[emrep]-/home/oracle >$OMS_HOME/bin/emctl stop oms Oracle Enterprise Manager 10g Release 5 Grid Control Copyright (c) 1996, 2009 Oracle Corporation. All rights reserved. Oracle Management Server is Down.

Set ORACLE_HOME to OMS_HOME oemhost1[emrep]-/u05/orastage/14375077 >export ORACLE_HOME=$OMS_HOME

Go to patch location and run opatch oemhost1[emrep]-/u05/orastage >which opatch /u04/app/oracle/oms10g/OPatch/opatch oemhost1[emrep]-/u05/orastage >cd 14375077 oemhost1[emrep]-/u05/orastage/14375077 >opatch apply End of patch should say succeeded. Return Code = 0 OPatch succeeded.

oemhost1[emrep]-/u05/orastage/14375077 >cd $OMS_HOME/sysman/preupgc oemhost1[emrep]-/u04/app/oracle/oms10g/sysman/preupgc >sqlplus /nolog SQL*Plus: Release 11.2.0.1.0 Production on Fri Sep 14 11:30:21 2012 Copyright (c) 1982, 2009, Oracle. All rights reserved. SQL> conn / as sysdba Connected. SQL> @puc_dblink_pkgdef.sql

Manageability

3 Session # 864

Package created. No errors. Package created. SQL> @puc_dblink_pkgbody.sql Package body created. No errors. Package body created. SQL>@pre_upg_console.sql … 1 row merged. optimized l_target_guids.COUNT : 4 oracle.sysman.beacon SMPADDON 10.2.0.5.0 Oracle Beacon Plugin oracle.sysman.emrep SMPADDON 10.2.0.5.0 Oracle Management And Repository Plugin oracle.sysman.oh SMPADDON 10.2.0.5.0 Oracle OracleHome oracle.sysman.csa SMPADDON 10.2.0.5.0 Oracle Client System Analyzer oracle.sysman.db SMPADDON 10.2.0.5.0 Oracle Database Plugin oracle.sysman.emas SMPADDON 10.2.0.5.0 Oracle Fusion Middleware Plugin oracle.sysman.empa ADDON 10.2.0.5.0 Oracle Siebel Plugin oracle.sysman.vt ADDON 10.2.0.5.0 Oracle Virtualization Plugin oracle.fmw.bhv ADDON 10.2.0.5.0 Oracle Beehive Plugin PL/SQL procedure successfully completed. PL/SQL procedure successfully completed. PL/SQL procedure successfully completed. Commit complete.

Start OMS oemhost1[emrep]-/u04/app/oracle/oms10g/sysman/preupgc >$OMS_HOME/bin/emctl start oms Oracle Enterprise Manager 10g Release 5 Grid Control Copyright (c) 1996, 2009 Oracle Corporation. All rights reserved. opmnctl: opmn is already running Starting HTTP Server ... Starting Oracle Management Server ... Checking Oracle Management Server Status ... Oracle Management Server is Up.

Manageability

4 Session # 864

2. Log into the EM Console and click on the Deployments tab.

A new Upgrade procedure and Enterprise Manager 12c Upgrade Console link appears (see Figure 1).

Figure 1 – Enterprise Manager 12c Upgrade Console link after apply Pre-Upgrade patch

Figure 2 below show the Pre-Upgrade Console

Manageability

5 Session # 864

Figure 2 – Upgrade Console

Manageability

6 Session # 864

3. Select the 2-System radio button to activate the procedures for this method.

4. Identify Host and Port for New Enterprise Manager System

Enter the fully qualified domain name of the target host and OMS HTTPS Upload Port and OMS HTTP Upload Port – 4899 and 4889 by default. If you are using a Server Load balance you should also check the associated box.

You may also disable the deferred data migration and accrued data migration jobs by checking the respective boxes. Click Save.

Figure 3 – Identify Host and Ports for new OMS

Manageability

7 Session # 864

5. Manage Software

Click the Manage Software Link in the Upgrade Console page. This allows you to define the path of the plug-ins and agents downloaded from OTN (see Figure 4). This should preferably be a shared location accessible by the new and old OMS. Plug-ins will be upgraded during the new OMS installation.

Figure 4 – Manage Software and plug-ins

6. Enter the path of the downloaded plug-ins and agent software for the platforms you want to manage. From the OTN page (see Figure 5) follow steps carefully to select the plug-ins and agents needed for your path.

Manageability

8 Session # 864

Figure 5 – Agent Software and Plug-ins for OEM upgrade

In this case we are following Path 1 – Upgrade from EM 10.2.0.5/11.1.0.1 to EM 12.1.0.2

The directory listing below shows all the files downloaded and staged. oemhost1[emrep]-/mnt/em_upgrade/em12cr2_plugins >ls -ltr total 2019520 -rw-r--r-- 1 oracle dba 1762394 Sep 14 10:03 12.1.0.2.0_oracle.sysman.oh_2000_0.opar -rw-r--r-- 1 oracle dba 119951825 Sep 14 10:06 12.1.0.2.0_oracle.sysman.beacon_2000_0.opar -rw-r--r-- 1 oracle dba 11758937 Sep 14 10:07 12.1.0.2.0_oracle.sysman.mos_2000_0.opar -rw-r--r-- 1 oracle dba 298841 Sep 14 10:07 12.1.0.2.0_oracle.sysman.emrep_2000_0.opar -rw-r--r-- 1 oracle dba 328765196 Sep 14 10:12 12.1.0.2.0_oracle.sysman.db_2000_20120804.opar -rw-r--r-- 1 oracle dba 333968065 Sep 14 10:16 12.1.0.2.0_AgentCore_212.zip -rw-r--r-- 1 oracle dba 233396034 Sep 14 10:17 12.1.0.2.0_AgentCore_226.zip

Manageability

9 Session # 864

7. Once the files have been downloaded and staged, you can add this path to software location and click Validate button. This will submit a job that will refresh and validate the targets for upgradeability.

Note: Agent Upgradeability and Target Upgradeability charts should be displayed in green. This will signify that all targets can be upgraded. If not, further remediation may be required before continuing with the rest of the upgrade.

Figure 6 – Refresh And Validate Target job submitted

8. Check the Section Under Software Required for Upgrade (see Figure 7) to verify that all plug-ins listed has been downloaded and staged. If there are missing plug-ins or agents you should download and stage them in the directory path specified by Software Location and retry the Validation.

Figure 7 – Verify Agent and Plug-in Software

Manageability

10 Session # 864

At this point there are two options

1) Backup repository before deploying the Agents

2) Deploy agents and then take the repository backup

The option you choose will depend on the number of targets you have to migrate, how often agents are added and deleted, and the duration of elapsed time before performing agent switchover. If the environment is fairly static either option can be chosen. If there are expected changes during the migration over a period of at least a week you may want to perform the backup after doing the agent deployment. Here we will deploy and configure agents before taking a backup of the repository database.

Deploy and Configure Agents

Since only 12c agents can communicate with 12c OMS, you must deploy the 12c agents before upgrading your OMS. Go to the Deploy and Configure Agents section of the Upgrade Console to deploy the agents to the existing targets.

Note: Agent deployment will create host level jobs that require the software installation owner i.e. oracle to be able to access the servers. If you don’t use local authentication (via /etc/passwd) then you should work with your systems administrator to setup PAM for emagents. (PAM modules exist for ldap, Kerberos, secureid etc.)

Add the agents you want to deploy to the procedure. You can filter agents by platform or version to limit a set of agents to be deployed. Enter the path for the agent base and instance directories (see Figure 8).

Note: The top-level directory chosen should either be owned by oracle or root. This may require installing the agent on a different filesystem.

Note: The procedure automatically uses ‘agent12g’ instead of ‘agent12c’ in the agent base and instance paths. You may want to use the latter nomenclature.

There is also the option of using the same path for all hosts (recommended). In some cases you may have to use separate paths.

Provide the agent credentials for the deployment. This is typically ‘oracle’. If you use a sudo or Power Broker for security purpose you may also check these settings.

Select the agents to be deployed and configured and click Next.

Manageability

11 Session # 864

Figure 8 – Deploy and Configure Agents

Manageability

12 Session # 864

Enter the root credentials to run the root.sh script. If you don’t have access to root credentials you can still enter any dummy username/password here. Deployment will still succeed but you will have to manually run root.sh scripts later.

Figure 9 – Pre-Deploy Options for Agent

Click Submit to create the Deploy agent job (see Figure 10).

Figure 10 – Submit Job to Deploy and Configure Agents

Manageability

13 Session # 864

Clicking on the job link will take you to the screen where you can monitor job progress (see Figure 11).

Figure 11 – Job Activity progress for Agent Deployment

If the valid root password was not provided during agent deployment you should run the root.sh scripts manually. If not ,you may get errors while executing host jobs using the new agent. (ERROR: NMO Not Setuid-root)

Backup repository database

In the case of migrating between similar platforms you can also use DBCA (recommended) or RMAN backups.

However, in order to do a platform migration from AIX to Linux only two options available.

1. Datapump Import/Export

2. Cross Platform Transportable Tablespaces

Note: Using either option above requires that both source and target databases use the same character set.

See Oracle Database Server 11g documentation on using Cross-Platform tablespaces.

For cross-platform migration refer to MOS note 388090.1

Both these cases require a pre-existing database that is certified with the 10.2.0.5 or 11.1 version of Enterprise Manager.

See Appendix A for database backup details.

Oracle EM 12.1.0.2 provides a seed database template which is already pre-configured for Enterprise Manager 12c installation. The template requires Oracle Database 11.2.0.3 software to be installed on the target server. For details on using the template refer to the Oracle Enterprise Manager Cloud Control Basic Installation Guide.

Before performing the repository backup ensure that the emkey has been copied to the repository using the following command. emctl config emkey –copy_to_repos –sysman_pwd <password>

Manageability

14 Session # 864

Once the database has been backed up successfully, verify that SYSMAN schema doesn’t contain invalid objects SELECT count(*) from user_objects WHERE status=‘INVALID’;

Provide Repository backup details

Enter the date/time when the repository backup was taken. This is used by the ADDMP and DDMP after the OMS is upgraded. You can find use the timestamp of the data pump log files to get the backup time.

Figure 12 – Repository Database Backup Date/Time

Upgrade OMS and Repository Manually

Next step will be to Upgrade the OMS and Repository.

It is important to verify that all your pre-requisites are met for the OMS and repository before beginning the upgrade.

Repository Database Pre-requisites

• Verify database initialization parameters

• PROCESSES parameter is at least 300. The installer can also make the changes if desired.

• Set JOB_QUEUE_PROCESSES to 0

• Set SHARED_POOL_SIZE to 700MB or greater

• PGA_AGGREGATE_TARGET should be at least 1GB

• SGA_TARGET should be at least twice PGA_AGGREGATE_TARGET

• Grant execute on DBMS_RANDOM to DBSNMP and SYSMAN

• Unlock ORACLE_OCM account

• Increase your redo log size to at least 300MB due to the high level of transactions during the repository.

• Check to make sure your emkey is in the repository

Tip: Create guaranteed restore point on both the old and new repository database so that changes can be easily flashed back; create restore point PRE_OMS_UPGRADE guarantee flashback database;

EM Prerequisite Kit Utility

EM Prerequisite Kit Utility can verify that the database pre-requisites are met before starting the upgrade. The example below shows the pre-requisites that will be checked. See Appendix B for additional details. EM12c_STAGE_DIR/install/requisites/bin/emprereqkit -executionType upgrade -prerequisiteXMLLoc install/requisites/list -connectString "(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=dbhost1)(PORT=1521)))(CONNECT_DATA=(SID=emrep)))" -dbUser SYS -dbPassword -dbRole sysdba -showPrereqs

Manageability

15 Session # 864

OMS Server Verification

• Verify that server meets pre-requisites

• Check ulimit to make sure it’s at least 4096

• ulimit –n

• If using RHEL 6.2 or Oracle Linux 6.2 – make sure you have the following package installed or else your installation may fail at OMS Configuration Assistant.

• glibc-2.12-1.47.0.2.el6_2.12.x86_64

• See MOS note: EM12c: Installation Fails On Linux 6 at "Start Oracle Management Service" step (Doc ID 1472216.1) for workaround

Start runInstaller

cd /stage/

./runInstaller

Figure 13 – Run Upgrade on the new OMS Server using the OEM12c OUI

Manageability

16 Session # 864

It is recommended to download software updates during upgrade. See MOS Note 1099123.1 for details on the “Auto Update” feature. This is not selected in the figure 14 below.

Figure 14 – Check for Software Updates

Manageability

17 Session # 864

Provide the location for the inventory and choose the OS group for the software owner. This will typically be ‘oinstall’ or ‘dba’ depending on your environment.

Figure 15 – Provide Inventory location and Select OS group

Manageability

18 Session # 864

Verify that all pre-requisites have been met. If there are any warnings or errors, examine and fix them before proceeding with the installation. In Figure 16 below, the glibc pre-requisite check failed. This is a known issue. See workaround in MOS note 1472216.1. This may not be applicable to more recent versions of Linux.

Figure 16 – Check Pre-requisites

bash $ rpm -qa|grep glibc-2.12-1.47 glibc-2.12-1.47.el6.x86_64 glibc-2.12-1.47.el6.i686

Manageability

19 Session # 864

The installer is checking for an exact match for glibc-2.12-1.47.0.2.el6_2.12.x86_64. Install this package if possible before continuing with the installation. If not, further workarounds may be necessary before completing the installation. In Figure 17 below, it is ignored to complete the demonstration.

Figure 17 – Pre-requisite Check for Packages Failed

Choose the “Upgrade an Existing Enterprise Manager System” option for Two System Upgrade as shown in Figure 18.

Manageability

20 Session # 864

Figure 18 – Select Upgrade Method

Manageability

21 Session # 864

Provide the repository database fully qualified hostname, port, service/SID as well as SYS and SYSMAN passwords.

Figure 19 – Repository Database Connection Details

Review any errors discovered in the configuration by reviewing the log files from the emprereqkit.

Manageability

22 Session # 864

Verify that all default plug-ins are selected as seen in figure 20 below. Only select plug-ins which are already available and downloaded into a location accessible by the OMS.

Figure 20 – Plug-in Deployment

Manageability

23 Session # 864

Provide a connect descriptor for the old repository database and the password for the SYSMAN user on the pre-12c repository. You may locate the repository connection information on the pre-12c OMS server by checking the $OMS_HOME/sysman/config/emoms.properties file for the oracle.sysman.eml.emdRepConnectDescriptor value with leading slashes removed (see Figure 21).

Figure 21 – Provide Pre-12c Repository Database Connection Information

Provide the path for the datafile used for the JVM Diagnostic Data Tablespace.

Manageability

24 Session # 864

Figure 22 – JVM Diagnostics Data Tablespace

Manageability

25 Session # 864

Review and verify the ports to be used for the new OEM system (see Figure 23). You can also check the target system to verify if the ports are in use or being blocked by a firewall. In the case of the latter, you should arrange with your network or security administrator to open up the required ports.

Figure 23 – OMS Port Summary

Review the installation summary.

Manageability

26 Session # 864

Figure 24 – Installation Summary Review

At the end of the installation you can verify whether if the OMS is running. [oracle@omshost bin]$ /u01/app/oracle/Middleware/oms/bin/emctl status oms -details Oracle Enterprise Manager Cloud Control 12c Release 2 Copyright (c) 1996, 2012 Oracle Corporation. All rights reserved. Enter Enterprise Manager Root (SYSMAN) Password : Console Server Host : omshost.example.com HTTP Console Port : 7788 HTTPS Console Port : 7802 HTTP Upload Port : 4889 HTTPS Upload Port : 4899 EM Instance Home : /u01/app/oracle/Middleware/gc_inst/em/EMGC_OMS1 OMS Log Directory Location : /u01/app/oracle/Middleware/gc_inst/em/EMGC_OMS1/sysman/log OMS is not configured with SLB or virtual hostname Agent Upload is unlocked. OMS Console is unlocked. Active CA ID: 2 Console URL: https://omshost.example.com:7802/em Upload URL: https://omshost.example.com:4899/empbs/upload WLS Domain Information Domain Name : GCDomain

Manageability

27 Session # 864

Admin Server Host: omshost.example.com Managed Server Information Managed Server Instance Name: EMGC_OMS1 Managed Server Instance Host: omshost.example.com WebTier is Up Oracle Management Server is Up

Go the the Console URL shown in the output and login as an Administrator

Figure 25 – OEM 12c Console

You are greeted with the License Agreement. Please Read and Accept the agreement to continue.

Figure 26 – License Agreement

Manageability

28 Session # 864

Choose your default Home Page.

Figure 27 – Default Home Page Selection

Manageability

29 Session # 864

You can preview home page before selecting it. This can be changed later under Username menu -> Select My Home.

Figure 28 – Default Home Page Preview

Manageability

30 Session # 864

Go to Pre-Upgrade Console in old EM to create a link between the old and new EM repository as shown in figure 29. The connect descriptor can be found by looking at Connect descriptor found in MW_HOME/gc_inst/em/EMGC_OMS1/emgc.properties

The link between the two repositories allows operations performed on the old repository to be propagated to the new repository.

Figure 29 – Create Repository Link

Agent Deployment

In order for the targets to be monitored in the new environment it is required that the EM12c agents be deployed and configured to the targets. After that is done we will need to generate and sign off on health check reports before switching over the agents to the new EM System. While this is being done all agents are still monitored up until the point of switcher. This stops the 10g agents and starts the 12c agent causing minimal disruption.

From the Upgrade Console Home page click on the name of the phase to Generate Health Report of Deployed Agents if the agents have already been deployed. If not, click on the Deploy and Configure Agents link to first deploy the agent.

Figure 120 – Agent Deployment

Manageability

31 Session # 864

Follow the steps in the workflow to Generate Health Reports. Select agents, which have been deployed by either loading from previous operation or by searching. Next select the checkboxes for the agents and provide the software owner username and password as shown in figure 31 then click the submit button to create the HealthCheck job.

Figure 31 – Generate Health Reports

Manageability

32 Session # 864

Once submitted, you can click on the job name to view its status (see figure 32).

Figure 32 – Health Report Job Status

Manageability

33 Session # 864

Figure 33 – Health Report Job Completed

After job completes successfully check and sign off on health reports. Select agents that had the health checks job was completed (Health Check Status should be Pending Verification Report) then click on the “View and Verify Health Check Reports” button as shown in figure 34 below.

Figure 34 – Health Check Report Verification

Manageability

34 Session # 864

Check that the Ping Test is passed and that Broken Targets is zero. There should also be enough free space in the state directory. To view detailed information such as metrics and target properties, select the agent and click on the View Detailed Report button.

Note: You can only view the detailed report of a single agent at a time.

Figure 35 – Sign off Health Report

Select the targets and click on the Verify and Sign Off button. Once done the User Verified column will contain green checks as shown below in figure 36.

Figure 36 – Verify and Sign Off

Switch Agents.

Before agent switchover, log into Enterprise Manager 12c Console and verify that the targets are in Pending Activation state as shown in figure 37 below.

During switchover, the pre-12c agents will be shutdown and the 12c agents will be started.

Manageability

35 Session # 864

Figure 37 – Targets in Pending Activation State in EM 12c Console

Manageability

36 Session # 864

From the Upgrade Console in the pre-12c OEM, add the targets that you want to switch over. You can either add targets manually or load targets from a previous operation.

Figure 38 – Agent Switchover

Select the targets and click on the Submit button to submit the agent switchover job (see figure 39).

Note: If you have an environment with many agents (greater than 100) you may want to consider doing the switchover in batches. For example, you could switchover non-production agents first and validate that they are working correctly first before doing production agents.

Manageability

37 Session # 864

Figure 39 – Agent Switchover Job Submitted

Clicking on the job name will allow you to view the status of the job.

Figure 40 – Switchover Job Activity

Manageability

38 Session # 864

Switched over agents now appear up and running in the OEM 12c Console as seen in figure 41.

Figure 41 – Switched Over Targets

These now appear in 10g EM as unreachable.

Figure 42 – Targets Unreachable in Pre-12c OEM Console

Post Upgrade

Stop the OCM Scheduler configured in the OMS home. oemhost1[emrep]-/home/oracle >export ORACLE_HOME=$OMS_HOME oemhost1[emrep]-/home/oracle >$ORACLE_HOME/ccr/bin/emCCR stop Oracle Configuration Manager - Release: 10.3.7.0.0 - Production

Manageability

39 Session # 864

Copyright (c) 2005, 2012, Oracle and/or its affiliates. All rights reserved. ------------------------------------------------------------------ Oracle Configuration Manager Stopped...

Delete Obsolete Targets

The following targets need to be deleted manually.

• EM Website

• EM Website System

• Oracle Application Server target of 10g OMS ORACLE HOME consisting of:

• Oracle Application Server

• OC4J

• Oracle HTTP Server

• Web Cache

Go to the All Targets page, right-click on the target name, select Target Setup -> Remove Target. Repeat the process for eacht target to be removed.

Figure 43 – Delete Obsolete Targets

By default Enterprise Manager 12c enables default incident Rule Sets. These may generate a lot of unnecessary alerts in your environment. Disable the default Incident Rulesets by clicking on Setup -> Incidents -> Incident Rules.

Figure 44 – Incident Rule Sets

Manageability

40 Session # 864

Select the Incident management Ruleset for all targets. Click on Actions and choose Disable from the drop-down menu as shown in figure 46.

Figure 46 – Disable Default Incident Rules

You may get a pop-up message such as shown in figure 47. Click OK. The Ruleset will be disabled.

Figure 47 – Information Pop-up After Disabling default Incident management Ruleset

Manageability

41 Session # 864

Verify that the Enabled column has been changed to “No”.

Perform Post-Upgrade Tasks from within the OEM 12c Console. Go to Setup -> Manage Cloud Control ->Post Upgrade Tasks.

Figure 48 – Post Upgrade Console

Deferred Data Migration • Data is migrated from the old EM system format the 12c format. A job is automatically submitted that runs in the

background after the EM system is up and running.

• The time taken to migrate data depends on the amount of data available in your old EM. Critical data is migrated first.

• Historical data is migrated after EM Cloud control is functioning.

Figure 49 – Deferred Data Migration Jobs

Accrued Data Migration Accrued Data is data that has been collected on the agents between the repository backup and agent switchover. To enable this data to be migrated, the ADMP jobs will automatically run if not disabled. You can also manually submit the ADMP job and track its status from the screen as shown in figure 50 below.

Manageability

42 Session # 864

Figure 50 – Accrued Data Migration Jobs

View Diff Reports

If any agents are added or removed from the old EM they will be shown in the Diff Report as shown in figure 51. Also any added jobs or Software Library entities will be displayed.

Figure 51 – Diff Reports

Once agents have been successfully migrated and validated, they can be removed by using the Sign Off tab in the Post Upgrade Console.

All jobs, reports, user-defined metrics (metric extensions) remained in tact. Also notifications, rules, and monitoring templates continued to work in the new environment.

Figure 52 below compares the custom reports in the EM 12c environment on the left with the EM 10g environment on the right.

Manageability

43 Session # 864

Figure 52 – Reports Comparison in OEM 10g and OEM 12c

Summary

We have seen how to perform a step-by-step migration and upgrade of Enterprise Manager Grid Control 10g environment to Enterprise Manager Cloud Control 12c environment involving a database upgrade and platform migration. The Enterprise Manager Console provides the workflow to accomplish this, which resulted in near zero downtime. During the entire workflow, all targets were still being managed and monitored by either the old EM before switchover or the new EM after the switchover.

Manageability

44 Session # 864

Appendices

Appendix A – Repository Backup using Data Pump

1. Copy emkey to repository before taking backup

a. $OMS_HOME/bin/emctl config emkey -copy_to_repos -sysman_pwd <password>

Oracle Enterprise Manager 10g Release 5 Grid Control

Copyright (c) 1996, 2009 Oracle Corporation. All rights reserved.

The Em Key is already present in the Management Repository.

b. Verify using emctl status emkey

[oemhost1]-/u02/orastage/em12c >$OMS_HOME/bin/emctl status emkey

Oracle Enterprise Manager 10g Release 5 Grid Control


Please enter repository password:

The Em Key is configured properly, but is not secure. Secure the Em Key by running "emctl config emkey -remove_from_repos".

To verify if the emkey is copied properly, log in to the repository as sys user and run the following sql command: SQL> select count(*) from MGMT_REPOS_TIME_COEFFICIENT; COUNT(*) ----------

1

(If the count is 1, then the emkey is copied.)

2. Use Datapump export/import method to migrate repository from 11gR1 database on AIX to 11gR2 database on RHEL 5 as described in MOS note: 388090.1

3. Remove EM Grid Control or DB Control repository from new target repository database if one exists.

To remove an existing EM Grid Control repository from the target database use RepManager utility to remove the objects from the database and drop the repository. $oemhost01>-$OMS_HOME/sysman/admin/emdrep/bin/RepManager dbhost1 1521 emrep -sys_password password -action drop

Enter repository user name : sysman

Getting temporary tablespace from database...

Found temporary tablespace: TEMP

Checking SYS Credentials ... rem error switch

OK.

rem error switch

Dropping the repository..

Checking for Repos User ... Exists.

Repos User exists..

Clearing EM Contexts ... OK.

Dropping EM users ...

Manageability

45 Session # 864

Done.

Dropping Repos User ... Done.

Dropping Roles/Synonymns/Tablespaces ... Done.

Dropped Repository Successfully.

The output of the SQL commands below when run in the target repository as a DBA user should not return any rows. If so, you may need to contact Oracle Support. select username,account_status from dba_users where username in ('SYSMAN', 'SYSMAN_MDS','MGMT_VIEW','SYSMAN_BIP','SYSMAN_APM','BIP','SYSMAN_OPSS','SYSMAN_RO') ; select owner,synonym_name from dba_synonyms where table_owner in ('SYSMAN', 'SYSMAN_MDS','MGMT_VIEW','SYSMAN_BIP','SYSMAN_APM','BIP','SYSMAN_OPSS','SYSMAN_RO') ; select tablespace_name from dba_tablespaces where tablespace_name like 'MGMT%'; select comp_name from SCHEMA_VERSION_REGISTRY;

NOTE: Not all objects dropped:

See MOS NOTE: 12c Cloud Control Repository: RepManager Dropall Command Fails To Drop all the Repository Objects [ID 1365820.1]

To remove DB Control, use the emca utility from the database home.

Appendix B – EM Prerequisite Kit

http://docs.oracle.com/cd/E24628_01/install.121/e22624/appdx_about_emprereqkit.htm#sthref430

The Enterprise Manager installer runs pre-requisite checks on the repository automatically during installation or upgrade. It can also fix some pre-requisites checks that fail before running the installer and revert changes afterwards. E.g. job_queue_process will be changed to 0 before installation/upgrade and reset back to original value after installation.

You can manually run the pre-requisite checks using the empreqkit command line.

[oracle@oemhost1 12.1.0.2]$ install/requisites/bin/emprereqkit -executionType upgrade -prerequisiteXMLLoc /orasource/Linux/EM/12.1.0.2/install/requisites/list -connectString "(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=dbrephost1)(PORT=1521)))(CONNECT_DATA=(SID=emrep1)))" -dbUser SYS -dbPassword password -dbRole sysdba -showPrereqs

Checking Temp space: must be greater than 400 MB. Actual 1922 MB Passed

Checking swap space: must be greater than 150 MB. Actual 3287 MB Passed

Preparing to Bootstrap Please wait ...

BootStrap To /tmp/emprereqkit_091712161748_BS Completed

Oracle Enterprise Manager Cloud Control 12c Release 12.1.0.2.0


Running EMPrereqKit ............

Result of prerequisite list, for scenario upgrade :

--------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Prerequisite Name | Status | Corrective Action Status | Post Corrective Action Status

--------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Manageability

46 Session # 864

Check for invalid objects in the EM repository | NE | NA | NA |

Check the CBO stats gathering job | NE | NE | NE |

Check nls_length_semantics parameter | NE | NA | NA |

Check for the EM Target policy | NE | NA | NA |

Check for automatic UNDO management | NE | NA | NA |

Ensure that the undo tablespace has at least one dataf| NE | NA | NA |

Check for a DBA policy on SYSMAN | NE | NA | NA |

Check DBA jobs running | NE | NA | NA |

Ensure that PK, FK constraints are not disabled | NE | NA | NA |

Ensure that all EM AQ queue tables are enabled for enq| NE | NA | NA |

Ensure that EM triggers are not disabled | NE | NA | NA |

Ensure that the SYSTEM tablespace has at least one dat| NE | NA | NA |

Ensure that the EM tablespace has at least one datafil| NE | NA | NA |

Ensure that the Oracle partitioning option is installe| NE | NA | NA |

Check remote_login_passwordfile parameter | NE | NA | NA |

Check emkey is copied to the repository | NE | NA | NA |

Check policies for MGMT_JOB table | NE | NA | NA |

Check EM_USER_CONTEXT is present in the repository | NE | NA | NA |

Check Abnormal condition in the Audit Master Table | NE | NA | NA |

Check EXEMPT ACCESS POLICY is grants | NE | NA | NA |

Check max_enabled_roles init parameter | NE | NA | NA |

Check execute privilege on DBMS_RANDOM | NE | NA | NA |

Check that no PAF executions are scheduled or running.| NE | NA | NA |

Check if all Agents are secured with latest CA | NE | NA | NA |

Check for the global stale percent | NE | NA | NA |

Check if SYSMAN account is locked | NE | NA | NA |

Check if MGMT_VIEW account is locked | NE | NA | NA |

Check if ORACLE_OCM account is locked | NE | NA | NA |

Check if SYSMAN schema is present for upgrade | NE | NA | NA |

Manageability

47 Session # 864

Check Policies for MGMT_CORRECTIVE_ACTION table | NE | NA | NA |

Check Fine-grained access control option | NE | NA | NA |

Check the Connector Config Table | NE | NA | NA |

Check if Snapshot logs are created on EM tables | NE | NA | NA |

Check if required DBMS packages are valid | NE | NA | NA |

Check the global_name when global_names enabled | NE | NA | NA |

Check the global_name when global_names disabled | NE | NA | NA |

Check the existence of current availability indexes | NE | NA | NA |

Check optimizer sql plan baseline capture. | NE | NA | NA |

Check if temporary tablespace group TEMP exist | NE | NA | NA |

Check for job type uniqueness in MGMT_JOB_TYPE_INFO. | NE | NA | NA |

Check the processes instance parameter value | NE | NA | NA |

Check the memory_target instance parameter value. | NE | NA | NA |

Check the workarea_size_policy instance parameter valu| NE | NA | NA |

Check the session_cached_cursors instance parameter va| NE | NA | NA |

Check the job_queue_processes instance parameter value| NE | NE | NE |

Check the open_cursors instance parameter value. | NE | NA | NA |

Check the log_buffer instance parameter value. | NE | NA | NA |

Check the shared_pool_size instance parameter value. | NE | NA | NA |

Check the db_cache_size instance parameter value. | NE | NA | NA |

Check the large_pool_size instance parameter value. | NE | NA | NA |

Check the java_pool_size instance parameter value. | NE | NA | NA |

Check the streams_pool_size instance parameter value. | NE | NA | NA |

Check the timed_statistics instance parameter value. | NE | NA | NA |

Check the statistics_level instance parameter value. | NE | NA | NA |

Check the redo log size. | NE | NA | NA |

Check the compatible instance parameter value. | NE | NA | NA |

--------------------------------------------------------------------------------------------------------------------------------------------------------------------------

NE = Not Executed NA = Not Applicable

--------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Manageability

48 Session # 864

For more details, check the log file at /orasource/Linux/EM/12.1.0.2/prerequisiteResults/log//LATEST/

Run and Execute prerequisites. [oracle@oemhost1 12.1.0.2]$ install/requisites/bin/emprereqkit -executionType upgrade -prerequisiteXMLLoc /orasource/Linux/EM/12.1.0.2/install/requisites/list -connectString "(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=dbrephost1)(PORT=1521)))(CONNECT_DATA=(SID=emrep1)))" -dbUser SYS -dbPassword password -dbRole sysdba -runPrerequisites

Checking Temp space: must be greater than 400 MB. Actual 1922 MB Passed

Checking swap space: must be greater than 150 MB. Actual 3287 MB Passed

Preparing to Bootstrap Please wait ...

BootStrap To /tmp/emprereqkit_091712162201_BS Completed

Oracle Enterprise Manager Cloud Control 12c Release 12.1.0.2.0


Running EMPrereqKit ............

Result of prerequisite-check , for scenario upgrade :

--------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Prerequisite Name | Status | Corrective Action | Recommendation

--------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Check for invalid objects in the EM repository | Fail | MANUAL | Compile all invalid objects in the EM reposi

Check the CBO stats gathering job | Fail | disable_stats_gather_job | Disable the stats gathering job

Check nls_length_semantics parameter | Pass | NA | NA

Check for the EM Target policy | Pass | NA | NA

Check for automatic UNDO management | Pass | NA | NA

Ensure that the undo tablespace has at least one dataf| Pass | NA | NA

Check for a DBA policy on SYSMAN | Pass | NA | NA

Check DBA jobs running | Pass | NA | NA

Ensure that PK, FK constraints are not disabled | Pass | NA | NA

Ensure that all EM AQ queue tables are enabled for enq| Fail | MANUAL | Enable all EM AQ queue tables for both enque

Ensure that EM triggers are not disabled | Pass | NA | NA

Ensure that the SYSTEM tablespace has at least one dat| Pass | NA | NA

Ensure that the EM tablespace has at least one datafil| Pass | NA | NA

Ensure that the Oracle partitioning option is installe| Pass | NA | NA

Check remote_login_passwordfile parameter | Pass | NA | NA

Manageability

49 Session # 864

Check emkey is copied to the repository | Pass | NA | NA

Check policies for MGMT_JOB table | Pass | NA | NA

Check EM_USER_CONTEXT is present in the repository | Pass | NA | NA

Check Abnormal condition in the Audit Master Table | Pass | NA | NA

Check EXEMPT ACCESS POLICY is grants | Pass | NA | NA

Check max_enabled_roles init parameter | Pass | NA | NA

Check execute privilege on DBMS_RANDOM | Pass | NA | NA

Check that no PAF executions are scheduled or running.| Pass | NA | NA

Check if all Agents are secured with latest CA | Pass | NA | NA

Check for the global stale percent | Pass | NA | NA

Check if SYSMAN account is locked | Pass | NA | NA

Check if MGMT_VIEW account is locked | Pass | NA | NA

Check if ORACLE_OCM account is locked | Fail | MANUAL | ORACLE_OCM account should be unlocked and un

Check if SYSMAN schema is present for upgrade | Pass | NA | NA

Check Policies for MGMT_CORRECTIVE_ACTION table | Pass | NA | NA

Check Fine-grained access control option | Pass | NA | NA

Check the Connector Config Table | Pass | NA | NA

Check if Snapshot logs are created on EM tables | Pass | NA | NA

Check if required DBMS packages are valid | Pass | NA | NA

Check the global_name when global_names enabled | Pass | NA | NA

Check the global_name when global_names disabled | Pass | NA | NA

Check the existence of current availability indexes | Pass | NA | NA

Check optimizer sql plan baseline capture. | Pass | NA | NA

Check if temporary tablespace group TEMP exist | Pass | NA | NA

Check for job type uniqueness in MGMT_JOB_TYPE_INFO. | Pass | NA | NA

Check the processes instance parameter value | Fail | MANUAL | The processes instance parameter needs to be

Check the memory_target instance parameter value. | Pass | NA | NA

Check the workarea_size_policy instance parameter valu| Pass | NA | NA

Check the session_cached_cursors instance parameter va| Pass | NA | NA

Manageability

50 Session # 864

Check the job_queue_processes instance parameter value| Pass | NA | NA

Check the open_cursors instance parameter value. | Pass | NA | NA

Check the log_buffer instance parameter value. | Pass | NA | NA

Check the shared_pool_size instance parameter value. | Pass | NA | NA

Check the db_cache_size instance parameter value. | Pass | NA | NA

Check the large_pool_size instance parameter value. | Pass | NA | NA

Check the java_pool_size instance parameter value. | Pass | NA | NA

Check the streams_pool_size instance parameter value. | Pass | NA | NA

Check the timed_statistics instance parameter value. | Pass | NA | NA

Check the statistics_level instance parameter value. | Pass | NA | NA

Check the redo log size. | Fail | MANUAL | The size of the EM database instance should

Check the compatible instance parameter value. | Pass | NA | NA

--------------------------------------------------------------------------------------------------------------------------------------------------------------------------

NE = Not Executed NA = Not Applicable

--------------------------------------------------------------------------------------------------------------------------------------------------------------------------

For more details, check the log file at /orasource/Linux/EM/12.1.0.2/prerequisiteResults/log//LATEST/

References OTN Enterprise Manager Page Announcing Enterprise Manager Cloud Control 12c Release 12.1.0.1 Bundle Patch 1(BP1) and 12.1.0.2 Plug-ins [ID 1395505.1] Upgrade Advisor: Enterprise Manager 2-System Upgrade from 10.2.0.5 or from 11.1.0.1 to 12.1.0.1 [ID 327.1] EM 12c: Upgrading to Enterprise Manager Cloud Control 12.1.0.1 on Linux x86_64 with the 2-System Method [ID 1367919.1] Grid Control Repository: Steps for Cross Platform Migration of the 10g Grid Control Repository [ID 388090.1] EM12c R2: How to Upgrade to EM 12c R2 - Two System Method [ID 1363162.1] EM 12c R2: How to Apply Upgrade Console Patch for Upgrading to EM 12.1.0.2 [ID 1362537.1] Grid Control Repository: How to Recompile Invalid SYSMAN Objects? [ID 372706.1]

Technology

UPGRADING FROM ORACLE ENTERPRISE MANAGER 10G TO CLOUD CONTROL 12C WITH ZERO DOWNTIME