UKOUG Tech15 - Deploying Oracle 12c Cloud Control in Maximum Availability Architecture

Our Awards:Our Awards:

Zahid Anwar, Senior Oracle DBA Consultant, Version 1

Clive Archibald, Senior Oracle DBA, Telefonica UK

9th December 2015

Deploying Oracle 12c Cloud Control in

a Maximum Availability Architecture

Introducing Version 1

A values-driven organisation that aims to prove that IT

can make a real difference to our clients’

businesses

IT Transformation to deliver business benefit in a

cost efficient manner through our service excellence,

innovation and service improvement

A circa. €75/£53m, 700-strong business with bases

across the UK and Ireland.

Broader and better than niche players, better

service than global players, nearshore and onshore

rather than offshore; values-driven approach to

delivering trusted advice to customers

http://www.businessawardseurope.com/vote/detail/the-business-of-the-year-award-with-turnover-of-26-150/14358

http://www.businessawardseurope.com/vote/detail/the-business-of-the-year-award-with-turnover-of-26-150/14358

Our History

Enterprise Architecture & Change

Microsoft Solutions

Oracle Solutions

Development Technologies and Services

Business Intelligence & Analytics

Infrastructure & Cloud Services

Licence Management

Section 1 - Introducing Version 1<Insert name>4 Main Sectors

3 Key Technology Specialisations 2 Delivery Practices

7 Areas of Deep Expertise

Business Solutions

Managed Services

About Version 1

26%

21%35%

18% Commercial

Financial

Public

Utilities

A bit about myself

• Senior Oracle DBA Consultant

• 10+ years experience

• Recently become an Oracle Certified Master, 2nd in Version 1

• Aspiring to become an Oracle Ace

• Oracle Certified RAC Expert, Exadata and ODA Specialist

• Follow me on:

– facebook.ZedDBA.co.uk (blog)

– twitter.ZedDBA.co.uk or @ZedDBA

– LinkedIn.ZedDBA.co.uk

– www.ZedDBA.co.uk (coming soon!)

facebook.ZedDBA.co.uk

twitter.ZedDBA.co.uk

LinkedIn.ZedDBA.co.uk

http://www.zeddba.co.uk/

A bit about Clive

• Senior Oracle Database Administrator

• 15+ years Oracle experience and 25+ years in IT

• Oracle Certified Professional (OCP) 10g & 11g

• Oracle Certified RAC Expert 10g

• Oracle Certified SQL Expert

• In the process of attaining the OCP 12c

• Keen wildlife photographer

• Follow me on:

– https://uk.linkedin.com/in/clivearchibald

– www.ukfozz.co.uk

https://uk.linkedin.com/in/clivearchibald

http://www.ukfozz.co.uk/

Smart Metering Project

• Large Government Project

• Telefonica won 2/3rds of the UK, +/- 20 million smart meters by 2020

• Database Infrastructure includes a number of:

– Oracle Exadata Machines X4-2

– Oracle Database Appliances X4-2

– Oracle Database Appliances X5-2

Assumptions

• Experienced DBAs who can:

– Understand and deploy RAC

– Understand and deploy Data Guard

• Technical Deep Dive:

– Will go into details for 12c Cloud Control

– But not for RAC and Data Guard

Key Components in a Cloud Control Deployment

Data Centre 2

Data Centre 1

Agents/Targets

Console & EMCLI

OMS

SW Lib

Repository<1 ms

Agents/Targets

JDBC

Oracle’s official way to

“Deploying 12c Cloud Control in a MAA”

with our tweaks

Links to Oracle’s White Paper are available at the end of the presentation

Requirements for 12c MAA

• For each Data Centre:

– Minimum of 2 node RAC database cluster for EM repository database

– Minimum of 2 servers for Oracle Management Servers (OMS)

– Server Load Balancer (e.g. f5 SLB)

– Storage appliance for OMS installations and Software Library

• Needs to support replication between primary and standby sites

• Needs to support snapshots so that point-in-time consistent copies

can be taken

• e.g. ZFS, NetApp, etc

RAC

WebTier Cluster

em.example.com

EMREPDC11

FAS3250FAS3250

Agents/Targets Console & EMCLI

DNS

Load Balancer

dc1em01.example.com

OMS1

SW Lib OMS1 OMS2

EMREPDC1

EMREPDC12

dc1em02.example.com

OMS2

FAS3250FAS3250

Load Balancer

dc2em01.example.com

SW Lib OMS1 OMS2

EMREPDC2

dc2em02.example.com

DC1 DC2Active Passive

Data Guard

Snap Mirror

Active Active PassivePassive

RAC

WebTier Cluster

EMREPDC21

OMS1

EMREPDC22

OMS2

GI 11.2.0.4RDBMS 11.2.0.4

OMS 12.1.0.4

GI 11.2.0.4RDBMS 11.2.0.4

OMS 12.1.0.4

dc1emrep01.example.com dc1emrep02.example.com dc2emrep01.example.com dc2emrep02.example.com

JDBC JDBC

NetworkLayer

WebTierLayer

DatabaseLayer

StorageLayer

Database Layer

Grid Infrastructure and Database Software Install

• Install the following on all nodes in the RAC cluster for the Repository database:

– Grid Infrastructure 11gR2 (11.2.0.4) binaries installed and configured

– Oracle Database 11gR2 (11.2.0.4) binaries installed and configured

– Apply latest PSU to both GI and RDBMS

– SCAN listeners configured

– ASM is optional but recommended

– dc1emrep01 and dc1emrep02 form dc1emrep cluster

– dc2emrep01 and dc2emrep02 form dc2emrep cluster

DC1 DC2

GI 11.2.0.4RDBMS 11.2.0.4

OMS 12.1.0.4

GI 11.2.0.4RDBMS 11.2.0.4

OMS 12.1.0.4


EMREP Database Creation

• Database Edition: Enterprise Edition

• Global Database Name: EMREP

• Database Unique Name: EMREPDC1

• Options: Oracle Partitioning

• Specify Fast Recovery Area

• Automatic Memory Management: 12G

• Database and National Character Set: AL32UTF8 and AL16UTF16

• Enable Archive Log Mode, Flashback and Force Logging required for Data Guard

• Processes: 800

• If using DBCA:

– Template: Custom database (Includes Datafiles: No)

– Don’t “Configure Enterprise Manager”

– Uncheck “Database Components”

EMREP Database Creation continued…

• ASM:

• +DATA for Data

• +FRA for Fast Recovery Area

• Cluster File System (e.g. ZFS, NetApp):

• /u01/app/oracle/oradata for data

– owner oracle:oinstall and file permissions 755

• /u01/app/oracle/fast_recovery_area for Fast Recovery Area

– owner oracle:oinstall and file permissions 755

EMREP Database Post Tasks

• Change the following and restart Database:

ALTER SYSTEM SET job_queue_processes=50 SCOPE=SPFILE;

ALTER SYSTEM SET open_cursors=400 SCOPE=SPFILE;

ALTER SYSTEM SET shared_pool_size=1G;

ALTER SYSTEM SET session_cached_cursors=250 SCOPE=SPFILE;

ALTER USER DBSNMP ACCOUNT UNLOCK;

• Create Database Service that runs only on primary for EM to connect to:

srvctl add service -d EMREP -s EMREPMAA -r EMREP1,EMREP2 -P

BASIC -l PRIMARY

RACEMREPDC11

EMREPDC1

EMREPDC12

DC1 DC2

GI 11.2.0.4RDBMS 11.2.0.4

OMS 12.1.0.4

GI 11.2.0.4RDBMS 11.2.0.4

OMS 12.1.0.4


Storage Layer

Preparing Shared Storage for OMSs and Software Library

• ZFS:

– Storage pool: dc1em (in DC1) and dc2em (in DC2) (300Gb – Double Parity)

• Project: OMS, share: OMS1 and OMS2 60Gb each, mount as /em,

continuous replication

• Project: SWLIB, share: swlib 100Gb, mount as /swlib, continuous

replication

• NetApp:

– Volume: dc1oms (in DC1) and dc2oms (in DC2), 120Gb each – on DP

Aggregate, mount as /em_share, SnapMirror every 15 minutes

– Volume: dc1swlib (in DC1) and dc2swlib (in DC2), 100Gb – on DP

Aggregate, mount as /swlib, SnapMirror every 5 minutes

RACEMREPDC11

FAS3250FAS3250

SW Lib OMS1 OMS2

EMREPDC1

EMREPDC12

DC1 DC2

GI 11.2.0.4RDBMS 11.2.0.4

OMS 12.1.0.4

GI 11.2.0.4RDBMS 11.2.0.4

OMS 12.1.0.4


OMS Layer

Preparing OMS Nodes (ZFS)

• Change ownership and file permissions:

chown oracle:oinstall /em

chown oracle:oinstall /swlib

chmod 775 /em

chmod 775 /swlib

• Create the software lib sub-directory:

mkdir –p /swlib/software

Preparing OMS Nodes (NFS)

• Change ownership and file permissions:

chown oracle:oinstall /em_share

chown oracle:oinstall /swlib

chmod 775 /em_share

chmod 775 /swlib

• Create the software lib sub-directory:

mkdir –p /swlib/software

Preparing OMS Nodes (NFS) continued…

• Create symbolic links on each OMS node to give a common path (/em) to a

shared storage (/em_share):

• 1st OMS node:

mkdir -p /em_share/OMS1

su -

ln -s /em_share/OMS1 /em

exit

• 2nd OMS node:

mkdir -p /em_share/OMS2

su -

ln -s /em_share/OMS2 /em

exit

Preparing OMS Nodes continued…

• On dc1em1 and dc1em02 add the following to /etc/hosts:

127.0.0.1 localhost localhost.localdomain

10.1.1.1 dc1em01.example.com dc1em01 ## Physical OMS Host Details


10.1.1.1 em01.example.com em01 ## Virtual OMS Host Details


• On dc2em01 and dc2em02 add the following to /etc/hosts:






RACEMREPDC11

FAS3250FAS3250

dc1em01.example.com

SW Lib OMS1 OMS2

EMREPDC1

EMREPDC12

dc1em02.example.com dc2em01.example.com dc2em02.example.com

DC1 DC2

GI 11.2.0.4RDBMS 11.2.0.4

OMS 12.1.0.4

GI 11.2.0.4RDBMS 11.2.0.4

OMS 12.1.0.4


• Before installing the OMS, we need to create shared Oracle Inventory that

will move with the OMS through the shared storage:

vi /etc/oraInst_OMS.loc

inventory_loc=/em/oraInventory

inst_group=oinstall

chmod 644 /etc/oraInst_OMS.loc

chown root:root /etc/oraInst_OMS.loc

./runInstaller -invPtrLoc /etc/oraInst_OMS.loc

• The default Oracle Inventory pointer file and location is intentionally not

used, which will become apparent later on

12c Cloud Control Install

12c Cloud Control Install continued…

• Important: the oracle inventory needs to be on shared storage /em:

/em/oraInventory


• Advanced install:

• Middleware and Agent on /em/Middleware and /em/Agent:


em01.example.com

Important Virtual Hostname to be used here!

Accept the warning that:“Installer has detected host name as dc1em01.example.com, you have changed that host name to em01.example.com.”

/em/Middleware/em/Agent


• Enter credentials and leave OMS Instance Base Location as default:

• Enter database connection details:


dc1emrep-scan.example.com

emrepmaa1521

••••••


• Enter credentials, datafiles and software library location (including sub-dir):

/swlib/software


• Ports that will be used in the installation:


./em/Middleware/oms/bin/emctl status oms –details

Oracle Enterprise Manager Cloud Control 12c Release 4

Copyright (c) 1996, 2014 Oracle Corporation. All rights reserved.

Enter Enterprise Manager Root (SYSMAN) Password :

Console Server Host : em01.example.com

HTTP Console Port : 7788

HTTPS Console Port : 7799

HTTP Upload Port : 4889

HTTPS Upload Port : 4900

EM Instance Home : /em/gc_inst/em/EMGC_OMS1

OMS Log Directory Location : /em/gc_inst/em/EMGC_OMS1/sysman/log

OMS is not configured with SLB or virtual hostname

Agent Upload is locked.

OMS Console is locked.

Active CA ID: 1

Console URL : https://em01.example.com:7799/em

Upload URL : https://em01.example.com:4900/empbs/upload

WLS Domain Information

Domain Name : GCDomain

Admin Server Host : em01.example.com

Admin Server HTTPS Port : 7101

Admin Server is RUNNING

Managed Server Information

Managed Server Instance Name : EMGC_OMS1

Managed Server Instance Host : em01.example.com

WebTier is Up

Oracle Management Server is Up


./em/Middleware/oms/bin/emctl config oms -list_repos_details



Repository Connect Descriptor :

(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=dc1emrep-scan.example.com)(PORT=1521)))

(CONNECT_DATA=(SERVICE_NAME=emrepmaa)))

Repository User : SYSMAN


./em/Middleware/oms/bin/emctl status agent



---------------------------------------------------------------

Agent Version : 12.1.0.4.0

OMS Version : 12.1.0.4.0

Protocol Version : 12.1.0.1.0

Agent Home : /em_share/OMS1/Agent/agent_inst

Agent Log Directory : /em_share/OMS1/Agent/agent_inst/sysman/log

Agent Binaries : /em/Agent/core/12.1.0.4.0

Agent Process ID : 9022

Parent Process ID : 8954

Agent URL : https://em01.example.com:3872/emd/main/

Local Agent URL in NAT : https://em01.example.com:3872/emd/main/

Repository URL : https://em01.example.com:4900/empbs/upload

Started at : 2015-01-20 18:27:32

Started by user : oracle

Operating System : Linux version 2.6.32-431.el6.x86_64 (amd64)

Last Reload : (none)

Last successful upload : 2015-01-21 09:52:00

Last attempted upload : 2015-01-21 09:52:00

Total Megabytes of XML files uploaded so far : 2.05

Number of XML files pending upload : 0

Size of XML files pending upload(MB) : 0

Available disk space on upload filesystem : 91.50%

Collection Status : Collections enabled

Heartbeat Status : Ok

Last attempted heartbeat to OMS : 2015-01-21 09:53:00

Last successful heartbeat to OMS : 2015-01-21 09:53:00

Next scheduled heartbeat to OMS : 2015-01-21 09:54:09

--------------------------------------------------------------

Agent is Running and Ready

• Backup EMREP database

• Patch OMS to latest PSU

• Change the lock file location from shared storage to local disk (at earliest convenience):

su root

mkdir –p /em_lock

chown oracle:oinstall /em_lock

chmod 774 /em_lock

vi /em/gc_inst/WebTierIH1/config/OHS/ohs1/httpd.conf

For the lines “MPM Prefork Module” & “MPM Worker Module”, change the LockFile to be:

LockFile "/em_lock/http_lock“

Restart OMS:

emctl stop oms

emctl start oms

12c Cloud Control Install – Post Tasks

RAC

WebTier Cluster

EMREPDC11

FAS3250FAS3250

dc1em01.example.com

OMS1

SW Lib OMS1 OMS2

EMREPDC1

EMREPDC12

dc1em02.example.com dc2em01.example.com dc2em02.example.com

DC1 DC2

GI 11.2.0.4RDBMS 11.2.0.4

OMS 12.1.0.4

GI 11.2.0.4RDBMS 11.2.0.4

OMS 12.1.0.4


JDBC

Network Layer

Configure SLBs – Create Health Monitors

Configure SLBs – Create TCP Profiles

• Create 2 TCP Profiles at each Data Centre (with default settings):

• 1 for the Secure Console

• 1 for the Secure Upload

Configure SLBs – Create Pools

dc1em01.example.com:7799dc1em02.example.com:7799




Configure SLBs – Persistence Profile

Configure SLBs – Create Virtual Servers

Configure SLBs – Summary

pool_ccsc7799

pool_ccsu4900

Update OMS Configuration

• Add Virtual Server Name into DNS:

em.example.com CNAME em_slb_dc1.example.com (TTL 60)

em_slb_dc1.example.com A 123.1.2.110

em_slb_dc2.example.com A 123.2.2.110

• Re-configuration in OMS:

emctl secure oms -sysman_pwd <sysman_pwd> -reg_pwd <agent_reg_password>

-host em.example.com -secure_port 4900 -slb_port 4900 -slb_console_port 443 -console

-lock_upload -lock_console

• Bounce OMS:

emctl stop oms -all

emctl start oms

Update OMS Configuration continued…

./em/Middleware/oms/bin/emctl status oms –details



Enter Enterprise Manager Root (SYSMAN) Password :

Console Server Host : em01.example.com

HTTP Console Port : 7788

HTTPS Console Port : 7799

HTTP Upload Port : 4889

HTTPS Upload Port : 4900

EM Instance Home : /em/gc_inst/em/EMGC_OMS1

OMS Log Directory Location : /em/gc_inst/em/EMGC_OMS1/sysman/log

SLB or virtual hostname : em.example.com

HTTPS SLB Upload Port : 4900

HTTPS SLB Console Port : 443

Agent Upload is locked.

OMS Console is locked.

Active CA ID: 1

Console URL : https://em.example.com:443/em

Upload URL : https://em.example.com:4900/empbs/upload

WLS Domain Information

Domain Name : GCDomain

Admin Server Host : em01.example.com

Admin Server HTTPS Port : 7101

Admin Server is RUNNING

Managed Server Information

Managed Server Instance Name: EMGC_OMS1

Managed Server Instance Host: em01.example.com

WebTier is Up

Oracle Management Server is Up


• Reconfigure OMS agent upload url (only agent deployed at this stage):

emctl secure agent –emdWalletSrcUrl https://em.example.com:4900/em

• Any new agent deployments will have the new agent upload url (em.example.com)


./em/Middleware/oms/bin/emctl status agent



---------------------------------------------------------------

Agent Version : 12.1.0.4.0

OMS Version : 12.1.0.4.0

Protocol Version : 12.1.0.1.0

Agent Home : /em_share/OMS1/Agent/agent_inst

Agent Log Directory : /em_share/OMS1/Agent/agent_inst/sysman/log

Agent Binaries : /em/Agent/core/12.1.0.4.0

Agent Process ID : 5815

Parent Process ID : 5749

Agent URL : https://em01.example.com:3872/emd/main/

Local Agent URL in NAT : https://em01.example.com:3872/emd/main/

Repository URL : https://em.example.com:4900/empbs/upload

Started at : 2015-01-26 11:52:40

Started by user : oracle

Operating System : Linux version 2.6.32-431.el6.x86_64 (amd64)

Last Reload : (none)

Last successful upload : 2015-01-26 11:52:53

Last attempted upload : 2015-01-26 11:52:53

Total Megabytes of XML files uploaded so far : 0.05

Number of XML files pending upload : 0

Size of XML files pending upload(MB) : 0

Available disk space on upload filesystem : 91.34%

Collection Status : Collections enabled

Heartbeat Status : Ok

Last attempted heartbeat to OMS : 2015-01-26 11:52:49

Last successful heartbeat to OMS : 2015-01-26 11:52:49

Next scheduled heartbeat to OMS : 2015-01-26 11:53:49

--------------------------------------------------------------

Agent is Running and Ready


• Reconfigure emcli:

emcli setup -url=https://em.example.com:7799/em -username=SYSMAN -trustall


emcli>status()

Oracle Enterprise Manager 12c Release 4 EM CLI with Scripting option.

Copyright (c) 1996, 2014 Oracle Corporation and/or its affiliates. All rights reserved.

Verb Jars Home (EMCLI_VERBJAR_DIR) : /em/Middleware/oms/bin/bindings/12.1.0.4.0/.emcli

EM CLI Home (EMCLI_INSTALL_HOME) : /em/Middleware/oms/bin

EM CLI Version : 12.1.0.4.0

Java Home : /em_share/OMS1/Middleware/jdk16/jdk/jre

Java Version : 1.6.0_43

Log file (EMCLI_LOG_LOC) : CONSOLE

Log level (EMCLI_LOG_LEVEL) : SEVERE

EM URL (EMCLI_OMS_URL) : https://em.example.com:7799/em

EM user (EMCLI_USERNAME) : SYSMAN

Auto login (EMCLI_AUTOLOGIN) : true

Trust all certificates (EMCLI_TRUSTALL) : true

Type help('client_properties') to see a list of available client properties.

emcli>

RAC

WebTier Cluster

em.example.com

EMREPDC11

FAS3250FAS3250


DNS

Load Balancer

dc1em01.example.com

OMS1

SW Lib OMS1 OMS2

EMREPDC1

EMREPDC12

dc1em02.example.com

Load Balancer

dc2em01.example.com dc2em02.example.com


Active

GI 11.2.0.4RDBMS 11.2.0.4

OMS 12.1.0.4

GI 11.2.0.4RDBMS 11.2.0.4

OMS 12.1.0.4


JDBC

OMS Layer

High Availability

• Before adding the 2nd OMS, we need to create shared Oracle Inventory that

will move with the OMS through shared storage:

vi /etc/oraInst.loc


inst_group=oinstall

chmod 644 /etc/oraInst.loc

chown root:root /etc/oraInst.loc

Adding the Second OMS

Adding the Second OMS continued…

• Push agent to host of second OMS using virtual address em02.example.com:

em02.example.com

ADD_HOST_SYSMAN_Jan_26_2015_12:14:48_AM_UTC

• Specify the same port and location for agent as 1st OMS:


• Run -> Enterprise | Provisioning and Patching | Procedure Library and then select the

“Add Management Service Deployment Procedure”:



• Tick all the tasks that should now be completed:


• Enter second OMS host using virtual address em02.example.com:

em02.example.com


• Use Shared Directory for Transfer Mode and keep port same as 1st OMS:


• Steps 1 and 2 already done, run root.sh for step 3:


• Finally you’ll see the 2 OMS in the EMGC_GCDomain Farm:

• Move the Oracle Inventory pointer file to match the 1st node:

mv /etc/oraInst.loc /etc/oraInst_OMS.loc

• Change the lock file location from shared storage to local disk (to avoid the

performance issues):

su root

mkdir –p /em_lock


chmod 774 /em_lock

vi /em/gc_inst/WebTierIH1/config/OHS/ohs1/httpd.conf

For the lines “MPM Prefork Module” & “MPM Worker Module”, change the LockFile to be:

LockFile "/em_lock/http_lock“

Restart OMS:

emctl stop oms

emctl start oms

Adding the Second OMS – Post Tasks

RAC

WebTier Cluster

em.example.com

EMREPDC11

FAS3250FAS3250


DNS

Load Balancer

dc1em01.example.com

OMS1

SW Lib OMS1 OMS2

EMREPDC1

EMREPDC12

dc1em02.example.com

OMS2

Load Balancer



Active Active

GI 11.2.0.4RDBMS 11.2.0.4

OMS 12.1.0.4

GI 11.2.0.4RDBMS 11.2.0.4

OMS 12.1.0.4


JDBC

Database Layer

Disaster Recovery

Creating Physical Standby

• Create Physical Standby EMREPDC2 ensuring you use the physical host

names via:

• Cloud Control

• RMAN Duplicate from Active Database

• Storage Replication

• Ensure you have set:

• Archive Log Mode

• Forced Logging

• Flashback on (for easier reinstatement)

Update OMS Connect String

• Add Virtual Server Name into DNS (to avoid reconfiguration at database layer):

emrep.example.com CNAME dc1emrep.example.com (TTL 60)

dc1emrep.example.com CNAME dc1emrep-scan.example.com

dc2emrep.example.com CNAME dc2emrep-scan.example.com

Update OMS Connect String continued…

• On both active OMS nodes run:

emctl stop oms -all

emctl start oms -admin_only

emctl config oms -store_repos_details -repos_conndesc

"(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)

(HOST=emrep.example.com)(PORT=1521)))

(CONNECT_DATA=(SERVICE_NAME=EMREPMAA)))" -repos_user sysman

emctl stop oms -all

emctl start oms

To avoid long TNS timeouts when hosts not available (DR scenario)

RAC

WebTier Cluster

em.example.com

EMREPDC11

FAS3250FAS3250


DNS

Load Balancer

dc1em01.example.com

OMS1

SW Lib OMS1 OMS2

EMREPDC1

EMREPDC12

dc1em02.example.com

OMS2

Load Balancer

dc2em01.example.com

EMREPDC2

dc2em02.example.com


Data Guard

Active Active

RACEMREPDC21 EMREPDC22

GI 11.2.0.4RDBMS 11.2.0.4

OMS 12.1.0.4

GI 11.2.0.4RDBMS 11.2.0.4

OMS 12.1.0.4


JDBC

OMS Layer

Disaster Recovery

Check OMS Pre-Requisites

• Run the OUI installer for OMS on the standby nodes to check pre-requisites

and then cancel and exit install

Replication for Shared Storage

• ZFS:

– Ensure the projects OMS and SWLIB have continuous replication from

storage pool dc1em (in DC1) to dc2em (in DC2)

• NetApp:

– Ensure the volume dc1oms (in DC1) is SnapMirror every 15 minutes to

it’s Snap Destination dc2oms (in DC2)

– Ensure the volume dc1swlib (in DC1) is SnapMirror every 5 minutes to

it’s Snap Destination dc2swlib (in DC2)

Preparing Standby OMS Nodes

• Create symbolic links on each OMS node (if using NFS):

su -

ln -s /em_share/OMS1 /em (on 1st node)

ln -s /em_share/OMS2 /em (on 2nd node)

exit

• Copy Oracle Inventory pointer file:

scp /etc/oraInst_OMS.loc root@dc2emrep01:/etc

scp /etc/oraInst_OMS.loc root@dc2emrep02:/etc

Preparing Standby OMS Nodes continued…

• Create lock folder for lock file:

su root

mkdir –p /em_lock


chmod 774 /em_lock

exit

• Install local agents on all 8 servers using physical hostnames

• On RAC nodes physical hostname:

– Discover local agent, host, databases, ASM if present, listeners, etc

• On OMS nodes physical hostname:

– Discover local agent and host only

– Remove any OMS targets if discovered

• On OMS nodes virtual hostname (agent already installed with OMS install):

– Discover shared agent, shared host, OMS, etc

Deployment Post Tasks

• Local Oracle Inventory:

/etc/oraInst.loc

inventory_loc=/u01/app/oracle/oraInventory

• Shared Oracle Inventory:

/etc/oraInst_OMS.loc


Deployment Post Tasks continued…

dc1em01.example.com dc1em02.example.com dc2em01.example.com dc2em02.example.com


Shared Inventory on /emShared AgentShared HostShared OMS

Local Inventory on /u01Local AgentLocal Host




Grid InfrastructureRDBMS









Moves with storage

WebTier (OMS)Layer

DatabaseLayer

DC2DC1

OMS 12c MAA

RAC

WebTier Cluster

em.example.com

EMREPDC11

FAS3250FAS3250


DNS

Load Balancer

dc1em01.example.com

OMS1

SW Lib OMS1 OMS2

EMREPDC1

EMREPDC12

dc1em02.example.com

OMS2

FAS3250FAS3250

Load Balancer

dc2em01.example.com

SW Lib OMS1 OMS2

EMREPDC2

dc2em02.example.com


Data Guard

Snap Mirror


RAC

WebTier Cluster

EMREPDC21

OMS1

EMREPDC22

OMS2

GI 11.2.0.4RDBMS 11.2.0.4

OMS 12.1.0.4

GI 11.2.0.4RDBMS 11.2.0.4

OMS 12.1.0.4


JDBC JDBC

NetworkLayer

WebTierLayer

DatabaseLayer

StorageLayer

Switchover

Switchover of OMS

• On all active OMS nodes, perform:

emctl stop oms -all

emctl stop agent

Unmount the /em or /em_share and /swlib

Switchover of OMS continued…

• On shared storage:

• If ZFS, perform on-demand replication for the SWLIB and OMS Projects, then perform role

reversal for these projects to make DC2 primary, ensure continuous replication schedule is

created.

• If NFS, take final SnapMirror of the dc1oms and dc1swlib volumes, break mirror, make DC2

volumes read/write and then reverse the SnapMirror relationship.


• Update DNS so as follows:



• Switchover EMREP database:

dgmgrl

connect sys/<password>

switchover to ‘EMREPDC2’


• On all inactive OMS nodes, perform:

Mount the /em or /em_share and /swlib

emctl start oms

emctl start agent

Failover

Failover of OMS

• On shared storage:

• If ZFS, perform role reversal for the SWLIB and OMS Projects to make DC2 primary, ensure

continuous replication schedule is created when DC1 comes back.

• If NFS, break the SnapMirror for the dc1oms and dc1swlib volumes, make DC2 volumes

read/write and then reverse the SnapMirror relationship when DC1 comes back.

Failover of OMS continued…

• Update DNS so as follows:



• Failover EMREP database:

dgmgrl


failover to ‘EMREPDC2’


• On all inactive OMS nodes, perform:

Mount the /em or /em_share and /swlib

• On the 1st OMS node, perform:

emctl start oms -admin_only

emctl resync repos -full -name "failover_resync"

emctl start agent

• On all the remaining OMS nodes, perform:

emctl start oms

emctl start agent


• When DC1 as a site is available again:

• For ZFS Shared Storage ensure continuous replication schedule is created for the SWLIB

and OMS Projects and for NFS Shared Storage the SnapMirror relationship is reversed for

dc1oms and dc1swlib volumes (so DC2 -> DC1)

• Reinstate EMREPDC1:

dgmgrl (on new primary DC2)


reinstate database ‘EMREPDC1’

Licensing

Implications

Licensing Implications

• The clustering of the OMS WebTier is covered as per the restricted-use license (free)

• Database Enterprise Edition license required for all servers except for 1 server which

can be used as part of restricted-use license

• RAC licenses required for all servers

• Data Guard is covered by the Database Enterprise Edition license

RAC

WebTier Cluster

em.example.com

EMREPDC11

FAS3250FAS3250


DNS

Load Balancer

dc1em01.example.com

OMS1

SW Lib OMS1 OMS2

EMREPDC1

EMREPDC12

dc1em02.example.com

OMS2

FAS3250FAS3250

Load Balancer

dc2em01.example.com

SW Lib OMS1 OMS2

EMREPDC2

dc2em02.example.com


Data Guard

Snap Mirror


RAC

WebTier Cluster

EMREPDC21

OMS1

EMREPDC22

OMS2

GI 11.2.0.4RDBMS 11.2.0.4

OMS 12.1.0.4

GI 11.2.0.4RDBMS 11.2.0.4

OMS 12.1.0.4


JDBC JDBC

RAC RAC

DB EE

RAC

DB EE

RAC

DB EE

R-UseFree

R-UseFree

R-UseFree

R-UseFree

References

• Deploying a Highly Available Enterprise Manager 12c Cloud Control:

http://www.oracle.com/technetwork/oem/framework-infra/wp-em12c-building-ha-level3-1631423.pdf

• Enterprise Manager Cloud Control 12c Disaster Recovery with Storage Replication:

http://www.oracle.com/technetwork/oem/cloud-mgmt/wp-em12cr3-dr-solution-1965514.pdf

• Oracle Real Application Cluster:

http://docs.oracle.com/cd/E11882_01/rac.112/e16795/toc.htm

• Oracle Data Guard:

http://docs.oracle.com/cd/E11882_01/server.112/e41134/toc.htm

http://www.oracle.com/technetwork/oem/framework-infra/wp-em12c-building-ha-level3-1631423.pdf

http://www.oracle.com/technetwork/oem/cloud-mgmt/wp-em12cr3-dr-solution-1965514.pdf

http://docs.oracle.com/cd/E11882_01/rac.112/e16795/toc.htm

http://docs.oracle.com/cd/E11882_01/server.112/e41134/toc.htm

References

• Architecture Principles and Implementation Practices for Remote Replication Using Oracle ZFS

Storage Appliance:

http://www.oracle.com/technetwork/server-storage/sun-unified-storage/documentation/zfssa-

replication-2014-1-2120969.pdf

• NetApp Snap Mirror:

http://www.netapp.com/uk/products/protection-software/snapmirror.aspx

• Enterprise Manager Restricted-use License:

http://docs.oracle.com/cd/E24628_01/doc.121/e24474/ch1_introduction.htm#OEMLI112

• Infrastructure Repository Databases:

http://docs.oracle.com/database/121/DBLIC/editions.htm#BABDFJIC

http://www.oracle.com/technetwork/server-storage/sun-unified-storage/documentation/zfssa-replication-2014-1-2120969.pdf

http://www.netapp.com/uk/products/protection-software/snapmirror.aspx

http://docs.oracle.com/cd/E24628_01/doc.121/e24474/ch1_introduction.htm#OEMLI112

http://docs.oracle.com/database/121/DBLIC/editions.htm#BABDFJIC

Thank you for listening!

• [email protected]













mailto:[email protected]



Bonus Material

Bonus Material

• Deploying 12c Cloud Control (DR) using 2 servers

• Deploying 12c Cloud Control (MAA) using 4 servers instead of 8

Deploying 12c CC (DR) using 2 servers

• Exactly the same as deployment with the following amendments:

• No SLB

• No WebTier Cluster

• No RAC Cluster

• But still gives resilience through the use of a 2nd Data Centre while

minimising the cost

Deploying 12c CC (DR) using 2 servers

• On dc1em1:


10.1.1.1 dc1em01.example.com dc1em01 ## Physical Host Details - DB

10.1.1.1 em01.example.com em01 ## Virtual Host Details - OMS

• On dc2em01:


10.1.2.1 dc2em01.example.com dc2em01 ## Physical Host Details - DB


em.example.com

EMREPDC11

FAS3250FAS3250


DNS

dc1em01.example.com

OMS1

SW Lib OMS1 OMS2

EMREPDC1

FAS3250FAS3250

dc2em01.example.com

SW Lib OMS1 OMS2

EMREPDC2


Data Guard

Snap Mirror

EMREPDC21

OMS1

GI 11.2.0.4RDBMS 11.2.0.4

OMS 12.1.0.4

GI 11.2.0.4RDBMS 11.2.0.4

OMS 12.1.0.4

Update OMS Configuration

• Add Virtual Server Name into DNS:

em.example.com CNAME dc1em01.example.com (TTL 60)

dc1em01.example.com A 10.1.1.1

dc2em01.example.com A 10.1.2.1

• Re-configuration in OMS (using the same ports):

emctl secure oms -sysman_pwd <sysman_pwd> -reg_pwd <agent_reg_password>

-host em.example.com -secure_port 4900 -slb_port 4900 -slb_console_port 7799 -console

-lock_upload -lock_console

• Bounce OMS:

emctl stop oms -all

emctl start oms

Pre-steps before update OMS Connect String

• Prior to updating the Connect String create DB Service:

exec DBMS_SERVICE.CREATE_SERVICE(SERVICE_NAME => 'EMREPMAA',

NETWORK_NAME => 'EMREPMAA');

Pre-steps before update OMS Connect String continued…

• And Service Startup Trigger:

CREATE OR REPLACE TRIGGER SYS.MANAGE_SERVICES AFTER STARTUP

ON DATABASE

DECLARE

ROLE VARCHAR (30);

OMODE VARCHAR (30);

BEGIN

SELECT DATABASE_ROLE INTO ROLE FROM V$DATABASE;

SELECT OPEN_MODE INTO OMODE FROM V$DATABASE;

IF ROLE = 'PRIMARY'

THEN

DBMS_SERVICE.START_SERVICE ('EMREPMAA');

END IF;

END;

/







Moves with storage

WebTier (OMS)and

DatabaseLayer

DC2DC1

Deploying 12c CC (MAA) using 4 instead of 8 servers

• Exactly the same as deploying on 8 server, however the OMS WebTier and

RAC cluster co-exist

Deploying 12c CC (MAA) using 4 instead of 8 servers

• On dc1em1 and dc1em02:


10.1.1.1 dc1em01.example.com dc1em01 ## Physical Host Details - RAC




• On dc2em01 and dc2em02:






RAC

WebTier Cluster

em.example.com

EMREPDC11

FAS3250FAS3250


DNS

Load Balancer

dc1em01.example.com

OMS1

SW Lib OMS1 OMS2

EMREPDC1

EMREPDC12

dc1em02.example.com

OMS2

FAS3250FAS3250

Load Balancer

dc2em01.example.com

SW Lib OMS1 OMS2

EMREPDC2

dc2em02.example.com


Data Guard

Snap Mirror


RAC

WebTier Cluster

EMREPDC21

OMS1

EMREPDC22

OMS2

GI 11.2.0.4RDBMS 11.2.0.4

OMS 12.1.0.4

GI 11.2.0.4RDBMS 11.2.0.4

OMS 12.1.0.4

• Before adding the 2nd OMS, we need to create shared Oracle Inventory that

will move with the OMS:

mv /etc/oraInst.loc /etc/oraInst_DB.loc

vi /etc/oraInst.loc


inst_group=oinstall

chmod 644 /etc/oraInst.loc

chown root:root /etc/oraInst.loc

Adding the Second OMS when using 4 instead of 8 servers

• Same as before but for the Oracle Inventory pointer file use these commands instead:

mv /etc/oraInst.loc /etc/oraInst_OMS.loc

mv /etc/oraInst_DB.loc /etc/oraInst.loc

• For the EMREP Virtual Server Name in DNS:


dc1emrep.example.com CNAME dc1em-scan.example.com

dc2emrep.example.com CNAME dc2em-scan.example.com

Adding the Second OMS – Post Tasks

dc1em01.example.com dc1em02.example.com dc2em01.example.com dc2em02.example.com











Moves with storage

WebTier (OMS)and

DatabaseLayer

DC2DC1

Thank you for listening!














mailto:[email protected]

