Overview of First4DB Monitoring

Oracle RAC with Data Guard and Golden Gate

Examination of 2 X 3-Node Clusters across 2 Data Centers

Steve RecskyComplete DBA Inc.

A Partner of First4 Database Partners Inc.

First4 Database Partners Inc.

Objectives Review and discuss the following:

MAA and it’s components Data Guard RAC (Real Application Clusters) Golden Gate Data Guard set up and configuration Oracle Cluster Resources and Services

Show a Data Guard Switchover Show connection to Service Discuss challenges for maintaining the

environment Questions


Oracle MAAMaximum Availability

ArchitectureAddress the following

unplanned downtime: Site failure Cluster wide failure Computer failure Storage failure Data corruption Human error Lost writes Hang or slowdown


Oracle MAA


Why RAC, Data Guard, Golden Gate?

RAC – Real Application ClustersHighly scalableSCAN (Single Client Access Name)

supportRemoves single-point-of-failure for

database server (Except for disk and/or site failure)


RAC


Oracle® Real Application ClustersAdministration and Deployment Guide 11g Release 2 (11.2) E41960-03


Data Guard Allows for the support of a Physical

standby (Redo Apply) and Logical standby (SQL Apply) database that replicates the exact contents of the Primary across the Oracle Net network layer

If licensed, the standby database can be opened in read-only mode to support the Active Data Guard option

Purpose-built for disaster protection and database rolling upgrades



Data Guard Protection LevelsMaximum Protection (zero data

loss, synchronous replication)Maximum Availability (same as

Protection but does not shut down if a fault prevents it from writing its redo stream to the remote standby)

Maximum Performance (asynchronous replication)


Data Guard



Golden Gate Supports subset replication to extract,

transform and aggregate data from numerous data sources into a central data store

Multi-master replication can be used to synchronize several databases

Used for planned maintenance that Data Guard can’t support (i.e. cross-endian platform migration, application upgrades, etc.)


Golden Gate – single direction


Golden Gate – bi-directional


FYI – Need to be aware of collisions with Sequences, Trigger processing and Cascade Deletes

Golden Gate Components


Combination of all 3 solutions


Environment


RACNode 1

Instance 1

Primary Secondary

Primary Data Center(PDC)

Secondary Data Center(SDC)

RACNode 2

Instance 2

RACNode 3

Instance 3

RACNode 1

Instance 1

RACNode 2

Instance 2

RACNode 3

Instance 3

Data Guard

RW Service

RO Service

Quick Steps to set up Data Guard


Make sure Primary database is in ARCHIVELOG mode Prepare Primary for Data Guard

alter database force logging; alter system set standby_file_management = 'AUTO'; alter database add standby logfile… set db_unique_name parameter alter system set fal_server =… alter system set log_archive_config = 'dg_config=(…)’;

Configure Oracle Net Configure Redo Transport

log_archive_dest_1 log_archive_dest_2

Prepare Standby Copy password file from $ORACLE_HOME/dba/orapw… Create standby spfile from the primary alter sytem set db_file_name_convert=… alter system set log_file_name_convert=…

Quick Steps to set up Data Guard


startup the Standby database with nomount… Create Standby Database

RMAN – duplicate target database … for standby from active database; Start the Redo Apply

ALTER DATABASE RECOVER MANAGED STANDBY DATABASE USING CURRENT LOGFILE DISCONNECT FROM SESSION;

Verify Redo Apply is working Primary and Standby: select DEST_ID, STATUS, DESTINATION, ERROR from V$ARCHIVE_DEST where

DEST_ID <=2; Primary: select SEQUENCE#, FIRST_TIME, NEXT_TIME, APPLIED, ARCHIVED from V$ARCHIVED_LOG where name = ‘…' order by FIRST_TIME; select STATUS, GAP_STATUS from V$ARCHIVE_DEST_STATUS where DEST_ID = 2;

Look for errors here select * from V$DATAGUARD_STATUS order by TIMESTAMP;

Oracle Cluster Services and RAC

pdcoradb1[DBATEST1](oracle):/home/oracle> srvctl status database -d DBATEST_PDCInstance DBATEST1 is running on node pdcoradb1Instance DBATEST2 is running on node pdcoradb2Instance DBATEST3 is running on node pdcoradb3


sdcoradb1[DBATEST1](oracle):/home/oracle> srvctl status database -d DBATEST_SDCInstance DBATEST1 is running on node sdcoradb1Instance DBATEST2 is running on node sdcoradb2Instance DBATEST3 is running on node sdcoradb3

PDC

SDC


pdcoradb1[DBATEST1](oracle):/home/oracle> srvctl config database -d DBATEST_PDCDatabase unique name: DBATEST_PDCDatabase name: DBATESTOracle home: /usr/oracle/product/11.2.0.2Oracle user: oracleSpfile: /oradata/pdcoraclu/admin/dbs/spfileDBATEST.oraDomain: PDCStart options: openStop options: immediateDatabase role: PRIMARYManagement policy: AUTOMATICServer pools: DBATEST_PDCDatabase instances: DBATEST1,DBATEST2,DBATEST3Disk Groups:Mount point paths:Services: DBATEST_RO.CORP,DBATEST_RW.CORPType: RACDatabase is administrator managed


PDC


sdcoradb1[DBATEST1](oracle):/home/oracle> srvctl config database -d DBATEST_SDCDatabase unique name: DBATEST_SDCDatabase name: DBATESTOracle home: /usr/oracle/product/11.2.0.2Oracle user: oracleSpfile: /oradata/sdcoraclu/admin/dbs/spfileDBATEST.oraDomain: SDCStart options: openStop options: immediateDatabase role: PHYSICAL_STANDBYManagement policy: AUTOMATICServer pools: DBATEST_SDCDatabase instances: DBATEST1,DBATEST2,DBATEST3Disk Groups:Mount point paths:Services: DBATEST_RO.CORP,DBATEST_RW.CORPType: RACDatabase is administrator managed


PDC

Data Guard BrokerDGMGRL> show database verbose "DBATEST_PDC"

Database - DBATEST_PDC

Role: PRIMARY Intended State: TRANSPORT-ON Instance(s): DBATEST1 DBATEST2 DBATEST3

Properties: DGConnectIdentifier = 'DBATEST.PDC' ObserverConnectIdentifier = ''<...snipped...> StandbyFileManagement = 'AUTO' ArchiveLagTarget = '1800' LogArchiveMaxProcesses = '4' LogArchiveMinSucceedDest = '1' DbFileNameConvert = '' LogFileNameConvert = '/oradata/DBATEST/redo/redoA, /oradata/DBATEST/redo/redoA, /oradata/DBATEST/redo/redoB, /oradata/DBATEST/redo/redoB' FastStartFailoverTarget = ''<..snipped..>

Database Status:SUCCESS


PDC/SDC

ArchiveLagTarget limits the amount of data that can be lost and effectively increases the availability of the standby database by forcing a log switch after the specified amount of time elapses.

Data Guard BrokerDGMGRL> show instance verbose "DBATEST1" on database "DBATEST_PDC"

Instance 'DBATEST1' of database 'DBATEST_PDC'

Host Name: pdcoradb1.CORP.ca PFILE: Properties: SidName = 'DBATEST1' StaticConnectIdentifier = '(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=10.96.44.38)(PORT=1521))(CONNECT_DATA=(SERVICE_NAME=DBATEST.PDC)(INSTANCE_NAME=DBATEST1)(SERVER=DEDICATED)))' StandbyArchiveLocation = '/oradata/DBATEST/arch' AlternateLocation = '' LogArchiveTrace = '0' LogArchiveFormat = 'DBATEST_%t_%s_%r.arc' TopWaitEvents = '(monitor)'

Instance Status:SUCCESS


PDC/SDC

Set StaticConnectIdentifier so as to not have to define a TNS Names entry called service_name_DGMGRL

Servicespdcoradb1[DBATEST1](oracle):/home/oracle> srvctl config service -s DBATEST_RW.CORP -d DBATEST_PDCService name: DBATEST_RW.CORPService is enabledServer pool: DBATEST_PDC_DBATEST_RW.CORPCardinality: 3Disconnect: falseService role: PRIMARYManagement policy: AUTOMATICDTP transaction: falseAQ HA notifications: trueFailover type: SELECTFailover method: BASICTAF failover retries: 0TAF failover delay: 0Connection Load Balancing Goal: LONGRuntime Load Balancing Goal: NONETAF policy specification: NONEEdition:Preferred instances: DBATEST1,DBATEST2,DBATEST3Available instances:


Servicessdcoradb1[DBATEST1](oracle):/home/oracle> srvctl config service -s DBATEST_RO.CORP -d DBATEST_SDCService name: DBATEST_RO.CORPService is enabledServer pool: DBATEST_SDC_DBATEST_RO.CORPCardinality: 3Disconnect: falseService role: PHYSICAL_STANDBYManagement policy: AUTOMATICDTP transaction: falseAQ HA notifications: trueFailover type: SELECTFailover method: BASICTAF failover retries: 0TAF failover delay: 0Connection Load Balancing Goal: LONGRuntime Load Balancing Goal: NONETAF policy specification: NONEEdition:Preferred instances: DBATEST1,DBATEST2,DBATEST3Available instances:


Servicessdcoradb1[DBATEST1](oracle):/home/oracle> sqlplus srecsky@dbatest_rw.CORP

SQL*Plus: Release 11.2.0.2.0 Production on Mon Dec 9 13:10:52 2013

Copyright (c) 1982, 2010, Oracle. All rights reserved.

Enter password:

Connected to:Oracle Database 11g Enterprise Edition Release 11.2.0.2.0 - 64bit ProductionWith the Partitioning, Real Application Clusters, OLAP, Data Miningand Real Application Testing options

13:10:57 DBATEST1 SRECSKY>select host_name from v$instance;

HOST_NAME----------------------------------------------------------------pdcoradb1.CORP.ca


DBATEST_RW.CORP =(DESCRIPTION = (ADDRESS_LIST= (ADDRESS = (PROTOCOL = TCP)(HOST = sdcoraclu-scan.CORP.ca)(PORT = 1521)) (ADDRESS = (PROTOCOL = TCP)(HOST = pdcoraclu-scan.CORP.ca)(PORT = 1521)) ) (CONNECT_DATA = (SERVICE_NAME = DBATEST_RW.CORP) (FAILOVER_MODE=(TYPE=SELECT)(METHOD=BASIC)(RETRIES=30)(DELAY=5)) ))

Data Guard Switchoverpdcoradb1[DBATEST1](oracle):/home/oracle> dgmgrlDGMGRL for Linux: Version 11.2.0.2.0 - 64bit Production


Welcome to DGMGRL, type "help" for information.DGMGRL> connect sys/****************@dbatest.corppdcConnected.DGMGRL> switchover to 'DBATEST_CORPSDC';Performing switchover NOW, please wait...New primary database "DBATEST_CORPSDC" is opening...Operation requires shutdown of instance "DBATEST3" on database "DBATEST_CORPPDC"Shutting down instance "DBATEST3"...ORACLE instance shut down.Operation requires startup of instance "DBATEST3" on database "DBATEST_CORPPDC"Starting instance "DBATEST3"...ORACLE instance started.Database mounted.Database opened.Switchover succeeded, new primary is "DBATEST_CORPSDC"DGMGRL>


Data Guard – RW Connection

H:\>sqlplus srecsky@dbatest_rw.corp

SQL*Plus: Release 11.2.0.1.0 Production on Tue Dec 10 14:37:47 2013


Enter password:


SQL> select host_name from v$instance;

HOST_NAME----------------------------------------------------------------sdcoradb2.corp.ca

SQL>


Data Guard – RO ConnectionH:\>sqlplus srecsky@dbatest_ro.corp

SQL*Plus: Release 11.2.0.1.0 Production on Tue Dec 10 14:51:07 2013


Enter password:


SQL> select host_name from v$instance;

HOST_NAME----------------------------------------------------------------pdcoradb3.corp.ca

SQL>


Data Guard - ConfigurationDGMGRL> show configuration

Configuration - DBATEST

Protection Mode: MaxPerformance Databases: DBATEST_CORPSDC - Primary database DBATEST_CORPPDC - Physical standby database

Fast-Start Failover: DISABLED

Configuration Status:SUCCESS

DGMGRL>


sdcoradb1[DBATEST1](oracle):/home/oracle> srvctl status service -s DBATEST_RW.CORP -d DBATEST_CORPSDCService DBATEST_RW.CORP is running on instance(s) DBATEST1,DBATEST2,DBATEST3

Golden Gate Challenge


RACNode 1

Instance 1

PrimarySecondary

Primary Data Center Secondary Data Center

RACNode 2

Instance 2RAC

Node 3Instance 3

RACNode 1

Instance 1RAC

Node 2Instance 2

RACNode 3

Instance 3

Oracle Database

Oracle Database

SecondaryPrimary

GoldenGate

Data Guard

Data Guard

GoldenGate

GG Checkpoint/Trail Files

Primary

Primary

GG Checkpoint/Trail Files

DBFS

DBFSOracle Database File System

Creates a standard file system interface on top of files and directories that are stored in database tables

Similar to NFS in that it provides a shared network file system that looks like a local file system

Files are stored as SecureFiles LOBs in a database table

A set of PL/SQL procedures implement the file system access primitives such as create, open, read, write, and list directory

Oracle states that there can be overhead on the primary database for the DBFS being part of it


DBFSOracle Database File System


Oracle® Database SecureFiles and Large Objects Developer's Guide11g Release 2 (11.2)E18294-04

Doc ID 1436913.1 Best Practice - Oracle GoldenGate 11gr2

integrated extract and Oracle Data Guard - Switchover/Fail-over Operations

Integrate GoldenGate integrated extract with Data Guard so that GoldenGate processes can also be switched-over/failed-over automatically with no manual intervention as part of Data Guard switchover/failover

Switchover/failover is actually done by a UNIX shell program which is triggered by the database system event "DB_ROLE_CHANGE" when switchover/failover occurs

Database must be 11.2.0.3 or higher to support Integrated Extract


Some Parting Thoughts Look at tools available to you as part of the MAA Follow best practices Build, test and document Be cognizant of the license requirements


The End

[email protected]


Documents

Overview of First4DB Monitoring