Oracle GoldenGate with Oracle Real Application Clusters

An Oracle White Paper

August 2013

Oracle Goldengate With Oracle Real Application Clusters Configuration

Oracle Maximum Availability Architecture Oracle GoldenGate with Oracle Real Application Clusters Configuration

Executive Overview ........................................................................... 2

Configuration Overview ..................................................................... 3

Oracle GoldenGate ........................................................................ 3

Oracle Real Application Clusters (RAC) ........................................ 3

Oracle Clusterware ........................................................................ 4

Oracle Database File System ........................................................ 4

Configuration Best Practices ............................................................. 5

Step 1: Set Up DBFS on Oracle RAC ............................................ 5

Step 2: Install Oracle GoldenGate ............................................... 10

Step 3: Configure Oracle GoldenGate and Database Parameters 10

Step 4: Set Up Checkpoint Files and Trail Files in DBFS ............. 12

Step 5: Configure Replicat Commit Behavior ............................... 14

Step 6: Configure Autostart of Extract, Data Pump, and Replicat Processes .................................................................................................... 14

Step 7: Oracle Clusterware Configuration .................................... 15

Appendix A: Creating Oracle GoldenGate Clusterware Resource ... 20

Recommendations When Deploying on Oracle RAC ................... 23

Appendix B: Example Agent Script .................................................. 24

References ...................................................................................... 30


Executive Overview

Oracle Real Application Clusters (Oracle RAC) and Oracle Clusterware allow Oracle Database

to run any packaged or custom application across a set of clustered servers. This capability

provides the best availability for node and instance failures and most planned maintenance

activities, and the most flexible scalability. If a clustered node fails, the Oracle database

continues running on the surviving nodes. When more processing power is needed, another

node can be added without interrupting user access to data.

Oracle Clusterware is a cluster manager that is designed specifically for the Oracle database.

In an Oracle RAC environment, Oracle Clusterware monitors all Oracle resources (such as

database instances and listeners). If a failure occurs, then Oracle Clusterware automatically

attempts to restart the failed resource. During outages, Oracle Clusterware relocates the

processing performed by the inoperative resource to a backup resource. For example, if a

node fails, then Oracle Clusterware relocates the database services being used by the

application to a surviving node in the cluster.

This white paper describes best practices for configuring Oracle GoldenGate to work with

Oracle RAC, Oracle Clusterware and Oracle Database File System (DBFS). Oracle

GoldenGate is instrumental for many reasons, including the following:

To migrate to an Oracle Database, incurring minimal downtime

As part of an application architecture that requires Oracle RAC plus the flexible availability

features provided by Oracle GoldenGate, such as active-active database for data distribution

and continuous availability, and zero or minimal downtime during planned outages for

system migrations, upgrades, and maintenance

To implement a near real-time data warehouse or consolidated database on Oracle RAC,

sourced from various, possibly heterogeneous source databases, populated by Oracle

GoldenGate

To capture from an OLTP application running on Oracle RAC to support further downstream

consumption such as a SOA type integration


3

This paper focuses on configuring Oracle GoldenGate to run on Oracle RAC, which can act as

the source database, as the target database, or in some cases as both source and target

databases for Oracle GoldenGate processing.

Configuration Overview

This section introduces Oracle GoldenGate, Oracle RAC, Oracle Clusterware, and Oracle Database

File System (DBFS). For more information about these features, see the References section at the end

of this white paper.

Oracle GoldenGate

Oracle GoldenGate provides real-time, log-based change data capture and delivery between

heterogeneous systems. This technology enables you to construct a cost-effective and low-impact real-

time data integration and continuous availability solution.

Oracle GoldenGate moves committed transactions with transaction integrity and minimal overhead on

your existing infrastructure. The architecture supports multiple data replication topologies such as one-

to-many, many-to-many, cascading, and bidirectional. Its wide variety of use cases includes real-time

business intelligence; query offloading; zero-downtime upgrades and migrations; and active-active

databases for data distribution, data synchronization, and high availability. Figure 1 shows the Oracle

GoldenGate architecture.

Figure 1. Oracle GoldenGate Architecture

Oracle Real Application Clusters

Oracle Real Application Clusters (Oracle RAC) enables multiple instances that are linked by an interconnect to share access to an Oracle database. In an Oracle RAC environment, Oracle Database runs on two or more systems in a cluster while concurrently accessing a


4

single shared database. The result is a single database system that spans multiple hardware systems, enabling Oracle RAC to provide high availability and redundancy during failures in the cluster. Oracle RAC accommodates all system types, from read-only data warehouse systems to update-intensive online transaction processing (OLTP) systems.

Oracle Clusterware

Oracle Clusterware enables servers to communicate with each other, so that they appear to function as

a collective unit. This combination of servers is commonly known as a cluster. Although the servers are

standalone servers, each server has additional processes that communicate with other servers. In this

way the separate servers appear as if they are one system to applications and end users.

Oracle Clusterware provides the infrastructure necessary to run Oracle RAC. Oracle Clusterware also

manages resources, such as virtual IP (VIP) addresses, databases, listeners, services, and so on.

There are APIs to register an application and instruct Oracle Clusterware regarding the way an

application is managed in a clustered environment. You use the APIs to register the Oracle

GoldenGate Manager process as an application managed through Oracle Clusterware. The Manager

process should then be configured to automatically start or restart other Oracle GoldenGate processes.

Oracle Database File System

The Oracle Database File System (DBFS) creates a file system interface to files stored in the database.

DBFS is similar to NFS in that it provides a shared network file system that looks like a local file

system. Because the data is stored in the database, the file system inherits all the high availability and

disaster recovery capabilities provided by the database.

With DBFS, the server is the Oracle Database. Files are stored as SecureFiles LOBs. PL/SQL

procedures implement file system access primitives such as create, open, read, write, and list directory.

The implementation of the file system in the database is called the DBFS SecureFiles Store. The DBFS

SecureFiles Store allows users to create file systems that can be mounted by clients. Each file system

has its own dedicated tables that hold the file system content.

http://www.oracle.com/pls/db112/lookup?id=CWADD91137


5

Configuration Best Practices

Step 1: Set Up DBFS on Oracle RAC

When setting up the configuration, the best practice is to store the Oracle GoldenGate trail files,

checkpoint files, bounded recovery and configuration files in DBFS to provide the best performance,

scalability, recoverability, and failover capabilities in the event of a system failure.

Using DBFS is fundamental to the continuing availability of the checkpoint and trail files in the event

of a node failure. Ensuring the availability of the checkpoint files clusterwide is essential to ensure that

after a failure occurs the Extract process can continue mining from the last known archived redo log

file position, and Replicat processes can start applying from the same trail file position before a failure

occurred. Using DBFS allows one of the surviving database instances to be the source of an

Extract/Data Pump processes or a destination for the Replicat processes.

It is recommended to run the DBFS database in ARCHIVELOG mode, so that recoverability is not

compromised in the event of media failures or corruptions.

Follow instructions in My Oracle Support note 869822.1 to install the required FUSE libraries. And

use the instructions in My Oracle Support note 1054431.1 to configure the DBFS database, required

users, and permissions on source or target environments. My Oracle Support note 1054431.1 also

provides detailed instructions for configuring DBFS with Oracle Clusterware which is covered later in

this paper.

Source Environment (Extract/Data Pump)

Create a single file system for storing the Oracle GoldenGate trail files, checkpoint files, bounded

recovery files, temp files, discard files, and parameter files.

It is recommended that you allocate enough trail file disk space to permit storage of up to 12 hours of

trail files. Doing this will give sufficient space to Extract trail file generation should a problem occur

with the target environment that prevents it from receiving new trail files. The amount of space needed

for 12 hours can only be determined by testing trail file generation rates with real production data.

Create the DBFS tablespace:

-- Connect to the DBFS database

SQL> connect system/<passwd>@<source_dbfs_tns_alias>

-- Create the tablespace:

SQL> create bigfile tablespace dbfs_gg_source_tbs datafile ‘+DBFS_DG’ size

200g autoextend on next 8g maxsize 400g LOGGING EXTENT MANAGEMENT LOCAL

AUTOALLOCATE SEGMENT SPACE MANAGEMENT AUTO;

Substitute the size parameters with your required trail file storage size.

https://support.oracle.com/CSP/main/article?cmd=show&type=NOT&id=869822.1



6

Create the file system:

% cd $ORACLE_HOME/rdbms/admin

% sqlplus dbfs_user/dbfs_password

SQL> start dbfs_create_filesystem dbfs_gg_source_tbs gg_source

The LOB segment used by DBFS should be configured with the storage options NOCACHE

LOGGING which is the default:


SQL> connect system/<passwd>@<dbfs_tns_alias>

-- View current LOB storage:

SQL> SELECT table_name, segment_name, logging, cache

FROM dba_lobs WHERE tablespace_name='DBFS_GG_SOURCE_TBS';

-- More than likely it will be something like this:

--

-- TABLE_NAME SEGMENT_NAME LOGGING CACHE

-- ------------------ ---------------------- ------- ----------

-- T_GOLDENGATE LOB_SFS$_FST_73 YES NO

If the LOB segment is not using NOCACHE LOGGING, alter it:

SQL> ALTER TABLE DBFS.<TABLE_NAME> MODIFY LOB (FILEDATA)

(NOCACHE LOGGING);

-- View the new LOB storage:


FROM dba_lobs WHERE tablespace_name='DBFS_GG_SOURCE_TBS';


-- ------------------ ---------------------- ------- ----------

-- T_GOLDENGATE LOB_SFS$_FST_73 YES NO

Starting with Oracle GoldenGate release 11.2.1 the dirtmp directory can be placed on DBFS. With

earlier releases of Oracle GoldenGate, the dirtmp directory was normally on the local file system or

another non-DBFS shared file system since data stored in dirtmp is transient and not required for any

Oracle GoldenGate startup. However, by placing dirtmp on DBFS, you get the additional benefit of

larger storage potential.

If dirtmp is on DBFS, use a NOCACHE NOLOGGING tablespace. For example:


7

Create the DBFS tablespace:


SQL> connect system/<passwd>@<source_dbfs_tns_alias>

-- Create the tablespace:

SQL> create bigfile tablespace dbfs_gg_dirtmp_tbs datafile ‘+DBFS_DG’ size

200g autoextend on next 8g maxsize 400g NOLOGGING EXTENT MANAGEMENT LOCAL

AUTOALLOCATE SEGMENT SPACE MANAGEMENT AUTO;

Substitute the size parameters with your required trail file storage size.

Create the file system:

% cd $ORACLE_HOME/rdbms/admin

% sqlplus dbfs_user/dbfs_password

SQL> start dbfs_create_filesystem dbfs_gg_dirtmp_tbs gg_dirtmp

The LOB segment used by DBFS should be configured with the storage options NOCACHE NOLOGGING which is the default when the tablespace is created with the NOLOGGING option:





FROM dba_lobs WHERE tablespace_name='DBFS_GG_DIRTMP_TBS';

-- More than likely it will be something like this:

--


-- ------------------ ---------------------- ------- ----------

-- T_GG_DIRTMP LOB_SFS$_FST_73 NO NO

Follow the instructions in My Oracle Support note 1054431.1 for configuring the newly created DBFS

file system so that the DBFS instance and mount point resources are automatically started by Cluster

Ready Services (CRS) after a node failure. When registering the resource with Oracle Clusterware, be

sure to create it as a cluster_resource instead of a local_resource as specified in the My

Oracle Support note. The reason for using cluster_resource is so the resource can only be

running on a single node preventing the accidental mounting of DBFS from concurrent nodes.

Example command to register the resource:

crsctl add resource $RESNAME \

-type cluster_resource \

-attr "ACTION_SCRIPT=$ACTION_SCRIPT, \

CHECK_INTERVAL=30,RESTART_ATTEMPTS=10, \

START_DEPENDENCIES='hard(ora.$DBNAMEL.db)pullup(ora.$DBNAMEL.db)',\



8

STOP_DEPENDENCIES='hard(ora.$DBNAMEL.db)',\

SCRIPT_TIMEOUT=300"

Once the file system is mounted, create directories in the newly created file system for storing the

Oracle GoldenGate files.

Example:

% cd /mnt/gg_source/goldengate

% mkdir dirchk

% mkdir dirpcs

% mkdir dirprm

% mkdir dirdat

% mkdir BR

Create symbolic links for the directories that are not controlled by Oracle GoldenGate parameters:

% ln –s /mnt/gg_source/goldengate/dirprm $GG_HOME/dirprm

% ln –s /mnt/gg_source/goldengate/dirchk $GG_HOME/dirchk

% ln –s /mnt/gg_source/goldengate/dirpcs $GG_HOME/dirpcs

% ln –s /mnt/gg_dirtmp $GG_HOME/dirtmp

The Bounded Recovery (BR) feature was added to Extract in Oracle GoldenGate version 11.1.1. This

feature guarantees an efficient recovery after Extract stops for any reason, planned or unplanned, no

matter how many open (uncommitted) transactions there were at the time that Extract stopped and no

matter how old they were. Bounded Recovery sets an upper boundary for the maximum amount of

time that it would take for Extract to recover to the point where it stopped and then resume normal

processing. The Bounded Recovery checkpoint files should be placed on a shared file system such that

in an event of a failover when there are open long running transactions, Extract can use Bounded

Recovery to reduce the time to taken to perform recovery. Starting in Oracle GoldenGate version

11.2.1 the Bounded Recovery files are supported and it is recommended that you place them on DBFS.

With earlier releases the Bounded Recovery files need to be stored on NFS storage such as Oracle ZFS

Storage Appliance. It is possible to store the checkpoint files on the local file system, but when Extract

performs recovery after a node failure, the standard checkpoint mechanism will be used until new local

Bounded Recovery checkpoint files are subsequently created. This will only be noticeable if there are

long running transactions at the time of the failure.

To set the Bounded Recovery file directory use the following Extract parameter:

BR BRDIR /mnt/gg_source/goldengate/BR


9

For more information on Bounded Recovery refer to the Oracle GoldenGate Windows and UNIX Reference

Guide:

http://docs.oracle.com/cd/E35209_01/doc.1121/e29399.pdf

The location of the Extract/Data Pump trail file directory is specified during process creation. For

Extract it is also specified in the parameter file with the EXTTRAIL.

Target Environment (Replicat)

On the target environment, where the Replicat processes read the trail files and apply the data to the

target database there is a requirement for two separate DBFS file systems to separate the different I/O

requirements of the trail and checkpoint files.

Trail files are written to by the Collection Server process on the target host using consecutive serial

I/O’s from the start to the end of the file, sized according to your Data Pump configuration. The same

trail files are read by each Replicat process, also using consecutive serial I/O requests. Once a portion

of the trail is read by a Replicat process it will not normally be read a second time by the same process.

When using multiple Replicat processes reading from the same trail files, it is rare that they remain in

sync, reading from the same portion of the trail file at the same time. Because of this, the best

configuration for DBFS is with NOCACHE LOGGING storage options. This is described above in

configuring the source environment.

The checkpoint files are small (approximately 4KB) but written to frequently, overwriting previous

data. The file doesn’t grow in size and is only read during process startup to determine the proper

starting point for recovery or initiation. Because the checkpoint file is written to over and over,

performance is best when the file is stored in DBFS with the CACHE LOGGING storage option.

Setting the CACHE option causes the small amount of data being written to the checkpoint files to be

written into the buffer cache of the DBFS instance, and not issuing direct writes to disk causing higher

waits on I/O. In testing this has shown to increase checkpoint performance by a factor of 2 to 5 times

compared to using the NOCACHE configuration with DBFS.

Create the second DBFS file system for the checkpoint files in much the same way as the file system

on the source environment (above). Some important notes:

The file system is only for checkpoint files, so it can be sized less than 100 MB.

Create the file system using the same user as the first file system created. It is important to make sure

the same user creates both file systems.

Change the LOB storage parameters to CACHE LOGGING:




SQL> SELECT table_name, segment_name,logging

FROM dba_lobs WHERE tablespace_name='DBFS_GG_CKPT_TBS';

-- Likely it will be something like this:

--



10


-- ------------------ ---------------------- ------- ----------

-- T_GOLDENGATE2 LOB_SFS$_FST_75 YES NO

SQL> ALTER TABLE DBFS.<TABLE_NAME> MODIFY LOB (FILEDATA)

(NOCACHE LOGGING);

-- View the new LOB storage:

SQL> SELECT table_name, segment_name,logging

FROM dba_lobs WHERE tablespace_name='DBFS_GG_CKPT_TBS';

TABLE_NAME SEGMENT_NAME LOGGING CACHE

------------------ ---------------------- ------- ----------

T_GOLDENGATE2 LOB_SFS$_FST_75 YES YES

Note: If you are using an Oracle GoldenGate Data Pump process to transfer the trail files from a

source host on the database machine using DBFS, then contact Oracle GoldenGate Support to obtain

the fix to Bug 10146318. This bug fix improves trail file creation performance on DBFS by the Oracle

GoldenGate server/collector process. This only affects Oracle GoldenGate versions earlier than

11.1.1.0.5.

Step 2: Install Oracle GoldenGate

1. Download the Oracle GoldenGate software from Oracle Technology Network (OTN) at:

http://www.oracle.com/technetwork/middleware/goldengate/downloads/index.html

2. Install Oracle GoldenGate locally on the primary source and target nodes in the Oracle RAC

configuration. Make sure the installation directory is the same on all nodes.

3. Once you have successfully configured Oracle GoldenGate on the primary source and/or

target nodes, shut down Extract/Replicat and copy the entire Oracle GoldenGate home

directory to the other source and target nodes.

4. Follow the generic installation instructions for the source and target machine installations

available in Chapter 2, “Installing GoldenGate” at:


Step 3: Configure Oracle GoldenGate and Database Parameters

It is recommended that you configure Oracle GoldenGate Extract in Integrated Capture Mode to take

advantage of the increased data type support, TDE, and data compression.

Introduced in Oracle GoldenGate version 11.2.1, Extract can be used in integrated capture mode. Extract

integrates with an Oracle database log mining server to receive change data from that server in the

form of logical change records (LCR). Extract can be configured to capture from a local or

downstream mining database. Because integrated capture is fully integrated with the database, no

additional setup is required to work with Oracle RAC, ASM, TDE, and data compression. Integrated

capture can be used starting with Oracle 11.2.0.3 with the patches described in My Oracle Support note

http://www.oracle.com/technetwork/middleware/goldengate/downloads/index.html




11

1411356.1. It can also be used to capture changes from Oracle versions starting with 10.2.0.4 in a

downstream mining deployment.

Extract can still be configured to capture directly from the redo logs for any supported Oracle version.

This configuration is now called classic capture mode.

1. Extract using Integrated Capture mode.

a. Set the database initialization parameter STREAMS_POOL_SIZE = 1.25GB X

#Integrated Capture Processes.

For further details about configuring Extract in Integrated Capture mode, refer to the

Oracle GoldenGate Installation and Setup Guide:


2. Extract in Classic Capture mode.

a. Use the default Oracle Automatic Storage Manager (Oracle ASM) naming convention for

the archived redo log files.

b. Configure the Oracle GoldenGate Extract parameter for the newer Oracle ASM log read

API.

Oracle GoldenGate release 11.1.1 introduces a new method of reading log files stored in

Oracle ASM. This new method uses the database server to access the redo and archived

redo log files, instead of connecting directly to the Oracle ASM instance. The database

must contain the libraries with the API modules. The libraries are currently included with

Oracle Database release 10.2.0.5, 11.2.0.2 and 11.2.0.3.

To successfully mine the Oracle archived redo log files located on the storage cells that

are managed by Oracle ASM, configure the Oracle GoldenGate Extract parameter as

follows:

Set the TRANLOGOPTIONS parameter to specify use of the new log read API. For

example:

TRANLOGOPTIONS DBLOGREADER

3. Configure Data Pump.

Configure the Data Pump with the PASSTHRU parameter if the process is not carrying out

any mappings or conversions. Using PASSTHRU reduces CPU by the Data Pump because it

does not have to look up table definitions, either from the database or from a data definitions

file.

For further details on Extract configuration or Data Pump with PASSTHRU, refer to the

Oracle GoldenGate Windows and UNIX Reference Guide:





12

Step 4: Set Up Checkpoint Files and Trail Files in DBFS

1. Set up checkpoint files.

Checkpoint files contain the current read and write positions of the Extract and Replicat

processes. Checkpoints provide fault tolerance by preventing the loss of data should the

system, the network, or an Oracle GoldenGate process need to be restarted.

Placing the checkpoint files on the local file system will not provide high availability in the

event of a database node failure. A checkpoint table can be used to record Replicat checkpoint

information to provide an alternative method of fault tolerance.

To store the checkpoint files on DBFS, the best practice is to create a symbolic link from the

Oracle GoldenGate home directory to a directory in DBFS. This is detailed in the DBFS

configuration in Step 1: Set Up DBFS on Oracle RAC.

Note: Oracle GoldenGate uses file locking on the checkpoint files to determine if the Extract

or Replicat processes are already running. This would normally prevent the process from

being started a second time on another Oracle RAC node that has access to the checkpoint

files. DBFS does not support this method of file locking. Mounting DBFS on a single Oracle

RAC node prevents access to the checkpoint files from other nodes. This will in turn prevent

the Extract or Replicat from being started concurrently on multiple nodes.

2. Set up trail files.

Trail files contain the data extracted from the archived redo log files. The trail files are

automatically generated by the Extract process.

To configure Oracle GoldenGate trail files on DBFS for the source database:

1. Create and mount the DBFS file system.

This is detailed in Step 1: Set Up DBFS on Oracle RAC.

2. Set the EXTTRAIL Extract parameter:

EXTTRAIL /mnt/dbfs/goldengate/dirdat/aa

3. After creating the Extract, use the same EXTTRAIL parameter value to add the local trail:

% ggsci

GGSCI (ggtest.oracle.com) 1> ADD EXTTRAIL

/mnt/dbfs/goldengate/dirdat/aa, EXTRACT ext_db, Megabytes 500


13

Further instructions about creating the Extract are available in the Oracle GoldenGate Windows

and UNIX Administration Guide at


To configure Oracle GoldenGate trail files on DBFS for the target database:

1. Make sure the DBFS directory is already created on the target environment.

2. Set the EXTTRAIL Replicat parameter, as follows:

EXTTRAIL /mnt/dbfs/goldengate/dirdat/aa

3. When adding the Replicat, use the same EXTTRAIL parameter value:

% ggsci

GGSCI (ggtest.oracle.com) 1> ADD REPLICAT rep_db1, EXTTRAIL

/mnt/dbfs/goldengate/dirdat/aa

Do not place trail files on the local file system because it will lengthen restart times in the

event of a node failure, reducing availability.

To configure Data Pump between a source and target database:

1. Make sure Extract and Replicat are configured

2. Set the RMTHOST Data Pump parameter to the IP or hostname that will be used for

connecting to the target. In Step 7: Oracle Clusterware Configuration, the Application Virtual

IP address is created with Cluster Ready Services (CRS) so that a single IP address can be

moved between compute nodes, so that Data Pump can continue to connect to the target

host when it moves from a failed node to a surviving node:

RMTHOST gg_dbmachine, MGRPORT 8901

3. Set the RMTTRAIL Data Pump parameter to the trail file location on the target host:

RMTTRAIL /mnt/dbfs/goldengate/dirdat/aa



14

4. Create a Data Pump process using the local trail file location on the source host:

% ggsci

GGSCI (ggtest.oracle.com) 1> ADD EXTRACT dpump_1, EXTTRAILSOURCE

/mnt/dbfs/goldengate/dirdat/aa

5. Use the ADD RMTTRAIL command to specify the remote trail file location on the target

host:

% ggsci

GGSCI (ggtest.oracle.com) 1> ADD RMTTRAIL

/mnt/dbfs/goldengate/dirdat/aa EXTRACT dpump_1, MEGABYTES 500

Further instructions about creating the Data Pump process are available in the Oracle

GoldenGate Administration Guide at


Step 5: Configure Replicat Commit Behavior

With Oracle GoldenGate version 11.2.1 and above, if a checkpoint table is configured, Replicat will

automatically operate using COMMIT NOWAIT. The Replicat processes will no longer wait at each

commit when applying transactions, increasing throughput performance.

For instructions on creating the checkpoint table, refer to the Oracle GoldenGate Oracle Installation and

Setup Guide, Chapter 4:


If you are using Oracle GoldenGate version 11.1.1 or earlier, setting the Replicat commit behavior to

COMMIT NOWAIT is configured separately from creating a checkpoint table. This should only be

considered when using a checkpoint table due to protection of recovery data during a checkpoint.

Set the Replicat parameter file to COMMIT NOWAIT as follows:

SQLEXEC "ALTER SESSION SET COMMIT_WRITE='NOWAIT'";

Step 6: Configure Autostart of Extract, Data Pump, and Replicat Processes

Configure the Extract, Data Pump (if used), and Replicat processes to automatically start when the

Manager process is started. Add the following parameter to the Manager parameter file:

AUTOSTART ER *

AUTORESTART ER *




15

If required, instead of using the Oracle GoldenGate process name wildcard (*), explicitly name the

processes you want to be restarted automatically.

Example:

AUTOSTART EXTRACT EXT_1A

AUTOSTART EXTRACT DPUMP_1A

AUTORESTART EXTRACT EXT_1A

AUTORESTART EXTRACT DPUMP_1A

Step 7: Oracle Clusterware Configuration

The following step-by-step procedure shows how to instruct Oracle Clusterware to start Oracle

GoldenGate using the 11.2.0.3 Oracle Grid Infrastructure Bundled Agent. If using a Grid

Infrastructure release earlier than 11.2.0.3, use the instructions in Appendix A: Creating Oracle

GoldenGate Clusterware Resourceto create and register an Oracle GoldenGate resource with Oracle

Clusterware.

1. Create an Application Virtual IP Address (VIP).

An application VIP is required to ensure the remote data pumps can communicate with the

target database machine, regardless of which node is hosting Oracle GoldenGate.

The VIP is a cluster resource that Oracle Clusterware manages. The VIP is assigned to a

cluster node and will be migrated to another node in the event of a node failure. This allows

Oracle GoldenGate data pump to continue transferring data to the newly assigned target

node.

Perform the following configuration steps:

a. To create the application VIP, run the following as the root user:

$GRID_HOME/bin/appvipcfg create -network=1 \

-ip=10.1.41.93 \

-vipname=gg_vip_source \

-user=root

In the example:

$GRID_HOME is the Oracle home in which Oracle 11g Release 2 Grid

infrastructure components have been installed (for example:

/u01/app/grid).

network is the network number that you want to use. With Oracle Clusterware

release 11.2.0.1, you can find the network number using the following command:

crsctl stat res -p |grep -ie .network -ie subnet |grep -ie

name -ie subnet


16

Consider the following sample output:

NAME=ora.net1.network

USR_ORA_SUBNET=10.1.41.0

net1 in NAME=ora.net1.network indicates this is network 1, and the

second line indicates the subnet on which the VIP will be created.

ip is the IP address provided by your system administrator for the new

Application VIP. This IP address must be in the same subnet as determined

above.

gg_vip_source is the name of the application VIP that you will create.

b. Run the following command to give the Oracle Database installation owner

permission to start the VIP:

$GRID_HOME/bin/crsctl setperm resource gg_vip_source -u

user:oracle:r-x

c. As the Oracle Database installation owner, start the VIP resource:

$GRID_HOME/bin/crsctl start resource gg_vip_source

d. To validate whether the VIP is running and on which node it is running, execute:

$GRID_HOME/bin/crsctl status resource gg_vip_source

See the Oracle Clusterware documentation for further details about creating an Application

VIP: http://docs.oracle.com/cd/E11882_01/rac.112/e16794/crschp.htm#CHDGHGJA

2. Configure Oracle Grid Infrastructure Bundled Agent.

Introduced in release 11.2.0.3 for 64bit Linux, the Oracle Infrastructure Bundled Agents

provide pre-defined Oracle Clusterware resources for Oracle GoldenGate, Siebel, and Apache

applications. Using the bundled agent for Oracle GoldenGate, it is simple to create

dependencies on the source/target database, the application VIP, and the DBFS mount point.

The agent command line utility (AGCTL) is used to start and stop Oracle GoldenGate and

can also be used to relocate Oracle GoldenGate between the nodes in the cluster.

The bundled agent for Oracle GoldenGate fully supports use of Extract running in classic or

integrated capture modes.

The current version certification matrix:

Grid Infrastructure Oracle GoldenGate Oracle Database

11.2.0.3.+ 11.2.1.+ 10.2.0.5, 11.2.0.2+

The bundled agent software should be downloaded from the following location:

http://docs.oracle.com/cd/E11882_01/rac.112/e16794/crschp.htm#CHDGHGJA


17

http://www.oracle.com/technetwork/products/clusterware/downloads/index.html

Follow the installation instructions provided in the readme.txt file.

Example configuration of the bundled agent:

a. Configure DBFS as detailed in My Oracle Support note 1054431.1 and test to make sure CRSCTL can be used to mount and unmount the file system. Mounting the file system should also start the DBFS instance if it is not already running. NOTE: It is important that the DBFS resource is registered with Oracle Clusterware for mounting and unmounting DBFS. If the Oracle Clusterware resource is not registered and tested, as described in MOS note 1054431.1, it will not be possible to use the Bundled Agent to automatically mount and unmount DBFS.

b. Use AGCTL to create the Oracle Clusterware resource:

On the Source environment

% agctl add goldengate GG_Source --gg_home

/home/oracle/goldengate \

--instance_type source \

--nodes dbm01db05,dbm01db06 \

--vip_name gg_vip_source \

--filesystems dbfs_mount --databases ggs \

--oracle_home /u01/app/oracle/product/11.2.0.3/dbhome_1 \

--monitor_extracts ext_1a,dpump_1a

On the Target environment:

% agctl add goldengate GG_Target --gg_home

/home/oracle/goldengate \

--instance_type target \

--nodes dbm03db01,dbm03db02 \

--vip_name gg_vip_target \

--filesystems dbfs_mount --databases ggt \

--oracle_home /u01/app/oracle/product/11.2.0.3/dbhome_1 \

--monitor_replicats rep_1a,rep_2a,rep_3a,rep_3b

NOTE: The dbfs_mount value for the --filesystems parameter is the name of

the resource registered with Oracle Clusterware. This should have been

carried out as described in My Oracle Support note 1054431.1.

Once the Oracle GoldenGate processes have been added to a bundled they should

only be started and stopped using AGCTL. The bundled agent starts the Oracle

GoldenGate processes by starting Manager which in turn will automatically start the

processes (Extract, Data Pump, Replicat) that have been configured to autostart. If

an Oracle GoldenGate process aborts due to a problem, as long as the manager

process is still running it is okay to use GGSCI to restart the failed process.

To check the status of Oracle GoldenGate:

http://www.oracle.com/technetwork/products/clusterware/downloads/index.html




18

% agctl status goldengate GG_Source

Goldengate instance 'GG_Source' is running on dbm01db06

GGCSI can be used to check the status of individual Oracle GoldenGate processes.

To start Oracle GoldenGate manager, and all processes that have autostart enabled:

% agctl start goldengate GG_Target [--serverpool serverpool_name

| --node node_name]

Note: Oracle GoldenGate will start up on the node you issue the command from,

unless a node name or serverpool name is specified.

To stop all Oracle GoldenGate processes:

% agctl stop goldengate GG_Target [--serverpool serverpool_name

| --node node_name]

To relocate Oracle GoldenGate to another node:

% agctl relocate goldengate GG_Source [--serverpool

serverpool_name | --node node_name]

Note: The Oracle GoldenGate resource MUST be running before relocating it.

To view the configuration parameters for the Oracle GoldenGate resource:

% agctl config goldengate GG_Target

GoldenGate location is: /home/oracle/goldengate

GoldenGate instance type is: source

Configured to run on Nodes: dbm01db05 dbm01db06

ORACLE_HOME location is:

/u01/app/oracle/product/11.2.0.3/dbhome_1

Databases needed: ggs

File System resources needed: dbfs_mount

Extracts to monitor: ext_1a,dpump_1a

Replicats to monitor:

To view more detailed clusterware configuration information:


19

% crsctl stat res -w "NAME = xag.GG_Source.goldengate" –p

To delete the Oracle GoldenGate resource:

% agctl remove goldengate GG_Source

Further information on the Oracle Grid Infrastructure Bundled Agent:

http://www.oracle.com/technetwork/products/clusterware/overview/ogiba-

reference-guide-v1-1844341.html

http://www.oracle.com/technetwork/products/clusterware/overview/ogiba-reference-guide-v1-1844341.html

http://www.oracle.com/technetwork/products/clusterware/overview/ogiba-reference-guide-v1-1844341.html


20

Appendix A: Creating Oracle GoldenGate Clusterware Resource

If the Grid Infrastructure release is earlier than 11.2.0.3, Oracle GoldenGate can still be registered as a

resource with Oracle Clusterware using the following instructions.

1. Create an agent script (if not using the Bundled Agent)

If you are not able to use the Oracle Grid Infrastructure Bundled Agent it is still possible to

use Oracle Clusterware to manage automatic starting and stopping of Oracle GoldenGate.

Oracle Clusterware runs resource-specific commands through an entity called an agent. The

agent script:

Must be able to accept five parameter values: start, stop, check, clean and

abort.

Must be stored in the same location on all nodes. In this example, it is stored in the

Grid Infrastructure ($GRID_HOME) ORACLE_HOME/crs/script directory.

Must be owned by the Oracle Grid Infrastructure user and have execute permissions.

Must be accessible at the same location on every node in the cluster.

Must include environment variable settings for ORACLE_HOME, ORACLE_SID,

PATH, TNS_ADMIN and LD_LIBRARY path so that CRS will be able to find the

correct program executables and Oracle Net configuration. If the correct sqlnet.ora,

tnsnames.ora or dbfs_client executable cannot be found when CRS tries to mount

DBFS it will fail.

See Appendix B: Example Agent Script for an example agent script starts and stops the

Oracle GoldenGate Manager, Extract, Data Pump, and Replicat processes.

It is important to manually test the agent script to start and stop the Oracle GoldenGate

processes before moving onto the next step.

2. Register a resource in Oracle Clusterware.

Register Oracle GoldenGate as a resource in Oracle Clusterware using the CRSCTL utility.

When using DBFS to store Oracle GoldenGate files, follow the configuration guidelines

provided in My Oracle Support note 1054431.1 (detailed above). Mounting of DBFS is carried

out by an Oracle Clusterware resource which in turn has a start dependency on the DBFS

instance. IF the DBFS resource is started and the instance is not running, it will be started

automatically. The DBFS mount resource is named as a start dependency for the Oracle

GoldenGate resource such that the required file systems are mounted BEFORE the Oracle

GoldenGate processes are started.

It is also recommended to list the source or target databases as start dependencies for the

Oracle GoldenGate resource so that the Extract or Replicat processes won’t fail when it can’t

connect to the database.



21

1. Determine the name of the DBFS or source/target database resource for the start

dependency:

crsctl status resource | grep -i dbfs|source/target DB name

2. Use the Oracle Grid Infrastructure user (oracle, in this example) to execute the

following:

$GRID_HOME/bin/crsctl add resource GG_Source \


-attr

"ACTION_SCRIPT=/u01/app/11.2.0/grid/crs/script/11gr2_gg_action.scr,

CHECK_INTERVAL=30,

START_DEPENDENCIES='hard(gg_vip_source,dbfs_mount,ora.ggs.db)

pullup(gg_vip_source)', STOP_DEPENDENCIES=’hard(gg_vip_source)’,

SCRIPT_TIMEOUT=300"

If an Application VIP is not used, issue the following command:



-attr


CHECK_INTERVAL=30,

START_DEPENDENCIES='hard(ora.ggs.db,dbfs_mount)

pullup(ora.ggs.db,dbfs_mount) attraction(ora.ggs.db)',

STOP_DEPENDENCIES='hard(intermediate:ora.ggs.db,intermediate:dbfs_m

ount)', SCRIPT_TIMEOUT=300"

If the source and target databases are within the same cluster, make sure the Extract and

Replicat is restricted to the designated cluster nodes:



-attr


CHECK_INTERVAL=30, START_DEPENDENCIES='hard(gg_vip_source,dbfs_mount)

pullup(gg_vip_source)', STOP_DEPENDENCIES=’hard(gg_vip_source)’,

HOSTING_MEMBERS=’dbm01db05 dbm01db06’, PLACEMENT=’restricted’,

SCRIPT_TIMEOUT=300"

For more information about the CRSCTL add resource command and its options, see

the Oracle Clusterware Administration and Deployment Guide at

http://docs.oracle.com/cd/E11882_01/rac.112/e16794/toc.htm

3. Start the resource.

Once the resource has been added, you should always use Oracle Clusterware to start Oracle

GoldenGate. Login as the Oracle Grid Infrastructure user (oracle) and execute the

following:

% $GRID_HOME/bin/crsctl start resource GG_Source

http://www.oracle.com/pls/db112/lookup?id=CWADD

http://docs.oracle.com/cd/E11882_01/rac.112/e16794/toc.htm


22

To check the status of the application, enter the command:

% $GRID_HOME/bin/crsctl status resource GG_Source

For example:

% crsctl status resource GG_Source

NAME=GG_Source

TYPE=cluster_resource

TARGET=ONLINE

STATE=ONLINE on dbm01db05

4. Manage the application.

To relocate Oracle GoldenGate onto a different cluster node, use the

‘$GRID_HOME/bin/crsctl relocate resource’ API and include the force (-f)

option. This command can be run on any node in the cluster as the Grid Infrastructure user

(oracle).

If there is a dependency on an Application VIP you need to relocate the VIP resource which

in turn will stop Oracle GoldenGate, relocate and restart it on the new node.

For example:

[oracle@dbm01db05 ~]$ crsctl relocate resource gg_vip_source –f

CRS-2673: Attempting to stop 'GG_Source' on 'dbm01db05'

CRS-2677: Stop of 'GG_Source' on 'dbm01db05' succeeded

CRS-2673: Attempting to stop 'gg_vip_source' on 'dbm01db05'

CRS-2677: Stop of 'gg_vip_source' on 'dbm01db05' succeeded

CRS-2672: Attempting to start 'gg_vip_source' on 'dbm01db06'

CRS-2676: Start of 'gg_vip_source' on 'dbm01db06' succeeded

CRS-2673: Attempting to stop 'dbfs_mount' on ' dbm01db05'

CRS-2677: Stop of 'dbfs_mount' on ' dbm01db05' succeeded

CRS-2672: Attempting to start 'dbfs_mount' on 'dbm01db06'

CRS-2676: Start of 'dbfs_mount' on 'dbm01db06' succeeded

CRS-2672: Attempting to start 'GG_Source' on 'dbm01db06'

CRS-2676: Start of 'GG_Source' on 'dbm01db06' succeeded

To stop the Oracle GoldenGate resource, enter the following command:

$GRID_HOME/bin/crsctl stop resource GG_Source

5. Perform CRS cleanup.

To remove Oracle GoldenGate from Oracle Clusterware management, perform the following

tasks:

a) Stop Oracle GoldenGate (login as the Oracle Grid Infrastructure (oracle) user):


23

$GRID_HOME/bin/crsctl stop resource GG_Source

b) As the root user, delete the Oracle GoldenGate resource:

$GRID_HOME/bin/crsctl delete resource GG_Source

c) If no longer needed, delete the agent action script: 11gr2_gg_action.scr.

This does not delete the Oracle GoldenGate or DBFS configuration, only the Oracle

Clusterware resource.

Recommendations When Deploying on Oracle RAC

When Oracle GoldenGate is configured in an Oracle RAC environment, follow these

recommendations:

Ensure that the DBFS database has instances running on all the database nodes involved in the

Oracle RAC configuration.

This action provides access to Oracle GoldenGate if it is restarted after a node failure.

Ensure that the DBFS file system is mountable on all database nodes in the Oracle RAC

configuration. Mounting DBFS using a Clusterware resource to prevent the Extract or Replicat

processes from being started on multiple nodes concurrently, mount the file system only on the

node where Oracle GoldenGate is running. Use the same mount point names on all the nodes to

ensure seamless failover.


24

Appendix B: Example Agent Script

The following example agent starts and stops the Oracle GoldenGate Manager, Extract, and Replicat

processes. The mounting and unmounting of DBFS is handled by the DBFS Clusterware resource,

created in step 2 of Appendix A: Creating Oracle GoldenGate Clusterware Resource. The agent script

accepts the parameter values: start, stop, check, clean, and abort.

#!/bin/bash

#11gr2_gg_action.scr

# Edit the following environment variables:

# NOTE: The ORACLE_SID will be different on each node

export ORACLE_HOME=/u01/app/oracle/product/11.2.0.3/dbhome_1

export ORACLE_SID=GGS1

GGS_HOME=/home/oracle/goldengate/latest

export PATH=$ORACLE_HOME/bin:$PATH

export TNS_ADMIN=$ORACLE_HOME/network/admin

#Include the GoldenGate home in the library path to start GGSCI

export LD_LIBRARY_PATH=${ORACLE_HOME}/lib:${LD_LIBRARY_PATH}:${GGS_HOME}

# Edit this to indicate the DBFS mount point

DBFS_MOUNT_POINT=/mnt/dbfs

# Edit this to indicate at least one of the file systems mounted in

# the DBFS mount point

DBFS_FILE_SYSTEM=/mnt/dbfs/goldengate

# Edit for correct Extract/Datapump name if running this script on

# the source (widlcard is appended to these names in the script):

EXTRACT=EXT

DATAPUMP=DPUMP

# Edit for current Replicat names if running script on the target

# (widlcard is appended to these names in the script):

REPLICAT=REP

# Specify delay after start before checking for successful start

start_delay_secs=5

### Syslog facility name (default user)


25

### Changed default from local3 to user for Solaris default support on 17-

FEB-2012

### This will allow us to log messages to the syslog

### (/var/log/messages on Linux, /var/adm/messages on Solaris)

LOGGER_FACILITY=user

###########################################

### No editing is required below this point

###########################################

### determine platform

UNAME_S=`uname -s`

if [ $UNAME_S = 'Linux' ]; then LINUX=1; SOLARIS=0;

elif [ $UNAME_S = 'SunOS' ]; then LINUX=0; SOLARIS=1;

fi

LOGGER="/bin/logger -t GoldenGate"

logit () {

### type: info, error, debug

type=$1

msg=$2

if [ "$type" = "info" ]; then

echo $msg

$LOGGER -p ${LOGGER_FACILITY}.info "$msg"

elif [ "$type" = "error" ]; then

echo $msg

$LOGGER -p ${LOGGER_FACILITY}.error "$msg"

elif [ "$type" = "debug" ]; then

echo $msg

$LOGGER -p ${LOGGER_FACILITY}.debug "$msg"

fi

}

# check_process validates that Manager/Extract/Replicat process is

# running #at PID that GoldenGate specifies.

check_process () {

PROCESS=$1

if [ ${PROCESS} = mgr ]

then

PFILE=MGR.pcm


26

elif [ ${PROCESS} = ext ]

then

PFILE=${EXTRACT}*.pce

elif [ ${PROCESS} = rep ]

then

PFILE=${REPLICAT}*.pcr

else

PFILE=${DATAPUMP}*.pce

fi

if ( [ -f "${GGS_HOME}/dirpcs/${PFILE}" ] )

then

pid=`cut -f8 "${GGS_HOME}/dirpcs/${PFILE}"`

if [ ${pid} = `ps -e |grep ${pid} |grep ${PROCESS} |cut -d " " -f2` ]

then

logit info "${SCRIPTNAME}(check_process) - Procese(s) ${PROCESS} IS

running"

exit 0

else

if [ ${pid} = `ps -e |grep ${pid} |grep ${PROCESS} |cut -d " " -f1` ]

then

logit info "${SCRIPTNAME}(check_process) - Procese(s) ${PROCESS} IS

running"

exit 0

else

logit error "${SCRIPTNAME}(check_process) - Procese(s) ${PROCESS} is

NOT running"

exit 1

fi

fi

else

logit error "${SCRIPTNAME}(check_process) - Procese(s) ${PROCESS} is NOT

running - no pid file"

exit 1

fi

}

# call_ggsci is a generic routine that executes a ggsci command

call_ggsci () {

ggsci_command=$1

ggsci_output=`${GGS_HOME}/ggsci << EOF

${ggsci_command}

exit


27

EOF`

}

stop_everything () {

# Before starting, make sure everything is shutdown and process files are

removed

#attempt a clean stop for all non-manager processes

logit info "${SCRIPTNAME}(stop_everything) - Stopping all processes"

call_ggsci 'stop er *'

#ensure everything is stopped

call_ggsci 'stop er *!'

#in case there are lingering processes

call_ggsci 'kill er *'

#stop Manager without (y/n) confirmation

call_ggsci 'stop manager!'

#Remove the process files:

rm -f $GGS_HOME/dirpcs/MGR.pcm

rm -f $GGS_HOME/dirpcs/*.pce

rm -f $GGS_HOME/dirpcs/*.pcr

}

case $1 in

'start')

# stop all GG processes and remove process files

logit info "${SCRIPTNAME} - start - Starting all processes"

stop_everything

sleep ${start_delay_secs}

#Now can start everything...

#start Manager

logit info "${SCRIPTNAME} - start - Starting Manager, autostarting

processes"

call_ggsci 'start manager'


28

#there is a small delay between issuing the start manager command

#and the process being spawned on the OS <96> wait before checking


#start Extract or Replicats

call_ggsci 'start er *'

#check whether Manager is running and exit accordingly

logit info "${SCRIPTNAME} - start - Checking Manager status"

check_process mgr


#Check whether Extract is running

logit info "${SCRIPTNAME} - start - Checking GoldenGate statuses"

check_process ext

check_process dpump

;;

'stop')


logit info "${SCRIPTNAME} - stop - Stopping all processes"

stop_everything

#exit success

exit 0

;;

'check')

logit info "${SCRIPTNAME} - check - Checking all processes"

check_process mgr

check_process ext

check_process dpump

check_process rep

;;

'clean')


logit info "${SCRIPTNAME} - clean - Stopping all processes"

stop_everything


29

#exit success

exit 0

;;

'abort')


logit info "${SCRIPTNAME} - abort - Stopping all processes"

stop_everything

#exit success

exit 0

;;

esac


30

References

Oracle GoldenGate Windows and UNIX Administration Guide version 11.2.1.0.1

Oracle GoldenGate Oracle Installation and Setup Guide version 11.2.1.0.1

Oracle GoldenGate Windows and UNIX Reference Guide version 11.2.1.0.1

Oracle Database SecureFiles and Large Object Developer’s Guide (DBFS)

Oracle Clusterware Administration and Deployment Guide

Oracle Maximum Availability Architecture Web site

http://www.otn.oracle.com/goto/maa




http://docs.oracle.com/cd/E11882_01/appdev.112/e18294.pdf

http://docs.oracle.com/cd/E11882_01/rac.112/e16794.pdf

http://www.otn.oracle.com/goto/maa

Oracle GoldenGate with Oracle Real

Application Clusters Configuration

August 2013

Author: Stephan Haisley

Contributing Authors: MAA team

Oracle Corporation

World Headquarters

500 Oracle Parkway

Redwood Shores, CA 94065

U.S.A.

Worldwide Inquiries:

Phone: +1.650.506.7000

Fax: +1.650.506.7200

oracle.com

Copyright © 2013, Oracle and/or its affiliates. All rights reserved. This document is provided for information purposes only and the

contents hereof are subject to change without notice. This document is not warranted to be error-free, nor subject to any other

warranties or conditions, whether expressed orally or implied in law, including implied warranties and conditions of merchantability or

fitness for a particular purpose. We specifically disclaim any liability with respect to this document and no contractual obligations are

formed either directly or indirectly by this document. This document may not be reproduced or transmitted in any form or by any

means, electronic or mechanical, for any purpose, without our prior written permission.

Oracle and Java are registered trademarks of Oracle and/or its affiliates. Other names may be trademarks of their respective owners.

AMD, Opteron, the AMD logo, and the AMD Opteron logo are trademarks or registered trademarks of Advanced Micro Devices.

Intel and Intel Xeon are trademarks or registered trademarks of Intel Corporation. All SPARC trademarks are used under license

and are trademarks or registered trademarks of SPARC International, Inc. UNIX is a registered trademark licensed through X/Open

Company, Ltd. 1010

Documents

Oracle GoldenGate with Oracle Real Application Clusters