ITNM 4.2 wOracle Active DataGuard - IBM Developer · 6 3 Installation of Oracle 12c Enterprise Edition Preparing VMs: o Login to primno1.fragglerock.wipro.com(10.162.64.172) as root

ITNM 4.2 DR with Oracle active dataguard

i

ORACLE 12C ADG WITH ITNM 4.2

Contents Contents ......................................................................................................................................................... i

1 Setup Details ......................................................................................................................................... 2

2 Architecture .......................................................................................................................................... 5

3 Installation of Oracle 12c Enterprise Edition ........................................................................................ 6

4 Preparing Databases for Active DataGuard ........................................................................................ 10

4.1 Configuring Primary Instance ...................................................................................................... 10

4.2 Configuring Standby Instance ........................................................................................................... 19

4.3 Configuring FarSync Instance ............................................................................................................ 26

5 Configuring DataGuard with DGMGRL................................................................................................ 33

5.1 Enabling Fast-Start Failover ........................................................................................................ 35

5.2 Enabling Fast-Start Failover and Starting the Observer .............................................................. 37

5.3 Performing a Manual Failover Operation ................................................................................... 42

6 Configuring ObjectServer Failover ...................................................................................................... 43

6.1 Configuring ObjectServers and gateways for failover ................................................................ 43

6.2 Sample configuration for the basic failover architecture (aggregation layer only) .................... 44

6.3 Connecting to an ObjectServer failover pair ............................................................................... 46

6.4 Configuring ObjectServer authentication ................................................................................... 46

7 Omnibus WebGUI/DASH HA Configuration ........................................................................................ 48

7.1 Configuring data source failover for the Tivoli Netcool/OMNIbus Web GUI ............................. 48

7.1.1 Sample ncwDataSourceDefinitions.xml configuration for data source failover ................. 50

7.2 Configuration needed on the DB2 server ................................................................................... 51

7.3 Configuration needed on each WebGUI server .......................................................................... 51

7.4 Creating users in Omnibus .......................................................................................................... 69

8 Installation of ITNM on primary and Standby Instances .................................................................... 70

8.1 Installing ITNM on Primary Instance ........................................................................................... 70

8.2 Installing ITNM on Physical Standby: .......................................................................................... 73

9 Enabling Rsync between Primary and Standby servers ...................................................................... 75

9.1 Creating rsync files ...................................................................................................................... 75

9.1.1 Create rsync files on primno1 : ........................................................................................... 75

9.1.2 Create rsync files on stbyno1 : ............................................................................................ 78

ii

9.2 Enabling Passwordless Authentication ....................................................................................... 80

9.3 Running rsyng scripts .................................................................................................................. 80

9.4 Stopping rsync ............................................................................................................................. 81

10 Role Change : Failover, Switchover and Switchback ....................................................................... 83

10.1 Fail-over operation with oracle ADG .......................................................................................... 83

10.1.1 Reinstating a Failed Primary Database ............................................................................... 83

10.2 Switchover (Primary to Standby) ................................................................................................ 88

10.2.1 ITNM Level .......................................................................................................................... 88

10.2.2 NCIM Level .......................................................................................................................... 88

10.2.3 Post Switchover Steps ......................................................................................................... 92

10.3 Switchback (Standby to Primary) ................................................................................................ 94

10.3.1 ITNM Level .......................................................................................................................... 94

10.3.2 NCIM Level .......................................................................................................................... 94

10.3.3 Post Switchover Steps ......................................................................................................... 99

11 Testing Oracle DataGuard Setup ................................................................................................... 100

11.1 Monitoring FRA space ............................................................................................................... 100

11.2 Testing ITNM functionality ........................................................................................................ 103

11.3 Testing Failover ......................................................................................................................... 107

11.4 Testing Rsync ............................................................................................................................. 111

iii

1

PART-I

Installing and Configuring the Setup

2

1 Setup Details

Primary Physical Standby FarSync

Hostname primno1.fragglerock.wipro.com stbyno1.fragglerock.wipro.com farsync.fragglerock.wipro.com

IP 10.162.64.172 10.45.41.202 10.162.64.137

Location Chennai DataCenter London DC Chennai DC

OS Linux CentOS Linux CentOS Linux CentOS

OS Version CentOS Linux release 7.6.1810

(Core)

CentOS Linux release 7.6.1810

(Core)

CentOS Linux release 7.6.1810

(Core)

Oracle

version

Oracle 12.1 R2 (12.1.0.2.0) Oracle 12.1 R2 (12.1.0.2.0) Oracle 12.1 R2 (12.1.0.2.0)

Adding exceptions:

The following range of ports will be used during the course of ITNM and there after. It is

mandatory to allow these ports through firewall to avoid any inconvenience in accessing

ITNM services from outside the ITNM server. The ports 16311-16320 are the default

ports used by ITNM and the port 1521 will be used by Oracle listener. ( If the ports are

not available, or ITNM/Oracle choose another port, those ports should be allowed

through firewall)

Example :

$ firewall-cmd --zone=public --add-port=16311/tcp --permanent











$ firewall-cmd --reload

$ sudo iptables-save

The above ports should be added as exception on all the three instances.

3

4

Adding Hostname aliasing:

Add the following hostname aliasing in “/etc/hosts” file on all the three instances.

#---------------------------------------------------------------------------------------------------

10.162.64.172 primno1 primno1.fragglerock.wipro.com

10.162.64.137 farsync farsync.fragglerock.wipro.com

10.45.41.202 stbyno1 stbyno1.fragglerock.wipro.com

#---------------------------------------------------------------------------------------------------

5

2 Architecture

6

3 Installation of Oracle 12c Enterprise Edition Preparing VMs:

o Login to primno1.fragglerock.wipro.com(10.162.64.172) as root.

o Create Users and Groups

# groupadd oinstall

# groupadd dba

# groupadd oper

# useradd -G dba,oper -g oinstall -d /home/oracle oracle

o Create password for oracle

# passwd oracle

- Enter a password

- Verify password

o Give permissions for oracle user

# chown -R oracle:oinstall /home/oracle/

# chmod 775 -R /home/oracle/

o Edit file /etc/sysctl.conf

# vi /etc/sysctl.conf

#---------------------------------------------------------------------------------------------------

fs.file-max = 6815744

kernel.sem = 250 32000 100 128

kernel.shmmni = 4096

kernel.shmall = 1073741824

kernel.shmmax = 4398046511104

net.core.rmem_default = 262144

net.core.rmem_max = 4194304

net.core.wmem_default = 262144

net.core.wmem_max = 1048576

fs.aio-max-nr = 1048576

net.ipv4.ip_local_port_range = 9000 65500

#---------------------------------------------------------------------------------------------------

Execute /sbin/sysctl -p to save changes

# /sbin/sysctl -p

7

o Set the limits

# vi /etc/security/limits.conf

#--------------------------------------------------------------------------------------------------

oracle soft nofile 1024

oracle hard nofile 65536

oracle soft nproc 16384

oracle hard nproc 16384

oracle soft stack 10240

oracle hard stack 32768

#---------------------------------------------------------------------------------------------------

o Set Environment Variables for oracle user

Switch user to oracle

# su - oracle

Edit bash profile

# vi .bash_profile

.Add following lines(Give the relevant hostname and unique_name)

#---------------------------------------------------------------------------------------------------

export TMP=/tmp

export TMPDIR=$TMP

export ORACLE_HOSTNAME=primno1.fragglerock.wipro.com

export ORACLE_UNQNAME=ODB

export ORACLE_BASE=/home/oracle/app/oracle

export ORACLE_HOME=$ORACLE_BASE/product/12.1.0/dbhome_1

export ORACLE_SID=ODB

export PATH=/usr/sbin:$PATH

export PATH=$ORACLE_HOME/bin:$PATH

export LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib

export CLASSPATH=$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib

#---------------------------------------------------------------------------------------------------

8

Installing Oracle 12c:

Navigate to the directory containing Installation files. Run the installer

# ./runInstaller

1.Uncheck "I wish to receive ****" next->

2.Install database software only next->

3.Single database instance next->

4.Enterprise Edition next->

5. Oracle base : /home/oracle/app/oracle

Software location /home/oracle/app/oracle/product/12.1.0/dbhome_1

next->

6.Create Inventory (leave as default)

next->

7.Operating system Groups(leave as default)

next->

8.Prereqs

If any packages are missing, run the following command

# yum install <Package_Name>

Check again

next->

9.Save Response file

10.Install

Once the installation is done, you’ll be asked to run two scripts

/home/oracle/oraInventory/orainstRoot.sh

/home/oracle/app/oracle/product/12.1.0/dbhome_1/root.sh

o Enter the full pathname of the local bin directory: [/usr/local/bin]:

(leave as default)(hit enter for it)

o Do you want to setup Oracle Trace File Analyzer (TFA) now? yes|[no]: No

9

The installation of Oracle 12C Enterprise Edition is completed on Primary instance.

Same procedure has to be followed to install Oracle 12c on both Physical Standby and FarSync

instances.

While configuring .bash_profile, following care should be taken.

Physical Standby :

export ORACLE_HOSTNAME=stbyno1.fragglerock.wipro.com

export ORACLE_UNQNAME=ODB_STBY

FarSync :

export ORACLE_HOSTNAME=farsync.fragglerock.wipro.com

export ORACLE_UNQNAME=ODB_FS

Database Creation:

The database should be created only on Primary instance.

# su - oracle

# $ORACLE_HOME/bin/dbca

You’ll see a pop-up window of Oracle DBCA(Database Configuration Assistance)

o Create a database

Next ->

o Typical configuration

Give relevant details

Next ->

o Finish

Note : Make sure that listener is running on port 1521. If listener port is other than 1521, it

should be allowed through firewall.

10

4 Preparing Databases for Active DataGuard

4.1 Configuring Primary Instance

Logon to primary instance as oracle.

# su - oracle

Create the following folders

# mkdir - p /home/oracle/app/oracle/oradata/ODB/onlinelog

# mkdir - p /home/oracle/app/oracle/fast_recovery_area/ODB/onlinelog

Run the following commands in SQL to make appropriate changes.

# sqlplus sys/Wipro_123@ODB as sysdba (Wipro_123 is the password set at the

creation of database)

SQL> SELECT log_mode FROM v$database;

LOG_MODE

------------

NOARCHIVELOG

SQL> SHUTDOWN IMMEDIATE;

SQL> STARTUP MOUNT;

SQL> ALTER DATABASE ARCHIVELOG;

SQL> ALTER DATABASE OPEN;

SQL> ALTER DATABASE FORCE LOGGING;

SQL> ALTER SYSTEM SWITCH LOGFILE;

SQL> ALTER DATABASE FLASHBACK ON;

SQL> ALTER SYSTEM SET STANDBY_FILE_MANAGEMENT=AUTO;

SQL> ALTER DATABASE ADD STANDBY LOGFILE thread 1 GROUP 4 (

'/home/oracle/app/oracle/oradata/ODB/onlinelog/stdbyrdo41.log',

'/home/oracle/app/oracle/fast_recovery_area/ODB/onlinelog/stdbyrdo42.log'

) SIZE 50M BLOCKSIZE 512;

SQL>

11

ALTER DATABASE ADD STANDBY LOGFILE thread 1 GROUP 5 (




SQL>





SQL>





Check the groups created :

SQL> select group#,bytes/1024/1024,members from v$log;

SQL> select member,group# from v$logfile;

Create tnsnames.ora file in $ORACLE_HOME/network/admin folder

# vi $ORACLE_HOME/network/admin/tnsnames.ora

#----------------------------------------------------------------------------------

ODB_FS =

(DESCRIPTION =

(ADDRESS_LIST =

12

(ADDRESS = (PROTOCOL = TCP)(HOST = farsync.fragglerock.wipro.com)(PORT =

1521))

)

(CONNECT_DATA =

(SID = ODB)

)

)

ODB_STBY =

(DESCRIPTION =

(ADDRESS_LIST =

(ADDRESS = (PROTOCOL = TCP)(HOST = stbyno1.fragglerock.wipro.com)(PORT =

1521))

)

(CONNECT_DATA =

(SID = ODB)

)

)

ODB =

(DESCRIPTION =

(ADDRESS = (PROTOCOL = TCP)(HOST = primno1.fragglerock.wipro.com)(PORT =

1521))

(CONNECT_DATA =

(SERVER = DEDICATED)

(SERVICE_NAME = ODB)

)

)

13

LISTENER_ODB =

(DESCRIPTION =


1521))

(CONNECT_DATA =

(SERVER = DEDICATED)

(SERVICE_NAME = ODB)

)

)

LISTENER_ODB_STBY =

(DESCRIPTION =

(ADDRESS_LIST =


1521))

)

(CONNECT_DATA =

(SID = ODB)

)

)

LISTENER_ODB_FS =

(DESCRIPTION =

(ADDRESS_LIST =


1521))

)

(CONNECT_DATA =

14

(SID = ODB)

)

)

#--------------------------------------------------------------------------------------------------------

Create a file for listener.

# vi $ORACLE_HOME/network/admin/listener.ora

#--------------------------------------------------------------------------------------------------------

LISTENER =

(DESCRIPTION_LIST =

(DESCRIPTION =


1521))

(ADDRESS = (PROTOCOL = IPC)(KEY = EXTPROC1521))

)

)

SID_LIST_LISTENER =

(SID_LIST =

(SID_DESC =

(GLOBAL_DBNAME = ODB)

(ORACLE_HOME = /home/oracle/app/oracle/product/12.1.0/dbhome_1)

(SID_NAME = ODB)

)

)

ADR_BASE_LISTENER = /home/oracle/app/oracle

#-----------------------------------------------------------------------------------------------

Restart the listener

15

# lsnrctl stop

#lsnrctl start

Edit $ORACLE_HOME/dbs/initODB.ora file to accommodate all configuration parameters.

#-----------------------------------------------------------------------------------------------

ODB.__data_transfer_cache_size=0

ODB.__db_cache_size=7918845952

#ODB.__inmemory_ext_roarea=0

#ODB.__inmemory_ext_rwarea=0

ODB.__java_pool_size=67108864

ODB.__large_pool_size=100663296

ODB.__oracle_base='/home/oracle/app/oracle'

#ORACLE_BASE set from environment

ODB.__pga_aggregate_target=3388997632

ODB.__sga_target=10099884032

ODB.__shared_io_pool_size=536870912

ODB.__shared_pool_size=1442840576

ODB.__streams_pool_size=0

*.audit_file_dest='/home/oracle/app/oracle/admin/ODB/adump'

*.audit_trail='db'

*.compatible='12.1.0.2'

*.control_files='/home/oracle/app/oracle/oradata/ODB/control01.ctl','/home/oracle/app/oracle/fas

t_recovery_area/ODB/control02.ctl'

*.db_block_size=8192

*.db_create_file_dest='/home/oracle/app/oracle/oradata'

*.db_name='ODB'

*.db_unique_name='ODB'

*.db_recovery_file_dest='/home/oracle/app/oracle/fast_recovery_area/ODB'

*.db_recovery_file_dest_size=8016m

16

*.diagnostic_dest='/home/oracle/app/oracle'

*.dispatchers='(PROTOCOL=TCP) (SERVICE=ODBXDB)'

#*.enable_pluggable_database=true

*.log_archive_dest_1='location=use_db_recovery_file_dest valid_for=(all_logfiles,all_roles)

db_unique_name=ODB'

*.log_archive_dest_2='service=ODB_FS SYNC AFFIRM alternate=log_archive_dest_3

valid_for=(online_logfiles,primary_role) db_unique_name=ODB_FS'

*.log_archive_dest_3='service=ODB_STBY ASYNC max_failure=1

alternate=log_archive_dest_2 valid_for=(online_logfiles,primary_role)

db_unique_name=ODB_STBY'

*.log_archive_dest_state_3=alternate

*.log_archive_config='DG_CONFIG=(ODB,ODB_FS,ODB_STBY)'

*.local_listener='LISTENER_ODB'

*.open_cursors=300

*.pga_aggregate_target=3201m

*.processes=640

*.remote_login_passwordfile='EXCLUSIVE'

*.sga_target=9603m

*.standby_file_management='AUTO'

*.undo_tablespace='UNDOTBS1'

*.fal_server='ODB_STBY'

*.dg_broker_start=false;

*.db_file_name_convert='/home/oracle/app/oracle/oradata/ODB_STBY/','/home/oracle/app/oracl

e/oradata/ODB/'

*.log_file_name_convert='/home/oracle/app/oracle/oradata/ODB_STBY/','/home/oracle/app/orac

le/oradata/ODB/'

#-----------------------------------------------------------------------------------------------

Restart the database:

SQL> shutdown immediate

17

SQL> create spfile from pfile;

SQL> startup mount

SQL> select status,instance_name,database_role,open_mode from v$database,v$Instance

STATUS INSTANCE_NAME DATABASE_ROLE OPEN_MODE

------------ ---------------- ---------------- --------------------

OPEN ODB PRIMARY READ WRITE

Create a password file for Standby and FarSync instances:

# orapwd file=$ORACLE_HOME/dbs/orapwODB password=Wipro_123 entries=10

Copy this file to both Standby and FarSync instances :

Create the Control File for Far SYNC:

SQL> alter database create far sync instance controlfile as

'$ORACLE_HOME/control01.ctl';

SQL> create pfile='$ORACLE_HOME/initODB_FS.ora' from spfile;

Create Control File for Physical Standby:

SQL> ALTER DATABASE CREATE STANDBY CONTROLFILE AS

'$ORACLE_HOME/control02.ctl';

Create the folders in far sync...

The destination folders in farsync instance should be in place prier to transferring files to

it. Refer the farsync folders creation section.

Copy files to FarSync instance :

# scp $ORACLE_HOME/dbs/orapwODB

oracle@farsync:/home/oracle/app/oracle/product/12.1.0/dbhome_1/dbs

# scp $ORACLE_HOME/initODB_FS.ora

oracle@farsync:/home/oracle/app/oracle/product/12.1.0/dbhome_1/dbs/initODB_FS.ora

18

# scp $ORACLE_HOME/control01.ctl

oracle@farsync:/home/oracle/app/oracle/oradata/ODB_FS/controlfile/control01_.ctl


oracle@farsync:/home/oracle/app/oracle/fast_recovery_area/ODB_FS/controlfile/control02_.ctl

Copy Control Files to Standby :

Check whether the destination folders are present on Standby instance or not. If not, create them

before transferring the files. Standby folders creation

# scp $ORACLE_HOME/dbs/orapwODB

oracle@stbyno1:/home/oracle/app/oracle/product/12.1.0/dbhome_1/dbs/orapwODB


oracle@stbyno1:/home/oracle/app/oracle/oradata/ODB_STBY/controlfile/control01_.ctl


oracle@stbyno1:/home/oracle/app/oracle/fast_recovery_area/ODB_STBY/controlfile/control02_

.ctl

19

4.2 Configuring Standby Instance

Logon to Physical standby server stbyno1 as oracle user

# su - oracle

Create the following folders

# mkdir -p /home/oracle/app/oracle/admin/ODB_STBY/adump

# mkdir -p /home/oracle/app/oracle/oradata/ODB_STBY/controlfile

# mkdir -p /home/oracle/app/oracle/fast_recovery_area/ODB_STBY/controlfile

# mkdir -p /home/oracle/app/oracle/admin/ODB/adump

# mkdir -p /home/oracle/app/oracle/oradata/ODB/controlfile

# mkdir -p /home/oracle/app/oracle/fast_recovery_area/ODB/controlfile

# mkdir -p /home/oracle/app/oracle/oradata/ODB_STBY/onlinelog

# mkdir -p /home/oracle/app/oracle/oradata/ODB/onlinelog/

# mkdir -p /home/oracle/app/oracle/fast_recovery_area/ODB/onlinelog/

Create the $ORACLE_HOME/dbs/initODB.ora file

# vi $ORACLE_HOME/dbs/initODB.ora

#-----------------------------------------------------------------------------------------------------------

ODB.__data_transfer_cache_size=0


#ODB.__inmemory_ext_roarea=0

#ODB.__inmemory_ext_rwarea=0



ODB.__oracle_base='/home/oracle/app/oracle'

#ORACLE_BASE set from environment



20




*.audit_file_dest='/home/oracle/app/oracle/admin/ODB_STBY/adump'

*.audit_trail='db'

*.compatible='12.1.0.2'

*.control_files='/home/oracle/app/oracle/oradata/ODB_STBY/controlfile/control01_.ctl

','/home/oracle/app/oracle/fast_recovery_area/ODB_STBY/controlfile/control02_.ctl'



*.db_name='ODB'

*.db_unique_name='ODB_STBY'

*.db_recovery_file_dest='/home/oracle/app/oracle/fast_recovery_area/ODB_STBY'



*.dispatchers='(PROTOCOL=TCP) (SERVICE=ODB_STBYXDB)'




*.log_archive_dest_2='service=ODB ASYNC valid_for=(online_logfiles,primary_role)

db_unique_name=ODB'


*.local_listener='LISTENER_ODB_STBY'

*.open_cursors=300


*.processes=640


*.sga_target=9603m


21


#*.dg_broker_start=false;

*.fal_server='ODB_FS','ODB'

*.db_file_name_convert='/home/oracle/app/oracle/oradata/ODB/','/home/oracle/app/oracle/orada

ta/ODB_STBY/'

*.log_file_name_convert='/home/oracle/app/oracle/oradata/ODB/','/home/oracle/app/oracle/orad

ata/ODB_STBY/'

#---------------------------------------------------------------------------------------------------------

Create tnsnames.ora file


#---------------------------------------------------------------------------------------------------------

LISTENER_ODB =

(DESCRIPTION =

(ADDRESS_LIST =


1521))

)

(CONNECT_DATA =

(SID = ODB)

)

)

ODB =

(DESCRIPTION =

(ADDRESS_LIST =


1521))

22

)

(CONNECT_DATA =

(SID = ODB)

)

)

LISTENER_ODB_FS =

(DESCRIPTION =

(ADDRESS_LIST =


1521))

)

(CONNECT_DATA =

(SID = ODB)

)

)

LISTENER_ODB_STBY =

(DESCRIPTION =

(ADDRESS_LIST =


1521))

)

(CONNECT_DATA =

(SID = ODB)

)

)

23

ODB_STBY =

(DESCRIPTION =

(ADDRESS_LIST =


1521))

)

(CONNECT_DATA =

(SID = ODB)

)

)

ODB_FS =

(DESCRIPTION =

(ADDRESS_LIST =


1521))

)

(CONNECT_DATA =

(SID = ODB)

)

)

#------------------------------------------------------------------------------------------------------------------

Create listener file :


#-----------------------------------------------------------------------------------------------------------------

24

LISTENER =

(DESCRIPTION_LIST =

(DESCRIPTION =

(ADDRESS = (PROTOCOL = TCP)(HOST = stbyno1.fragglerock.wipro.com) (PORT =

1521))


)

)

SID_LIST_LISTENER =

(SID_LIST =

(SID_DESC =

(GLOBAL_DBNAME = ODB_STBY)


(SID_NAME = ODB)

)

)


#-----------------------------------------------------------------------------------------------------

Restart the listener :

# lsnrctl stop

# lsnrctl start

# export ORACLE_SID=ODB

Start the database session with created parameter file

# sqlplus / as sysdba

25

SQL> STARTUP NOMOUNT PFILE='$ORACLE_HOME/dbs/initODB.ora';

Duplicate the Primary database to Standby Database with RMAN using Duplicate command

# rman

rman> connect TARGET sys/Wipro_123@ODB

rman> connect AUXILIARY sys/Wipro_123@ODB_STBY

rman> DUPLICATE TARGET DATABASE

FOR STANDBY

FROM ACTIVE DATABASE

DORECOVER

SPFILE

SET db_unique_name='ODB_STBY' COMMENT 'Is standby'

NOFILENAMECHECK;

Restart the database :

# sqlplus sys/Wipro_123@ODB_STBY as sysdba


SQL> startup mount



------------ ---------------- ---------------- --------------------

MOUNTED ODB PHYSICAL STANDBY MOUNTED

26

4.3 Configuring FarSync Instance

Logon to farsync instance as oracle

# su - oracle

Create the following folders :

# mkdir -p /home/oracle/app/oracle/oradata/ODB_FS/controlfile/

# mkdir -p /home/oracle/app/oracle/oradata/ODB_FS/controlfile/

# mkdir -p /home/oracle/app/oracle/oradata/ODB_FS/onlinelog/

# mkdir -p /home/oracle/app/oracle/oradata/ODB_FS/pdbseed

# mkdir -p /home/oracle/app/oracle/oradata/ODB_FS/pdb1

# mkdir -p /home/oracle/app/oracle/oradata/fastpwd_recovery_area/ODB_FS

# mkdir -p /home/oracle/app/oracle/admin/ODB_FS/adump

# mkdir -p /home/oracle/app/oracle/fast_recovery_area/ODB_FS/controlfile/

# mkdir -p /home/oracle/app/oracle/admin/ODB/adump

# mkdir -p /home/oracle/app/oracle/oradata/ODB_FS/onlinelog

# mkdir -p /home/oracle/app/oracle/fast_recovery_area/ODB_FS/onlinelog

Rename initODB_FS.ora file as initODB.ora in $ORACLE_HOME/dbs

# cd $ORACLE_HOME/dbs

# mv initODB_FS.ora initODB.ora

Edit the initODB.ora file on the Far Sync server to change the location of control_files

*.control_files='/home/oracle/app/oracle/oradata/ODB_FS/controlfile/control01_.ctl','/home/or

acle/app/oracle/fast_recovery_area/ODB_FS/controlfile/control02_.ctl'

Check the parameters in initODB.ora file and add/modify the following if not present

# vi $ORACLE_HOME/dbs/initODB.ora

#--------------------------------------------------------------------------------------------------------------

DB.__data_transfer_cache_size=0




ODB.__oracle_base='/home/oracle/app/oracle'#ORACLE_BASE set from environment




27



*.audit_file_dest='/home/oracle/app/oracle/admin/ODB_FS/adump'

*.audit_trail='db'

*.compatible='12.1.0.2.0'

*.control_files='/home/oracle/app/oracle/oradata/ODB_FS/controlfile/control01_.ctl','/home/orac

le/app/oracle/fast_recovery_area/ODB_FS/controlfile/control02_.ctl'


*.db_domain=''


*.db_name='ODB'

*.db_unique_name='ODB_FS'

*.db_recovery_file_dest='/home/oracle/app/oracle/fast_recovery_area/ODB_FS'



*.dispatchers='(PROTOCOL=TCP) (SERVICE=ODB_FSXDB)'



db_unique_name=ODB_FS'

*.log_archive_dest_2='service=ODB_STBY ASYNC valid_for=(standby_logfiles,standby_role)



*.local_listener='LISTENER_ODB_FS'

*.open_cursors=300


*.processes=300


*.sga_target=4740m


28


*.fal_server='ODB'

*.dg_broker_start=true;

*.db_file_name_convert='/home/oracle/app/oracle/oradata/ODB/','/home/oracle/app/oracle/orada

ta/ODB_FS/'

*.log_file_name_convert='/home/oracle/app/oracle/oradata/ODB/','/home/oracle/app/oracle/orad

ata/ODB_FS/'

#-----------------------------------------------------------------------------------------------------

Create tnsnames.ora file :


#-------------------------------------------------------------------------------------------------------------------

LISTENER_ODB =

(DESCRIPTION =

(ADDRESS_LIST =


1521))

)

(CONNECT_DATA =

(SID = ODB)

)

)

ODB =

(DESCRIPTION =

(ADDRESS_LIST =


1521))

)

(CONNECT_DATA =

29

(SID = ODB)

)

)

LISTENER_ODB_FS =

(DESCRIPTION =

(ADDRESS_LIST =


1521))

)

(CONNECT_DATA =

(SID = ODB)

)

)

LISTENER_ODB_STBY =

(DESCRIPTION =

(ADDRESS_LIST =


1521))

)

(CONNECT_DATA =

(SID = ODB)

)

)

ODB_STBY =

30

(DESCRIPTION =

(ADDRESS_LIST =


1521))

)

(CONNECT_DATA =

(SID = ODB)

)

)

ODB_FS =

(DESCRIPTION =

(ADDRESS_LIST =


1521))

)

(CONNECT_DATA =

(SID = ODB)

)

)

#----------------------------------------------------------------------------------------------------------

Create listener file :


#-----------------------------------------------------------------------------------------------------------

ODB_FS =

(DESCRIPTION_LIST =

(DESCRIPTION =


1521))


31

)

)

SID_LIST_LISTENER =

(SID_LIST =

(SID_DESC =

(GLOBAL_DBNAME = ODB_FS)


(SID_NAME = ODB)

)

)


#-------------------------------------------------------------------------------------------------------

Restart the listener :

# lsnrctl stop

# lsnrctl start

Start the database in mount mode

# sqlplus sys/Wipro_123@ODB_FS as sysdba

SQL> startup mount;

SQL> select GROUP#,MEMBER,TYPE from v$logfile;

SQL> select group#,bytes/1024/1024,members from v$log;

Check if the below groups exists and add if not.

SQL>


'/home/oracle/app/oracle/oradata/ODB_FS/onlinelog/stdbyrdo41.log',

'/home/oracle/app/oracle/fast_recovery_area/ODB_FS/onlinelog/stdbyrdo42.log'

32


SQL>





SQL>





SQL>





ReStart the database instance after creating the groups:

# sqlplus sys/Wipro_123@ODB_FS as sysdba


SQL> create spfile from pfile='$ORACLE_HOME/dbs/initODB.ora';

SQL> startup mount



------------ ---------------- ---------------- --------------------

MOUNTED ODB FAR SYNC MOUNTED

33

5 Configuring DataGuard with DGMGRL

Configuring BROKER and DATA GUARD in Primary :

Enable the broker on all 3 instances

SQL> alter system set dg_broker_start=true;

On Primary Instance

# dgmgrl sys/Wipro_123@ODB

Create the configuration with name ODB_CONF_NEW with primary database ODB

DGMGRL> CREATE CONFIGURATION ODB_CONF_NEW AS PRIMARY

DATABASE IS ODB CONNECT IDENTIFIER IS ODB;

( If any error raised with log_archive_dest_n parameter, note down the parameter value

somewhere else and remove the parameter having ‘service’ field in it on all 3 instances.

SQL> show parameter log_archive_dest_2

SQL> alter system set log_archive_dest_2='' scope=both;

SQL> alter system set log_archive_dest_3='' scope=both;

Add the Physical Standby database ODB_STBY as ODB_STBY to the configuration

DGMGRL> ADD DATABASE ODB_STBY AS CONNECT IDENTIFIER IS

ODB_STBY MAINTAINED AS PHYSICAL;

Add the FarSync instance ODB_FS to the configuration as ODB_FS

DGMGRL> ADD FAR_SYNC ODB_FS AS CONNECT IDENTIFIER IS ODB_FS;

Find the status of the configuration :

DGMGRL>

(Set log_archive_dest_n values before enabling the configuration)

DGMGRL> enable configuration

Enabled.

DGMGRL> show configuration

Configuration - odb_conf_new

Protection Mode: MaxPerformance

34

Members:

odb - Primary database

odb_stby - Physical standby database

odb_fs - Far sync instance

Fast-Start Failover: DISABLED

Configuration Status:

SUCCESS (status updated 42 seconds ago)

DGMGRL>

35

5.1 Enabling Fast-Start Failover

Fast-start failover allows the broker to automatically fail over to a previously chosen standby

database in the event of loss of the primary database. Fast-start failover quickly and reliably fails

over the target standby database to the primary database role, without requiring you to perform any

manual steps to invoke the failover. Fast-start failover can be used only in a broker configuration and

can be configured only through DGMGRL or Enterprise Manager.

Before Fast-Start Failover: Data Guard is operating in a steady state, with the primary

database transmitting redo data to the target standby database and the observer monitoring

the state of the entire configuration.

FastStart Failover Ensues: Disaster strikes the primary database and its network

connections to both the observer and the target standby database are lost. Upon detecting the

break in communication, the observer attempts to reestablish a connection with the primary

database for the amount of time defined by the FastStartFailoverThreshold property before

initiating a fast-start failover. If the observer is unable to regain a connection to the primary

database within the specified time, and the target standby database is ready for fast-start

failover, then fast-start failover ensues.

After Fast-Start Failover: The fast-start failover has completed and the target standby

database is running in the primary database role. After the former primary database has been

repaired, the observer reestablishes its connection to that database and reinstates it as a new

standby database. The new primary database starts transmitting redo data to the new standby

database.

Before Fast-Start Failover enabled :

DGMGRL> show configuration verbose



Members:

odb_stby - Primary database

odb - Physical standby database


36

Properties:

FastStartFailoverThreshold = '30'

OperationTimeout = '30'

TraceLevel = 'USER'

FastStartFailoverLagLimit = '30'

CommunicationTimeout = '180'

ObserverReconnect = '0'

FastStartFailoverAutoReinstate = 'TRUE'

FastStartFailoverPmyShutdown = 'TRUE'

BystandersFollowRoleChange = 'ALL'

ObserverOverride = 'FALSE'

ExternalDestination1 = ''


PrimaryLostWriteAction = 'CONTINUE'



SUCCESS

37

5.2 Enabling Fast-Start Failover and Starting the Observer We can enable fast-start failover from any site, including the observer site, while connected to any

database in the broker configuration. Enabling fast-start failover does not trigger a failover. Instead, it

allows the observer to begin observing the primary and standby databases and initiate a fast-start

failover should conditions warrant a failover.

Step 1 Ensure standby redo logs are configured on the primary and target standby databases.

Standby redo logs must be configured on the primary and standby databases. You must stop log apply services prior to configuring standby redo logs.

Step 2 Ensure the LogXptMode Property is set to SYNC.

The LogXptMode configurable database property must be set to SYNC on the primary and target

standby databases.

To set the redo transport service that corresponds to the protection mode you plan to set, use

the EDIT DATABASE (property) command on the primary and target standby databases. For

example, if the protection mode to be set is MAXAVAILABILITY, you must set

the LogXptMode property to SYNC on the primary database and on the target standby database, as

shown in the following examples:

DGMGRL> EDIT DATABASE ODB SET PROPERTY 'LogXptMode'='SYNC';

Property "LogXptMode" updated

DGMGRL> EDIT DATABASE ODB_STBY SET PROPERTY 'LogXptMode'='SYNC';

Property "LogXptMode" updated

The broker does not allow these commands to succeed unless the databases are configured with standby redo log files.

Step 3 Set the FastStartFailoverTarget configuration property.

If you have two or more standby databases, set up the FastStartFailoverTarget configuration property on the primary database to indicate the desired target standby database. For example:

DGMGRL> EDIT DATABASE ODB SET PROPERTY FastStartFailoverTarget='ODB_STBY';

Property "FastStartFailoverTarget" updated

DGMGRL> EDIT DATABASE ODB_STBY SET PROPERTY FastStartFailoverTarget='ODB';

38

Property "FastStartFailoverTarget" updated

Step 4 Upgrade the protection mode to MAXAVAILABILITY, if necessary.

If it is necessary to upgrade the protection mode, use the following DGMGRL EDIT

CONFIGURATION command. For example:

DGMGRL> EDIT CONFIGURATION SET PROTECTION MODE AS MAXAVAILABILITY;

Step 5 Enable Flashback Database on the primary and target standby databases, if necessary.

If it is not already enabled on both primary and standby databases, enable Flashback Database by issuing the following statements on each database:

SQL> ALTER SYSTEM SET UNDO_RETENTION=3600 SCOPE=SPFILE;

SQL> ALTER SYSTEM SET UNDO_MANAGEMENT='AUTO' SCOPE=SPFILE;

SQL> SHUTDOWN IMMEDIATE;

SQL> STARTUP MOUNT;

SQL> SHOW PARAMETER UNDO;

SQL> ALTER SYSTEM SET DB_FLASHBACK_RETENTION_TARGET=4320 SCOPE=BOTH;

SQL> ALTER DATABASE ARCHIVELOG;

SQL> ALTER DATABASE FLASHBACK ON;

SQL> ALTER DATABASE OPEN;

On Standby only, start the redo apply processes:

SQL> ALTER DATABASE RECOVER MANAGED STANDBY DATABASE DISCONNECT;

Ensure the UNDO_RETENTION and DB_FLASHBACK_RETENTION_TARGET initialization parameters are set to sufficiently large values so that reinstatement is still possible after a prolonged outage.

Step 6 Start the observer.

Start the observer by logging into the observer computer and running DGMGRL. Connect to the

configuration as SYS and then issue the START OBSERVER command. Note that the command does not return; that is you will not get DGMGRL prompt after issuing the command.

39

DGMGRL> CONNECT sys@ODB;

Password:

Connected.

DGMGRL> START OBSERVER;

Observer started

Step 7 Enable fast start failover.

You can enable fast-start failover while connected to any database system in the broker configuration. For example:

DGMGRL> ENABLE FAST_START FAILOVER;

Enabled.

Step 8 Verify the fast-start failover configuration.

Use the SHOW FAST_START FAILOVER command to display the fast-start failover settings:

DGMGRL> SHOW FAST_START FAILOVER;

Fast-Start Failover: ENABLED

Threshold: 30 seconds

Target: odb

Observer: farsync.fragglerock.wipro.com

Lag Limit: 30 seconds (not in use)

Shutdown Primary: TRUE

Auto-reinstate: TRUE

Observer Reconnect: (none)

Observer Override: FALSE

Configurable Failover Conditions

Health Conditions:

Corrupted Controlfile YES

Corrupted Dictionary YES

40

Inaccessible Logfile NO

Stuck Archiver NO

Datafile Offline YES

Oracle Error Conditions:

(none)

Configuration :



Protection Mode: MaxAvailability

Members:


odb - (*) Physical standby database


(*) Fast-Start Failover target

Properties:



TraceLevel = 'USER'








41






Target: odb








SUCCESS

There is no configuration changes needed on ITNM level for failover process

42

5.3 Performing a Manual Failover Operation If fast-start failover is enabled, you can perform a manual failover only to the standby database that was specified as the target of a fast-start failover and only when the observer is running.

Step 1 Connect to the target standby database.

To perform the failover operation, you must connect to the standby database to which you want to

fail over to as a user that has the SYSDBA privilege. For example:

DGMGRL> CONNECT sys@ODB_STBY;

Password:

Connected.

Step 2 Issue the failover command.

Now you can issue the failover command to make the target standby database the new primary

database for the configuration. Note that after the failover completes, the original primary

database cannot be used as a standby database of the new primary database unless it is reenabled.

DGMGRL> FAILOVER TO ODB_STBY;

43

6 Configuring ObjectServer Failover

Assuming that the Omnibus (v7.3 or more) is already installed on the server.

The way in which you configure ObjectServer failover is dependent on the Tivoli Netcool/OMNIbus

version.

In a Tivoli Netcool/OMNIbus installation, each computer on which the Tivoli Netcool/OMNIbus

components run must be configured with server communication information that enables the

components in the architecture to run and communicate with one another. Configure the connections

data file with all the component details, as follows:

Linux:Update the communication information for all the Tivoli Netcool/OMNIbus server components in

your deployment by manually editing the connections data file $NCHOME/etc/omni.dat, which is used

to create the interfaces file.

A suggested good practice is to add all the components in the entire deployment to a single omni.dat

file, which can then be distributed to the $NCHOME/etc directory in all the computers in the

deployment. You can then generate the interfaces file from each computer by running the

$NCHOME/bin/nco_igen command. (Interfaces files are named $NCHOME/etc/interfaces.arch, where

arch is the operating system name.)

6.1 Configuring ObjectServers and gateways for failover

1. create the primary aggregation ObjectServer AGG_P on the primary server and apply the SQL

customization by running the nco_dbinit command with the supplied aggregation.sql import file:

$NCHOME/omnibus/bin/nco_dbinit -server AGG_P -customconfigfile

$NCHOME/omnibus/extensions/multitier/objectserver/aggregation.sql

If the ObjectServer is already installed and running, apply the aggregation.sql import file against

the ObjectServer, as follows:

$NCHOME/omnibus/bin/nco_sql -server AGG_P -user user_name –password password <

$NCHOME/omnibus/extensions/multitier/objectserver/aggregation.sql

2. Start the primary ObjectServer (if necessary):

$NCHOME/omnibus/bin/nco_objserv -name AGG_P &

If you installed Tivoli Netcool/OMNIbus using the Network Manager installer, you can, as an

alternative, run the itnm_start command in the $NCHOME/precision/bin directory:

itnm_start nco

44

3. Create (or update) the backup aggregation ObjectServer AGG_B on standby server, and apply

the SQL customization, as described in step 1. When you apply the SQL customization, the

BackupObjectServer property is automatically set to TRUE and the automations required by the

backup ObjectServer are enabled.

4. Start the backup ObjectServer (if necessary), as described in step 2.

5. On the computer where the backup ObjectServer is installed, configure the bidirectional

aggregation ObjectServer Gateway AGG_GATE:

a. Copy the multitiered property files for the gateway from the

$NCHOME/omnibus/extensions/multitier/gateway location, to the default location

($NCHOME/omnibus/etc) where configuration and properties files are held:

i. AGG_GATE.map

ii. AGG_GATE.props

iii. AGG_GATE.tblrep.def

b. Start the gateway AGG_GATE:

i. $NCHOME/omnibus/bin/nco_g_objserv_bi -propsfile

$NCHOME/omnibus/etc/AGG_GATE.props &

6.2 Sample configuration for the basic failover architecture (aggregation layer only)

The following sample configuration shows the server communications details for the basic failover

architecture in the $NCHOME/etc/omni.dat file, where:

AGG_P is the name of the primary ObjectServer.

AGG_B is the name of the backup ObjectServer.

AGG_V is the name of the virtual ObjectServer pair.

AGG_GATE is the name of the birdirectional ObjectServer Gateway.

NCO_PA represents the default name for the process agent. (If you have configured process

agents to manage the Tivoli Netcool/OMNIbus processes and run external procedures, each

uniquely-named process agent must be added with the appropriate host name and port

number.)

NCO_PROXY represents the default name for the proxy server. (If you have configured one or

more proxy servers to reduce the number of direct probe connections to the ObjectServers,

each uniquely-named proxy server must be added with the appropriate host name and port

number.)

45

$NCHOME/etc/omni.dat

[AGG_P] { Primary: primno1.fragglerock.wipro.com 4100 } [AGG_B] { Primary: stbyno1.fragglerock.wipro.com 4150 } [AGG_V] { Primary: primno1.fragglerock.wipro.com 4100 Backup: stbyno1.fragglerock.wipro.com 4150 } [AGG_GATE] { Primary: stbyno1.fragglerock.wipro.com 4105 } [NCO_PA] { Primary: primno1.fragglerock.wipro.com 4200 } [NCO_PROXY] { Primary: primno1.fragglerock.wipro.com 4400 }

46

6.3 Connecting to an ObjectServer failover pair Each Network Manager installation that connects to an ObjectServer needs a copy of the Tivoli

Netcool/OMNIbus interfaces file.

Assuming that server communication information has been configured in your Tivoli Netcool/OMNIbus

installations, the $NCHOME/etc/interfaces.arch file (where arch represents the operating system name)

should be available in the NCHOME installation location.

When Tivoli Netcool/OMNIbus is installed on the same server as Network Manager, they must

both be installed into the same NCHOME location. In such cases, no further action is required to

ensure that Network Manager processes can connect to an ObjectServer failover pair.

If Network Manager and Tivoli Netcool/OMNIbus are installed on different servers, perform the

following steps on the primary and backup Network Manager servers:

Copy the $NCHOME/etc/interfaces.arch file from the Tivoli Netcool/OMNIbus NCHOME location to the

NCHOME installation location on the server where Network Manager is installed.

6.4 Configuring ObjectServer authentication

If you are using an ObjectServer as the central user registry for user management and authentication,

and you want the ObjectServer to be in a federated repository, you must use the script provided with

Dashboard Application Services Hub to configure the Virtual Member Manager adapter for the

ObjectServer. Configure the adapter for both of the ObjectServers in the failover pair.

On each Dashboard Application Services Hub server where the Network Manager Web applications and

the Web GUI are installed:

1. Go to the $WAS_HOME/bin directory.

2. Enter the following command at the command line:

./confvmm4ncos jazzsm_profile_dir user password address port address2 port2

Where:

o jazzsm_profile_dir is the Jazz for Service Management profile directory.

o user is the ID of a user with administrative privileges for the ObjectServers.

o password is the password for the user ID.

o address is the IP address of the primary ObjectServer.

o port is the port number used by the primary ObjectServer.

o address2 is the IP address of the backup ObjectServer.

47

o port2 is the port number used by the backup ObjectServer.

3. Restart the Dashboard Application Services Hub server.

Ref :

https://www.ibm.com/support/knowledgecenter/SSSHRK_4.2.0/install/task/ins_confobjectserverfai

lover.html?view=kc

https://www.ibm.com/support/knowledgecenter/SSSHRK_4.2.0/install/task/ins_connobjserverfailo

verpair.html?view=kc

https://www.ibm.com/support/knowledgecenter/SSSHRK_4.2.0/install/task/ins_confobjservauthen

tication.html?view=kc

https://www.ibm.com/support/knowledgecenter/SSSHRK_4.2.0/install/task/ins_confobjectserverfailover.html?view=kc

https://www.ibm.com/support/knowledgecenter/SSSHRK_4.2.0/install/task/ins_confobjectserverfailover.html?view=kc

https://www.ibm.com/support/knowledgecenter/SSSHRK_4.2.0/install/task/ins_connobjserverfailoverpair.html?view=kc

https://www.ibm.com/support/knowledgecenter/SSSHRK_4.2.0/install/task/ins_connobjserverfailoverpair.html?view=kc

48

7 Omnibus WebGUI/DASH HA Configuration

Note : All the servers that will be part of the cluster MUST have the exact same versions and

components installed

The two instances of DASH/OmnibusWebGUI should have the exact version of components installed.

7.1 Configuring data source failover for the Tivoli Netcool/OMNIbus Web GUI

If you have a failover pair of ObjectServers to which the Web GUI should connect, you can configure

data source failover by using the ncwDataSourceDefinitions.xml data source configuration file in your

Web GUI installation.

This file is located in webgui_home_dir/etc/datasources, where webgui_home_dir is the installation

directory for Web GUI; for example, $NCHOME/omnibus_webgui.

To configure data source failover:

On the Dashboard Application Services Hub server where the Web GUI is installed, edit the data

source configuration file as follows:

o Use the name attribute of the <ncwDataSourceEntry> element to specify a label for the

failover pair of ObjectServers; for example, VirtualObjectServerPair.

o Define the connection details for the primary and backup ObjectServers by using the

<ncwDataSourceDefinition> element and its child elements.

Note : The name attribute values of both the <ncwDataSourceEntry> and <ncwDataSourceDefinition>

elements must be identical. You must also define the ObjectServer connections by using the

ObjectServer host names and port numbers, rather than the ObjectServer names that are configured in

the omni.dat or sql.ini file.

o Restart the Dashboard Application Services Hub server for the changes to take effect.

Use the following command:

startServer.sh server1

You must also set WebTopDataSource value in ModelNcimDb.<DOMAIN>.cfg file to the same

value as the <ncwDataSourceEntry> is set to in the ncwDataSourceDefinitions.xml file. Using the

settings in the Sample ncwDataSourceDefinitions.xml configuration for data source failover, the

following example shows what changes you need to make:

o Go to $NCHOME/etc/precision/ModelNcimDb.<DOMAIN>.cfg file and open it for editing.

o Find the insert that defines the WebTopDataSource:

49

insert into dbModel.access ( EnumGroupFilter, TransactionLength, ValidateCacheFile, WebTopDataSource ) values ( "enumGroup in ('ifAdminStatus', 'ifOperStatus', 'sysServices', 'ifType', 'cefcFRUPowerAdminStatus', 'cefcFRUPowerOperStatus', 'TruthValue', 'entSensorType', 'entSensorScale', 'entSensorStatus', 'cefcModuleAdminStatus', 'cefcModuleOperStatus', 'ipForwarding', 'cefcPowerRedundancyMode', 'EntityType', 'ospfIfState', 'ospfIfType', 'dot3StatsDuplexStatus', 'accessProtocol')", 500, 0, "OS" );

o Change the WebTopDataSource value in the following insert query to match the data

source configured in the <ncwDataSourceEntry> (in this case, change the value OS to

VirtualObjectServerPair):

insert into dbModel.access ( EnumGroupFilter, TransactionLength, ValidateCacheFile, WebTopDataSource ) values ( "enumGroup in ('ifAdminStatus', 'ifOperStatus', 'sysServices', 'ifType', 'cefcFRUPowerAdminStatus', 'cefcFRUPowerOperStatus', 'TruthValue', 'entSensorType', 'entSensorScale', 'entSensorStatus', 'cefcModuleAdminStatus', 'cefcModuleOperStatus', 'ipForwarding', 'cefcPowerRedundancyMode', 'EntityType', 'ospfIfState', 'ospfIfType', 'dot3StatsDuplexStatus', 'accessProtocol')", 500, 0, "AGG_V" );

Note : The Web GUI data source name is the name for the connection, and it has to be the same as what

is set in the Web GUI. The name might not always be the same as the ObjectServer name

50

o Make this change on both the primary and backup core Network Manager servers.

o Restart ncp_ctrl

7.1.1 Sample ncwDataSourceDefinitions.xml configuration for data source failover In the following sample code, the bold text identifies the values that are applicable to data source

failover

<ncwDefaultDataSourceList>

<ncwDataSourceEntry name="AGG_V"/>

</ncwDefaultDataSourceList>

...

<ncwDataSourceDefinition type="singleServerOSDataSource" name="AGG_V"

enabled="true">

<ncwFailOverPairDefinition>



<ncwPrimaryServer>

<ncwOSConnection host="primno1.fragglerock.wipro.com" port="4100"

ssl="false" minPoolSize="5" maxPoolSize="10"/>

</ncwPrimaryServer>



<ncwBackUpServer>

<ncwOSConnection host="stbyno1.fragglerock.wipro.com" port="4150"

ssl="false" minPoolSize="5" maxPoolSize="10"/>

</ncwBackUpServer>

</ncwFailOverPairDefinition>

</ncwDataSourceDefinition>

51

7.2 Configuration needed on the DB2 server

Login to DB2 with the DB2 instance owner user, in this example the default db2inst1 user has been

used.

Start DB2 database by running the following command: db2start

Create an empty database, you can name it for example WASHA

db2 create database WASHA

$ db2 create database WASHA

$ db2 connect to WASHA

Database Connection Information

Database server = DB2/LINUXX8664 10.5.3

SQL authorization ID = DB2INST1

Local database alias = WASHA

7.3 Configuration needed on each WebGUI server On the first WebGUI server:

1. Login to the WebGUI server and open WebSphere Administrative Console

52

2. From WAS go to Resources -> JDBC -> JDBC providers

3. Select instead of “All scopes” the option Node=JazzSMNode01, Server=server1:

53

4. Create a new JDBC provider by clicking on the New option:

For provider type select DB2 universal JDBC driver provider:

S elect DB2 for database type:

54

For implementation type select connection pool data source:

C lick next .

55

On the server search for db2jcc.jar file paths. There should be one under JazzSM directory which

is required for native library path and one under WebSphere directory which is required for the

first field.

Enter the following path to the directory location for the mentioned jar files:

/home/oracle/IBM/WebSphere/AppServer/deploytool/itp/plugins/com.ibm.datatools.db2_2.1.110.v20

121008_1514/driver/db2jcc.jar

A nd the following path for t he native directory:

56

Native library path

Directory location which is saved as WebSphere variable

${DB2UNIVERSAL_JDBC_DRIVER_NATIVEPATH}

/home/oracle/IBM/JazzSM/lib/db2

Click next.

Click Save to save the configuration (you will need to do this each time you get this screen):

5. Create a new Data Source for JDBC.

Click finish .

57

Select instead of “All scopes” the option Node=JazzSMNode01, Server=server1:

Click on "New"

Enter tipds (this should be always named as this) and jdbc/tipds for JNDI name (this should be

always named as this):

Click on Next.

G o to " R esources" - > JDB C - > D ata S ources

58

Select the option "Select an existing JDBC provider" and select the “DB2 universal JDBC driver

provider”:

Click on Next.

Within this screen you will have to enter the name of the database that you have created e.g.

WASHA and also the server hostname and the port number where DB2 is installed.

59

Within this screen you don't have to select anything, we'll complete this later.

Click next.

Click finish.

6. Click on the data source that was created e.g. "tipds":

C lick next .

C lick S ave to store the configuration

60

Select "JAAS - J2C authentication data" under the Related Items section.

Enter a name as alias – in this example the following name was used: DB2_alias

Enter the db2inst1 user (the instance owner user from DB2) and its password.

Save the configuration.

C lick on new:

C lick ok .

61

7. Return to the tipds data source and go to Security Settings section:

Select JazzSMNode01/DB2_alias for component-managed authentication alias:

Select DefaultPrincipalMapping for mapping-configuration alias:

Select JazzSMNode01/DB2_alias for container-manager authentication alias:

Click ok.

Click Save to store the configuration.

62

The output should be the below one:

9. From WAS menu -> Servers -> Server Types -> WebSphere application servers

Under Server Infrastructure menu-> Java and Process Management => Process Definition

8. C heck tipds data s ource connection :

C lick on server1 :

63

Click on Java Virtual Machine under the Additional Properties section:

Click on Custom Properties under the Additional Properties section:

64

Enter com.ibm.isc.ha for the Name property and true for the Value property:

10. On the server, edit the server.init file from the webgui etc directory and set the following 2

properties as per above:

cluster.mode:on

timedtasks.enabled:true

C lick on N ew :

C lick apply and save .

65

Afterwards, you will need to restart webgui.

Then on the webgui server run the following command:

./consolecli.sh ListHANodes --username smadmin --password netcool

You should get your webgui server on the list.

Repeat all the above steps from 1 to 10 on all the other WebGUI servers that you want to

add to this cluster setup.

Afterwards, with both servers configured you will need to enable server to server trust by following

the steps described within the following link:

https://www.ibm.com/support/knowledgecenter/en/SSEKCU_1.1.2.1/com.ibm.psc.doc/tip_original/t

tip_config _loadbal_trust.html

e.g. repeat the below steps from 1 to 5 for each WebGUI server:

1. Edit ssl.client.props properties file

/home/oracle/IBM/JazzSM/profile/properties/ssl.client.props

Uncomment the section that starts with com.ibm.ssl.alias=AnotherSSLSettings so that it looks

like this:

https://www.ibm.com/support/knowledgecenter/en/SSEKCU_1.1.2.1/com.ibm.psc.doc/tip_original/ttip_config_loadbal_trust.html




66

2. Uncomment the section that starts with com.ibm.ssl.trustStoreName=AnotherTrustStore

so that it looks like this:

3. Update the location of the trust store that the signer should be added to in the

com.ibm.ssl.trustStore property of AnotherTrustStore by replacing the default value

com.ibm.ssl.trustStore=${user.root}/etc/trust.p12 with the correct path for your trust

store. Example:

com.ibm.ssl.trustStore=${user.root}/config/cells/JazzSMNode01Cell/nodes/JazzSMNode01/trust.p1

2

67

4. Save file.

5. Restart webgui.

Repeat the same steps from 1 to 5 on the other servers.

Afterwards, run the following command on each node for each myremotehost (that is, for every

node that you want to enable trust with) in the cluster.

JazzSM_WAS_Profile/bin/retrieveSigners.sh NodeDefaultTrustStore AnotherTrustStore -host

myremotehost -port remote_SOAP_port

where: myremotehost is the name of the server to enable trust with; remote_SOAP_port is the

SOAP connector port number (16313 is the default). If you have installed with non-default ports,

check JazzSM_WAS_Profile/properties/portdef.props for the value of

SOAP_CONNECTOR_ADDRESS and use that.

So, on server 1: run the following command – the host added in command line is the one from the

second server:

./retrieveSigners.sh NodeDefaultTrustStore AnotherTrustStore -host loaf1.castle.fyre.ibm.com -

port 16313

Click yes to add the signer to the trust store :

68

On server 2 – run the same but add the host of the webgui server1, example:

./retrieveSigners.sh NodeDefaultTrustStore AnotherTrustStore -host bazarz1.castle.fyre.ibm.com -

port 16313

En ter smadmin credentials and click ok :

Restart all webgui serve rs again .

69

At the end, you will have your HA environment configured.

Check status by running on each webgui server the following command:

./consolecli.sh ListHANodes -username smadmin -password netcool

7.4 Creating users in Omnibus

The users (itnmadmin, itnmuser, etc) should be created manuallyonce the ObjectServer, DASH HA

configuration is finished. Generally, these users are created by ITNM at the time of creation of

ObjectServer in default mode. Since we are providing already existing ObjectServer to ITNM, ITNM will

not alter any components in existing ObjectServer.

Run the following command :

/home/oracle/IBM/netcool/gui/omnibus_webgui/configtool/linux.gtk.x86_64/ncwConfigUI -WASUserID

smadmin -WASPassword netcool

Configure an advance setup

o Host : primno1.fragglerock.wipro.com

o Port : 4100

o Admin User : root

o Password : (Default password - empty string)

70

8 Installation of ITNM on primary and Standby Instances

8.1 Installing ITNM on Primary Instance Assuming the customer already has the following packages

1. IBM Installation Manager for Linux x86_64

2. ITNM Dependencies (WAS v8.5.5.9, Omnibus v8.1.0.5, Omnibus WebGUI 8.1.0.14,

Jazz v1.1.2.1, IBM DASH v3.1.3.0 )

3. ITNM v4.2 FP5

Installing IBM Installation Manager :

Open the folder containing IBM Installation Manager and run the installer

# cd <ibm_install_manager>

#./install (./userinst if the user is non-root)

Give the location where IBMIM to be installed.

Next >

Install>

Installing ITNM Dependencies :

Navigate to location where IBMIM is installed. (default location <userHome>

/IBM/InstallationManager)

Start the IBMIM

# eclipse/IBMIM

Navigate to File-> Preferences…

Add the respective repository in which all the dependencies are present.

Apply>

OK>

Click on the “Install” option, Next>

Select all the dependencies Next>

Give appropriate details Next>

Install Next>

After the installation is completed, select None Finish>

71

Creating and Populating Oracle database on Primary Instance :

Start IBMIM

Navigate to File> Preference

Uncheck the repository for dependencies and add another one containing ITNM packages

Apply>

OK>

Click Install>

Select “Network Manager Topology Database Creation Scripts” package and proceed.

Once the package is installed, the database schema are to be created on Primary instance.

Login as oracle user,

# su - oracle

Navigate to $NCHOME/precision/scripts/sql/oracle (default NCHOME

<userHome>/IBM/netcool/core)

Create users for the database

# ./create_oracle_ncadmin_user.sh sys <password>

Create database schema

# ./create_oracle_database.sh system <password>

Create the tables on the database schema

# ./populate_oracle_database.sh ncim ncim

Installing ITNM on Primary Instance :

Create Group and user for ITNM

# groupadd wiprosrc

# useradd -d /home/itnmadm -g wiprosrc itnm_poller

# passwd itnm_poller

# id itnm_poller

72

Start the IBMIM

Install>

Select the Network manager Core and GUI components

Next>

Give the relevant details that are given at the time of installing dependencies.

At ObjectServer :

ObjectServer Name : AGG_V (Virtual ObjectServer)

Host : primno1.fragglerock.wipro.com

Port : 4100 ( Port on which AGG_P is running)

User : root

Password : < root password>

At the database :

Select Oracle database,

Service Name : ODB

Host : primno1.fragglerock.wipro.com

Port : 1521 (port on which listener is running)

User : ncim

Password : ncim

Next>

Install>

73

8.2 Installing ITNM on Physical Standby:

Before installing the ITNM on Standby, the switchover has to be performed. (Ref section : 11.1)

Similar procedure has to be followed to install ITNM on Physical Standby Instance.

The population of database is not needed as the database schema is already created and

populated when primno1 is as primary instance.

At the time of installing ITNM core and GUI, while providing details, give the following

details…

At ObjectServer :

ObjectServer Name : AGG_V (Virtual ObjectServer)

Host : stbyno1.fragglerock.wipro.com

Port : 4150 ( Port on which AGG_B is running)

User : root

Password : < root password>

At the database :

Select Oracle database,

Service Name : ODB_STBY

Host : stbyno1.fragglerock.wipro.com

Port : 1521 (port on which listener is running)

User : ncim

Password : ncim

Next>

Install>

It is always mandatory to keep ITNM process ideal (not running) at all time when the

server is acting as standby instance.

74

75

9 Enabling Rsync between Primary and Standby servers

Rsync is a utility tool used to sync the files between servers more effectively. It can copy

locally/remotely from/to another host over any remote shell. It provides a numerous options that

control every aspect of its behavior. It reduces the amount of data sent over the network by sending

only the differences between the source files and existing files in the destination.

In Active DataGuard setup, only the NCIM database of primary is highly available for the standby which

is connected through broker configuration. The second most important files which are present in the

primary ITNM instance (Discovery cache files, other config files) resides and available only on primary

ITNM. These are not accessible to standby ITNM. Inorder to facilitate this accessibility and make the

High Availability more meaningful, Rsync utility in association with InotifyWait is used to sync these files

between primary ITNM and standby ITNM.

inotify is an API utility used to moniter a file/directory and notify the user whenever a particular event is

occurred. When a directory/file is monitored, inotify will return the events for the directory itself and for

all the files inside the directory.

Setting Up Rsync :

Install the inotifywait and rsync utilities on both Primary and Standby servers

# yum -y install epel-release

# yum install inotify-tools

# yum install rsync

9.1 Creating rsync files

9.1.1 Create rsync files on primno1 :

Synchronizing $NCHOME/etc/precision

$ vim /home/oracle/bin/rsync_nchome_etc_precision.bash

#!/usr/bin/bash

while inotifywait -r /home/oracle/IBM/tivoli/netcool/etc/precision; do

rsync -azu --delete --exclude={'DbLogins*','MibDbLogin*','ServiceData.cfg'}

/home/oracle/IBM/tivoli/netcool/etc/precision/

oracle@stbyno1:/home/oracle/IBM/tivoli/netcool/etc/precision/

done

76

Synchronizing $NCHOME/precision/aoc

$ vim /home/oracle/bin/rsync_nchome_prec_aoc.bash

#!/usr/bin/bash

while inotifywait -r /home/oracle/IBM/tivoli/netcool/precision/aoc; do

rsync -azu --delete /home/oracle/IBM/tivoli/netcool/precision/aoc/

oracle@stbyno1:/home/oracle/IBM/tivoli/netcool/precision/aoc/

done

Synchronizing $NCHOME/precision/collectors/perlCollectors

$ vim /home/oracle/bin/rsync_nchome_prec_coll_perlcol.bash

#!/usr/bin/bash

while inotifywait -r /home/oracle/IBM/tivoli/netcool/precision/collectors/perlCollectors; do

rsync -azu --delete /home/oracle/IBM/tivoli/netcool/precision/collectors/perlCollectors/

oracle@stbyno1:/home/oracle/IBM/tivoli/netcool/precision/collectors/perlCollectors/

done

Synchronizing $NCHOME/precision/disco/agents

$ vim /home/oracle/bin/rsync_nchome_prec_disco_agent.bash

#!/usr/bin/bash

while inotifywait -r /home/oracle/IBM/tivoli/netcool/precision/disco/agents; do

rsync -azu --delete /home/oracle/IBM/tivoli/netcool/precision/disco/agents/

oracle@stbyno1:/home/oracle/IBM/tivoli/netcool/precision/disco/agents/

done

Synchronizing $NCHOME/precision/eventGateway/stitchers

$ vim /home/oracle/bin/rsync_nchome_prec_evgate_stch.bash

#!/usr/bin/bash

while inotifywait -r /home/oracle/IBM/tivoli/netcool/precision/eventGateway/stitchers; do

77

rsync -azu --delete /home/oracle/IBM/tivoli/netcool/precision/eventGateway/stitchers/

oracle@stbyno1:/home/oracle/IBM/tivoli/netcool/precision/eventGateway/stitchers/

done

Synchronizing $NCHOME/precision/mibs

$ vim /home/oracle/bin/rsync_nchome_prec_mibs.bash

#!/usr/bin/bash

while inotifywait -r /home/oracle/IBM/tivoli/netcool/precision/mibs; do

rsync -azu --delete /home/oracle/IBM/tivoli/netcool/precision/mibs/

oracle@stbyno1:/home/oracle/IBM/tivoli/netcool/precision/mibs/

done

Synchronizing $NCHOME/var/precision

$ vim /home/oracle/bin/rsync_nchome_var_precision.bash

#!/usr/bin/bash

while inotifywait -r /home/oracle/IBM/tivoli/netcool/var/precision; do

rsync -azu --delete --exclude='storm' /home/oracle/IBM/tivoli/netcool/var/precision/

oracle@stbyno1:/home/oracle/IBM/tivoli/netcool/var/precision/

done

Give execution permissions for above bash scripts

# chmod +x /home/oracle/bin/rsync_*.bash

78

9.1.2 Create rsync files on stbyno1 :

Synchronizing $NCHOME/etc/precision

$ vim /home/oracle/bin/rsync_nchome_etc_precision.bash

#!/usr/bin/bash

while inotifywait -r /home/oracle/IBM/tivoli/netcool/etc/precision; do

rsync -azu --delete --exclude={'DbLogins*','MibDbLogin*','ServiceData.cfg'}

/home/oracle/IBM/tivoli/netcool/etc/precision/

oracle@primno1:/home/oracle/IBM/tivoli/netcool/etc/precision/

done

Synchronizing $NCHOME/precision/aoc

$ vim /home/oracle/bin/ rsync_nchome_prec_aoc.bash

#!/usr/bin/bash

while inotifywait -r /home/oracle/IBM/tivoli/netcool/precision/aoc; do

rsync -azu --delete /home/oracle/IBM/tivoli/netcool/precision/aoc/

oracle@primno1:/home/oracle/IBM/tivoli/netcool/precision/aoc/

done

Synchronizing $NCHOME/precision/collectors/perlCollectors

$ vim /home/oracle/bin/rsync_nchome_prec_coll_perlcol.bash

#!/usr/bin/bash

while inotifywait -r /home/oracle/IBM/tivoli/netcool/precision/collectors/perlCollectors; do

rsync -azu --delete /home/oracle/IBM/tivoli/netcool/precision/collectors/perlCollectors/

oracle@primno1:/home/oracle/IBM/tivoli/netcool/precision/collectors/perlCollectors/

done

Synchronizing $NCHOME/precision/disco/agents

$ vim /home/oracle/bin/rsync_nchome_prec_disco_agent.bash

#!/usr/bin/bash

while inotifywait -r /home/oracle/IBM/tivoli/netcool/precision/disco/agents; do

rsync -azu --delete /home/oracle/IBM/tivoli/netcool/precision/disco/agents/

oracle@primno1:/home/oracle/IBM/tivoli/netcool/precision/disco/agents/

done

79

Synchronizing $NCHOME/precision/eventGateway/stitchers

$ vim /home/oracle/bin/rsync_nchome_prec_evgate_stch.bash

#!/usr/bin/bash

while inotifywait -r /home/oracle/IBM/tivoli/netcool/precision/eventGateway/stitchers; do

rsync -azu --delete /home/oracle/IBM/tivoli/netcool/precision/eventGateway/stitchers/

oracle@primno1:/home/oracle/IBM/tivoli/netcool/precision/eventGateway/stitchers/

done

Synchronizing $NCHOME/precision/mibs

$ vim /home/oracle/bin/rsync_nchome_prec_mibs.bash

#!/usr/bin/bash

while inotifywait -r /home/oracle/IBM/tivoli/netcool/precision/mibs; do

rsync -azu --delete /home/oracle/IBM/tivoli/netcool/precision/mibs/

oracle@primno1:/home/oracle/IBM/tivoli/netcool/precision/mibs/

done

Synchronizing $NCHOME/var/precision

$ vim /home/oracle/bin/rsync_nchome_var_precision.bash

#!/usr/bin/bash

while inotifywait -r /home/oracle/IBM/tivoli/netcool/var/precision; do

rsync -azu --delete --exclude='storm' /home/oracle/IBM/tivoli/netcool/var/precision/

oracle@primno1:/home/oracle/IBM/tivoli/netcool/var/precision/

done

Give execution permissions for above bash scripts

# chmod +x /home/oracle/bin/rsync_*.bash

80

9.2 Enabling Passwordless Authentication For the rsync to work, the password is to be provided everytime the files are transferred to remote

destination. For this process has to be done automatically, a passwordless authentication must be

enabled from primno1 to stbyno1 and as well as from stbyno1 to primno1. Follow the below steps.

On primno1,

Log on to oracle user,

# su - oracle

# cd ~/.ssh

Check if id_rsa.pub file is present. If not, create the file

# ssh-keygen -t rsa

Give passphrase as blank(Hit Enter)

Copy the string generated in .ssh/id_rsa.pub in local server

On remote server(stbyno1), go to <user>/.ssh/authorized_keys file (if not present, create the file)

Append the string copied earlier

Similarly, paste the id_rsa.pub file of stbyno1 in the authorized_keys of primno1.

This should enable the rsync to transfer files continuously without any authentication.

Make sure that relevant directories/files have sufficient permissions to them for passwordless ssh to

work.

Example :

drwx------. 29 oracle oinstall 4096 Jun 3 04:43 /home/oracle

drwx------. 2 oracle oinstall 4096 May 29 05:54 /home/oracle/.ssh

-rw-r--r--. 1 oracle oinstall 837 May 29 05:54 /home/oracle/.ssh/authorized_keys

9.3 Running rsyng scripts Initiate the rsync on both primno1 and stbyno1 by running following command

# nohup /home/oracle/bin/rsync_var_precision.bash >> var_prec.log &

The above command appends the logs to var_prec.log file.

Similarly, we can sync dncim database with the following command.

# nohup /home/oracle/bin/rsync_etc_precision.bash >> etc_prec.log &

81

# nohup /home/oracle/bin/rsync_nchome_precision.bash >> nchome_prec.log &

**Caution : As storm runs continuously, the logs are generated in huge amount. It would be good

practice to monitor the size of var_prec.log file from time to time.

For two way syncing, these scripts has to be run on both primno1 and stbyno1.

9.4 Stopping rsync The rsync can be stopped by conventional method of killing rsync Process ID.

# ps - ef | grep rsync_var_precision.bash

# kill -9 <PID>

82

PART-II

Role Change : FailOver, SwitchOver

83

10 Role Change : Failover, Switchover and Switchback

A switchover is a role reversal between the primary database and one of its standby databases. A

switchover guarantees no data loss and is typically done for planned maintenance of the primary

system. During a switchover, the primary database transitions to a standby role, and the standby

database transitions to the primary role

10.1 Fail-over operation with oracle ADG

10.1.1 Reinstating a Failed Primary Database

10.1.1.1 System Level

Incase the entire server that is acting as primary is down (restarted otherwise), follow the below steps to

make it work as primary/standby again

Make sure that network connection is restored between this server and other servers(farsync

,standby)

Make sure that you are able to ping the other servers with their IP, short-hand hostname, full-

fledged hostname

Make sure that oracle listener is running

o Listener status : $ORACLE_HOME/bin/lsnrctl status

o Start listener : $ORACLE_HOME/bin/lsnrctl start

Make sure that you are able to get a response through tnsping

o Syntax :

tnsping <hostname>

tnsping <DB name>

o Example :

tnsping stbyno1.fragglerock.wipro.com

tnsping stbyno1

tnsping odb_stby

tnsping farsync.fragglerock.wipro.com

tnsping farsync

tnsping odb_fs

10.1.1.2 Pre-requisite Level

If the DB2, that is used to configure HA of DASH/WebGUI is on the same server, start the DB2

database

Start ObjectServer

o $NCHOME/omnibus/bin/nco_objserv -name AGG_P &

Start DASH

o <jazz_home>/profile/bin/startServer.sh server1

Check the DASH HA status

84

o <jazz_home>/ui/bin/consolecli.sh ListHANodes --username smadmin --password netcool

10.1.1.3 Database Level

If your primary database had been configured with Flashback Database, you can easily reinstate

the failed primary database as a standby database of the new primary database. The failed

primary database will be reinstated as a standby type that matches the old standby database. For

example, if you failed over to a physical standby database, the old primary will be reinstated as a

physical standby database.

To reinstate the failed primary database, start it to the mounted state. Then run DGMGRL,

connect to the new primary database and reinstate the old primary database.

Step 1 Restart the Old Primary Database

[oracle@primno1 ~]$ sqlplus sys/Wipro_123 as sysdba

SQL*Plus: Release 12.1.0.2.0 Production on Mon Jan 28 04:16:57 2019

Copyright (c) 1982, 2014, Oracle. All rights reserved.

Connected to an idle instance.

SQL> startup mount

ORACLE instance started.

Total System Global Area 1.0100E+10 bytes

Fixed Size 3721224 bytes

85

Variable Size 1879050232 bytes

Database Buffers 8187281408 bytes

Redo Buffers 29831168 bytes

Database mounted.

SQL>

Step 2 Reinstate the old primary database

Then run DGMGRL, connect to the new primary database and reinstate the old primary database.

% dgmgrl connect sys

Password:

Connected.

DGMGRL> REINSTATE DATABASE 'odb';

Reinstating database "odb", please wait...

Operation requires shutdown of instance "sales1" on database "odb"

Shutting down instance " odb "...

ORA-01109: database not open

Database dismounted.

ORACLE instance shut down.

Operation requires startup of instance " odb" on database "odb"

Starting instance "odb "...


Database mounted.

Continuing to reinstate database "odb " ...

Reinstatement of database "odb " succeeded

After the primary has been reinstated, issue the SHOW CONFIGURATION and SHOW

DATABASE commands to confirm that the old primary has been successfully reinstated

Step 3 Show the Configuration and Databases


86



Members:


odb_stby - (*) Physical standby database





DGMGRL> show database odb

Database - odb

Role: PRIMARY

Intended State: TRANSPORT-ON

Instance(s):

ODB

Database Status:

SUCCESS

DGMGRL> show database odb_stby

Database - odb_stby

Role: PHYSICAL STANDBY

Intended State: APPLY-ON

Transport Lag: 0 seconds (computed 0 seconds ago)

Apply Lag: 0 seconds (computed 1 second ago)

87

Average Apply Rate: 9.00 KByte/s

Real Time Query: OFF

Instance(s):

ODB

Database Status:

SUCCESS

88

10.2 Switchover (Primary to Standby)

10.2.1 ITNM Level

Primary Instance :

Stop all the ITNM processes using ‘itnm_stop’ command

Make sure that rsync scripts are running

10.2.2 NCIM Level

Check status on both Primary and STANDBY

SQL> select status,instance_name,database_role,open_mode from v$database,v$Instance;

Primary:


------------ ---------------- ---------------- --------------------


Standby:


------------ ---------------- ---------------- --------------------


Find the switch over status on primary and standby :

SQL> select switchover_status from v$database;

Primary:

SWITCHOVER_STATUS

--------------------

TO STANDBY

Standby :

SWITCHOVER_STATUS

--------------------

NOT ALLOWED

89

DGMGRL> validate database odb

Database Role: Primary database

Ready for Switchover: Yes

DGMGRL> validate database odb_stby

Database Role: Physical standby database

Primary Database: odb


Ready for Failover: Yes (Primary Running)

Temporary Tablespace File Information:

odb TEMP Files: 2

odb_stby TEMP Files: 2

DGMGRL> validate far_sync odb_fs

Member Role: Far Sync Instance

When Primary Is: odb

Active Redo Source: ODB

Redo Destinations:

(none)

Thread # Online Redo Log Groups Standby Redo Log Groups Status

odb odb_fs

1 3 4 Sufficient SRLs

DGMGRL>

Check log_archive_dest_2 parameter on 3 instances:

SQL> show parameter log_archive_dest_2

Add log_archive_dest_2 on Primary ( primno1 10.162.64.172) (if not already present):

90

SQL> alter system set log_archive_dest_2='service=ODB_FS SYNC AFFIRM


db_unique_name=ODB_FS';

Add log_archive_dest_2 on Standby ( stbyno1 10.45.41.202) (if not already present):

SQL> alter system set log_archive_dest_2='service=ODB ASYNC

valid_for=(online_logfiles,primary_role) db_unique_name=ODB';

Add log_archive_dest_2 on FarSync (farsync 10.16264.137) (if now already present);

SQL> alter system set log_archive_dest_2='service=ODB_STBY ASYNC

valid_for=(standby_logfiles,standby_role) db_unique_name=ODB_STBY';

Verify the status on primary:

SQL> alter database switchover to ODB_STBY verify;

Database altered.

Switching over to Standby,

On Primary

SQL> alter database switchover to ODB_STBY;

Database altered.

After switching over;

On new primary stbyno1 (old standby stbyno1)



------------ ---------------- ---------------- --------------------

MOUNTED ODB PRIMARY MOUNTED

91

SQL> alter database open;

Database altered.



------------ ---------------- ---------------- --------------------


new Standby (primno1)

The database will be in shutdown mode. Start it in nomount mode,

SQL> startup nomount







SQL> alter database mount

Database altered.



------------ ---------------- ---------------- --------------------


92

Start Applying redo data to the new standby

SQL> alter database recover managed standby database disconnect;

Database altered.

Connect to broker configuration through new primary :

# dgmgrl sys/Wipro_123@ODB_STBY

Disable Fast_Start failover if already enabled

DGMGRL> disable fast_start failover

Disabled.

DGMGRL> disable configuration

Disabled


Enabled.




Members:


odb - Physical standby database





DGMGRL>

10.2.3 Post Switchover Steps

93

Check the ITNM processes on new primary instance :

On ITNM level, once the switch over is completed, it is best practice to check the ITNM

processes.

Start the Omnibus if not already running

# /home/oracle/IBM/tivoli/netcool/omnibus/bin/nco_objserv -name AGG_B &

( AGG_B is the objectserver name that is configured at the time of installation of ITNM)

Set the environment and Check database access

# cd /home/oracle/IBM/tivoli/netcool

# . env.sh

# /home/oracle/IBM/tivoli/netcool/precision/scripts/perl/scripts/ncp_db_access.pl -domain

NCOMS

(NCOMS is the default domain whose database is configured through dataguard)

If all the database connections are ‘OKAY’, then start the ITNM processes

# itnm_start

On new Standby, stop the rsync processes and start them from new primary to new standby

94

10.3 Switchback (Standby to Primary)

10.3.1 ITNM Level

Primary Instance :

Stop all the ITNM processes using ‘itnm_stop’ command

Make sure that rsync scripts are running

10.3.2 NCIM Level


Primary:


------------ ---------------- ---------------- --------------------


Standby:


------------ ---------------- ---------------- --------------------


On Primary


SWITCHOVER_STATUS

--------------------

TO STANDBY

On Standby


95

SWITCHOVER_STATUS

--------------------

NOT ALLOWED

Add log_archive_dest_2 on Standby ( primno1 10.162.64.172) (if not already present):

SQL> alter system set log_archive_dest_2='service=ODB_FS SYNC AFFIRM


db_unique_name=ODB_FS';

Add log_archive_dest_2 on Primary ( stbyno1 10.45.41.202) (if not already present):

SQL> alter system set log_archive_dest_2='service=ODB ASYNC

valid_for=(online_logfiles,primary_role) db_unique_name=ODB';

Add log_archive_dest_2 on FarSync (farsync 10.162.64.137) (if now already present);

SQL> alter system set log_archive_dest_2='service=ODB_STBY ASYNC

valid_for=(standby_logfiles,standby_role) db_unique_name=ODB_STBY';

Verify the database switchover status on DG Broker configuration

DGMGRL> validate database odb_stby

Database Role: Primary database


Flashback Database Status:

odb_stby: Off

DGMGRL> validate database odb

Database Role: Physical standby database

Primary Database: odb_stby


Ready for Failover: Yes (Primary Running)

Flashback Database Status:

96

odb_stby: Off

odb: Off

DGMGRL> validate far_sync odb_fs

Member Role: Far Sync Instance

When Primary Is: odb_stby

Active Redo Source: ODB_STBY

Redo Destinations:

(none)

Thread # Online Redo Log Groups Standby Redo Log Groups Status

odb_stby odb_fs

1 3 4 Sufficient SRLs

SwitchOver verification the status on primary:

SQL> alter database switchover to ODB verify;

Database altered.

Switching over to Standby,

On Primary

SQL> alter database switchover to ODB;

Database altered.

After switching over;

On new primary(old standby)

97



------------ ---------------- ---------------- --------------------

MOUNTED ODB PRIMARY MOUNTED

SQL> alter database open;



------------ ---------------- ---------------- --------------------


On new Standby (stbyo1)

The database will be in shutdown mode.

SQL> startup nomount







SQL> alter database mount

Database altered.



98

------------ ---------------- ---------------- --------------------


Start MRP processes on new standby

SQL> alter database recover managed standby database disconnect;

Database altered.

Re-enable the broker configuration from primary(primno1)

# dgmgrl sys/Wipro_123@ODB

Disable Fast_Start failover if already enabled

DGMGRL> disable fast_start failover

Disabled.

DGMGRL> disable configuration

Disabled.


Enabled.




Members:


odb_stby - Physical standby database





DGMGRL>

99

10.3.3 Post Switchover Steps

Check the ITNM processes on new primary instance :

Start the Omnibus if not already running

# /home/oracle/IBM/tivoli/netcool/omnibus/bin/nco_objserv -name AGG_P &

( AGG_P is the objectserver name that is configured at the time of installation of ITNM)

Set the environment and Check database access

# cd /home/oracle/IBM/tivoli/netcool

# . env.sh

# /home/oracle/IBM/tivoli/netcool/precision/scripts/perl/scripts/ncp_db_access.pl -domain

NCOMS

(NCOMS is the default domain whose database is configured through dataguard)

If all the database connections are ‘OKAY’, then start the ITNM processes

# itnm_start

On new Standby, stop the rsync processes and start them from new primary to new standby

100

11 Testing Oracle DataGuard Setup

11.1 Monitoring FRA space

It is important to monitor space usage in the fast recovery area to ensure that it is large enough to

contain backups and other recovery-related files.

Query 1:

SQL >SELECT ROUND((A.SPACE_LIMIT / 1024 / 1024 / 1024), 2) AS FLASH_IN_GB,

ROUND((A.SPACE_USED / 1024 / 1024 / 1024), 2) AS FLASH_USED_IN_GB,

ROUND((A.SPACE_RECLAIMABLE / 1024 / 1024 / 1024), 2) AS

FLASH_RECLAIMABLE_GB,

SUM(B.PERCENT_SPACE_USED) AS PERCENT_OF_SPACE_USED

FROM

V$RECOVERY_FILE_DEST A,

V$FLASH_RECOVERY_AREA_USAGE B

GROUP BY

SPACE_LIMIT,

SPACE_USED ,

SPACE_RECLAIMABLE ;

Query 2 :

SQL> SELECT * FROM V$RECOVERY_AREA_USAGE;

Query 3 :

SQL> SELECT trunc(first_time) DAY,

count(*) NB_SWITCHS,

trunc(count(*)*log_size/1024/1024) TOTAL_SIZE_MB,

to_char(count(*)/24,'9999.9') AVG_SWITCHS_PER_HOUR

FROM v$loghist,

101

(select avg(bytes) log_size from v$log) GROUP BY trunc(first_time),log_size;

If the available FRA space is nearly 0 in any of the instance, the redo-logs may not be shipped which

effects the functionality of ADG. So, Oracle provides 3 methods to tackle this issue.

Method 1 : Increase the FRA space

SQL> ALTER SYSTEM SET db_recovery_file_dest_size=xxG;

Where xx is the new size(should be greater than earlier value) of FRA in GigaBytes.

Method 2 : Relocate the FRA to another location

SQL> SHOW PARAMETER DB_RECOVERY_FILE_DEST

SQL> ALTER SYSTEM SET DB_RECOVERY_FILE_DEST='<New_Location>';

Method 3 : Disable the FRA policy

SQL> ALTER SYSTEM SET DB_RECOVERY_FILE_DEST='' SCOPE=BOTH SID="*";

Removing FRA logs

rman target=/

RMAN>

run {

crosscheck archivelog all;

crosscheck foreign archivelog all;

list expired archivelog all;

delete expired archivelog all;

delete noprompt archivelog all;

delete noprompt backup of database;

delete noprompt copy of database;

delete expired foreign archivelog all;

delete foreign archivelog all;

102

}

103

11.2 Testing ITNM functionality On Primary Instance :

1. Run BVT on Primary ITNM Instance

1. start ITNM and all available domains on Primary server

Result:All processes and domains started successfully

2. open DASH on browser and navigate to https://10.162.64.172:16311/ibm/console

Result : DASH Web page loaded successfully

3. enter login details

smadmin/*******

Result:successfully logged in

Log out and log in as

itnmadmin/********

Result : Successfully logged in

4. Run full discovery

Discovery->Network Discovery-> Network Discovery Status , select a DUT(Domain Under Test),

run full disco

Result : Discovery started successfully

5. Create a new NetworkView

Incident->Network Availability->Network Views create a network view for DUT

Result : Created successfully

6. Display the newly created NetworkView

Incident->Network Availability->Network Views, display network views for DUT

Result : Network view displayed successfully

8. RightClick on a device in Network View, FIND IN->Network Hop View

Result : New tab is opened and device is listed in hopview

9. RightClick on a device, SHOW DEVICE STRUCTURE

Result : Structure Browser of device is opened

2. Run Full Discovery on Primary

1. Connect to Dash as itnmadmin/********

104

2. Discovery->Network Discovery-> Network Discovery Status , select a DUT(Domain Under

Test), run full disco

Result : Devices are discovered for a given scope

3. Network polling

1. Administration->Network->Network Polling

Select a polling policy and note what are the devices polled

4. Network Views

1. Connect to Dash as itnmadmin/********

2. Incident->Network Availability->Network Views, display network views for DUT

Result: Network views are displayed for selected domain

5. Network views with user other than itnmadmin

1. Login to Dash as itnmuser/*******

2. Incident->Network Availability->Network Views

Under Libraries, create a network view of any kind

Reult : Desired network view is created.

6. Root Cause Analysis

1. Create a fake event on any one of the discovered devices using inject_fake_events.pl script

$NCHOME/precision/bin/ncp_perl

$NCHOME/precision/scripts/perl/scripts/inject_fake_events.pl -domain NCOMS -

entityNameString 172.20.1.3

2. Login to Dash as itnmadmin, Incident-> Event -> Event Viewer

3. Find 172.20.1.3, Right Click-> Show Suppressed Events

4. Note the events of devices suppressed because of 172.20.1.3

SwitchOver to Standby (Refer SwitchOver section )

After Switching Over,

105

On new primary instance :

1. Check Omnibus running

1. ps -ef | grep objserv

2. Start all itnm processes

1. navigate to /home/oracle/IBM/tivoli/netcool

cd /home/oracle/IBM/tivoli/netcool

2. set the environment variable

. env.sh

3. start itnm processes

itnm_start

4. check all the processes are running

itnm_status

3. Open Dash, check login

1. check Jazz server running

ps -ef | grep Jazz

If not, start manually

/home/oracle/IBM/JazzSM/profile/bin/startServer.sh server1

2. open https://10.162.64.172:16311/ibm/console in firefox or IE

3. login as itnmadmin/********

4. Start Discovery

1. Discovery->Network Discovery-> Network Discovery Status

select the domain, Check what are the devices listed

run full discovery

5. Network Views


Create a network view similar to that of on primary instance, display network views for

the domain

6. Network views for itnmuser

Logon to DASH as itnmuser

106


Check the presense of Network View that is created by itnmuser on Primary Instance

7. Network Hop Views

1. Incident->Network Hop View

select the domain for which disco is run, seed any device to see network hop views

8. Network Polling

1. Administration->Network->Network Polling

Select the polling policy(same as in primary instance) and note the devices polled. Cross

check them with the devices from primary list

9. Root Cause Analysis

1. Login to Dash as itnmadmin

2. Navigate to Incident->Events->Event Viewer

3. Find 172.20.1.3, Right Click->Show Supressed Events

4. Cross check the events of devides with that of before switching over

10. Network Health Dashboard


Bookmark the network view that is already created

2. Incident->Network Availability->Network Health Dashboard

Select bookmarked network view and check the health of the devices

107

11.3 Testing Failover




Members:


odb_stby - (*) Physical standby database


(*) Fast-Start Failover target

Properties:



TraceLevel = 'USER'













Target: odb_stby


108







SUCCESS

SQL> select FS_FAILOVER_STATUS,FS_FAILOVER_OBSERVER_PRESENT from

v$database;

FS_FAILOVER_STATUS FS_FAIL

------------------------------------------------

SYNCHRONIZED YES

Initiate Failover :-

To simulate a fast start failover, kill the smon proceess in primary database so that it looses the connectivity with the observer and the standby database.

(Shutting down of primary instance also will fulfill the same task)

[oracle@primno101 ~]$ ps -ef | grep smon

root 49190 49113 0 04:43 pts/4 00:00:00 grep --color=auto smon

oracle 127742 1 0 Jan25 ? 00:00:02 ora_smon_ODB

[root@primno1 home]# kill -9 127742

Killed

109

On the observer terminal, the following status will be logged…

04:46:11.80 Monday, January 28, 2019

Initiating Fast-Start Failover to database "odb_stby"...

Performing failover NOW, please wait...

Failover succeeded, new primary is "odb_stby"

04:46:27.91 Monday, January 28, 2019

110




Members:


Warning: ORA-16817: unsynchronized fast-start failover configuration

odb - (*) Physical standby database (disabled)

ORA-16661: the standby database needs to be reinstated




WARNING (status updated 4 seconds ago)

After Reiteration of lost primary database,




Members:


odb - (*) Physical standby database





111

11.4 Testing Rsync As two-way rsync is enabled, there can be two scenarios. Scenario 1 : primno1 is Primary instance

On primno1, check whether rsync is running

# ps -ef | grep rsync Expected result : Two bash scripts rsync_var_prec.bash , rsync_etc_precision.bash, rsync_nchome_precision.bash must be running.

Tail the var_prec.log file

# tail -f /home/oracle/bin/var_prec.log

Expected result : This should log the files those have been transferred to stbyno1

On stbyno1, check whether the files are getting updated as shown in the var_prec.log of primno1

Repeat the above test for rsync_etc_precision.bash, rsync_nchome_precision.bash scripts.

Scenario 2 : stbyno1 is Primary instance

On stbyno1, check whether rsync is running

# ps -ef | grep rsync Expected result : Two bash scripts Rsync_var_prec.bash rsync_etc_precision.bash,

rsync_nchome_precision.bash must be running.

Tail the var_prec.log file

# tail -f /home/oracle/bin/var_prec.log

Expected result : This should log the files those have been transferred to stbyno1

On primno1, check whether the files are getting updated as shown in the var_prec.log of stbyo1

Repeat the above test for rsync_etc_precision.bash, rsync_nchome_precision.bash scripts.

Documents

ITNM 4.2 wOracle Active DataGuard - IBM Developer · 6 3 Installation of Oracle 12c Enterprise Edition Preparing VMs: o Login to primno1.fragglerock.wipro.com(10.162.64.172) as root