Data guard

What Would You Do???It’s Monday morning. Fatal production system hardware problem,

or production hardware location is now a smoking hole

Expect Production to be down for an unforseen amount of time

It is year-end, you have a new manager and your senior DBA is on vacation

Jeff Slavitz - NorCal OAUG Training Day 2010 v3

What is your backup situation?

Do you have a cold backup? On-site or off-site?

Do you have a hot backup. On disk? On tape? On another accessible site? How recent is it?

Will you be able to read your tape backups?How recent are the archive logs that are now

on the system that is down?Where are your archive log backups since the

last hot backup?Jeff Slavitz - NorCal OAUG Training Day 2010 v3

OptionsRestore and clone latest backup to a test box.

How familiar is your DBA with RMAN recover? Is the box big enough to run the company? Will interfaces work (EDI, Payroll, …)

Rent or buy a box. How long would it take to find and setup a new box?

How long will it take to get the company up and running? Have you practiced this process or is this a fire drill? How many P1s will you need?

How much data will the company lose? What does the company do while Oracle is down?



How Does Your Manager Feel?

Or…In 15 minutes you’ve fully switched Oracle

Applications over to a standby databaseYou’ve tested the procedure with all DBAsThe failover procedure is fully supported and

documented by OracleInterfaces are tested on the standby databaseYou run the entire company once a month for

a short period of time on the standby databaseYou have a test standby database that DBAs

practice on and you use for reporting purposes



How Does Your Manager Feel?

My Goal TodayTell you how you can be relaxed, not stressed, in

a disaster recovery situationWhat is Data GuardDescribe added value Data Guard and Flashback

provide in addition to disaster recoveryOn a high level, review steps to implement Data

Guard and FlashbackLessons learned - what Metalink notes don’t tell

youPlease ask questions as I go alongApologies to the non-DBAs


What is Data Guard?Data Guard is part of Oracle, not a

separate product, which provides a set of services that create, maintain, manage, and monitor one or more standby databases

Data Guard maintains these standby databases as transactionally consistent copies of the production database

Data Guard can switch any standby database to the production role


Without Data Guard


Oracle® Data Guard Concepts and Administration 10g Release 2

With Data Guard


Oracle® Database High Availability Best Practices 10g Release 2 (10.2)

Why Not Use Data Guard?Too expensive – not! You already have and license it if

you have Oracle Enterprise Edition. Nothing to purchase in order to implement what we discuss today.

Too bleeding edge. Data Guard has been around since Oracle 9i and Oracle 8i as Standby Server.

Too difficult to implement. Well documented by Oracle and completely supported. This presentation is a roadmap through the process plus my own observations and experiences.

No DR box(Data Resident box). Use test box for standby. Doesn’t protect against smoking hole but it’s better than nothing.

How often do you really need DR? Is once enough? Plus a standby database can be used for other purposes.


Why You Should Use Data GuardIf your Production Oracle Applications instance

went away would anybody notice?Our job is to protect the company’s dataYour manager will thank you – DR is important.

Data Guard plus Flashback provide added

value: Instant clone Reporting instance Many exciting possibilities!


What Is Flashback?Introduced in Oracle 9iFlashback is part of your Oracle database –

no licensing requirementProvides a rewind button for your dataFlashback Database - return database to a

previous point in timeFlashback Table - return a table to a previous

point in timeFlashback Query – see data at a previous

point in timeJeff Slavitz - NorCal OAUG Training Day 2010 v3

Data Guard and Flashback TogetherMany possibilities when used together …

Scenario: Show stopper problem in ProductionNeed to test a patch or data fix with current

production data on test system ASAP

Old school: Clone Time to complete: ??? Hours

Data Guard: Use standby database as an instant clone

Time to complete: 10 minutesJeff Slavitz - NorCal OAUG Training Day 2010 v3

Instant CloneActivate standby databasePerform testing on standby databaseWhen testing complete, flashback standby

database to activation point in timeTest again? Allows destructive testing.When all done, flashback and re-activate

standby database


Reporting DatabaseScenario: You want a data warehouse which is a snapshot

of Production as of 9pm the day before. Users need access via Oracle Applications and Discoverer

Old School: Daily clone

Data Guard: Use standby database as a reporting database


Reporting DatabaseActivate standby databaseUsers login to standby database with Oracle

Applications, Discoverer, or any other tool. Users can make changes in Application though they won’t be saved after Flashback.

Archive logs accumulate on standby databaseAt 9pm each day flashback database to the way it

was at 9pm the day before, apply archive logs created since then, reopen database

Can have multiple standby databases – one for DR, one for reporting, one for ???

Oracle 11i Active Data Guard allows read-only access to standby database while it is being updated from primary


Bumps Along the WayTime implementation so it’s not at year-endBumpy transition into production – low Test

database activity is not an accurate picture of how Data Guard and Flashback will work in Production

3mb network was not enough for standby to keep up with primary - needed 10mb just for Data Guard

Do periodic full company test of standby systemData Guard and Flashback implementation are

well documented but there is a learning curveJeff Slavitz - NorCal OAUG Training Day 2010 v3

Implementing Data Guard and Flashback


Hardware Configuration


Database

Application

Primary(California)

Database

Application

Standby(Nevada)

45mb of which 8-10mb used by

Data Guard

Software ConfigurationDatabase

10.2.0.4Data Guard Physical Bundle patch 7936993Data Guard Logical Bundle patch 7937113Data Guard Broker Bundle patch 7936793

Applications 11.5.10.2


Logical vs PhysicalTwo types of standby databasesPhysical

Byte-for-byte duplicate of the primary databaseArchived redo logs transferred from primary database

are directly applied to the standby databaseWhen standby in recovery mode it is mounted not open

LogicalDifferent structure than the primary databaseOracle uses SQL statements to update standbyStandby can be open while it is being updated

Today we are talking about Physical standby databases


Maximum Protection ModeThree types of protection – Maximum

Protection is highest level of data securityPrimary database changes are not committed

until it has been confirmed that the data is available on at least one standby database

If Oracle determines that the redo data cannot be transferred from the primary server to the standby server, it will automatically stop the primary database instance

Data transmitted synchronously - network implications


Maximum Performance ModeDefault protection modeThe commit operation on the primary

database is not contingent upon the data being received by the standby server

If all of the standby servers become unavailable, processing will continue on the primary database

Data transmitted asynchronouslyThis performance mode is what are

discussing today


Maximum Availability ModeSecond highest level of data securityPrimary database changes are not committed

until it has been confirmed that the data is available on at least one standby database

If the standby database becomes unavailable for any reason, the protection mode is temporarily lowered to maximum performance until the problem has been corrected


Data Guard Installation OverviewDevelop a naming convention for your primary

and standby database servers e.g. PROD_<primary server>, PROD_<standby server>

Create standby redo logs: one more than number of redo logs, same size as redo logs, do not multiplex

Clone database to standbyCopy datafiles to standby with closed database

or with open database usingTest primary and standbyClone application to standbyYou are now ready to turn on Data Guard


Start Data GuardStart primary and second database and listenersPrimary: startupStandby: startup mountTurn on archive redo log transport on primaryalter system set log_archive_dest_state_2=enable;Put standby in recovery modeRecover managed standby database using current

logfile disconnect;Confirm logs are shipping to standbyPrimary: alter system archive log current;Standby: Check archive directory or query

v$archive_logJeff Slavitz - NorCal OAUG Training Day 2010 v3

Care and Feeding of Data GuardCheck alert log of primary and standby on a

regular basisSet Enterprise Manager to email you when an

error shows up in alert logAutomatic archive log gap resolution usually works

but network connectivity issue can require you to manually fix

Primary alert log will notify you when a gap exists:FAL[client]: Failed to request gap sequence GAP - thread 1 sequence 435-442

Copy missing archive logs from primary to standbyOn standby: alter database register logfile …


Flashback db_recovery_file_dest

defines Flash recovery area disk location This is the default location for control files, online redo logs,

archived redo logs, flashback logs, RMAN backups Archive logs are placed in flashback area in sub-directory with date DO NOT delete files in flashback area manually; database

calculates space available based on what RMAN has done db_recovery_file_dest_size

Specifies limit on the total space to be used by the flash recovery area

Check space usage using v$flash_recovery_area_usage When recovery area is full database stops!

db_flashback_retention_target Defines in minutes, how long you want to keep flashback files Since using guaranteed flashback don't need this to be too big

To turn flashback on: mount database and alter database flashback on


Switchover – Database StepsSeamless switch from primary to standbySwitch existing primary to new standby:

Shutdown applicationConfirm all logs received and applied on standbyalter database commit to switchover to standby with session

shutdown;Recover managed standby database using current logfile

disconnectSwitch existing standby to new primary:

alter database commit to switchover to primary;alter database open;alter system set log_archive_dest_state_2=enable;

Confirm logs are shipping to standby by checking alert log, archive directory or query v$archive_log


Switchover – Application StepsSome of these steps are not in Metalink note but are

required

Clear application context: exec fnd_conc_clone.setup_clean

Run adautocfg on database and application tiersRun cmclean.sql (note 134007.1) to avoid Output Post

Processing manager not startingUpdate fnd_concurrent_requests

Change logfile_name and outfile_name to new APPLCSF path

Change logfile_node_name and outfile_node_name to new host name

Update fnd_conc_req_outputs (for XML reports)Change file_name to new APPLCSF patchChange file_node_name to new host nameJeff Slavitz - NorCal OAUG Training Day

2010 v3

Switchover – Application StepsUpdate wf_notification_attributes.text_value with new

node nameUpdate wf_item_attribute_values.text value with new

node nameCheck for any other profile values that contain old node

nameClear Apache cacheStart application and send users new login urlSynchronize APPLCSF log and out filesUse rsync

Faster than scp and only copies changed filesrsync -avz

$APPLCSF/out/${TWO_TASK}*/ - applprod@lnxdrapp1:$APPLCSF/out/${TWO_TASK}*

Consider setting cron job to synchornize APPLCSF in case of system failure


FailoverImmediate switch of primary to standby systemSome archive log data may be lost due to

network latencyOn standby-soon-to-be-primary:

Cancel database recoveryalter database commit to switchover to primary

Configure application as shown in SwitchoverEven with cron job synchronizing APPLCSF log

and out files you probably won’t get all report output

Repair former primary systemJeff Slavitz - NorCal OAUG Training Day 2010 v3

Failover – Sync Old Primary as New StandbyEstablish new standby by copying primary

datafiles using RMAN, cold backup or flashback. To use flashback:Find the SCN when the existing primary

database became the primary: select to_char(standby_became_primary_scn) from

v$database;

On new standby: Startup mount flashback database to scn <above #> alter database convert to physical standby; recover managed standby database using current

logfile disconnect;Jeff Slavitz - NorCal OAUG Training Day 2010 v3

Instant Clone or Reporting DatabaseOn standby:

Cancel standby recovery

create restore point STBY_ON guarantee flashback database

Active and open database

Configure application as in Switchover Users can now login to the standby database through Oracle

Applications, Discoverer or anything.Apply patches, test, update data in standby database.To revert to restore point STBY_ON and do further testing as

needed:

Shutdown immediate

Startup mount

Flashback database to guaranteed restore point STBY_ON

Alter database openJeff Slavitz - NorCal OAUG Training Day 2010 v3

Resume Standby OperationOn Standby:Shutdown applicationShutdown databaseStartup mountFlashback database to restore point

STBY_ONAlter database convert to physical standby


Lessons LearnedScript as much as possible – you will make

typos under time pressureUse Enterprise Manager to see how Data

Guard affects your system and tune accordingly

Implement Data Guard and flashback at different times

Setup one test database using Data Guard for testing and training


For the FutureTry Maximum Availability protectionUse Enterprise Manager to manage Data

Guard. This requires Data Guard Broker implementation which requires use of spfile.

Completely script switchover and failover as shown in note 452056.1

Use RMAN to backup standby database onlyUpgrade database to 11G and experiment

with Active Data Guard. This allows you to create a read-only database that is open and is constantly updated from primary.


For More InformationGoogle (it knows everything)http://blogs.oracle.com/stevenchanNote 452056.1 – Business Continuity for Oracle Apps

Release 12 on Database 10GR2 (works fine for 11i)Note 216212.1 – Business Continuity for Oracle Apps

Release 11i, Database Releases 9i and 10GNote 753241.1 – Configuring Standby Database on R12

using RMAN Hot Backup (works fine for 11i)Note 805438.1 – How to Open Physical Standby For Read

Write Testing and FlashbackData Guard Redo Transport & Network Best Practices

Oracle 10GR2 White PaperOracle Data Guard Concepts and Administration manualHigh Availability Best Practices manualBackup and Recovery Reference manual


Questions?


Education

Data guard