From 20 hours to 20 minutes Oracle e-business cloning using Oracle Data Guard

ORACLE E-Business Suite DBA { APPS DBA } onsite at Medscheme

Target Audience• Oracle APPS DBAs• Oracle DBAs• Technical Managers• Oracle Sysadmins• Disaster Recovery planners• Technical Project Managers• Technology architects'

Technology Used

Current setup

Archives Shipped To DGuard

Production Instance Dev Instance

What is Active DataGuard

• The ability to use your Physical Standby for much more than just High Availability or Disaster Recovery!

Backup done to tape

3 Hours to restore from Tape

Sent to Bryanston

Implementation of Data Guard and Oracle RMAN

Archives shipped to Data Guard

Archives backed

up to Disk

- Archives backed up every 30 minutes

– Database backup every

Friday

New Business Requirement / Audit finding

• Ebus support team must not have access to production environment

• Oracle Developers must not have access to production data

• Only directors and managers can have access to sensitive data

• Sysadmin login should be controlled

Requirement

• Create a exact copy of Production daily• Scramble Salary information• Close all open workflows• Reset all database passwords { system,dbsnmp}• Reset application passwords { apps, applsys}• Reset Sysadmin password• Set workflow over-ride address• Reset functional support team passwords• No budget for this !!!!

• Create a exact copy of Production daily using dataGuard as an engine

IDEA was born !!!

Mirror image of Production

Requirements for daily copy • Every patch which is applied to Production is applied TEST { Strict change

control}• ORACLE HOMES between the servers are at the same patch levels• We do not copy log and out files { $APPLCSF }• All custom forms and reports are copied both to TST and Production

environment• If a Data file is added on the dbTier on the Target then you would have to

recreate the xml file on the dbTier on Test environment• Printers which are configured on production must be configured using the

same name on development servers• Not tested on Windows • Initial clone is done on the source system to create the xml file using

Cloning Oracle Applications Release 12 with Rapid Clone [ID 406982.1]

Datafiles

Log and out {APPLCSF}

Below is a diagram on the components and how they are refreshed onsite

Step 1 Application Server shutdown

Step 2 Shutdown Dataguard instance#### SHUTDOWN DATAGUARD FOR EBUS CLONING ###########00 1 * * * /home/oradg/refresh/shutdownDG.sh > /home/oradg/refresh/log/shutdownDG.log################ STARTUP DATAGUARD FOR EBUS CLONING ################################15 2 * * * /home/oradg/refresh/startDG.sh > /home/oradg/refresh/log/startupDG.logoradg@ebusdb:~> more /home/oradg/refresh/shutdownDG.sh

##### Running environment file #####################################. /home/oradg/env_DG.sh

########### Shuting down the listener ##############################

lsnrctl stop dg

######## Shutting down the DB ######################################

sqlplus '/as sysdba' << 0xffselect DB_UNIQUE_NAME from v\$database;recover managed standby database cancel;shutdown immediate;exit0xffdatedate

##### DATAGUARD ENVIRONMENT IS SHUTDOWN ##########################oradg@ebusdb:~>

Step 3 Database Tier05 1 * * * /home/oratst/refresh/clone.sh > /home/oratst/refresh/log/Dailyclone-$(date +\%Y\%m\%d_\%H:\%M:\%S).log 2>&1oratst@ebusdb:~> more /home/oratst/refresh/clone.sh. /home/oratst/env_tstdb.shexport ORACLE_HOME=/tstdb001/11.2/TSTexport ORACLE_SID=tstexport TNS_ADMIN=/tstdb001/11.2/TST/network/admin/tst_ebusdbexport PATH=$PATH:$ORACLE_HOME/binecho CLEANING OUT THE INVENTORYexport PERL5LIB=/tstdb001/11.2/TST/perl/lib/site_perl/5.10.0:/tstdb001/11.2/TST/appsutil/perlrm -rf /home/oratst/oraInventory/*rm -rf /tstdb001/11.2/TST/appsutil.bakrm -rf /tstdb001/11.2/TST/admin/tst_ebusdb/diag/rdbms/tst/tst/incident/*> /tstdb001/11.2/TST/admin/tst_ebusdb/diag/tnslsnr/ebusdb/tst/trace/tst.logrm /tstdb001/11.2/TST/log/diag/tnslsnr/ebusdb/tst/alert/*.xmlrm /tstdb001/11.2/TST/admin/tst_ebusdb/diag/rdbms/tst/tst/trace/*.trcrm /tstdb001/11.2/TST/admin/tst_ebusdb/diag/rdbms/tst/tst/trace/*.trmrm /tstdb001/11.2/TST/admin/tst_ebusdb/diag/rdbms/tst/tst/trace/alert_tst.logrm /tstdb001/11.2/TST/admin/tst_ebusdb/diag/rdbms/tst/tst/trace/*.outrm /tstdb001/11.2/TST/appsutil/tst_ebusdb.xml.bak*rm /admin/utl_file_dir/tst/*tmp

Log file cleanup

##### KILLING LOCAL PROCESSES #######echo KILLING LOCAL PROCESSESdate. /home/oratst/.profilekill_local > /home/oratst/refresh/log/kill_local.shchmod 777 /home/oratst/refresh/log/kill_local.sh. /home/oratst/refresh/log/kill_local.shdate#### SHUTING DOWN LISTENER #########echo SHUTING DOWN LISTENERcd /tstdb001/11.2/TST/bin./lsnrctl stop tst#### SHUTING DOWN DATABASE ###############echo SHUTING DOWN DATABASEcd /tstdb001/11.2/TST/bin./sqlplus '/as sysdba' << 0xffselect SYSDATE,NAME from v\$database;shutdown immediate;exit0xffdate################## DELETING DATAFILES ########echo DELETING DATAFILES 1rm /tstdb001/oradata/*.dbfecho DELETING DATAFILES 2rm /tstdb002/oradata/*.dbfecho DELETING DATAFILES 3rm /tstdb003/oradata/*.dbfecho DELETING DATAFILES 4rm /tstdb004/oradata/*.dbfecho DATAFILES DELETEDdate

########## COPING DATAFILES ##################echo COPY DATAFILES 1cp /dg001/oradata/*.dbf /tstdb001/oradataecho COPY DATAFILES 2cp /dg002/oradata/*.dbf /tstdb002/oradataecho COPY DATAFILES 3cp /dg003/oradata/*.dbf /tstdb003/oradataecho COPY DATAFILES 4cp /dg004/oradata/*.dbf /tstdb004/oradataecho DATAFILES COPY DONEdatedate

########## DOING CLONE ###################echo STARTING DBTIER CLONEcd /tstdb001/11.2/TST/appsutil/clone/binperl adcfgclone.pl dbTier /home/oratst/refresh/contextfile/tst_ebusdb.xml << EOFappsEOFdate########echo REFRESH OF PRODUCTION DONE

Step 3 Start Dataguard instance#### SHUTDOWN DATAGUARD FOR EBUS CLONING ###########00 1 * * * /home/oradg/refresh/shutdownDG.sh > /home/oradg/refresh/log/shutdownDG.log################ STARTUP DATAGUARD FOR EBUS CLONING ################################15 2 * * * /home/oradg/refresh/startDG.sh > /home/oradg/refresh/log/startupDG.logoradg@ebusdb:~> more /home/oradg/refresh/startDG.sh

##### Running environment file #####################################. /home/oradg/env_DG.sh

########### Shuting down the listener ##############################

lsnrctl start dg

######## Shutting down the DB ######################################

sqlplus '/as sysdba' << 0xffstartup nomount;alter database mount standby database; alter database open readonly;recover managed standby database using current logfile disconnect;select DB_UNIQUE_NAME from v\$database;exit0xffdate

##### DATAGUARD ENVIRONMENT IS UP ##########################oradg@ebusdb:~>

Step 4 Appstier00 4 * * * /home/appltst/refresh/cloneapps.sh > /home/appltst/refresh/log/CLONE-APPS--$(date +\%Y\%m\%d_\%H:\%M:\%S).log 2>&1appltst@ebusapp:/tstapp001/tstapp/inst/apps/tst_ebusapp/admin/scripts> more /home/appltst/refresh/cloneapps.shecho CLEARING THE INVENTORYrm -rf /home/appltst/oraInventory/*echo STARTING APPS CLONEcd /tstapp001/tstapp/apps/apps_st/comn/clone/binperl adcfgclone.pl appsTier /home/appltst/refresh/contextfile/tst_ebusapp.xml << EOFappsnEOFdateecho APPS CLONE DONEecho CHANGING SYSADMIN PASSWORD. /tstapp001/tstapp/apps/apps_st/appl/APPStst_ebusapp.envcd /tstapp001/tstapp/apps/apps_st/appl/fnd/12.0.0/bin./FNDCPASS apps/apps 0 Y system/apps USER SYSADMIN appsecho SYSADMIN PASSWORD CHANGED TO apps./FNDCPASS apps/apps 0 Y system/apps USER MEDSUSER appsecho MEDSUSER PASSWORD CHANGED TO appsecho ALL CLONE OF PRODUCTION DONEecho Clearing the CACHErm /tstapp001/tstapp/inst/apps/tst_ebusapp/logs/ora/10.1.2/reports/cache/*dateecho Cache Doneecho Clearing APPLTMPrm -rf /tstapp001/tstapp/inst/apps/tst_ebusapp/appltmp/*echo Clearing APPLTMP donedateecho ALL CLONE OF PRODUCTION DONE

appltst@ebusapp:/tstapp001/tstapp/inst/apps/tst_ebusapp/admin/scripts> more /home/appltst/refresh/changeappspassword.sh. /tstapp001/tstapp/apps/apps_st/appl/APPStst_ebusapp.envcd /tstapp001/tstapp/apps/apps_st/appl/fnd/12.0.0/bin./FNDCPASS apps/apps0 Y system/apps ORACLE APPLSYSPUB PUB./FNDCPASS apps/apps 0 Y system/apps SYSTEM APPLSYS appsecho CHANGED APPS PASSWORDecho CHANGED FILEecho START APPS SERVER. /tstapp001/tstapp/apps/apps_st/appl/APPStst_ebusapp.envcd /tstapp001/tstapp/inst/apps/tst_ebusapp/admin/scriptscp /home/appltst/refresh/customtst_ebusapp.env /tstapp001/tstapp/apps/apps_st/appl./adautocfg.sh << EOFappsEOFecho AUTOCONFIG COMPLETED SUCCESSFULLYsleep 180./adstrtal.sh apps/appsecho ALL STARTEDdateecho deleting old log filescd /tstapp001/tstapp/inst/apps/tst_ebusapp/logs/appl/conc/logfind /tstapp001/tstapp/inst/apps/tst_ebusapp/logs/appl/conc/log -mtime +1 -exec rm -f {} \;dateecho deleting old out filescd /tstapp001/tstapp/inst/apps/tst_ebusapp/logs/appl/conc/outfind /tstapp001/tstapp/inst/apps/tst_ebusapp/logs/appl/conc/out -mtime +1 -exec rm -f {} \;dateecho cleaning out Apache log filesfind /tstapp001/tstapp/inst/apps/tst_ebusapp/logs/ora/10.1.3/Apache -mtime +1 -exec rm -f {} \;dateecho CLONE COMPLETEDappltst@ebusapp:/tstapp001/tstapp/inst/apps/tst_ebusapp/admin/scripts>

Log and out file cleanup

Additional steps

• Close all workflowsUpdate apps.wf_notificationsSet mail_status='SENT',status='CLOSED'where mail_status='MAIL'and status='OPEN';commit;

• Close all alertsupdate apps.alr_alertsset enabled_flag='N'where enabled_flag='Y'and upper(alert_name) like 'MEDS%';Update apps. fnd_concurrent_requestsset phase_code = 'C',status_code = 'D'where concurrent_program_id in (select concurrent_program_idfrom apps.fnd_concurrent_programswhere concurrent_program_name in ('ALECDC', 'ALECTC'));

• Set override addressupdate fnd_user set email_address = 'rynod@medscheme.co.za';update wf_local_roles set email_address = 'rynod@medscheme.co.za';commit;

Scramble Salary informationupdate peR_pay_proposals set proposed_salary_n = ROUND((proposed_salary_n/assignment_id)*4732,2);

Updating Site ParameterDECLAREstat boolean;BEGINdbms_output.disable;dbms_output.enable(100000);stat := FND_PROFILE.SAVE('SITENAME', '****DAILY CLONE OF PRODUCTION****TST******DAILY CLONE OF PRODUCTION***Data from '||to_char(trunc(sysdate)-1,'DD-MON-YYYY')||'(12:00AM)*****TST**', 'SITE');IF stat THENdbms_output.put_line( 'Stat = TRUE - profile updated' );ELSEdbms_output.put_line( 'Stat = FALSE - profile NOT updated' );END IF;commit;END;/

We have noticed the following cost saving to business

• No requirements of tapes• No requirements for assistance of Linux admin• No requirements for restores• No human interaction , only to check logs• No functional support or super users on production• Time saved from 24 working hours to 20 minutes• DR available immediately• DataGuard environment is read only which means it can be used for reporting• Production Servers are up and running 24 X 7 X 365• Clones to DEV, UAT, Training and dba environments takes 1 hour as clones on local servers• Users and support team can test anything at anytime• Copy of production available 7 days a week.• Backup of production can be verified by dataguard.• Rman backups can be run from the DataGuard instance.• No need to purchase expensive software for cloning.• Daily copy of production which functional support team can use• Assistance with OWC with Oracle as scripts and updates can be tested with production data immediately• We noticed a reduction in TARS as support team could test solutions from metalink• Patches can be tested immediately using Production data• No more P1 SRs• Super users can test changes to the system • Demos can be given to Directors using latest data• Training can be given with data from Production• Disaster recovery environment is available immediately• Changes can be made on TEST system and user acceptance testing can take place easy• Emergency changes can be tested• No additional money was spent on this solution

We have noticed the following cost saving to business

• No requirements of tapes• No requirements for assistance of Linux admin• No requirements for restores• No human interaction , only to check logs• No functional support or super users on production• Time saved from 24 working hours to 20 minutes• DR available immediately• DataGuard environment is read only which means it can be used for reporting• Production Servers are up and running 24 X 7 X 365• Clones to DEV, UAT, Training and dba environments takes 1 hour as clones on local servers• Users and support team can test anything at anytime• Copy of production available 7 days a week.• Backup of production can be verified by dataguard.• Rman backups can be run from the DataGuard instance.• No need to purchase expensive software for cloning.• Daily copy of production which functional support team can use• Assistance with OWC with Oracle as scripts and updates can be tested with production data immediately• We noticed a reduction in TARS as support team could test solutions from metalink• Patches can be tested immediately using Production data• No more P1 SRs• Super users can test changes to the system • Demos can be given to Directors using latest data• Training can be given with data from Production• Disaster recovery environment is available immediately• Changes can be made on TEST system and user acceptance testing can take place easy• Emergency changes can be tested• No additional money was spent on this solution

NO ADDITONAL

COST !!!!

Backups c

an

be test

ed

daily

Disaster Recovery• Shutdown DataGuard

– Stop the listener {lsnrctl stop dg}– Stop DataGuard { recover managed standby database cancel; }– Shutdown the database {shutdown immediate; }– Copy the datafiles to any instance.

• On the DB server run post-clone– cd $ORACLE_HOME/appsutil/clone/bin { taking into account you know the locations}– perl adcfgclone.pl dbTier { configure your dbTier}

• On your Apps Server { we normally use our TST environment or have a dedicated disk }– cd $COMMON_TOP/clone/bin { taking into account you know the locations}– perl adcfgclone.pl appsTier

• Users can login onto your DR/ DataGuard environment• Update your log and out files locations update apps.fnd_concurrent_requests set logfile_node_name = <New APPS SERVER>, outfile_node_name = <New APPS SERVER> where logfile_node_name = <OLD APPS SERVER> and outfile_node_name = <OLD APPS SERVER>;

Database CleanupFiles which can be deleted

Space Used – 12 GB

APPS Server CleanupFiles which can be deleted

Application – 34 GB

Summary of Environment

What does functional support team say !

Due to audit requirements that none of support staff are allowed to have Production access, this solution has been ‘forced’ upon us with actually very positive results:1) We’re still able to support users timeously2) We’re able to replicate Production issues with up-to-date data and configuration without any risk to Production

Shounese Khan HR Functional Specialist Oracle

•Assistance can be given the next day if people are having trouble with a specific transaction on self-service / professional forms as the data would have been replicated and no need to wait for a scheduled refresh.•You can log on as the user to confirm the error they have and assist then to resolve it (this could be to get print screens to show them where to click ect…)

Wayne WrightOracle System Administrator

We are also using this environment for post-implementation testing – in cases where testing requires that transactions need to be created, this environment is ideal as no Production data is compromised

Shalantha RampersadOracle HR Developer

Ryno DurrheimFinance Functional Support

Juan FerreiraOracle Finance Developer

Alwin WeirSenior Manager: Corporate IT Systems and Data Warehouse

The benefit that we have is that we can now experiment (play around) with the latest records when problems are experienced by users.

TST gives me the ability to work with “real” data and cases, and find proper solutions that almost guarantees flawless implementations to PROD.

The Daily Clone has allowed us to meet our audit requirements has we now have the ability to remove all supports staff’s access to production and they can use the daily clone for most of the support issues.

Fortunate & FrikkieMedscheme Linux Administrators

Johann LaurensSenior Manager: Infrastructure

The old process took extremely long and had a lot of manual steps. After the new process was implemented we don’t even know when the APPS DBAs are cloning. All is done in the background, we no more work overtime on development environmentsSaving us time to concentrate on production hardware supportmade our life's easier

My team { APPS DBAs & linux admins } can now concentrate on Production support and supporting Medscheme users instead of spending time cloning development environments. This new process has freed our resources for business critical tasks

Mohomed SulimanSenior Manager : Payroll

Tshivhiya MulaudziSystems and Support Specialist HRMS | Human Capital Shared Services

We use TST on a daily basis for any issues we come across on PROD that we first want to test out and it is great that TST has the updated information from PROD. We are then easily able to resolve payroll run, elements, formula, leave queries or personal record issues. We have also found benefit using TST where we can run tax year end processes and payroll runs to test out new patch functionalities as well as new element builds and formula results using the data. We find the daily clone of PROD to be very valuable for the Payroll department.

We use TST to replicate the problem which we experience in production and try to fix it on TST first before we do it on production. I use it for Performance Management, I-Recruitment and Learning Management ,Employee Self Server and Manager Self Service

Tabita VenterPerformance Consultant | HC - Centre of Excellence

TST is very helpful and has enabled us to assist our end users with their system queries – logging into a system and being able to see what they see. It has also been very useful in testing setup changes and confirming minimal impact on the system. We also make use of it for training purposes, where users can log in and go and test the different performance processes, before working on the live environment.

I used TST to assist with my up skilling on Purchase order training when it was handed over from Yolande Mathews, I have since used TST for training sessions to train PA’s on the purchase order process. We also use it to train new staff on the employee self service component- on TST they are given an opportunity to explore the navigation and practice how to use the applicable update functionality on Oracle employee self service.

Marius DreyerGeneral Manager : Technology

By applying Oracle’s active Data Guard solution we’ve solved key issues we had in the past both from a recovery, usability and audit perspective. The automated solution deployed by the IT Technology DBA team has decreased the refresh turnaround time to such and extend that it has really become a non-issue for us. We’ve solved real business issues/requirements with no additional costs apart from the Data Guard license costs.

The customized automated clone has saved us many repetitive man hours. With system and human errors likely to occur with the previous manual cloning, this new process guarantees our environment is available and accessible on time every day

Questions & Answers

Feedback