Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Near-Zero Downtime Database Migration Case Study Presented by Starwood Hotels &

Oracle Confidential – Internal/Restricted/Highly RestrictedCopyright © 2014, Oracle and/or its affiliates. All rights reserved. |

Near-Zero Downtime Database Migration

Case Study Presented by Starwood Hotels & Oracle Advanced Customer Support

Harish Patel DBA Lead, AccentureNalin Sahoo Sr. Principal Advanced Support EngineerShampa Mukhopadhyay Sr. Technical Account ManagerOracle Advanced Customer SupportOctober 02, 2014

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Oracle Confidential – Internal/Restricted/Highly Restricted 3

Safe Harbor StatementThe following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.


Agenda

Oracle Advanced Customer Support

Migration Planning and Analysis

Migration Process

Lesson Learned

Summary

Q&A

1

2

3

4

5

6


Agenda



Migration Process

Lesson Learned

Summary

Q&A

1

2

3

4

5

6


Oracle Advanced Customer Support (ACS) delivers personalized and proactive mission-critical support for organizations seeking to maximize the availability, performance, and value of their Oracle solutions

6


About Oracle Advanced Customer Support

We Operate Globally as part of Oracle Customer Support Services

We Bring Expertise and Experience across the Complete Oracle Stack

ACS Works Closely with Oracle Development to Enhance Supportability

Our #1 Focus is Customer Success

7

#1

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | 8

Global 50 Governments by GDP 84%Global Fortune 100 Companies

Telecommunications Companies

Banks

High Tech Companies

Auto Companies

Aerospace and Defense Companies

82%5 of top 5

4 of top 5

5 of top 5

4 of top 5

5 of top 5ACS works with 4,000+

public and private sector organizations

Oracle Advanced Customer Support: Our Customers


Oracle Advanced Customer Support: What We Help You Achieve

MAXIMUM AVAILABILITY MAXIMUM PERFORMANCE & VALUE

9


Supportability at Every Step of Your Lifecycle

Plan & Design

Build & Deploy

Support &Maintain

Optimize &Modernize


Oracle Advanced Customer Support Offerings

Personalized and Proactive Support from a Named Teamof Oracle Engineers

Priority Service Request Handling for Your Most Critical Oracle Solutions

Oracle On-Site Support

Advanced Support Engineers at Your Site to Provide Immediate Assistance

Oracle SystemsOptimization

Support

Startup, Go-live, Tuning, Hardware and Software Changes, and More

Oracle AdvancedSupport Cloud

Services

Connectivity Enables a Wide Range of EngineeredSupport Offerings

Oracle PrioritySupport

Oracle SolutionSupport Center

1 2 3 4 5


Starwood at a glance

~ 1200 HOTELS & RESORTSACROSS 9 RENOWNED BRANDS

Nearly 1,200 properties worldwide in 100 countries

Starwood doubled its global luxury footprint over the past six years

181,400 employees at its owned and managed properties

In 2013, signed 152 new hotel agreements, the highest number since before the global economic crisis

FORTUNE 500 COMPANY


Oracle and Starwood• Oracle Advanced Customer Support

– Worked with Starwood for over 4 years

– Solution Support Center

• Introducing Starwood– Harish Patel, DBA Team Lead, Accenture

– Arup Nanda, Principal Database Architect, Starwood


Agenda



Migration Process

Lesson Learned

Summary

Q&A

1

2

3

4

5

6


Database Migration Planning & Analysis:

~150+ Oracle Databases – OLTP & DWDW - 11gR2 on ExadataGoldenGate Implementations – 11.2.xEnterprise Manager

15

Oracle Database Footprint at Starwood


Source DB : 4 node 10G RAC on HP ItaniumTarget DB : 4 node 11G R2 RAC on AIX P-series

Houston Data Center Phoenix Data Center

16

Cross-Platform, Multi-Version Database move

Target 4 Node 11G R2 on AIX

Source 4 Node 10G R2 on HP Itanium




OLTP system for reservation across multiple channel including internet

Production Database NDS on 4-node 10gR2 RAC on HP-UX

High transaction volume, 14 TB databases – high revenue of Billions

24X7 Business Critical Systems

17

Business Critical Global Reservation System



NDS DB is the heart of Starwood’s global reservation systemZero data loss tolerance. Cannot have downtime for migrationAny outage has a significant financial impact Once migrated, cannot revert to old systemsThe Migration is very complex

– Cross-platform (HP Itanium to AIX)– Multiple Versions Upgrade (10G R2 to 11G R2)– Geographic distance of data centers (Houston to Phoenix)

Any transaction taking place on 10g infrastructure must get propagated to target 11g systems during migration until cut-over

18

Challenges for Zero Downtime Migration



Upgrade planning and analysis Analyze & gather information about environment Review documentation, MOS Notes, White Papers & Known Issues Determine upgrade path Prepare strategy for testing and downgrade

Migration Prepare for database migration Establish performance baseline/metrics before upgrade Migrate database Test migrated database with applications and reports Ensure adequate performance by comparing metrics gathered before and after upgrade

Post-Migration Support Monitor and tune Work with Oracle Support on issues

Cross-Platform, Multi-Version Database move


Migration Planning & Analysis: Upgrade & Migration Options

Stay on same OS?N Y

UPGRADE/MIGRATIONexp/imp, expdp/impdp

Oracle Golden Gate

DBUA

Downtime >30 min ? Inline Upgrade

Y

NSQL Apply &

Transient Standby

CLI

$> catctl.pl

Full Transportable Export/Import

Transportable Tablespaces

CTAS, COPY, SQL Loader


Agenda



Migration Process

Lesson Learned

Summary

Q&A

2

3

4

5

6

1


Database Migration ProcessEvaluation of Available Options

exp/imp, expdp/impdp

Oracle GoldenGate•Downtime < 120 min•Cross Platforms must be handled

Transportable Tablespaces

CTAS, COPY, SQL Loader

Oracle GoldenGateTransportable Tablespaces

Standby Database RMAN Bkp/Restore

Two Conditions to Evaluate


Database Migration Process:

Data Movement Options for initial copy of 14 TB Starwood decision:

• Use licensed EMC Business Continuance Volume(BCV) ~ 24 hours• SRDF BCV copy to Destination ~ 72 hours

Database Conversion Options Many options evaluated vs X-TTSStarwood decision:

• Cross-Platform Transportable Tablespace

Incremental Update/Synchronization of Data Starwood decision:

• GoldenGate

Final selection of Method of Migration



Create BCV copy of production 14 TB 4 node RAC database running on HP server in Houston

Split BCV copy in Houston and SRDF to intermediate HP server in PhoenixRecover 10G Database on the intermediate HP server in PhoenixPrepare 11g RAC database on target IBM AIX serversPerform trasportable tablespace migration from HP to AIXPerform post migration tasks to get pl/sql objects, types, views, synonyms and other

objectsValidate system performanceCapture incremental changes in existing production and then apply changes to target

database and cutover when ready

High Level Steps A.K.A The Blueprint


Apps Servers120 Middleware Servers

OGG 11G R2 RAC on AIX

BCV COPY

10G R2 RAC on HP

Database Migration Process:Database Migration of NDS Database – Houston to Phoenix

Oracle GoldenGate

Apps Servers300 Middleware servers

New version of Websphere

SRDF

PhoenixHouston

Intermediate HP Server


Database Migration Process:Migration, Application Workload & Performance Testing

HP-UX Itanium AIX 64bit

PROD

HP-UX Itanium

IntermediateServer

Migration , Application workload testing ,

DB replay and SQL Performance Analyzer

Prod Load

R2

DESTINATION

R2

X-PlatformTTS

R2



Concept / High level steps: Start capturing changes on source Create an upgraded copy on target Start applying changes to target Once in sync, monitor and maintain a lag of 15 minutes Cut-over gracefully after all captured transactions are applied

Synchronization using GoldenGate


Oracle 10.2.0.4HP-UX Itanium

14 TB

Oracle 11.2.0.3AIX 64bit

NEW

14 TB

Extract Trail Pump Trail Replicat

Database Migration Process:Synchronization using GoldenGate (continued)



Is the Migration successful? Make sure all tablespaces are available at destination Make sure all datafiles are available and online

Did we miss any objects in the migration process?

With help of meta data export/ import, all objects were made available into Target DatabaseObject comparison by schema was performed

Can Applications run on new System?

Functional testing was performed by APP teams to validate applications.

Can new DB take the same app load / Perform adequately?

SQL Performance Analyzer was utilized to identify regressed SQLs and corrective actions were taken to improve performance.RAT- DB capture and replay were utilized to determine transactional load handling capacity

Is the new Database ready for GO-Live?

UAT was performed by Application teams for final approval of GO-LIVE

Check-Points for Validation



Gracefully Stop the Applications on Source DBSwitch log files repeatedly to capture / apply changesAllow 30 minutes to completely drain the trail files by replicate processOnce in sync, allow 45 minutes for app teams to validate data on both

source and destinationDuring this 45 minutes DBAs will complete the sequence number

adjustment and enable any constraint which was disabled in processOnce, GO-LIVE approval is obtained, start the services on Target Database

and allow business traffic

Cut-Over Tasks


Agenda



Migration Process

Lesson Learned

Summary

Q&A

1

3

5

6

2

4


Lesson Learned:

Test Upgrade • The application runs correctly?

- Application, queries, report performance degradation issue gather statistics, application rewrite

Real application load testing • SQL Performance Analyzer – Regression in execution?• Database replay issues

- Preprocessing of large capture files- ORA-979 error because 11g syntax change- XA transaction cannot be replayed

Millions+ transaction replayed

During Validation Phase


Lesson Learned:

Go-live • Yes ! Easier than expected• Total upgrade and migration time: As expected

Overall upgrade • As per the test migration/upgrade

Post Go-Live Issues • Parallel degree - Automatic Degree of Parallelism feature. Limit maximum number of

parallel processes using Resource Manager• SQL performance issue

Used SQL profile for plan stability

• Oracle bug related instance eviction during DRM/synchronization• Conflict of automatic statistic gather job with user scheduled statistics collection job.

During Post Go-live


Agenda



Migration Process

Lesson Learned

Summary

Q&A

1

3

4

6

2

5


SummaryNot all best practices may be applicable to your environment. Also, they

are not a substitute for planning, preparation, and testingMajority of time spent during an upgrade project should be in planning,

preparation, and testingEnsure test database is identical to production – in terms of data volume

and distributionShorten test cycle by using Real Application Testing featuresTest upgrade process completely before upgrading production

– Capture lessons learned from upgrade testingEngage Oracle Support immediately upon encountering an issue

– Oracle ACS can help you in all phases of upgrade to Oracle 11g or 12c!


Agenda



Migration Process

Lesson Learned

Summary

Q&A

1

3

4

5

2

6


Next StepsContact Oracle AdvancedCustomer Support Services

http://www.oracle.com/acs [email protected] Q&A

Documents

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Near-Zero Downtime Database Migration Case Study Presented by Starwood Hotels &