© 2
011
Amad
eus
IT G
roup
SA
Brighter, Bolder, Better
© 2
011
Amad
eus
IT G
roup
SA
Brighter, Bolder, Better
1
Amadeus Journey toOracle Grid Infrastructure 11gR2
© 2
011
Amad
eus
IT G
roup
SA
Brighter, Bolder, Better
2
Edgar Canaan System Programmer
Franz Fischer Project Manager
Vitor Pacheco Manager
Nicolae Hachi DBA Manager
© 2
011
Amad
eus
IT G
roup
SA
Brighter, Bolder, Better
3 2010 movie following the well known internet series “ Did you know? …”
© 2
011
Amad
eus
IT G
roup
SA
Brighter, Bolder, Better
4
The largest transactionprocessor in travel
Common / overlapping platforms & applications
Common data centre
Common customers
Common sales & marketing infrastructure
DISTRIBUTION BUSINESS
Provision of indirect distribution
services
713 airlines (over 436 bookable)
116 Insurance companies
50+ cruise and ferry lines
204+ tour operators
87,000+ hotel properties
29 car rental companies
103 railways
Travel providers
Consumers/General public
Corporate traveldepartments
Travel buyers
Travel agencies
Travel Management companiesBusiness travel agenciesLeisure travel agenciesOnline travel agencies
Consolidators Single-site agency
Travel search companiesAirline sale offices and airline websites
connected to Amadeus direct sell technology
IT SOLUTIONSIncluding direct distribution
technology
Transaction-based business model Operate globally in the growing travel and technology market Two highly synergistic and profitable businesses: Distribution and IT Solutions
© 2
011
Amad
eus
IT G
roup
SA
Brighter, Bolder, Better
© 2
011
Amad
eus
IT G
roup
SA
Brighter, Bolder, Better
5
© 2
011
Amad
eus
IT G
roup
SA
Brighter, Bolder, Better
Amadeus technology is present across the entire journey
7
Traveler checks-in and receives boarding pass
Traveler receives notification via mobile that return flight schedule has changed
Action Travel agency / traveler searches for best air fare and hotel content
Traveler needs to change their travel date after ticket has been issued
Travel agency/ Traveler makes air and hotel bookings and issues air e-ticket/ hotel voucher paying with credit card
Altéa Inventory
Altéa Departure Control System
Solutions for mobile applications
Amadeus technology
Availability & shopping technology
TA front office Altéa Reservation Revenue maximization
(dynamic availability, journey data)
Self-booking tool for corporations
Ticket Changer GDS booking
Altéa e-commerce
E-ticketing & voucher
Credit card authorization
Sales management solution
Revenue integrity
JourneyTrip planning & pricing
Booking & ticketing On the moveCheck-in & boardingChange ticketing Post-trip
Traveler completes travel expense report
Travel agency issues invoice and reconciles travel to invoiced data
Travel provider / travel agency conducts revenue management and accounting activities
© 2
011
Amad
eus
IT G
roup
SA
Brighter, Bolder, Better
© 2
011
Amad
eus
IT G
roup
SA
Brighter, Bolder, Better
8
UXs3 ProjectUniX stack 3 Qualification
© 2
011
Amad
eus
IT G
roup
SA
Brighter, Bolder, Better
Objectives
Validate and introduce a new software stack (s3/s4) for the Oracle DB tier Operating System:
HP-UX V3 – s3 RedHat 5.7 – s4
Oracle GI (ASM & Oracle Clusterware 11gR2) Database Engine: Oracle 11gR2
RAC (services Active/Passive) – s3 RAC One Node – s4
Migration of all existing Oracle Databases to the new stack in both RAC and RAC One Node configurations
Prepare foundation for future DB Tier platform on Linux x-86
9
© 2
011
Amad
eus
IT G
roup
SA
Brighter, Bolder, Better
Project Motivation
Reduce the number of vendors & complexity of two cluster layers (Oracle and Symantec)
Avoid resulting problems with validation, integration, patching, upgrading, change / problem & incident management
Cost avoidance by reducing Symantec licensing costs Oracle product lifecycle compliance (10.2.0.3 off prime support) Oracle announcement for discontinuation of support of Itanium Green IT (reduce growth of Data Center power & cooling
requirements)
10
© 2
011
Amad
eus
IT G
roup
SA
Brighter, Bolder, Better
11
Scope
Validation of the new s3/s4 stack Migration strategy customized for different use cases
RAC, RAC One Node, HP-UX, Linux etc. Different migration windows - 15 minutes or 30 minutes
Integration of the new s3/s4 stack with spread & thin provisioning technologies Address increasing storage costs
Train operational & systems teams on Oracle 11gR2 GI Ensure operational readiness of support processes for Oracle
11gR2 GI
© 2
011
Amad
eus
IT G
roup
SA
Brighter, Bolder, BetterTimeline overview
12
Migration Phase HP-UX -> HP-UX
Migration Phase HP-UX -> Linux RAC One Node
Migration Phase HP-UX -> Linux RAC
System Qualification
Funct.Qual. Part1
Funct.Qual. Part2
Funct.Qual. Part2
Migration Phase Linux -> Linux RAC One Node
IP NW Resiliency
Server Disk Access & SAN
DB RAC Server HA
MW Porting to 11g
GG PoC
OAQ’s dependency removal /Extension of the migration process
RAT
Patch Strategy
Backup
RDBMS Feat. Eval.
Test Refresh
APF
SecurityConcept
DBMonitoring
Migr. Strategy
ZBO
ADPStats
OEM GC
PlanStability
DR Replication
EHA
May11
Jan11 UXs3 Project
May10
Sep10
Sep11
Jan12
May12
Sep12
Jan13
May13
1st D
eploym
ent
1st M
igration
© 2
011
Amad
eus
IT G
roup
SA
Brighter, Bolder, Better
Main challenges
Short time frame 32 months for validation and
migration ~ 70 production Databases ~ 8 test databases for each Prod DB
Expectations Every team has
3 different wishes, 3 different constraints
End customer base distributed across the globe
Development vs. Operations targets SW functionality vs. high availability
and stability Marketing milestone
Alignment with ongoing Projects Technical Customer driven
Capacity Requirements Double capacity during migration Budget constraints HW Delivery and provisioning
processes
13
© 2
011
Amad
eus
IT G
roup
SA
Brighter, Bolder, Better
© 2
011
Amad
eus
IT G
roup
SA
Brighter, Bolder, Better
14
System Infrastructure
© 2
011
Amad
eus
IT G
roup
SA
Brighter, Bolder, BetterUNIX OBE –May 2011 global figures
Application: 83 applications (with 15 being DB
management related) 569 Oracle DBs 105 MySQL DBs
Cluster: 63 clusters 764 Service Groups 6333 Resources
Servers: 25 SuperDomes 82 HP-UX partitions 71 Linux servers (rack mounted)
Storage (estimated growth of 4TB/day): 53 disk arrays 650 TB of Filesystem and ASM
volumes 1.1 PB of DG 2.1 PB of raw storage (estimation
considering we are mainly R1 and adding BCV/clone space)
10K LUNs
15
© 2
011
Amad
eus
IT G
roup
SA
Brighter, Bolder, BetterSystems Infrastructure – building blocs
16
Oracle11gR2 GI
HPUXv3or
RHEL 5…
1A lib
Oracle Clusterware
S-Checkmonitoring
EMC DMX 4 or VMax
XiV
1A libASM ASM
MPIOOS
MPIOOS
node 1 node n
VOMSRM
VCSOne‘orchestration framework’
10 GbE
FC
© 2
011
Amad
eus
IT G
roup
SA
Brighter, Bolder, BetterSystems Infrastructure – RAC One Node scale-out approach
17
© 2
011
Amad
eus
IT G
roup
SA
Brighter, Bolder, Better
Future DB platform – IB with RDS and iSer for low latency transport
18
© 2
011
Amad
eus
IT G
roup
SA
Brighter, Bolder, Better
1Alib highlights
Making sure Storage, Systems and DBA teams refer to the same storage component.
Quickly troubleshooting hot spots is cumbersome without meaningful name common to all three groups (DBA, UNIX, SAN) when issues arise.
Provisioning Storage safely with an enterprise unique schema based on World Wide Identifier (WWUID)
Repurposing LUNs can be a risky task. Interaction with DBAs and SAN Administrator a must.
19
© 2
011
Amad
eus
IT G
roup
SA
Brighter, Bolder, Better
1Alib:: iostat
22More 1Alib command examples in the Backup Slides section
© 2
011
Amad
eus
IT G
roup
SA
Brighter, Bolder, BetterSystems Feedback
Still no RHEL 6 support. No thin provisioning awareness. Lack of official white papers with architecture and concept
documentation. No possibility for 3rd party off host mirroring integration like EMC TF
BCV/CLONE. No specific performance report 'a la AWR' for GI components (also
AWR should report in microseconds and not milliseconds for I/O related statistics)
Patching is complex and lengthy. No POSIX conformant interface.
25
© 2
011
Amad
eus
IT G
roup
SA
Brighter, Bolder, Better
Application flows
Application access to the DB performed through Amadeus own middleware Simple validation of Middleware
Client upgrade decoupled from the Server upgrade We maintain 10gR2 client for 2 months past the migration
Plan stability enforced through baselines Depends on plan completeness in the source environment Attention to deprecated hints in 11g – i.e. “ordered”
Application specific qualification done as part of the pre-migration validation Qualification to start up to 3 months prior to migration date
27
© 2
011
Amad
eus
IT G
roup
SA
Brighter, Bolder, Better
Backup and RecoveryCurrent backup solution
RMAN with Flashback Recovery Area Integration with Backup server
Tivoly Data Protection for Oracle Complemented by
Data Guard in Maximum Performance Mode Second DB in primary site
Data Guard in Archiver transport mode Disaster Recovery
EMC Clone Off host backup solution Not available in the new 11gR2 environment
Alternative being studied29
© 2
011
Amad
eus
IT G
roup
SA
Brighter, Bolder, Better
Amadeus Test Systems
30
OBO
MIG
OBO clone
PDT
UAT SKL
OBO clone
RMAN restore
Data guardPPT
DES
DEV
FVT
RMAN restore PHY STBY
BTPRD QRT
QAT
AP1W
W
3*D 3*D
4W4W4W
8W 8W8W
W
OnReqOnReq
Split mirror from OBO
Split mirror from OBO
Full disk image
Clone (copy on write)
OnReqInSyncWithPRD
© 2
011
Amad
eus
IT G
roup
SA
Brighter, Bolder, Better
Amadeus Test Systems
31
OBO
MIG
OBO clone
PDT
UAT SKL
OBO clone
RMAN restore
Data guardPPT
DES
DEV
FVT
RMAN restore PHY STBY
BTPRD QRT
QAT
AP1
Split mirror from OBO
Split mirror from OBO
Full disk image
Clone (copy on write)
Up to 10 Test Systems per Application Virtualization as a mean to minimize Storage investments Short outage windows Large DB volume – up to 12 TB
Sum of all Test: 1.2 PB Storage snaps on write not yet available in ASM Symantec Volume Manager will be maintained in most of the Test
environments
© 2
011
Amad
eus
IT G
roup
SA
Brighter, Bolder, Better
ASM
Symantec Volume Manager
Amadeus Test Systems
32
OBO
MIG
OBO clone
PDT
UAT SKL
OBO clone
RMAN restore
Data guardPPT
DES
DEV
FVT
RMAN restore PHY STBY
BTPRD QRT
QAT
AP1
Split mirror from OBO
Split mirror from OBO
Full disk image
Clone (copy on write)
© 2
011
Amad
eus
IT G
roup
SA
Brighter, Bolder, Better
© 2
011
Amad
eus
IT G
roup
SA
Brighter, Bolder, Better
33
Migration Scenarios
© 2
011
Amad
eus
IT G
roup
SA
Brighter, Bolder, Better
10g-11g Migration alternatives
Data Guard Positives:
Robust technology Used in the 9i-10g migration Extensive experience Process industrialized Consistency confidence
Negatives Oracle 11gR2 does not install in HP-
UX V2 No fallback possible after migration
Upgrade duration in the range of 15+ min
Not possible cross endian Data dictionary persistency No reorganization possible (soft)
GoldenGate Positives
Cross platform No need to upgrade during outage
window = short outage window Physical reorganization is possible
Negatives Complex and effort driven Technology bugs Difficult to industrialize Consistency between the two DB’s
needs to be evaluated regularly Licensing costs
34
1 2
© 2
011
Amad
eus
IT G
roup
SA
Brighter, Bolder, Better
10g-11g Migration alternatives
Functional migration Positives
No down time: products migrated one by one
No extra license costs Application exposed to 11g
progressively No big-bang approach
Negatives Development effort (very hard) Long coexistence of 10g and
11g (soft)
Data Pump Positives
Rehearse effort is small Simple process
Negatives Long duration for DB’s larger
than 10GB Constraint can be addressed
using low latency technology
35
3 4
© 2
011
Amad
eus
IT G
roup
SA
Brighter, Bolder, Better
36
Amadeus Planned Outages
Amadeus classifies its services in 3 different categories Mission Critical Application
Target: continuous availability Planned outages: aligned to Quarterly Outage Windows (QOW)
Mission Critical for Reservations Target: 99.99% availability Planned outages: QOW – quarterly 15 min
Other Target: 99.95% availability Planed outages: Monthly Maintenance Window – monthly 30 min
© 2
011
Amad
eus
IT G
roup
SA
Brighter, Bolder, Better
37
Migration strategy
Linux-to-Linux Data Guard Migration
HP-UX to Linux Option 1: Data pump, if respective outage window is acceptable Option 2: GoldenGate
HP-UX to HP-UX Option 1: GoldenGate – in particular to MCA’s Option 2: Data Guard – if no fallback is acceptable
1
24
21
© 2
011
Amad
eus
IT G
roup
SA
Brighter, Bolder, Better
Golden GateMain Challenges
Activation of Supplemental Logging on source DB Dealing with ‘resource busy’ situations for highly accessed tables Additional amount of generated redo
Handling of unsupported data types Ex. dedicated handling of Oracle Advanced Queues
Limitation of effort / risk related to DDL replication Avoiding of DB Loads during active GG processes Disabling of batch jobs during active GG processes
Awareness and/or handling of all data model ‘specialties’, like Unused columns, Triggers, Cascading deletes, Missing Primary / Unique
Key Dedicated GG configuration needed per application
Complex qualification: 4 test runs for each Application Long preparation and test phase - 8 to 12 weeks duration
A series of new builds were required to fix an IOT related bug
38
© 2
011
Amad
eus
IT G
roup
SA
Brighter, Bolder, Better
39
Golden GateTuning Challenge
Special performance based GG configuration required for DBs with high data modification rate Parallel GG processes for
Identification of independent sets of tables
Splitting table replication ondata range level
Various GG tuning parameters DB based tuning
(e.g. creation of unique keys)
Veridata Configuration Comparison of large amount of data in a reasonable time
and with a limited amount of resources Comparison of databases with on-going data changes
© 2
011
Amad
eus
IT G
roup
SA
Brighter, Bolder, Better
Conclusions 1/2
40
11gR2 RDBMS part
Not a major transition Good toolkit
Grid Infrastructure Major evolution from previous Oracle versions Toolkit not yet there
IO Performance metrics not fulfilling our expectation
Migration is the biggest challenge High effort demand in the DBA teams
UXs3 Project Project extended to the end of 2012 New stacks included
S5 – RedHat 6.x, GI 11gR2, RAC (services Active/Passive) S6 – RedHat 6.x, GI 11gR2, Low latency, RAC (services
Active/Active)
© 2
011
Amad
eus
IT G
roup
SA
Brighter, Bolder, Better
Conclusions 2/2
41
What we have achieved until now New stack s3 and s4 qualified
S3 completed by May 2011 S4 completed by August 2011
4 Production DB’s migrated to the s3 stack 2 migrations using Golden Gate 3 migrations achieved in less than 10 minutes
5 new DB’s deployed natively in the new s3 stack S4 platform ready for Production deployment
Ongoing Application migration qualification First migrations to the s4 stack planned for the 13th of
November
© 2
011
Amad
eus
IT G
roup
SA
Brighter, Bolder, Better
Special thanks
Oracle RAC Pack Team Martina Keippel Sanjay Singh Erich Kreisler
Oracle ACS Florian Zahneisen and his
team Oracle Germany
Klaus Thielen Wolfgang Thiem Rogee Biondi
42