Database-Migration and -Upgrade with Transportable Tablespaces

Basel · Baden Bern · Brugg · Lausanne Zurich Düsseldorf · Frankfurt/M. · Freiburg i. Br. Hamburg · Munich · Stuttgart · Vienna

Database-Migration and –Upgrade using Transportable Tablespaces –report from a running project

Markus Flechtner

TechEvent April 2010

© 20102

Agenda

§ The setting

§ Why Transportable Tablespaces?

§ The TTS Approach

§ Tuning TTS

§ Problems and challenges

§ Outlook

Life, the universe and everything.

Database-Migration and -Upgrade using Transportable Tablespaces

© 2010Database-Migration and -Upgrade using Transportable Tablespaces 3

The setting (1) – customer situation

§ Our customer is one of the largest telecommunications equipment suppliers in the world

§ They develop and sell hardware for cellular networks (GSM)

§ And they sell a „network performance management & capacity management software“

§ A lot of mobile network operators worldwide use this software


The setting (2) – the infrastructure

§ Current platform of the end customer:ú Server Hardware: PA-RISC ú Server OS: HP-UX ú Oracle-Software: 10.2.0.4ú File Storage: Filesystem

... But the PA-RISC processor architecture will be desupported in the near future and they have to migrate to a new platform

§ Target platform of the end customer is:ú Server Hardware: Intelú Server OS: Red Hat Linux 64bitú Oracle-Software: 11.2ú RACú File Storage: ASM

© 2010Database-Migration and –Upgrade using Transportable Tablespaces 5

The setting (3) – the project

§ First customer for data migration is the largest customer of the application:ú 4 databases, about 50.000 tables per databaseú Each DB Server: 32 CPUs, 96 GB RAMú DB Size between 3.2 and 5.5 TB, up to 1 TB redo log amount/dayú Databases running in NOARCHIVELOG-mode (performance!)

ú New SANú No chance to connect the old SAN to the new serversú Just a network connection available (Gigabit ethernet)

ú RAC with 42 nodes planned

§ Requirement: Maximum downtime of the application of 5 – 6 hours!

© 2010Database-Migration and –Upgrade using Transportable Tablespaces 6

The setting (4) - summary

§ Large databases, running in NOARCHIVELOG-Mode

§ Platform change HP-UX à Linux

§ Oracle Upgrade: 10.2 à 11.2

§ From single instance to RAC

§ From file system to ASM

§ Limited downtime

§ Experiences from the current project will be used for future customers


Agenda

§ The setting


§ The TTS Approach

§ Tuning TTS


§ Outlook

There's more than one way to skin a cat …


Why Transportable Tablespaces (TTS)?

§ Exp/imp and Datapump è Performance?

§ Tools like „Golden Gate“ or „Quest Shareplex“?è Additional license costs

§ Streams? è ARCHIVELOG-mode required.

§ Data Guard / Physical Standby è Not applicable due to platform change from HP-UX à Linux

§ Replication è Materialized views on 50.000 tables?

§ Self-made methods had bad performance


Agenda

§ The setting


§ The TTS Approach

§ Tuning TTS


§ Outlook

Data is always part of the game.


The TTS approach (1) - Overview

§ „Transportable Tablespaces“ are used to transfer data between databases by copying the data files and transfering the corresponding metadata

§ Starting with Oracle 10g, this works even between different operating systems (e.g. from HP-UX to Linux), changing the „endianess“ of a datafile is possible using rman

§ It is a well known procedure


The TTS approach (2) – Specials

§ The set of tablespaces which is transferred must be „self-contained“:

ú no reference to another database object in a tablespace outside the tablespace set

ú This can be checked in advance and is usually no problem when transferring a complete database

• Sequences, synonyms, temporary tables, privileges, objects owned by SYSTEM or placed in the SYSTEM-tablespace etc. must be transferred separately


The TTS approach (3) – The procedure (simplified)

1. Create an "empty" target database

2. Set the tablespaces to be transferred in „READ ONLY“ mode

3. Export the metadata using Oracle export or Oracle Datapump („which objects are in the tablespaces?“)

4. Transfer the export file and the datafiles to the target system.- If necessary, convert the data files using rman

5. Import the metadata (import or Datapump)

6. Copy additional objects like sequences, synonyms etc. using Datapump via Network Link

7. Set the tablespaces „READ WRITE“ in the target database


The TTS approach (4) – File Conversion

§ rman is used to change the endianess of the files and to move the files from the file system into ASM

ú Big Endianess: HP-UX, Solaris (Sparc), AIX, Linux on zOS or PowerPC

ú Little Endianess: Windows, Linux (Intel-based), VMS

rman> CONVERT DATAFILE 'copy_of_db_file_in_filesystem' FROM PLATFORM 'HP-UX (64-bit)'TO PLATFORM 'Linux x86 64-bit'FORMAT 'file_location_in_ASM'PARALLELISM 4;


Agenda

§ The setting


§ The TTS Approach

§ Tuning TTS


§ Outlook

You cannot increase the speed of light.


Tuning TTS (1) - Overview

§ Resize the files to the High Water Mark before the transferú Less data to be read from the source systemú Less data to be transferred via networkú Less data to be converted using rman

§ Compression (for data transfer)

§ Parallelizationú For all operations where possible (background jobs)ú For transfer & conversion: transfer and convert multiple files in

parallel

§ Automation (using scripts)


Tuning TTS (2) – Reduce File Size

§ For all datafiles involved (pseudocode):

§ ORA-3297 "file contains used data beyond requested RESIZE value"

§ Resize can be done "online" during normal business

§ Estimated result: Reduction of the database size to about 85 –90 % of original size J

§ Files will be resized in target database after transfer

FOR ALL DATAFILESSET AUTOEXTEND ON MAXSIZE 32G;REPEAT

RESIZE FILE TO 95% of FILE_SIZE;UNTIL ORA-3297;


Tuning TTS (3) – Compression

§ Facts:ú When transfering a lot of large database files via network, network

bandwidth can be a bottleneckú Using a simple „gzip“, typical database files can be compressed to

less than 20% of their original size

Þ Why not use a „compressed transfer“?

§ Like:ssh source_server "gzip remote_file –c"|gunzip >local_file


Tuning TTS (4) – Compression

Theoretical network bandwidth between old DB-Server and new RAC-System

1 Gigabit/second

This is about 450 GB/hourLet‘s take 75% of this theoretical value 337 GB/hour

Database Size (largest DB, without resize) 5.500 GBWithout SYSTEM- , SYSAUX- and UNDO-Tablespace 5.000 GBResized to 90 % original size 4.500 GBCompressed to approx. 25% of original size 1.125 GB

Theoretical duration of the „compressed transfer“ 3,4 hours+ metadata export/import


Tuning TTS (5) – Parallelization

§ Level 1:ú Transferring and converting multiple files to one node in parallel

Source-Server(32 CPU)

One RAC node Target Database(on ASM)

File transfer(ssh / gzip)

Conversion(rman)


Tuning TTS (6) – Parallelization

§ Level 2:ú Let's use the RAC and use all nodes for transfer & conversion

Source-Server(32 CPU)

RAC nodes Target Database(on ASM)

File transfer(ssh / gzip)

Conversion(rman)


Tuning TTS (7) – Automation

§ Basic database migration using TTS is a well-documented straight-forward operation which can be scripted

§ One RAC node is the "target master" which coordinates the work and starts all the scripts on the other nodes

§ Communication between the nodes and the scripts is done via a table in the target database

ú contains information on file transfer statusú contains information on tablespace transfer status (metadata)ú files which could not be copied or converted using rman get an „ERROR“-

status and have to be transferred again


Agenda

§ The setting


§ The TTS Approach

§ Tuning TTS


§ Outlook

It‘s not as simple as that!


Problems and challenges (1) – Metadata Export

§ In the lab exporting the metadata for the whole database using DataPump took 12 hours

§ è Oracle said: "Use Original Export"

§ With "Original Export", it took just 6 hours J

§ But running the metadata export using "Original Export" in another database resulted in an:EXP-0008 / ORA-29336: Internal error … from DBMS_PLUGTS

§ è Oracle said: "Use DataPump"

§ L



§ Solution: Create self-contained subsets of the tablespaces and export them in parallel using datapump

§ Preparation (Pseudocode):

§ The "grouping operation" can be very time-consuming (450+ tablespaces), but can be executed during normal operation

REPEATCREATE A NEW GROUP;ADD TABLESPACE TO NEW GROUP;

REPEATCHECK TS-GROUP FOR SELFCONTAINMENT(DBMS_TTS.TRANSPORT_CHECK_SET)IF THERE ARE ROWS IN SYS.TRANSPORT_SET_VIOLATIONS

ADD THESE TABLESPACES TO GROUP;UNTIL SYS.TRANSPORT_SET_VIOLATIONS IS EMPTY;

UNTIL ALL TABLESPACES ARE ASSIGNED TO A GROUP;



§ The „grouping operation“ resulted in about 120 small tablespaces groups

§ These tablespace groups are grouped into larger "metagroups" and every target node transfers a given number of metagroups:

ú 1. export metadata (in Background)ú 2. transfer and convert datafiles (multiple jobs in parallel)ú 3. import metadata

§ It takes just about 1,5 hours for all export jobs to complete J

§ Status information is stored in the database


Problems and challenges (4) – Job Coordination

§ Coordination of all the jobs between by the "target master"-nodeú Keeping control over all the parallel background jobs on source

server, RAC nodes, etc.ú Communication via database table

§ The number of parallel transfer jobs is limited by:ú CPU power on source ú available disk space on the RAC nodesèjob monitoring needed to avoid overloading or space problems

§ Error handling / Restart


Agenda

§ The setting


§ The TTS Approach

§ Tuning TTS


§ Outlook

What has the future in store for us?


Outlook (1) – The Procedure

§ The project is still ongoing ..

§ The procedure works J

§ The lab is too small for performance tests:ú Target cluster has 4 nodes with 2 Quadcores and 32 GB RAMú But source server has just 2 CPUs and 8 GB RAMú Running more than 8 transfer jobs in parallel (2 per RAC node) kills

the source server

§ (End-)customer has a larger server (8 CPUs), but no RAC at the moment L

§ Performance not tested so far …


Outlook (2) – The Plan

§ We're pretty confident that the procedure will work in production

§ For the small databases (2 databases with less than 4 TB) we'll use the TTS-approach only

§ Depending on the results with these databases, we can change the strategy for the larger databases:

ú 1. Use an offline backup of the database (volume split) as a source for a TTS-based data transfer

ú 2. Use an already existing self-developed replication based on the timestamps in the records for the remaining data

§ Plan B: Use "Golden Gate"


References

§ Oracle MAA-Whitepaper „Platform Migration using Transportable Tablespaces“http://www.oracle.com/technology/deploy/availability/pdf/maa_wp_11g_platformmigrationtts.pdf

§ Oracle MAA-Whitepaper „Database Upgrade using Transportable Tablespaceshttp://www.oracle.com/technology/deploy/availability/pdf/maa_wp_11g_upgradetts.pdf

§ "Minimum Downtime Upgrade with Transportable Tablespaces"http://blogs.oracle.com/UPDATE/

§ "Moving Oracle Databases Across Platforms without Export/Import"http://www.dbspecialists.com/files/presentations/changing_platforms.ppt

Basel · Baden Bern · Brugg · Lausanne Zurich Düsseldorf · Frankfurt/M. · Freiburg i. Br. Hamburg · Munich · Stuttgart · Vienna

Thanks for your attention! .. any questions?

Technology

Database-Migration and -Upgrade with Transportable Tablespaces