Oracle GoldenGate :::::::::Overview:::::::::::
Sreenivaas BolisettyFor more info : [email protected]
Agenda
Oracle Golden Gate
Overview
Deep-dive Architecture
Demo
• Oracle GoldenGate provides real-time data integration to support high availability capabilities across heterogeneous systems to achieve business continuity.
• GoldenGate provides the ability to capture and deliver updates of critical data as the changes occur and provide continuous synchronized data across heterogeneous environments.
• Use Cases: Online banking, Claims processing and Online shopping can use GoldenGate to operate without any disruption.
Overview of Oracle Golden Gate
Overview of Golden Gate … Contd.
Log Based, Real-Time Change Data Capture
Heterogeneous Source Systems
EDWODS
EDW
Active-Active HighAvailability
Zero Downtime Migration and Upgrades
Real-time BI
Fully Active Distributed Database
ReportingDatabase
ETL
ETL
Query Offloading
Data Distribution
New DB/OS/HW/App
Global Data Centers
SOA/EDA
GoldenGate
Provides Low-impact capture Routing Transformation and Delivery of transactional data across Heterogeneous environments in Real time
Key Features
Non-intrusive, low-impact, sub-second latency
Open, modular architecture - Supports heterogeneous sources and targets
Maintains transactional integrity - Resilient against interruptions and failures
Performance
Flexible / Extensible
Reliable
Oracle GoldenGate
LAN/WANInternet
TCP/IP
CaptureTrail
Pump DeliveryTrail
Trail: stages and queues data for routing.
Pump: distributes data for routing to target(s). Route: data is compressed, encrypted for routing to target(s).
Delivery: applies data with transaction integrity, transforming the data as required.
SourceOracle & Non-OracleDatabase(s)
TargetOracle & Non-OracleDatabase(s)
Capture: Committed transactions are captured (and can be filtered) real time, from the DB Redo logs.
Oracle GoldenGate
LAN/WANInternet
TCP/IP
CaptureTrail
Pump Delivery (SQL)Trail
Trail: stages and queues data for routing.
Pump: distributes data for routing to target(s). Route: data is compressed, encrypted for routing to target(s).
Delivery: applies data with transaction integrity, transforming the data as required.
SourceOracle & Non-OracleDatabase(s)
TargetDatabase(s)Bi-directional
Capture: Committed transactions are captured (and can be filtered) real time, from the DB Redo logs.
GoldenGate Deployment Topologies
Unidirectional Query OffloadingZero-Downtime Migration
Bi-DirectionalHot Standby orActive-Active for HA
Peer-to-PeerLoad Balancing, Multi-Master
Broadcast /Fan-out Data Distribution
Integration/Consolidation/Fan-inData Warehouse
Data Distribution via Messaging
BPM
Architecture Components (Processes) • Log-based• VAM-based• Local/Remote Queuing• Filtering• Parallel Coordination• Local/Remote
• Native database API• ODBC• Batched Operations• Filtering• Transaction/Operations Grouping• Parallel Coordination• Local/Remote
CAPTURE DELIVERY
PUMP (routes from one side to other) MANAGER (Start/Stop/Report on thresholds, trail purging etc.)
• Filtering• Local/Remote• TCP/IP• Compression• Encryption
• Queue (Trail) Management• Process Management (Dynamic)• Monitoring and Administration (Lag reports)• Reporting (Events, Errors, Thresholds)• Receive/Route requests from UI
On-disk Components
• Trail Files• Checkpoint Files (Capture, Pump, Delivery)• Data Definition Files (SOURCEDEFS/TARGETDEFS)• Configuration Files• Discard Files• Report/Log Files• Executables
TargetSource
TCP/IP
Network
Process StartShared Memory (for Monitoring)
TargetDatabase
Capture DeliveryPumpSource Trail Target TrailCollector
GGSCIInteractiveInterface
Manager ManagerGGSCIInteractiveInterface
Oracle GoldenGate Components Communications and Startup
Capture
Memory Manager
Database Access – To gather Metadata (to interpret log data) & for SQL interface for data lookup, advanced filtering etc.
Redo API(Log reader interface)
Metadata Manager
SQL Interface
FilteringFormatting Transformation and Mapping
Checkpoint Manager
Transformation Engine
Serialization and Routing
Messaging Subsystem
Transaction Log Reader
Transaction Logs
GoldenGate Trail Files
GoldenGate Event Log
GoldenGate Checkpoint File
Parser
GoldenGate Parameter File
Command and Control
GoldenGate Command Interpreter
Metadata Manager
Source Database
GoldenGate Capture
Delivery
Memory Manager
Database Access – For Metadata access & SQL interface
Read API
SQL Interface
FilteringFormatting Transformation and Mapping
Checkpoint Manager
Transformation Engine
SQL Generation
Messaging Subsystem
GoldenGate Trail Reader
GoldenGate Event Log
GoldenGate Checkpoint File
Parser
GoldenGate Parameter File
GoldenGate Command Interpreter
Command and Control
Metadata Manager
GoldenGate Trail Files
Target Database
GoldenGate Delivery
GoldenGate InstantiationStart delivery at or after given backup CSN (245 in this case)
Current source Commit Sequence Number (CSN) is 356
Once Delivery is current the source and target are in sync
Capture DeliveryPumpCommit Ordered Source Trail
Commit Ordered Target Trail
SourceDatabase
TargetDatabase
CSNs 222 through 356
Deliver transactions after CSN 245
Current CSN is 356
Consistent as of CSN 356In Sync!
Take Backup as of CSN 245
Apply Backup as of CSN 245
Initial Data LoadSource:
Create the Initial data extract process ‘load1′
ADD EXTRACT load1, SOURCEISTABLE
Create Parameter file for Load1: Edit params load1
Target:
Create initial data load task ‘load2’
ADD REPLICAT load2, SPECIALRUN
Create Parameter file for Load2: Edit params load2
Start Initial Load:
START EXTRACT load1
INFO EXTRACT load1
Online Change Synchronization1. Create a GoldenGate Checkpoint table
EDIT PARAMS ./GLOBALS
GGSCHEMA GGS_OWNERCHECKPOINTTABLE GGS_OWNER.CHKPTAB
DBLOGIN USERID ggs_owner, PASSWORD ggs_ownerADD CHECKPOINTTABLE GGS_OWNER.CHKPTAB
2. Create an Extract groups on Source
ADD EXTRACT ext1, TRANLOG, BEGIN NOW
ADD EXTRACT dpump, EXTTRAILSOURCE /ora/app/stage/GoldenGate-Zip/ggs/dirdat/lt
1. Create a parameter file for the online Extract group
EDIT PARAMS ext1
2. Create a Trail on Source & Target
On the source/target system, the trail files will be created having a prefix ‘rt’/’lt’ which will be used by the Replicat/Extract processes also running on the target /source system(s)
ADD EXTTRAIL /ora/app/stage/GoldenGate-Zip/ggs/dirdat/lt, EXTRACT EXT1
ADD RMTTRAIL /ora/app/stage/GoldenGate-Zip/ggs/dirdat/rt, EXTRACT dpump
1. Create a Replicat group on the Target
1. ADD REPLICAT rep1, exttrail /ora/app/stage/GoldenGate-Zip/ggs/dirdat/rt, checkpointtable chkptab
• Create a parameter file for the online Replicat group
EDIT PARAMS rep1
Start Online Change Sync: START EXTRACT ext1
START EXTRACT dpump
START REPLICAT rep1
Significance of Data PumpWhen Data Pump is not used, the Extract process writes to a remote trail that is located on the target system using TCP/IP
When Data Pump is configured, Extract process writes to a local trail for Data Pump to read and write over the network to the remote trail located on the target system
The advantages of Data Pump can be seen as it protects against a network failure as in the absence of a storage device on the local system, the Extract process writes data into memory before the same is sent over the network
Any failures in the network could then cause the Extract process to abort (“abend”)
Additionally, if there are any complex data transformations or filtering, Data Pump can be used to perform these
Recommended
