Coherence GoldenGate
Oracle Coherence & GoldenGateTechnical Overview
Insert Picture Here
The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions.The development, release, and timing of any features or functionality described for Oracle’s products remain at the sole discretion of Oracle.
Agenda• Coherence Overview• GoldenGate Overview• Integration
Agenda• Coherence Overview• GoldenGate Overview• Integration
Copyright © 2013 Oracle and/or its affiliates. All rights reserved.5
The Need for an In-Memory Data GridScalability, Performance and Reliability at Internet Scale
ScalabilityPerformance
Less # hitsBetter cost
DataSupply
ApplicationServers
Web Servers
Universe of Users
Copyright © 2013 Oracle and/or its affiliates. All rights reserved.6
Modern Caching Architecture Pioneered by Coherence
First Generation Cache Solutions(still commonly seen today)
• Difficult to scale – must scale vertically only• Complex to keep cache synchronized• Often performance problems due to large
memory usage, large data sizes
Application
Collocated
Local Cache
Application
Collocated
Local Cache
• Easy to scale – scales horizontally & vertically• Clustered caching avoids synchronization issues• Can dramatically improve performance with
parallel query capabilities and compute
Coherence Cache Solutions
Application
Collocated
Local Cache
Application
Collocated
Local Cache
Data Fabric
Copyright © 2013 Oracle and/or its affiliates. All rights reserved.7
The Data Grid at a Glance
• Lower cost / Backend systems protection- Less #hits to backend systems- Match data supply with data demand by without scaling-up backends- Extreme scale-out using commodity hardware
• Enhanced performance / Better user-experience- In-memory data access – information close in memory, not in backend
systems• Enhanced availability / Better user-experience
- Reliable in-memory data grid providing high-availability to applications- Reliable transaction processing at any load
Major Benefits
Copyright © 2013 Oracle and/or its affiliates. All rights reserved.8
Oracle Coherence Through 12cPioneer & Market Leader Since 2001
2001 ….. 2008 20092000 2007 2010
Coherence
1.0
2011
Tangosol founded to
pioneer distributed
caching
Cool Vendor Award
Oracle acquires Tangosol
Coherence
3.7.1.1
Coherence
3.6
Coherence
3.5
Coherence
3.7
Increasing market share and feature capabilities for In Memory Data Grid. Market widened All Categories – Financial Services, E-Commerce, Travel, Insurance,
Retail, Manufacturing, Telco …
Deep adoption in Fusion Middleware, Oracle Horizontal and Vertical and Business Applications and Exalogic
2012 2013
Coherence
12.1.2
Alignment with Fusion Middleware
Releases
Copyright © 2013 Oracle and/or its affiliates. All rights reserved.9
Oracle Coherence DefinedDistributed in-Memory Object-Oriented Data Grid
No cost per object assembly
Object – Oriented
Distributed Grid / Clustered Caching
• Identified by a key (key-value pair)• Any serializable object • No proprietary classes to extend• Not forced to use relational models, ORMs, SQL • Fully native Java, .NET , C++ interoperability
KEY
• Data distribution > massive scale-out• Backup replicas > High availability• No single points of failure (SPoF)• No single points of bottleneck (SPoB)
Copyright © 2013 Oracle and/or its affiliates. All rights reserved.10
Coherence Clustered Caching ExplainedPartitioned, Fault Tolerant, Self-Healing Cache
• All nodes verify health of each other• In the event a node is unhealthy, other nodes diagnose
state?
• Unhealthy node isolated from cluster• Remaining nodes redistribute primary and back-up
responsibilities to healthy nodes
X
• Cluster of nodes holding % of primary data locally
• Back-up of primary data is distributed across all other nodes
• Logical view of all data from any node
Copyright © 2013 Oracle and/or its affiliates. All rights reserved.11
Data Distribution Options
• Data is load balanced across the data grid• Data and processing capacity scales linearly • Ownership responsibilities also partitioned• Access and update latency are constant
Partitioned Caching > Scalability
Copyright © 2013 Oracle and/or its affiliates. All rights reserved.12
Data Distribution Options
• Automatically manages the fault tolerance of your data• Backups guaranteed to be on separate node / machine /
rack / site from primary• Backup for data owned by one node is shared across the
others
Partitioned Caching w/ Backup Replicas > High Availability
Backup Storage
Primary Storage
Copyright © 2013 Oracle and/or its affiliates. All rights reserved.13
Data Distribution Options
• Storage enabled grid members > Cache Servers• Storage disabled grid members > Cache Clients
Cache Server vs. Cache Client > Application Protection
Application Application Application Cache Clients
Cache Servers
Copyright © 2013 Oracle and/or its affiliates. All rights reserved.14
Data Distribution Options
• 2-level cache• Most recently/frequently used data is stored locally
Near Caching > Enhanced Performance
Cache Clients
Cache Servers
Application Application Application
Front Local Cache
Back Distributed Cache
Copyright © 2013 Oracle and/or its affiliates. All rights reserved.15
Querying The Data grid
• Programmatic & SQL-like query mechanism• Queries performed in parallel across the grid• Index support for better performance• Aggregation support
Parallel Queries & Aggregation
Application
Copyright © 2013 Oracle and/or its affiliates. All rights reserved.16
Querying The Data grid
• Snapshot automatically updated as back entries change• Event – based
Continuous Query Cache
Application
Copyright © 2013 Oracle and/or its affiliates. All rights reserved.17
Event Notifications
• JavaBean Model (addListener), key and filter based events• Real-time notifications• Full-state & Lite events
Real Time Event Driven Architectures
Application
!
Application
!
Application
!
Copyright © 2013 Oracle and/or its affiliates. All rights reserved.18
Processing on the grid
• Sends the processing to where the data is in the grid- Once and only once guarantees / automatically fault-tolerant
• Key(s) and filter based processing
In-place Data Processing and Map/Reduce Aggregation
Application
ProcessingUnit
Copyright © 2013 Oracle and/or its affiliates. All rights reserved.19
Managing PersistenceCaching Patterns
• Cache Aside – Architecture manages cache‒ Check the cache before reading from data source
‒ Put data into cache after reading from data source
‒ Evict or update cache when updating data source Cache
DAO
CacheDAO
CacheDAO
• Cache On Top – All data reads/writes occur through cache‒ Read Through/Write Through
• Cache miss causes load from data source automatically
• Updates to cache written synchronously to the data source
‒ Write Behind
• Updates to cache written asynchronously to the data source
Copyright © 2013 Oracle and/or its affiliates. All rights reserved.20
Use casesCachingReduced latency / Backend (DB, Mainframe) downsizing
Reliability / ScalabilityData Grid acts as a transactional system of record, hosting data and business logic
Parallel Computing & EventsParallel queries and processing / Automated processing based on event
InteroperabiltiyData exchange between heterogeneous apps (Java, .Net, C++)
Shared contextsHTTP Session externalization / Data consolidation (profiles)
Copyright © 2013 Oracle and/or its affiliates. All rights reserved.21
Join the Coherence Communityhttp://coherence.oracle.com
@OracleCoherence
facebook.com/OracleCoherence
blogs.oracle.com/OracleCoherence
Oracle Coherence Users
youtube.com/OracleCoherence
coherence.oracle.com/display/CSIGCoherence Special Interest Group
Agenda• Coherence Overview• GoldenGate Overview• Integration
Copyright © 2013 Oracle and/or its affiliates. All rights reserved.23
Oracle GoldenGate Defined
• Oracle GoldenGate is a real-time, log-based change data capture and replication software platform
• The software provides:- Capture- Routing- Transformation- Delivery
• of transactional data across heterogeneous data sources in real time
No-Impact, Real-Time Data Integration & Transactional Replication
Copyright © 2013 Oracle and/or its affiliates. All rights reserved.24
The Need for a Change Data Capture SystemCapturing Changes and Applying them Throughout the Enterprise
New DB/HW/OS/APP
Fully Active Distributed DB
Reporting Database
Data Warehouse
Global Data Centers
Zero Downtime Upgrade & Migration
Query OffloadingMainframe Offloading
Data Synch Across the Enterprise
Real-time BI, Operational Reporting, MDM
Event Driven Architecture, SOA
Highly Available / Disaster Recovery
Log-based,
changed data
Database
Legacy
Message Bus
HW Migration
Mainframe modernization
Mainframe downsizing
Web specialization
Active - Active
Disaster Recovery
Exadata
Copyright © 2013 Oracle and/or its affiliates. All rights reserved.25
SourceOracle & Non-Oracle DBs
TargetOracle & Non-Oracle DBs
Capture: committed transactions are captured (and can be filtered) as they occur by reading the transaction logs.
How Oracle GoldenGate Works
Copyright © 2013 Oracle and/or its affiliates. All rights reserved.26
Capture: committed transactions are captured (and can be filtered) as they occur by reading the transaction logs.
Trail: stages and queues data for routing.
How Oracle GoldenGate Works
SourceOracle & Non-Oracle DBs
TargetOracle & Non-Oracle DBs
Copyright © 2013 Oracle and/or its affiliates. All rights reserved.27
Capture: committed transactions are captured (and can be filtered) as they occur by reading the transaction logs.
Trail: stages and queues data for routing.
Pump: distributes data for routing to target(s).
How Oracle GoldenGate Works
SourceOracle & Non-Oracle DBs
TargetOracle & Non-Oracle DBs
Copyright © 2013 Oracle and/or its affiliates. All rights reserved.28
Capture: committed transactions are captured (and can be filtered) as they occur by reading the transaction logs.
Trail: stages and queues data for routing.
Pump: distributes data for routing to target(s).
Route: data is compressed, encrypted for routing to target(s).
How Oracle GoldenGate Works
SourceOracle & Non-Oracle DBs
TargetOracle & Non-Oracle DBs
Copyright © 2013 Oracle and/or its affiliates. All rights reserved.29
Capture: committed transactions are captured (and can be filtered) as they occur by reading the transaction logs.
Trail: stages and queues data for routing.
Pump: distributes data for routing to target(s).
Route: data is compressed, encrypted for routing to target(s).
Delivery: applies data with transaction integrity, transforming the data as required.
How Oracle GoldenGate Works
SourceOracle & Non-Oracle DBs
TargetOracle & Non-Oracle DBs
Copyright © 2013 Oracle and/or its affiliates. All rights reserved.30
Capture: committed transactions are captured (and can be filtered) as they occur by reading the transaction logs.
Trail: stages and queues data for routing.
Pump: distributes data for routing to target(s).
Route: data is compressed, encrypted for routing to target(s).
Delivery: applies data with transaction integrity, transforming the data as required.
Bi-directional
How Oracle GoldenGate Works
SourceOracle & Non-Oracle DBs
TargetOracle & Non-Oracle DBs
Copyright © 2013 Oracle and/or its affiliates. All rights reserved.31
UnidirectionalQuery Offloading
Zero-Downtime Migration
Bi-DirectionalHot Standby or
Active-Active for HA
Peer-to-PeerLoad Balancing
Multi-Master
BroadcastData Distribution
Integration/ConsolidationData Warehouse
BPM
BAM
OEP
Data Distributionvia Messaging
Differentiator: FlexibilityOne-to-One, One-to-Many, Many-to-One, Many-to-Many
Java
Copyright © 2013 Oracle and/or its affiliates. All rights reserved.32
Oracle GoldenGateDifferentiators
• Zero-impact capture, routing, transformation, and delivery of transactional data
• Sub-second latency for data movement
• Open, modular architecture• Supports heterogeneous sources and targets• One-to-one, one-to-many, many-to-many, many-to-one
• Maintains transactional integrity• Resilient against interruptions and failures• Fast Recovery after process interruptions.
• Conflict resolution in active-active configurations• Filtering, mapping between different data models• Simple transformations• Encryption, compression,…
Performance
Flexible & Extendable
Reliable
Logical replication
Copyright © 2013 Oracle and/or its affiliates. All rights reserved.33
Oracle GoldenGate
• Filtering & “Compression”- Unnecessary change data is discarded immediately- Updates and deletes are “compressed” by default- Before images are discarded by default
• Transformations and mappings - Within either the Capture or Delivery modules - no middle tier server is
needed - Built-in functions, PL/SQL, custom functions (user-exits)
• Flexible topology support- Variety of topologies, one-to-one, one-to-many, many-to-one, many-to-
many, and cascading and bidirectional configurations.
Differentiators
Copyright © 2013 Oracle and/or its affiliates. All rights reserved.34
Oracle GoldenGate
• Bidirectional configuration support- Oracle GoldenGate enables both active-passive and active-active solutions
for maximum system availability. • Conflict detection and resolution
- Implemented globally, object by object, based on data values and complex filters, or through event-driven criteria.
• Routing and compression - Oracle GoldenGate uses TCP/IP for sending data, so no geographical
distance constraints are imposed.• Security
- Data encryption (trail files, data sent)
Differentiators
Copyright © 2013 Oracle and/or its affiliates. All rights reserved.35
Oracle Golden Gate
• Transaction integrity- Transaction boundaries recorded in GoldenGate trail file- Checkpointing and recovery are based on transaction boundaries- Original commit sequence is maintained by the delivery process
• Resiliency & fast recovery- Individual processes can be restarted automatically- Tolerance to network outages (configurable)- Recovery ensures that no operations are skipped or duplicated after failure
of any kind
Differentiators
Copyright © 2013 Oracle and/or its affiliates. All rights reserved.36 36
Oracle GoldenGate Supported PlatformsDatabases O/S and Platforms
Oracle GoldenGate Capture and Delivery: Oracle DB2 Microsoft SQL Server Sybase ASE Teradata Enscribe SQL/MP SQL/MX MySQL v 5.5 JMS message queues
Oracle GoldenGate Delivery: All listed above, plus:TimesTen, Postgres Netezza, Greenplum, HP Neoview ETL products
Linux
Sun Solaris v11
Windows 2000, 2003, XP
HP NonStop
HP-UX
IBM AIX v7.1
IBM z Series
zLinux
IBM i Series
Copyright © 2013 Oracle and/or its affiliates. All rights reserved.37
Use Cases
New DB/HW/OS/APP
Fully Active Distributed DB
Reporting Database
Data Warehouse
Global Data Centers
Zero Downtime Upgrade & Migration
Query OffloadingMainframe Offloading
Data Synch Across the Enterprise
Real-time BI, Operational Reporting, MDM
Event Driven Architecture, SOA
Highly Available / Disaster Recovery
Log-based,
changed data
Database
Legacy
Message Bus
HW Migration
Mainframe modernization
Mainframe downsizing
Web specialization
Active - Active
Disaster Recovery
Exadata
Copyright © 2013 Oracle and/or its affiliates. All rights reserved.38
Join the GoldenGate Communityhttp://www.oracle.com/technetwork/middleware/goldengate/overview/index.html
twitter.com/ORCLGoldenGate
facebook.com/OracleDataIntegration
blogs.oracle.com/dataintegration
Oracle Data Integration
youtube.com/oraclegoldengate
Agenda• Coherence Overview• GoldenGate Overview• Integration
Copyright © 2013 Oracle and/or its affiliates. All rights reserved.40
Coherence HotCacheSolve the Stale Cache Problem
Coherence Application
Coherence Application
3rd Party Application Read / Write
Read / Write
HotCache
Copyright © 2013 Oracle and/or its affiliates. All rights reserved.41
Coherence HotCache
• Propagate database changes to cache in real time• Leverage existing technologies
- TopLink Grid- GoldenGate
Solve the Stale Cache Problem
• Event-driven• Low latency• No application
changes
Copyright © 2013 Oracle and/or its affiliates. All rights reserved.42
GoldenGate as a Coherence FeedTurn the Database Into an Event Source
Read / Write
HotCache
3rd PartyApplications
Batch Process